Parallelizing Scripts - Revision history

Neteler: fix broken URLs

2023-12-27T19:50:01Z

fix broken URLs

← Older revision		Revision as of 19:50, 27 December 2023
Line 50:		Line 50:
	</source>		</source>

	* This approach has been used in the {{~~AddonCmd~~\|r3.in.xyz}} ~~addon~~ script.		* This approach has been used in the {{cmd\|r3.in.xyz}} script.
	* Another example using r.sun Mode 2 can be found on the [[r.sun]] wiki page.		* Another example using r.sun Mode 2 can be found on the [[r.sun]] wiki page.
	* See the {{cmd\|i.landsat.rgb}} and {{cmd\|i.oif}} examples.		* See the {{cmd\|i.landsat.rgb}} and {{cmd\|i.oif}} examples.
Line 153:		Line 153:

	=== GNU Parallel ===		=== GNU Parallel ===
	* [~~http~~://www.gnu.org/software/parallel/ GNU Parallel] is an advanced version of {{wikipedia\|xargs}} which makes it easy to write parallel shell scripts. On many Linux systems GNU parallel defaults to the "--tollef" style syntax. Adding the "--gnu" flag will cause GNU parallel to revert to native syntax. See also the unrelated C "parallel" program which comes with the Linux "moreutils" package. It is tighter but less featureful than GNU Parallel.		* [https://www.gnu.org/software/parallel/ GNU Parallel] is an advanced version of {{wikipedia\|xargs}} which makes it easy to write parallel shell scripts. On many Linux systems GNU parallel defaults to the "--tollef" style syntax. Adding the "--gnu" flag will cause GNU parallel to revert to native syntax. See also the unrelated C "parallel" program which comes with the Linux "moreutils" package. It is tighter but less featureful than GNU Parallel.

	<source lang="bash">		<source lang="bash">
Line 175:		Line 175:

	When tasks are limited by disk I/O parallel processing may be counter productive. A I/O monitoring tool such as [http://dag.wiee.rs/home-made/dstat/ dstat] can help diagnose cases where the use of GNU parallel fails to saturate all available CPU cores. A solid state disk can usually help cases where dstat reports a "wait" value that is larger than about 10%.		When tasks are limited by disk I/O parallel processing may be counter productive. A I/O monitoring tool such as [http://dag.wiee.rs/home-made/dstat/ dstat] can help diagnose cases where the use of GNU parallel fails to saturate all available CPU cores. A solid state disk can usually help cases where dstat reports a "wait" value that is larger than about 10%.


	GNU Parallel can also distribute work to other computers, see the video on how:		GNU Parallel can also distribute work to other computers, see the video on how:
	: ~~http~~://www.youtube.com/watch?v=LlXDtd_pRaY		: https://www.youtube.com/watch?v=LlXDtd_pRaY

	=== xargs ===		=== xargs ===

Neteler: + see also: r.mapcalc.tiled + r.texture.tiled

2020-10-10T14:49:38Z

+ see also: r.mapcalc.tiled + r.texture.tiled

← Older revision		Revision as of 14:49, 10 October 2020
Line 199:		Line 199:
	pnmtopng > \`basename {} .pdf\`.png"		pnmtopng > \`basename {} .pdf\`.png"
	</source>		</source>

			== See also ==

			* {{AddonCmd\|r.mapcalc.tiled}}: Runs {{cmd\|r.mapcalc}} in parallel over tiles
			* {{AddonCmd\|r.texture.tiled}}: Runs {{cmd\|r.texture}} in parallel over tiles

	[[Category: Development]]		[[Category: Development]]
	[[Category: Parallelization]]		[[Category: Parallelization]]

Neteler: manual url updates

2015-12-15T21:52:37Z

manual url updates

← Older revision		Revision as of 21:52, 15 December 2015
Line 52:		Line 52:
	* This approach has been used in the {{AddonCmd\|r3.in.xyz}} addon script.		* This approach has been used in the {{AddonCmd\|r3.in.xyz}} addon script.
	* Another example using r.sun Mode 2 can be found on the [[r.sun]] wiki page.		* Another example using r.sun Mode 2 can be found on the [[r.sun]] wiki page.
	* See the {{cmd\|i.landsat.rgb~~\|version=65~~}} and {{cmd\|i.oif~~\|version=65~~}} examples ~~in 6.5svn~~.		* See the {{cmd\|i.landsat.rgb}} and {{cmd\|i.oif}} examples.


Line 72:		Line 72:

	Similar to the Bourne shell example above, but using the '''subprocess''' python module.		Similar to the Bourne shell example above, but using the '''subprocess''' python module.
	The {{~~Cmd~~\|i.oif~~\|version=70~~}} script in GRASS7 is using this method.		The {{cmd\|i.oif}} script in GRASS7 is using this method.

	<source lang="python">		<source lang="python">

⚠️DylanBeaudette: /* GNU Parallel */

2015-10-16T19:32:49Z

GNU Parallel

← Older revision		Revision as of 19:32, 16 October 2015
Line 153:		Line 153:

	=== GNU Parallel ===		=== GNU Parallel ===
	* [http://www.gnu.org/software/parallel/ GNU Parallel] is an advanced version of {{wikipedia\|xargs}} which makes it easy to write parallel shell scripts.		* [http://www.gnu.org/software/parallel/ GNU Parallel] is an advanced version of {{wikipedia\|xargs}} which makes it easy to write parallel shell scripts. On many Linux systems GNU parallel defaults to the "--tollef" style syntax. Adding the "--gnu" flag will cause GNU parallel to revert to native syntax. See also the unrelated C "parallel" program which comes with the Linux "moreutils" package. It is tighter but less featureful than GNU Parallel.
	: See also the unrelated C "parallel" program which comes with the Linux "moreutils" package. It is tighter but less featureful than GNU Parallel.

	<source lang="bash">		<source lang="bash">
Line 168:		Line 167:
	time={} beam_rad=rad1_test.${DAY}_{}_beam --quiet		time={} beam_rad=rad1_test.${DAY}_{}_beam --quiet
	</source>		</source>

			Another example using the standard GNU parallel syntax. The "-k" flag will ensure that data are written to the output file in the same order than input maps are processed.
			<source lang="bash">
			# extract raster data from a time-series stack at named vector location
			t.rast.list -s my_raster_stack columns="id" \| parallel -k --gnu r.what map="{}" points=my_vector_map > output.dat
			</source>

			When tasks are limited by disk I/O parallel processing may be counter productive. A I/O monitoring tool such as [http://dag.wiee.rs/home-made/dstat/ dstat] can help diagnose cases where the use of GNU parallel fails to saturate all available CPU cores. A solid state disk can usually help cases where dstat reports a "wait" value that is larger than about 10%.


	GNU Parallel can also distribute work to other computers, see the video on how:		GNU Parallel can also distribute work to other computers, see the video on how:

⚠️HamishBowman: /* Bourne shell script */

2014-03-23T08:44:28Z

Bourne shell script

← Older revision		Revision as of 08:44, 23 March 2014
Line 12:		Line 12:
	fi		fi
	done		done
			wait
	</source>		</source>

⚠️HamishBowman: /* Bourne shell script */ another poor man's example

2014-03-23T08:42:18Z

Bourne shell script: another poor man's example

@@ Line 2: / Line 2: @@
 * '''Poor-man's multithreading''' using Bourne shell script & backgrounding. WARNING: not all GRASS modules and scripts are safe to have other things happening in the same mapset while they are running. Try at your own risk after performing a suitable safety audit. e.g. Make sure {{cmd|g.region}} is not run, externally changing the region settings.
-Example:
+Example 1:
 <source lang="bash">
   ### r.sun mode 1 loop ###

⚠️Landa: /* Bourne shell script */

2013-07-21T16:50:29Z

Bourne shell script

← Older revision		Revision as of 16:50, 21 July 2013
Line 1:		Line 1:
	=== Bourne shell script ===		=== Bourne shell script ===
	* '''Poor-man's multithreading''' using Bourne shell script & backgrounding. WARNING: not all GRASS modules and scripts are safe to have other things happening in the same mapset while they are running. Try at your own risk after performing a suitable safety audit. e.g. Make sure g.region is not run, externally changing the region settings.		* '''Poor-man's multithreading''' using Bourne shell script & backgrounding. WARNING: not all GRASS modules and scripts are safe to have other things happening in the same mapset while they are running. Try at your own risk after performing a suitable safety audit. e.g. Make sure {{cmd\|g.region}} is not run, externally changing the region settings.

	Example:		Example:

⚠️Erget: /* Python */ Added another (simple) implementation of a parallelization script

2012-08-07T14:48:58Z

Python: Added another (simple) implementation of a parallelization script

← Older revision		Revision as of 14:48, 7 August 2012
Line 97:		Line 97:


			This can also be accomplished fairly simply by using tracking the number of workers available and using grass.start_command as long as another job can be performed:

			<source lang="python">
			import os
			import multiprocessing as multi

			import grass.script as grass

			# Find number of workers that can be used on system. This variable could
			# also be set manually.
			workers = multi.cpu_count()
			# This is only a set of examples for r.slope.aspect jobs where the maps are
			# named serially.
			jobs = range(20)

			# Check if workers are already being used
			if workers is 1 and "WORKERS" in os.environ:
			workers = int(os.environ["WORKERS"])
			if workers < 1:
			workers = 1

			# Initialize process dictionary
			proc = {}

			# Loop over jobs
			for i in range(jobs):
			# Insert job into dictinoary to keep track of it
			proc[i] = grass.start_command('r.slope.aspect',
			elevation='elev_' + str(i),
			slope='slope_' + str(i))
			# If the workers are used up, wait for all of them from the last group to
			# finish.
			if i % workers is 0:
			for j in range(workers):
			proc.[i - j].wait()

			# Make sure all workers are finished.
			for i in range(jobs):
			if proc[i].wait() is not 0:
			grass.fatal(_('Problem running analysis on evel_' + str(i) + '.')
			</source>

	=== GNU Parallel ===		=== GNU Parallel ===

⚠️HamishBowman: /* Bourne shell script */

2012-06-19T05:59:13Z

Bourne shell script

← Older revision		Revision as of 05:59, 19 June 2012
Line 36:		Line 36:
	wait # wait for background jobs to finish to avoid race conditions		wait # wait for background jobs to finish to avoid race conditions
	</source>		</source>

			* This approach has been used in the {{AddonCmd\|r3.in.xyz}} addon script.
			* Another example using r.sun Mode 2 can be found on the [[r.sun]] wiki page.
			* See the {{cmd\|i.landsat.rgb\|version=65}} and {{cmd\|i.oif\|version=65}} examples in 6.5svn.


	Backgrounding code which sets environment variables with `eval` requires the use of grouping within ()s:		Backgrounding code which sets environment variables with `eval` requires the use of grouping within ()s:
Line 46:		Line 51:
	)`		)`
	</source>		</source>


	* This approach has been used in the {{AddonCmd\|r3.in.xyz}} addon script.
	* Another example using r.sun Mode 2 can be found on the [[r.sun]] wiki page.
	* See the {{cmd\|i.landsat.rgb\|version=65}} and {{cmd\|i.oif\|version=65}} examples in 6.5svn.

	=== Python ===		=== Python ===

⚠️HamishBowman: /* Bourne shell script */

2012-06-19T05:58:30Z

Bourne shell script

← Older revision		Revision as of 05:58, 19 June 2012
Line 37:		Line 37:
	</source>		</source>

	Backgrounding code which sets environment variables with `eval` requires the use of grouping within ()s.		Backgrounding code which sets environment variables with `eval` requires the use of grouping within ()s:
	<source lang="bash">		<source lang="bash">
	eval `(		eval `(