Parallel GRASS jobs - Revision history

Neteler: fix broken URLs

2023-12-27T19:43:52Z

fix broken URLs

← Older revision		Revision as of 19:43, 27 December 2023
Line 115:		Line 115:
	=== MPI Programming ===		=== MPI Programming ===

	There is a sample implementation at module level in [~~http~~://~~trac~~.~~osgeo.org~~/~~grass/browser~~/grass-addons/~~grass7~~/imagery/i.vi.mpi i.vi.mpi]		There is a sample implementation at module level in [https://github.com/OSGeo/grass-addons/tree/grass8/src/imagery/i.vi.mpi i.vi.mpi]

	=== GPU Programming ===		=== GPU Programming ===
Line 125:		Line 125:
	* See [[GPU]]		* See [[GPU]]

	Configure GRASS 7 with:		Configure GRASS GIS with:
	./configure --with-opencl		./configure --with-opencl

Line 168:		Line 168:
	== Cloud computing ==		== Cloud computing ==

	GRASS GIS 7 is running in the cloud as web processing service backend. Have a look at:		GRASS GIS is running in the cloud as web processing service backend. Have a look at:

	{{YouTube\|jg2pb_Xjq8Y\|desc=GRASS 7 in the cloud (by Sören Gebbert)}}		{{YouTube\|jg2pb_Xjq8Y\|desc=GRASS 7 in the cloud (by Sören Gebbert)}}
Line 179:		Line 179:
	* wps-grass-bridge latest svn		* wps-grass-bridge latest svn
	* QGIS 1.7 with a modified QWPS plugin		* QGIS 1.7 with a modified QWPS plugin

			For latest development, visit: https://github.com/actinia-org/actinia-core

	== GRASS GIS on VPS ==		== GRASS GIS on VPS ==
Line 184:		Line 186:
	Instructions to run GRASS GIS on a commercial VPS to do some memory-intensive operations:		Instructions to run GRASS GIS on a commercial VPS to do some memory-intensive operations:

	~~http~~://plantarum.ca/~~code~~/medium-performance-cluster/		https://plantarum.ca/2014/08/19/medium-performance-cluster-computing/

	== Hints for NFS users ==		== Hints for NFS users ==
Line 202:		Line 204:

	== Misc Tips & Tricks ==		== Misc Tips & Tricks ==
	See the poor-man's multi-processing script on the [[Parallelizing Scripts]] wiki page. This approach has been used in the {{~~AddonCmd~~\|r3.in.xyz}} ~~addon~~ script.		See the poor-man's multi-processing script on the [[Parallelizing Scripts]] wiki page. This approach has been used in the {{cmd\|r3.in.xyz}} script.

	== Workshop on Parallelization ==		== Workshop on Parallelization ==
Line 231:		Line 233:

	Elsewhere:		Elsewhere:
	* [~~http~~://~~gfoss~~.~~blogspot.com~~/~~2008/11~~/building-cluster-for-grass-gis-and~~.html~~ Building a cluster for GRASS GIS and other software from the OSGeo stack]		* [https://neteler.org/blog/building-a-cluster-for-grass-gis-and-other-software-from-the-osgeo-stack/ Building a cluster for GRASS GIS and other software from the OSGeo stack]
	* [https://research.csc.fi/geocomputing Using supercomputers for spatial analysis: ]		* [https://research.csc.fi/geocomputing Using supercomputers for spatial analysis]
	* [https://github.com/csc-training/geocomputing/tree/master/grass GRASS batch job and parallelization examples for a ~~supercomptuer~~ with SLURM]		* [https://github.com/csc-training/geocomputing/tree/master/grass GRASS batch job and parallelization examples for a supercomputer with SLURM]

	[[Category:Parallelization]]		[[Category:Parallelization]]
	[[Category: massive data analysis]]		[[Category: massive data analysis]]

Neteler: +Jupyter Notebook

2023-10-09T08:36:26Z

+Jupyter Notebook

← Older revision		Revision as of 08:36, 9 October 2023
Line 217:		Line 217:
	Session 2		Session 2
	* [Lab] Parallelization Case Study: Urban Growth Modeling		* [Lab] Parallelization Case Study: Urban Growth Modeling

			Jupyter Notebook:
			* https://github.com/ncsu-geoforall-lab/opengeohub-2023
			** https://github.com/ncsu-geoforall-lab/opengeohub-2023/blob/main/01_intro_to_GRASS_parallelization.ipynb

	== See also ==		== See also ==

Neteler: Workshop on Parallelization: agenda added

2023-10-09T08:01:57Z

Workshop on Parallelization: agenda added

← Older revision		Revision as of 08:01, 9 October 2023
Line 208:		Line 208:
	Haedrich, C., Petrasova, A. Parallelization for big EO data processing, OpenGeoHub Summer School 2023		Haedrich, C., Petrasova, A. Parallelization for big EO data processing, OpenGeoHub Summer School 2023
	* https://doi.org/10.5446/63123		* https://doi.org/10.5446/63123

			'''Schedule'''

			Session 1
			* [Slides] Introduction to Parallelization, GRASS GIS and Python
			* [Lab] Introduction to Parallelization with GRASS GIS and Python Notebook

			Session 2
			* [Lab] Parallelization Case Study: Urban Growth Modeling

	== See also ==		== See also ==

Neteler: +Workshop on Parallelization

2023-10-09T08:00:29Z

+Workshop on Parallelization

← Older revision		Revision as of 08:00, 9 October 2023
Line 203:		Line 203:
	== Misc Tips & Tricks ==		== Misc Tips & Tricks ==
	See the poor-man's multi-processing script on the [[Parallelizing Scripts]] wiki page. This approach has been used in the {{AddonCmd\|r3.in.xyz}} addon script.		See the poor-man's multi-processing script on the [[Parallelizing Scripts]] wiki page. This approach has been used in the {{AddonCmd\|r3.in.xyz}} addon script.

			== Workshop on Parallelization ==

			Haedrich, C., Petrasova, A. Parallelization for big EO data processing, OpenGeoHub Summer School 2023
			* https://doi.org/10.5446/63123

	== See also ==		== See also ==

⚠️Ktiits: Fix CSC links

2023-04-26T05:24:16Z

Fix CSC links

← Older revision		Revision as of 05:24, 26 April 2023
Line 214:		Line 214:
	Elsewhere:		Elsewhere:
	* [http://gfoss.blogspot.com/2008/11/building-cluster-for-grass-gis-and.html Building a cluster for GRASS GIS and other software from the OSGeo stack]		* [http://gfoss.blogspot.com/2008/11/building-cluster-for-grass-gis-and.html Building a cluster for GRASS GIS and other software from the OSGeo stack]
	* https://research.csc.fi/supercomputers-and~~-gis~~		* [https://research.csc.fi/geocomputing Using supercomputers for spatial analysis: ]
			* [https://github.com/csc-training/geocomputing/tree/master/grass GRASS batch job and parallelization examples for a supercomptuer with SLURM]

	[[Category:Parallelization]]		[[Category:Parallelization]]
	[[Category: massive data analysis]]		[[Category: massive data analysis]]

Neteler: added "Tips for parallelization in GRASS GIS"

2022-08-27T15:47:57Z

added "Tips for parallelization in GRASS GIS"

@@ Line 1: / Line 1: @@
 == Parallel GRASS jobs ==
-NOTE: GRASS 6 libraries are NOT thread safe (except for GPDE, see below).
+The idea of parallel GRASS GIS jobs is to speed up the computation.
 GRASS doesn't perform any locking on the files within a GRASS
@@ Line 25: / Line 42: @@
 See below for ways around these limitations.
 == Working with tiles ==

⚠️Mmetz: GRASS on a cluster: cross-link wiki pages

2018-03-10T21:39:38Z

GRASS on a cluster: cross-link wiki pages

← Older revision		Revision as of 21:39, 10 March 2018
Line 156:		Line 156:
	# deletes the temporary mapset and GISRC file		# deletes the temporary mapset and GISRC file

	Such a script should not call grassXY and must be executable outside GRASS, because it is establishing a GRASS session, performing some processing, and closing the GRASS session, all by itself.		Such a script should not call grassXY and must be executable outside GRASS, because it is establishing a GRASS session, performing some processing, and closing the GRASS session, all by itself. See also [[GRASS_and_Shell\|GRASS and Shell]].

	Such a script will take arguments to specify the particular data to be processed.		Such a script will take arguments to specify the particular data to be processed.

⚠️Mmetz: /* GRASS on a cluster */

2018-03-10T21:35:17Z

GRASS on a cluster

@@ Line 146: / Line 146: @@
 * write a script that initializes GRASS, creates a unique mapset, executes the script with the actual processing, and copies the results to a common mapset
 * add that script as a task to a job, create one job for each data chunk
 The common bottleneck when using GRASS on a cluster is often disk I/O. Try to start the jobs with nice/ionice to reduce strain on the storage devices.

⚠️Mmetz: Cluster and Grid computing: update, more generic

2018-02-07T21:02:39Z

Cluster and Grid computing: update, more generic

Show changes

Neteler: grass70 --> grass72

2017-04-27T11:18:01Z

grass70 --> grass72

← Older revision		Revision as of 11:18, 27 April 2017
Line 83:		Line 83:
	=== Python ===		=== Python ===

	[http://grass.osgeo.org/~~grass70~~/manuals/libpython/pygrass.modules.interface.html?highlight=parallelmodulequeue#pygrass.modules.interface.module.ParallelModuleQueue PyGRASS ParallelModuleQueue]		[http://grass.osgeo.org/grass73/manuals/libpython/pygrass.modules.interface.html?highlight=parallelmodulequeue#pygrass.modules.interface.module.ParallelModuleQueue PyGRASS ParallelModuleQueue]

	=== pthreads ===		=== pthreads ===

	Note: only used in the r.mapcalc parser!		Note: only used in the r.mapcalc ''parser''!

	Good for a single system with a multi-core CPU.		Good for a single system with a multi-core CPU.
Line 94:		Line 94:
	./configure --with-pthread		./configure --with-pthread

	The {{cmd\|r.mapcalc~~\|version=70~~}} in GRASS 7 has been parallelized using GNU {{wikipedia\|pthreads}}.		The ''parser'' of {{cmd\|r.mapcalc}} in GRASS 7 has been parallelized using GNU {{wikipedia\|pthreads}}. The computation itself is executed serially.

	=== Bourne and Python Scripts ===		=== Bourne and Python Scripts ===