Difference between revisions of "Memory issues"

From GRASS-Wiki
Jump to navigation Jump to search
Line 52: Line 52:

[[Category: massive data analysis]]

Revision as of 12:46, 14 August 2012

thanks to Gordon Keith for these tips

Adding temporary swap space in Linux

To monitor your swap usage you can use utilities like top, xosview etc (they're the ones I use, but there are many others that will do the job).

If you are running out of swap space it is fairly easy to create a swap file and add it to your current swap area:

dd if=/dev/zero of=/var/swapfile bs=1024 count=2048000
mkswap /var/swapfile
swapon /var/swapfile

Will create a 2GB (approx) swapfile called /var/swapfile, format it as swapfile and add it to the swap space.

The file can reside anywhere on the local file system, so long as it it created using the preceding commands. On a faster disk is of course better for performance. Swapfiles bigger than 2GB only work on recent kernels, so if you have an older system I'd recommend sticking with 2GB or less and use more than one if you want more than 2GB extra swap space. (I used 2x2GB swapfiles and a 1.5GB swap partition when I was playing with s.surf.idw).

If you no longer need the swap space but need the disk space you can free it with:

swapoff /var/swapfile
rm /var/swapfile

The swapoff command will only work when there are no pages currently in the swapfile. See man pages for mkswap, swapon and swapoff for details.

Limiting process memory usage in Linux

On Linux using bash it can be useful to limit the amount of memory any process can have so that if you get the region size wrong the job dies rather than tying up the machine.

For example my .bashrc on our shared server has:

ulimit -S -v 20000000

This sets the soft limit for virtual memory to around 20GB which is reasonable for a machine with 32GB RAM and 64GB swap. I use a soft limit so that if I have a job that needs more I can up the limit. But generally if a module wants more than 20GB something is wrong.

It doesn't solve all out-of-memory problems, but may make it less painful. There is probably something similar on other platforms, but the mechanics may differ (eg under csh: limit vm 20000m).