Memory Issue When Processing Large Amount of Data

dlevitas · September 29, 2018, 7:51pm

Hello,

I am trying to pre-process a subject where a large amount of data was collected (8 runs; 870 volumes per run), but have been running into memory issues. On my university’s high performance cluster, I have been requesting 16 cores and 80gb of virtual memory, yet after ~1.5 hours, I exceed my memory limit. I wanted to see if I should just request more virtual memory (perhaps 128gb), or if I should use the --low-mem and --mem-mb options instead?

If I use the --mem-mb options, should that value match how much virtual memory I request in my job? In other words, if I request 20gb on my job, I should do mem-mb 20000?

As reference, this was my command when the memory issue arose:

	unset PYTHONPATH; singularity run $data_dir/$s/fmriprep_*.img 	\
	$data_dir/$s $data_dir/$s 										\
	participant 													\
	--participant-label $s 											\
	--verbose 														\
	--fs-license-file $data_dir/$s/license.txt 						\
	--fs-no-reconall												\
	--use-syn-sdc 													\
	--force-syn 													\
	--use-aroma 													\
	--output-space template 										\
	--write-graph 													\
	--nthread 16												    \
	--omp-nthreads 16											    \
	--stop-on-first-crash 											\
	-w $data_dir/$s

Thank you,

Dan

ChrisGorgolewski · September 29, 2018, 10:02pm

I would try with removing --omp-nthreads (there is no benefit from going above 8) and adding --mem-mb 75000 (slightly less than required).

dlevitas · September 29, 2018, 10:43pm

Thanks Chris,

Could I also use --low-mem since my working directory is using scratch space, and therefore disk usage isn’t an issue for me?

dlevitas · October 2, 2018, 1:01pm

I’ve been noticing in the log output that even when requesting larger amounts of virtual memory (250gb), I still run into memory allocation issues:

Fatal Python error: Segmentation fault

Thread 0x00002ae34d4ee700 (most recent call first):
File “/usr/local/miniconda/lib/python3.6/selectors.py”, line 376 in select
File “/usr/local/miniconda/lib/python3.6/subprocess.py”, line 1514 in _communicate
File “/usr/local/miniconda/lib/python3.6/subprocess.py”, line 843 in communicate
File “/usr/local/miniconda/lib/python3.6/site-packages/nipype/interfaces/base/core.py”, line 834 in run_command
File “/usr/local/miniconda/lib/python3.6/site-packages/nipype/interfaces/base/core.py”, line 1030 in _run_interface
File “/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/interfaces/fixes.py”, line 24 in _run_interface
File “/usr/local/miniconda/lib/python3.6/site-packages/nipype/interfaces/base/core.py”, line 521 in run
File “/usr/local/miniconda/lib/python3.6/site-packages/fmriprep/interfaces/itk.py”, line 262 in _applytfms
File “/usr/local/miniconda/lib/python3.6/concurrent/futures/thread.py”, line 56 in run
File “/usr/local/miniconda/lib/python3.6/concurrent/futures/thread.py”, line 69 in _worker
File “/usr/local/miniconda/lib/python3.6/threading.py”, line 864 in run
File “/usr/local/miniconda/lib/python3.6/threading.py”, line 916 in _bootstrap_inner
File “/usr/local/miniconda/lib/python3.6/threading.py”, line 884 in _bootstrap

Thread 0x00002ae34d8f0700 (most recent call first):
File “/usr/local/miniconda/lib/python3.6/selectors.py”, line 376 in select
File “/usr/local/miniconda/lib/python3.6/subprocess.py”, line 1514 in _communicate
File “/usr/local/miniconda/lib/python3.6/subprocess.py”, line 843 in communicate
File “/usr/local/miniconda/lib/python3.6/site-packages/nipype/interfaces/base/core.py”, line 834 in run_command
File “/usr/local/miniconda/lib/python3.6/site-packages/nipype/interfaces/base/core.py”, line 1030 in _run_interface
File “/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/interfaces/fixes.py”, line 24 in _run_interface
File “/usr/local/miniconda/lib/python3.6/site-packages/nipype/interfaces/base/core.py”, line 521 in run
File “/usr/local/miniconda/lib/python3.6/site-packages/fmriprep/interfaces/itk.py”, line 262 in _applytfms
File “/usr/local/miniconda/lib/python3.6/concurrent/futures/thread.py”, line 56 in run
File “/usr/local/miniconda/lib/python3.6/concurrent/futures/thread.py”, line 69 in _worker
File “/usr/local/miniconda/lib/python3.6/threading.py”, line 864 in run
File “/usr/local/miniconda/lib/python3.6/threading.py”, line 916 in _bootstrap_inner
File “/usr/local/miniconda/lib/python3.6/threading.py”, line 884 in _bootstrap

Thread 0x00002ae34d2ed700 (most recent call first):
File “/usr/local/miniconda/lib/python3.6/selectors.py”, line 376 in select
File “/usr/local/miniconda/lib/python3.6/subprocess.py”, line 1514 in _communicate
File “/usr/local/miniconda/lib/python3.6/subprocess.py”, line 843 in communicate
File “/usr/local/miniconda/lib/python3.6/site-packages/nipype/interfaces/base/core.py”, line 834 in run_command
File “/usr/local/miniconda/lib/python3.6/site-packages/nipype/interfaces/base/core.py”, line 1030 in _run_interface
File “/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/interfaces/fixes.py”, line 24 in _run_interface
File “/usr/local/miniconda/lib/python3.6/site-packages/nipype/interfaces/base/core.py”, line 521 in run
File “/usr/local/miniconda/lib/python3.6/site-packages/fmriprep/interfaces/itk.py”, line 262 in _applytfms
File “/usr/local/miniconda/lib/python3.6/concurrent/futures/thread.py”, line 56 in run
File “/usr/local/miniconda/lib/python3.6/concurrent/futures/thread.py”, line 69 in _worker
File “/usr/local/miniconda/lib/python3.6/threading.py”, line 864 in run
File “/usr/local/miniconda/lib/python3.6/threading.py”, line 916 in _bootstrap_inner
File “/usr/local/miniconda/lib/python3.6/threading.py”, line 884 in _bootstrap

Thread 0x00002ae34d6ef700 (most recent call first):
File “/usr/local/miniconda/lib/python3.6/selectors.py”, line 376 in select
File “/usr/local/miniconda/lib/python3.6/subprocess.py”, line 1514 in _communicate
File “/usr/local/miniconda/lib/python3.6/subprocess.py”, line 843 in communicate
File “/usr/local/miniconda/lib/python3.6/site-packages/nipype/interfaces/base/core.py”, line 834 in run_command
File “/usr/local/miniconda/lib/python3.6/site-packages/nipype/interfaces/base/core.py”, line 1030 in _run_interface
File “/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/interfaces/fixes.py”, line 24 in _run_interface
File “/usr/local/miniconda/lib/python3.6/site-packages/nipype/interfaces/base/core.py”, line 521 in run
File “/usr/local/miniconda/lib/python3.6/site-packages/fmriprep/interfaces/itk.py”, line 262 in _applytfms
File “/usr/local/miniconda/lib/python3.6/concurrent/futures/thread.py”, line 56 in run
File “/usr/local/miniconda/lib/python3.6/concurrent/futures/thread.py”, line 69 in _worker
File “/usr/local/miniconda/lib/python3.6/threading.py”, line 864 in run
File “/usr/local/miniconda/lib/python3.6/threading.py”, line 916 in _bootstrap_inner
File “/usr/local/miniconda/lib/python3.6/threading.py”, line 884 in _bootstrap

Current thread 0x00002ae34def3700 (most recent call first):
File “/usr/local/miniconda/lib/python3.6/site-packages/numpy/linalg/linalg.py”, line 2010 in det
File “/usr/local/miniconda/lib/python3.6/site-packages/nibabel/nifti1.py”, line 1015 in set_qform
File “/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/interfaces/utils.py”, line 175 in _copyxform
File “/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/interfaces/fixes.py”, line 29 in _run_interface
File “/usr/local/miniconda/lib/python3.6/site-packages/nipype/interfaces/base/core.py”, line 521 in run
File “/usr/local/miniconda/lib/python3.6/site-packages/fmriprep/interfaces/itk.py”, line 262 in _applytfms
File “/usr/local/miniconda/lib/python3.6/concurrent/futures/thread.py”, line 56 in run
File “/usr/local/miniconda/lib/python3.6/concurrent/futures/thread.py”, line 69 in _worker
File “/usr/local/miniconda/lib/python3.6/threading.py”, line 864 in run
File “/usr/local/miniconda/lib/python3.6/threading.py”, line 916 in _bootstrap_inner
File “/usr/local/miniconda/lib/python3.6/threading.py”, line 884 in _bootstrap

Thread 0x00002ae34e0f4700 (most recent call first):
File “/usr/local/miniconda/lib/python3.6/selectors.py”, line 376 in select
File “/usr/local/miniconda/lib/python3.6/subprocess.py”, line 1514 in _communicate
File “/usr/local/miniconda/lib/python3.6/subprocess.py”, line 843 in communicate
File “/usr/local/miniconda/lib/python3.6/site-packages/nipype/interfaces/base/core.py”, line 834 in run_command
File “/usr/local/miniconda/lib/python3.6/site-packages/nipype/interfaces/base/core.py”, line 1030 in _run_interface
File “/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/interfaces/fixes.py”, line 24 in _run_interface
File “/usr/local/miniconda/lib/python3.6/site-packages/nipype/interfaces/base/core.py”, line 521 in run
File “/usr/local/miniconda/lib/python3.6/site-packages/fmriprep/interfaces/itk.py”, line 262 in _applytfms
File “/usr/local/miniconda/lib/python3.6/concurrent/futures/thread.py”, line 56 in run
File “/usr/local/miniconda/lib/python3.6/concurrent/futures/thread.py”, line 69 in _worker
File “/usr/local/miniconda/lib/python3.6/threading.py”, line 864 in run
File “/usr/local/miniconda/lib/python3.6/threading.py”, line 916 in _bootstrap_inner
File “/usr/local/miniconda/lib/python3.6/threading.py”, line 884 in _bootstrap

Thread 0x00002ae34dcf2700 (most recent call first):
File “/usr/local/miniconda/lib/python3.6/selectors.py”, line 376 in select
File “/usr/local/miniconda/lib/python3.6/subprocess.py”, line 1514 in _communicate
File “/usr/local/miniconda/lib/python3.6/subprocess.py”, line 843 in communicate
File “/usr/local/miniconda/lib/python3.6/site-packages/nipype/interfaces/base/core.py”, line 834 in run_command
File “/usr/local/miniconda/lib/python3.6/site-packages/nipype/interfaces/base/core.py”, line 1030 in _run_interface
File “/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/interfaces/fixes.py”, line 24 in _run_interface
File “/usr/local/miniconda/lib/python3.6/site-packages/nipype/interfaces/base/core.py”, line 521 in run
File “/usr/local/miniconda/lib/python3.6/site-packages/fmriprep/interfaces/itk.py”, line 262 in _applytfms
File “/usr/local/miniconda/lib/python3.6/concurrent/futures/thread.py”, line 56 in run
File “/usr/local/miniconda/lib/python3.6/concurrent/futures/thread.py”, line 69 in _worker
File “/usr/local/miniconda/lib/python3.6/threading.py”, line 864 in run
File “/usr/local/miniconda/lib/python3.6/threading.py”, line 916 in _bootstrap_inner
File “/usr/local/miniconda/lib/python3.6/threading.py”, line 884 in _bootstrap

Thread 0x00002ae34daf1700 (most recent call first):
File “/usr/local/miniconda/lib/python3.6/selectors.py”, line 376 in select
File “/usr/local/miniconda/lib/python3.6/subprocess.py”, line 1514 in _communicate
File “/usr/local/miniconda/lib/python3.6/subprocess.py”, line 843 in communicate
File “/usr/local/miniconda/lib/python3.6/site-packages/nipype/interfaces/base/core.py”, line 834 in run_command
File “/usr/local/miniconda/lib/python3.6/site-packages/nipype/interfaces/base/core.py”, line 1030 in _run_interface
File “/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/interfaces/fixes.py”, line 24 in _run_interface
File “/usr/local/miniconda/lib/python3.6/site-packages/nipype/interfaces/base/core.py”, line 521 in run
File “/usr/local/miniconda/lib/python3.6/site-packages/fmriprep/interfaces/itk.py”, line 262 in _applytfms
File “/usr/local/miniconda/lib/python3.6/concurrent/futures/thread.py”, line 56 in run
File “/usr/local/miniconda/lib/python3.6/concurrent/futures/thread.py”, line 69 in _worker
File “/usr/local/miniconda/lib/python3.6/threading.py”, line 864 in run
File “/usr/local/miniconda/lib/python3.6/threading.py”, line 916 in _bootstrap_inner
File “/usr/local/miniconda/lib/python3.6/threading.py”, line 884 in _bootstrap

Thread 0x00002ae315603440 (most recent call first):
File “/usr/local/miniconda/lib/python3.6/threading.py”, line 295 in wait
File “/usr/local/miniconda/lib/python3.6/concurrent/futures/_base.py”, line 427 in result
File “/usr/local/miniconda/lib/python3.6/concurrent/futures/_base.py”, line 586 in result_iterator
File “/usr/local/miniconda/lib/python3.6/site-packages/fmriprep/interfaces/itk.py”, line 143 in _run_interface
File “/usr/local/miniconda/lib/python3.6/site-packages/nipype/interfaces/base/core.py”, line 521 in run
File “/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py”, line 644 in _run_command
File “/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py”, line 564 in _run_interface
File “/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py”, line 480 in run
File “/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/plugins/multiproc.py”, line 69 in run_node
File “/usr/local/miniconda/lib/python3.6/concurrent/futures/process.py”, line 175 in _process_worker
File “/usr/local/miniconda/lib/python3.6/multiprocessing/process.py”, line 93 in run
File “/usr/local/miniconda/lib/python3.6/multiprocessing/process.py”, line 258 in _bootstrap
File “/usr/local/miniconda/lib/python3.6/multiprocessing/spawn.py”, line 118 in _main
File “/usr/local/miniconda/lib/python3.6/multiprocessing/forkserver.py”, line 231 in _serve_one
File “/usr/local/miniconda/lib/python3.6/multiprocessing/forkserver.py”, line 196 in main
File “”, line 1 in
exception calling callback for <Future at 0x2b7d6fb67ba8 state=finished raised BrokenProcessPool>
Traceback (most recent call last):
File “/usr/local/miniconda/lib/python3.6/concurrent/futures/_base.py”, line 324, in _invoke_callbacks
callback(self)
File “/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/plugins/multiproc.py”, line 143, in _async_callback
result = args.result()
File “/usr/local/miniconda/lib/python3.6/concurrent/futures/_base.py”, line 425, in result
return self.__get_result()
File “/usr/local/miniconda/lib/python3.6/concurrent/futures/_base.py”, line 384, in __get_result
raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

Looking online led me to this discussion, which suggests that setting the plugin to LegacyMultiProc may help. I was wondering how to do this from the fmriprep end, if it’s possible?

Thanks again.

effigies · October 2, 2018, 1:45pm

Hi @dlevitas,

See this comment by Mathias for an example plugin.yml file. You’ll need to use fMRIPrep>=1.1.6 for it to work correctly.

Chris

dlevitas · October 2, 2018, 2:50pm

Thanks @effigies

I’ve now downloaded version 1.1.7 to my server. I’ve read Mathias’s comment, but just to clarify, would this be the proper call in fmriprep?

--use-plugin plugin_args: {maxtasksperchild: 1, memory_gb: 250, n_procs: 16, raise_insufficient: false}

Or do I need to pass this into a file somewhere?

effigies · October 2, 2018, 3:04pm

You would create a file, e.g., plugin.yml with contents:

plugin: LegacyMultiProc
plugin_args: {maxtasksperchild: 1, memory_gb: 250, n_procs: 16, raise_insufficient: false}

And then pass --use-plugin plugin.yml.