Process was killed when using fmriprep via docker

Community Chat
1

After running for a few hours, my process was killed. Hoping to know how to resolve this situation.
Thanks a lot !!!

210326-08:53:11,779 nipype.workflow ERROR:
Node t1w_dseg failed to run on host cda4a5e8e74e.
210326-08:53:14,43 nipype.workflow ERROR:
Saving crash info to /out/fmriprep/sub-CBDPC0413B/log/20210326-061458_b2c9e15d-f1d1-4dd7-ba7a-c51d1e9c077e/crash-20210326-085311-root-t1w_dseg-fabc65bd-9fcf-4ac4-8c71-7d360fb169b8.txt
Traceback (most recent call last):
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py”, line 67, in run_node
result[“result”] = node.run(updatehash=updatehash)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py”, line 516, in run
result = self._run_interface(execute=True)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py”, line 635, in _run_interface
return self._run_command(execute)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py”, line 741, in _run_command
result = self._interface.run(cwd=outdir)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py”, line 419, in run
runtime = self._run_interface(runtime)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py”, line 814, in _run_interface
self.raise_exception(runtime)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py”, line 745, in raise_exception
).format(**runtime.dictcopy())
RuntimeError: Command:
fast -N -p -g -S 1 /scratch/fmriprep_wf/single_subject_CBDPC0413B_wf/anat_preproc_wf/t1w_dseg/sub-CBDPC0413B_ses-s01_T1w_valid_corrected_xform_masked.nii.gz
Standard output:

Standard error:
Killed
Return code: 137

exception calling callback for <Future at 0x7f1d23406320 state=finished raised BrokenProcessPool>
Traceback (most recent call last):
File “/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py”, line 324, in _invoke_callbacks
callback(self)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py”, line 159, in _async_callback
result = args.result()
File “/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py”, line 425, in result
return self.__get_result()
File “/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py”, line 384, in __get_result
raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.
exception calling callback for <Future at 0x7f1d22cea710 state=finished raised BrokenProcessPool>
Traceback (most recent call last):
File “/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py”, line 324, in _invoke_callbacks
callback(self)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py”, line 159, in _async_callback
result = args.result()
File “/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py”, line 425, in result
return self.__get_result()
File “/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py”, line 384, in __get_result
raise self._exception
File “/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py”, line 324, in _invoke_callbacks
callback(self)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py”, line 159, in _async_callback
result = args.result()
File “/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py”, line 425, in result
return self.__get_result()
File “/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py”, line 384, in __get_result
raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.
exception calling callback for <Future at 0x7f1d22ceabe0 state=finished raised BrokenProcessPool>

210326-08:54:05,639 nipype.workflow CRITICAL:
fMRIPrep failed: A child process terminated abruptly, the process pool is not usable anymore

2

It would help to know more about how you are running Docker. E.g., Is this your own computer or are you running on a cluster? Are you running on a single subject or a whole directory?

I suspect, as return code 137 indicated an out-of-memory event, that you have not dedicated enough memory in Docker to allow fMRIPrep to run. Try upping it in settings and rerun. If it doesn’t work we can debug more from there.

Best,
Steven

3

Hi, Steven
Thanks a lot!! I am running Docker on a graphic workstation with 4×16GB of memory. And I am running a whole directory, nearly 100 subjects. The command I use is: fmriprep ./bids_dir ./output_dir participant -w ./cache_dir --output-spaces MNI152NLin6Asym:res-2 --use-aroma --fs-license-file ./fs-license.txt
I wanna ask for your help about a better parameter to run fmriprep. And how much CPU is required to run a large number of subjects in a directory, how about the optimum value and maximum value of the memory?
Thanks so much again.

4

Is that 4X16GB RAM for your computer or for Docker specifically? I was talking about about upping the limit on Docker, which you can do in settings. I would try giving 16 GB to it. Your command looks fine, but you can also try testing with just running a single subject too with the --participant-label argument.

1 Like