BrokenProcessPool error when running XCP-d

Software Support
1

Summary of what happened:

Hello - I’m trying to run xcp-d using my laptop (4 cores, 16GB RAM) on an fMRIprep directory living on an external hard drive. Running into a BrokenProcessPool error during regress_and_filter_bold. From my limited reading this may be a memory error so I tried limiting RAM and using low memory flag but it did not help. Would appreciate any suggestions!

Command used (and if a helper script was used, a link to the helper script or the command generated):

#!/bin/bash
set -euxo pipefail

export FREESURFER_HOME=/Applications/freesurfer/7.2.0
export HOME_DIR=/Volumes/Seagate/

docker run --rm -it \
   -v ${HOME_DIR}BIDS/derivatives:/fmriprep:ro \
   -v ${HOME_DIR}work_xcp:/work:rw \
   -v ${HOME_DIR}BIDS/derivatives/xcp_d:/out:rw \
   -v ${FREESURFER_HOME}/license.txt:/out/fs_license:rw \
   pennlinc/xcp_d:latest \
   /fmriprep /out participant \
   --despike -w /work --smoothing 6 --fs-license-file /out/fs_license --mem-gb 15 --low-mem```

Version:

0.7.5

Environment (Docker, Singularity / Apptainer, custom installation):

Docker

Data formatted according to a validatable standard? Please provide the output of the validator:

NA but it's valid from the fMRIprep run.

Relevant log outputs (up to 20 lines):

	 [Node] Executing "regress_and_filter_bold" <xcp_d.interfaces.nilearn.DenoiseNifti>
exception calling callback for <Future at 0x7fa6c7629e10 state=finished raised BrokenProcessPool>
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.10/concurrent/futures/_base.py", line 342, in _invoke_callbacks
    callback(self)
  File "/usr/local/miniconda/lib/python3.10/site-packages/nipype/pipeline/plugins/multiproc.py", line 159, in _async_callback
    result = args.result()
  File "/usr/local/miniconda/lib/python3.10/concurrent/futures/_base.py", line 451, in result
    return self.__get_result()
  File "/usr/local/miniconda/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.
exception calling callback for <Future at 0x7fa6c7628190 state=finished raised BrokenProcessPool>
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.10/concurrent/futures/_base.py", line 342, in _invoke_callbacks
    callback(self)
  File "/usr/local/miniconda/lib/python3.10/site-packages/nipype/pipeline/plugins/multiproc.py", line 159, in _async_callback
    result = args.result()
  File "/usr/local/miniconda/lib/python3.10/concurrent/futures/_base.py", line 451, in result
    return self.__get_result()
  File "/usr/local/miniconda/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/usr/local/miniconda/lib/python3.10/concurrent/futures/_base.py", line 342, in _invoke_callbacks
    callback(self)
  File "/usr/local/miniconda/lib/python3.10/site-packages/nipype/pipeline/plugins/multiproc.py", line 159, in _async_callback
    result = args.result()
  File "/usr/local/miniconda/lib/python3.10/concurrent/futures/_base.py", line 451, in result
    return self.__get_result()
  File "/usr/local/miniconda/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.
^C^C

Screenshots / relevant information:

2

Hi @ejcorn,

How much data are you trying to process? Can you try a minimal run with run subject and one bold file?

Best,
Steven

3

Hi Steve,

Appreciate your quick reply! I have 11 subjects with 10 sessions and 4 runs of resting state BOLD each (minus some sessions for a couple subjects who dropped out). Overall about 400 bold scans. Running a single subject/session/run now and will let you know what happens.

Best,
Eli

4

Single bold scan run worked … so just write a for loop and batch it manually?

5

Hi @ejcorn,

Yah that’s an option. And you might be safe going a level higher and doing a subject at a time (instead of per BOLD file).

Best,
Steven

6

Hi Steven,

Just wanted to give an update on this error. I continue to hit the same error unless I run XCP with a single bold scan as input. I’m able to work around this with a for loop, although it’s a little bit frustrating. Also, if one of the BOLD scans has too many frames scrubbed due to motion errors, then the pipeline terminates rather than completing with an error. This kills the for loop so I have to manually restart my script each time this happens.

Any suggestions for how to overcome that issue? If not, hopefully this report helps you all improve the package and I appreciate your quick replies and assistance.

Best,
Eli

7

Hi @ejcorn,

You have options, non-mutually exclusive. You can change the --min-time to something lower (default 240), you can lower the --fd-thresh (default 0.3), or in a bash script add the line set +e before the loop to not stop execution on error.

Best,
Steven

8

Thanks Steven! Much appreciated.