MRIQC crashing in middle of scan

mriqc

#1

Hi, I am new to MRIQC and linux (ubuntu 18.04LTS) in general. I am trying to run a participant level analysis on approx 100 subjects. The program crashes after a day or so into the analysis.

This is an abbreviated version of the output:

2018-12-25 06:57:43,234 concurrent.futures:ERROR exception calling callback for <Future at 0x7fcb23f47fd0 state=finished raised BrokenProcessPool>

Traceback (most recent call last):

File "/usr/local/miniconda/lib/python3.6/concurrent/futures/_base.py", line 324, in _invoke_callbacks

callback(self)

File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/plugins/multiproc.py", line 143, in _async_callback

result = args.result()

File "/usr/local/miniconda/lib/python3.6/concurrent/futures/_base.py", line 425, in result

return self.__get_result()

File "/usr/local/miniconda/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result

raise self._exception

concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

2018-12-25 06:57:43,414 concurrent.futures:ERROR exception calling callback for <Future at 0x7fcb281fbcf8 state=finished raised BrokenProcessPool>

Traceback (most recent call last):

File "/usr/local/miniconda/lib/python3.6/concurrent/futures/_base.py", line 324, in _invoke_callbacks

callback(self)

File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/plugins/multiproc.py", line 143, in _async_callback

result = args.result()

File "/usr/local/miniconda/lib/python3.6/concurrent/futures/_base.py", line 425, in result

return self.__get_result()

File "/usr/local/miniconda/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result

raise self._exception

concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

I would really appreciate some suggestions as I have been stuck on this for some time now. I have read various forums which mostly suggest increasing the memory allotment which I have done and still have the issue. (Gold star for the right answer!)


#2

What is the exact command line you are trying to run?


#3

This is the code:

sudo docker run --cpus 7.0 -m=20G -it --rm -v /mnt/7056E0E656E0AE52/Trajectories_Data/MB_HP_REGTRJ/REGTRJ_Main:/data:ro -v /mnt/7056E0E656E0AE52/Trajectories_Data/MRIQC_OUTPUT/rest/:/out poldracklab/mriqc:latest /data /out participant --participant_label regtrj003 regtrj004 regtrj005 regtrj006 regtrj007 regtrj008 regtrj009 regtrj010 regtrj011 regtrj012 regtrj013 regtrj014 regtrj015 regtrj016 regtrj017 regtrj019 regtrj020 regtrj021 regtrj022 regtrj024 regtrj025 regtrj026 regtrj027 regtrj028 regtrj029 regtrj030 regtrj031 regtrj032 regtrj033 regtrj034 regtrj035 regtrj036 regtrj037 regtrj038 regtrj040 regtrj041 regtrj042 regtrj043 regtrj044 regtrj045 regtrj046 regtrj047 regtrj048 regtrj049 regtrj050 regtrj053 regtrj054 regtrj055 regtrj056 regtrj057 regtrj059 regtrj060 regtrj061 regtrj062 regtrj063 regtrj064 regtrj065 regtrj066 regtrj067 regtrj068 regtrj069 regtrj070 regtrj071 regtrj072 regtrj073 regtrj074 regtrj075 regtrj076 regtrj077 regtrj078 regtrj079 regtrj080 regtrj081 regtrj082 regtrj083 regtrj084 regtrj085 regtrj086 regtrj088 regtrj089 regtrj090 regtrj091 regtrj092 regtrj093 regtrj094 regtrj095 regtrj096 regtrj097 regtrj098 regtrj099 regtrj101 regtrj102 regtrj103 regtrj104 regtrj105 regtrj106 regtrj107 regtrj108 regtrj109 regtrj110 regtrj111 regtrj112 regtrj113 regtrj115 regtrj116 regtrj117 regtrj118 regtrj119


#4

Pulling all subjects together will create a huge workflow. That BrokenProcessPoolError is telling you that the memory limits were likely hit.

Please, try running batches of one or two subjects at most. If you are running in Windows or Mac, please make sure you gave access Docker to those 20GB RAM you are setting via -m=20G