Fmriprep docker resource overload

1

Hello Neurostars,

I was trying to batch process a large dataset of resting state and structural scans with fmriprep, but was running into some resource allocation issues. In brief, I have one server with 40 cores and I used the following syntax

docker run --rm -v /home/jamielh:/opt/freesurfer/license.txt:ro -v /home/jamielh/Volumes/Hanson/NKI_HealthyBrainNetwork/RU/R1/:/data:ro -v /home/jamielh/Volumes/Hanson/NKI_HealthyBrainNetwork/RU/R1/derivatives/:/out -v /usr/share/fsl/5.0/data/standard/MNI152_T1_2mm_brain.nii.gz:/imports/MNI152_T1_2mm_brain.nii.gz:ro poldracklab/fmriprep:1.0.3 /data /out participant --participant_label ${i} --output-space T1w --no-freesurfer --ignore fieldmaps --output-space template T1w --use-aroma --ignore-aroma-denoising-errors --nthreads 1 --output-grid-reference /imports/MNI152_T1_2mm_brain.nii.gz --work-dir /home/jamielh/Volumes/Hanson/NKI_HealthyBrainNetwork/RU/R1/derivatives/working

I have my qsub queue limited to run 31 subjects simultaneously, but my server resources get ramped up (and then a few things go to hell), e.g., the load goes incredibly high… ~120-130; and a few times recently my qsub sun grid engine has stopped working. In general, it looks like antsRegistration is using a fair amount of cpu (here’s some pasting from top)

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
181850 root 20 0 2576912 1.877g 30876 R 651.0 1.5 131:33.62 antsRegistratio

I thought the ‘–nthreads 1’ flag above would have limited fmriprep’s resource utilization? Is there something I’m doing incorrectly? (Is it a positional argument?) Has anyone else run into this issue?

Thanks much,
Jamie.

2

Hi,

You can use --omp-nthreads argument to limit the number of threads ants and other OMP tools use. http://fmriprep.readthedocs.io/en/latest/usage.html#Options to handle performance

1 Like
3

Ah my confusion… I thought, --nthreads would have limited the thread use across all processes? To limit resources, is it best for me to use ‘–omp-nthreads’ AND ‘–nthreads’ flags?

In addition, fmriprep seems to be messing up my sun grid engine (qsub) functioning… I just tried to qdel all the processes I had in my queue and ran into this error:

jamielh@pfc:~$ qdel -u jamielh
error: commlib error: got select error (Connection refused)
unable to send message to qmaster using port 6444 on host “localhost”: got send error

Have other folks run into this? This issue has only started popping up since I’ve been using fmriprep (and I’ve had to restart my server three or four times this week).

1 Like
4

Correct. We would be happy to adjust this if something else would be more intuitive.

1 Like
5

I will think about potential changes (and will also try to figure my qsub/queue issues). Thanks much!

1 Like