Fmriprep: Workaround for running subjects in parallel

fmriprep
#1

Currently, running subjects in parallel with fmriprep will sometimes cause errors. For example, when running 13 subjects in parallel, 4 were unsuccessful due to the following error caused by running them simultaneously:

FileNotFoundError: [Errno 2] No such file or directory: ‘/scratch/03201/jbwexler/openneuro_fmriprep/data/ds000003_work/ds000003-download/derivatives/fmriprep-1.4.0/fmriprep/logs/CITATION.md’

I’ve created an issue on the fmriprep github page, but until it is fixed, there is an easy workaround: Add a one minute delay between each task. For example, instead of:
fmriprep data/bids_root/ out/ participant --participant_label 01 -w work/
fmriprep data/bids_root/ out/ participant --participant_label 02 -w work/
fmriprep data/bids_root/ out/ participant --participant_label 03 -w work/
fmriprep data/bids_root/ out/ participant --participant_label 04 -w work/

Try the following:
sleep 1m && fmriprep data/bids_root/ out/ participant --participant_label 01 -w work/
sleep 2m && fmriprep data/bids_root/ out/ participant --participant_label 02 -w work/
sleep 3m && fmriprep data/bids_root/ out/ participant --participant_label 03 -w work/
sleep 4m && fmriprep data/bids_root/ out/ participant --participant_label 04 -w work/

3 Likes
#2

Thanks very much

I think this is a good question to add to the FAQs, @franklin

1 Like
#3

Thank you for the workaround @jbwexler !

Have added to the tips and tricks - here is the PR with it included: https://github.com/poldracklab/fmriprep/pull/1622/files

#4

I’m becoming less and less sure that this is a good workaround. Been running into lots of issues lately. Maybe wait for me to do some more testing before adding it to the FAQ

#5

@franklin Ok, I just updated the post with a workaround that seems to work a lot better

1 Like
#6

Hey, not sure if it is relevant but I just wanted to flag that I had lot of troubles when running ~150 subjects in parallel on our cluster, i.e., the pipeline would crash for about half the subjects with similar error as the one reported above, then I would tried again and it would work for some of those subjects but still crash on other subjects etc - until I tried to run freesurfer first, placed the output in the freesurfer output directory where it should be if you run it through the pipeline, and then tried to run fmriprep again. Not sure if this workaround is specific to our cluster, but doing that fixed the issues and all the subjects run perfectly fine.

#7

Hi Elena, thanks for sharing. Unfortunately, I get the errors regardless of whether or not freesurfer has already been run. Btw, do you know about the “–anat-only” flag for fmriprep? You can use it run just freesurfer and it will put everything in the correct place to run fmriprep afterward.