Fmriprep partial success when accounting for SDC

So I am at a point where I need help with fmriprep. Please note, I am running it with Singularity and have run it successfully many times. However I have discovered I needed to add the “IntendedFor:” field into my fmap json files in the BIDS data(which has passed bids validator), to account for SDC. After doing this I tested it and it HAS run succesfully. After running a batch script to get all subjects done, most of the subjects are partially finishing (the ~/func directory and the report sub-X.html is not being completed). When looking at my error output the strange thing is the errors aren’t consistent, sometimes I am getting out of memory errors(which I highly doubt because the memory allocated for this is beyond sufficient and because some subjects run successfully) but for instance that error looks like…

OSError: [Errno 28] No space left on device: '/home_dir/projects/niblab/bids_projects/Experiments/Bevel/fmriprep/sub-009/fmriprep_wf/single_subject_009_wf/func_preproc_task_prob_run_4_wf/bold_confounds_wf/add_header/_node.pklz'

or I get something like…

FileNotFoundError: No such file: '/home_dir/projects/niblab/bids_projects/Experiments/Bevel/fmriprep/sub-005/fmriprep_wf/single_subject_005_wf/func_preproc_task_prob_run_1_wf/bold_mni_trans_wf/bold_to_mni_transform/vol0112_xform-00112.nii.gz'

and for reference the command looks like…

singularity exec -B /:/home_dir /projects/niblab/bids_projects/Singularity_Containers/fmriprep.simg \
fmriprep /home_dir/projects/niblab/bids_projects/Experiments/Bevel/BIDS /home_dir/projects/niblab/bids_projects/Experiments/Bevel/fmriprep/sub-001 \
    participant  \
    --participant-label sub-001 \
    --fs-license-file /home_dir/projects/niblab/bids_projects/freesurfer/license.txt \
    --fs-no-reconall \
    --omp-nthreads 16 --n_cpus 16 \
    --ignore slicetiming  \
    --bold2t1w-dof 12 \
    --output-space template --template MNI152NLin2009cAsym \
    --debug \
    -w /home_dir/projects/niblab/bids_projects/Experiments/Bevel/fmriprep/sub-001 \
    --resource-monitor --write-graph --stop-on-first-crash 

Like I mentioned above, this command had worked before I modified the json files and even after the modification some of my subjects have run successfully(and I cannot notice a difference in the BIDS data between successful and unsuccessful runs). That being said I do not think it is a singularity or BIDS issue. My next guess is possibly a package issue or possibly something I may need the cluster administrators to adjust but I am not sure. I am really at a loss here, any help is appreciated. Thank you.


Hi Nikki,

The error OSError: [Errno 28] No space left on device suggest the there is not enough hard disk space in the working directory specified via -w /home_dir/projects/niblab/bids_projects/Experiments/Bevel/fmriprep/sub-001. The second error could be related to the inability to write files. You can check disk space on unix systems via df -h, but your mileage might vary due to different ways disk quotas are implemented on multi-user clusters. This also could’ve been a temporary disk space shortage or filesystem malfunction on your cluster.

BTW on HPCs we recommend using local scratch (for example /tmp) for the working directory - it’s usually faster than the mounted network file systems. On some systems it is unfortunately not big enough.

Hope this helps!


Ahh, okay that makes sense. Thanks for the fast reply Chris I’ll mess around with this!!

1 Like