Fmriprep 1.5.0 errors

Hello,

I’m gotten a series of errors for a participant that I’ve tried to run through fmriprep three separate times using the following command:

export SINGULARITYENV_TEMPLATEFLOW_HOME=/N/dcwan/projects/irf/templateflow

		unset PYTHONPATH; singularity run -B /N/dcwan/projects/irf/templateflow:/opt/templateflow /N/dcwan/projects/irf/containers/fmriprep-1.5.0.simg \
		$bids_root_dir $bids_root_dir/derivatives						\
		participant														\
		--skip-bids-validation 											\
		--participant-label $s 											\
		--fs-license-file $FREESURFER_HOME/license.txt 					\
		--ignore slicetiming 											\
		--use-aroma 													\
		--aroma-melodic-dimensionality 100 								\
		--output-spaces T1w fsnative fsaverage MNI152NLin6Asym:res-2	\
		--nthread $nthreads												\
		--fd-spike-threshold 0.5										\
		--dvars-spike-threshold 5 										\
		--stop-on-first-crash 											\
		--resource-monitor 												\
		--low-mem 														\
		--mem_mb $mem_mb 												\
		--use-plugin $bids_root_dir/fmriprep_plugin_${s}.yml 			\
		--verbose 														\
		-w $bids_root_dir/derivatives

I’m attaching the error log, since it’s rather long, but I’m wondering if it has something to do with nipype. Nothing else about the data suggests that it should fail. fmriprep_errors.txt (31.4 KB)

Thanks for the help

Hi @dlevitas. Have you tried with a clean working directory? There’ve been some changes in recent nipype that have been causing problems with reusing working directories.

Thanks @effigies. My working directory $bids_root_dir/derivatives is the one I use for all subjects in my dataset (with the fmriprep, fmriprep_wf, freesurfer directories), so I can see if using a clean directory solves the issue. To clarify though, do I need to specify a completely new working directory location, or could is be something like $bids_root_dir/derivatives/sub-${s}?

You can really call it whatever you want. In general, we tend to recommend not putting your working directory inside your BIDS data directory, but where you’ve placed it inside derivatives should not be problematic. Since the workflow directory will be WORKDIR/fmriprep_wf, changing to $bids_root_dir/derivatives/sub-${s} would be fine.

It may not be as obvious on first glance that derivatives/ is not itself a BIDS derivative directory, but this isn’t a technical problem. As long as you’re not confusing yourself or anybody that comes after you, do things that make sense to you.

Also note that, once done, there’s no problem with removing the working directories. If you’re keeping them around because there are contents that would be useful to have in the actual outputs, please let us know. Adding derivatives isn’t a big deal… probably the only useful derivative we’d be hesitant to add are more full-size BOLD series.

Is there a general recommendation for where the working directories should be? I’ve only put everything in /derivatives because it seemed like an intuitive place to puts things and since there’s no inherent BIDS structure to it. Would it be okay to still have the freesurfer, fmriprep and other directories in my derivatives, even if I have my fmriprep_wf outside the bids root directory?

I do normally delete the working directories; it was just that I wanted to keep them while testing out the new fmriprep version (in case something failed and didn’t want to completely restart the processing stream). But in the future I’ll be sure to remove them sooner.

Thanks again

It should be durable enough for reuse or inspection if your job errors or is interrupted, so not on a scratch that will be purged when a job completes, but because so many operations happen in it so quickly, it’s good to have a fast, synchronous filesystem if possible. Slower filesystems can cause some performance degradation on I/O heavy portions of the workflow, while ones with weaker consistency guarantees can cause some problems when a file just written isn’t available for reading immediately.

The derivatives directory will be mostly write-only (FreeSurfer is a notable exception, as it doesn’t distinguish working and output directories), so a slower network filesystem is usually not a problem. (Not that you can’t use a faster filesystem, but there’s often a durability-vs-speed tradeoff on HPC environments.)

Sure, that’s pretty common. I personally prefer to invert the relationship:

dsXYZ-fmriprep/
    fmriprep/
        sourcedata/ -> $BIDS_DIR
        sub-01/
        sub-02/
        ...
    freesurfer/
        sub-01/
        sub-02/
        ...
Click to see semi-off-topic datalad discussion

You can do this with the following approach in regular shell…

mkdir -p $OUTPUT_DIR/fmriprep $OUTPUT_DIR/freesurfer
mv $BIDS_DIR $OUTPUT_DIR/fmriprep/sourcedata/
fmriprep $OUTPUT_DIR/fmriprep/sourcedata/ $OUTPUT_DIR participant ...

or use the datalad equivalent, that will record the changes to the dataset in a git history (assuming $BIDS_DIR is a datalad dataset)…

datalad create -c text2git $OUTPUT_DIR
datalad install -g --dataset $OUTPUT_DIR --source $BIDS_DIR $OUTPUT_DIR/fmriprep/sourcedata
datalad run fmriprep $OUTPUT_DIR/fmriprep/sourcedata $OUTPUT_DIR participant ...

If you’re used to thinking of derivatives as sub-datasets, this will feel a bit unnatural, but it makes a lot of sense to me. This links the contents of your derivatives directory with the state of your source dataset when the derivatives were produced. You can see an example of this at ///labs/poldrack/ds003_fmriprep (in this case I didn’t set it up to include the FreeSurfer outputs.).

If you’re thinking about data organization, you might have a look at YODA.

Yeah, it is sometimes worth trying to re-run with the old working directory, but it’s pretty brittle across versions. We haven’t yet come up with a very good way to automatically determine whether a working directory is incompatible.