It should be durable enough for reuse or inspection if your job errors or is interrupted, so not on a scratch that will be purged when a job completes, but because so many operations happen in it so quickly, it’s good to have a fast, synchronous filesystem if possible. Slower filesystems can cause some performance degradation on I/O heavy portions of the workflow, while ones with weaker consistency guarantees can cause some problems when a file just written isn’t available for reading immediately.
The derivatives directory will be mostly write-only (FreeSurfer is a notable exception, as it doesn’t distinguish working and output directories), so a slower network filesystem is usually not a problem. (Not that you can’t use a faster filesystem, but there’s often a durability-vs-speed tradeoff on HPC environments.)
Sure, that’s pretty common. I personally prefer to invert the relationship:
sourcedata/ -> $BIDS_DIR
Click to see semi-off-topic datalad discussion
You can do this with the following approach in regular shell…
mkdir -p $OUTPUT_DIR/fmriprep $OUTPUT_DIR/freesurfer
mv $BIDS_DIR $OUTPUT_DIR/fmriprep/sourcedata/
fmriprep $OUTPUT_DIR/fmriprep/sourcedata/ $OUTPUT_DIR participant ...
or use the datalad equivalent, that will record the changes to the dataset in a git history (assuming $BIDS_DIR is a datalad dataset)…
datalad create -c text2git $OUTPUT_DIR
datalad install -g --dataset $OUTPUT_DIR --source $BIDS_DIR $OUTPUT_DIR/fmriprep/sourcedata
datalad run fmriprep $OUTPUT_DIR/fmriprep/sourcedata $OUTPUT_DIR participant ...
If you’re used to thinking of derivatives as sub-datasets, this will feel a bit unnatural, but it makes a lot of sense to me. This links the contents of your derivatives directory with the state of your source dataset when the derivatives were produced. You can see an example of this at ///labs/poldrack/ds003_fmriprep (in this case I didn’t set it up to include the FreeSurfer outputs.).
If you’re thinking about data organization, you might have a look at YODA.
Yeah, it is sometimes worth trying to re-run with the old working directory, but it’s pretty brittle across versions. We haven’t yet come up with a very good way to automatically determine whether a working directory is incompatible.