Fmriprep: pre-download templateflow

1

I’m running fmriprep in a singularity container and my computing nodes do not have internet access. Therefore, I’ve pre-downloaded the templates as suggested. However, even though I now have the templates already downloaded, fmriprep still tries to download the files and inevitably fails. Can someone please tell me what I’m doing wrong here? Thanks in advance!

export freesurfer_license=$HOME/licenses/freesurfer/license.txt
export SINGULARITYENV_TEMPLATEFLOW_HOME=/templateflow

sing_home=$SCRATCH/anm/newdata_032320/singularity_logs
outdir=$SCRATCH/anm/newdata_032320/derivatives
workdir=$SCRATCH/anm/newdata_032320/work/fmriprep
indir=$SCRATCH/anm/newdata_032320/rawdata

mkdir -p ${sing_home} ${outdir} ${workdir}

singularity run --no-home \
  -H ${sing_home} \
  -B ${indir}:/bids \
  -B ${outdir}:/output \
  -B ${freesurfer_license}:/freesurfer_license.txt \
  -B ${workdir}:/workdir \
  -B ${TEMPLATEFLOW_HOME:-$SCRATCH/anm/newdata_032320/singularity_logs/.cache/templateflow}:/templateflow \
  --cleanenv \
  $SCRATCH/singularity_containers/fmriprep-20.1.1.simg \
      /bids /output participant \
      --participant_label Cc0029 \
      --use-aroma \
      --nthreads 13 \
      --omp-nthreads 13 \
      --mem_mb 20000 \
      --fs-license-file /freesurfer_license.txt \
      --work-dir /workdir \
      --output-spaces MNI152NLin6Asym MNI152NLin2009cAsym \
      --skull-strip-template OASIS30ANTs \
      --notrack
2

I would try running the templateflow commands inside the singularity container. It should be more illuminating.

The following is copied from you, except I replace run with exec and drop all of the fMRIPrep arguments, instead running python.

export freesurfer_license=$HOME/licenses/freesurfer/license.txt
export SINGULARITYENV_TEMPLATEFLOW_HOME=/templateflow

sing_home=$SCRATCH/anm/newdata_032320/singularity_logs
outdir=$SCRATCH/anm/newdata_032320/derivatives
workdir=$SCRATCH/anm/newdata_032320/work/fmriprep
indir=$SCRATCH/anm/newdata_032320/rawdata

singularity exec --no-home \
  -H ${sing_home} \
  -B ${indir}:/bids \
  -B ${outdir}:/output \
  -B ${freesurfer_license}:/freesurfer_license.txt \
  -B ${workdir}:/workdir \
  -B ${TEMPLATEFLOW_HOME:-$SCRATCH/anm/newdata_032320/singularity_logs/.cache/templateflow}:/templateflow \
  --cleanenv \
  $SCRATCH/singularity_containers/fmriprep-20.1.1.simg python

This should produce a shell:

>>> from templateflow import api as tf
>>> tf.get(...)

That should be targeting the same location as fMRIPrep does.

3

Hi, I’m having the same issue as the original poster, could you expand on what you mean by running the commands inside the container? Sorry I don’t have much experience with bash or singularity

For reference, this is the code I’m running, which is failing because it tries to download templateflow rather than use what I have stored locally. I have the templates stored in the path specified in the TEMPLATEFLOW_DIR.

module load singularity/3.7.3

TEMPLATEFLOW_DIR=/home/tun47039/work/tads_bids/archive/templateflow
export SINGULARITYENV_TEMPLATEFLOW_HOME=/home/fmriprep/.cache/templateflow

singularity run --cleanenv -B ${TEMPLATEFLOW_DIR}:/home/fmriprep/.cache/templateflow
/home/tun47039/fmriprep-20.2.6.simg
/home/tun47039/work/tads_bids /home/tun47039/work/tads_bids/derivatives
-w /home/tun47039/scratch/tads_scratch
participant --participant_label 5000
[various fmriprep arguments]

4

My guess is that you’re not able to bind TEMPLATEFLOW_DIR. I would check, by first dropping something into your templateflow directory:

TEMPLATEFLOW_DIR=/home/tun47039/work/tads_bids/archive/templateflow
export SINGULARITYENV_TEMPLATEFLOW_HOME=/home/fmriprep/.cache/templateflow

touch $TEMPLATEFLOW_DIR/canary

Then use singularity shell to look at the filesystem from inside the container:

singularity shell --cleanenv -B ${TEMPLATEFLOW_DIR}:/home/fmriprep/.cache/templateflow /home/tun47039/fmriprep-20.2.6.simg

Inside the shell:

ls /home/fmriprep/.cache/templateflow

You should see your canary. If not, the bind is failing, probably due to system policy. I would try putting your templateflow directory somewhere like a scratch directory.

5

Hi, Thanks so much for the reply. Unfortunately, this did not solve the issue, as I can see my canary after following your exact code - screenshot below:

image
image743×298 10.8 KB

6

The fact that you see your canary doesn’t mean that you can actually write to that folder from within the container. I’d say you are having some permission issues with that folder.

7

Hi,
I don’t believe it turned out to be a permissions error. I’m posting the solution that worked for me below, in case it is ever of use to someone else.

First, I exported the same location as my previous code:
export SINGULARITYENV_TEMPLATEFLOW_HOME=/home/fmriprep/.cache/templateflow

Then in singularity, I binded a /opt directory, instead of the /fmriprep/.cache directory
singularity run --cleanenv -B /home/tun47039/work/tads_bids/archive/templateflow:/opt/templateflow
[fmriprep & arguments]

Not sure why this worked, but it did. I’m Running singularity 3.7.3 and fmriprep 20.2.6.

8

Cool! Happy to hear you figured it out. Thanks much for reporting the solution :slight_smile:

1 Like