Unable to use custom template fMRIPrep 22.1.1

snp2003 · March 21, 2023, 7:09pm

Dear team,

I am facing a similar issue as mentioned in this link when using custom template in a fMRIPrep docker container (v 22.1.1). Following are the options I have tried:

1] Created a custom template in MNI152NLin6Asym space with a resolution of 3mm and named it as “MNI152NLin6Asym3mm”. I saved this template with following structure

/CustomTemplateFlow
└── tpl-MNI152NLin6Asym3mm
├── template-description.json
├── tpl-MNI152NLin6Asym3mm_res-01_desc-brain_mask.nii.gz
├── tpl-MNI152NLin6Asym3mm_res-01_T1w.nii.gz
├── tpl-MNI152NLin6Asym3mm_res-03_desc-brain_mask.nii.gz
└── tpl-MNI152NLin6Asym3mm_res-03_T1w.nii.gz

2] I then ran my docker command by mounting this directory inside the container and get an error “ValueError: space identifier “/CustomTemplateFlow/MNI152NLin6Asym3mm” is invalid”

docker run -ti --rm -v /BIDS_nifti:/data:ro -v /fmriprep_output:/out \
	-v /fmriprep_work:/work -v ${fslicense}:/opt/freesurfer/license.txt \
    -v /CustomTemplateFlow:/CustomTemplateFlow:rw nipreps/fmriprep:22.1.1 \
    /data /out participant --participant-label sub-01 --fs-subjects-dir /out/freesurfer \
	--skip-bids-validation -v --use-syn-sdc --force-syn --use-aroma \
    --output-spaces /CustomTemplateFlow/MNI152NLin6Asym3mm:res-3 --low-mem --track-carbon -w /work

3] When above failed I tried setting TEMPLATEFLOW_HOME environment variable while mounting the docker and changed my command to:

docker run -ti --rm -v /BIDS_nifti:/data:ro -v /fmriprep_output:/out \
	-v /fmriprep_work:/work -v ${fslicense}:/opt/freesurfer/license.txt -v /CustomTemplateFlow:/CustomTemplateFlow:rw \
    -e TEMPLATEFLOW_HOME='/CustomTemplateFlow' nipreps/fmriprep:22.1.1 /data /out participant \
	--participant-label sub-01 --fs-subjects-dir /out/freesurfer \
	--skip-bids-validation -v --use-syn-sdc --force-syn --use-aroma --output-spaces \
    MNI152NLin6Asym3mm:res-3 --low-mem \
    --track-carbon -w /work

This gives me an error “datalad.support.exceptions.NoDatasetFound: No installed dataset found at /CustomTemplateFlow”. I do have templates necessary for fmriprep that I downloaded using “fetch_templates.py” script. Following are my templates saved under this custom template folder. Most niftis inside these sub-folder are zero size, given that they are only placeholders and niftis specifically listed in “fetch_templates.py” script are non-zero size.

/CustomTemplateFlow/
├── tpl-Fischer344
├── tpl-fsaverage
├── tpl-fsLR
├── tpl-MNI152Lin
├── tpl-MNI152NLin2009aAsym
├── tpl-MNI152NLin2009aSym
├── tpl-MNI152NLin2009bAsym
├── tpl-MNI152NLin2009bSym
├── tpl-MNI152NLin2009cAsym
├── tpl-MNI152NLin2009cSym
├── tpl-MNI152NLin6Asym
├── tpl-MNI152NLin6Asym3mm
├── tpl-MNI152NLin6Sym
├── tpl-MNI305
├── tpl-MNIColin27
├── tpl-MNIInfant
├── tpl-MNIPediatricAsym
├── tpl-MouseIn
├── tpl-NKI
├── tpl-NMT31Sym
├── tpl-OASIS30ANTs
├── tpl-PNC
├── tpl-RESILIENT
├── tpl-UNCInfant
├── tpl-VALiDATe29
└── tpl-WHS

Inside the container when I compared templates that were downloaded inherently by fmriprep docker inside ${HOME}/.cache/template and the one I mounted, it seems that placeholder for some templates are missing, for example I could not find templates named “tpl-MNI152Lin_desc-SPM12*” inside “tpl-MNI152Lin” sub-folder under my custom template folder. I do not think it is necessarily required by fmriprep pipeline, and it is not fetched in my custom folder with python API either. Any suggestions to make this work?

Thank you!

jsein · March 21, 2023, 7:59pm

Hi @snp2003,

For me, here is what worked (with singularity):

1/ I downloaded the templateflow archive in a local directory, namely $HOME/.templateflow.
2/ I added my CustomTemplate (named “tpl-MIITRA” in my case) in this $HOME/.templateflow folder, just as you did for tpl-MNI152NLin6Asym3mm.

3/ Here is the command I used:

export SINGULARITYENV_TEMPLATEFLOW_HOME=/home/fmriprep/.cache/templateflow

singularity run -B /scratch/jsein/BIDS:/work,$HOME/.templateflow:/home/fmriprep/.cache/templateflow --cleanenv \
 		 /scratch/jsein/my_images/fmriprep-21.0.2.simg \
		 --fs-license-file /work/freesurfer/license.txt /work/$study /work/$study/derivatives/fmriprep_MIITRA  \
		 participant --participant-label $sub \
		 -w /work/temp_dataMIITRA_${study}\
		 --mem-mb 50000 --omp-nthreads 10 --nthreads 12  \
		 --fd-spike-threshold 0.5 --dvars-spike-threshold 2.0 --bold2t1w-dof 9 --skull-strip-template MIITRA   \
		 --output-spaces MNI152NLin2009cAsym MIITRA T1w  --ignore slicetiming --debug compcor \
		 --fs-subjects-dir /work/$study/derivatives/fmriprep_MIITRA/sourcedata/freesurfer

Otherwise, what is the native resolution of your bold images? If there are close to 3mm isotropic, you could use the option --output-spaces MNI152NLin6Asym:res-native which would produce the result you want without the need of a custom template.

snp2003 · March 21, 2023, 9:13pm

Thank you for your prompt response.

I believe the closest option to export SINGULARITYENV_TEMPLATEFLOW_HOME=/home/fmriprep/.cache/templateflow in docker is setting an environment variable in the docker command while mounting which I am hoping is similar to -e TEMPLATEFLOW_HOME='/CustomTemplateFlow' flag I used, unless I misunderstood it?

Similar to your workflow, first I downloaded the templateflow archive to /CustomTemplateFlow folder using python client, which created a skeleton for all the templates. Then I copied my custom tpl-MNI152NLin6Asym3mm folder to CustomTemplateFlow folder. Since I got “datalad.support.exceptions.NoDatasetFound: No installed dataset found at /CustomTemplateFlow” error, I also used “fetch_templates.py” standalone script from fmriprep to fetch and update initial placeholders of templates and that did’nt work as well. So I am running out of ideas.

Native resolution of our fmri data is 4mm, which is a bit large, and we though that resampling to 2mm grid (already available within template flow archive) might cost us a lot of space. So we decided to resample it to 3mm resolution instead.

jsein · March 21, 2023, 9:32pm

I am not sure of this suggestion, but could you try:

-v /CustomTemplateFlow:/home/fmriprep/.cache/templateflow  -e TEMPLATEFLOW_HOME=/home/fmriprep/.cache/templateflow

To see if you get a different result?

Also, you are sure that /CustomTemplateFlow is indeed the absolute path to your Custom Template Flow folder?

snp2003 · March 21, 2023, 9:52pm

I did try this option as well, but with this my custom template is not copied inside docker’s
/home/fmriprep/.cache/templateflow folder. I believe it is the way fmriprep docker is setup in which it downloads templates to this folder which is its default path, and when I explicitly try to mount it, it does not allow it as /home/fmriprep/.cache/templateflow is not empty.

/CustomTemplateFlow is actually inside the mount folder in my server. Entire path is /mnt/data/snp2003/CustomTemplateFlow which is what I use it in my commands. I redacted it in the post so it is easy to read.

jsein · March 21, 2023, 9:56pm

I see.
I was thinking that mounting your local folder with : -v /CustomTemplateFlow:/home/fmriprep/.cache/templateflow would allow docker to see its content without the need to copy it.
It looks that docker behaviour is different from the singularity behaviour. Let’s hope somebody using fmriprep with docker can give better suggestions here. Good luck!

snp2003 · March 22, 2023, 1:42pm

Thank you, yes it does seem like it. Hoping if there is a solution to it.

effigies · March 22, 2023, 2:24pm

What is the command line you’re using now?

snp2003 · March 22, 2023, 3:45pm

I am using following docker command

docker run -ti --rm -v /BIDS_nifti:/data:ro -v /fmriprep_output:/out \
	-v /fmriprep_work:/work -v ${fslicense}:/opt/freesurfer/license.txt -v /CustomTemplateFlow:/CustomTemplateFlow:rw \
    -e TEMPLATEFLOW_HOME='/CustomTemplateFlow' nipreps/fmriprep:22.1.1 /data /out participant \
	--participant-label sub-01 --fs-subjects-dir /out/freesurfer \
	--skip-bids-validation -v --use-syn-sdc --force-syn --use-aroma --output-spaces \
    MNI152NLin6Asym3mm:res-3 --low-mem \
    --track-carbon -w /work

Error that I get with this command is:

“datalad.support.exceptions.NoDatasetFound: No installed dataset found at /CustomTemplateFlow”

However, insider the container I do see templates inside mounted /CustomTemplateFlow folder. Most are just placeholders except the ones downloaded with “fetch_templates.py” script and tpl-MNI152NLin6Asym3mm.

Thank you!

effigies · March 22, 2023, 3:48pm

I assume the error came with a whole traceback? Could you provide that? The full log would be helpful.

snp2003 · March 22, 2023, 4:34pm

Yes, following is what I see on the terminal output. I kill my command after I get this error as the only thing that runs after this exception is output from --track-carbon. I do not see a crash report under the output log sub-folder though.

You are using fMRIPrep-22.1.1, and a newer version of fMRIPrep is available: 23.0.0.
Please check out our documentation about how and when to upgrade:
https://fmriprep.readthedocs.io/en/latest/faq.html#upgrading
--topup-max-vols input is no longer used and the flag will be removed in a future release.
230322-16:13:37,880 cli INFO:
         Telemetry system to collect crashes and errors is enabled - thanks for your feedback!. Use option ``--notrack`` to opt out.
230322-16:17:39,975 nipype.workflow IMPORTANT:
         CodeCarbon tracker started ...
230322-16:17:39,975 nipype.workflow IMPORTANT:
         Using country_iso_code: CAN
230322-16:17:39,975 nipype.workflow IMPORTANT:
         Saving logs at: /out/logs
[codecarbon INFO @ 16:17:39] offline tracker init
[codecarbon INFO @ 16:17:39] [setup] RAM Tracking...
[codecarbon INFO @ 16:17:39] [setup] GPU Tracking...
[codecarbon INFO @ 16:17:39] No GPU found.
[codecarbon INFO @ 16:17:39] [setup] CPU Tracking...
[codecarbon INFO @ 16:17:39] Tracking Intel CPU via RAPL interface
[codecarbon INFO @ 16:17:41] >>> Tracker's metadata:
[codecarbon INFO @ 16:17:41]   Platform system: Linux-4.4.0-142-generic-x86_64-with-glibc2.31
[codecarbon INFO @ 16:17:41]   Python version: 3.9.12
[codecarbon INFO @ 16:17:41]   Available RAM : 503.806 GB
[codecarbon INFO @ 16:17:41]   CPU count: 40
[codecarbon INFO @ 16:17:41]   CPU model: Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
[codecarbon INFO @ 16:17:41]   GPU count: None
[codecarbon INFO @ 16:17:41]   GPU model: None
[codecarbon INFO @ 16:17:56] Energy consumed for RAM : 0.000787 kWh. RAM Power : 188.92708683013916 W
[codecarbon INFO @ 16:17:56] Energy consumed for all CPUs : 0.000211 kWh. All CPUs Power : 50.54294495304314 W
[codecarbon INFO @ 16:17:56] 0.000998 kWh of electricity used since the begining.
[codecarbon INFO @ 16:18:11] Energy consumed for RAM : 0.001575 kWh. RAM Power : 188.92708683013916 W
[codecarbon INFO @ 16:18:11] Energy consumed for all CPUs : 0.000419 kWh. All CPUs Power : 49.911324710202315 W
[codecarbon INFO @ 16:18:11] 0.001995 kWh of electricity used since the begining.
[codecarbon INFO @ 16:18:26] Energy consumed for RAM : 0.002357 kWh. RAM Power : 188.92708683013916 W
[codecarbon INFO @ 16:18:26] Energy consumed for all CPUs : 0.000632 kWh. All CPUs Power : 51.28802216214774 W
[codecarbon INFO @ 16:18:26] 0.002989 kWh of electricity used since the begining.
230322-16:18:29,676 nipype.workflow IMPORTANT:
         Running fMRIPrep version 22.1.1

         License NOTICE ##################################################
         fMRIPrep 22.1.1
         Copyright 2022 The NiPreps Developers.

         This product includes software developed by
         the NiPreps Community (https://nipreps.org/).

         Portions of this software were developed at the Department of
         Psychology at Stanford University, Stanford, CA, US.

         This software redistributes the versioneer Python package, which is
         Public domain source code.

         This software is also distributed as a Docker container image.
         The bootstraping file for the image ("Dockerfile") is licensed
         under the MIT License.

         This software may be distributed through an add-on package called
         "Docker Wrapper" that is under the BSD 3-clause License.
         #################################################################
[codecarbon INFO @ 16:18:41] Energy consumed for RAM : 0.003144 kWh. RAM Power : 188.92708683013916 W
[codecarbon INFO @ 16:18:41] Energy consumed for all CPUs : 0.000845 kWh. All CPUs Power : 51.21811490853103 W
[codecarbon INFO @ 16:18:41] 0.003989 kWh of electricity used since the begining.
230322-16:18:44,716 nipype.workflow IMPORTANT:
         Building fMRIPrep's workflow:
           * BIDS dataset path: /data.
           * Participant list: ['FCS02AMC'].
           * Run identifier: 20230322-161333_72fd6cfa-e290-4a9f-ad03-18a379caf5f5.
           * Output spaces: MNI152NLin6Asym3mm:res-3.
           * Pre-run FreeSurfer's SUBJECTS_DIR: /out/freesurfer.
It is highly recommended to configure Git before using DataLad. Set both 'user.name' and 'user.email' configuration variables.
Process Process-2:
Traceback (most recent call last):
  File "/opt/conda/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/opt/conda/lib/python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/conda/lib/python3.9/site-packages/fmriprep/cli/workflow.py", line 115, in build_workflow
    retval["workflow"] = init_fmriprep_wf()
  File "/opt/conda/lib/python3.9/site-packages/fmriprep/workflows/base.py", line 91, in init_fmriprep_wf
    single_subject_wf = init_single_subject_wf(subject_id)
  File "/opt/conda/lib/python3.9/site-packages/fmriprep/workflows/base.py", line 297, in init_single_subject_wf
    anat_preproc_wf = init_anat_preproc_wf(
  File "/opt/conda/lib/python3.9/site-packages/smriprep/workflows/anatomical.py", line 387, in init_anat_preproc_wf
    anat_norm_wf = init_anat_norm_wf(
  File "/opt/conda/lib/python3.9/site-packages/smriprep/workflows/norm.py", line 149, in init_anat_norm_wf
    template_meta = get_metadata(template.split(":")[0])
  File "/opt/conda/lib/python3.9/site-packages/templateflow/conf/__init__.py", line 65, in wrapper
    return func(*args, **kwargs)
  File "/opt/conda/lib/python3.9/site-packages/templateflow/api.py", line 170, in get_metadata
    _datalad_get(filepath)
  File "/opt/conda/lib/python3.9/site-packages/templateflow/api.py", line 205, in _datalad_get
    api.get(filepath, dataset=str(TF_LAYOUT.root))
  File "/opt/conda/lib/python3.9/site-packages/datalad/interface/utils.py", line 447, in eval_func
    return return_func(*args, **kwargs)
  File "/opt/conda/lib/python3.9/site-packages/datalad/interface/utils.py", line 439, in return_func
    results = list(results)
  File "/opt/conda/lib/python3.9/site-packages/datalad/interface/utils.py", line 357, in generator_func
    for r in _process_results(
  File "/opt/conda/lib/python3.9/site-packages/datalad/interface/utils.py", line 544, in _process_results
    for res in results:
  File "/opt/conda/lib/python3.9/site-packages/datalad/distribution/get.py", line 857, in __call__
    refds = require_dataset(
  File "/opt/conda/lib/python3.9/site-packages/datalad/distribution/dataset.py", line 595, in require_dataset
    raise NoDatasetFound(
datalad.support.exceptions.NoDatasetFound: No installed dataset found at /CustomTemplateFlow
[codecarbon INFO @ 16:18:56] Energy consumed for RAM : 0.003931 kWh. RAM Power : 188.92708683013916 W
[codecarbon INFO @ 16:18:56] Energy consumed for all CPUs : 0.001028 kWh. All CPUs Power : 43.91478422821897 W
[codecarbon INFO @ 16:18:56] 0.004959 kWh of electricity used since the begining.
[codecarbon INFO @ 16:19:11] Energy consumed for RAM : 0.004718 kWh. RAM Power : 188.92708683013916 W
[codecarbon INFO @ 16:19:11] Energy consumed for all CPUs : 0.001227 kWh. All CPUs Power : 47.666851996791266 W
[codecarbon INFO @ 16:19:11] 0.005945 kWh of electricity used since the begining.

effigies · March 22, 2023, 5:21pm

Okay. So I don’t really understand why it’s attempting to use datalad, as it doesn’t appear that you’ve enabled it by setting the TEMPLATEFLOW_USE_DATALAD environment variable. But the “simple” solution would just be to install the templateflow superdataset and add your template to it.

datalad install -r https://github.com/templateflow/templateflow.git
cd templateflow
datalad create -d . tpl-MNI152NLin6Asym3mm
# copy your files into the new dataset
datalad save -r . -m "Add tpl-MNI152NLin6Asym3mm template"

I suppose I’m also not sure why -v /path/to/MNI152NLin6Asym3mm:/home/fmriprep/.cache/templateflow/MNI152NLin6Asym3mm didn’t work. Or did you try that?

snp2003 · March 22, 2023, 6:08pm

Let me try this solution and update you.

I did try -v /CustomTemplateFlow/MNI152NLin6Asym3mm:/home/fmriprep/.cache/templateflow/MNI152NLin6Asym3mm option, it did not work though. With this option all templates are visible in /home/fmriprep/.cache/templateflow directory except MNI152NLin6Asym3mm, and I end up with an error

“ValueError: space identifier “MNI152NLin6Asym3mm” is invalid”

My understanding is the way fmriprep docker is setup in which it is pulling these templates early on and when I am trying to mount it, it is not able to as the existing /home/fmriprep/.cache/templateflow directory is non-empty.

effigies · March 22, 2023, 6:41pm

I don’t really understand this. Mounting to existing directories is generally not a problem in Docker.

effigies · March 22, 2023, 7:05pm

Just as an example, here I’m mounting my existing MNI152NLin6Asym directory into the Docker image:

$ docker run --rm -it \
    -v /home/chris/.cache/templateflow/tpl-MNI152NLin6Asym:/home/fmriprep/.cache/templateflow/tpl-MNI152NLin6Asym3mm \
    --entrypoint=bash nipreps/fmriprep:22.1.1
(base) root@035e86989976:/tmp# ls -l /home/fmriprep/.cache/templateflow
total 236
drwxrwxrwx 1 root     root      4096 Nov 30 20:26 tpl-Fischer344
drwxrwxrwx 1 root     root      4096 Nov 30 20:26 tpl-MNI152Lin
drwxrwxrwx 1 root     root      4096 Nov 30 20:26 tpl-MNI152NLin2009aAsym
drwxrwxrwx 1 root     root      4096 Nov 30 20:26 tpl-MNI152NLin2009aSym
drwxrwxrwx 1 root     root      4096 Nov 30 20:26 tpl-MNI152NLin2009bAsym
drwxrwxrwx 1 root     root      4096 Nov 30 20:26 tpl-MNI152NLin2009bSym
drwxrwxrwx 1 root     root     12288 Nov 30 20:26 tpl-MNI152NLin2009cAsym
drwxrwxrwx 1 root     root      4096 Nov 30 20:26 tpl-MNI152NLin2009cSym
drwxrwxrwx 1 root     root     12288 Nov 30 20:26 tpl-MNI152NLin6Asym
drwxrwxr-x 6 fmriprep fmriprep 12288 Jan 30 15:02 tpl-MNI152NLin6Asym3mm
drwxrwxrwx 1 root     root      4096 Nov 30 20:26 tpl-MNI152NLin6Sym
drwxrwxrwx 1 root     root      4096 Nov 30 20:26 tpl-MNI305
drwxrwxrwx 1 root     root      4096 Nov 30 20:26 tpl-MNIColin27
drwxrwxrwx 1 root     root      4096 Nov 30 20:26 tpl-MNIInfant
drwxrwxrwx 1 root     root       165 Nov 30 20:26 tpl-MNIPediatricAsym
drwxrwxrwx 1 root     root      4096 Nov 30 20:26 tpl-MouseIn
drwxrwxrwx 1 root     root      4096 Nov 30 20:26 tpl-NKI
drwxrwxrwx 1 root     root     90112 Nov 30 20:26 tpl-NMT31Sym
drwxrwxrwx 1 root     root      4096 Nov 30 20:26 tpl-OASIS30ANTs
drwxrwxrwx 1 root     root      4096 Nov 30 20:26 tpl-PNC
drwxrwxrwx 1 root     root       118 Nov 30 20:26 tpl-RESILIENT
drwxrwxrwx 1 root     root       117 Nov 30 20:26 tpl-UNCInfant
drwxrwxrwx 1 root     root      4096 Nov 30 20:26 tpl-VALiDATe29
drwxrwxrwx 1 root     root      4096 Nov 30 20:26 tpl-WHS
drwxrwxrwx 1 root     root      4096 Nov 30 20:26 tpl-fsLR
drwxrwxrwx 1 root     root      4096 Nov 30 20:26 tpl-fsaverage

snp2003 · March 22, 2023, 7:13pm

Ahh, I figured it out, what an exercise! When I was mounting it to default template flow directory I used ${HOME} to point to /home/fmriprep instead of explicitly specifying the home path inside the container, as in I did -v /CustomTemplateFlow:${HOME}/.cache/templateflow. I now tried mounting it with /CustomTemplateFlow:/home/fmriprep/.cache/templateflow and it shows up.

Thank you so much for all the help!

ari1 · March 22, 2024, 1:45pm

Hi, I was wondering if you could post your full code & folder structures that led to the successful custom template? I am also struggling to use a custom template for fmriprep! Thank you