Fmriprep using singularity on HPC can't find pediatric MNI template

,
1

Hi there! I have previously run fmriprep on my university’s high performance computing cluster with no problems, but am now using the same script on the same cluster with the same data, just using the pediatric template (MNIPediatricAsym) instead of the standard adult template (MNI152NLin2009cAsym). However, now fmriprep crashes with no output after about half an hour and I get this error:

Screen Shot 2022-01-27 at 9.28.36 PM
Screen Shot 2022-01-27 at 9.28.36 PM1134×682 231 KB

It seems that although I have my templates in a directory on the cluster (called “templateflow”), it is still trying to download the specified template, which is not possible because our cluster does not have internet access. Does anyone know how to solve this? Perhaps I’m not correctly pointing to where the templates live on the cluster.

Here is my script, which I run as a job in batches from my project directory on the cluster:

module load singularity

export 
SINGULARITYENV_TEMPLATEFLOW_HOME=/home/fmriprep/.cache/templateflow
                                 
rm -f archive/job_scripts/cmd_fmriprep_${PBS_JOBID}.txt
touch archive/job_scripts/cmd_fmriprep_${PBS_JOBID}.txt

for subj in ${subjects}; do
    echo singularity run --cleanenv \
            -B ~/work/project_dir/archive/templateflow:/opt/templateflow \
            ~/work/project_dir/archive/fmriprep-20.2.6.simg \
            ~/work/project_dir ~/work/project_dir/derivatives \
            participant --participant-label ${subj} \
            --output-spaces MNIPediatricAsym:res-2:cohort-2 \
            --fs-license-file ~/work/license.txt \
            --notrack \
            --stop-on-first-crash \
            --nthreads 8 --omp-nthreads 8 \
            --skip_bids_validation >> archive/job_scripts/cmd_fmriprep_${PBS_JOBID}.txt
done
                                                                         
torque-launch -p archive/job_scripts/chk_fmriprep_${PBS_JOBID}.txt 
archive/job_scripts/cmd_fmriprep_${PBS_JOBID}.txt

As stated, this script ran fine and produced the expected output when --output-spaces was specified as the adult template rather than the pediatric template. Thanks in advance for any advice!

2

Could you download the template on your local machine and then transfer it to your cluster’s templateflow directory?

If fMRIPrep was able to run before with no error, then this is probably not the case.

Steven

3

Hi Steven! I downloaded all the fmriprep templates into the directory where I have the -B flag, ~/work/project_dir/archive/templateflow which is on the cluster in my project directory, and which I’ve confirmed does indeed contain the template I’m trying to use. I’m wondering if there is a mismatch between that directory and when I set up template flow at the beginning with SINGULARITYENV_TEMPLATEFLOW_HOME=/home/fmriprep/.cache/templateflow? Someone informed me that the adult MNI template is included in the singularity image, while other templates like the pediatric one are not. So perhaps my template flow directory was always problematic, but I got around it before by fmriprep pulling the template from the .simg? Not sure if that’s true (or possible?), though!

4

Confirm that the actual templates are downloaded, and not just the git identifiers to them. You should see that the files are on the order of megabytes, not kilobytes, and you should be able to open a file in fslview/fsleyes, for example.

Whatever SINGULARITYENV_TEMPLATEFLOW_HOME is set to is where fMRIprep will look for templates. So you may want to move files to /home/fmriprep/.cache/templateflow or change the SINGULARITYENV_TEMPLATEFLOW_HOME variable.

Steven

5

Thanks for these recommendations! I checked and the file was indeed empty! So I downloaded it and moved it into my directory, and re-checked to confirm it’s an actual nifty file now, which it is. However, I’m still getting the same error with it trying to download the template instead of using the existing one. I followed the instructions in this post to confirm I can write to my directory and even checked within the /home/fmriprep/.cache/templateflow directory and confirmed the template is there as well. So I can’t figure out why fmriprep insists on trying to download it from the internet when it’s already downloaded into the directory I’m setting as my templateflow home.

6

This issue persists! I tried to run one subject on my lab’s linux machine and, although it successfully completed, I ran into a whole host of other issues, so I’d really prefer to be able to get this working on the cluster where I previously ran fmriprep with the adult template. Any other advice would be welcome! Thanks in advance.

7

For anyone who might come to this post with a similar issue, I was able to solve it through a different combination of exporting and binding directories on our HPC. A breakdown of the troubleshooting and final successful script can be found here: https://github.com/nipreps/fmriprep/issues/2717#issuecomment-1047238017

8

Hi Susan, Clayton here (from your original Twitter thread on this). I actually have found myself with a very similar problem (though, potentially not crash-inducing, because our server has internet). I saw your link to your solution, but it’s not 1:1 because I’m using Docker, not Singularity. Please don’t ask me why… I imagine there is also a Goldilocks solution of binding / mounting to our issue, but basically the scenario is that FMRIPREP seems to be working, but it’s unnecessarily downloading the pediatric template on every instance of our parallelized container runs, and this is adding a small but not insignificant amount to our runtime.

If you have a moment to take a look at this and anything stands out, please let me know.

Like you, I was originally storing my template in ~/.cache/templateflow, and found that I was actually only storing the skeleton (not sure why) and soon after downloaded the actual MNIPediatricAsym template (filesize on order of MB). This is still problematic, because Docker containers cannot see /home/username, a new /home is created in the container. So I instead moved my templateflow repo to a user-independent path, and mounted that directory to Docker as well. I then set the environment variable TEMPLATEFLOW_HOME equal to that directory for the Docker container. AFAIK, there is no equivalent of SINGULARITYENV_TEMPLATEFLOW_HOME for the Docker version.

        docker run --rm -u $( id -u)\
                --name FMRIPREP_$(basename $(dirname $1))_$sub \
                -v $1:/data:ro \
                -v $1/derivatives:/out \
                -v $1/work:/work \
                -v $(dirname $1)/bids-database:/bids-database \
                -v $FREESURFER_HOME/license.txt:/opt/freesurfer/license.txt \
                -v /path/to/server-wise/templateflow:/templateflow \
                -e TEMPLATEFLOW_HOME='/templateflow' \
                nipreps/fmriprep:latest \
                /data /out/fmriprep \
                participant \
                -w /work \
                --longitudinal \
                --skip_bids_validation \
                --use-aroma \
                --nthreads 16 \
                --low-mem \
                --mem-mb 15000 \
                --output-spaces MNIPediatricAsym:cohort-2:res-native \
                --participant-label $sub &

If you spot anything in my call here, or with how I’m running this, please let me know. Thanks!

9

Hi @claytonjschneider! I’m sorry to hear you’re running into this issue. I’m not super familiar with the constraints of using Docker, and honestly really only found my directory-binding solution for Singularity by trial and error.

My first suggestion would’ve been what you’ve done already, which is to confirm that the templates you want to use are actually physical nifty files in your templateflow directory and not just metadata. Another suggestion is to make sure that’s the case for the other required template directories (there are templates needed for fmriprep to run in general, in addition to the one to which you’d like to register your data).

Another thing to consider is your use of “res-native” here - I know this is theoretically an option per fmriprep’s website, but on the same page, it also says “Please mind that the selected resolutions specified must exist within TemplateFlow.” I wonder if you can’t use native resolution for the pediatric templates for some reason. You could try, e.g., res-2 and see if that runs and if that will maybe solve your problem?

The last thing I’d suggest is just to double check that your templates live wherever Docker is pointing. I’m not sure the whereabouts of these directories:

 -v /path/to/server-wise/templateflow:/templateflow \
 -e TEMPLATEFLOW_HOME='/templateflow' \

but assuming you’re binding in the first command and then directing back to it in the second, can you confirm your templates actually end up where they’re supposed to and that you’re pointing to that location correctly?

I’m quite an amateur in this domain and can’t offer much outside my immediate experience haha so hoping someone much more knowledgeable than I am (e.g. @Steven?) can come to your aid if none of this works! Best of luck and let me know if you figure it out!!

10

Thanks for getting back to me, @susanb! I think it’s comical that I ended up running into basically the same issue 2 months down the road. I suspect that using res-native is the problem, because I ran almost this exact script on another dataset without issue. Basically everything is identical, but the parameters of the acquisitions themselves. I felt really confident that I had a good understanding going into this one that my bindings were correct, and so res-native is highly suspect. By the way, that /path/to/server-wise path looks odd because I wrote in a fudged one for the sake of this post being public.

I will try again using the desired res and let you know! Thanks so much, your immediate experience in this domain has proved very informative!