Issue with QSIprep templateflow on HPC host with no internet access

Software Support
10

Ah, I figured it out! sorry about all the confusion, this was an issue with --cleanenv --containall, and environment variables not properly set…

TL;DR

The following script works:

export TEMPLATEFLOW_HOME=path/to/.templateflow
apptainer run --cleanenv --containall \
  -B ${TEMPLATEFLOW_HOME}:${TEMPLATEFLOW_HOME} \
  --env "TEMPLATEFLOW_HOME=$TEMPLATEFLOW_HOME" \
  /path/to/qsiprep_${VERSION}.sif etc.
  • run it once on a node with internet until downloads finish,
  • then it can be submitted as a job to node without internet.

Details

My $TEMPLATEFLOW_HOME was not set in my submission script. When I was submitting it on the dev node via the terminal, it already had it in the environment so it worked. But not in the script, so it was going back to default home directory.

To your earlier point @Steven, I think my $HOME/.cache/templateflow folder was out of date. I used to run this script without the flags --cleanenv --containall, so templateflow would download to $HOME/.cache/templateflow automatically, which would be available both to the dev node (with internet) and the compute nodes (without).

But since I started using these flags, dev node runs were downloading the templates into their own qsiprep working directory, which are not available elsewhere on the cluster. This led job runs to not find the files, try to download them, and fail.

Thank you Steven for your help, really appreciate it! I’ll look into submitting some more info in the qsiprep docs regarding this.

1 Like