Template Flow Issue on Singularity HPC

mattonim · December 18, 2023, 6:31pm

Summary of what happened:

I am trying to use XCP_D 0.6.0 on data preprocessed with fmriprep 23.1.3. Similar to a previous post, I’m running into an issue where XCP is trying to download templateflow files but the HPC has no internet access. I’m making a new post here as this seems like a separate issue from the previous post, which seems like it was solved in the github issue.

Command used (and if a helper script was used, a link to the helper script or the command generated):

I first tried:

singularity run -B /home/tun47039 -B --cleanenv xcp_d.simg \
/home/tun47039/work/CATS/derivatives /home/tun47039/work/postproc \
participant --smoothing 6 --custom_confounds /home/tun47039/work/UpdatedConfounds --disable_bandpass_filter \
-r auto -f 1.5 -w /home/tun47039/work/postprocWORK

Then, based on how I use fmriprep on the HPC, I also tried:

export APPTAINERENV_TEMPLATEFLOW_HOME="/templates"

singularity run -B /home/tun47039 -B /home/tun47039/work/templateflow/templateflow:/templates --cleanenv xcp_d.simg \
/home/tun47039/work/CATS/derivatives /home/tun47039/work/postproc \
participant --smoothing 6 --custom_confounds /home/tun47039/work/UpdatedConfounds --disable_bandpass_filter \
-r auto -f 1.5 -w /home/tun47039/work/postprocWORK

Both produced the same error.

fmriprep command example:
singularity run -B /home/tun47039 -B /home/tun47039/work/templateflow/templateflow:/templates --cleanenv /home/tun47039/fmriprep_23.1.3.simg /home/tun47039/work/CATS /home/tun47039/work/CATS/derivatives participant --participant_label 2002 -w $TMPDIR --fs-license-file /home/tun47039/license.txt --stop-on-first-crash --nthreads 8 --omp-nthreads 8 --skip-bids-validation --notrack --use-syn-sdc --fs-no-reconall --fd-spike-threshold 1.5

Version:

XCP 0.6.0. fmriprep 23.1.3.

Environment (Docker, Singularity, custom installation):

Singularity, HPC without internet access

Data formatted according to a validatable standard? Please provide the output of the validator:

BIDS Standard. Unedited after fmriprep.

Relevant log outputs (up to 20 lines):

231218-12:56:42,239 nipype.utils IMPORTANT:
         Collected run data for sub-2002_task-posmood_space-MNI152NLin2009cAsym_desc-preproc_bold.nii.gz:
boldmask: /gpfs/work/tun47039/CATS/derivatives/sub-2002/func/sub-2002_task-posmood_space-MNI152NLin2009cAsym_desc-brain_mask.nii.gz
boldref: /gpfs/work/tun47039/CATS/derivatives/sub-2002/func/sub-2002_task-posmood_space-MNI152NLin2009cAsym_boldref.nii.gz
confounds: /gpfs/work/tun47039/CATS/derivatives/sub-2002/func/sub-2002_task-posmood_desc-confounds_timeseries.tsv
confounds_json: /gpfs/work/tun47039/CATS/derivatives/sub-2002/func/sub-2002_task-posmood_desc-confounds_timeseries.json

Downloading https://templateflow.s3.amazonaws.com/tpl-MNI152NLin6Asym/tpl-MNI152NLin6Asym_from-MNI152NLin2009cAsym_mode-image_xfm.h5

Process Process-2:
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.8/site-packages/urllib3/connection.py", line 159, in _new_conn
    conn = connection.create_connection(
  File "/usr/local/miniconda/lib/python3.8/site-packages/urllib3/util/connection.py", line 84, in create_connection
    raise err
  File "/usr/local/miniconda/lib/python3.8/site-packages/urllib3/util/connection.py", line 74, in create_connection
    sock.connect(sa)
TimeoutError: [Errno 110] Connection timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/miniconda/lib/python3.8/site-packages/urllib3/connectionpool.py", line 381, in _make_request
    self._validate_conn(conn)
  File "/usr/local/miniconda/lib/python3.8/site-packages/urllib3/connectionpool.py", line 978, in _validate_conn
    conn.connect()
  File "/usr/local/miniconda/lib/python3.8/site-packages/urllib3/connection.py", line 309, in connect
    conn = self._new_conn()
  File "/usr/local/miniconda/lib/python3.8/site-packages/urllib3/connection.py", line 171, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x2aaacc477130>: Failed to establish a new connection: [Errno 110] Connection timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.8/site-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
  File "/usr/local/miniconda/lib/python3.8/site-packages/urllib3/connectionpool.py", line 726, in urlopen
    retries = retries.increment(
  File "/usr/local/miniconda/lib/python3.8/site-packages/urllib3/util/retry.py", line 446, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='templateflow.s3.amazonaws.com', port=443): Max retries exceeded with url: /tpl-MNI152NLin6Asym/tpl-MNI152NLin6Asym_from-MNI152NLin2009cAsym_mode-image_xfm.h5 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x2aaacc477130>: Failed to establish a new connection: [Errno 110] Connection timed out'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/local/miniconda/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/miniconda/lib/python3.8/site-packages/xcp_d/cli/run.py", line 977, in build_workflow
    retval["workflow"] = init_xcpd_wf(
  File "/usr/local/miniconda/lib/python3.8/site-packages/xcp_d/workflows/base.py", line 199, in init_xcpd_wf
    single_subj_wf = init_subject_wf(
  File "/usr/local/miniconda/lib/python3.8/site-packages/xcp_d/workflows/base.py", line 713, in init_subject_wf
    postprocess_bold_wf = init_postprocess_bold_wf(
  File "/usr/local/miniconda/lib/python3.8/site-packages/xcp_d/workflows/bold.py", line 465, in init_postprocess_nifti_wf
    qc_report_wf = init_qc_report_wf(
  File "/usr/local/miniconda/lib/python3.8/site-packages/xcp_d/workflows/plotting.py", line 234, in init_qc_report_wf
    get_template(
  File "/usr/local/miniconda/lib/python3.8/site-packages/templateflow/conf/__init__.py", line 65, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/miniconda/lib/python3.8/site-packages/templateflow/api.py", line 83, in get
    _s3_get(filepath)
  File "/usr/local/miniconda/lib/python3.8/site-packages/templateflow/api.py", line 226, in _s3_get
    r = requests.get(url, stream=True)
  File "/usr/local/miniconda/lib/python3.8/site-packages/requests/api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
  File "/usr/local/miniconda/lib/python3.8/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/miniconda/lib/python3.8/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/miniconda/lib/python3.8/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/miniconda/lib/python3.8/site-packages/requests/adapters.py", line 519, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='templateflow.s3.amazonaws.com', port=443): Max retries exceeded with url: /tpl-MNI152NLin6Asym/tpl-MNI152NLin6Asym_from-MNI152NLin2009cAsym_mode-image_xfm.h5 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x2aaacc477130>: Failed to establish a new connection: [Errno 110] Connection timed out'))

Thank you for any help, I am new to XPC and relatively inexperienced with HPC work (but learning!)

Steven · December 18, 2023, 6:59pm

Hi @mattonim,

Did you try downloading the templateflow directory locally (i.e., on your own machine), and then transfer it to your cluster some some SSHFT client (e.g., Cyberduck)?

Best,
Steven

mattonim · December 18, 2023, 7:03pm

Hi, thanks for the help.

Yep, I have it in /home/tun47039/work/templateflow/templateflow (replicated name intended), which I succesfully used to use fmriprep on this hpc, but the same strategy didn’t seem to work here.

Steven · December 18, 2023, 7:08pm

Have you used datalad to actually download the images? What you have might just be the git identifiers that link to the objects.

mattonim · December 18, 2023, 7:15pm

I didn’t use datalad, but I believe I have the actual images. Would I have been able to use fmriprep without them? Here is what the directory looks like:

And an example of one of the subfolders:

mattonim · December 18, 2023, 7:55pm

Also I just specifically checked that the tpl-MNI152NLin6Asym_from-MNI152NLin2009cAsym_mode-image_xfm.h5 file it is looking for exists nested in the directory I tried binding in the 2nd singularity command.

Steven · December 18, 2023, 8:20pm

Hi @mattonim,

It’s not just about whether the file exists. The file has to be downloaded from git. Templateflow is a datalad repository meaning that files may only be git identifiers to the objects and need to get pulled from the cloud. What are the file sizes of the nifti objects in the template folder?

Best,
Steven

mattonim · December 19, 2023, 2:55pm

Ah, I think you are right, thanks. I will work on downloading the actual images with datalad and transferring over. After that, should I need to bind that directory in the singularity command, like so?

export APPTAINERENV_TEMPLATEFLOW_HOME="/templates"

singularity run -B /home/tun47039 -B /home/tun47039/work/templateflow/templateflow:/templates --cleanenv xcp_d.simg \
/home/tun47039/work/CATS/derivatives /home/tun47039/work/postproc \
participant --smoothing 6 --custom_confounds /home/tun47039/work/UpdatedConfounds --disable_bandpass_filter \
-r auto -f 1.5 -w /home/tun47039/work/postprocWORK

Steven · December 19, 2023, 3:16pm

As long as you have apptainer installed, that should work (if you have an older version before singularity became apptainer then you’ll have to use SINGULARITYENV_xxxxx).