Error Connection to templateflow timed out

cindylucerog · November 21, 2024, 1:22am

Summary of what happened:

when trying to run ASLprep in the Niagara cluster via Singularity, I get an error (see below). I downloaded templates, as advised here: FAQ - Frequently Asked Questions — fmriprep version documentation
After this, I bound the Template Path, but still get this error.

Command used (and if a helper script was used, a link to the helper script or the command generated):

singularity run --cleanenv \
  --bind /home/m/mchakrav/cindylu/scratch/analysis/ASLprep/data3subjects:/data:ro \
  --bind /home/m/mchakrav/cindylu/scratch/analysis/ASLprep/output3subjects:/out \
  --bind /home/m/mchakrav/cindylu/scratch/analysis/ASLprep/intermediatefiles:/work \
  --bind /home/m/mchakrav/cindylu/scratch/analysis/ASLprep/freesurferlicense:/freesurfer_license \
  --bind /home/m/mchakrav/cindylu/scratch/analysis/ASLprep/templateflow:/templateflow \
  /home/m/mchakrav/cindylu/scratch/analysis/aslprep-0.7.4.simg \
  /data /out participant \
  --output-spaces T1w MNI152NLin2009aSym \
  --basil \
  --fs-license-file /freesurfer_license/license.txt \
  --work-dir /work

Version:

0.7.4

Environment (Docker, Singularity / Apptainer, custom installation): Singularity

Singularity

Data formatted according to a validatable standard? Please provide the output of the validator:

PASTE VALIDATOR OUTPUT HERE

Relevant log outputs (up to 20 lines):

requests.exceptions.ConnectTimeout: HTTPSConnectionPool(host='templateflow.s3.amazonaws.com', port=443): Max retries exceeded with url: /tpl-MNI152NLin2009aSym/tpl-MNI152NLin2009aSym_res-1_T1w.nii.gz (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x2b55e3eef010>, 'Connection to templateflow.s3.amazonaws.com timed out. (connect timeout=10)'))

Screenshots / relevant information:

Steven · November 21, 2024, 1:26am

Hi @cindylucerog,

Did you set export SINGULARITYENV_TEMPLATEFLOW_HOME=/templateflow? And have you used datalad get to download the files before beginning the job, as I am assuming your compute nodes have no internet?

Best,
Steven

cindylucerog · November 21, 2024, 1:36am

Hi Steven,

I think I do. Before running ASLprep I export two paths:

export FS_LICENSE=/home/m/mchakrav/cindylu/scratch/analysis/ASLprep/freesurferlicense/license.txt
export TEMPLATEFLOW_HOME=/home/m/mchakrav/cindylu/scratch/analysis/ASLprep/templateflow

singularity run --cleanenv \
  --bind /home/m/mchakrav/cindylu/scratch/analysis/ASLprep/data3subjects:/data:ro \
  --bind /home/m/mchakrav/cindylu/scratch/analysis/ASLprep/output3subjects:/out \
  --bind /home/m/mchakrav/cindylu/scratch/analysis/ASLprep/intermediatefiles:/work \
  --bind /home/m/mchakrav/cindylu/scratch/analysis/ASLprep/freesurferlicense:/freesurfer_license \
  --bind /home/m/mchakrav/cindylu/scratch/analysis/ASLprep/templateflow:/templateflow \
  /home/m/mchakrav/cindylu/scratch/analysis/aslprep-0.7.4.simg \
  /data /out participant \
  --output-spaces T1w MNI152NLin2009aSym \
  --basil \
  --fs-license-file /freesurfer_license/license.txt \
  --work-dir /work

I first downloaded the templateflow folder on my personal laptop and uploaded it to the cluster, this is advised here: FAQ - Frequently Asked Questions — fmriprep version documentation

Steven · November 21, 2024, 1:38am

Hi @cindylucerog,

You have to set SINGULARITYENV_TEMPLATEFLOW_HOME, and it should be set to where the templateflow directory will be inside the container (e.g., you rename it to /templateflow).

When you first download templateflow, everything is only symlinks. Files/folder within need to be retrieved with datalad get.

Best,
Steven

cindylucerog · November 21, 2024, 2:37am

Hi,

I verified I had the Niftii files, and I rerun, finding the same issue. Do you know of other resources? thank you very much for your time helping me.

export FS_LICENSE=/home/m/mchakrav/cindylu/scratch/analysis/ASLprep/freesurferlicense/license.txt
export TEMPLATEFLOW_HOME=/home/m/mchakrav/cindylu/scratch/analysis/ASLprep/templateflow
export APPTAINERENV_FS_LICENSE=/freesurfer_license/license.txt
export APPTAINERENV_TEMPLATEFLOW_HOME=/templateflow

singularity run --cleanenv \
  --bind /home/m/mchakrav/cindylu/scratch/analysis/ASLprep/data3subjects:/data:ro \
  --bind /home/m/mchakrav/cindylu/scratch/analysis/ASLprep/output3subjects:/out \
  --bind /home/m/mchakrav/cindylu/scratch/analysis/ASLprep/intermediatefiles:/work \
  --bind /home/m/mchakrav/cindylu/scratch/analysis/ASLprep/freesurferlicense:/freesurfer_license \
  --bind /home/m/mchakrav/cindylu/scratch/analysis/ASLprep/templateflow:/templateflow \
  /home/m/mchakrav/cindylu/scratch/analysis/aslprep-0.7.4.simg \
  /data /out participant \
  --output-spaces T1w MNI152NLin2009aSym \
  --basil \
  --fs-license-file /freesurfer_license/license.txt \
  --work-dir /work

error:

Matplotlib created a temporary cache directory at /tmp/matplotlib-eab1xei6 because the default path (/home/m/mchakrav/cindylu/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing.
(node:66976) Warning: Closing directory handle on garbage collection
(Use `node --trace-warnings ...` to show where the warning was created)
Downloading https://templateflow.s3.amazonaws.com/tpl-MNI152NLin2009aSym/tpl-MNI152NLin2009aSym_res-1_T1w.nii.gz
Process Process-2:
Traceback (most recent call last):
  File "/opt/conda/envs/aslprep/lib/python3.11/site-packages/urllib3/connection.py", line 198, in _new_conn
    sock = connection.create_connection(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/aslprep/lib/python3.11/site-packages/urllib3/util/connection.py", line 85, in create_connection
    raise err
  File "/opt/conda/envs/aslprep/lib/python3.11/site-packages/urllib3/util/connection.py", line 73, in create_connection
    sock.connect(sa)
TimeoutError: timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/conda/envs/aslprep/lib/python3.11/site-packages/urllib3/connectionpool.py", line 793, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/aslprep/lib/python3.11/site-packages/urllib3/connectionpool.py", line 491, in _make_request
    raise new_e
  File "/opt/conda/envs/aslprep/lib/python3.11/site-packages/urllib3/connectionpool.py", line 467, in _make_request
    self._validate_conn(conn)
  File "/opt/conda/envs/aslprep/lib/python3.11/site-packages/urllib3/connectionpool.py", line 1099, in _validate_conn
    conn.connect()
  File "/opt/conda/envs/aslprep/lib/python3.11/site-packages/urllib3/connection.py", line 616, in connect
    self.sock = sock = self._new_conn()
                       ^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/aslprep/lib/python3.11/site-packages/urllib3/connection.py", line 207, in _new_conn
    raise ConnectTimeoutError(
urllib3.exceptions.ConnectTimeoutError: (<urllib3.connection.HTTPSConnection object at 0x2b8ca38f5c90>, 'Connection to templateflow.s3.amazonaws.com timed out. (connect timeout=10)')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/conda/envs/aslprep/lib/python3.11/site-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "/opt/conda/envs/aslprep/lib/python3.11/site-packages/urllib3/connectionpool.py", line 847, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/aslprep/lib/python3.11/site-packages/urllib3/util/retry.py", line 515, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='templateflow.s3.amazonaws.com', port=443): Max retries exceeded with url: /tpl-MNI152NLin2009aSym/tpl-MNI152NLin2009aSym_res-1_T1w.nii.gz (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x2b8ca38f5c90>, 'Connection to templateflow.s3.amazonaws.com timed out. (connect timeout=10)'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/envs/aslprep/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/opt/conda/envs/aslprep/lib/python3.11/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/conda/envs/aslprep/lib/python3.11/site-packages/aslprep/cli/workflow.py", line 101, in build_workflow
    retval['workflow'] = init_aslprep_wf()
                         ^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/aslprep/lib/python3.11/site-packages/aslprep/workflows/base.py", line 68, in init_aslprep_wf
    single_subject_wf = init_single_subject_wf(subject_id)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/aslprep/lib/python3.11/site-packages/aslprep/workflows/base.py", line 275, in init_single_subject_wf
    anat_fit_wf = init_anat_fit_wf(
                  ^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/aslprep/lib/python3.11/site-packages/smriprep/workflows/anatomical.py", line 726, in init_anat_fit_wf
    anat_reports_wf = init_anat_reports_wf(
                      ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/aslprep/lib/python3.11/site-packages/smriprep/workflows/outputs.py", line 137, in init_anat_reports_wf
    template_iterator_wf = init_template_iterator_wf(spaces=spaces, sloppy=sloppy)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/aslprep/lib/python3.11/site-packages/smriprep/workflows/outputs.py", line 1150, in init_template_iterator_wf
    fetch_template_files(template, specs=None, sloppy=sloppy)
  File "/opt/conda/envs/aslprep/lib/python3.11/site-packages/smriprep/interfaces/templateflow.py", line 205, in fetch_template_files
    files['t1w'] = tf.get(name[0], desc=None, suffix='T1w', **specs)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/aslprep/lib/python3.11/site-packages/templateflow/conf/__init__.py", line 72, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/aslprep/lib/python3.11/site-packages/templateflow/api.py", line 174, in get
    _s3_get(filepath)
  File "/opt/conda/envs/aslprep/lib/python3.11/site-packages/templateflow/api.py", line 329, in _s3_get
    r = requests.get(url, stream=True, timeout=TF_GET_TIMEOUT)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/aslprep/lib/python3.11/site-packages/requests/api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/aslprep/lib/python3.11/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/aslprep/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/aslprep/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/aslprep/lib/python3.11/site-packages/requests/adapters.py", line 507, in send
    raise ConnectTimeout(e, request=request)
requests.exceptions.ConnectTimeout: HTTPSConnectionPool(host='templateflow.s3.amazonaws.com', port=443): Max retries exceeded with url: /tpl-MNI152NLin2009aSym/tpl-MNI152NLin2009aSym_res-1_T1w.nii.gz (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x2b8ca38f5c90>, 'Connection to templateflow.s3.amazonaws.com timed out. (connect timeout=10)'))

Best regards,
-Cindy

Steven · November 21, 2024, 2:45am

Hi @cindylucerog,

In the future, please format your code with the </> button in the text editor (or surrounding your text in tickmarks: `). You can see I have edited your posts for you so far.

Can you verify they are full files (e.g., reporting the file sizes of the MNI brain and displaying it in an image viewer)?

Best,
Steven

cindylucerog · November 23, 2024, 2:02pm

Thank you for advising how to frame my code Steven, and for the help with the error. You were correct, my templates were not properly downloaded. In case this helps others, to download them I had to:

Step 1: Activate Python environment
source /path/to/your/env/bin/activate

Step 2: Load required module
module load git-annex/8.20200618

Step 3: Clone the TemplateFlow dataset
datalad clone -r ///templateflow

Step 4: Navigate into the repository
cd templateflow

Step 5: Fetch all files recursively
datalad get -r .

The error disappeared after all the templates had been downloaded. For reference, this was my script:

#!/bin/bash
#SBATCH --job-name=ASLprep
#SBATCH --output=ASLprep.out    # Output file
#SBATCH --error=ASLprep.err     # Error file
#SBATCH --ntasks=1                         # Specify 1 task
#SBATCH --nodes=1                          # Request 1 node
#SBATCH --time=05:30:00
#SBATCH --output=aslprep_%j.log
#SBATCH --cpus-per-task=16
#SBATCH --mail-user=punkigirl5@gmail.com
#SBATCH --mail-type=ALL

export FS_LICENSE=/home/m/mchakrav/cindylu/scratch/analysis/ASLprep/freesurferlicense/license.txt
export TEMPLATEFLOW_HOME=/home/m/mchakrav/cindylu/scratch/analysis/templateflow
export APPTAINERENV_FS_LICENSE=/freesurfer_license/license.txt
export APPTAINERENV_TEMPLATEFLOW_HOME=/templateflow


singularity run --cleanenv \
  --env TEMPLATEFLOW_HOME=/templateflow \
  --bind /home/m/mchakrav/cindylu/scratch/analysis/ASLprep/data3subjects:/data:ro \
  --bind /home/m/mchakrav/cindylu/scratch/analysis/ASLprep/output3subjects:/out \
  --bind /home/m/mchakrav/cindylu/scratch/analysis/ASLprep/intermediatefiles:/work \
  --bind /home/m/mchakrav/cindylu/scratch/analysis/ASLprep/freesurferlicense:/freesurfer_license \
  --bind /home/m/mchakrav/cindylu/scratch/analysis/templateflow:/templateflow \
  /home/m/mchakrav/cindylu/scratch/analysis/ASLprep/aslprep-0.7.4.simg \
  /data /out participant \
  --skip_bids_validation \
  --output-spaces T1w \
  --basil \
  --fs-license-file /freesurfer_license/license.txt \
  --work-dir /work

Best regards,

-Cindy