fMRIPrep in Compute Canada

Software Support
,
1

Summary of what happened:

Hello neuro aficionados!

I am trying to run fMRIPrep on some successfully BIDS formatted fMRI data of mine. I have asked one of the Digital Research Alliance of Canada employees to add a singularity image to one of my personal directories since I had difficulties setting it up myself. This image seems to operate well as I can successfully submit a fMRIPrep command.

Has anyone run into a similar issue and have any suggestions as to how to alleviate this issue?

Command used (and if a helper script was used, a link to the helper script or the command generated):

module restore my_modules

subject=032

source $FREESURFER_HOME/SetUpFreeSurfer.sh
source $FREESURFER_HOME/FreeSurferEnv.sh


license=/home/coopapap/.local/easybuild/software/2020/Core/freesurfer/6.0.1/license.txt
mkdir -p /project/6001995/coopapap/data/garrison/BIDS/out/

singularity run --cleanenv -B /project /project/6001995/coopapap/downloads/fmriprep/fmriprep-23.2.0.simg \
    /project/6001995/coopapap/data/garrison/BIDS/meditators \
    /project/6001995/coopapap/data/garrison/BIDS/out/ \
    participant \
    --participant-label $subject \
    --skip-bids-validation

Version:

23.2.0

Environment (Docker, Singularity / Apptainer, custom installation):

Singularity

Data formatted according to a validatable standard? Please provide the output of the validator:

PASTE VALIDATOR OUTPUT HERE

Relevant log outputs (up to 20 lines):

*ANAT Stage 2: Preparing brain extraction workflow*
*Downloading https://templateflow.s3.amazonaws.com/tpl-OASIS30ANTs/tpl-OASIS30ANTs_res-01_T1w.nii.gz*
*Process Process-2:*
*Traceback (most recent call last):*
*  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/urllib3/connection.py", line 203, in _new_conn*
*    sock = connection.create_connection(*
*  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/urllib3/util/connection.py", line 85, in create_connection*
*    raise err*
*  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/urllib3/util/connection.py", line 73, in create_connection*
*    sock.connect(sa)*
*ConnectionRefusedError: [Errno 111] Connection refused*

Screenshots / relevant information:

2
3

Hi @coopapap, and welcome to neurostars!

As stated on Compute Canada’s documentation:

In the context of the Alliance, SSH tunnelling is necessary in certain cases, because compute nodes on Niagara, Béluga and Graham do not have direct access to the Internet, nor can the compute nodes be contacted directly from the Internet.

On your local machine you can download the templateflow cache with datalad, and then transfer it to your machine with something like Cyberduck.io. Then you would need to make sure that directory is bound to your Singularity command with the -B argument, and that fmriprep knows where to look for it by defining the SINGULARITYENV_TEMPLATEFLOW_HOME=$templateflow_dir.

@bnintzan uses Compute Canada and I helped her with a similar issue, maybe she can send you her local templateflow directory.

Best,
Steven

4

got it! Thank you kindly Steven :slight_smile:
I’m in the process of downloading the templateflow directories with datalad.

In the meantime I just want to verify that I shoiuld now bind (-B) both my fmriprep singularity image and now also the location to the templateflow cache.

Right now I have my command looking like this:

singularity run --cleanenv \
    -B /project,$SINGULARITYENV_TEMPLATEFLOW_HOME \
    /project/6001995/coopapap/downloads/fmriprep/fmriprep-23.2.0.simg \
    /project/6001995/coopapap/data/garrison/BIDS/meditators \
    /project/6001995/coopapap/data/garrison/BIDS/out/ \
    participant \
    --participant-label $subject \
    --skip-bids-validation

Could you let me know if it looks right to you?

Thanks again for your very quick help, I appreciate it greatly!

Austin

5

Hi @coopapap,

Looks good to me! Although I recommend specifying a working directory (-w) and not skipping BIDS validation.

Best,
Steven

6

Hey Steven,

I still get the same type of error:

*240306-21:07:52,439 nipype.workflow IMPORTANT:
         Building fMRIPrep's workflow:
           * BIDS dataset path: /project/6001995/coopapap/data/garrison/BIDS/meditators.
           * Participant list: ['032'].
           * Run identifier: 20240306-210546_510cd0c4-7e67-4644-b342-5d60973ae6a7.
           * Output spaces: MNI152NLin2009cAsym:res-native.
           * Pre-run FreeSurfer's SUBJECTS_DIR: /project/6001995/coopapap/data/garrison/BIDS/out/sourcedata/freesurfer.
240306-21:07:53,426 nipype.workflow INFO:
         ANAT Stage 1: Adding template workflow
240306-21:07:54,346 nipype.workflow INFO:
         ANAT Stage 2: Preparing brain extraction workflow
Downloading https://templateflow.s3.amazonaws.com/tpl-OASIS30ANTs/tpl-OASIS30ANTs_res-01_T1w.nii.gz
Process Process-2:
Traceback (most recent call last):
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/urllib3/connection.py", line 203, in _new_conn
    sock = connection.create_connection(
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/urllib3/util/connection.py", line 85, in create_connection
    raise err
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/urllib3/util/connection.py", line 73, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

The above exception was the direct cause of the following exception:*

and the error goes on, but I figure this displays the most important information.

The main method I used for getting the templateflow cache was with the command: datalad install -r ///templateflow

I downloaded this and then additionally submitted this command: datalad get -r tpl-OASIS30ANTs

I then moved all these files from my computer to $SINGULARITYENV_TEMPLATEFLOW_HOME location in Compute Canada.

Again, the main command I submitted in compute canada is:

singularity run --cleanenv \
    -B /project \
    -B $SINGULARITYENV_TEMPLATEFLOW_HOME \
    /project/6001995/coopapap/downloads/fmriprep/fmriprep-23.2.0.simg \
    /project/6001995/coopapap/data/garrison/BIDS/meditators \
    /project/6001995/coopapap/data/garrison/BIDS/out/ \
    participant \
    --participant-label $subject

And I can certainly add a work directory, I’m just doubting that that’s the issue.

Any ideas?

7

Hi @coopapap,

Can you show your full script, including where you define $SINGULARITYENV_TEMPLATEFLOW_HOME? Also, try this templateflow repository I zipped up. It should have files you need, and more. templateflow.zip - Google Drive

Best,
Steven

8

Brilliant! Thanks so much!!

I’ll check out what you zipped.

Here is my entire script:

#!/bin/bash
#SBATCH --time=0-10:00
#SBATCH --account=def-amirs
#SBATCH --ntasks=1
#SBATCH --mem=20G
#SBATCH --cpus-per-task=1


module restore my_modules

subject=032

source $FREESURFER_HOME/SetUpFreeSurfer.sh
source $FREESURFER_HOME/FreeSurferEnv.sh


license=/home/coopapap/.local/easybuild/software/2020/Core/freesurfer/6.0.1/license.txt
mkdir -p /project/6001995/coopapap/data/garrison/BIDS/out/

export SINGULARITYENV_TEMPLATEFLOW_HOME=/project/6001995/coopapap/atlases/templateflow

singularity run --cleanenv \
    -B /project \
    -B $SINGULARITYENV_TEMPLATEFLOW_HOME \
    /project/6001995/coopapap/downloads/fmriprep/fmriprep-23.2.0.simg \
    /project/6001995/coopapap/data/garrison/BIDS/meditators \
    /project/6001995/coopapap/data/garrison/BIDS/out/ \
    participant \
    --participant-label $subject
9

Thanks @coopapap, keep us updated.

However,

That is not a lot a time (fmriprep takes on the order of hours-to-days depending on data input and resources), especially for a single CPU as you specify. Can you use more resources?

Best,
Steven

10

Yes, I will request more time once I can verify that the script runs successfully.

It’s just that I get the error after a couple of minutes.

Currently uploading the new templateflow directory and will test the script with this!

Will update shortly and thanks again!

Austin

11

Wow!

That seemed to do the trick and now it’s running:

[coopapap@bc11256 fmriprep]$ singularity run --cleanenv \
>     -B /project \
>     -B $SINGULARITYENV_TEMPLATEFLOW_HOME \
>     /project/6001995/coopapap/downloads/fmriprep/fmriprep-23.2.0.simg \
>     /project/6001995/coopapap/data/garrison/BIDS/meditators \
>     /project/6001995/coopapap/data/garrison/BIDS/out/ \
>     participant \
>     --participant-label $subject \
>     --fs-license-file $license
INFO:    Environment variable SINGULARITYENV_TEMPLATEFLOW_HOME is set, but APPTAINERENV_TEMPLATEFLOW_HOME is preferred
bids-validator@1.14.0
(node:38530) Warning: Closing directory handle on garbage collection
(Use `node --trace-warnings ...` to show where the warning was created)
        1: [WARN] The recommended file /README is missing. See Section 03 (Modality agnostic files) of the BIDS specification. (code: 101 - README_FILE_MISSING)

        Please visit https://neurostars.org/search?q=README_FILE_MISSING for existing conversations about this issue.

        2: [WARN] The Authors field of dataset_description.json should contain an array of fields - with one author per field. This was triggered because there are no authors, which will make DOI registration from dataset metadata impossible. (code: 113 - NO_AUTHORS)

        Please visit https://neurostars.org/search?q=NO_AUTHORS for existing conversations about this issue.

        Summary:                 Available Tasks:        Available Modalities:
        281 Files, 3.45GB        task-ca1                MRI
        21 - Subjects            task-ca2
        1 - Session              task-conc1
                                 task-conc2
                                 task-metta2
                                 task-metta1


        If you have any questions, please post on https://neurostars.org/tags/bids.
240306-22:05:30,35 nipype.workflow IMPORTANT:
         Running fMRIPrep version 23.2.0

         License NOTICE ##################################################
         fMRIPrep 23.2.0
         Copyright 2023 The NiPreps Developers.

         This product includes software developed by
         the NiPreps Community (https://nipreps.org/).

         Portions of this software were developed at the Department of
         Psychology at Stanford University, Stanford, CA, US.

         This software is also distributed as a Docker container image.
         The bootstrapping file for the image ("Dockerfile") is licensed
         under the MIT License.

         This software may be distributed through an add-on package called
         "Docker Wrapper" that is under the BSD 3-clause License.
         #################################################################
240306-22:05:30,584 nipype.workflow IMPORTANT:
         Building fMRIPrep's workflow:
           * BIDS dataset path: /project/6001995/coopapap/data/garrison/BIDS/meditators.
           * Participant list: ['032'].
           * Run identifier: 20240306-220407_13b8dc9e-d926-4e97-b282-68ce2b78e539.
           * Output spaces: MNI152NLin2009cAsym:res-native.
           * Pre-run FreeSurfer's SUBJECTS_DIR: /project/6001995/coopapap/data/garrison/BIDS/out/sourcedata/freesurfer.
240306-22:05:31,372 nipype.workflow INFO:
         ANAT Stage 1: Adding template workflow
240306-22:05:32,196 nipype.workflow INFO:
         ANAT Stage 2: Preparing brain extraction workflow
240306-22:05:32,320 nipype.workflow INFO:
         ANAT Stage 3: Preparing segmentation workflow
240306-22:05:32,328 nipype.workflow INFO:
         ANAT Stage 4: Preparing normalization workflow for ['MNI152NLin2009cAsym']
240306-22:05:32,341 nipype.workflow INFO:
         ANAT Stage 5: Preparing surface reconstruction workflow
240306-22:05:32,373 nipype.workflow INFO:
         ANAT Stage 6: Preparing mask refinement workflow
240306-22:05:32,377 nipype.workflow INFO:
         ANAT No T2w images provided - skipping Stage 7
240306-22:05:32,398 nipype.workflow INFO:
         ANAT Stage 8: Creating GIFTI surfaces for ['white', 'pial', 'midthickness', 'sphere_reg', 'sphere']
240306-22:05:32,439 nipype.workflow INFO:
         ANAT Stage 8: Creating GIFTI metrics for ['thickness', 'sulc']
240306-22:05:32,450 nipype.workflow INFO:
         ANAT Stage 8a: Creating cortical ribbon mask
240306-22:05:32,455 nipype.workflow INFO:
         ANAT Stage 9: Creating fsLR registration sphere
240306-22:05:32,461 nipype.workflow INFO:
         ANAT Stage 10: Creating MSM-Sulc registration sphere
240306-22:05:32,686 nipype.workflow INFO:
         No single-band-reference found for sub-032_task-ca1_bold.nii.gz.
240306-22:05:32,824 nipype.workflow INFO:
         Stage 1: Adding HMC boldref workflow
240306-22:05:32,835 nipype.workflow INFO:
         Stage 2: Adding motion correction workflow
240306-22:05:32,848 nipype.workflow INFO:
         Stage 3: Adding coregistration boldref workflow
240306-22:05:32,905 nipype.workflow IMPORTANT:
         BOLD series will be slice-timing corrected to an offset of 0.96s.
240306-22:05:33,61 nipype.workflow INFO:
         No single-band-reference found for sub-032_task-ca2_bold.nii.gz.
240306-22:05:33,127 nipype.workflow INFO:
         Stage 1: Adding HMC boldref workflow
240306-22:05:33,136 nipype.workflow INFO:
         Stage 2: Adding motion correction workflow
240306-22:05:33,143 nipype.workflow INFO:
         Stage 3: Adding coregistration boldref workflow
240306-22:05:33,195 nipype.workflow IMPORTANT:
         BOLD series will be slice-timing corrected to an offset of 0.96s.
240306-22:05:33,343 nipype.workflow INFO:
         No single-band-reference found for sub-032_task-conc1_bold.nii.gz.
240306-22:05:33,407 nipype.workflow INFO:
         Stage 1: Adding HMC boldref workflow
240306-22:05:33,417 nipype.workflow INFO:
         Stage 2: Adding motion correction workflow
240306-22:05:33,424 nipype.workflow INFO:
         Stage 3: Adding coregistration boldref workflow
240306-22:05:33,477 nipype.workflow IMPORTANT:
         BOLD series will be slice-timing corrected to an offset of 0.96s.
240306-22:05:33,625 nipype.workflow INFO:
         No single-band-reference found for sub-032_task-conc2_bold.nii.gz.
240306-22:05:33,689 nipype.workflow INFO:
         Stage 1: Adding HMC boldref workflow
240306-22:05:33,698 nipype.workflow INFO:
         Stage 2: Adding motion correction workflow
240306-22:05:33,706 nipype.workflow INFO:
         Stage 3: Adding coregistration boldref workflow
240306-22:05:33,759 nipype.workflow IMPORTANT:
         BOLD series will be slice-timing corrected to an offset of 0.96s.
240306-22:05:33,906 nipype.workflow INFO:
         No single-band-reference found for sub-032_task-metta1_bold.nii.gz.
240306-22:05:33,970 nipype.workflow INFO:
         Stage 1: Adding HMC boldref workflow
240306-22:05:33,980 nipype.workflow INFO:
         Stage 2: Adding motion correction workflow
240306-22:05:33,987 nipype.workflow INFO:
         Stage 3: Adding coregistration boldref workflow
240306-22:05:34,40 nipype.workflow IMPORTANT:
         BOLD series will be slice-timing corrected to an offset of 0.96s.
240306-22:05:34,186 nipype.workflow INFO:
         No single-band-reference found for sub-032_task-metta2_bold.nii.gz.
240306-22:05:34,252 nipype.workflow INFO:
         Stage 1: Adding HMC boldref workflow
240306-22:05:34,261 nipype.workflow INFO:
         Stage 2: Adding motion correction workflow
240306-22:05:34,268 nipype.workflow INFO:
         Stage 3: Adding coregistration boldref workflow
240306-22:05:34,321 nipype.workflow IMPORTANT:
         BOLD series will be slice-timing corrected to an offset of 0.96s.
240306-22:05:38,938 nipype.workflow INFO:
         fMRIPrep workflow graph with 1141 nodes built successfully.
240306-22:05:55,649 nipype.workflow IMPORTANT:
         fMRIPrep started!

How long do you suppose I should run it for without having it cut short?

12

Hi @coopapap,

Great!

It depends on how much data you’re running. In my first job for any dataset, I like to give it a lot of time and memory (like 64GB and 2 days), and check the time/memory usage after the fact to guide future jobs.

If you keep it as single threaded, expect up to multiple days.

Best,
Steven

13

Continuing the discussion from fMRIPrep in Compute Canada:

Hi Steven!

I’ve finally gotten the script to run through completely (it took a while for the hob to be accepted into Compute Canada queue.

I have a question that I hope will be quick for you to answer.

I’d like to have all my subjects fMRI data preprocessed into fsaverage space, and I see that is not now (I can only find data in the volume space). Will adding --output-space fsaverage to my job submission solve this? My job submission would then be this:

 singularity run --cleanenv \
    -B /project \
    -B $SINGULARITYENV_TEMPLATEFLOW_HOME \
    /project/6001995/coopapap/downloads/fmriprep/fmriprep-23.2.0.simg \
    /project/6001995/coopapap/data/garrison/BIDS/meditators \
    /project/6001995/coopapap/data/garrison/BIDS/out/ \
    participant \
    --participant-label $subject \
    --fs-license-file $license \
    --output-space fsaverage

I just want to make sure whether this is or is not possible as I see some other forum threads suggesting that I will need to run the volume to surface transformation manually.

Warm regards,
Austin

14

Hi @coopapap,

Yup!

Best,
Steven