fMRIPrep 1.5.0 fails in singularity container with Read-only file system

1

I get:

Traceback (most recent call last):
  File "/usr/local/miniconda/bin/fmriprep", line 10, in <module>
    sys.exit(main())
  File "/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/cli/run.py", line 311, in main
    opts = get_parser().parse_args()
  File "/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/cli/run.py", line 284, in get_parser
    latest = check_latest()
  File "/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/cli/version.py", line 22, in check_latest
    cachefile.parent.mkdir(parents=True, exist_ok=True)
  File "/usr/local/miniconda/lib/python3.7/pathlib.py", line 1241, in mkdir
    self._accessor.mkdir(self, mode)
OSError: [Errno 30] Read-only file system: '/home/fmriprep/.cache/fmriprep'

This was the command run:

/cm/shared/singularity/bin/singularity run \
    -B /data:/data -B /data1:/data1 -B /data2:/data2 -B /data3:/data3 -B /cm/shared:/cm/shared \
    --cleanenv \
    /cm/shared/singularity/images/fmriprep-1.5.0.simg \
    /data/ajanes/testing \
    /data/ajanes/testing/derivatives \
    participant \
    --fs-license-file /cm/shared/freesurfer-6.0.1/license.txt \
    --participant_label acj \
    --output-spaces "MNI152NLin2009cAsym:res-2 anat func fsaverage" \
    --n_cpus 4 \
    --mem-mb 8192 \
    --use-plugin /data/ddrucker/workaround.yml \
    --use-aroma --ignore-aroma-denoising-errors \
    --use-syn-sdc \
    -w  /data/ajanes/testing/fmriprep-work
2

@dmd did you happen to download the image off of singularity hub? I’ve seen this issue from kasbohm’s image. What works for me is to add:

-B /some/scratch/dir:/home/fmriprep

Let me know if that works out or if the error is still thrown.

3

Yes, that causes it it to not immediately die. (I can’t speak to what happens over the next few hours yet, but at least it doesn’t die on launch.)

4

Now it fails with

requests.exceptions.ConnectionError: HTTPSConnectionPool(host='templateflow.s3.amazonaws.com', port=443): Max retries exceeded with url: /tpl-OASIS30ANTs/tpl-OASIS30ANTs_res-01_T1w.nii.gz (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x2aaacc203198>: Failed to establish a new connection: [Errno 110] Connection timed out'))

but that’s a different issue.

5

I was able to download https://templateflow.s3.amazonaws.com/tpl-OASIS30ANTs/tpl-OASIS30ANTs_res-01_T1w.nii.gz from a shell using the same image.

Can you attempt to download it with curl from the network that you are running the singularity container on to confirm you can reach it? If you can curl it outside of the singularity image but not from inside of it that might narrow the issue down. If the only place that url is not accessible is from inside the singularity container it might be worth trying to recreate the singularity image from docker manually:
https://fmriprep.readthedocs.io/en/stable/installation.html#singularity-container

6

Ok, sorry, looks like the problem is our compute nodes don’t have internet access. That said, the 1.4.1 image did not attempt to download things from the internet; is there a reason for this change?

7

Assuming this is something 1.5.0 needs, is there a way to tell the process inside the container to use a proxy?

8

Looks like you can download the templates to a directory mounted in singularity and fmriprep will read an environment variable on where to look for the templates:

1 Like