Fmriprep singularity path does not exist error

,
1

Hi there,

I’m running fmriprep on my hospital HPC by slrum, at first it ran perfectly fine under my home directory (input, output, freesurfer license and fmriprep image all under home dir)

However when I changed the output directory to the scratch, it gave me some error like this: OSError: [Errno 30] Read-only file system: ‘/gpfs/scratch’

So I added a line to bind the scratch dir and home dir together:

bids_dir=/gpfs/home/suw08/Desktop/PE/bidsdata
out_dir=/gpfs/scratch/suw08/outputdata
file_dir=/gpfs/home/suw08/my_images
singularity run --cleanenv
–bind /gpfs/scratch/suw08:/gpfs/home/suw08
/$file_dir/fmriprep-20.2.3.simg
–fs-license-file $file_dir/license.txt --notrack --verbose
–participant-label sub-00$1
–nthreads 6 --omp-nthreads 6
–longitudinal --ignore slicetiming --bold2t1w-dof 6
–output-spaces anat func fsnative fsaverage:den-41k
MNI152NLin2009cAsym MNI152NLin6Asym
–work-dir $out_dir/work
/$bids_dir $out_dir/ participant

Now it gave me another error:
fmriprep: error: Path does not exist: <//gpfs/home/suw08/my_images/license.txt>.

However, indeed the license.txt did exist under this directory!

[suw08@cn-0038 my_images]$ ls
fmriprep-20.2.3.simg license.txt

Any idea how to fix this?

2

Hi @WenjunSu,

Why do you rename the drive like this?

I would not put working directory in the output directory. Try to put it in a different folder.

Try changing it to this:

bids_dir=/gpfs/home/suw08/Desktop/PE/bidsdata
out_dir=/gpfs/scratch/suw08/outputdata
file_dir=/gpfs/home/suw08/my_images
work_dir=/path/to/work/dir # REPLACE WITH YOUR OWN
singularity run -e --containall
-B ${bids_dir},${out_dir},${file_dir},${work_dir}
/$file_dir/fmriprep-20.2.3.simg
–fs-license-file $file_dir/license.txt --notrack --verbose
–participant-label sub-00$1
–nthreads 6 --omp-nthreads 6
–longitudinal --ignore slicetiming --bold2t1w-dof 6
–output-spaces anat func fsnative fsaverage:den-41k
MNI152NLin2009cAsym MNI152NLin6Asym
–work-dir ${work_dir}
/$bids_dir $out_dir/ participant

Best,
Steven

3

Hi Steven! Thank you very much. I followed your steps and it showed another error: OSError: [Errno 28] No space left on device. I tried to set the work directory under the scratch and lab directory and neither of them worked.
However, I do believe there is enough space on my scratch or my lab directory. Any ideas how to fix that?

Loading singularity/3.9.8
Loading requirement: git/2.17.0 squashfs-tools/4.3 libarchive/3.3.2
openmpi/gcc/64/3.1.0 go/1.16
You are using fMRIPrep-20.2.3, and a newer version of fMRIPrep is available: 23.1.4.
Please check out our documentation about how and when to upgrade:
https://fmriprep.readthedocs.io/en/latest/faq.html#upgrading
Downloading https://templateflow.s3.amazonaws.com/tpl-OASIS30ANTs/tpl-OASIS30ANTs_res-01_T1w.nii.gz

0%| | 0.00/32.4M [00:00<?, ?B/s]
2%|▏ | 683k/32.4M [00:00<00:04, 6.80MB/s]
7%|▋ | 2.30M/32.4M [00:00<00:02, 12.3MB/s]
12%|█▏ | 4.00M/32.4M [00:00<00:01, 14.4MB/s]
18%|█▊ | 5.66M/32.4M [00:00<00:01, 15.3MB/s]
23%|██▎ | 7.37M/32.4M [00:00<00:01, 15.9MB/s]
28%|██▊ | 9.08M/32.4M [00:00<00:01, 16.3MB/s]
33%|███▎ | 10.7M/32.4M [00:00<00:01, 16.4MB/s]
38%|███▊ | 12.4M/32.4M [00:00<00:01, 16.5MB/s]
44%|████▎ | 14.1M/32.4M [00:00<00:01, 16.5MB/s]
48%|████▊ | 15.7M/32.4M [00:00<00:01, 15.7MB/s]
Process Process-2:
Traceback (most recent call last):
File “/usr/local/miniconda/lib/python3.7/site-packages/templateflow/api.py”, line 233, in _s3_get
f.write(data)
OSError: [Errno 28] No space left on device

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/usr/local/miniconda/lib/python3.7/multiprocessing/process.py”, line 297, in _bootstrap
self.run()
File “/usr/local/miniconda/lib/python3.7/multiprocessing/process.py”, line 99, in run
self._target(*self._args, **self._kwargs)
File “/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/cli/workflow.py”, line 82, in build_workflow
retval[“workflow”] = init_fmriprep_wf()
File “/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/workflows/base.py”, line 64, in init_fmriprep_wf
single_subject_wf = init_single_subject_wf(subject_id)
File “/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/workflows/base.py”, line 252, in init_single_subject_wf
t1w=subject_data[‘t1w’],
File “/usr/local/miniconda/lib/python3.7/site-packages/smriprep/workflows/anatomical.py”, line 321, in init_anat_preproc_wf
normalization_quality=‘precise’ if not debug else ‘testing’)
File “/usr/local/miniconda/lib/python3.7/site-packages/niworkflows/anat/ants.py”, line 180, in init_brain_extraction_wf
in_template, template_spec=template_spec
File “/usr/local/miniconda/lib/python3.7/site-packages/niworkflows/utils/misc.py”, line 58, in get_template_specs
tpl_target_path = get_template(in_template, **template_spec)
File “/usr/local/miniconda/lib/python3.7/site-packages/templateflow/api.py”, line 78, in get
_s3_get(filepath)
File “/usr/local/miniconda/lib/python3.7/site-packages/templateflow/api.py”, line 234, in _s3_get
t.update(len(data))
OSError: [Errno 28] No space left on device

4

Hi @WenjunSu,

Promising move forward! This might depend on whatever computing system you use, but what I would guess, based on other times I’ve seen this message, is that Templateflow is writing to your home drive (e.g. the ~ drive), and in many high performance computing systems, that is very space-limited. You can change where templateflow downloads by defining SINGULARITYENV_TEMPLATEFLOW_HOME before the fmriprep command. Wherever you set it, make sure to bind it in the fmriprep command too!

Best,
Steven