Mriqc No space left on device error

Jeremy_Purcell · July 30, 2020, 7:15am

Hello,
I seem to be running out of space when trying to run anatomical images through mriqc. The functional data is run through fine. I am using the following command:
singularity run --cleanenv /home/mriqc_latest.simg /home/TPC_Project/bids /home/TPC_Project/fmriprep/mriqc participant --participant_label tpc000 --work-dir /home/TPC_Project/work_directory -m T1w T2w

Below is the output. The most relevant section is bolded below. I am guessing that I may need to expand the size of my singularity image? I am using linux Centos 7. The current size of the image is 2.8G. I’m not sure if this is/was the right direction to go in but, I did try to expand the singularity image using “singularity image.expand mriqc_latest.simg”, but I get an error (“This is not a writable Singularity image format”). Thanks for any input!!

200730-06:36:52,796 cli IMPORTANT:
Running MRIQC version 0.15.2:
* BIDS dataset path: /home/TPC_Project/bids.
* Output folder: /home/TPC_Project/fmriprep/mriqc.
* Analysis levels: [‘participant’].

200730-06:37:07,136 cli WARNING:
IMPORTANT: Anonymized quality metrics (IQMs) will be submitted to MRIQC’s metrics repository. Submission of IQMs can be disabled using the --no-sub argument. Please visit https://mriqc.readthedocs.io/en/latest/dsa.html to revise MRIQC’s Data Sharing Agreement.
Could not check for version updates:
Connection to server could not be made
The use of template_resolution is deprecated
Downloading https://templateflow.s3.amazonaws.com/tpl-MNI152NLin2009cAsym/tpl-MNI152NLin2009cAsym_res-02_T2w.nii.gz

0%| | 0.00/1.68k [00:00<?, ?B/s]
61%|██████████████████████████████████████████████████████████████████████████████████████████████████████████▊ | 1.02k/1.68k [00:00<00:00, 11.2kB/s]
200730-06:37:32,531 nipype.workflow WARNING:
Storing result file without outputs
200730-06:37:32,532 nipype.workflow WARNING:
[Node] Error on “mriqc_wf.anatMRIQC.SpatialNormalization.SpatialNormalization” (/home/jpurcel8/MRI_DATA/TPC_Project/work_directory/mriqc_wf/anatMRIQC/SpatialNormalization/in_file…home…TPC_Project…bids…sub-tpc000…ses-mnc…anat…sub-tpc000_ses-mnc_acq-1norm_T2w.nii.gz/SpatialNormalization)
200730-06:37:33,312 nipype.workflow ERROR:
** Node SpatialNormalization.a1 failed to run on host **
200730-06:37:33,319 nipype.workflow ERROR:
** Saving crash info to /home/TPC_Project/fmriprep/mriqc/logs/crash-20200730-063733-jpurcel8-SpatialNormalization.a1-08809e5c-9dd3-4514-b483-53893b375b0f.txt**
Traceback (most recent call last):
** File “/usr/local/miniconda/lib/python3.7/site-packages/templateflow/api.py”, line 192, in _s3_get**
** f.write(data)**
OSError: [Errno 28] No space left on device

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py”, line 67, in run_node
result[“result”] = node.run(updatehash=updatehash)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py”, line 516, in run
result = self._run_interface(execute=True)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py”, line 635, in _run_interface
return self._run_command(execute)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py”, line 741, in _run_command
result = self._interface.run(cwd=outdir)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py”, line 397, in run
runtime = self._run_interface(runtime)
File “/usr/local/miniconda/lib/python3.7/site-packages/niworkflows/interfaces/mni.py”, line 125, in _run_interface
ants_args = self._get_ants_args()
File “/usr/local/miniconda/lib/python3.7/site-packages/niworkflows/interfaces/mni.py”, line 354, in _get_ants_args
default_resolution=default_resolution)
File “/usr/local/miniconda/lib/python3.7/site-packages/niworkflows/utils/misc.py”, line 53, in get_template_specs
tpl_target_path = get_template(in_template, **template_spec)
File “/usr/local/miniconda/lib/python3.7/site-packages/templateflow/api.py”, line 67, in get
_s3_get(filepath)
File “/usr/local/miniconda/lib/python3.7/site-packages/templateflow/api.py”, line 192, in _s3_get
f.write(data)
OSError: [Errno 28] No space left on device

Traceback (most recent call last):
File “/usr/local/miniconda/bin/mriqc”, line 10, in
sys.exit(main())
File “/usr/local/miniconda/lib/python3.7/site-packages/mriqc/cli/run.py”, line 69, in main
mriqc_wf.run(**config.nipype.get_plugin())
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/workflows.py”, line 632, in run
runner.run(execgraph, updatehash=updatehash, config=self.config)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/base.py”, line 164, in run
self._clean_queue(jobid, graph, result=result)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/base.py”, line 227, in _clean_queue
raise RuntimeError("".join(result[“traceback”]))
RuntimeError: Traceback (most recent call last):
File “/usr/local/miniconda/lib/python3.7/site-packages/templateflow/api.py”, line 192, in _s3_get
f.write(data)
OSError: [Errno 28] No space left on device

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py”, line 67, in run_node
result[“result”] = node.run(updatehash=updatehash)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py”, line 516, in run
result = self._run_interface(execute=True)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py”, line 635, in _run_interface
return self._run_command(execute)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py”, line 741, in _run_command
result = self._interface.run(cwd=outdir)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py”, line 397, in run
runtime = self._run_interface(runtime)
File “/usr/local/miniconda/lib/python3.7/site-packages/niworkflows/interfaces/mni.py”, line 125, in _run_interface
ants_args = self._get_ants_args()
File “/usr/local/miniconda/lib/python3.7/site-packages/niworkflows/interfaces/mni.py”, line 354, in _get_ants_args
default_resolution=default_resolution)
File “/usr/local/miniconda/lib/python3.7/site-packages/niworkflows/utils/misc.py”, line 53, in get_template_specs
tpl_target_path = get_template(in_template, **template_spec)
File “/usr/local/miniconda/lib/python3.7/site-packages/templateflow/api.py”, line 67, in get
_s3_get(filepath)
File “/usr/local/miniconda/lib/python3.7/site-packages/templateflow/api.py”, line 192, in _s3_get
f.write(data)
OSError: [Errno 28] No space left on device

Jeremy_Purcell · August 3, 2020, 2:21am

FYI, the solution was to pre-download the templateflow folder and put it in my home directory, and then put “setenv SINGULARITYENV_TEMPLATEFLOW_HOME /home/templateflow” at the start of my mriqc run command. This is similar to what was needed previously for fmriprep, as noted here: Problems using pediatric template from TemplateFlow
I’m not sure why I was only getting this error when using mriqc (not fmriprep) and even then only for anatomical analyses, but this seemed to fix it.
Best,
Jeremy

pan · December 21, 2021, 9:26am

Hi @Jeremy_Purcell,
can you please copy the command you gave including the SINGULARITYENV_…? I don’t understand how I should run this.

I try to use this suggestion to run fmriprep. It was running before but now gives me this “OSError: [Errno 28] No space left on device”.

I have set my workingdir & have available space. Our HPC has recently an SSL issue i imagine this may be related.

I run:

singularity --verbose  run -B /storage/users, $HOME/.cache/templateflow:/opt/templateflow /usr/local/bin/fmriprep-20.2.6.simg $bids_root_dir $out_dir participant \
--participant-label $subj \
--bids-filter-file /storage/users/iliopoul/PHD/data/bids_filter_memtrain2_run1.json \
-w $workdir \
--md-only-boilerplate \
--fs-license-file /storage/users/iliopoul/fs_license.txt \
--fs-no-reconall \
--output-spaces MNI152NLin2009cAsym:res-2 \
--verbose \
--stop-on-first-crash \
-t itemobject \
--write-graph \