fmriPrep 1.5.1rc1 EOFError: Ran out of input

azg · November 2, 2019, 2:42am

Hello! Have been using fmriPrep 1.51.rc1 since its been available and ran through 6 or so subs successfully. But since then have run into the following errors for any batch I run. Any suggestions much appreciated

191031-22:51:06,897 nipype.workflow WARNING
Storing result file without outputs
191031-22:51:06,912 nipype

.workflow WARNING:
[Node] Error on “fmriprep_wf.single_subject_40_wf.anat_preproc_wf.anat_norm_wf.registration” (/tmp/work/fmriprep_wf/single_subject_40_wf/anat_preproc_wf/anat_norm_wf/_template_MNI152NLin2009cAsym/registration)
191031-22:51:06,934 nipype.utils WARNING:
No metadata was found in the pkl file. Make sure you are currently using the same Nipype version from the generated pkl.
exception calling callback for <Future at 0x7f62b337ef60 state=finished raised EOFError>
concurrent.futures.process._RemoteTraceback:
“”"
OSError: [Errno 28] No space left on device

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py”, line 662, in _run_command
result = self._interface.run(cwd=outdir)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py”, line 382, in run
runtime = self._run_interface(runtime)
File “/usr/local/miniconda/lib/python3.7/site-packages/niworkflows/interfaces/mni.py”, line 162, in _run_interface
print(cmd + “\n”, file=cmdfile)
OSError: [Errno 28] No space left on device

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py”, line 69, in run_node
result[‘result’] = node.run(updatehash=updatehash)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py”, line 479, in run
result = self._run_interface(execute=True)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py”, line 569, in _run_interface
return self._run_command(execute)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py”, line 668, in _run_command
rebase=str2bool(self.config[‘execution’][‘use_relative_paths’]))
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/utils.py”, line 239, in save_resultfile
savepkl(resultsfile, result)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/utils/filemanip.py”, line 795, in savepkl
pickle.dump(record, pkl_file)
File “/usr/local/miniconda/lib/python3.7/gzip.py”, line 319, in close
myfileobj.close()
OSError: [Errno 28] No space left on device

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/usr/local/miniconda/lib/python3.7/concurrent/futures/process.py”, line 232, in process_worker
r = call_item.fn(*call_item.args, **call_item.kwargs)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py”, line 72, in run_node
result[‘result’] = node.result
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py”, line 198, in result
op.join(self.output_dir(), 'result%s.pklz’ % self.name))
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/utils.py”, line 296, in load_resultfile
result = loadpkl(results_file)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/utils/filemanip.py”, line 744, in loadpkl
raise e
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/utils/filemanip.py”, line 726, in loadpkl
unpkl = pickle.loads(pkl_contents)
EOFError: Ran out of input
“”"

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File “/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py”, line 324, in _invoke_callbacks
callback(self)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py”, line 149, in _async_callback
result = args.result()
File “/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py”, line 425, in result
return self.__get_result()
File “/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py”, line 384, in __get_result
raise self._exception
EOFError: Ran out of input

oesteban · November 2, 2019, 11:02pm

It seems you run out of space on the filesystem you set your working directory.

azg · November 3, 2019, 10:17pm

I’ve been getting this error on and off while using fmriprep, but the device seems to have plenty of storage (a little over 200 gb) so I’m not sure why it says that. I’ve had some instances where it says that but then I rerun the subject and it completes without issue.

oesteban · November 4, 2019, 10:39pm

How many subjects are you trying to process? Are you positive you checked the working directory? What is the command line you are using? What platform (containers, custom installation, etc.)?

wem3 · November 22, 2019, 3:58pm

I’ve received a similar error for several subjects (when trying to process about 40 subjects on an HPC via SLURM job array). I haven’t been able to reproduce it consistently, though. When I re-run fmriprep-1.5.0, subjects for whom the error was thrown previously seem to run without a hitch…

For each job, I’ve allocated 16 CPUs with 4GB RAM per CPU.

command line (for sub-08):
singularity run --cleanenv -B /ncf/cikaralab/earnBurn/ds_earnburn/inputs/data/nifti:/data -B /tmp/wem3/sub-08:/home/fmriprep -B /tmp/wem3/sub-08:/work -B /tmp/wem3/sub-08/.cache/templateflow:/templateflow /ncf/cikaralab/earnBurn/ds_earnburn/.datalad/environments/fmriprep/image /data /ncf/cikaralab/earnBurn/ds_earnburn/outputs/derivatives/fmriprep-1.5.0 participant --participant-label 08 -w /work/ -vv --omp-nthreads 8 --nthreads 16 --mem_mb 32000 --skull-strip-template MNI152NLin2009cAsym --output-spaces MNI152NLin2009cAsym:res-2 MNI152NLin6Asym:res-2 fsnative fsaverage5 --use-aroma --use-syn-sdc --fs-no-reconall --ignore slicetiming --resource-monitor --write-graph

OS: CentOS 7
fmriprep-1.5.0 (via Singularity container)
python 3.7 (via miniconda)

oesteban · December 10, 2019, 5:52pm

What is the exact error?

wem3 · December 10, 2019, 7:00pm

I think the relevant part of the error is:

exception calling callback for <Future at 0x2b245e038e80 state=finished raised EOFError>
concurrent.futures.process._RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 69, in run_node
    result['result'] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 479, in run
    result = self._run_interface(execute=True)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 569, in _run_interface
    return self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 684, in _run_command
    rebase=str2bool(self.config['execution']['use_relative_paths']))
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/utils.py", line 249, in save_resultfile
    savepkl(resultsfile, result)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/utils/filemanip.py", line 795, in savepkl
    pickle.dump(record, pkl_file)
  File "/usr/local/miniconda/lib/python3.7/gzip.py", line 319, in close
    myfileobj.close()
OSError: [Errno 28] No space left on device

but here’s the entire standard error file: fmriprep_5.err.txt (8.6 KB) :

wem3 · December 10, 2019, 7:05pm

Please accept my apology for the sloppy provenance, but I updated to 1.5.1rc2 and re-ran fmriprep without a problem for this subject, and so I’m not 100% certain whether or not the file attached above is the most relevant standard error (I just did grep -r "No space left on device" on my directory with old logs to find this one).