Errors about .nfs busy and resource errors when running fmriprep through singularity

Ursula_Tooley · January 14, 2019, 11:42pm

I have been running into these errors, both IMPORTWARNINGS and SystemExit: 0 and the Device or resource busy errors. The import warnings seem not to affect performance, though I tried to solve by unsetting my PYTHONPATH–but no dice, they seem above my pay grade as some searching reveals they’re related to a Cython bug. I’m on an HPC using singularity containers.

The other two are what I’m more worried about. Output below (I took out the boilerplate). The wf continues to run, but I’m not sure where these errors come from? The command I’m using to run is also below.

90114-23:33:07,44 nipype.workflow IMPORTANT:

Running fMRIPREP version 1.2.5:

* BIDS dataset path: /mnt.

* Participant list: ['CBPD0002'].

* Run identifier: 20190114-233307_450ce602-5940-4814-a163-2accafc777fb.

190114-23:33:09,368 nipype.workflow IMPORTANT:

Creating bold processing workflow for "/mnt/sub-CBPD0002/func/sub-CBPD0002_task-rest_run-01_bold.nii.gz" (0.15 GB / 150 TRs). Memory resampled/largemem=0.61/0.84 GB.

190114-23:33:09,377 nipype.workflow IMPORTANT:

No single-band-reference found for sub-CBPD0002_task-rest_run-01_bold.nii.gz

190114-23:33:11,364 nipype.workflow IMPORTANT:

Slice-timing correction will be included.

190114-23:33:11,392 nipype.workflow WARNING:

SDC: no fieldmaps found or they were ignored (/mnt/sub-CBPD0002/func/sub-CBPD0002_task-rest_run-01_bold.nii.gz).

190114-23:33:12,783 nipype.workflow IMPORTANT:

Creating BOLD surface-sampling workflow.

/usr/local/miniconda/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__

return f(*args, **kwds)

/usr/local/miniconda/lib/python3.6/site-packages/skimage/__init__.py:80: ResourceWarning: unclosed file &lt;_io.TextIOWrapper name='/usr/local/miniconda/lib/python3.6/site-packages/pytest.py' mode='r' encoding='utf-8'&gt;

imp.find_module('pytest')

/usr/local/miniconda/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__

return f(*args, **kwds)

/usr/local/miniconda/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__

return f(*args, **kwds)

/usr/local/miniconda/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__

return f(*args, **kwds)

/usr/local/miniconda/lib/python3.6/site-packages/bids/layout/bids_layout.py:121: ResourceWarning: unclosed file &lt;_io.TextIOWrapper name='/mnt/dataset_description.json' mode='r' encoding='UTF-8'&gt;

self.description = json.load(open(target, 'r'))

/usr/local/miniconda/lib/python3.6/site-packages/grabbit/core.py:436: ResourceWarning: unclosed file &lt;_io.TextIOWrapper name='/usr/local/miniconda/lib/python3.6/site-packages/bids/layout/config/bids.json' mode='r' encoding='UTF-8'&gt;

domain = json.load(open(domain, 'r'))

/usr/local/miniconda/lib/python3.6/site-packages/bids/layout/bids_layout.py:208: ResourceWarning: unclosed file &lt;_io.TextIOWrapper name='/mnt/task-rest_bold.json' mode='r' encoding='utf-8'&gt;

encoding='utf-8'))

/usr/local/miniconda/lib/python3.6/site-packages/bids/layout/bids_layout.py:208: ResourceWarning: unclosed file &lt;_io.TextIOWrapper name='/mnt/sub-CBPD0002/func/sub-CBPD0002_task-rest_run-01_bold.json' mode='r' encoding='utf-8'&gt;

encoding='utf-8'))

/usr/local/miniconda/lib/python3.6/site-packages/bids/layout/bids_layout.py:208: ResourceWarning: unclosed file &lt;_io.TextIOWrapper name='/mnt/sub-CBPD0002/fmap/sub-CBPD0002_run-01_epi.json' mode='r' encoding='utf-8'&gt;

encoding='utf-8'))

Captured warning (&lt;class 'ImportWarning'&gt;): can't resolve package from __spec__ or __package__, falling back on __name__ and __path__

Captured warning (&lt;class 'ResourceWarning'&gt;): unclosed file &lt;_io.TextIOWrapper name='/usr/local/miniconda/lib/python3.6/site-packages/pytest.py' mode='r' encoding='utf-8'&gt;

Captured warning (&lt;class 'ImportWarning'&gt;): can't resolve package from __spec__ or __package__, falling back on __name__ and __path__

Captured warning (&lt;class 'ImportWarning'&gt;): can't resolve package from __spec__ or __package__, falling back on __name__ and __path__

Captured warning (&lt;class 'ImportWarning'&gt;): can't resolve package from __spec__ or __package__, falling back on __name__ and __path__

/usr/local/miniconda/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__

return f(*args, **kwds)

/usr/local/miniconda/lib/python3.6/site-packages/skimage/__init__.py:80: ResourceWarning: unclosed file &lt;_io.TextIOWrapper name='/usr/local/miniconda/lib/python3.6/site-packages/pytest.py' mode='r' encoding='utf-8'&gt;

imp.find_module('pytest')

/usr/local/miniconda/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__

return f(*args, **kwds)

/usr/local/miniconda/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__

return f(*args, **kwds)

/usr/local/miniconda/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__

return f(*args, **kwds)

Traceback (most recent call last):

File "/usr/local/miniconda/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap

self.run()

File "/usr/local/miniconda/lib/python3.6/multiprocessing/process.py", line 93, in run

self._target(*self._args, **self._kwargs)

File "/usr/local/miniconda/lib/python3.6/multiprocessing/managers.py", line 547, in _run_server

server.serve_forever()

File "/usr/local/miniconda/lib/python3.6/multiprocessing/managers.py", line 167, in serve_forever

sys.exit(0)

SystemExit: 0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

File "/usr/local/miniconda/lib/python3.6/multiprocessing/util.py", line 262, in _run_finalizers

finalizer()

File "/usr/local/miniconda/lib/python3.6/multiprocessing/util.py", line 186, in __call__

res = self._callback(*self._args, **self._kwargs)

File "/usr/local/miniconda/lib/python3.6/shutil.py", line 480, in rmtree

_rmtree_safe_fd(fd, path, onerror)

File "/usr/local/miniconda/lib/python3.6/shutil.py", line 438, in _rmtree_safe_fd

onerror(os.unlink, fullname, sys.exc_info())

File "/usr/local/miniconda/lib/python3.6/shutil.py", line 436, in _rmtree_safe_fd

os.unlink(name, dir_fd=topfd)

OSError: [Errno 16] Device or resource busy: '.nfs0000000000683d9e00005833'

190114-23:33:33,636 nipype.workflow WARNING:

Some nodes demand for more threads than available (1).

Captured warning (&lt;class 'PendingDeprecationWarning'&gt;): the matrix subclass is not the recommended way to represent matrices or deal with linear algebra (see https://docs.scipy.org/doc/numpy/user/numpy-for-matlab-users.html). Please adjust your code to use regular ndarray.

Command running fmriprep:
qsub -j y -l h_vmem=25.1G,s_vmem=25G ${SCRIPTS_DIR}/fmriprep_cmd.sh ${SUB}
with:

unset PYTHONPATH;
singularity run --cleanenv -B ${BIDS_folder}:/mnt,${working_dir}:/tmp ${tools_dir}/fmriprep-1.2.5.simg \
/mnt/ /mnt/derivatives \
participant \
-w /tmp \
--participant-label ${subject} \
--fs-license-file $HOME/license.txt \
--use-aroma \
--output-space T1w template fsaverage5 \
--skip-bids-validation \
--cifti-output \
--nthreads 1 \

ChrisGorgolewski · January 14, 2019, 11:48pm

Could you try setting the ${working_dir} to a local filesystem (not NFS)? /tmp is usually a good choice.

Ursula_Tooley · January 15, 2019, 2:18pm

That does it! I still get a bunch of import warnings, but I’m assuming I can ignore those safely. None of the other system exit errors or resource busy errors have popped up (as of yet, subjects are still running from last night).

Thanks so much! I assume I’ll just proceed with the assumption that I won’t need anything out of the working directory. I initially wanted to keep it someplace as in the past using Nipype, sometimes it was useful to go back and find something that was only saved in the working directory and not the output directory. But it seems like everything I’ll need should be there.

Ursula_Tooley · January 15, 2019, 2:25pm

A follow-up question–should I be worried about these errors affecting mriqc output? I ran my MRIQC singularity containers with /tmp bound to a directory, which generates the same errors, but the output seems okay (with the exception of those kinds of errors in the logs).