Hey Chris,
I’m working with Nick and we are running these processes on a server. I’m logging system usage on the server. When we close out a session we can see the system usage go to zero, when logging back in we see that the system usage ramp back up to what we normally see when running fmriprep.
In normal circumstances our data takes about 15 hours to go through fmriprep. But when trying to run jobs over the weekend, the job won’t complete until the user resumes the session at which point the job will pick-up where it left off.
We think that this is more so an issue with singularity 2.x.x rather than fmriprep.
We’ve found suggestions that upgrading singularity to a version 3.x.x would help with this issue. But after upgrading singularity we see the above ‘Read-Only’ error messages.
Then the process will stop within about a minute after throwing the following:
Captured warning (<class 'UserWarning'>): [Errno 30] Read-only file system. joblib will operate in serial mode
Captured warning (<class 'ImportWarning'>): can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
Captured warning (<class 'DeprecationWarning'>): invalid escape sequence \k
Captured warning (<class 'DeprecationWarning'>): invalid escape sequence \s
Captured warning (<class 'DeprecationWarning'>): invalid escape sequence \
Captured warning (<class 'DeprecationWarning'>): invalid escape sequence \.
Captured warning (<class 'ImportWarning'>): can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
Captured warning (<class 'ImportWarning'>): can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
Captured warning (<class 'DeprecationWarning'>): invalid escape sequence \#
Captured warning (<class 'DeprecationWarning'>): invalid escape sequence \c
Captured warning (<class 'DeprecationWarning'>): Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
Captured warning (<class 'DeprecationWarning'>): invalid escape sequence \s
Captured warning (<class 'DeprecationWarning'>): invalid escape sequence \g
Captured warning (<class 'DeprecationWarning'>): invalid escape sequence \.
Captured warning (<class 'DeprecationWarning'>): invalid escape sequence \d
Captured warning (<class 'DeprecationWarning'>): invalid escape sequence \{
Captured warning (<class 'DeprecationWarning'>): invalid escape sequence \%
Captured warning (<class 'DeprecationWarning'>): invalid escape sequence \S
Captured warning (<class 'DeprecationWarning'>): invalid escape sequence \|
Captured warning (<class 'DeprecationWarning'>): invalid escape sequence \[
Captured warning (<class 'ResourceWarning'>): unclosed file <_io.TextIOWrapper name='/usr/local/miniconda/lib/python3.7/site-packages/pytest.py' mode='r' encoding='utf-8'>
Captured warning (<class 'DeprecationWarning'>): invalid escape sequence \A
Captured warning (<class 'DeprecationWarning'>): invalid escape sequence \h
Captured warning (<class 'DeprecationWarning'>): invalid escape sequence \s
Captured warning (<class 'DeprecationWarning'>): invalid escape sequence \*
Captured warning (<class 'DeprecationWarning'>): invalid escape sequence \m
Captured warning (<class 'DeprecationWarning'>): invalid escape sequence \-
Captured warning (<class 'DeprecationWarning'>): invalid escape sequence \D
Traceback (most recent call last):
File "/usr/local/miniconda/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
self.run()
File "/usr/local/miniconda/lib/python3.7/multiprocessing/process.py", line 99, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/miniconda/lib/python3.7/multiprocessing/managers.py", line 577, in _run_server
server.serve_forever()
File "/usr/local/miniconda/lib/python3.7/multiprocessing/managers.py", line 173, in serve_forever
sys.exit(0)
SystemExit: 0
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/miniconda/lib/python3.7/multiprocessing/util.py", line 265, in _run_finalizers
finalizer()
File "/usr/local/miniconda/lib/python3.7/multiprocessing/util.py", line 189, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/usr/local/miniconda/lib/python3.7/shutil.py", line 485, in rmtree
_rmtree_safe_fd(fd, path, onerror)
File "/usr/local/miniconda/lib/python3.7/shutil.py", line 443, in _rmtree_safe_fd
onerror(os.unlink, fullname, sys.exc_info())
File "/usr/local/miniconda/lib/python3.7/shutil.py", line 441, in _rmtree_safe_fd
os.unlink(entry.name, dir_fd=topfd)
OSError: [Errno 16] Device or resource busy: '.nfs000000000cb6003f00003f66'
Traceback (most recent call last):
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 136, in __init__
mp_context = mp.context.get_context(
AttributeError: module 'multiprocessing.context' has no attribute 'get_context'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/miniconda/bin/fmriprep", line 11, in <module>
load_entry_point('fmriprep==1.3.2', 'console_scripts', 'fmriprep')()
File "/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/cli/run.py", line 436, in main
fmriprep_wf.run(**plugin_settings)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/workflows.py", line 583, in run
runner = plugin_mod(plugin_args=plugin_args)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 144, in __init__
self.pool = ProcessPoolExecutor(max_workers=self.processors)
File "/usr/local/miniconda/lib/python3.7/concurrent/futures/process.py", line 542, in __init__
pending_work_items=self._pending_work_items)
File "/usr/local/miniconda/lib/python3.7/concurrent/futures/process.py", line 158, in __init__
super().__init__(max_size, ctx=ctx)
File "/usr/local/miniconda/lib/python3.7/multiprocessing/queues.py", line 42, in __init__
self._rlock = ctx.Lock()
File "/usr/local/miniconda/lib/python3.7/multiprocessing/context.py", line 67, in Lock
return Lock(ctx=self.get_context())
File "/usr/local/miniconda/lib/python3.7/multiprocessing/synchronize.py", line 162, in __init__
SemLock.__init__(self, SEMAPHORE, 1, 1, ctx=ctx)
File "/usr/local/miniconda/lib/python3.7/multiprocessing/synchronize.py", line 59, in __init__
unlink_now)
OSError: [Errno 30] Read-only file system
Traceback (most recent call last):
File "/usr/local/miniconda/lib/python3.7/multiprocessing/util.py", line 265, in _run_finalizers
finalizer()
File "/usr/local/miniconda/lib/python3.7/multiprocessing/util.py", line 189, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/usr/local/miniconda/lib/python3.7/shutil.py", line 489, in rmtree
onerror(os.rmdir, path, sys.exc_info())
File "/usr/local/miniconda/lib/python3.7/shutil.py", line 487, in rmtree
os.rmdir(path)
OSError: [Errno 39] Directory not empty: '/tmp/pymp-us46vzab'
I guess our main question is, are these any know compatibility issues with fmriprep-1.3.2 and singularity version 3.2.0 that could be causing these errors?
Thanks,
Mitch