Permission denied to /tmp?

Ursula_Tooley · March 12, 2019, 2:48pm

Not sure if this is related to other permissions issues recently, but I thought not as I’m running 1.2.6 rather than the newer 1.3.0 to maintain consistency with a previous batch of subjects. Every other job (at least, thus far), gives the error below, PermissionError: [Errno 13] Permission denied: '/tmp/fmriprep_wf/index.html'. Some subjects seem to work, and others don’t. I didn’t have this problem previously. Is this something at the level of my HPC that I should investigate, or related to fmriprep?

I’m running with the flag -w /tmp, as running without it writes to my /home directory and rapidly fills it up.

Update: In desperation, I tried changing permissions on the input data to world-read and writable, but no difference. The same subjects continue to fail.

Full traceback:

Making sure the input data is BIDS compliant (warnings can be ignored in most cases).
190312-14:35:48,141 nipype.workflow IMPORTANT:
	 
    Running fMRIPREP version 1.2.6-1:
      * BIDS dataset path: /mnt.
      * Participant list: ['CBPD0136'].
      * Run identifier: 20190312-143548_339a24c6-9b69-4e32-9c7f-f5372205e0ae.
    
190312-14:35:50,457 nipype.workflow IMPORTANT:
	 Creating bold processing workflow for "/mnt/sub-CBPD0136/func/sub-CBPD0136_task-rest_run-01_bold.nii.gz" (0.16 GB / 170 TRs). Memory resampled/largemem=0.66/0.93 GB.
190312-14:35:50,465 nipype.workflow IMPORTANT:
	 No single-band-reference found for sub-CBPD0136_task-rest_run-01_bold.nii.gz
190312-14:35:52,280 nipype.workflow IMPORTANT:
	 Slice-timing correction will be included.
190312-14:35:52,305 nipype.workflow WARNING:
	 SDC: no fieldmaps found or they were ignored (/mnt/sub-CBPD0136/func/sub-CBPD0136_task-rest_run-01_bold.nii.gz).
190312-14:35:53,582 nipype.workflow IMPORTANT:
	 Creating BOLD surface-sampling workflow.
190312-14:35:55,841 nipype.workflow IMPORTANT:
	 Creating bold processing workflow for "/mnt/sub-CBPD0136/func/sub-CBPD0136_task-rest_run-02_bold.nii.gz" (0.17 GB / 176 TRs). Memory resampled/largemem=0.68/0.98 GB.
190312-14:35:55,849 nipype.workflow IMPORTANT:
	 No single-band-reference found for sub-CBPD0136_task-rest_run-02_bold.nii.gz
190312-14:35:57,457 nipype.workflow IMPORTANT:
	 Slice-timing correction will be included.
190312-14:35:57,480 nipype.workflow WARNING:
	 SDC: no fieldmaps found or they were ignored (/mnt/sub-CBPD0136/func/sub-CBPD0136_task-rest_run-02_bold.nii.gz).
190312-14:35:58,755 nipype.workflow IMPORTANT:
	 Creating BOLD surface-sampling workflow.
190312-14:36:03,641 nipype.workflow IMPORTANT:
	 Works derived from this fMRIPrep execution should include the following boilerplate:


Results included in this manuscript come from preprocessing
performed using *fMRIPprep* 1.2.6-1
(@fmriprep1; @fmriprep2; RRID:SCR_016216),
which is based on *Nipype* 1.1.7
(@nipype1; @nipype2; RRID:SCR_002502).

Many internal operations of *fMRIPrep* use
*Nilearn* 0.5.0 [@nilearn, RRID:SCR_001362],
mostly within the functional processing workflow.
For more details of the pipeline, see [the section corresponding
to workflows in *fMRIPrep*'s documentation](https://fmriprep.readthedocs.io/en/latest/workflows.html "FMRIPrep's documentation").


### References


/usr/local/miniconda/lib/python3.7/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
  return f(*args, **kwds)
/usr/local/miniconda/lib/python3.7/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
  return f(*args, **kwds)
/usr/local/miniconda/lib/python3.7/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
  return f(*args, **kwds)
/usr/local/miniconda/lib/python3.7/site-packages/nilearn/datasets/neurovault.py:16: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
  from collections import Container
/usr/local/miniconda/lib/python3.7/site-packages/skimage/__init__.py:80: ResourceWarning: unclosed file <_io.TextIOWrapper name='/usr/local/miniconda/lib/python3.7/site-packages/pytest.py' mode='r' encoding='utf-8'>
  imp.find_module('pytest')
/usr/local/miniconda/lib/python3.7/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
  return f(*args, **kwds)
/usr/local/miniconda/lib/python3.7/site-packages/networkx/classes/reportviews.py:95: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
  from collections import Mapping, Set, Iterable
/usr/local/miniconda/lib/python3.7/site-packages/bids/layout/bids_layout.py:121: ResourceWarning: unclosed file <_io.TextIOWrapper name='/mnt/dataset_description.json' mode='r' encoding='UTF-8'>
  self.description = json.load(open(target, 'r'))
/usr/local/miniconda/lib/python3.7/site-packages/grabbit/core.py:436: ResourceWarning: unclosed file <_io.TextIOWrapper name='/usr/local/miniconda/lib/python3.7/site-packages/bids/layout/config/bids.json' mode='r' encoding='UTF-8'>
  domain = json.load(open(domain, 'r'))
/usr/local/miniconda/lib/python3.7/site-packages/bids/layout/bids_layout.py:208: ResourceWarning: unclosed file <_io.TextIOWrapper name='/mnt/task-rest_bold.json' mode='r' encoding='utf-8'>
  encoding='utf-8'))
/usr/local/miniconda/lib/python3.7/site-packages/bids/layout/bids_layout.py:208: ResourceWarning: unclosed file <_io.TextIOWrapper name='/mnt/sub-CBPD0136/func/sub-CBPD0136_task-rest_run-01_bold.json' mode='r' encoding='utf-8'>
  encoding='utf-8'))
/usr/local/miniconda/lib/python3.7/site-packages/bids/layout/bids_layout.py:208: ResourceWarning: unclosed file <_io.TextIOWrapper name='/mnt/sub-CBPD0136/fmap/sub-CBPD0136_run-01_epi.json' mode='r' encoding='utf-8'>
  encoding='utf-8'))
/usr/local/miniconda/lib/python3.7/site-packages/bids/layout/bids_layout.py:208: ResourceWarning: unclosed file <_io.TextIOWrapper name='/mnt/sub-CBPD0136/func/sub-CBPD0136_task-rest_run-02_bold.json' mode='r' encoding='utf-8'>
  encoding='utf-8'))
Captured warning (<class 'ImportWarning'>): can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
Captured warning (<class 'ImportWarning'>): can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
Captured warning (<class 'ImportWarning'>): can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
Captured warning (<class 'ImportWarning'>): can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
Captured warning (<class 'DeprecationWarning'>): Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
Captured warning (<class 'ResourceWarning'>): unclosed file <_io.TextIOWrapper name='/usr/local/miniconda/lib/python3.7/site-packages/pytest.py' mode='r' encoding='utf-8'>
Captured warning (<class 'ImportWarning'>): can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
/usr/local/miniconda/lib/python3.7/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
  return f(*args, **kwds)
/usr/local/miniconda/lib/python3.7/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
  return f(*args, **kwds)
/usr/local/miniconda/lib/python3.7/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
  return f(*args, **kwds)
/usr/local/miniconda/lib/python3.7/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
  return f(*args, **kwds)
/usr/local/miniconda/lib/python3.7/site-packages/nilearn/datasets/neurovault.py:16: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
  from collections import Container
/usr/local/miniconda/lib/python3.7/site-packages/skimage/__init__.py:80: ResourceWarning: unclosed file <_io.TextIOWrapper name='/usr/local/miniconda/lib/python3.7/site-packages/pytest.py' mode='r' encoding='utf-8'>
  imp.find_module('pytest')
/usr/local/miniconda/lib/python3.7/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
  return f(*args, **kwds)
Traceback (most recent call last):
  File "/usr/local/miniconda/bin/fmriprep", line 11, in <module>
    sys.exit(main())
  File "/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/cli/run.py", line 432, in main
    fmriprep_wf.run(**plugin_settings)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/workflows.py", line 598, in run
    self._write_report_info(self.base_dir, self.name, execgraph)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/workflows.py", line 623, in _write_report_info
    op.join(report_dir, 'index.html'))
  File "/usr/local/miniconda/lib/python3.7/shutil.py", line 121, in copyfile
    with open(dst, 'wb') as fdst:
PermissionError: [Errno 13] Permission denied: '/tmp/fmriprep_wf/index.html'

franklin · March 12, 2019, 5:22pm

Hi @Ursula_Tooley

May you please share what your input command is? I am assuming you are running this using singularity?

Ursula_Tooley · March 12, 2019, 7:09pm

Yes, @franklin, so sorry–I totally forgot about that and edited to add the working dir to the original post, but the whole command is below. Running through singularity on an HPC as an array job, 4/7 of the jobs work just fine, 3/7 fail with this error.

unset PYTHONPATH;
echo 'job is running'
singularity run --cleanenv -B ${BIDS_folder}:/mnt ${tools_dir}/fmriprep-1.2.6-1.simg \
/mnt/ /mnt/derivatives \
participant \
-w /tmp \
--participant-label ${subject} \
--fs-license-file $HOME/license.txt \
--use-aroma \
--output-space T1w template fsaverage5 \
--cifti-output \
--nthreads 1 \

franklin · March 12, 2019, 8:58pm

Thank you for sharing your command! I am thinking it could be server permissions issue depending on the ownership of the /tmp folder (i.e. needing to have root permissions on the HPC). Are you the owner of the /tmp folder (vs. having just read/write on it)? Though interesting that it works for some runs, but not for others. Is there something different between the subjects that work and don’t work?

As you said above there are some leftover issues with this version so it would be interesting to see if a current version would solve this problem, but understand your version constraints.

Ursula_Tooley · March 12, 2019, 9:10pm

I can’t figure out any differences between the subjects it works for and those it doesn’t–believe me, that was the first thing I thought of! I thought it must be permissions but couldn’t think why half of them would work, in that case.

Having a working directory somewhere bound to the singularity container (in /derivatives for example) seems to fix the issue, but it’s very odd. I’m building the latest container and if I end up trying it, I’ll post again.

franklin · March 12, 2019, 9:20pm

Permissions can be a weird issue! I found a couple issues that when using -w with a bound directory it worked (in that case it was calling docker), but interesting it works for singularity too! (this issue had the same Error 13 you have). Sounds good!

ChrisGorgolewski · March 13, 2019, 4:49am

I wonder if perchance your jobs might be running on a node someone else is running FMRIPREP on in /tmp. If the other person created the /tmp/fmriprep_wf folder you would not have permission to it. Could you try running:

unset PYTHONPATH;
echo 'job is running'
singularity run --cleanenv -B ${BIDS_folder}:/mnt ${tools_dir}/fmriprep-1.2.6-1.simg \
/mnt/ /mnt/derivatives \
participant \
-w /tmp/utooley${subject} \
--participant-label ${subject} \
--fs-license-file $HOME/license.txt \
--use-aroma \
--output-space T1w template fsaverage5 \
--cifti-output \
--nthreads 1 \

oesteban · March 13, 2019, 6:41am

I wonder whether /tmp is automatically bound on your singularity installation. If not, this permission denied would be expected because paths in the singularity image are read-only. Only paths made accessible from the host can be written (singularity automatically sets some mount points). Can you try the following?

singularity run --cleanenv -B ${BIDS_folder}:/mnt -B /tmp:/tmp \
                 ${tools_dir}/fmriprep-1.2.6-1.simg \
                 /mnt/ /mnt/derivatives \
                 participant \
                 -w /tmp \
                 --participant-label ${subject} \
                 --fs-license-file $HOME/license.txt \
                 --use-aroma \
                 --output-space T1w template fsaverage5 \
                 --cifti-output \
                 --nthreads 1

Ursula_Tooley · March 13, 2019, 6:38pm

I tried the command @ChrisGorgolewski posted, and it worked (meaning it continue to run past the writing of index.html). Given that I was running my jobs in an array, I could have been the other user writing to /tmp/fmriprep_wf, right? Except I guess then I’d have permissions for that folder, so wouldn’t have run into this issue. I guess this is a good enough workaround for now, I’ll repost if it continues to occur (it’s happened with participants from two different datasets now).

I think @oesteban, if /tmp weren’t automatically bound on my singularity installation, then none of my jobs would work (correct me if I have this wrong). It’s puzzling mostly because some jobs ran and some didn’t.