QSIPrep - raw_src_qc - os.link(self.inputs.src_file, linked_src_file) - PermissionError: [Errno 1] Operation not permitted

vinpetersen · April 16, 2021, 1:00pm

Dear Community,

I currently try to deploy a preconfigured QSIPrep singularity container on our CentOS HPC. Aiming for some test cycles with a single subject I run in a permission problem within the container. The error concerns hard link creation via python’s “os.link()”. All other nodes seem to run flawlessly.

[Node] Setting-up “qsiprep_wf.single_subject_control01_wf.dwi_preproc_wf.pre_hmc_wf.merge_and_denoise_wf.dwi_qc_wf.raw_src_qc” in “/sngl/scratch/qsiprep_wf/single_subject_control01_wf/dwi_preproc_wf/pre_hmc_wf/merge_and_denoise_wf/dwi_qc_wf/raw_src_qc”.
210416-09:42:20,902 nipype.workflow INFO:
[Node] Setting-up “qsiprep_wf.single_subject_control01_wf.dwi_preproc_wf.pre_hmc_wf.dwi_qc_wf.raw_src_qc” in “/sngl/scratch/qsiprep_wf/single_subject_control01_wf/dwi_preproc_wf/pre_hmc_wf/dwi_qc_wf/raw_src_qc”.
210416-09:42:21,205 nipype.workflow INFO:
[Node] Running “raw_src_qc” (“qsiprep.interfaces.dsi_studio.DSIStudioSrcQC”)
210416-09:42:21,205 nipype.workflow INFO:
[Node] Running “raw_src_qc” (“qsiprep.interfaces.dsi_studio.DSIStudioSrcQC”)
210416-09:42:21,217 nipype.workflow WARNING:
Storing result file without outputs
210416-09:42:21,218 nipype.workflow WARNING:
Storing result file without outputs
210416-09:42:21,221 nipype.workflow WARNING:
[Node] Error on “qsiprep_wf.single_subject_control01_wf.dwi_preproc_wf.pre_hmc_wf.dwi_qc_wf.raw_src_qc” (/sngl/scratch/qsiprep_wf/single_subject_control01_wf/dwi_preproc_wf/pre_hmc_wf/dwi_qc_wf/raw_src_qc)
210416-09:42:21,221 nipype.workflow WARNING:
[Node] Error on “qsiprep_wf.single_subject_control01_wf.dwi_preproc_wf.pre_hmc_wf.merge_and_denoise_wf.dwi_qc_wf.raw_src_qc” (/sngl/scratch/qsiprep_wf/single_subject_control01_wf/dwi_preproc_wf/pre_hmc_wf/merge_and_denoise_wf/dwi_qc_wf/raw_src_qc)
210416-09:42:22,505 nipype.workflow ERROR:
Node raw_src_qc failed to run on host node309.
210416-09:42:22,529 nipype.workflow ERROR:
Saving crash info to /sngl/qsiprep-output/qsiprep/sub-control01/log/20210416-094111_6865a7b6-176e-4c40-807f-674909d16ce1/crash-20210416-094222-fatx405-raw_src_qc-8f455aeb-590c-4cc6-b3ab-e71cfb9ca67a.txt
Traceback (most recent call last):
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py”, line 67, in run_node
result[“result”] = node.run(updatehash=updatehash)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py”, line 516, in run
result = self._run_interface(execute=True)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py”, line 635, in _run_interface
return self._run_command(execute)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py”, line 741, in _run_command
result = self._interface.run(cwd=outdir)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py”, line 434, in run
runtime = self._run_interface(runtime)
File “/usr/local/miniconda/lib/python3.7/site-packages/qsiprep/interfaces/dsi_studio.py”, line 108, in _run_interface
os.link(self.inputs.src_file, linked_src_file)
PermissionError: [Errno 1] Operation not permitted: ‘/sngl/scratch/qsiprep_wf/single_subject_control01_wf/dwi_preproc_wf/pre_hmc_wf/dwi_qc_wf/raw_src_qc/sub-control01_dwi_merged.src.gz’ → ‘/sngl/scratch/qsiprep_wf/single_subject_control01_wf/dwi_preproc_wf/pre_hmc_wf/dwi_qc_wf/raw_src_qc/src_qc/sub-control01_dwi_merged.src.gz’
210416-09:42:22,542 nipype.workflow ERROR:
Node raw_src_qc failed to run on host node309.
210416-09:42:22,549 nipype.workflow ERROR:
Saving crash info to /sngl/qsiprep-output/qsiprep/sub-control01/log/20210416-094111_6865a7b6-176e-4c40-807f-674909d16ce1/crash-20210416-094222-fatx405-raw_src_qc-979b9de1-8ed0-4a3b-94fa-63f0b1ed02e0.txt
Traceback (most recent call last):
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py”, line 67, in run_node
result[“result”] = node.run(updatehash=updatehash)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py”, line 516, in run
result = self._run_interface(execute=True)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py”, line 635, in _run_interface
return self._run_command(execute)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py”, line 741, in _run_command
result = self._interface.run(cwd=outdir)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py”, line 434, in run
runtime = self._run_interface(runtime)
File “/usr/local/miniconda/lib/python3.7/site-packages/qsiprep/interfaces/dsi_studio.py”, line 108, in _run_interface
os.link(self.inputs.src_file, linked_src_file)
PermissionError: [Errno 1] Operation not permitted: ‘/sngl/scratch/qsiprep_wf/single_subject_control01_wf/dwi_preproc_wf/pre_hmc_wf/merge_and_denoise_wf/dwi_qc_wf/raw_src_qc/sub-control01_dwi_merged.src.gz’ → ‘/sngl/scratch/qsiprep_wf/single_subject_control01_wf/dwi_preproc_wf/pre_hmc_wf/merge_and_denoise_wf/dwi_qc_wf/raw_src_qc/src_qc/sub-control01_dwi_merged.src.gz’

This error always involves os.link(file1,file2) and appears in 3 other instances further down the pipeline which eventually terminates prematurely. Whole slurm output file is here: slurm_output.txt (287.1 KB)
I tried to reproduce the error when shelling into the container and I am indeed not able to write with os.link() in the directories concerned.

Writing with for example touch or mkdir works fine. When trying to write to these directories outside the container there is no problem at all. I presume there is something wrong with my environment influencing Python within the container but I don’t know how to further debug it.

Further information:

Singularity version on HPC

HPC. 3.5.2
Local machine: 3.7.1

QSIPrep versions (tried multiple preconfigured containers from dockerhub)

0.13.0RC2
0.13.0RC1
0.12.2

Command. Whole script: qsiprep.sh.txt (1.9 KB)

singularity run --cleanenv --no-home --userns -B /work/fatx405/projects/2104_dwiproc_hchs/02_env:/mnt
-B $raw_bids:/sngl/data
-B $derivatives:/sngl/out
-B $derivatives/qsiprep:/sngl/qsiprep-output
-B $intermediate:/sngl/scratch
-B $env_dir:/sngl/env
/work/fatx405/projects/2104_dwiproc_hchs/02_env/qsiprep-0.13.0RC2
/sngl/data /sngl/qsiprep-output participant
-w /sngl/scratch/
–participant-label ${subjs_subarr[@]} \ #${subjs_subarr[@]} contains the sub ID
–recon_input /sngl/out/qsiprep
–recon_spec mrtrix_singleshell_ss3t
–fs-license-file /sngl/env/freesurfer_license.txt
–output-resolution 1.3
–output-space T1w
–use-syn-sdc \

I also tried to follow some recommendations from NeuroStars and fMRIprep documentation by adding --cleanenv --no-home --userns to singularity and unsetting PYTHONPATH but without avail.

Thanks in advance for your help. Happy to provide further information.

vinpetersen · April 20, 2021, 2:17pm

I contacted the admins of our HPC as well and they had a solution. BeeGFS which underlies the disk I was writing to does not allow hard links between directories. Exchanging the initial $intermed_dir with /scratch solved the problem because /scratch allows hard link creation.

mattcieslak · May 1, 2021, 12:20am

This is fixed in the upcoming release

mattcieslak · May 6, 2021, 6:10pm

0.13.0 is up, could you let me know if this resolves the issue?

vinpetersen · May 14, 2021, 3:25pm

I ran a more recent version of the script using 0.13.0 on the partition with BeeGFS and everything went fine.

Thanks for your help and effort.