Fmri prep crashing at autorecon1 on hpc due to permissions error(I think)

Summary of what happened:

I am getting the same failure at autorecon1 that others have posted. I went through the github comment related to this issue and troubleshot as suggested by -shell ing into the container and running

<workdir>/fmriprep_22_0_wf/single_subject_<subject>_wf/anat_preproc_wf/surface_recon_wf/autorecon1/command.txt

the output was:

bash: /wynton/group/rsl/ABCD/code/working/slashofregas/rsfmri_pipeline/fmri_

Command used (and if a helper script was used, a link to the helper script or the command generated):

This is using fmriprep-22.0.2.simg in singularity remotely on an HPC. My singularity call is a bash script that looks like this(all the variables are filepaths):

singularity run --cleanenv -B $hostpath	$fmriprep							\
    $bids_raw_dir  $bids_deriv_dir	participant			\
    --participant-label $sub 							\
    -w $working_dir								\
    --fs-license-file $fs_license 						\
    --output-spaces T1w MNI152NLin6Asym:res-2           \

Version:

22.0.2

Environment (Docker, Singularity, custom installation):

Singularity

Data formatted according to a validatable standard? Please provide the output of the validator:

Yes, at least according to the fmri-prep built in BIDS validator.

Relevant log outputs (up to 20 lines):

Here are the lines from the terminal output when it errors out(it claims 'no such file or directory but the way I am thinking about it is that this has to do with the permissions error stated above):

[Node] Error on "fmriprep_22_0_wf.single_subject_hifuRS01a_wf.anat_preproc_wf.surface_recon_wf.autorecon1" (/wynton/group/rsl/ABCD/code/working/slashofregas/rsfmri_pipeline/fmri_prep/working/fmriprep_22_0_wf/single_subject_hifuRS01a_wf/anat_preproc_wf/surface_recon_wf/autorecon1)
221207-09:32:33,648 nipype.workflow ERROR:
         Node autorecon1 failed to run on host dev1.wynton.ucsf.edu.
221207-09:32:33,657 nipype.workflow ERROR:
         Saving crash info to /wynton/group/rsl/ABCD/code/working/slashofregas/rsfmri_pipeline/Data/hifu_rsfmri_bids/derivatives/sub-hifuRS01a/log/20221207-093126_8a2fb94f-5a20-4688-9877-8d4ce05b6b79/crash-20221207-093233-slashofregas-autorecon1-0840c0c3-4797-4135-8a1e-a91a7b67dd48.txt
Traceback (most recent call last):
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
    result = self._run_interface(execute=True)
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface
    return self._run_command(execute)
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command
    raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node autorecon1.

Cmdline:
        recon-all -autorecon1 -i /wynton/group/rsl/ABCD/code/working/slashofregas/rsfmri_pipeline/Data/hifu_rsfmri_bids/rawdata/sub-hifuRS01a/ses-pre/anat/sub-hifuRS01a_ses-pre_T1w.nii -noskullstrip -noT2pial -noFLAIRpial -openmp 8 -subjid sub-hifuRS01a -sd /wynton/group/rsl/ABCD/code/working/slashofregas/rsfmri_pipeline/Data/hifu_rsfmri_bids/derivatives/sourcedata/freesurfer
Stdout:

Stderr:
        /wynton/home/sugrue/slashofregas/rsfmri_pipeline/fmri_prep/working/fmriprep_22_0_wf/single_subject_hifuRS01a_wf/anat_preproc_wf/surface_recon_wf/autorecon1: No such file or directory.
Traceback:
        RuntimeError: subprocess exited with code 1.

221207-09:33:39,376 nipype.workflow ERROR:
         could not run node: fmriprep_22_0_wf.single_subject_hifuRS01a_wf.anat_preproc_wf.surface_recon_wf.autorecon1
221207-09:33:39,445 nipype.workflow CRITICAL:
         fMRIPrep failed: Traceback (most recent call last):
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
    result = self._run_interface(execute=True)
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface
    return self._run_command(execute)
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command
    raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node autorecon1.

Screenshots / relevant information:

It seems like the autorecon1 command from within singularity is having a permissions issue creating a folder or doing something but this is my first time using singularity and not sure how to make it play nice with the HPC.To note all the working dirs and everything else going on with fmriprep are subirectories that come after $hostpath so, at least the way I understand it, they should all be bound to the simg.

Just tried running again and got the same error(have also tried running it with the server /scratch folders bound and the same thing happens) .

Hi,

A few questions:

  1. Were you using premade FreeSurfer outputs?
  2. Does the error persist if you try again with a fresh working directory and/or have fMRIPrep do FreeSurfer from scratch?
  3. Is this error subject specific or for everyone you have tried in your dataset?
  4. If you manually give that file executable permissions ( chmod u+x command.txt) does the error persist?

Best,
Steven

Not using premade surf outputs - getting fMRIprep to do everything and yes I have tried it with a clean working directory. Haven’t tried it with other subjects yet.

I just tried using chmod u+x on the autorecon command and tried running it manually both from inside and outside the shell and these are the logs I got and running it as part of the singularity image and none of it worked.

Hi, did you mean to attach new logs to this message for after you changed the permissions?

Best,
Steven

Ah yes sorry - here they are. Similar errors. Not sure if this is a general singularity~ issue or if this is specific to fmriprep on our system. Thanks

(base) [slashofregas@dev1 slashofregas]$ /wynton/group/rsl/ABCD/code/working/slashofregas/rsfmri_pipeline/fmri_prep/working/fmriprep_22_0_wf/single_subject_hifuRS01a_wf/anat_preproc_wf/surface_recon_wf/autorecon1/command.txt
bash: /wynton/group/rsl/ABCD/code/working/slashofregas/rsfmri_pipeline/fmri_prep/working/fmriprep_22_0_wf/single_subject_hifuRS01a_wf/anat_preproc_wf/surface_recon_wf/autorecon1/command.txt: Permission denied
(base) [slashofregas@dev1 slashofregas]$ chmod u+x /wynton/group/rsl/ABCD/code/working/slashofregas/rsfmri_pipeline/fmri_prep/working/fmriprep_22_0_wf/single_subject_hifuRS01a_wf/anat_preproc_wf/surface_recon_wf/autorecon1/command.txt
(base) [slashofregas@dev1 slashofregas]$ /wynton/group/rsl/ABCD/code/working/slashofregas/rsfmri_pipeline/fmri_prep/working/fmriprep_22_0_wf/single_subject_hifuRS01a_wf/anat_preproc_wf/surface_recon_wf/autorecon1/command.txt
/wynton/group/rsl/ABCD/code/working/slashofregas/rsfmri_pipeline/fmri_prep/working/fmriprep_22_0_wf/single_subject_hifuRS01a_wf/anat_preproc_wf/surface_recon_wf/autorecon1/command.txt: line 1: recon-all: command not found
Singularity> /wynton/group/rsl/ABCD/code/working/slashofregas/rsfmri_pipeline/fmri_prep/working/fmriprep_22_0_wf/single_subject_hifuRS01a_wf/anat_preproc_wf/surface_recon_wf/autorecon1/command.txt
bash: /wynton/group/rsl/ABCD/code/working/slashofregas/rsfmri_pipeline/fmri_prep/working/fmriprep_22_0_wf/single_subject_hifuRS01a_wf/anat_preproc_wf/surface_recon_wf/autorecon1/command.txt: No such file or directory



Are you sure you were singularity shelled into the container when running that? It seems weird that recon-all would not be found.

Yeah - I mean the command line prompt says ‘singularity’ so can’t imagine I wasn’t. Freesurfer is built into the singularity image right?

Some more fiddling around and still nothing:

Singularity> pwd
/wynton/group/rsl/ABCD/code/working/slashofregas/rsfmri_pipeline/fmri_prep/working/fmriprep_22_0_wf/single_subject_hifuRS01a_wf/anat_preproc_wf/surface_recon_wf/autorecon1
Singularity> ls
_inputs.pklz  _node.pklz  _report  command.txt  result_autorecon1.pklz
Singularity> command.txt
bash: command.txt: command not found
Singularity> chmod u+x command.txt
Singularity> ./command.txt 
/wynton/home/sugrue/slashofregas/rsfmri_pipeline/fmri_prep/working/fmriprep_22_0_wf/single_subject_hifuRS01a_wf/anat_preproc_wf/surface_recon_wf/autorecon1: No such file or directory.

When you singularity shelled in to the container, did you bind the same drives that you did when you ran singularity for fmriprep?

Yep! (I tried it both ways and got the same thing both times).

Is it possible that I’m not doing my binds correctly? my -B $hostpath in my singularity call is just a filepath but in the fmriprep singularity usage documentation they add a : and then another directory before adding the path to the .simg. Could this be affecting things?

The -B name1:name2 is to rename your local path to a new name inside the container. It is usually just a convenience measure, but not necessary.

/wynton/home/sugrue/slashofregas/rsfmri_pipeline/fmri_prep/working/fmriprep_22_0_wf/single_subject_hifuRS01a_wf/anat_preproc_wf/surface_recon_wf/autorecon1: No such file or directory.

Can you find out at which point the directory is no longer found? Do you find earlier paths, such as /wynton/home/sugrue/slashofregas/rsfmri_pipeline/fmri_prep/working/? What are you defining as $hostpath?

I am not sure if this helps but we solved our autorecon1 error with:

singularity run --cleanenv --env TMP=$TMPDIR,TMPDIR=$TMPDIR,TEMP=$TMPDIR,TEMPDIR=$TMPDIR [..]

Where TMPDIR is the scratch directory available from within the container

Thanks Steven and Marcel.

Looking at it again it does seem like there is an issue with it figuring out where to look for autorecon and not just a permissions thing.

@Marcel_Zwiers how did figure out the pathname for TMPDIR/can you give me a brief overview of how you debugged?

Hello everyone putting this here for posterity for anyone working on an HPC system that might have confusing directory implementation. Still not sure exactly what the issue was but I do know that it has something to do with the filepath to the fmriprep.simg not having my home directory as it base. So I added the --home argument to my singularity call.

Final call that worked(for now) is:

apptainer run --cleanenv -B $pathtodata:$pathtodata --home $hostpath	$fmriprep							\
    $bids_raw_dir  $bids_deriv_dir	participant			\
    --participant-label $sub 							\
    -w $working_dir								\
    --fs-license-file $fs_license 						\
    --output-spaces T1w MNI152NLin6Asym:res-2           \

where $hostpath is hpc/group/mygroup, which different than what I get when I type in $HOME which is /hpc/home/mygroup. Kinda weird and kinda not sure why its causing problems(i can imagine it vaguely) but this is what worked! Thanks for the input all.