fMRIPrep failed: [Errno 2] No such file or directory

Hi,

I am trying to run fmriprep on HPC using singularity image. FMRIprep successfully builds the workflow. However, it crashes out with a rather uninformative output error before it can execute. Here is the command:
singularity run --cleanenv -B /zl-data/mahtakp/fMRIPrep/:/data -B $FREESURFER_HOME:/fs fmriprep.simg /data/input /data/output participant --fs-license-file /fs/license.txt --nthreads 1 -vvv

Here is the output:

You are using fMRIPrep-20.1.1, and a newer version of fMRIPrep is available: 20.1.3.
Please check out our documentation about how and when to upgrade:
https://fmriprep.readthedocs.io/en/latest/faq.html#upgrading
200917-19:17:00,352 cli INFO:
	 Making sure the input data is BIDS compliant (warnings can be ignored in most cases).
bids-validator@1.4.0

	1: [WARN] The recommended file /README is missing. See Section 03 (Modality agnostic files) of the BIDS specification. (code: 101 - README_FILE_MISSING)

	Please visit https://neurostars.org/search?q=README_FILE_MISSING for existing conversations about this issue.


        Summary:                  Available Tasks:        Available Modalities: 
        18 Files, 484.11MB                                T1w                   
        1 - Subject                                       bold                  
        1 - Session                                       fieldmap              


	If you have any questions, please post on https://neurostars.org/tags/bids.

200917-19:17:10,48 nipype.workflow IMPORTANT:
	 
    Running fMRIPREP version 20.1.1:
      * BIDS dataset path: /data/input.
      * Participant list: ['tesPAC12Aug'].
      * Run identifier: 20200917-191659_afd4c5bd-d706-45ae-8f42-e97475f3524d.
      * Output spaces: MNI152NLin2009cAsym:res-native.
      * Pre-run FreeSurfer's SUBJECTS_DIR: /data/output/freesurfer.
200917-19:17:41,118 nipype.workflow INFO:
	 No single-band-reference found for sub-tesPAC12Aug_ses-01_task-rest_bold.nii.gz.
200917-19:18:11,728 nipype.workflow IMPORTANT:
	 Slice-timing correction will be included.
200917-19:18:22,580 nipype.workflow INFO:
	 fMRIPrep workflow graph with 357 nodes built successfully.
200917-19:18:33,190 nipype.workflow VERBOSE:
	 fMRIPrep config:
		[environment]
		cpu_count = 40
		exec_env = "singularity"
		free_mem = 34.4
		overcommit_policy = "heuristic"
		overcommit_limit = "50%"
		nipype_version = "1.5.0"
		templateflow_version = "0.6.2"
		version = "20.1.1"
		
		[execution]
		bids_dir = "/data/input"
		boilerplate_only = false
		debug = false
		fs_license_file = "/fs/license.txt"
		fs_subjects_dir = "/data/output/freesurfer"
		layout = "BIDS Layout: .../data/input | Subjects: 1 | Sessions: 1 | Runs: 6"
		log_dir = "/data/output/fmriprep/logs"
		log_level = 10
		low_mem = false
		md_only_boilerplate = false
		notrack = false
		output_dir = "/data/output"
		output_spaces = "MNI152NLin2009cAsym:res-native"
		reports_only = false
		run_uuid = "20200917-191659_afd4c5bd-d706-45ae-8f42-e97475f3524d"
		participant_label = [ "tesPAC12Aug",]
		templateflow_home = "/home/fmriprep/.cache/templateflow"
		work_dir = "/home/mahtakp/work"
		write_graph = false
		
		[workflow]
		anat_only = false
		aroma_err_on_warn = false
		aroma_melodic_dim = -200
		bold2t1w_dof = 6
		bold2t1w_init = "register"
		cifti_output = false
		fmap_bspline = false
		force_syn = false
		hires = true
		ignore = []
		longitudinal = false
		medial_surface_nan = false
		regressors_all_comps = false
		regressors_dvars_th = 1.5
		regressors_fd_th = 0.5
		run_reconall = true
		skull_strip_fixed_seed = false
		skull_strip_template = "OASIS30ANTs"
		skull_strip_t1w = "force"
		spaces = "MNI152NLin2009cAsym:res-native"
		use_aroma = false
		use_syn_sdc = false
		
		[nipype]
		crashfile_format = "txt"
		get_linked_libs = false
		nprocs = 1
		omp_nthreads = 8
		plugin = "MultiProc"
		resource_monitor = false
		stop_on_first_crash = false
		
		[seeds]
		master = 41710
		ants = 56949
		
		[nipype.plugin_args]
		maxtasksperchild = 1
		raise_insufficient = false
200917-19:18:33,190 nipype.workflow IMPORTANT:
	 fMRIPrep started!
200917-19:18:33,206 nipype.workflow DEBUG:
	 [MultiProc] Starting (n_procs=1, mem_gb=453.42, cwd=/home/mahtakp)
200917-19:18:33,221 nipype.workflow CRITICAL:
	 fMRIPrep failed: [Errno 2] No such file or directory

I saw an error on GitHub that suggested this could be a multiprocessing issue. When I examined run/shm, I found that there were too many layers of symbolic links. However, I found reducing the thread count will not resolve the issue. One possible concern is I have limited space in my home directory, and templates might not be able to fit.

Do you have any suggestions or ideas what might be causing this?

Hi @12345,

welcome to neurostars!

My guess is that the directory /home/mahtakp is not accessible within the singularity container.
To see if that is the issue, try running the following command:

singularity exec --cleanenv \
-B /zl-data/mahtakp/fMRIPrep/:/data \
-B $FREESURFER_HOME:/fs \
fmriprep.simg \
ls /home/mahtakp

if the output is something like:

ls: cannot access '/home/mahtakp': No such file or directory

then I believe binding your home directory into the container should resolve your issue:

singularity run --cleanenv \
-B /zl-data/mahtakp/fMRIPrep/:/data \
-B $FREESURFER_HOME:/fs \
-B /home/mahtakp \
fmriprep.simg /data/input /data/output participant \
--fs-license-file /fs/license.txt --nthreads 1 -vvv

Hope that helps!
James

Thank you James. I really appreciate your quick response. I think home/mahtakp is accessible within the singularity. When I ran the command that you recommended it shows list of contents of my home directory. I also tried binding my home directory into the container using the following command

singularity run --cleanenv -B /zl-data/mahtakp/fMRIPrep/:/data -B $FREESURFER_HOME:/fs -B /home/mahtakp fmriprep.simg /data/input /data/output participant --fs-license-file /fs/license.txt --nthreads 1 -vvv

and it still shows fMRIPrep failed: [Errno 2] No such file or directory.

Do you have any thoughts?

Many thanks,
Mahta

Hmm, there are a couple things I would try:

  • binding /run/shm as was done here

  • check the paths listed below to make sure you have access to them and/or have permission to create them.

I haven’t been following how templateflow is incorporated into fmriprep, but my intuition would have suggested "/home/fmriprep/.cache/templateflow" should be:
"/home/mahtakp/.cache/templateflow".

I’m curious what the output of this command is:

singularity exec --cleanenv \
-B /zl-data/mahtakp/fMRIPrep/:/data \
-B $FREESURFER_HOME:/fs \
fmriprep.simg \
echo ${HOME}

I empathize that [Errno 2] No such file or directory. is not so helpful of an error message when you don’t know what file or directory does not exist.