fMRIPrep failed: [Errno 2] No such file or directory

1

Hi,

I am trying to run fmriprep on HPC using singularity image. FMRIprep successfully builds the workflow. However, it crashes out with a rather uninformative output error before it can execute. Here is the command:
singularity run --cleanenv -B /zl-data/mahtakp/fMRIPrep/:/data -B $FREESURFER_HOME:/fs fmriprep.simg /data/input /data/output participant --fs-license-file /fs/license.txt --nthreads 1 -vvv

Here is the output:

You are using fMRIPrep-20.1.1, and a newer version of fMRIPrep is available: 20.1.3.
Please check out our documentation about how and when to upgrade:
https://fmriprep.readthedocs.io/en/latest/faq.html#upgrading
200917-19:17:00,352 cli INFO:
	 Making sure the input data is BIDS compliant (warnings can be ignored in most cases).
bids-validator@1.4.0

	1: [WARN] The recommended file /README is missing. See Section 03 (Modality agnostic files) of the BIDS specification. (code: 101 - README_FILE_MISSING)

	Please visit https://neurostars.org/search?q=README_FILE_MISSING for existing conversations about this issue.


        Summary:                  Available Tasks:        Available Modalities: 
        18 Files, 484.11MB                                T1w                   
        1 - Subject                                       bold                  
        1 - Session                                       fieldmap              


	If you have any questions, please post on https://neurostars.org/tags/bids.

200917-19:17:10,48 nipype.workflow IMPORTANT:
	 
    Running fMRIPREP version 20.1.1:
      * BIDS dataset path: /data/input.
      * Participant list: ['tesPAC12Aug'].
      * Run identifier: 20200917-191659_afd4c5bd-d706-45ae-8f42-e97475f3524d.
      * Output spaces: MNI152NLin2009cAsym:res-native.
      * Pre-run FreeSurfer's SUBJECTS_DIR: /data/output/freesurfer.
200917-19:17:41,118 nipype.workflow INFO:
	 No single-band-reference found for sub-tesPAC12Aug_ses-01_task-rest_bold.nii.gz.
200917-19:18:11,728 nipype.workflow IMPORTANT:
	 Slice-timing correction will be included.
200917-19:18:22,580 nipype.workflow INFO:
	 fMRIPrep workflow graph with 357 nodes built successfully.
200917-19:18:33,190 nipype.workflow VERBOSE:
	 fMRIPrep config:
		[environment]
		cpu_count = 40
		exec_env = "singularity"
		free_mem = 34.4
		overcommit_policy = "heuristic"
		overcommit_limit = "50%"
		nipype_version = "1.5.0"
		templateflow_version = "0.6.2"
		version = "20.1.1"
		
		[execution]
		bids_dir = "/data/input"
		boilerplate_only = false
		debug = false
		fs_license_file = "/fs/license.txt"
		fs_subjects_dir = "/data/output/freesurfer"
		layout = "BIDS Layout: .../data/input | Subjects: 1 | Sessions: 1 | Runs: 6"
		log_dir = "/data/output/fmriprep/logs"
		log_level = 10
		low_mem = false
		md_only_boilerplate = false
		notrack = false
		output_dir = "/data/output"
		output_spaces = "MNI152NLin2009cAsym:res-native"
		reports_only = false
		run_uuid = "20200917-191659_afd4c5bd-d706-45ae-8f42-e97475f3524d"
		participant_label = [ "tesPAC12Aug",]
		templateflow_home = "/home/fmriprep/.cache/templateflow"
		work_dir = "/home/mahtakp/work"
		write_graph = false
		
		[workflow]
		anat_only = false
		aroma_err_on_warn = false
		aroma_melodic_dim = -200
		bold2t1w_dof = 6
		bold2t1w_init = "register"
		cifti_output = false
		fmap_bspline = false
		force_syn = false
		hires = true
		ignore = []
		longitudinal = false
		medial_surface_nan = false
		regressors_all_comps = false
		regressors_dvars_th = 1.5
		regressors_fd_th = 0.5
		run_reconall = true
		skull_strip_fixed_seed = false
		skull_strip_template = "OASIS30ANTs"
		skull_strip_t1w = "force"
		spaces = "MNI152NLin2009cAsym:res-native"
		use_aroma = false
		use_syn_sdc = false
		
		[nipype]
		crashfile_format = "txt"
		get_linked_libs = false
		nprocs = 1
		omp_nthreads = 8
		plugin = "MultiProc"
		resource_monitor = false
		stop_on_first_crash = false
		
		[seeds]
		master = 41710
		ants = 56949
		
		[nipype.plugin_args]
		maxtasksperchild = 1
		raise_insufficient = false
200917-19:18:33,190 nipype.workflow IMPORTANT:
	 fMRIPrep started!
200917-19:18:33,206 nipype.workflow DEBUG:
	 [MultiProc] Starting (n_procs=1, mem_gb=453.42, cwd=/home/mahtakp)
200917-19:18:33,221 nipype.workflow CRITICAL:
	 fMRIPrep failed: [Errno 2] No such file or directory

I saw an error on GitHub that suggested this could be a multiprocessing issue. When I examined run/shm, I found that there were too many layers of symbolic links. However, I found reducing the thread count will not resolve the issue. One possible concern is I have limited space in my home directory, and templates might not be able to fit.

Do you have any suggestions or ideas what might be causing this?

2

Hi @12345,

welcome to neurostars!

My guess is that the directory /home/mahtakp is not accessible within the singularity container.
To see if that is the issue, try running the following command:

singularity exec --cleanenv \
-B /zl-data/mahtakp/fMRIPrep/:/data \
-B $FREESURFER_HOME:/fs \
fmriprep.simg \
ls /home/mahtakp

if the output is something like:

ls: cannot access '/home/mahtakp': No such file or directory

then I believe binding your home directory into the container should resolve your issue:

singularity run --cleanenv \
-B /zl-data/mahtakp/fMRIPrep/:/data \
-B $FREESURFER_HOME:/fs \
-B /home/mahtakp \
fmriprep.simg /data/input /data/output participant \
--fs-license-file /fs/license.txt --nthreads 1 -vvv

Hope that helps!
James

3

Thank you James. I really appreciate your quick response. I think home/mahtakp is accessible within the singularity. When I ran the command that you recommended it shows list of contents of my home directory. I also tried binding my home directory into the container using the following command

singularity run --cleanenv -B /zl-data/mahtakp/fMRIPrep/:/data -B $FREESURFER_HOME:/fs -B /home/mahtakp fmriprep.simg /data/input /data/output participant --fs-license-file /fs/license.txt --nthreads 1 -vvv

and it still shows fMRIPrep failed: [Errno 2] No such file or directory.

Do you have any thoughts?

Many thanks,
Mahta

4

Hmm, there are a couple things I would try:

  • binding /run/shm as was done here

  • check the paths listed below to make sure you have access to them and/or have permission to create them.

I haven’t been following how templateflow is incorporated into fmriprep, but my intuition would have suggested "/home/fmriprep/.cache/templateflow" should be:
"/home/mahtakp/.cache/templateflow".

I’m curious what the output of this command is:

singularity exec --cleanenv \
-B /zl-data/mahtakp/fMRIPrep/:/data \
-B $FREESURFER_HOME:/fs \
fmriprep.simg \
echo ${HOME}

I empathize that [Errno 2] No such file or directory. is not so helpful of an error message when you don’t know what file or directory does not exist.

5

hello! was this ever figured out?

i just updgraded to fmriprep 23.0.0, with singularity 2.5.0, i get the same error running:

export SINGULARITYENV_TEMPLATEFLOW_HOME=$HOME/.cache/templateflow \ singularity run --cleanenv -B $HOME/.cache/templateflow --home $HOME ~/fmriprep/fmriprep_23.0.0.simg ~/data/input ~/data/output participant --fs-license-file ~/fmriprep/license.txt -vvv

the error is [Errno 2] No such file or directory. i also receive the warning Non existent bind point (file) in container: '/etc/localtime'.

i have tried:

  • rebuilding the singularity image using sudo
  • binding run/shm using -B /run/shm:/run/shm
  • binding the home directory (was notified that this path was already mounted in the container)
  • tried specifying --nthreads 1

here is the output:

30323-23:42:29,191 cli INFO:
	 Telemetry system to collect crashes and errors is enabled - thanks for your feedback!. Use option ``--notrack`` to opt out.
230323-23:43:35,399 nipype.workflow IMPORTANT:
	 Running fMRIPrep version 23.0.0

         License NOTICE ##################################################
         fMRIPrep 23.0.0
         Copyright 2023 The NiPreps Developers.
         
         This product includes software developed by
         the NiPreps Community (https://nipreps.org/).
         
         Portions of this software were developed at the Department of
         Psychology at Stanford University, Stanford, CA, US.
         
         This software is also distributed as a Docker container image.
         The bootstrapping file for the image ("Dockerfile") is licensed
         under the MIT License.
         
         This software may be distributed through an add-on package called
         "Docker Wrapper" that is under the BSD 3-clause License.
         #################################################################
230323-23:43:35,400 nipype.workflow WARNING:
	 Previous output generated by version 0+unknown found.
230323-23:43:43,414 nipype.workflow IMPORTANT:
	 Building fMRIPrep's workflow:
           * BIDS dataset path: /home/cgrossman/spacetaxi/nii_spacetaxi.
           * Participant list: ['JODspacetaxi029'].
           * Run identifier: 20230323-234227_b78984b8-3a51-48f3-a6d9-c622ac4860f8.
           * Output spaces: MNI152NLin2009cAsym:res-native.
           * Pre-run FreeSurfer's SUBJECTS_DIR: /home/cgrossman/spacetaxi/fmriprepped/fmriprep/sourcedata/freesurfer.
230323-23:43:49,965 nipype.workflow INFO:
	 B0 field inhomogeneity map will be estimated with  the following 2 estimators: [<EstimatorType.PHASEDIFF: 3>, <EstimatorType.PEPOLAR: 2>].
230323-23:43:50,508 nipype.workflow INFO:
	 Using single-band reference file(s) sub-JODspacetaxi029_task-taxi_acq-intermed_run-1_sbref.nii.gz.
230323-23:43:50,509 nipype.workflow INFO:
	 Found usable B0-map (fieldmap) estimator(s) <auto_00001> to correct </home/cgrossman/spacetaxi/nii_spacetaxi/sub-JODspacetaxi029/func/sub-JODspacetaxi029_task-taxi_run-1_bold.nii.gz> for susceptibility-derived distortions.
230323-23:43:51,752 nipype.workflow INFO:
	 Using single-band reference file(s) sub-JODspacetaxi029_task-taxi_acq-intermed_run-2_sbref.nii.gz.
230323-23:43:51,753 nipype.workflow INFO:
	 Found usable B0-map (fieldmap) estimator(s) <auto_00001> to correct </home/cgrossman/spacetaxi/nii_spacetaxi/sub-JODspacetaxi029/func/sub-JODspacetaxi029_task-taxi_run-2_bold.nii.gz> for susceptibility-derived distortions.
230323-23:43:52,690 nipype.workflow INFO:
	 Using single-band reference file(s) sub-JODspacetaxi029_task-taxi_acq-intermed_run-3_sbref.nii.gz.
230323-23:43:52,690 nipype.workflow INFO:
	 Found usable B0-map (fieldmap) estimator(s) <auto_00001> to correct </home/cgrossman/spacetaxi/nii_spacetaxi/sub-JODspacetaxi029/func/sub-JODspacetaxi029_task-taxi_run-3_bold.nii.gz> for susceptibility-derived distortions.
230323-23:43:53,489 nipype.workflow INFO:
	 Setting-up fieldmap "auto_00000" (EstimatorType.PHASEDIFF) with <sub-JODspacetaxi029_acq-fmap_run-1_phase1.nii.gz, sub-JODspacetaxi029_acq-fmap_run-1_phase2.nii.gz, sub-JODspacetaxi029_acq-fmap_run-1_magnitude1.nii.gz, sub-JODspacetaxi029_acq-fmap_run-1_magnitude2.nii.gz>
230323-23:43:53,489 nipype.workflow INFO:
	 Setting-up fieldmap "auto_00001" (EstimatorType.PEPOLAR) with <sub-JODspacetaxi029_acq-taxi_dir-AP_run-1_epi.nii.gz, sub-JODspacetaxi029_acq-taxi_dir-AP_run-2_epi.nii.gz, sub-JODspacetaxi029_acq-taxi_dir-AP_run-3_epi.nii.gz, sub-JODspacetaxi029_acq-taxi_dir-PA_run-1_epi.nii.gz, sub-JODspacetaxi029_acq-taxi_dir-PA_run-2_epi.nii.gz, sub-JODspacetaxi029_acq-taxi_dir-PA_run-3_epi.nii.gz>
230323-23:44:11,197 nipype.workflow INFO:
	 fMRIPrep workflow graph with 851 nodes built successfully.
230323-23:44:46,399 nipype.workflow VERBOSE:
	 fMRIPrep config:
		[environment]
		cpu_count = 16
		exec_env = "singularity"
		free_mem = 7.4
		overcommit_policy = "heuristic"
		overcommit_limit = "50%"
		nipype_version = "1.8.5"
		templateflow_version = "23.0.0"
		version = "23.0.0"
		
		[execution]
		bids_dir = "/home/cgrossman/spacetaxi/nii_spacetaxi"
		bids_database_dir = "/home/cgrossman/spacetaxi/fmriprepped/fmriprep/work/20230323-234227_b78984b8-3a51-48f3-a6d9-c622ac4860f8/bids_db"
		bids_description_hash = "56ffc36e50a3d53cf60e5a335938ac476c3a08b2e48728363e476e094aa1b5e8"
		boilerplate_only = false
		sloppy = false
		debug = []
		fmriprep_dir = "/home/cgrossman/spacetaxi/fmriprepped/fmriprep"
		fs_license_file = "/home/cgrossman/fmriprep/license.txt"
		fs_subjects_dir = "/home/cgrossman/spacetaxi/fmriprepped/fmriprep/sourcedata/freesurfer"
		layout = "BIDS Layout: ...ossman/spacetaxi/nii_spacetaxi | Subjects: 37 | Sessions: 0 | Runs: 109"
		log_dir = "/home/cgrossman/spacetaxi/fmriprepped/fmriprep/logs"
		log_level = 10
		low_mem = false
		md_only_boilerplate = true
		notrack = false
		track_carbon = false
		country_code = "CAN"
		output_dir = "/home/cgrossman/spacetaxi/fmriprepped/fmriprep"
		me_output_echos = false
		output_layout = "bids"
		output_spaces = "MNI152NLin2009cAsym:res-native"
		reports_only = false
		run_uuid = "20230323-234227_b78984b8-3a51-48f3-a6d9-c622ac4860f8"
		participant_label = [ "JODspacetaxi029",]
		templateflow_home = "/home/cgrossman/.cache/templateflow"
		work_dir = "/home/cgrossman/spacetaxi/fmriprepped/fmriprep/work"
		write_graph = false
		
		[workflow]
		anat_only = false
		aroma_err_on_warn = false
		aroma_melodic_dim = -200
		bold2t1w_dof = 6
		bold2t1w_init = "register"
		cifti_output = false
		fmap_bspline = false
		force_syn = false
		hires = true
		ignore = [ "slicetiming",]
		longitudinal = false
		medial_surface_nan = false
		project_goodvoxels = false
		regressors_all_comps = false
		regressors_dvars_th = 1.5
		regressors_fd_th = 0.5
		run_reconall = false
		skull_strip_fixed_seed = false
		skull_strip_template = "OASIS30ANTs"
		skull_strip_t1w = "force"
		slice_time_ref = 0.5
		spaces = "MNI152NLin2009cAsym:res-native"
		use_aroma = false
		use_syn_sdc = false
		
		[nipype]
		crashfile_format = "txt"
		get_linked_libs = false
		nprocs = 1
		omp_nthreads = 8
		plugin = "MultiProc"
		resource_monitor = false
		stop_on_first_crash = true
		
		[seeds]
		master = 18074
		ants = 25573
		numpy = 27727
		
		[nipype.plugin_args]
		maxtasksperchild = 1
		raise_insufficient = false
230323-23:44:46,399 nipype.workflow IMPORTANT:
	 fMRIPrep started!
230323-23:44:46,418 nipype.workflow DEBUG:
	 [MultiProc] Starting (n_procs=1, mem_gb=56.60, cwd=/home/cgrossman)
230323-23:44:46,441 nipype.workflow CRITICAL:
	 fMRIPrep failed: [Errno 2] No such file or directory
6

Hi @cooper, and welcome to neurostars!

Any way you can try upgrading to most recent singularity version 3? Some other things I may try:

  1. instead of the tilda to indicate home, trying to write the full path.
  2. does that echo command work for you as suggested by the comment above yours?

Best,
Steven

7

thanks for the suggestions, @Steven!

i was dragging my feet on upgrading singularity because it is installed on a shared hpc cluster. but i will try this next.

  1. replacing the tilde with the full path did not change anything
  2. the echo and ls commands described above produced the appropriate outputs