fMRIPrep Crashing on Vagrant/Singularity

Hello all,

I am trying to complete the BIDS/fMRIPrep tutorial from Andy’s Brain Book and am running into problems in preprocessing.

Here’s the config from the verbose command:

fMRIPrep config:
                [environment]
                cpu_count = 24
                exec_env = "singularity"
                free_mem = 190.5
                overcommit_policy = "heuristic"
                overcommit_limit = "50%"
                nipype_version = "1.6.1"
                templateflow_version = "0.7.2"
                version = "21.0.4"

                [execution]
                bids_dir = "/home/vagrant/DATA/BIDS_tutorial"
                bids_database_dir = "/home/vagrant/DATA/work/20221127-005903_c7f81f20-7f31-4640-9b66-1e2f2129f41f/bids_db"
                bids_description_hash = "642036db1ea7da4f7ce4c915f6c75e81cc5535d7d4bde327cb228d982ffff16b"
                boilerplate_only = false
                sloppy = false
                debug = []
                fmriprep_dir = "/home/vagrant/DATA/BIDS_tutorial/derivatives"
                fs_license_file = "/home/vagrant/DATA/Resources/license.txt"
                fs_subjects_dir = "/home/vagrant/DATA/BIDS_tutorial/derivatives/sourcedata/freesurfer"
                layout = "BIDS Layout: ...ome/vagrant/DATA/BIDS_tutorial | Subjects: 1 | Sessions: 1 | Runs: 2"
                log_dir = "/home/vagrant/DATA/BIDS_tutorial/derivatives/logs"
                log_level = 15
                low_mem = true
                md_only_boilerplate = true
                notrack = false
                output_dir = "/home/vagrant/DATA/BIDS_tutorial/derivatives"
                me_output_echos = false
                output_layout = "bids"
                output_spaces = "MNI152NLin2009cAsym:res-2"
                reports_only = false
                run_uuid = "20221127-005903_c7f81f20-7f31-4640-9b66-1e2f2129f41f"
                participant_label = [ "01",]
                templateflow_home = "/home/vagrant/DATA/Resources/templateflow"
                work_dir = "/home/vagrant/DATA/work"
                write_graph = false

                [workflow]
                anat_only = false
                aroma_err_on_warn = false
                aroma_melodic_dim = -200
                bold2t1w_dof = 6
                bold2t1w_init = "register"
                cifti_output = false
                fmap_bspline = false
                force_syn = false
                hires = true
                ignore = []
                longitudinal = false
                medial_surface_nan = false
                regressors_all_comps = false
                regressors_dvars_th = 1.5
                regressors_fd_th = 0.5
                run_reconall = false
                skull_strip_fixed_seed = false
                skull_strip_template = "OASIS30ANTs"
                skull_strip_t1w = "force"
                slice_time_ref = 0.5
                spaces = "MNI152NLin2009cAsym:res-2"
                topup_max_vols = 5
                use_aroma = false
                use_syn_sdc = false

                [nipype]
                crashfile_format = "txt"
                get_linked_libs = false
                memory_gb = 195.0
                nprocs = 16
                omp_nthreads = 8
                plugin = "MultiProc"
                resource_monitor = false
                stop_on_first_crash = true

                [seeds]
                master = 55317
                ants = 40395
                numpy = 18737

                [nipype.plugin_args]
                maxtasksperchild = 1
                raise_insufficient = false

It works fine until it attempts to start node fsl2itk. Here are the warning and error messages:

[Node] Finished "fmriprep_wf.single_subject_01_wf.func_preproc_ses_01_task_bart_run_01_wf.initial_boldref_wf.enhance_and_skullstrip_bold_wf.pre_mask_dilate".
221128-02:47:49,857 nipype.workflow WARNING:
         Storing result file without outputs
221128-02:47:49,875 nipype.workflow WARNING:
         [Node] Error on "fmriprep_wf.single_subject_01_wf.func_preproc_ses_01_task_bart_run_01_wf.bold_hmc_wf.fsl2itk" (/home/vagrant/DATA/work/fmriprep_wf/single_subject_01_wf/func_preproc_ses_01_task_bart_run_01_wf/bold_hmc_wf/fsl2itk)
221128-02:47:49,895 nipype.workflow ERROR:
         Node fsl2itk failed to run on host vagrant.
221128-02:47:49,924 nipype.workflow ERROR:
         Saving crash info to /home/vagrant/DATA/BIDS_tutorial/derivatives/sub-01/log/20221128-023819_0e821647-f1ec-476a-a7d1-1bb91f9ea8a8/crash-20221128-024749-root-fsl2itk-0d871f02-4eba-4f5e-bc90-83d568d2cdff.txt
Traceback (most recent call last):
  File "/opt/conda/lib/python3.8/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/opt/conda/lib/python3.8/site-packages/nipype/pipeline/engine/nodes.py", line 516, in run
    result = self._run_interface(execute=True)
  File "/opt/conda/lib/python3.8/site-packages/nipype/pipeline/engine/nodes.py", line 635, in _run_interface
    return self._run_command(execute)
  File "/opt/conda/lib/python3.8/site-packages/nipype/pipeline/engine/nodes.py", line 741, in _run_command
    result = self._interface.run(cwd=outdir)
  File "/opt/conda/lib/python3.8/site-packages/nipype/interfaces/base/core.py", line 428, in run
    runtime = self._run_interface(runtime)
  File "/opt/conda/lib/python3.8/site-packages/niworkflows/interfaces/itk.py", line 94, in _run_interface
    itk_outs = [
  File "/opt/conda/lib/python3.8/tempfile.py", line 827, in __exit__
    self.cleanup()
  File "/opt/conda/lib/python3.8/tempfile.py", line 831, in cleanup
    self._rmtree(self.name)
  File "/opt/conda/lib/python3.8/tempfile.py", line 813, in _rmtree
    _shutil.rmtree(name, onerror=onerror)
  File "/opt/conda/lib/python3.8/shutil.py", line 722, in rmtree
    onerror(os.rmdir, path, sys.exc_info())
  File "/opt/conda/lib/python3.8/shutil.py", line 720, in rmtree
    os.rmdir(path)
OSError: [Errno 26] Text file busy: '/home/vagrant/DATA/work/fmriprep_wf/single_subject_01_wf/func_preproc_ses_01_task_bart_run_01_wf/bold_hmc_wf/fsl2itk/tmp-otsvqilg'

221128-02:47:50,44 nipype.workflow CRITICAL:
         fMRIPrep failed: Traceback (most recent call last):
  File "/opt/conda/lib/python3.8/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/opt/conda/lib/python3.8/site-packages/nipype/pipeline/engine/nodes.py", line 516, in run
    result = self._run_interface(execute=True)
  File "/opt/conda/lib/python3.8/site-packages/nipype/pipeline/engine/nodes.py", line 635, in _run_interface
    return self._run_command(execute)
  File "/opt/conda/lib/python3.8/site-packages/nipype/pipeline/engine/nodes.py", line 741, in _run_command
    result = self._interface.run(cwd=outdir)
  File "/opt/conda/lib/python3.8/site-packages/nipype/interfaces/base/core.py", line 428, in run
    runtime = self._run_interface(runtime)
  File "/opt/conda/lib/python3.8/site-packages/niworkflows/interfaces/itk.py", line 94, in _run_interface
    itk_outs = [
  File "/opt/conda/lib/python3.8/tempfile.py", line 827, in __exit__
    self.cleanup()
  File "/opt/conda/lib/python3.8/tempfile.py", line 831, in cleanup
    self._rmtree(self.name)
  File "/opt/conda/lib/python3.8/tempfile.py", line 813, in _rmtree
    _shutil.rmtree(name, onerror=onerror)
  File "/opt/conda/lib/python3.8/shutil.py", line 722, in rmtree
    onerror(os.rmdir, path, sys.exc_info())
  File "/opt/conda/lib/python3.8/shutil.py", line 720, in rmtree
    os.rmdir(path)
OSError: [Errno 26] Text file busy: '/home/vagrant/DATA/work/fmriprep_wf/single_subject_01_wf/func_preproc_ses_01_task_bart_run_01_wf/bold_hmc_wf/fsl2itk/tmp-otsvqilg'

221128-02:47:53,723 cli ERROR:
         Preprocessing did not finish successfully. Errors occurred while processing data from participants: 01 (1). Check the HTML reports for details.

I have not been able to find any help online and would appreciate any insight!

Hi and welcome to neurostars!

To help us best answer your question, would you mind answering the following questions?

  1. Are your data BIDS valid?
  2. What version of fMRIPrep are you using, if not the most recent can you try updating?
  3. Is this error subject specific or for everyone?
  4. Does the error persist when you retry rerunning with a new working directory? More specifically, a working directory that it not in /home/vagrant/DATA?
  5. Can you link us to the tutorial you are referring to?

Best,
Steven

1 Like

Thank you for the reply and welcome to the site! After correcting the JSON sidecars and updating fMRIPrep, it ran successfully. I will add some information in case anyone runs into the same issue.

  1. I used the online BIDS validator and found a weird error with the IntendedFor field. I re-ran dcm2bids on the dataset without re-checking with the validator and in the process an extra forward slash was placed in the IntendedFor field in the JSON sidecars of the field maps (i.e. func//images). I manually corrected this instead of re-running dcm2bids.
  2. I was using version 21.0.4 instead of 22.0.2. After this step, fMRIPrep ran to completion.
  3. The dataset only included one subject .
  4. The data was processed successfully without moving the working directory. Should the working directory generally not be in the same folder as the BIDS directory?
  5. The tutorial was from Andy’s Brain Book and I was following the tutorial on BIDS and fMRIPrep.

Thanks again,

John

Glad to hear it!

I brought this up not because there’s any rule against it, but having the work directory in your BIDS directory where data are getting read will lead to a lot of read/write action in that one drive, which may cause it to be overused. So, I put them on different drives, but that’s a personal preference,