Fmriprep failing during preprocessing can't find itk_transorm

bnintzan · December 20, 2023, 5:45pm

### Summary of what happened:
Hi there,

I’m trying to run fmriprep (version 23.1.3) using singularity (well Compute Canada’s version of that - “apptainer”). My data are bids-validated, except one component of the dwi data, which should not affect this. I have tried to run this on multiple participants (whilst clearing the output and work directory each time). I always receive this as the error (will provide the full one further below):
FileNotFoundError: No such file or directory 'output/work/fmriprep_23_1_wf/single_subject_VIA6614_wf/func_preproc_task_rest_wf/bold_hmc_wf/fsl2itk/tmp-3nktl6vi/MAT_0000_itk-00000.txt' for output 'itk_transform' of a C3dAffineTool interface

I have searched all the forums, google, etc to try to figure out what I might be doing wrong, I’m hoping it’s something simple and related to me being a newbie with fmriprep.

Command used (and if a helper script was used, a link to the helper script or the command generated):

apptainer run --cleanenv -B /home/bnintzan/scratch/CCNA:/CCNA -B /home/bnintzan/scratch/output:/output ../fmriprep_23.1.3.sif /CCNA /output participant -w /output/work --participant-label sub-VIA6614 --fs-license-file /CCNA/license.txt --fs-no-reconall --skip_bids_validation

### Version:
23.1.3

### Environment (Docker, Singularity, custom installation):
Singularity like environment - apptainer

### Data formatted according to a validatable standard? Please provide the output of the validator:

(note that most of the errors are in the tmp_dcm2bids folder, which is a temporary folder)

(Use `node --trace-warnings ...` to show where the warning was created)
        1: [ERR] Files with such naming scheme are not part of BIDS specification. This error is most commonly caused by typos in file names that make them not BIDS compatible.                                                     Please consult the specification and make sure your files are named correctly. If this is not a file naming issue (for example when including files not yet covered by the BIDS                                                     specification) you should include a ".bidsignore" file in your dataset (see https://github.com/bids-standard/bids-validator#bidsignore for details). Please note that derived (                                                    processed) data should be placed in /derivatives folder and source data (such as DICOMS or behavioural logs in proprietary formats) should be placed in the /sourcedata folder.                                                     (code: 1 - NOT_INCLUDED)
                ./license.txt
                        Evidence: license.txt
                ./sub-BCT9968/anat/sub-BCT9968_PDT2e1.json
                        Evidence: sub-BCT9968_PDT2e1.json
                ./sub-BCT9968/anat/sub-BCT9968_PDT2e1.nii.gz
                        Evidence: sub-BCT9968_PDT2e1.nii.gz
                ./sub-BCT9968/anat/sub-BCT9968_PDT2e2.json
                        Evidence: sub-BCT9968_PDT2e2.json
                ./sub-BCT9968/anat/sub-BCT9968_PDT2e2.nii.gz
                        Evidence: sub-BCT9968_PDT2e2.nii.gz
                ./tmp_dcm2bids/log/sub-BCT8650_20231215-133423.log
                        Evidence: sub-BCT8650_20231215-133423.log
                ./tmp_dcm2bids/log/sub-BCT9503_20231215-170113.log
                        Evidence: sub-BCT9503_20231215-170113.log
                ./tmp_dcm2bids/log/sub-BCU0330_20231215-171106.log
                        Evidence: sub-BCU0330_20231215-171106.log
                ./tmp_dcm2bids/log/sub-BCU3351_20231215-171012.log
                        Evidence: sub-BCU3351_20231215-171012.log
                ./tmp_dcm2bids/log/sub-BCU4649_20231215-170913.log
                        Evidence: sub-BCU4649_20231215-170913.log
                ... and 102902 more files having this issue (Use --verbose to see them all).

        Please visit https://neurostars.org/search?q=NOT_INCLUDED for existing conversations about this issue.

        2: [ERR] Sub label contain an Illegal Character hyphen or underscore. Please edit the filename as per BIDS spec. (code: 62 - SUBJECT_VALUE_CONTAINS_ILLEGAL_CHARACTER)
                ./tmp_dcm2bids/sub-BCT8650/001_sub-BCT8650_tmp_AAHead_Scout_20191122145958.json
                        Evidence: sub name contains illegal character:/tmp_dcm2bids/sub-BCT8650/001_sub-BCT8650_tmp_AAHead_Scout_20191122145958.json
                ./tmp_dcm2bids/sub-BCT8650/002_sub-BCT8650_tmp_AAHead_Scout_20191122145958_i00001.json
                        Evidence: sub name contains illegal character:/tmp_dcm2bids/sub-BCT8650/002_sub-BCT8650_tmp_AAHead_Scout_20191122145958_i00001.json
                ./tmp_dcm2bids/sub-BCT8650/002_sub-BCT8650_tmp_AAHead_Scout_20191122145958_i00002.json
                        Evidence: sub name contains illegal character:/tmp_dcm2bids/sub-BCT8650/002_sub-BCT8650_tmp_AAHead_Scout_20191122145958_i00002.json
                ./tmp_dcm2bids/sub-BCT8650/002_sub-BCT8650_tmp_AAHead_Scout_20191122145958_i00003.json
                        Evidence: sub name contains illegal character:/tmp_dcm2bids/sub-BCT8650/002_sub-BCT8650_tmp_AAHead_Scout_20191122145958_i00003.json
                ./tmp_dcm2bids/sub-BCT8650/002_sub-BCT8650_tmp_AAHead_Scout_20191122145958_i00004.json
                        Evidence: sub name contains illegal character:/tmp_dcm2bids/sub-BCT8650/002_sub-BCT8650_tmp_AAHead_Scout_20191122145958_i00004.json
                ./tmp_dcm2bids/sub-BCT8650/003_sub-BCT8650_tmp_AAHead_Scout_20191122145958_i00001.json
                        Evidence: sub name contains illegal character:/tmp_dcm2bids/sub-BCT8650/003_sub-BCT8650_tmp_AAHead_Scout_20191122145958_i00001.json
                ./tmp_dcm2bids/sub-BCT8650/003_sub-BCT8650_tmp_AAHead_Scout_20191122145958_i00002.json
                        Evidence: sub name contains illegal character:/tmp_dcm2bids/sub-BCT8650/003_sub-BCT8650_tmp_AAHead_Scout_20191122145958_i00002.json
                ./tmp_dcm2bids/sub-BCT8650/003_sub-BCT8650_tmp_AAHead_Scout_20191122145958_i00003.json
                        Evidence: sub name contains illegal character:/tmp_dcm2bids/sub-BCT8650/003_sub-BCT8650_tmp_AAHead_Scout_20191122145958_i00003.json
                ./tmp_dcm2bids/sub-BCT8650/004_sub-BCT8650_tmp_AAHead_Scout_20191122145958_i00001.json
                        Evidence: sub name contains illegal character:/tmp_dcm2bids/sub-BCT8650/004_sub-BCT8650_tmp_AAHead_Scout_20191122145958_i00001.json
                ./tmp_dcm2bids/sub-BCT8650/004_sub-BCT8650_tmp_AAHead_Scout_20191122145958_i00002.json
                        Evidence: sub name contains illegal character:/tmp_dcm2bids/sub-BCT8650/004_sub-BCT8650_tmp_AAHead_Scout_20191122145958_i00002.json
                ... and 206 more files having this issue (Use --verbose to see them all).

        Please visit https://neurostars.org/search?q=SUBJECT_VALUE_CONTAINS_ILLEGAL_CHARACTER for existing conversations about this issue.

        3: [ERR] Subject label in the filename doesn't match with the path of the file. File seems to be saved in incorrect subject directory. (code: 64 - SUBJECT_LABEL_IN_FILE                                                    NAME_DOESNOT_MATCH_DIRECTORY)
                ./tmp_dcm2bids/log/sub-BCT8650_20231215-133423.log
                        Evidence: File: /tmp_dcm2bids/log/sub-BCT8650_20231215-133423.log is saved in incorrect subject directory as per sub-id in filename.
                ./tmp_dcm2bids/log/sub-BCT9503_20231215-170113.log
                        Evidence: File: /tmp_dcm2bids/log/sub-BCT9503_20231215-170113.log is saved in incorrect subject directory as per sub-id in filename.
                ./tmp_dcm2bids/log/sub-BCU0330_20231215-171106.log
                        Evidence: File: /tmp_dcm2bids/log/sub-BCU0330_20231215-171106.log is saved in incorrect subject directory as per sub-id in filename.
                ./tmp_dcm2bids/log/sub-BCU3351_20231215-171012.log
                        Evidence: File: /tmp_dcm2bids/log/sub-BCU3351_20231215-171012.log is saved in incorrect subject directory as per sub-id in filename.
                ./tmp_dcm2bids/log/sub-BCU4649_20231215-170913.log
                        Evidence: File: /tmp_dcm2bids/log/sub-BCU4649_20231215-170913.log is saved in incorrect subject directory as per sub-id in filename.
                ./tmp_dcm2bids/log/sub-BCU7939_20231215-201254.log
                        Evidence: File: /tmp_dcm2bids/log/sub-BCU7939_20231215-201254.log is saved in incorrect subject directory as per sub-id in filename.
                ./tmp_dcm2bids/log/sub-BCU8620_20231215-193718.log
                        Evidence: File: /tmp_dcm2bids/log/sub-BCU8620_20231215-193718.log is saved in incorrect subject directory as per sub-id in filename.
                ./tmp_dcm2bids/log/sub-BRI1997_20231215-193204.log
                        Evidence: File: /tmp_dcm2bids/log/sub-BRI1997_20231215-193204.log is saved in incorrect subject directory as per sub-id in filename.
                ./tmp_dcm2bids/log/sub-BRI3721_20231215-180417.log
                        Evidence: File: /tmp_dcm2bids/log/sub-BRI3721_20231215-180417.log is saved in incorrect subject directory as per sub-id in filename.
                ./tmp_dcm2bids/log/sub-CBH0371_20231215-210328.log
                        Evidence: File: /tmp_dcm2bids/log/sub-CBH0371_20231215-210328.log is saved in incorrect subject directory as per sub-id in filename.
                ... and 62 more files having this issue (Use --verbose to see them all).

        Please visit https://neurostars.org/search?q=SUBJECT_LABEL_IN_FILENAME_DOESNOT_MATCH_DIRECTORY for existing conversations about this issue.

        4: [ERR] No BIDS compatible data found for at least one subject. (code: 67 - NO_VALID_DATA_FOUND_FOR_SUBJECT)
                ./sub-HEJ0194_tmp
                ./sub-HEJ0851_tmp
                ./sub-HEJ1366_tmp
                ./sub-HEJ2061_tmp
                ./sub-HEJ2145_tmp
                ./sub-HEJ2428_tmp
                ./sub-HEJ2436_tmp
                ./sub-HEJ4990_tmp
                ./sub-NSH5240
                ./sub-NSH5240_tmp

        Please visit https://neurostars.org/search?q=NO_VALID_DATA_FOUND_FOR_SUBJECT for existing conversations about this issue.

        Please visit https://neurostars.org/search?q=README_FILE_MISSING for existing conversations about this issue.

        Summary:                     Available Tasks:        Available Modalities:
        109927 Files, 39.28GB                                MRI
        369 - Subjects
        1 - Session

### Relevant log outputs (up to 20 lines):

Node: fmriprep_23_1_wf.single_subject_VIA6614_wf.func_preproc_task_rest_wf.bold_hmc_wf.fsl2itk
Working directory: /output/work/fmriprep_23_1_wf/single_subject_VIA6614_wf/func_preproc_task_rest_wf/bold_hmc_wf/fsl2itk

Node inputs:

in_files = <undefined>
in_reference = <undefined>
in_source = <undefined>
num_threads = 8

Traceback (most recent call last):
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
    result = self._run_interface(execute=True)
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface
    return self._run_command(execute)
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command
    raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node fsl2itk.

Traceback:
        Traceback (most recent call last):
          File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 453, in aggregate_outputs
            setattr(outputs, key, val)
          File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/interfaces/base/traits_extension.py", line 330, in validate
            value = super(File, self).validate(objekt, name, value, return_pathlike=True)
          File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/interfaces/base/traits_extension.py", line 135, in validate
            self.error(objekt, name, str(value))
          File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/traits/base_trait_handler.py", line 74, in error
            raise TraitError(
        traits.trait_errors.TraitError: The 'itk_transform' trait of a C3dAffineToolOutputSpec instance must be a pathlike object or string representing an existing file, but a value of '/output/work/fmriprep_23_1_wf/single_subject_VIA6614_wf/func_preproc_task_rest_wf/bold_hmc_wf/fsl2itk/tmp-3nktl6vi/MAT_0000_itk-00000.txt' <class 'str'> was specified.

        During handling of the above exception, another exception occurred:

        Traceback (most recent call last):
          File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 397, in run
            runtime = self._run_interface(runtime)
          File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/niworkflows/interfaces/itk.py", line 78, in _run_interface
            itk_outs = list(
          File "/opt/conda/envs/fmriprep/lib/python3.10/concurrent/futures/_base.py", line 621, in result_iterator
            yield _result_or_cancel(fs.pop())
          File "/opt/conda/envs/fmriprep/lib/python3.10/concurrent/futures/_base.py", line 319, in _result_or_cancel
            return fut.result(timeout)
          File "/opt/conda/envs/fmriprep/lib/python3.10/concurrent/futures/_base.py", line 458, in result
            return self.__get_result()
          File "/opt/conda/envs/fmriprep/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
            raise self._exception
          File "/opt/conda/envs/fmriprep/lib/python3.10/concurrent/futures/thread.py", line 58, in run
            result = self.fn(*self.args, **self.kwargs)
          File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/niworkflows/interfaces/itk.py", line 225, in _mat2itk
            ).run()
          File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 400, in run
            outputs = self.aggregate_outputs(runtime)
          File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 460, in aggregate_outputs
            raise FileNotFoundError(msg)
        FileNotFoundError: No such file or directory '/output/work/fmriprep_23_1_wf/single_subject_VIA6614_wf/func_preproc_task_rest_wf/bold_hmc_wf/fsl2itk/tmp-3nktl6vi/MAT_0000_itk-00000.txt' for output 'itk_transform' of a C3dAffineTool interface

### Screenshots / relevant information:

Steven · December 20, 2023, 5:56pm

Hi @bnintzan and welcome to neurostars!

Thanks for the report.

Here is what I would suggest:

Upgrade to 23.1.4 (or try 23.2.0a3)
Do not put your working directory within your output directory
Do not use --fs-no-reconall
Fix the BIDS validation errors, or add a .bidsignore text file to your project root with the following lines:

tmp_dcm2bids/
license.txt
sub*/anat/*PDTe*
sub*_tmp/
sub-NSH5240/

Then try rerunning.

Best,
Steven

bnintzan · December 21, 2023, 2:38pm

Hi Steven,

Thanks for the quick response.

Unfortunately, doing all these steps still produces the exact same error. Any other ideas of what might be going on? I tried it again for two separate participants and made sure to give eahc unique directories (including your above suggestions).

Let me know!

Thanks,

Brittany

Steven · December 21, 2023, 2:45pm

Hi @bnintzan,

Would you mind sharing your new command and the new BIDS validation output?

Best,
Steven

bnintzan · December 28, 2023, 5:17pm

Hi @Steven ,

I can definitely share the new command -

apptainer run --cleanenv -B /home/bnintzan/scratch/CCNA:/CCNA -B /home/bnintzan/scratch/output_new:/output -B /home/bnintzan/scratch/work_new:/work fmriprep_23.1.4.sif /CCNA /output participant -w /work --participant-label sub-JGH0062 --fs-license-file /CCNA/license.txt

I still get the exact same error when I updated to 23.1.4. I changed the working directory and removed --fs-no-reconall.

The other issue I have is even when the bidsignore is in the project root, and I try to run the bids validation I still get the same BIDs errors, so it seems as though it won’t read the .bidsignore file (I have ensured the permissions have also been changed).

bnintzan · January 4, 2024, 6:50pm

Hi @Steven

Just wanted to follow up to see if you had any other thoughts on how to move forward with this?

Thanks again!

Best,

Brittany

Steven · January 4, 2024, 10:19pm

Hi @bnintzan,

Can you try to enter the container with

apptainer shell --cleanenv -B /home/bnintzan/scratch/CCNA:/CCNA -B /home/bnintzan/scratch/output_new:/output -B /home/bnintzan/scratch/work_new:/work fmriprep_23.1.4.sif

and then run

ls /work/fmriprep_23_1_wf/single_subject_VIA6614_wf/func_preproc_task_rest_wf/bold_hmc_wf/fsl2itk/tmp*/

to see what files are there?

bnintzan · January 5, 2024, 3:33am

Hi @Steven ,

When I ls into this directory I am told that there are no files or directories matching this. However, when I go up one directory, i.e., get rid of the ‘tmp’ component, there are files there. So it seems as though the tmp directory is not being created for some reason…

Steven · January 5, 2024, 4:17am

Hi @bnintzan,

Would you be able to share one subject’s data so I can try processing it and see if I can reproduce the error?

Best,
Steven

bnintzan · January 5, 2024, 6:20pm

Hi @Steven ,

Absolutely, is there an email that I can send the participant to? Its fairly large…and just for confidentiality etc.

What specific files would you need? Or do you just prefer the entire participant folder with all the imaging data?

Steven · January 5, 2024, 6:35pm

Yup! smeisler@g.harvard.edu. I’d prefer the entire participant folder. If too large for email message, google drive works well.

bnintzan · January 5, 2024, 9:16pm

perfect - shared via google drive - let me know if you didn’t receive the link.

Steven · January 27, 2024, 4:42pm

This error was solved offline by making sure proper resources were available on the job scheduler.