File not found error - possibly related to templates

Software Support
1

What happened?

We recently installed fMRI prep and successfully ran two test participants with no errors reported. We then attempted to run a participant with a pediatric template and it produced file not found errors. We then reverted to using the default MNI adult template but the same errors were produced. We suspect that the errors might be related to template files being moved when attempting to switch to the pediatric template (I do not know what files were moved, someone else did this).

The errors do not stop the final html file from being produced, and the results in that file look roughly the same as the ones from the original test participants that had no errors in them.

We have run through all of the system checks for fMRIprep and solved the few issues that arose with that, rerun a participant, and produced the same error ~30 minutes into the run.

We are using a data set downloaded from OpenNeuro (link if it is helpful)

What command did you use?

RUNNING: docker run --rm -e DOCKER_VERSION_8395080871=25.0.2 -it -v /Volumes/ITALY/license.txt:/opt/freesurfer/license.txt:ro -v /Volumes/ITALY/raw:/data:ro -v /Volumes/ITALY/derivatives:/out -v /Volumes/ITALY/tmp:/scratch nipreps/fmriprep:23.2.0 /data /out participant --skip_bids_validation --participant_label 016 --dummy-scans 0 -w /scratch --output-spaces MNI152NLin2009cAsym

What version of fMRIPrep are you running?

23.2.0

How are you running fMRIPrep?

Docker

Please copy and paste any relevant log output.

Node: fmriprep_23_2_wf.sub_016_wf.anat_fit_wf.ds_template_wf.ds_t1w_ref_xfms
Working directory: /scratch/fmriprep_23_2_wf/sub_016_wf/anat_fit_wf/ds_template_wf/ds_t1w_ref_xfms

Traceback (most recent call last):
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/utils.py", line 94, in nodelist_runner
    result = node.run(updatehash=updatehash)
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 543, in run
    write_node_report(self, result=result, is_mapnode=isinstance(self, MapNode))
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/utils.py", line 208, in write_node_report
    report_file.write_text("\n".join(lines), encoding='utf-8')
  File "/opt/conda/envs/fmriprep/lib/python3.10/pathlib.py", line 1154, in write_text
    with self.open(mode='w', encoding=encoding, errors=errors, newline=newline) as f:
  File "/opt/conda/envs/fmriprep/lib/python3.10/pathlib.py", line 1119, in open
    return self._accessor.open(self, mode, buffering, encoding, errors,
FileNotFoundError: [Errno 2] No such file or directory: '/scratch/fmriprep_23_2_wf/sub_016_wf/anat_fit_wf/ds_template_wf/ds_t1w_ref_xfms/mapflow/_ds_t1w_ref_xfms1/_report/report.rst'
2

Hi @Rachel_S,

What pediatric output-space gave you errors? Was it related to this error: 23.2.0a3: `template_iterator_wf.select_xfm` failing with `MNIPediatricAsym` when cohort specified · Issue #407 · nipreps/smriprep · GitHub?

In (what I presume is) your github post (File not found error - possibly related to template folder · Issue #3233 · nipreps/fmriprep · GitHub), you mention you did not use any precomputed outputs. Does this include using a fresh working directory on each run?

Best,
Steven

3

Hi @Steven,

We used output-spaces MNIPediatricAsym:cohort-5:res-1 when we tried it with the pediatric template. That does look very similar to one of the errors we got when we ran that one.

Correct that is my github post, I meant to link it but clearly forgot. And we emptied out the derivatives and .tmp files before each run if that is what you mean by using a fresh working directory?

We did try to run it with precomputed derivatives that are available on Open Neuro, using -d /Volumes/ITALY/derivatives/mriqc but it kept popping up an error about expecting 2 values and only getting one, so we dropped it and just ran the code included in the original post.

4

Hi @Rachel_S,

I mean using a different argument for -w or clearing out /scratch for each run.

I believe it has to be -d anat|func|fmap DIRECTORY, depending on what modality you want to use. What derivative do you want to use from mriqc? I am not sure I would not recommend using MRIQC derivatives, as the preprocessing for that pipeline are not as rigorous as fMRIPrep…

Best,
Steven

5

Hi Steve:

Thanks for your answers but I think we found another possible culprit. We noticed that the summary for each output gives an error reporting the pediatric template and the adult template while our code does not call for the pediatric template.

fmriprep-docker /Volumes/ITALY/raw/ /Volumes/ITALY/derivatives/ participant --skip_bids_validation --participant_label 006 --dummy-scans 0 --fs-license-file /Volumes/ITALY/license.txt --output-spaces MNI152NLin2009cAsym -w /Volumes/ITALY/tmp/

Screenshot 2024-02-21 at 12.54.22 PM

1- could this be the cause of our errors?
2- How can we remove it from the pipeline?

Thank you,
Ilaria

6

Did you clear the working directory? Also, fmriprep updates the html report, so if you didn’t delete the old html, you should expect the pediatric space from your old run to still be present.

7

Yes, we did clear the working directory. Tried again but the same error popped up again about 30 minutes into the run. We are running it all the way through to see if the final output looks complete (takes about 5 hours) but wonder why we can’t run it again without issues.

Is it possibly related to running it with 0 dummy scans? That is the one thing we changed from the original code - we had 6 dummy scans originally but then after getting warnings about 0-1 non-steady-state volumes automatically recognized we reviewed the information on Open Neuro and realized that they had already been removed from the data
TIA

8

Probably not.

But otherwise, I am stumped at the moment. Are final outputs created in the requested output space, and if so do they look okay?

9

Hi Steven, apologies for the delayed response.

We wiped our system and reinstalled everything. Again, the first time we ran a participant, it went through with no errors. We then ran a set of 3 participants over night and got similar errors, though each participant had a different number of crash reports, and for seemingly random runs (two of the three had crashes with the anatomy scan, all of them had at least one crash from various functional scans). I will include the final lines of the error message from one participant below identifying which file could not be found.

'/scratch/fmriprep_23_2_wf/sub_011_wf/anat_fit_wf/ds_template_wf/ds_t1w_ref_xfms/mapflow/_ds_t1w_ref_xfms1/result__ds_t1w_ref_xfms1.pklz'
'/scratch/fmriprep_23_2_wf/sub_011_wf/bold_ses_T1_task_Num_run_01_wf/bold_confounds_wf/acc_msk_brain/mapflow/_acc_msk_brain2/result__acc_msk_brain2.pklz'
'/scratch/fmriprep_23_2_wf/sub_011_wf/bold_ses_T2_task_Sub_run_01_wf/bold_confounds_wf/acc_msk_brain/mapflow/_acc_msk_brain2/result__acc_msk_brain2.pklz'
'/scratch/fmriprep_23_2_wf/sub_011_wf/bold_ses_T1_task_Rhyming_wf/bold_confounds_wf/acc_msk_brain/mapflow/_acc_msk_brain2/result__acc_msk_brain2.pklz'
'/scratch/fmriprep_23_2_wf/sub_011_wf/bold_ses_T1_task_Sub_run_02_wf/bold_confounds_wf/acc_msk_bin/mapflow/_acc_msk_bin1/result__acc_msk_bin1.pklz'

What is even more confusing is that I ran this same participant today (still running, 6.5 hours in, the first one that finished only took 5 hours) and got a similar crash report pop up but for a different functional run than any of the four runs that created crash reports the first time around:

'No such file or directory: '/scratch/fmriprep_23_2_wf/sub_011_wf/bold_ses_T1_task_Num_run_02_wf/bold_confounds_wf/acc_msk_bin/mapflow/_acc_msk_bin2/result__acc_msk_bin2.pklz'

We skimmed over the output files and they look normal/similar to the files from our runs with no errors, but none of us have a ton of experience with the fMRIprep outputs so if something was off, it would have to be pretty significant for us to catch it.

10

Hi @Rachel_S,

How much memory/cpu are you devoting to the jobs?

Best,
Steven

11

We set docker permissions for 28 GB (Mac Mini with 32 max) and it is the only thing running on the computer. I don’t know off hand exact CPUs but can look at it when I’m back in lab tomorrow.

12

28GB might not be enough for multiple subjects concurrently. Are you running in serial or parallel?

13

Serial (I assume, we just put the participant numbers in order in the participant_label argument (006 011 016), but we get the error even when just running one participant.

The one from yesterday finished after 7 hours and the error above was the only error that popped up.

CPU load was regularly above 90% when it was running and CPU% for fMRIprep was really high, I want to say 800-900%, but it might have gone higher than that at points.

14

If you put all participant labels in one fmriprep command it will run in parallel (to the best of its ability, based on the number of cpus).

8 runs could also be a lot of data (I’ve seen it push above 32 gb for single subject). As a test, can you try using a --bids-filter-file to just run a single run for a single subject, to see if it is memory related or something else?

Best,
Steven

15

Hi Steven,

Before seeing your comment we ran the same participant using the low-mem tag and without free surfer, but ran into the same anatomy error as above.

We then ran it with just a single run for the same subject using the --bids-filter-file tag you suggested (and --low-mem and without free surfer) and it ran without errors.

For the project I am working on, we only need 4 runs from each of the two time points, so we modified the filter file for just those 8 (json text below), but again got the same file not found anatomy type scan. One possible difference is that when I ran it for just the one run, I also only used the T1 anatomy scan versus when I ran it will all 8, we included the anatomy scans at both time points.

{
    "t1w": {
        "datatype": "anat",
        "session": "*",
        "acquisition": null,
        "suffix": "T1w"
    },
    "bold": {
        "datatype": "func",
        "session": "*",
        "task": ["Num", "Sub"],
        "suffix": "bold"
    }
}

terminal command:

fmriprep-docker /Volumes/ITALY/raw/ \
/Volumes/ITALY/derivatives6/ \
participant --skip_bids_validation \
--participant_label 011 \
--dummy-scans 0 \
--fs-license-file /Volumes/ITALY/license.txt \
--fs-no-reconall \
--low-mem \
--output-spaces MNI152NLin2009cAsym \
-w /Volumes/ITALY/tmp6/ \
--bids-filter-file /Volumes/ITALY/num_sub_only.json

We compared the final .html files for the single scan that ran without errors and the equivalent scan in one that ran with errors. It seems like the only differences are minor changes in the noise estimates (ex: 26 components explaining 50% of the variance versus 27 components). Do you think that these errors mean we should not use any of the outputs or should they be considered more of warnings that noise estimates might be slightly off?

We are going to run another subject with the filter for 8 runs to verify that it is not something specifically wrong with 011, but after that we are out of ideas.

16

I wouldn’t read too much into noise component differences. The ICA in compcor isn’t deterministic.

You can feel free to DM me a subject worth of raw data and I can see if I replicate error on my HPC. Do you have access to a more powerful computing cluster? You might also consider brainlife.io

17

The data is all on OpenNeuro if that would be easier for you to access - if not I will send you a subject.

And unfortunately we don’t have access to any computing clusters. I did briefly try brainlife.io at some point, I can’t remember what issue popped up, could look into it again if you think it would be worth it.

18

Thanks, I’ll give it a shot

1 Like
19

Updating: the second participant (016) also produced errors, both the anat related one and one related to a functional scan (when we ran it in a batch of 3 participants a couple of days ago this participant only produced the functional scan error - though it was for the same task both times - see below - including the whole error just in case there is something I’m not noticing in the middle)

Node: fmriprep_23_2_wf.sub_016_wf.bold_ses_T2_task_Num_run_01_wf.bold_confounds_wf.acc_msk_brain
Working directory: /scratch/fmriprep_23_2_wf/sub_016_wf/bold_ses_T2_task_Num_run_01_wf/bold_confounds_wf/acc_msk_brain

Node inputs:

in_file = ['/scratch/fmriprep_23_2_wf/sub_016_wf/bold_ses_T2_task_Num_run_01_wf/bold_confounds_wf/acc_msk_tfm/mapflow/_acc_msk_tfm0/acompcor_csf_trans.nii.gz', '/scratch/fmriprep_23_2_wf/sub_016_wf/bold_ses_T2_task_Num_run_01_wf/bold_confounds_wf/acc_msk_tfm/mapflow/_acc_msk_tfm1/acompcor_wm_trans.nii.gz', '/scratch/fmriprep_23_2_wf/sub_016_wf/bold_ses_T2_task_Num_run_01_wf/bold_confounds_wf/acc_msk_tfm/mapflow/_acc_msk_tfm2/acompcor_wmcsf_trans.nii.gz']
in_mask = <undefined>
threshold = 0.5

Traceback (most recent call last):
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/utils.py", line 94, in nodelist_runner
    result = node.run(updatehash=updatehash)
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 543, in run
    write_node_report(self, result=result, is_mapnode=isinstance(self, MapNode))
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/utils.py", line 208, in write_node_report
    report_file.write_text("\n".join(lines), encoding='utf-8')
  File "/opt/conda/envs/fmriprep/lib/python3.10/pathlib.py", line 1154, in write_text
    with self.open(mode='w', encoding=encoding, errors=errors, newline=newline) as f:
  File "/opt/conda/envs/fmriprep/lib/python3.10/pathlib.py", line 1119, in open
    return self._accessor.open(self, mode, buffering, encoding, errors,
FileNotFoundError: [Errno 2] No such file or directory: '/scratch/fmriprep_23_2_wf/sub_016_wf/bold_ses_T2_task_Num_run_01_wf/bold_confounds_wf/acc_msk_brain/mapflow/_acc_msk_brain2/_report/report.rst'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
    result = self._run_interface(execute=True)
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 1380, in _run_interface
    result = self._collate_results(
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 1249, in _collate_results
    for i, nresult, err in nodes:
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/utils.py", line 99, in nodelist_runner
    result = node.result
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 223, in result
    return _load_resultfile(
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/utils.py", line 293, in load_resultfile
    result = loadpkl(results_file)
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/utils/filemanip.py", line 621, in loadpkl
    with pklopen(str(infile), "rb") as pkl_file:
  File "/opt/conda/envs/fmriprep/lib/python3.10/gzip.py", line 58, in open
    binary_file = GzipFile(filename, gz_mode, compresslevel)
  File "/opt/conda/envs/fmriprep/lib/python3.10/gzip.py", line 174, in __init__
    fileobj = self.myfileobj = builtins.open(filename, mode or 'rb')
FileNotFoundError: [Errno 2] No such file or directory: '/scratch/fmriprep_23_2_wf/sub_016_wf/bold_ses_T2_task_Num_run_01_wf/bold_confounds_wf/acc_msk_brain/mapflow/_acc_msk_brain2/result__acc_msk_brain2.pklz'

When I run the same participant with only the task with the error (T2, Num, run 01), no errors are reported :upside_down_face:

20

Hi @Rachel_S,

I ran subject 011 yesterday with no errors on my compute cluster. When using 32 cores (16 max per task), the job maxed out at 26.17GB used (out of 64 available total) and took 3 hours and 16 minutes. I used all default flags (related to fmriprep steps, at least) except for adding --longitudinal. I doubt that flag is relevant to the errors you get, which, at least to me, sounds like they’re due to not having enough resources.

Best,
Steven