Error populating the input of a node in fMRIPrep - the results file of the source node does not exist

1

Hi @oesteban, not sure if I should have actually open a new thread - sorry if this was the case. The pipeline with the MNIPediatricAsym ran fine for some subjects but it crashed for several others. For those subjects, it created the anat and figures folders, but not the func one. The errors seems pretty similars between subjects (around mcflirt, fsl2itk, normalize, calc dummy scans, init aff), I am posting some examples:

  Node: fmriprep_wf.single_subject_F066_wf.func_preproc_task_rest_wf.bold_hmc_wf.mcflirt
Working directory: /scratch/punim0025/work/fmriprep_wf/single_subject_F066_wf/func_preproc_task_rest_wf/bold_hmc_wf/mcflirt

Node inputs:

args = <undefined>
bins = <undefined>
cost = <undefined>
dof = <undefined>
environ = {'FSLOUTPUTTYPE': 'NIFTI_GZ'}
in_file = <undefined>
init = <undefined>
interpolation = <undefined>
mean_vol = <undefined>
out_file = <undefined>
output_type = NIFTI_GZ
ref_file = <undefined>
ref_vol = <undefined>
rotation = <undefined>
save_mats = True
save_plots = True
save_rms = <undefined>
scaling = <undefined>
smooth = <undefined>
stages = <undefined>
stats_imgs = <undefined>
use_contour = <undefined>
use_gradient = <undefined>

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 69, in run_node
    result['result'] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 411, in run
    cached, updated = self.is_cached()
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 297, in is_cached
    hashed_inputs, hashvalue = self._get_hashval()
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 494, in _get_hashval
    self._get_inputs()
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 525, in _get_inputs
    (%s) does not contain any outputs.""" % (key, self.name, results_file))
RuntimeError: Error populating the input "ref_file" of node "mcflirt": the results file of the source node (/scratch/punim0025/work/fmriprep_wf/single_subject_F066_wf/func_preproc_task_rest_wf/bold_reference_wf/gen_ref/result_gen_ref.pklz) does not contain any outputs.

Node: fmriprep_wf.single_subject_F058_wf.func_preproc_task_rest_wf.bold_hmc_wf.fsl2itk
Working directory: /scratch/punim0025/work/fmriprep_wf/single_subject_F058_wf/func_preproc_task_rest_wf/bold_hmc_wf/fsl2itk

Node inputs:

in_files = <undefined>
in_reference = <undefined>
in_source = <undefined>
num_threads = 8

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 69, in run_node
    result['result'] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 411, in run
    cached, updated = self.is_cached()
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 297, in is_cached
    hashed_inputs, hashvalue = self._get_hashval()
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 494, in _get_hashval
    self._get_inputs()
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 525, in _get_inputs
    (%s) does not contain any outputs.""" % (key, self.name, results_file))
RuntimeError: Error populating the input "in_files" of node "fsl2itk": the results file of the source node (/scratch/punim0025/work/fmriprep_wf/single_subject_F058_wf/func_preproc_task_rest_wf/bold_hmc_wf/mcflirt/result_mcflirt.pklz) does not contain any outputs.

I have previously ran fmriprep on the same subjects using the standard template (with fmriprep 14.0 though) and it worked just fine for all of them. Here I tried with fmriprep 1.4.1 and 15.0 because I wanted to use the paediatric template and I get the same errors across versions. I am not sure if there is some issue with the data, rather than the pipeline, and I am not sure why it worked with the previous version/standard template?

Any help would be really appreciated! Thank you.

2

I have split the issue since it seemed different.

Can you attach the output log from fMRIPrep? I’m worried you are just looking at cascaded errors from prior faults.

3

Thanks @oesteban! I am attaching the full log for one of the subject for who the pipeline crashed slurm-11960222_19.txt (92.2 KB). It indeed seems that there are other errors upstreams, e.g., Error while checking node hash, forcing re-run

I just did a bit of googling of this error - not sure if this is the problem but actually I have not created a new clean working directory in /scratch/punim0025/work, when I re-ran fmriprep 14.1 and 15.0 with the paediatric template output space option.

Not sure why it failed for just some subjects though? If you think this could be causing the problem, I can create a new working directory and run the pipeline again and let you know.

Thanks again for your help!

4

Looks like a corrupt working directory. Can you perhaps rename /scratch/punim0025/work (or delete it if you are 100% that’s safe)?

Otherwise, you can run fmriprep with the -w /scratch/punim0025/work2 argument to point intermediate results to a clean directory.

5

Hi @oesteban, thank you for your help. I re-ran the pipeline deleting the /scratch/punim0025/work directory and it worked fine for all the subjects (hooray) - except one. I am attaching the log for that subject - it seems that the errors are Node truncate_images failed to run on host followed by FileNotFoundError: [Errno 2] No such file or directory: 'result_anat_validate.pklz'.
slurm-12036976_40.txt (60.8 KB)
Sorry for the multiple questions, and thanks!

2 Likes
6

I’ve got the same issue.