After running for less than 10 hours I encountered the following error:
Node Name: fmriprep_wf.single_subject_01_wf.func_preproc_task_faces_wf.bold_reg_wf.bbreg_wf.lta_concat
File: /data/group/edu/faces/derivatives/fmriprep/sub-01/log/20171031-135044_ed3d9809-7ae3-4ca3-8290-91ef3712d491/crash-20171031-180645-k1469765-lta_concat-6440e3ed-af1e-47ce-b22c-c38210f0e86b.txt
Working Directory: /data/group/edu/faces/work/fmriprep_wf/single_subject_01_wf/func_preproc_task_faces_wf/bold_reg_wf/bbreg_wf/lta_concat
Inputs:
args:
environ: {‘SUBJECTS_DIR’: ‘/software/system/freesurfer/freesurfer-6.0.0/subjects’}
ignore_exception: False
in_lta1: /data/group/edu/faces/work/fmriprep_wf/single_subject_01_wf/func_preproc_task_faces_wf/bold_reg_wf/bbreg_wf/bbregister/uni_masked_xform_bbreg_sub-01.lta
in_lta2: /data/group/edu/faces/work/fmriprep_wf/single_subject_01_wf/anat_preproc_wf/surface_recon_wf/fsnative_2_t1_xfm/T1_robustreg.lta
invert_1:
invert_2:
invert_out:
out_file: out.lta
out_type:
subject:
subjects_dir: /software/system/freesurfer/freesurfer-6.0.0/subjects
tal_source_file:
tal_template_file:
terminal_output:
Traceback (most recent call last):
File “/cns_zfs/system/u1604/fmriprep/1.0.0_rc7/lib/python3.6/site-packages/niworkflows/nipype/pipeline/plugins/multiproc.py”, line 51, in run_node
result[‘result’] = node.run(updatehash=updatehash)
File “/cns_zfs/system/u1604/fmriprep/1.0.0_rc7/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py”, line 407, in run
self._run_interface()
File “/cns_zfs/system/u1604/fmriprep/1.0.0_rc7/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py”, line 517, in _run_interface
self._result = self._run_command(execute)
File “/cns_zfs/system/u1604/fmriprep/1.0.0_rc7/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py”, line 650, in _run_command
result = self._interface.run()
File “/cns_zfs/system/u1604/fmriprep/1.0.0_rc7/lib/python3.6/site-packages/niworkflows/nipype/interfaces/freesurfer/base.py”, line 162, in run
return super(FSCommand, self).run(**inputs)
File “/cns_zfs/system/u1604/fmriprep/1.0.0_rc7/lib/python3.6/site-packages/niworkflows/nipype/interfaces/base.py”, line 1089, in run
outputs = self.aggregate_outputs(runtime)
File “/cns_zfs/system/u1604/fmriprep/1.0.0_rc7/lib/python3.6/site-packages/niworkflows/nipype/interfaces/base.py”, line 1154, in aggregate_outputs
predicted_outputs = self._list_outputs()
File “/cns_zfs/system/u1604/fmriprep/1.0.0_rc7/lib/python3.6/site-packages/fmriprep/interfaces/freesurfer.py”, line 181, in _list_outputs
with open(outputs[‘out_file’], ‘r’) as f:
TypeError: ‘NoneType’ object is not subscriptable
I don’t really understand what has led to this error and any help would be much appreciated. I’m sure it’s a user error, I just can’t figure out what went wrong.
I don’t know if this is related, but I did notice that Freesurfer had removed a lot of the brain. Only about 1/3 is left after autorecon1 creates brainmask.mgz. Here is a link to the html output, in case it’s any use:
Is there anyway to bypass whatever causes Freesurfer to do this?
I’d like to use freesurfer for surface registration so I’m not that keen to use the --no-freesurfer flag.
Sure, I don’t mind sharing the T1:
I hope you can figure out what’s going wrong.
The only other thing I’ve noticed that I haven’t mentioned yet was that the sub-01_T1w_template.nii.gz that is the input for autorecon1 seems a bit odd. It’s almost bank, there is just one slice, which the very bottom slice of the original T1. Since this is the bottom slice it also contains next to no brain. I’m not sure, but I don’t think this is right?
The version of fmriprep you’re using may be depending on niworkflows behavior that has not yet been released. (These unreleased versions don’t cause problems when running with Docker/Singularity, but manual installs work most smoothly with releases, and we haven’t made niworkflows releases alongside the past two fmriprep releases.)
I think the easiest thing to do would be for us to make a niworkflows release, and see if that fixes your issue. There is also a possibility that you’ve found a bug that’s arisen due to some recent refactoring in nipype; if so, we’ll need to fix that and issue a new fmriprep release.
Hi, just FYI we did release niworkflows 0.1.9 today. If you want to upgrade and check if your problem persists, that would be useful, even if you end up using Docker or Singularity going forward.
I’m sorry it has taken me so long to get back to you!
I’ve checked and the people who manage the network where we analyse our data are not planning to install Singularity or Docker anytime soon. They prefer manual installation.
So they have now installed the new niworkflows release (0.1.9) and I tested it a few times, but the same error persists. It looks to me that it all goes wrong during t1_merge and the template that it creates is still just one slice showing the participant’s neck. There seems to be no brain in the template.
From what I can tell everything looks fine just before this step, but after this step everything seems to go a bit wrong.
Any ideas what could be going wrong? I still have a feeling that it might be just a user error, in my experience it usually is.
Given that you only have a single T1w image, t1_merge should not be doing anything except passing through a reference to your original T1w image. So my best guess at the moment is that you got an error that resulted from a dependency mismatch (our fault!), and that the intermediate result stored in your working directory is now triggering a bug in nipype (again, our fault). This should be fixable with a fresh scratch directory.
I normally do that anyway as otherwise I find I get errors straightaway saying some folders already exist. Still, to double check, I am running it again after clearing the directory, and so far it’s not looking good. No error yet, but it looks like that’s where it is headed. As before, the template in the T1_merge directory is still just one slice from the participant’s neck and the Freesurfer brainmask seems to have removed about 2/3 of the brain. If it helps at all everything looks exactly the same as before: when it removes too much brain, it does so exactly the same way every time. So at least it isn’t doing this randomly.
Did you remove your FreeSurfer directory in your derivatives? If that contained badly reconstructed data due to bad inputs, then it needs to be removed, as well. (fmriprep assumes that a pre-existing FreeSurfer directory has been checked by the user, since it’s such a high cost preprocessing step.)
I have tried running fmriprep on your data (without FreeSurfer, since it does take so long), and I have not found any issues with either merging – it simply links to the original T1w image – or skull-stripping. At this point, I can only recommend upgrading to the newest version of fmriprep (ideally using fmriprep-docker and starting with fresh working and derivatives directories. If the problem persists, please send us the latest crash files.
Sorry if this isn’t very helpful. I’m a bit confused as to what’s happening here, as well.
Just to give a quick update, it seems to have finished running without error this time after installing the new version, but all the Freesurfer bits look pretty bad. It looks like I’ll have to run it without Freesurfer.
Oh, wow. I just ran through the FreeSurfer bit. Yeah, that’s a bug. It looks like there’s a resampling issue from the ANTs-extracted brain mask to the FreeSurfer space. I’ve opened an issue over at the GitHub page.
In the short term, this can be resolved by running the following Python code on your T1w image before running fmriprep:
import nibabel as nb
img = nb.load('sub-01_T1w.nii.gz')
img.get_data() # Loads all data into memory so we can rewrite the file
# Sync qform/sform affines with best available information
xform = img.afffine
_, sform_code = img.get_sform(coded=True)
_, qform_code = img.get_qform(coded=True)
xform_code = int(sform_code) or int(qform_code) or 1
img.set_sform(xform, xform_code)
img.set_qform(xform, xform_code)
img.to_filename('sub-01_T1w.nii.gz')
I’m not sure where the mismatch has come from. My colleagues tell me that all that has been done to the images is that origin was set using SPM. Could that be it?