Looking for the fieldwarp transform in fmriprep working directory

Hi I am looking to get some of the intermediate files from the fmriprep working directory. I am using fmriprep 23.0.2.

Specifically, I am looking for all of the transforms used here

The registration and normalization are available in the derivatives folder, but HMC and SDC are in the working directory right? I think I have found the HMC transform, but can’t seem to find the SDC (I can only find concat_xfm where it seems like they have already been concatenated? I could be wrong).

Could someone point me in the right direction?

Basically I am looking for these files

Hi,

The transformation command to project from bold native space to bold standard space is located in a path of this form:

PATH_TO_YOUR_FMRIPREP_WORKING_DIRECTORY/fmriprep_22_0_wf/single_subject_pilote2_wf/func_preproc_task_rest_run_01_wf/bold_std_trans_wf/_std_target_MNI152NLin2009cAsym.resnative/bold_to_std_transform/command.txt

And the transformation for one volume of a bold run looks like this:

antsApplyTransforms --default-value 0 --float 1 --input /work/temp_data_GesteSigne/fmriprep_22_0_wf/single_subject_pilote2_wf/func_preproc_task_rest_run_01_wf/bold_split/vol0000.nii.gz --interpolation LanczosWindowedSinc --output /work/temp_data_GesteSigne/fmriprep_22_0_wf/single_subject_pilote2_wf/func_preproc_task_rest_run_01_wf/bold_std_trans_wf/_std_target_MNI152NLin2009cAsym.resnative/bold_to_std_transform/vol0000_xform-00000.nii.gz --reference-image /work/temp_data_GesteSigne/fmriprep_22_0_wf/single_subject_pilote2_wf/func_preproc_task_rest_run_01_wf/bold_std_trans_wf/_std_target_MNI152NLin2009cAsym.resnative/gen_ref/tpl-MNI152NLin2009cAsym_res-02_T1w_reference.nii.gz --transform /work/temp_data_GesteSigne/fmriprep_22_0_wf/single_subject_pilote2_wf/anat_preproc_wf/anat_norm_wf/_template_MNI152NLin2009cAsym/registration/ants_t1_to_mniComposite.h5 --transform /work/temp_data_GesteSigne/fmriprep_22_0_wf/single_subject_pilote2_wf/func_preproc_task_rest_run_01_wf/bold_reg_wf/bbreg_wf/concat_xfm/out_fwd.tfm --transform /work/temp_data_GesteSigne/fmriprep_22_0_wf/single_subject_pilote2_wf/func_preproc_task_rest_run_01_wf/unwarp_wf/resample/vol0000_xfm.nii.gz --transform /work/temp_data_GesteSigne/fmriprep_22_0_wf/single_subject_pilote2_wf/func_preproc_task_rest_run_01_wf/bold_std_trans_wf/_std_target_MNI152NLin2009cAsym.resnative/bold_to_std_transform/tmp-65jj8aod/mat2itk_pos-003_xfm-00000.txt

Where all the transformation are concatenated in one step. If you look closely, you should be able to find there the path to the SDC transform.

Does this help?

got it

so in your example

NORMALIZATION XFM --transform /work/temp_data_GesteSigne/fmriprep_22_0_wf/single_subject_pilote2_wf/anat_preproc_wf/anat_norm_wf/_template_MNI152NLin2009cAsym/registration/ants_t1_to_mniComposite.h5 

REGISTRATION? --transform /work/temp_data_GesteSigne/fmriprep_22_0_wf/single_subject_pilote2_wf/func_preproc_task_rest_run_01_wf/bold_reg_wf/bbreg_wf/concat_xfm/out_fwd.tfm 

FIELDWARP(?) what is this one? --transform /work/temp_data_GesteSigne/fmriprep_22_0_wf/single_subject_pilote2_wf/func_preproc_task_rest_run_01_wf/unwarp_wf/resample/vol0000_xfm.nii.gz 

?? this one also looks related to normalization --transform /work/temp_data_GesteSigne/fmriprep_22_0_wf/single_subject_pilote2_wf/func_preproc_task_rest_run_01_wf/bold_std_trans_wf/_std_target_MNI152NLin2009cAsym.resnative/bold_to_std_transform/tmp-65jj8aod/mat2itk_pos-003_xfm-00000.txt

Also, here the unwarp xfm looks like a NIFTI, i was expecting something in ITK format. Is that normal? Where is the HMC?

For me, my command there looks like this

antsApplyTransforms --default-value 0 --float 1 
--input /quest_scratch/fmriprep_23_0_wf/single_subject_HUBS02_wf/func_preproc_ses_1_task_rest_run_01_echo_1_wf/split_opt_comb/vol0000.nii.gz 
--interpolation LanczosWindowedSinc 
--output /quest_scratch/fmriprep_23_0_wf/single_subject_HUBS02_wf/func_preproc_ses_1_task_rest_run_01_echo_1_wf/bold_std_trans_wf/_std_target_MNI152NLin6Asym.res2/bold_to_std_transform/vol0000_xform-00000.nii.gz 
--reference-image /home/zla5191/.cache/templateflow/tpl-MNI152NLin6Asym/tpl-MNI152NLin6Asym_res-02_T1w.nii.gz 
--transform /quest_scratch/fmriprep_23_0_wf/single_subject_HUBS02_wf/anat_preproc_wf/anat_norm_wf/_template_MNI152NLin6Asym/registration/ants_t1_to_mniComposite.h5 
--transform /quest_scratch/fmriprep_23_0_wf/single_subject_HUBS02_wf/func_preproc_ses_1_task_rest_run_01_echo_1_wf/bold_reg_wf/bbreg_wf/concat_xfm/out_fwd.tfm 
--transform identity 
--transform identity

So I see registration and normalization xfms in mine… but I don’t see head motion correction or sdc. Were those already folded into to concat_xfm/out_fwd.tfm? What does transform identity mean? Why don’t I see one for sdc?

Thank you for this discussion. It is indeed interesting to look deeper at what is “under the hood” of fmriprep.

I am not a developper, I can only guess at this point, but one should expect a deformation field to correct for SDC so I m not surprised to see an NIFTI here, that must be the deformation field calculated by the unwarp module.

This:
--transform /work/temp_data_GesteSigne/fmriprep_22_0_wf/single_subject_pilote2_wf/func_preproc_task_rest_run_01_wf/bold_std_trans_wf/_std_target_MNI152NLin2009cAsym.resnative/bold_to_std_transform/tmp-65jj8aod/mat2itk_pos-003_xfm-00000.txt is a text file and must be the linear (6dof) transformation for motion correction. I don’t see anymore this temporary folder tmp-65jj8aod/ in the working directory but if I look in temp_data_GesteSigne/fmriprep_22_0_wf/single_subject_pilote2_wf/func_preproc_task_rest_run_01_wf/bold_hmc_wf/fsl2itk/ I indeed see a mat2itk.txt file that contain all the 6 dof transformation for all the volumes of the run. I guess that this mat2itk.txt file is split at one point to get one file for each volume to be concatenated in the final transformation.

In your case it is indeed strange, it looks like neither the SDC nor the motion correction was done. Could you share the part of the HTML file where they summarize what was done on preprocessing for this particular run? For example, for the example I shared, it looks like this:

Reports for: task rest, run 1.

Summary

  • Original orientation: LAS
  • Repetition time (TR): 1.13s
  • Phase-encoding (PE) direction: Anterior-Posterior
  • Single-echo EPI sequence.
  • Slice timing correction: Not applied
  • Susceptibility distortion correction: PEB/PEPOLAR (phase-encoding based / PE-POLARity)
  • Registration: FreeSurfer bbregister (boundary-based registration, BBR) - 9 dof
  • Non-steady-state volumes: 2

Otherwise, at the very end of the HTML report, do you see any error reported?

--transform /quest_scratch/fmriprep_23_0_wf/single_subject_HUBS02_wf/func_preproc_ses_1_task_rest_run_01_echo_1_wf/bold_reg_wf/bbreg_wf/concat_xfm/out_fwd.tfm 
--transform identity 
--transform identity

You’re using multiecho data. The thing about multi-echo is that you need to correct each echo before you can calculate the T2* map and the optimal combination. Therefore, the optimal combination is already slice-timing/head-motion/susceptibility-distortion corrected, and you can’t reapply them. That’s why you’re seeing identity: the workflow expects to get SDC and HMC transforms, so we need to tell it to use identity transforms in this case.

Now, you should find a bold_preproc_trans_wf.bold_transform node where the echos are corrected.

This is correct. ANTs produces ITK transforms as .mat files for affines and .nii.gz for warps, or .h5 when you have a chain of transforms (which is often the case). For SDC, there is no shift or rotation, just a warp.

I’m going to get a bit in the weeds here, but it may be useful to have this written down for reference. In recent fMRIPrep (I’ll need to check which versions), we actually perform SDC in two ways. One for the BOLD native space (which is relevant to the multi-echo case), and one for the T1w/MNI space.


Here’s an example for the BOLD native one from our test multi-echo dataset:

$ ls fmriprep_23_1_wf/single_subject_02_wf/func_preproc_task_cuedSGT_run_01_echo_1_wf/unwarp_wf/_echoidx_0/resample/
00000                                     _node.pklz                                vol0001_unwarped.nii.gz                   vol0003_xfm.nii.gz                      
00001                                     _report/                                  vol0001_xfm.nii.gz                        vol0004_field.nii.gz                    
00002                                     result_resample.pklz                      vol0002_field.nii.gz                      vol0004_unwarped.nii.gz                 
00003                                     vol0000_field.nii.gz                      vol0002_unwarped.nii.gz                   vol0004_xfm.nii.gz                      
00004                                     vol0000_unwarped.nii.gz                   vol0002_xfm.nii.gz                                                                
_0x4113a609562a41f17c930c47cec3b759.json  vol0000_xfm.nii.gz                        vol0003_field.nii.gz                                                              
_inputs.pklz                              vol0001_field.nii.gz                      vol0003_unwarped.nii.gz                                                 

What you’re seeing here is:

  • 00000 - Head motion transform in ITK format. (Bad name because it’s not used anywhere else.)
  • vol00000_field.nii.gz - The B0 fieldmap resampled into vol00000 using the HMC transform
  • vol00000_xfm.nii.gz - The ITK format deformation field for volume 00000, derived from the field.
  • vol00000_unwarped.nii.gz - The corrected vol00000

(Since only *_unwarped.nii.gz are used by later nodes, the others will generally be deleted unless you have nipype settings configured to avoid deleting unused outputs, which I do for debugging.)

This is an early version of a technique we’ve been developing to reduce the interactions of susceptibility effects and head motion. Basically, the inhomogeneities that cause the distortion happen based on the location of structures in the head, but the distortion occurs in one direction relative to the scanner bore. When you move relative to that direction vector, the distortion changes direction, so can’t be fully corrected by performing head motion correction and then applying a warp as if the head was motionless.

That said, this approach is wildly expensive and gets more complicated when you add additional transforms. So we hadn’t implemented it for other spaces yet, keeping the resampling we’ve been doing for years. For small motion, the effect should be small.

Now, it turns out it does not need to be wildly expensive to do the correction correctly. The 23.2 release will resample all series in the same way, and it will be much cheaper. If you really want to dive into it, I’ve written up the theory here: https://hackmd.io/@effigies/rethinking-resampling

1 Like

Hi @effigies thank you for this! I am learning a lot about the insides of fmriprep.

Re: multi-echo. What you are saying makes sense to me. Basically, HMC and SDC are before optimal combination (at echo-level) and registration to T1 + normalization to MNI are after optimal combination (run-level).

I don’t see a bold_preproc_trans_wf folder i see ones for T1 and std space, but not preproc. Am I missing it?

In the main derivatives folder (not working directory), I do get echo-level (and one OC) outputs which I assumed have had HMC and SDC applied to them. Is that the same info that would have been in bold_preproc_trans_wf?

Also, that last transform from-scanner_to-boldref_mode-image_xfm… what does that include? just SDC? is this the initial boldref that was used for HMC? Also, final question - if I provide multiple echos of sbrefs (i have 5 echos, 4 sbref echos), how does it pick which one to use for the boldref? I assume the boldref is not an optimal combination of sbrefs

Sorry, instead of bold_preproc_trans_wf, it’s just the unwarp_wf.

unwarp_wf/_echoidx_0/resample/

Yes, they have.

That’s the head motion transforms.

Shortest echo.

Thank you @effigies for all of this.

I am looking at that data (in native space but already optimally combined with HMC and SDC applied) and for some reason it does not seem to be registered well with the initial_boldref.

Shouldn’t they match extremely well (since HMC used the intial boldref?)?

The orange is desc-preproc_bold.nii.gz (optimally combined preprocessed native bold) and the gray is the initial boldref that i got from the working directory

Also, on the preprocessed native space data, I am noticing a strange amount of head movement. which is confusing because HMC has already been applied. Do you have any idea why this is?

screenclip2_hmc.mov - Google Drive (updated to have higher quality vid)

I would compare to the boldref in your output directory. It’s possible that your motion estimates (volume to boldref) are off due to the use of the sbref for motion correction. That’s something that will be fixed in the next release.

As to your motion question, I’m not sure if it’s just the size of the GIF, but I’m not seeing noticeable motion even full-screened. Ah, yes. I see that. My first guess would be that it’s also the sbref issue. You could try running with --ignore sbref and see if you get better results.

sorry! that was my fault - the video got compressed when I made the gif.

Here is the video screenclip2_hmc.mov - Google Drive

It looks like you were the OP of the other recentish thread on this, so you’re probably aware of the sbref issue: Poor func/anatomical registration with fmriprep (21.0.2) on multi-echo data set

Oh yes you’re right it does look better aligned to this boldref (this is final_boldref right? based on the worse gray/white contrast, I presume it is).

From the documentation I thought that HMC was to initial_boldref?

I will run it with --ignore sbref and see how that goes. Why would that result in motion?

Yes! In the interim, I am trying to make a script where we could run fMRIprep and then I could use the output transforms + intermediate files and just swap out the registration step with our own registration (hence why I was looking for the SDC transform) and apply all the transforms together at the end

Yes, but the final boldref is currently calculated from the corrected time series. So I guess it had to be aligned, as I think about it.

According to @jsein’s post ([mcflirt related behavour] FMRIPREP: Strange motion estimations in confounds_timeseries.tsv (lots of zeros)), MCFLIRT under-estimates motion when aligning BOLD to sbref if the contrasts are sufficiently different. So it’s not inducing motion so much as failing to correct for motion.

Got it! I will give this a try and report back

@effigies FYI after ignoring sbref, the motion in the native optimally combined preprocessed BOLD looks much much better. I will look forward to that fix in the next version of fmriprep

1 Like