ValueError: Fieldmap-less (SyN) estimation was requested, but PhaseEncodingDirection information appears to be absent. fMRIprep v. 22.0.2

Nuttaon · October 27, 2022, 10:09pm

Hi,

I added the -B arguments. The script ran successfully but it still showed “susceptibility distortion correction: none” in the report.

Here is the script

#User inputs:
bids_root_dir=/scratch/g/jgoveas/R01_nii/baseline_mid/PROJECT/bidstest5/
subj=3172101920
nthreads=4
mem=40 #gb
container=singularity #docker or singularity

#Begin:

#Convert virtual memory from gb to mb
mem=echo "${mem//[!0-9]/}"
mem_mb=echo $(((mem*1000)-5000))

export FS_LICENSE=/scratch/g/jgoveas/R01_nii/baseline_mid/bids/license.txt \
singularity run
-B /scratch/g/jgoveas/R01_nii/baseline_mid
-B /scratch/g/jgoveas/R01_nii/baseline_mid/PROJECT
-B /scratch/g/jgoveas/R01_nii/baseline_mid/PROJECT/bidstest5
–cleanenv /scratch/g/jgoveas/R01_nii/baseline_mid/fmriprep-22.0.2.simg $bids_root_dir $bids_root_dir/derivatives
participant
–participant-label $subj
–skip-bids-validation
–md-only-boilerplate
–fs-license-file /scratch/g/jgoveas/R01_nii/baseline_mid/bids/license.txt
–fs-no-reconall
–output-spaces MNI152NLin2009cAsym:res-2
–output-spaces func:res-native
–output-spaces MNI152NLin2009cAsym:res-native
–nthreads $nthreads
–mem_mb mem_mb \ --ignore slicetiming \ -w /scratch/g/jgoveas/R01_nii/baseline_mid/TOPUP_{subj}

Here is the tree hierarchy:
sub-3172101920
└── ses-1
├── anat
│ ├── sub-3172101920_ses-1_FLAIR.json
│ ├── sub-3172101920_ses-1_FLAIR.nii
│ ├── sub-3172101920_ses-1_T1w.json
│ └── sub-3172101920_ses-1_T1w.nii
├── fmap
│ ├── sub-3172101920_ses-1_acq-mid1_dir-AP_epi.json
│ ├── sub-3172101920_ses-1_acq-mid1_dir-AP_epi.nii
│ ├── sub-3172101920_ses-1_acq-mid1_dir-PA_epi.json
│ ├── sub-3172101920_ses-1_acq-mid1_dir-PA_epi.nii
│ ├── sub-3172101920_ses-1_acq-mid2_dir-AP_epi.json
│ ├── sub-3172101920_ses-1_acq-mid2_dir-AP_epi.nii
│ ├── sub-3172101920_ses-1_acq-mid2_dir-PA_epi.json
│ └── sub-3172101920_ses-1_acq-mid2_dir-PA_epi.nii
└── func
├── sub-3172101920_ses-1_task-mid1_bold.json
├── sub-3172101920_ses-1_task-mid1_bold.nii
├── sub-3172101920_ses-1_task-mid2_bold.json
└── sub-3172101920_ses-1_task-mid2_bold.nii

Here is one of the fmap.json
{
“Modality”: “MR”,
“MagneticFieldStrength”: 3,
“ImagingFrequency”: 127.772,
“Manufacturer”: “GE”,
“PulseSequenceName”: “epi”,
“InternalPulseSequenceName”: “EPI”,
“ManufacturersModelName”: “SIGNA Premier”,
“InstitutionName”: “MCW Premier”,
“DeviceSerialNumber”: “00000000FLORIO1X”,
“StationName”: “MCWPREM”,
“BodyPartExamined”: “BRAIN”,
“PatientPosition”: “HFS”,
“ProcedureStepDescription”: “GovGriefO1”,
“SoftwareVersions”: “27\LX\MR Software release:RX27.0_R03_1850.a”,
“MRAcquisitionType”: “2D”,
“SeriesDescription”: “SE Map PE2”,
“ProtocolName”: “Goveas_GovGriefR01_10052”,
“ScanningSequence”: “EP\RM”,
“SequenceVariant”: “NONE”,
“ScanOptions”: “SAT_GEMS\MP_GEMS\EPI_GEMS\FS”,
“ImageType”: [“ORIGINAL”, “PRIMARY”, “OTHER”],
“SeriesNumber”: 26,
“AcquisitionTime”: “10:20:1.000000”,
“AcquisitionNumber”: 1,
“TriggerDelayTime”: 14677,
“SliceThickness”: 2.4,
“SpacingBetweenSlices”: 2.4,
“SAR”: 0.20231,
“EchoTime”: 0.08,
“RepetitionTime”: 7.4,
“FlipAngle”: 90,
“PhaseEncodingPolarityGE”: “Unflipped”,
“CoilString”: “RM:32NovaHead2”,
“PercentPhaseFOV”: 100,
“PercentSampling”: 100,
“AcquisitionMatrixPE”: 90,
“ReconMatrixPE”: 128,
“EffectiveEchoSpacing”: 0.000403654,
“TotalReadoutTime”: 0.051264,
“IntendedFor”: “ses-1/func/sub-3172101920_ses-1_task-mid1_bold.nii”,
“PixelBandwidth”: 3906.25,
“PhaseEncodingDirection”: “j-”,
“SliceTiming”: [
0,
3.7,
0.123333,
3.82333,
0.246667,
3.94667,
0.37,
4.07,
0.493333,
4.19333,
0.616667,
4.31667,
0.74,
4.44,
0.863333,
4.56333,
0.986667,
4.68667,
1.11,
4.81,
1.23333,
4.93333,
1.35667,
5.05667,
1.48,
5.18,
1.60333,
5.30333,
1.72667,
5.42667,
1.85,
5.55,
1.97333,
5.67333,
2.09667,
5.79667,
2.22,
5.92,
2.34333,
6.04333,
2.46667,
6.16667,
2.59,
6.29,
2.71333,
6.41333,
2.83667,
6.53667,
2.96,
6.66,
3.08333,
6.78333,
3.20667,
6.90667,
3.33,
7.03,
3.45333,
7.15333,
3.57667,
7.27667 ],
“ImageOrientationPatientDICOM”: [
1,
-0,
0,
-0,
1,
0 ],
“InPlanePhaseEncodingDirectionDICOM”: “COL”,
“ConversionSoftware”: “dcm2niix”,
“ConversionSoftwareVersion”: “v1.0.20210317”
}

Please let me know what is missing.

Thank you for your help.

Steven · October 27, 2022, 11:44pm

You need the phase encoding direction set in your BOLD json file to use SYN-SDC.

Nuttaon · October 28, 2022, 12:30am

Hi,

Thank you for your help. I have “PhaseEncodingDirection”: “j-”, in the BOLD.json. Is this ok or do I need to modify it?

Thanks again.

jsein · October 28, 2022, 1:24pm

If you want to use your fmap/_epi images for Fieldmap correction (since in your last version of your script you don’t use --ignore fieldmaps), you need to provide the "IntendedFor" tag in the fmap/epi.json files or "B0FieldSource"/"B0FieldIdentifier" tags in func.json and fmap/epi.json files respectively.

Nuttaon · October 29, 2022, 2:23pm

Hi,

I added "B0FieldSource" /"B0FieldIdentifier" tags in func.json and fmap/epi.json files respectively and still have "IntendedFor" tag in the fmap/epi.json files. However, the susceptibility distortion correction is still not run.

Note: Bids validated. I cleaned the work folder every time. The data was collected using GE scanner.

Here is the fmap.json:

"EffectiveEchoSpacing": 0.000403654,
"TotalReadoutTime": 0.051264,
    "IntendedFor": "ses-1/func/sub-3172101920_ses-1_task-mid1_bold.nii",
"PixelBandwidth": 3906.25,
"PhaseEncodingDirection": "j-",
    "B0FieldIdentifier": "pepolar_fmap0",

Here is the corresponding Bold.json:

"EffectiveEchoSpacing": 0.000588944,
"TotalReadoutTime": 0.052416,
"PixelBandwidth": 5555.56,
"PhaseEncodingDirection": "j-",
    "B0FieldSource": "pepolar_fmap0",

Here is the fmriprep command:

singularity run \
-B /scratch/g/jgoveas/R01_nii/baseline_mid \
-B /scratch/g/jgoveas/R01_nii/baseline_mid/PROJECT \
-B /scratch/g/jgoveas/R01_nii/baseline_mid/PROJECT/bidstest5 \
--cleanenv /scratch/g/jgoveas/R01_nii/baseline_mid/fmriprep-22.0.2.simg \
$bids_root_dir $bids_root_dir/derivatives \
participant \
--participant-label $subj \
--skip-bids-validation \
--md-only-boilerplate \
--fs-license-file /scratch/g/jgoveas/R01_nii/baseline_mid/bids/license.txt \
--fs-no-reconall \
--output-spaces MNI152NLin2009cAsym:res-2 \
--nthreads $nthreads \
--mem_mb $mem_mb \
--ignore slicetiming \
-w /scratch/g/jgoveas/R01_nii/baseline_mid/TOPUP_${subj}

Thank you for your help.

jsein · October 29, 2022, 9:29pm

Thank you for providing those details. We must be still missing something here.

I guess you correctly wrote the B0FiledIdentifier and ÌntendedFor fields in both AP and PA _epi.json files in the fmap/ folder?
Did you check that dir-AP_epi.json and dir-PA_epi.json files have actually opposite phase encoding direction (i.e "-j"and "j"respectively)?
Why do you keep --skip-bids-validationoption in your command? Would you get an error if you remove it?
Would you mind sharing the beginning of the output log of FMRIPREP, where details are thrown about the detection of fieldmaps and the identification of functional runs to be corrected for SDC?

Nuttaon · October 30, 2022, 8:07pm

Thank you for being patient with me.

Yes, I believed that I have the B0FiledIdentifier and ÌntendedFor fields in both AP and PA _epi.json files in the fmap/ folder; and the AP and PA were also correct.

I reran with --skip-bids-validation. There were some warning regarding missing README file. The fmriprep ran successfully otherwise.

Below is the beginning of the output log. There were nothing about sdc or fieldmaps. (nothing showed up when using the find function for those two keywords either).

221030-09:34:17,968 nipype.workflow IMPORTANT:
Building fMRIPrep’s workflow:
* BIDS dataset path: /scratch/g/jgoveas/R01_nii/baseline_mid/PROJECT/bidstest5.
* Participant list: [‘3172101920’].
* Run identifier: 20221030-093359_ba671b65-2560-4ecb-afdc-1dbd47b3ca7a.
* Output spaces: MNI152NLin2009cAsym:res-2.
* Pre-run FreeSurfer’s SUBJECTS_DIR: /scratch/g/jgoveas/R01_nii/baseline_mid/PROJECT/bidstest5/derivatives/sourcedata/freesurfer.
221030-09:34:19,758 nipype.workflow INFO:
No single-band-reference found for sub-3172101920_ses-1_task-mid1_bold.nii.
221030-09:34:19,977 nipype.workflow INFO:
No single-band-reference found for sub-3172101920_ses-1_task-mid2_bold.nii.
221030-09:34:23,182 nipype.workflow INFO:
fMRIPrep workflow graph with 522 nodes built successfully.
221030-09:34:36,142 nipype.workflow IMPORTANT:
fMRIPrep started!
221030-09:34:39,390 nipype.workflow INFO:
[Node] Setting-up “fmriprep_22_0_wf.single_subject_3172101920_wf.anat_preproc_wf.brain_extraction_wf.full_wm” in “/scratch/g/jgoveas/R01_nii/baseline_mid/TOPUP2_3172101920/fmriprep_22_0_wf/single_subject_3172101920_wf/anat_preproc_wf/brain_extraction_wf/full_wm”.
221030-09:34:39,413 nipype.workflow INFO:
[Node] Executing “full_wm” <nipype.interfaces.utility.wrappers.Function>
221030-09:34:39,570 nipype.workflow INFO:
[Node] Setting-up “fmriprep_22_0_wf.single_subject_3172101920_wf.anat_preproc_wf.brain_extraction_wf.lap_tmpl” in “/scratch/g/jgoveas/R01_nii/baseline_mid/TOPUP2_3172101920/fmriprep_22_0_wf/single_subject_3172101920_wf/anat_preproc_wf/brain_extraction_wf/lap_tmpl”.
221030-09:34:39,573 nipype.workflow INFO:
[Node] Setting-up “fmriprep_22_0_wf.single_subject_3172101920_wf.anat_preproc_wf.brain_extraction_wf.res_tmpl” in “/scratch/g/jgoveas/R01_nii/baseline_mid/TOPUP2_3172101920/fmriprep_22_0_wf/single_subject_3172101920_wf/anat_preproc_wf/brain_extraction_wf/res_tmpl”.
221030-09:34:39,592 nipype.workflow INFO:
[Node] Executing “res_tmpl” <niworkflows.interfaces.nibabel.RegridToZooms>
221030-09:34:39,592 nipype.workflow INFO:
[Node] Executing “lap_tmpl” <nipype.interfaces.ants.utils.ImageMath>
221030-09:34:40,113 nipype.workflow INFO:
[Node] Finished “full_wm”, elapsed time 0.690009s.

Thank you.

jsein · October 30, 2022, 9:45pm

Being only a regular user of FMRIPREP, I am running out of suggestions to offer. One last thing, which should be unrelated, is the presence of --fs-no-reconall which was not advised to use earlier by @Steven.

When you run the bids validator, does fmap shows up as one available modality?

It looks like the fmap folder is not even seen by FMRIPREP!

I would be curious to try a minimal sample of your dataset on my side!

Nuttaon · October 30, 2022, 10:31pm

Yep, the bids validator does see fmap folder. Yes, I totally agree. It seems that fMRIprep doesn’t see it at all.

When I used the previous version (20.2.3) to run with fmap or without (–use-syn-sdc), both worked.

I still wonder what “ValueError: Fieldmap-less (SyN) estimation was requested, but PhaseEncodingDirection information appears to be absent.” mean when using --use-syn-sdc.

That would be great if you can try running it. Thanks!

Steven · November 1, 2022, 8:05pm

Are you absolutely sure you are using fMRIPrep 22.0.2? Because as you can see in the changelog (What’s new — fmriprep version documentation) there was a bug with using SYN and --ignore-fieldmaps that was not patched until 22.0.2. If you run fmriprep -v using your 22.0.2 container, does the terminal return 22.0.2?

Nuttaon · November 3, 2022, 8:36pm

Hi,

Yep, it shows 22.0.2.

Thank you.

jsein · November 4, 2022, 12:11pm

Hi @Nuttaon , I played with your dataset and indeed, you may have uncovered a bug!

I run your dataset with either FMRIPREP v21.02 or v22.02, and in one case (v21.0.2) SDC correction is run, and in the other case (v22.0.2) , it is not run!

I used exactly the same dataset, and the same script, just changing the singularity image and the name of the working directory, the output directory and the freesurfer directory.

singularity run -B /scratch/jsein/BIDS:/work,$HOME/.templateflow:/opt/templateflow --cleanenv /scratch/jsein/my_images/fmriprep-22.0.2.simg \
		 --fs-license-file /work/freesurfer/license.txt /work/$study /work/$study/derivatives/fmriprep22  \
		 participant --participant-label $sub \
		 -w /work/temp_data22_${study}\
		 --mem-mb 50000 --omp-nthreads 10 --nthreads 12  \
		 --fd-spike-threshold 0.5 --dvars-spike-threshold 2.0 --bold2t1w-dof 6 --debug fieldmaps --write-graph  \
		 --output-spaces T1w MNI152NLin2009cAsym --ignore slicetiming  --fs-subjects-dir /work/$study/derivatives/fmriprep22/sourcedata/freesurfer \

or

singularity run -B /scratch/jsein/BIDS:/work,$HOME/.templateflow:/opt/templateflow --cleanenv /scratch/jsein/my_images/fmriprep-21.0.2.simg \
		 --fs-license-file /work/freesurfer/license.txt /work/$study /work/$study/derivatives/fmriprep21  \
		 participant --participant-label $sub \
		 -w /work/temp_data21_${study}\
		 --mem-mb 50000 --omp-nthreads 10 --nthreads 12  \
		 --fd-spike-threshold 0.5 --dvars-spike-threshold 2.0 --bold2t1w-dof 6 --debug fieldmaps --write-graph  \
		 --output-spaces T1w MNI152NLin2009cAsym --ignore slicetiming  --fs-subjects-dir /work/$study/derivatives/fmriprep21/sourcedata/freesurfer \

jsein · November 4, 2022, 3:02pm

It is very strange, I don’t understand what is going on.

I can’t wrap my mind around the problem right now:

What is sure is that FMRIPREP v21.0.2 systematically “detects” the fieldmaps.
I had previously used FMRIPREP 22.0.2 where the fieldmaps were correctly detected. But If I create a new directory in which I copy-paste a subject from the project that worked, and I re-run FMRIPREP v22.0.2 (same code that worked previously) on this new project (but the same subject!), the field maps are not detected anymore!
-Crazier: if now I copy the subject from @Nuttaon into the BIDS directory for which FMRIPREP v22.0.2 was detecting the fieldmaps, it does detect the fieldmap!!!

I don’t know FMRIPREP deep enough to know what is the heuristic for the selection of the workflows, to debug further.

Nuttaon · November 4, 2022, 7:31pm

Hi @jsein ,

Thank you so much for testing this! I still have no luck. My experience is that FMRIPREP version 20.2.3 consistently detects the fieldmaps. While, version 22.0.2 never detected them (I tried singularity, docker, and python on two different platforms: OS and Linux, and on different task and different subjects). I also tried using the updated dcm2niix.

This is a silly question. Can it be dataset_description.json?

jsein · November 4, 2022, 7:42pm

I went down the rabbit hole: what makes FMRIPREP v22.0.2 not “recognize” the fieldmaps is the fact that in your dataset_description.json , you use:

    "DatasetType": "derivative",
    "GeneratedBy": [
        {
            "Name": "Manual"
        }
    ],

I don’t know what is the meaning of these two fields, and how they are related to the behavior of the fieldmap detection by FMRIPREP v22.0.2, but it is fact. If you remove those fields from dataset_description.json, the SDC correction is run by FMRIPREP v 22.0.2!

This does not affect FMRIPREP v21.0.2 or earlier.

jsein · November 4, 2022, 9:01pm

A long topic but I learnt something in the process.

This DatasetType defined in dataset_description.json defines the “scope” and how pipeline with interact with the dataset.

If we look here:

github.com

nipreps/sdcflows/blob/master/sdcflows/utils/wrangler.py

# emacs: -*- mode: python; py-indent-offset: 4; indent-tabs-mode: nil -*-
# vi: set ft=python sts=4 ts=4 sw=4 et:
#
# Copyright 2021 The NiPreps Developers <nipreps@gmail.com>
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# We support and encourage derived works from this project, please read
# about our expectations at
#

This file has been truncated. show original

line: 271:

    base_entities = {
        "subject": subject,
        "extension": [".nii", ".nii.gz"],
        "scope": "raw",  # Ensure derivatives are not captured
    }

This may suggest that SDCFlows does not deal with datasets whose “scope” (The "DataType" defined in dataset_description.json) is "derivative"`.

PyBIS seems to be involved in this too. Well, a whole new world…

Nuttaon · November 9, 2022, 4:03pm

Hi @jsein,

Thanks you so much for looking into this and helping me solve this issue! I learnt a lot during this whole process.

Nuttaon

AustinBipolar · May 5, 2023, 11:21pm

Hey, just wanted to say that this is still an issue for me in fMRI prep v 23.0.2. I never would have even considered this issue if I hadn’t read it here. Is it fixable?

Jen

Steven · May 5, 2023, 11:29pm

@AustinBipolar Please open a new issue under the Software Support category making sure to fill in all requested information in the prepopulated template.

Best,
Steven

Steven · May 5, 2023, 11:30pm