Have multiple AP and PA images to generate multiple fieldmaps (1 for each func run) but fmriprep claims only 1 fieldmap can be generated/is available

Software Support
1

Summary of what happened:

I have an AP and PA that were acquired prior to each func run, but fmriprep claims that only 1 fieldmap can be generated: A total of 1 fieldmaps were found available within the input BIDS structure for this particular subject. A B0-nonuniformity map (or fieldmap) was estimated based on two (or more) echo-planar imaging (EPI) references with topup (Andersson, Skare, and Ashburner (2003); FSL 6.0.5.1:57b01774). This is copied from the html.

I have specified in each .json file of the AP and PA images which func run they are intended for like so: "IntendedFor": ["func/sub-SPIN001_task-rest_run-02_bold.nii.gz"],

Everything else with fmriprep seems quite perfect. I made sure to pass the bids-validator with no errors prior to running fmriprep.

Command used (and if a helper script was used, a link to the helper script or the command generated):

singularity run --cleanenv -B /data $sif_loc $input_dir $output_dir participant --participant-label $subject --nthreads 16 --omp-nthreads 16 --mem_mb 32000 --skip-bids-validation --output-spaces MNIPediatricAsym:cohort-4:res-2 --use-aroma --skull-strip-template NKI --fs-license-file $fs_license --work-dir $work_dir

Version:

fMRIPrep v23.0.0rc0

Environment (Docker, Singularity, custom installation):

Singularity

Data formatted according to a validatable standard? Please provide the output of the validator:

Yes. There is one strange warning:

	1: [WARN] Not all subjects contain the same files. Each subject should contain the same number of files with the same naming unless some files are known to be missing. (code: 38 - INCONSISTENT_SUBJECTS)
		./sub-SPIN001/fmap/sub-SPIN001_acq-rest_dir-AP_epi.json
			Evidence: Subject: sub-SPIN001; Missing file: sub-SPIN001_acq-rest_dir-AP_epi.json
		./sub-SPIN001/fmap/sub-SPIN001_acq-rest_dir-AP_epi.nii.gz
			Evidence: Subject: sub-SPIN001; Missing file: sub-SPIN001_acq-rest_dir-AP_epi.nii.gz
		./sub-SPIN001/fmap/sub-SPIN001_acq-rest_dir-AP_run-03_epi.json
			Evidence: Subject: sub-SPIN001; Missing file: sub-SPIN001_acq-rest_dir-AP_run-03_epi.json
		./sub-SPIN001/fmap/sub-SPIN001_acq-rest_dir-AP_run-03_epi.nii.gz
			Evidence: Subject: sub-SPIN001; Missing file: sub-SPIN001_acq-rest_dir-AP_run-03_epi.nii.gz
		./sub-SPIN001/fmap/sub-SPIN001_acq-rest_dir-PA_epi.json
			Evidence: Subject: sub-SPIN001; Missing file: sub-SPIN001_acq-rest_dir-PA_epi.json
		./sub-SPIN001/fmap/sub-SPIN001_acq-rest_dir-PA_epi.nii.gz
			Evidence: Subject: sub-SPIN001; Missing file: sub-SPIN001_acq-rest_dir-PA_epi.nii.gz
		./sub-SPIN001/fmap/sub-SPIN001_acq-rest_dir-PA_run-03_epi.json
			Evidence: Subject: sub-SPIN001; Missing file: sub-SPIN001_acq-rest_dir-PA_run-03_epi.json
		./sub-SPIN001/fmap/sub-SPIN001_acq-rest_dir-PA_run-03_epi.nii.gz
			Evidence: Subject: sub-SPIN001; Missing file: sub-SPIN001_acq-rest_dir-PA_run-03_epi.nii.gz
		./sub-SPIN001/func/sub-SPIN001_task-rest_bold.json
			Evidence: Subject: sub-SPIN001; Missing file: sub-SPIN001_task-rest_bold.json
		./sub-SPIN001/func/sub-SPIN001_task-rest_bold.nii.gz
			Evidence: Subject: sub-SPIN001; Missing file: sub-SPIN001_task-rest_bold.nii.gz
		... and 1592 more files having this issue (Use --verbose to see them all).

The above output is extremely weird because I don’t even have some of the above files. I did do some file cleaning/removing and ran the bids-validator after. The files should be:

ender.540> cd sub-SPIN001
ender.541> ls
anat  fmap  func
ender.542> ls -R
.:
anat  fmap  func

./anat:
sub-SPIN001_T1w.json  sub-SPIN001_T1w.nii.gz

./fmap:
sub-SPIN001_acq-rest_dir-AP_run-01_epi.json    sub-SPIN001_acq-rest_dir-PA_run-01_epi.json
sub-SPIN001_acq-rest_dir-AP_run-01_epi.nii.gz  sub-SPIN001_acq-rest_dir-PA_run-01_epi.nii.gz
sub-SPIN001_acq-rest_dir-AP_run-02_epi.json    sub-SPIN001_acq-rest_dir-PA_run-02_epi.json
sub-SPIN001_acq-rest_dir-AP_run-02_epi.nii.gz  sub-SPIN001_acq-rest_dir-PA_run-02_epi.nii.gz

./func:
sub-SPIN001_task-rest_run-01_bold.json    sub-SPIN001_task-rest_run-02_bold.json
sub-SPIN001_task-rest_run-01_bold.nii.gz  sub-SPIN001_task-rest_run-02_bold.nii.gz

Relevant log outputs (up to 20 lines):

None.

Screenshots / relevant information:

None.

2

Hi @Hannah.Choi,

Looks like fMRIPrep is doing SDC as intended, in that it is making a fieldmap based off of your AP/PA pair of images. Do SDC results look good?

Best,
Steven

3

Hi @Steven,

Yes, fMRIPrep is indeed doing SDC as intended and it looks pretty good, but I was wondering if it could make separate fieldmaps for each functional bold run since we have two pairs, one for each bold run. Our scanning protocol is so that the functional runs are acquired some time part (T2w and FLAIR are acquired in between). Does fMRIPrep only generate one fieldmap per sub despite there being 2 pairs and different IntendedFor specifications?

Thanks again,
Hannah

4

Ah sorry I misunderstood your original post, I thought you were asking one distortion performance for AP and PA separately, my bad.

Do you get this result for all of your subjects? Just in case can you try upgrading to the stable release?

Best,
Steven

5

No worries! I tested for a single 2-run subject, so I will try to test more subs. I will first make sure to upgrade to the stable release. I will let you know how it goes - probably sometime tomorrow.

Thanks for the suggestions,
Hannah

6

The most reliable approach is using B0FieldIdentifier/B0FieldSource. 23.1.0 will do a better job of respecting IntendedFor, with RF: Split PEPolar fieldmaps by intent, if available by effigies · Pull Request #342 · nipreps/sdcflows · GitHub.

7

@Steven @effigies
Thank you again for your suggestions. I tried the upgrade to 23.0.2 and handpicked some subjects with 2 runs. Everything runs great.
I just checked/learned something. I went to the fmap directory and opened up the .json.

{
  "AnatomicalReference": "sub-SPIN039_acq-rest_fmapid-auto00000_desc-epi_fieldmap.nii.gz",
  "AssociatedCoefficients": [
    "sub-SPIN039_acq-rest_fmapid-auto00000_desc-coeff_fieldmap.nii.gz"
  ],
  "B0FieldIdentifier": "auto_00000",
  "IntendedFor": [
    "func/sub-SPIN039_task-rest_run-01_bold.nii.gz",
    "func/sub-SPIN039_task-rest_run-02_bold.nii.gz"
  ],
  "RawSources": [
    "/data/ncl-mb13/SPIN_REST/sub-SPIN039/fmap/sub-SPIN039_acq-rest_dir-AP_run-01_epi.nii.gz",
    "/data/ncl-mb13/SPIN_REST/sub-SPIN039/fmap/sub-SPIN039_acq-rest_dir-AP_run-02_epi.nii.gz",
    "/data/ncl-mb13/SPIN_REST/sub-SPIN039/fmap/sub-SPIN039_acq-rest_dir-PA_run-01_epi.nii.gz",
    "/data/ncl-mb13/SPIN_REST/sub-SPIN039/fmap/sub-SPIN039_acq-rest_dir-PA_run-02_epi.nii.gz"
  ],
  "Units": "Hz"
}

So, I’m assuming fmriprep is using all AP and PA images to generate a single fieldmap. I guess it is not the worst thing ever because the images are all for the subject, but I’m guessing that it is more appropriate to have the fieldmaps generated according to run (especially since the AP and PA pairs were acquired prior to each func run).

@effigies I looked into B0FieldIdentifier /B0FieldSource. I also chatted a bit with ChatGPT to see if I understood correctly. Would the following be an appropriate way to handle it?

image
image1216×1470 101 KB

8

hmm that shouldn’t be happening. I just looked at my own data run on v23 and it makes different fieldmaps for each bold run as intended. Can you share the contents of the JSON files?

Best,
Steven

9

@Steven I tried experimenting with B0FieldIdentifier and B0FieldSource and believe I have something working. Say for a run, the jsons of the AP and PA images have:

"B0FieldIdentifier": "sub-SPIN117_run-01",
"IntendedFor": ["func/sub-SPIN117_task-rest_run-01_bold.nii.gz"],

and the func image has:

"B0FieldSource": "sub-SPIN117_run-01",

Then, we get a fieldmap json from fmriprep looking like:

{
  "AnatomicalReference": "sub-SPIN117_acq-rest_run-01_fmapid-sub-SPIN117run-01_desc-epi_fieldmap.nii.gz",
  "AssociatedCoefficients": [
    "sub-SPIN117_acq-rest_run-01_fmapid-sub-SPIN117run-01_desc-coeff_fieldmap.nii.gz"
  ],
  "B0FieldIdentifier": "sub-SPIN117_run-01",
  "IntendedFor": [
    "func/sub-SPIN117_task-rest_run-01_bold.nii.gz"
  ],
  "RawSources": [
    "/data/ncl-mb13/SPIN_REST/sub-SPIN117/fmap/sub-SPIN117_acq-rest_dir-AP_run-01_epi.nii.gz",
    "/data/ncl-mb13/SPIN_REST/sub-SPIN117/fmap/sub-SPIN117_acq-rest_dir-PA_run-01_epi.nii.gz"
  ],
  "Units": "Hz"
}

Also, the final report html mentions that 2 fieldmaps have been produced and used, which is great.

If I just used intended for, like I did before, I get the above json. I’ll just copy it over here again:

{
  "AnatomicalReference": "sub-SPIN039_acq-rest_fmapid-auto00000_desc-epi_fieldmap.nii.gz",
  "AssociatedCoefficients": [
    "sub-SPIN039_acq-rest_fmapid-auto00000_desc-coeff_fieldmap.nii.gz"
  ],
  "B0FieldIdentifier": "auto_00000",
  "IntendedFor": [
    "func/sub-SPIN039_task-rest_run-01_bold.nii.gz",
    "func/sub-SPIN039_task-rest_run-02_bold.nii.gz"
  ],
  "RawSources": [
    "/data/ncl-mb13/SPIN_REST/sub-SPIN039/fmap/sub-SPIN039_acq-rest_dir-AP_run-01_epi.nii.gz",
    "/data/ncl-mb13/SPIN_REST/sub-SPIN039/fmap/sub-SPIN039_acq-rest_dir-AP_run-02_epi.nii.gz",
    "/data/ncl-mb13/SPIN_REST/sub-SPIN039/fmap/sub-SPIN039_acq-rest_dir-PA_run-01_epi.nii.gz",
    "/data/ncl-mb13/SPIN_REST/sub-SPIN039/fmap/sub-SPIN039_acq-rest_dir-PA_run-02_epi.nii.gz"
  ],
  "Units": "Hz"
}

Thanks again for the helpful suggestions!

10

Hi Hannah @Hannah.Choi

I also have a similar situation of AP/PA collected before each BOLD func task. I haven’t worked with B0 field source and B0 field identifier before. I’m messing around with my old intendedfor.py script that is only for one field map AP/PA but it doesn’t seem to be working. I have an AP and PA that were acquired prior to each BOLD func run, and one AP/PA collected before DWI diffusion scan. Could you please share your intendedfor.py script as to how you added intendedfor to the .json files. My tree structure is below:

TREE
TREE461×709 118 KB

*In this pic there are 2 runs of each task. So for example AP/PA run-01 would be for task-mid_run-01_bold.json and task-mid_run-02_bold.json and AP/PA run-02 would be for task-nback_run-01_bold.json and task-nback_run-02_bold.json. Run 05 of AP/PA is for DWI.

Did you add it according to acquisition time?

Thanks.

–Subbi Madhavan

11

Hi Subbi,
I’m adding my code below:


def insert_IntendedFor(main_dir):
    """
    Each AP and PA must have an IntendedFor field in their .json. Should be IntendedFor func_file_of_matching_run.
    Syntax: "IntendedFor": ["func/sub-SPIN000_task-rest_bold.nii.gz"],
    """
    for subdir in os.listdir(main_dir):
        if subdir.startswith('sub-'):
            fmap_dir = os.path.join(main_dir, subdir, 'fmap')
            fmap_json_files = [f for f in os.listdir(fmap_dir) if f.endswith('.json')]
            func_dir = os.path.join(main_dir, subdir, 'func')
            if len(fmap_json_files) == 2:
                func_nii_file = None
                for f in os.listdir(func_dir):
                    if f.endswith('.nii.gz'):
                        func_nii_file = f
                        break
                if func_nii_file is not None:
                    for file in fmap_json_files:
                        file_path = os.path.join(fmap_dir, file)
                        with open(file_path, 'r+') as f:
                            lines = f.readlines()
                            if not lines[1].startswith('    "IntendedFor": ["func/{}"],'.format(func_nii_file)):
                                lines.insert(1, '    "IntendedFor": ["func/{}"],\n'.format(func_nii_file))
                                f.seek(0)
                                f.writelines(lines)
                                f.truncate()
            else:
                func_nii_files = {int(re.findall(r'run-(\d+)_', f)[0]): f for f in os.listdir(func_dir) if f.endswith('.nii.gz')}
                for file in fmap_json_files:
                    file_path = os.path.join(fmap_dir, file)
                    with open(file_path, 'r+') as f:
                        lines = f.readlines()
                        run_number = int(re.findall(r'run-(\d+)_', file)[0])
                        if run_number in func_nii_files and not lines[1].startswith('    "IntendedFor": ["func/{}"],'.format(func_nii_files[run_number])):
                            lines.insert(1, '    "IntendedFor": ["func/{}"],\n'.format(func_nii_files[run_number]))
                            f.seek(0)
                            f.writelines(lines)
                            f.truncate()

def check_IntendedFor(main_dir):
    """
    Make sure each AP and PA .json has an IntendedFor field. 
    """
    num_json_files = 0
    num_intended_for_files = 0
    for subject in os.listdir(main_dir):
        if not subject.startswith('sub-'):
            continue

        fmap_dir = os.path.join(main_dir, subject, 'fmap')
        if not os.path.exists(fmap_dir):
            continue

        for file in os.listdir(fmap_dir):
            if file.endswith('.json'):
                num_json_files += 1
                file_path = os.path.join(fmap_dir, file)
                with open(file_path, 'r') as json_file:
                    json_data = json.load(json_file)
                    if 'IntendedFor' in json_data:
                        num_intended_for_files += 1

    print(f"Found {num_intended_for_files} JSON files with IntendedFor fields out of {num_json_files} total JSON files in {main_dir}.")
    if num_intended_for_files == num_json_files:
        print("All JSON files have an IntendedFor field.")
    else:
        print("Not all JSON files have an IntendedFor field.")

def insert_B0FieldSource(main_dir):
    """
    Inserts a `B0FieldSource` field in the JSON files of each functional run.
    The value of the `B0FieldSource` field is `subjectdirectory_runnumber` if the
    functional file has a run number, and `subjectdirectory` otherwise.

    Parameters
    ----------
    main_dir : str
        Path to the directory containing subject directories (e.g., `/data/ncl-mb13/SPIN_REST`).
    """
    # loop through all subject directories in the main directory
    for subject_dir in os.listdir(main_dir):
        if subject_dir.startswith('sub-'):
            func_dir = os.path.join(main_dir, subject_dir, 'func')
            # loop through all JSON files in the func directory
            for file in os.listdir(func_dir):
                if file.endswith('.json'):
                    # get the run number from the filename if it exists
                    match = re.search(r'run-(\d+)', file)
                    if match:
                        run_number = match.group(1)
                    else:
                        run_number = None
                    
                    # create the B0FieldSource value
                    if run_number:
                        b0_field_source = f"{subject_dir}_run-{run_number}"
                    else:
                        b0_field_source = subject_dir
                    
                    # insert the B0FieldSource field in the JSON file if it's not already present
                    file_path = os.path.join(func_dir, file)
                    with open(file_path, 'r+') as f:
                        lines = f.readlines()
                        if not lines[1].startswith('    "B0FieldSource":'):
                            lines.insert(1, f'    "B0FieldSource": "{b0_field_source}",\n')
                            f.seek(0)
                            f.writelines(lines)
                            f.truncate()

def insert_B0FieldIdentifier(main_dir):
    """
    Inserts a `B0FieldIdentifier` field in the JSON files of each fmap run.
    The value of the `B0FieldIdentifier` field is `subjectdirectory_runnumber` if the
    functional file has a run number, and `subjectdirectory` otherwise.

    Parameters
    ----------
    main_dir : str
        Path to the directory containing subject directories (e.g., `/data/ncl-mb13/SPIN_REST`).
    """
    # loop through all subject directories in the main directory
    for subject_dir in os.listdir(main_dir):
        if subject_dir.startswith('sub-'):
            fmap_dir = os.path.join(main_dir, subject_dir, 'fmap')
            # loop through all JSON files in the fmap directory
            for file in os.listdir(fmap_dir):
                if file.endswith('.json'):
                    # get the run number from the filename if it exists
                    match = re.search(r'run-(\d+)', file)
                    if match:
                        run_number = match.group(1)
                    else:
                        run_number = None
                    
                    # create the B0FieldIdentifier value
                    if run_number:
                        b0_field_identifier = f"{subject_dir}_run-{run_number}"
                    else:
                        b0_field_identifier = subject_dir
                    
                    # insert the B0FieldIdentifier field in the JSON file if it's not already present
                    file_path = os.path.join(fmap_dir, file)
                    with open(file_path, 'r+') as f:
                        lines = f.readlines()
                        if not lines[2].startswith('    "B0FieldIdentifier":'):
                            lines.insert(2, f'    "B0FieldIdentifier": "{b0_field_identifier}",\n')
                            f.seek(0)
                            f.writelines(lines)
                            f.truncate()

def check_B0Field_values_match(main_dir):
    """
    Checks if the B0FieldIdentifier value in fmap JSON files and B0FieldSource value
    in func JSON files match for each subject's run with the same run number.

    Parameters
    ----------
    main_dir : str
        Path to the directory containing subject directories (e.g., `/data/ncl-mb13/SPIN_REST`).

    Returns
    -------
    bool
        True if all values match, False otherwise.
    """
    all_values_match = True

    # loop through all subject directories in the main directory
    for subject_dir in os.listdir(main_dir):
        if subject_dir.startswith('sub-'):
            func_dir = os.path.join(main_dir, subject_dir, 'func')
            fmap_dir = os.path.join(main_dir, subject_dir, 'fmap')

            # loop through all JSON files in the func directory
            for func_file in os.listdir(func_dir):
                if func_file.endswith('.json'):
                    func_file_path = os.path.join(func_dir, func_file)
                    with open(func_file_path, 'r') as f:
                        func_data = json.load(f)
                        b0_field_source = func_data.get("B0FieldSource", None)

                    # get the run number from the func filename if it exists
                    match = re.search(r'run-(\d+)', func_file)
                    if match:
                        func_run_number = match.group(1)
                    else:
                        func_run_number = None

                    # loop through all JSON files in the fmap directory
                    for fmap_file in os.listdir(fmap_dir):
                        if fmap_file.endswith('.json'):
                            # check if the fmap file has the same run number as the func file
                            fmap_match = re.search(r'run-(\d+)', fmap_file)
                            if fmap_match and fmap_match.group(1) == func_run_number:
                                fmap_file_path = os.path.join(fmap_dir, fmap_file)
                                with open(fmap_file_path, 'r') as f:
                                    fmap_data = json.load(f)
                                    b0_field_identifier = fmap_data.get("B0FieldIdentifier", None)

                                if b0_field_source and b0_field_identifier and b0_field_source != b0_field_identifier:
                                    all_values_match = False
                                    print(f"Mismatch found in subject {subject_dir}:")
                                    print(f"  B0FieldSource: {b0_field_source} (in {func_file})")
                                    print(f"  B0FieldIdentifier: {b0_field_identifier} (in {fmap_file})")

    return all_values_match

Let me know if it makes sense!

Best,
Hannah

12

Hi @Hannah.Choi Thanks for sharing this. Sorry for the late response. Work has been hectic. I’ll let you know if I have any follow up questions.

–Subbi Madhavan