How to best insert "IntendedFor" field in jsons

Hi,

I have beeen searching for an answer to this, but I didn’t really find anything. Feel free to point me to an existing thread if it has already been adressed.

I am wondering how people generally add the “IntendedFor” in their jsons? I am using dcm2niix to convert, so the field is not there. There may be a new version that I am not aware of? Or if there is another converter that automatically adds this; that might be the best solution. I have tried googling for how to edit jsons automatically using both bash and Python scripts, but none of the solutions I found worked.

I have been adding the field manually up until now, but it’s very cumbersome and obviously prone to errors. My dataset has three sessions and two task runs for each subject, so there are 12 correction scans per person (we use the PE-polar approach). I am guessing this is also a challenge for those working with the huge open datasets that are out there.

Any tips are very much appreciated!

According to this post, fw-heudiconv inserts the IntendedFor field automatically. I don’t have any experience with that tool, but it looks like you set your IntendedFor field in the heuristic file. In my program, we generally use a custom function called intended_for_gen() that finds the most recent field map before each functional scan after conversion to BIDS. That way you can remove bad field maps before you assign the IntendedFor field.

I think it’s probably a mistake to assume that everyone uses the same approach for their field maps, however. While I generally default to using the closest field map before each fMRI scan, it might make more sense to grab the closest field map to each scan regardless of whether it’s before or after. I’m not sure there’s a one-size-fits-all solution to this, but those two options can probably help.

EDIT: There is also this heudiconv thread, in which @satra mentions an approach that sounds better (at least for Siemens scanners), although I don’t know if there’s a straightforward implementation out there. The method described compares the shims from different files to assign field maps to scans.

2 Likes

Not particularly elegant, but I’ve done it with R text-wrangling: reading in a json, concatenating the new lines, then writing it out again. An example is in https://openneuro.org/crn/datasets/ds002737/snapshots/1.0.1/files/code:convert_sub01_ses02.R, starting line 108.

2 Likes

EDIT: got caught by preformatted text. Try again.

An additional, inelegant but effective method for bash that I include in a script that does the dicom to nii conversion - it adds the Intended for right after the EchoTime details

cd ${niidir}/sub-$subj
line=$(grep -n '"EchoTime": 0.033,' fmap/sub-${subj}_dir-PA_run-1_epi.json | cut -d : -f 1)
next=1
lineout=$(($line + $next))

array=()
array=(`find func/*bold.nii.gz -type f`)
var=$( IFS=$'\n'; printf "\"${array[*]}"\" )
filenames=$(echo $var | sed 's/ /", "/g')
textin=$(echo -e '"IntendedFor": ['$filenames'],')
sed -i "${lineout}i $textin " fmap/sub-${subj}_dir-PA_run-1_epi.json
2 Likes

So I’ve been testing the R and bash solutions, and they both seem to work (at least for one fieldmap - one run). Thanks to all of you for your great help! I will keep testing and hopefully I can apply it to my 3 ses + 2 runs + 2 fieldmap-per-run-setup :slight_smile:

@tsalo thanks for pointing this out and for your suggestions. I saw that dcm2niix is recommended by the BIDS specification, so I might stick with using that… Also yes, probably there are many ways to acquire the fieldmaps as well! We have them actually after each BOLD run. Our fieldmaps are named the same as the corresponding run (e.g. task_run-1, rest), so it shouldn’t be too hard to work into a script.

A little late to the party but here’s been my solution for one field map with python. I had a version that would point different fieldmaps to different runs but I’ll have to go rooting around for that.

2 Likes

Thanks alot for this!

Here is the code that I wrote which I have been using for my fieldmaps. I hope this can help.

import os
import glob
import json

subjectsPath = os.path.join('/data', 'D2', 'Nifti', 'sub-*')
subjects = glob.glob(subjectsPath)


for subject in subjects:

	#Edit 'ses-1' to be the same as your session
	fmapsPath = os.path.join(subject, 'ses-1', 'fmap', '*.json')
	fmaps = glob.glob(fmapsPath)
	funcsPath = os.path.join(subject, 'ses-1', 'func', '*.nii.gz')
	funcs = glob.glob(funcsPath)

	#substring to be removed from absolute path of functional files
	pathToRemove = subject + '/'
	funcs = list(map(lambda x: x.replace(pathToRemove, ''), funcs))
	for fmap in fmaps:
		with open(fmap, 'r') as data_file:
			fmap_json = json.load(data_file)
		fmap_json['IntendedFor'] = funcs

		with open(fmap, 'w') as data_file:
			fmap_json = json.dump(fmap_json, data_file)
1 Like

@skebaas Thank you! Wow this is very simple and efficient :slight_smile:
and @Jeff_Dennison I only just got around to testing yours as well. It works!

I’m pretty sure I could easily adapt both of these scripts to applying the relevant runs for each json myself. So many solutions, I’m very grateful to you all!

I’m just curious about something, @skebaas. I see that the new jsons in your version, contain all lines on the same line (so everything is dumped on line 1). Does that work / is that like… OK? I am not able to validate either version atm cause I don’t have immediate access to our data.
This is one of the reasons I didn’t manage to do this myself, I thought the json format was very strict and I had to keep the “new line” format it was originally in.

@hannesm from what I understand there are two types of json objects. Normal JSON and JSON lines. JSON lines separates each object onto one line, and I assume it is recognizing all of the data as one object and shoving it onto that line. This can be fixed by converting it to normal JSON.

@hannessm
Also, if you’re interested I have a script that automatically removes the “IntendedFor” statements from the jsons in a subject directory (to be able to quickly fix my mistakes when adding them in). Its useful when you are in the testing phase of adding them in.
Best,
Alex

Sure, thanks! I’m mostly making copies to discard before testing, but maybe someone else also finds this useful:)

To answer your question about making the json data appear on multiple lines, it appears that specifiying indent with the json.dumps() function fixes the problem.

fmap_json = json.dump(fmap_json, data_file, indent=4)

Here is the code for removing the IntendedFor statements from your jsons. This could be useful.

#! /usr/bin/python3
import os 
import glob
import json
'''Remove IntendedFor statement from json files if a mistake was made'''
#grab the epi jsons and store them in a list

#files = os.path.join('/home', 'skebaas', 'Nifti', 'sub-*', 'ses-1', 'fmap', '*.json')
files = os.path.join('/data', 'MNA', 'data', 'Nifti', 'sub-*', 'ses-1', 'fmap', '*.json')
fmaps = glob.glob(files)

#next search through each fmap json and remove 'IntendedFor'
for file in fmaps:

	with open(file, 'r') as data_file:
		data = json.load(data_file)
	try:
		print(f"removing IntendedFor statement from {file}")
		del data['IntendedFor']
	except KeyError:
		print(f"ERROR: could not find IntendedFor statement for {file}")
	with open(file, 'w') as data_file:
		data = json.dump(data, data_file)


1 Like