Using heudiconv with anonymized dicoms?

Summary of what happened:

I’ve downloaded the PARDIP dataset from the NDA repository, and the dicoms were all anonymized prior to being put in the repository. I was able to use heudiconv to pull out a template on a single subject, as a first step, but the dicominfo.tsv file showed the protocol field is ‘anonymous’, the patient id, study description, series description are all ‘anonymous’. The only way to kind of determine what the output nifti file should be called was to look at the sequence_name field. There is an epifid2d1_64 for the resting state, tfl3d1_ns for the T1, *tse2d2_7 for the T2, etc.

I have not successfully used heudiconv before; I got a walkthrough from someone who uses it on their own data so they haven’t dealt with anonymized dicoms. But I worked through their scripts and converted them, to try on the same single subject that I got the dicominfo.tsv file from (running heudiconv with the -c none option, same subject.)

It seems to be working, in that it finds the files, and does actually convert the anatomy and T2 dicoms, puts them in the right place with the right names, etc.

But it doesn’t output the resting state or DTI files–and the output is full of error warnings. I did check that dcm2niix will work by itself on these images, so the error must be something else I’ve done in the heuristic file in trying to get the names and BIDS formats right.

But the error below shows that “Anonymous” is trying to get converted to a float, during the embedder process? Is that an error I need to work around? If I can ignore that I can move on to the other errors.

Command used (and if a helper script was used, a link to the helper script or the command generated):

(the bold and italics is something weird in formatting here; those are all variable names that do actually evaluate properly)

singularity run --cleanenv
/fs/ess/PCON0501/Heudiconv_scripts/heudiconv_latest.sif
–files {dirpath}/{Data}/{subject}_files/*.dcm \ -o {dirpath}/{Study}/Analyses/ \ -f {dirpath}/${Data}/PARDIPheuristic.py
-s $subject
-ss $session
-c dcm2niix
-b
–overwrite ;

Version:

latest --I did the singularity pull command just five days ago.

Environment (Docker, Singularity, custom installation):

Singularity

Data formatted according to a validatable standard? Please provide the output of the validator:

No it isn’t, that’s the problem.

Relevant log outputs (up to 20 lines):

Starting BIDS conversion for subject S0078JWR 1
INFO: Running heudiconv version 0.13.1 latest 0.13.1
INFO: Analyzing 710 dicoms
INFO: Generated sequence info for 1 studies with 8 entries total
WARNING: Heuristic is missing an infotoids method, assigning empty method and using provided subject id S0078JWR. Provide session and locator fields for best results.
INFO: Study session for StudySessionInfo(locator=None, session=‘1’, subject=‘S0078JWR’)

INFO: PROCESSING STARTS: {‘subject’: ‘S0078JWR’, ‘outdir’: ‘/fs/ess/PCON0501/PARDIP/Analyses/’, ‘session’: ‘1’}
INFO: Processing 8 pre-sorted seqinfo entries
INFO: Doing conversion using dcm2niix

INFO: [Node] Finished “embedder”, elapsed time 0.018789s.
230830-10:58:38,750 nipype.workflow WARNING:
Storing result file without outputs
WARNING: Storing result file without outputs
230830-10:58:38,751 nipype.workflow WARNING:
[Node] Error on “embedder” (/tmp/embedmetaiobs88ab/embedder)
WARNING: [Node] Error on “embedder” (/tmp/embedmetaiobs88ab/embedder)
ERROR: Embedding failed: Exception raised while executing Node embedder.

Traceback:
Traceback (most recent call last):
File “/opt/miniconda-py39_4.12.0/lib/python3.9/site-packages/nipype/interfaces/base/core.py”, line 397, in run
runtime = self._run_interface(runtime)
File “/opt/miniconda-py39_4.12.0/lib/python3.9/site-packages/nipype/interfaces/utility/wrappers.py”, line 142, in _run_interface
out = function_handle(**args)
File “”, line 35, in embed_dicom_and_nifti_metadata
File “/opt/miniconda-py39_4.12.0/lib/python3.9/site-packages/dcmstack/dcmstack.py”, line 1156, in parse_and_stack
results = parse_and_group(src_paths,
File “/opt/miniconda-py39_4.12.0/lib/python3.9/site-packages/dcmstack/dcmstack.py”, line 1060, in parse_and_group
meta = extractor(dcm)
File “/opt/miniconda-py39_4.12.0/lib/python3.9/site-packages/dcmstack/extract.py”, line 504, in call
value = self._get_elem_value(elem)
File “/opt/miniconda-py39_4.12.0/lib/python3.9/site-packages/dcmstack/extract.py”, line 421, in _get_elem_value
value = self.conversionselem.VR
ValueError: could not convert string to float: ‘Anonymous’

And it ends with:
INFO: stdout 2023-08-30T16:44:06.005821:Conversion required 0.730743 seconds (0.067720 for core code
).
230830-16:44:06,26 nipype.workflow INFO:
[Node] Finished “convert”, elapsed time 0.780368s.
INFO: [Node] Finished “convert”, elapsed time 0.780368s.
Traceback (most recent call last):
File “/opt/miniconda-py39_4.12.0/bin/heudiconv”, line 8, in
sys.exit(main())
File “/src/heudiconv/heudiconv/cli/run.py”, line 30, in main
workflow(**kwargs)
File “/src/heudiconv/heudiconv/main.py”, line 463, in workflow
prep_conversion(
File “/src/heudiconv/heudiconv/convert.py”, line 255, in prep_conversion
convert(
File “/src/heudiconv/heudiconv/convert.py”, line 635, in convert
tuneup_bids_json_files(bids_outfiles)
File “/src/heudiconv/heudiconv/bids.py”, line 334, in tuneup_bids_json_files
assert HEUDICONV_VERSION_JSON_KEY not in json_
AssertionError

Screenshots / relevant information:

LSB Version: :core-4.1-amd64:core-4.1-noarch:cxx-4.1-amd64:cxx-4.1-noarch:desktop-4.1-amd64:desktop-4.1-noarch:languages-4.1-amd64:languages-4.1-noarch:printing-4.1-amd64:printing-4.1-noarch

(if that is relevant. I’m pretty sure the problem is something about the anonymized dicom tags that I need to set some specific flag for…)

@JessicaTurner - try using the --minmeta flag as an option, which will bypass the dcmstack embedding step, which i think is causing the issue at the moment.

Thanks Satra! That did get rid of a lot of the errors in the output file from Embeddings; but it still ends with an AssertionError after running dcm2niix on the T2 scans (and throwing a Warning: Siemens MoCo? Bogus slice timing" notice on them). I’ll see if I can figure out how to format a dcmconfig file to say -m, which should get rid of the slice timing warnings. But I’m not at all sure how to get rid of that Assertion Error that the JSON_KEY is missing.

You can remove headiconv from the equation and see how dcm2niix performs. Make sure you have the latest version (v1.0.20230411) by running dcm2niix with no arguments. To convert your DICOMs you would run:

dcm2niix /path/to/DICOMs

In general, dcm2niix works fine with anonymization as well as the private tags that provide sequence details are preserved. The tags vary from manufacturer, but assuming Siemens MR, the CSA header is crucial.

You can always see what dcm2niix sees by running is in logorrheic mode. However, this mode generates so much output that I would just get a single DICOM file and place it in a folder without any other files. Then you can run:

dcm2niix -v 2 /path/to/single/DICOM

This will show all the tags that remain in the DICOM data. If the CSA tag is missing for Siemens MR, you will not have a lot of sequence details including slice timing, gradient direction, etc.

Thank you–yes, in desperation I did do that. dcm2niix works fine, but I fear the anonymization steps have removed more than is warranted (and it wasn’t consistent–some of the sessions still have their sequence names but not all, and most are missing the CSA tag). I think I will have to convert/name/bidsify based on subject specific information from a master control file extracted from all the various csv files that come with NDA data. I’ll come back to making heudiconv work when I have datasets that have the same dicom tags available consistently within a study… :wink: