Overwriting heudiconv sub.auto.txt and sub.edit.txt

Hi All,

I’ve got a workable solution for this, but would be very grateful if anyone can suggest something cleaner than my hack! Wondering if maybe I’ve missed a flag for heudiconv…?

My heudiconv conversion of a bunch of participants goes like this:

Step 1: Generate dicominfo.tsv files for all participants using the conversion file flag (-c) as “none”.

Step 2: Select one ‘reference’ participant dicominfo.tsv to create a heuristic (“BIDSConvert_03_heur_dicominfoRef.py” below).

Step 3: Compare all other dicominfo.tsv files against this reference to see if they can be appropriately converted with the same heuristic (to make sure I don’t convert the ‘bad’ T1w image if two were acquired, for example). Convert if they match or skip if they don’t.

However, I’m running into a problem with the “.auto.txt” and “.edit.txt” files generated in step 1.

After running with no conversion, the .auto.txt and .edit.txt files start with “(‘run{item:03d}’, (‘nii.gz’,), None):”. The ‘converted’ files from step 3 are then outputted as e.g. “run003.nii.gz”. Heudiconv tells me it is “Reloading existing filegroup.json because /base/sourcedata/BIDS/.heudiconv/subID/info/subID.edit.txt exists” and the reference heuristic isn’t applied (=> “WARNING: Failed to detect fullfilled BIDS layout. No scans.tsv file(s) will be produced for /base/sourcedata/BIDS/run003.json”).

It all works fine if I delete the .auto.txt and .edit.txt files after step 1. This is easy enough to do in the same loop as they are created, but it feels a bit dirty. I tried adding the “–overwrite” flag when I call heudiconv, but that doesn’t seem to make a difference.

This is effectively the same issue as https://github.com/nipy/heudiconv/issues/179, which was solved by not “running the “dry pass” step in the tutorial http://nipy.org/heudiconv/#6”, i.e. not creating the .auto and .edit.txt files.

Is there a way of running a minimal heudiconv or else to create only the dicominfo.tsv files and not the .auto/.edit.txt files? Or can I specify that existing .auto/.edit.txt files shouldn’t be reloaded?

My calls to create the dicominfo (step 1) and convert the files (step 3) are below. My heuristic (step 3) is also below for reference.

For ref, the singularity image was created with “singularity pull docker://nipy/heudiconv:0.5.4”

Step 1:

singularity run -B \
/vols/Scratch/brc_em/7DP:/base \
/vols/Scratch/brc_em/7DP/code/BIDSConversion/heudiconv_0.5.4.sif\
-d /base/sourcedata/dicom/{subject}/* \
-o /base/sourcedata/BIDS/ \
-f convertall \
-s $subject  \
-c none \
--overwrite

Step 3:

singularity run -B \
/vols/Scratch/brc_em/7DP:/base \
/vols/Scratch/brc_em/7DP/code/BIDSConversion/heudiconv_0.5.4.sif \
-d /base/sourcedata/dicom/{subject}/* \
-o /base/sourcedata/BIDS/ \
-f /base/code/BIDSConversion/BIDSConvert_03_heur_dicominfoRef.py \
-s $subject  \
-c dcm2niix -b \
--overwrite

Step 2:

 # BIDSConvert_03_heur_dicominfoRef.py

# Creates a "heuristic" for heudicomv to follow in converting the dicomsimport os

    def create_key(template, outtype=('nii.gz',), annotation_classes=None):
        if template is None or not template:
            raise ValueError('Template must be a valid format string')
        return template, outtype, annotation_classes


    def infotodict(seqinfo):
        """Heuristic evaluator for determining which runs belong where
        allowed template fields - follow python string module:
        item: index within category
        subject: participant id
        seqitem: run number during scanning
        subindex: sub index within group
        """

        # h2mri-update: create variables for each file type to be converted
        # then create the appropriate 'key' for conversion based on the BIDS format

        t1w                = create_key('sub-{subject}/anat/sub-{subject}_T1w')
        faces_bold         = create_key('sub-{subject}/func/sub-{subject}_task-faces_bold')
        faces_sbref        = create_key('sub-{subject}/func/sub-{subject}_task-faces_sbref')
        hipp_bold          = create_key('sub-{subject}/func/sub-{subject}_task-hipp_bold')
        hipp_sbref         = create_key('sub-{subject}/func/sub-{subject}_task-hipp_sbref')
        rest_bold          = create_key('sub-{subject}/func/sub-{subject}_task-rest_bold')
        rest_sbref         = create_key('sub-{subject}/func/sub-{subject}_task-rest_sbref')
        fmap_taskMag       = create_key('sub-{subject}/fmap/sub-{subject}_acq-task_magnitude')
        fmap_taskPhase     = create_key('sub-{subject}/fmap/sub-{subject}_acq-task_phasediff')
        fmap_restMag       = create_key('sub-{subject}/fmap/sub-{subject}_acq-rest_magnitude')
        fmap_restPhase     = create_key('sub-{subject}/fmap/sub-{subject}_acq-rest_phasediff')

        # h2mri-update: edit the below to match the file type variables create above
        info = {
                        t1w: [],
                        faces_bold: [],
                        faces_sbref: [],
                        hipp_bold: [],
                        hipp_sbref: [],
                        rest_bold: [],
                        rest_sbref: [],
                        fmap_taskMag: [],
                        fmap_taskPhase: [],
                        fmap_restMag: [],
                        fmap_restPhase: [],
                    }

        for s in seqinfo:
            """
            The namedtuple `s` contains the following fields:
            * total_files_till_now
            * example_dcm_file
            * series_id
            * dcm_dir_name
            * unspecified2
            * unspecified3
            * dim1
            * dim2
            * dim3
            * dim4
            * TR
            * TE
            * protocol_name
            * is_motion_corrected
            * is_derived
            * patient_id
            * study_description
            * referring_physician_name
            * series_description
            * image_type
            """

            # h2mri-update: use the file BIDSConvert_02_dicominfRef.tsv to update the below for your sequences
            # amend these to match the dicominfo.tsv column "protocol_name" and "series_id" if you need to
            # idenitfy a specific dicom series (e.g. to seperate the sbref image from the main sequence)

            # idenitfy the un-biasfield corrected T1 image. fmriprep does bias field correction
            if s.protocol_name == 't1_mpr_ax_1mm_iso_32ch_v2'and s.series_id == '6-t1_mpr_ax_1mm_iso_32ch_v2':
                info[t1w] = [s.series_id]
            if s.protocol_name == 'bold_mbep2d_MB4P2_task_faces' and s.series_id == '9-bold_mbep2d_MB4P2_task_faces':
                info[faces_bold] = [s.series_id]
            if s.protocol_name == 'bold_mbep2d_MB4P2_task_faces_SBRef' and s.series_id == '8-bold_mbep2d_MB4P2_task_faces_SBRef':
                info[faces_sbref] = [s.series_id]
            if s.protocol_name == 'BOLD_2p4_MB6_P1_TE30_TR800_AP_hippocampal' and s.series_id == '13-BOLD_2p4_MB6_P1_TE30_TR800_AP_hippocampal':
                info[hipp_bold] = [s.series_id]
            if s.protocol_name == 'BOLD_2p4_MB6_P1_TE30_TR800_AP_hippocampal_SBRef' and s.series_id == '12-BOLD_2p4_MB6_P1_TE30_TR800_AP_hippocampal_SBRef':
                info[hipp_sbref] = [s.series_id]
            if s.protocol_name == 'RS_BOLD_2p5_MB3_P2_TE15-36-57_TR1600_PA' and s.series_id == '19-RS_BOLD_2p5_MB3_P2_TE15-36-57_TR1600_PA':
                info[rest_bold] = [s.series_id]
            if s.protocol_name == 'RS_BOLD_2p5_MB3_P2_TE15-36-57_TR1600_PA_SBRef' and s.series_id == '18-RS_BOLD_2p5_MB3_P2_TE15-36-57_TR1600_PA_SBRef':
                info[rest_sbref] = [s.series_id]
            if s.protocol_name == 'fieldmap_2pmm_FoV_216x216x114' and s.series_id == '10-fieldmap_2pmm_FoV_216x216x114':
                info[fmap_taskMag] = [s.series_id]
            if s.protocol_name == 'fieldmap_2pmm_FoV_216x216x114' and s.series_id == '11-fieldmap_2pmm_FoV_216x216x114':
                info[fmap_taskPhase] = [s.series_id]
            if s.protocol_name == 'fieldmap_2pmm_FoV_210x210x150' and s.series_id == '20-fieldmap_2pmm_FoV_210x210x150':
                info[fmap_restMag] = [s.series_id]
            if s.protocol_name == 'fieldmap_2pmm_FoV_210x210x150' and s.series_id == '21-fieldmap_2pmm_FoV_210x210x150':
                info[fmap_restPhase] = [s.series_id]

        return info

With curiosity and gratitude!

Cass