fMRIprep duplication issue - Discrepancy in processing runs

Software Support
1

We have noticed an issue running fMRIprep where it seems to overwrite/duplicate files sometimes. Our input data consists of fMRI runs from two different sessions (baselineYear1 and 2YearFollowUp) and the count of runs output from fMRI prep for each of these sessions do not match the input counts.

Here are the parameters we are using in our pipeline:

Fmriprep version: 21.0.2

Dcm2bids version: 2.1.7 (Singularity)

Session json file:

baselineYear1Arm1.json:

{
    "t1w": {
        "datatype": "anat",
        "session": "baselineYear1Arm1",
	"acquisition": null,
        "suffix": "T1w"
    },
    "bold": {
        "datatype": "func",
        "session": "baselineYear1Arm1",
        "suffix": "bold"
    }
}

2YearFollowUpYArm1.json:

{
    "t1w": {
        "datatype": "anat",
        "session": "2YearFollowUpYArm1",
	"acquisition": null, 
	"suffix": "T1w"
    },
    "bold": {
        "datatype": "func",
        "session": "2YearFollowUpYArm1",
        "suffix": "bold"
    }
}

Abcd_bids.conf file:

{
    "descriptions": [
        {
            "dataType": "anat",
            "modalityLabel": "T1w",
            "customLabels": "rec-normalized",
            "criteria": {
                "SeriesDescription": "*ABCD-T1-NORM*"
            }
        },
        {
            "dataType": "anat",
            "modalityLabel": "T1w",
            "criteria": {
                "SeriesDescription": "ABCD-T1,*"
            }
        },
        {
            "dataType": "func",
            "modalityLabel": "bold",
            "customLabels": "task-MID",
            "criteria": {
                "SeriesDescription": "ABCD-MID-fMRI*"
            }
        },
        {
            "dataType": "func",
            "modalityLabel": "bold",
            "customLabels": "task-rest",
            "criteria": {
                "SeriesDescription": "ABCD-rsfMRI*"
            }
        },
        {
            "dataType": "func",
            "modalityLabel": "bold",
            "customLabels": "task-SST",
            "criteria": {
                "SeriesDescription": "ABCD-SST-fMRI*"
            }
        },
        {
            "dataType": "func",
            "modalityLabel": "bold",
            "customLabels": "task-nback",
            "criteria": {
                "SeriesDescription": "ABCD-nBack-fMRI*"
            }
        }
    ],

	"dcm2niixOptions": "-b y -ba y -z y -v 2 -d 9 -f '%3s_%f_%p_%t'"
}

The process goes as follows:

  • extract the individual tar files from different session/runs to the respective subject folder

  • convert the data to bids format with dcm2bids using abcd_bids.conf as configuration - via singularity image

  • run fmriprep for each session by using json files, with the following flags:

fMRIPrep command: /opt/conda/bin/fmriprep /scratch/bids_template /scratch/ABCD_NDARINVXMP1CA5E_2YearFollowUpYArm1 participant --participant-label sub-NDARINVXMP1CA5E --fs-license-file /scratch/license.txt --notrack --output-spaces MNI152NLin2009cAsym:res-2 --ignore fieldmaps --skull-strip-t1w auto --bids-filter-file /scratch/2YearFollowUpYArm1.json --low-mem --random-seed 42 --use-aroma -vv --omp-nthreads 8 --n_cpus 16 --mem-mb 49200 -w /scratch/temp_work

  • We don’t stop fmriprep if processing for one file (bold run) fails.

  • We use singularity image for fmriprep.

  • We run everything on HCP, mention the range of resources we provide.

  • Time: 4-8 hours allocated for time

  • CPU: 16

  • Mem: 492

Below are a few examples of the behavior we are seeing:

Example #1

These are the raw counts of resting state fMRI runs for subject NDARINVV96LUWY8, we have 2 runs in baselineYear1 and 4 runs in 2YearFollowUp:

./77/NDARINVV96LUWY8_2YearFollowUpYArm1_ABCD-rsfMRI_20191207120420.tgz

./77/NDARINVV96LUWY8_2YearFollowUpYArm1_ABCD-rsfMRI_20191207121015.tgz

./77/NDARINVV96LUWY8_2YearFollowUpYArm1_ABCD-rsfMRI_20191207123359.tgz

./77/NDARINVV96LUWY8_2YearFollowUpYArm1_ABCD-rsfMRI_20191207123946.tgz

./77/NDARINVV96LUWY8_2YearFollowUpYArm1_ABCD-SST-fMRI_20191207131322.tgz

./77/NDARINVV96LUWY8_2YearFollowUpYArm1_ABCD-SST-fMRI_20191207131947.tgz

./77/NDARINVV96LUWY8_2YearFollowUpYArm1_ABCD-T1_20191207120235.tgz

./77/NDARINVV96LUWY8_baselineYear1Arm1_ABCD-rsfMRI_20171209140727.tgz

./77/NDARINVV96LUWY8_baselineYear1Arm1_ABCD-rsfMRI_20171209141441.tgz

./77/NDARINVV96LUWY8_baselineYear1Arm1_ABCD-T1_20171209140542.tgz

After running fMRIprep the output is only available for the 2YearFollowUp and has 6 runs:

./sub-NDARINVV96LUWY8_ses-2YearFollowUpYArm1_task-rest_run-4_space-MNI152NLin2009cAsym_res-2_boldref.nii.gz

./sub-NDARINVV96LUWY8_ses-2YearFollowUpYArm1_task-rest_run-1_space-MNI152NLin2009cAsym_res-2_boldref.nii.gz

./sub-NDARINVV96LUWY8_ses-2YearFollowUpYArm1_task-rest_run-3_space-MNI152NLin2009cAsym_res-2_boldref.nii.gz

./sub-NDARINVV96LUWY8_ses-2YearFollowUpYArm1_task-rest_run-6_space-MNI152NLin2009cAsym_res-2_boldref.nii.gz

./sub-NDARINVV96LUWY8_ses-2YearFollowUpYArm1_task-rest_run-2_space-MNI152NLin2009cAsym_res-2_boldref.nii.gz

./sub-NDARINVV96LUWY8_ses-2YearFollowUpYArm1_task-rest_run-5_space-MNI152NLin2009cAsym_res-2_boldref.nii.gz

Example #2

These are the raw counts of resting state fMRI runs for subject NDARINVXMP1CA5E, we have 4 runs in baselineYear1 and 0 runs in 2YearFollowUp:

/83/NDARINVXMP1CA5E_baselineYear1Arm1_ABCD-nBack-fMRI_20170604153423.tgz

./83/NDARINVXMP1CA5E_baselineYear1Arm1_ABCD-nBack-fMRI_20170604154004.tgz

./83/NDARINVXMP1CA5E_baselineYear1Arm1_ABCD-rsfMRI_20170604141013.tgz

./83/NDARINVXMP1CA5E_baselineYear1Arm1_ABCD-rsfMRI_20170604141603.tgz

./83/NDARINVXMP1CA5E_baselineYear1Arm1_ABCD-rsfMRI_20170604143913.tgz

./83/NDARINVXMP1CA5E_baselineYear1Arm1_ABCD-rsfMRI_20170604144523.tgz

./83/NDARINVXMP1CA5E_baselineYear1Arm1_ABCD-SST-fMRI_20170604150504.tgz

./83/NDARINVXMP1CA5E_baselineYear1Arm1_ABCD-SST-fMRI_20170604151133.tgz

./83/NDARINVXMP1CA5E_baselineYear1Arm1_ABCD-T1_20170604140531.tgz

After running fMRIprep the output has 4 resting state runs for baselineYear1 and 4 resting state runs for 2YearFollowUp:

./sub-NDARINVXMP1CA5E_ses-baselineYear1Arm1_task-rest_run-4_space-MNI152NLin2009cAsym_res-2_boldref.nii.gz

./sub-NDARINVXMP1CA5E_ses-baselineYear1Arm1_task-rest_run-1_space-MNI152NLin2009cAsym_res-2_boldref.nii.gz

./sub-NDARINVXMP1CA5E_ses-baselineYear1Arm1_task-rest_run-3_space-MNI152NLin2009cAsym_res-2_boldref.nii.gz

./sub-NDARINVXMP1CA5E_ses-baselineYear1Arm1_task-rest_run-2_space-MNI152NLin2009cAsym_res-2_boldref.nii.gz

./sub-NDARINVXMP1CA5E_ses-2YearFollowUpYArm1_task-rest_run-3_space-MNI152NLin2009cAsym_res-2_boldref.nii.gz

./sub-NDARINVXMP1CA5E_ses-2YearFollowUpYArm1_task-rest_run-1_space-MNI152NLin2009cAsym_res-2_boldref.nii.gz

./sub-NDARINVXMP1CA5E_ses-2YearFollowUpYArm1_task-rest_run-4_space-MNI152NLin2009cAsym_res-2_boldref.nii.gz

./sub-NDARINVXMP1CA5E_ses-2YearFollowUpYArm1_task-rest_run-2_space-MNI152NLin2009cAsym_res-2_boldref.nii.gz

Example #3

NDAR_INVWGVKC3KK

These are the raw counts of resting state fMRI runs for subject NDAR_INVWGVKC3KK, we have 5 runs in baselineYear1 and 4 runs in 2YearFollowUp:

./80/NDARINVWGVKC3KK_2YearFollowUpYArm1_ABCD-MID-fMRI_20191220121509.tgz

./80/NDARINVWGVKC3KK_2YearFollowUpYArm1_ABCD-nBack-fMRI_20191220115739.tgz

./80/NDARINVWGVKC3KK_2YearFollowUpYArm1_ABCD-nBack-fMRI_20191220120258.tgz

./80/NDARINVWGVKC3KK_2YearFollowUpYArm1_ABCD-rsfMRI_20191220105018.tgz

./80/NDARINVWGVKC3KK_2YearFollowUpYArm1_ABCD-rsfMRI_20191220105609.tgz

./80/NDARINVWGVKC3KK_2YearFollowUpYArm1_ABCD-rsfMRI_20191220112529.tgz

./80/NDARINVWGVKC3KK_2YearFollowUpYArm1_ABCD-rsfMRI_20191220113058.tgz

./80/NDARINVWGVKC3KK_2YearFollowUpYArm1_ABCD-SST-fMRI_20191220113838.tgz

./80/NDARINVWGVKC3KK_2YearFollowUpYArm1_ABCD-SST-fMRI_20191220114509.tgz

./80/NDARINVWGVKC3KK_2YearFollowUpYArm1_ABCD-T1_20191220104839.tgz

./80/NDARINVWGVKC3KK_2YearFollowUpYArm1_ABCD-T1_20191220110804.tgz

./80/NDARINVWGVKC3KK_baselineYear1Arm1_ABCD-MID-fMRI_20180119113953.tgz

./80/NDARINVWGVKC3KK_baselineYear1Arm1_ABCD-MID-fMRI_20180119114547.tgz

./80/NDARINVWGVKC3KK_baselineYear1Arm1_ABCD-nBack-fMRI_20180119112817.tgz

./80/NDARINVWGVKC3KK_baselineYear1Arm1_ABCD-nBack-fMRI_20180119113333.tgz

./80/NDARINVWGVKC3KK_baselineYear1Arm1_ABCD-rsfMRI_20180119102357.tgz

./80/NDARINVWGVKC3KK_baselineYear1Arm1_ABCD-rsfMRI_20180119102933.tgz

./80/NDARINVWGVKC3KK_baselineYear1Arm1_ABCD-rsfMRI_20180119105217.tgz

./80/NDARINVWGVKC3KK_baselineYear1Arm1_ABCD-rsfMRI_20180119105753.tgz

./80/NDARINVWGVKC3KK_baselineYear1Arm1_ABCD-rsfMRI_20180119110547.tgz

./80/NDARINVWGVKC3KK_baselineYear1Arm1_ABCD-SST-fMRI_20180119111407.tgz

./80/NDARINVWGVKC3KK_baselineYear1Arm1_ABCD-SST-fMRI_20180119112037.tgz

After running fMRIprep the output has 9 resting state runs for baselineYear1 and 9 resting state runs for 2YearFollowUp:

./sub-NDARINVWGVKC3KK_ses-baselineYear1Arm1_task-rest_run-7_space-MNI152NLin2009cAsym_res-2_boldref.nii.gz

./sub-NDARINVWGVKC3KK_ses-baselineYear1Arm1_task-rest_run-3_space-MNI152NLin2009cAsym_res-2_boldref.nii.gz

./sub-NDARINVWGVKC3KK_ses-baselineYear1Arm1_task-rest_run-9_space-MNI152NLin2009cAsym_res-2_boldref.nii.gz

./sub-NDARINVWGVKC3KK_ses-baselineYear1Arm1_task-rest_run-4_space-MNI152NLin2009cAsym_res-2_boldref.nii.gz

./sub-NDARINVWGVKC3KK_ses-baselineYear1Arm1_task-rest_run-6_space-MNI152NLin2009cAsym_res-2_boldref.nii.gz

./sub-NDARINVWGVKC3KK_ses-baselineYear1Arm1_task-rest_run-1_space-MNI152NLin2009cAsym_res-2_boldref.nii.gz

./sub-NDARINVWGVKC3KK_ses-baselineYear1Arm1_task-rest_run-5_space-MNI152NLin2009cAsym_res-2_boldref.nii.gz

./sub-NDARINVWGVKC3KK_ses-baselineYear1Arm1_task-rest_run-8_space-MNI152NLin2009cAsym_res-2_boldref.nii.gz

./sub-NDARINVWGVKC3KK_ses-baselineYear1Arm1_task-rest_run-2_space-MNI152NLin2009cAsym_res-2_boldref.nii.gz

./sub-NDARINVWGVKC3KK_ses-2YearFollowUpYArm1_task-rest_run-7_space-MNI152NLin2009cAsym_res-2_boldref.nii.gz

./sub-NDARINVWGVKC3KK_ses-2YearFollowUpYArm1_task-rest_run-3_space-MNI152NLin2009cAsym_res-2_boldref.nii.gz

./sub-NDARINVWGVKC3KK_ses-2YearFollowUpYArm1_task-rest_run-4_space-MNI152NLin2009cAsym_res-2_boldref.nii.gz

./sub-NDARINVWGVKC3KK_ses-2YearFollowUpYArm1_task-rest_run-9_space-MNI152NLin2009cAsym_res-2_boldref.nii.gz

./sub-NDARINVWGVKC3KK_ses-2YearFollowUpYArm1_task-rest_run-6_space-MNI152NLin2009cAsym_res-2_boldref.nii.gz

./sub-NDARINVWGVKC3KK_ses-2YearFollowUpYArm1_task-rest_run-1_space-MNI152NLin2009cAsym_res-2_boldref.nii.gz

./sub-NDARINVWGVKC3KK_ses-2YearFollowUpYArm1_task-rest_run-8_space-MNI152NLin2009cAsym_res-2_boldref.nii.gz

./sub-NDARINVWGVKC3KK_ses-2YearFollowUpYArm1_task-rest_run-5_space-MNI152NLin2009cAsym_res-2_boldref.nii.gz

./sub-NDARINVWGVKC3KK_ses-2YearFollowUpYArm1_task-rest_run-2_space-MNI152NLin2009cAsym_res-2_boldref.nii.gz

2

Summary screenshot for each fmriprep output example:

sub-NDARINVV96LUWY8_2year:

  • Subject ID: NDARINVV96LUWY8
  • Structural images: 1 T1-weighted
  • Functional series: 12
    • Task: MID (2 runs)
    • Task: SST (2 runs)
    • Task: nback (2 runs)
    • Task: rest (6 runs)
  • Standard output spaces: MNI152NLin2009cAsym, MNI152NLin6Asym
  • Non-standard output spaces:
  • FreeSurfer reconstruction: Run by fMRIPrep

sub-NDARINVWGVKC3KK_baseline:

  • Subject ID: NDARINVWGVKC3KK
  • Structural images: 1 T1-weighted
  • Functional series: 21
    • Task: MID (4 runs)
    • Task: SST (4 runs)
    • Task: nback (4 runs)
    • Task: rest (9 runs)
  • Standard output spaces: MNI152NLin2009cAsym, MNI152NLin6Asym
  • Non-standard output spaces:
  • FreeSurfer reconstruction: Run by fMRIPrep

sub-NDARINVWGVKC3KK_year2:

  • Subject ID: NDARINVWGVKC3KK
  • Structural images: 1 T1-weighted
  • Functional series: 21
    • Task: MID (4 runs)
    • Task: SST (4 runs)
    • Task: nback (4 runs)
    • Task: rest (9 runs)
  • Standard output spaces: MNI152NLin2009cAsym, MNI152NLin6Asym
  • Non-standard output spaces:
  • FreeSurfer reconstruction: Run by fMRIPrep

sub-NDARINVXMP1CA5E_baseline:

  • Subject ID: NDARINVXMP1CA5E
  • Structural images: 1 T1-weighted
  • Functional series: 16
    • Task: MID (4 runs)
    • Task: SST (4 runs)
    • Task: nback (4 runs)
    • Task: rest (4 runs)
  • Standard output spaces: MNI152NLin2009cAsym, MNI152NLin6Asym
  • Non-standard output spaces:
  • FreeSurfer reconstruction: Run by fMRIPrep

sub-NDARINVXMP1CA5E_2year:

  • Subject ID: NDARINVXMP1CA5E
  • Structural images: 1 T1-weighted
  • Functional series: 16
    • Task: MID (4 runs)
    • Task: SST (4 runs)
    • Task: nback (4 runs)
    • Task: rest (4 runs)
  • Standard output spaces: MNI152NLin2009cAsym, MNI152NLin6Asym
  • Non-standard output spaces:
  • FreeSurfer reconstruction: Run by fMRIPrep