Dcm2bids - No pairing issue

1

I am using a new cluster that runs Centos7. Since using the new cluster, I am getting pairing issues when using dcm2bids. I have tried dcm2bids vs. 2.1.4, 2.1.5, and 2.1.6.

module load anaconda/2020.07
python --version
Python 3.8.3
source activate dcm2bids
python --version
Python 3.9.2
which dcm2bids
~/.conda/envs/dcm2bids/bin/dcm2bids
which dcm2niix
~/.conda/envs/dcm2bids/bin/dcm2niix

I run the dcm2bids command in a bash script

dcm2bids -d {dicom_subdir_nospaces} -p {SubjectID} -s 1 -c /dfs4/som/tvanerp_col/r21cbv/scripts/bids/config_HippoCBV.json -o $output_dir --forceDcm2niix --clobber

It randomly pairs or does not pair with the .json configuration side car.

Does anyone have an idea why this may be happening. Not sure if this is relevant but this never happened on our old cluster which was running Centos6.

::::::::::::::

sub-201949700004_ses-1_2021-03-01T094042.574451.log

::::::::::::::

INFO:dcm2bids.dcm2bids:— dcm2bids start —

INFO:dcm2bids.dcm2bids:OS:version: Linux-3.10.0-1127.19.1.el7.x86_64-x86_64-with-glibc2.17

INFO:dcm2bids.dcm2bids:python:version: 3.9.2 | packaged by conda-forge | (default, Feb 21 2021, 05:02:46) [GCC 9.3.0]

INFO:dcm2bids.dcm2bids:dcm2bids:version: 2.1.6

INFO:dcm2bids.dcm2bids:dcm2niix:version: v1.0.20201224

INFO:dcm2bids.dcm2bids:participant: sub-201949700004

INFO:dcm2bids.dcm2bids:session: ses-1

INFO:dcm2bids.dcm2bids:config: /dfs4/som/tvanerp_col/r21cbv/scripts/bids/config_HippoCBV.json

INFO:dcm2bids.dcm2bids:BIDS directory: /dfs4/som/tvanerp_col/r21cbv/dicom/201949700004_DOS_02-28-20/bids

INFO:dcm2bids.utils:Running dcm2niix -b y -ba y -z y -f ‘%3s_%f_%p_%t’ -o …/…/dicom/201949700004_DOS_02-28-20/bids/tmp_dcm2bids/sub-201949700004_ses-1 …/…/dicom/201949700004_DOS_02-28-20/Mri_Brain_Wo-W_Contrast_20200228

INFO:dcm2bids.dcm2niix:Check log file for dcm2niix output

INFO:dcm2bids.sidecar:Sidecars pairing:

INFO:dcm2bids.sidecar:No Pairing ← 001_Mri_Brain_Wo-W_Contrast_20200228_LOCALIZER_20200228194619_i00001

INFO:dcm2bids.sidecar:No Pairing ← 001_Mri_Brain_Wo-W_Contrast_20200228_LOCALIZER_20200228194619_i00002

INFO:dcm2bids.sidecar:No Pairing ← 001_Mri_Brain_Wo-W_Contrast_20200228_LOCALIZER_20200228194619_i00003

INFO:dcm2bids.sidecar:No Pairing ← 002_Mri_Brain_Wo-W_Contrast_20200228_2D_Midline_Sagittal_20200228194619

INFO:dcm2bids.sidecar:No Pairing ← 003_Mri_Brain_Wo-W_Contrast_20200228_SAG_FBIRN_T1w_TR2300_.9iso_20200228194619

INFO:dcm2bids.sidecar:No Pairing ← 004_Mri_Brain_Wo-W_Contrast_20200228_OBL_COR_t2_tse_NEW_ZOOM_3mm_thck_20200228194619

INFO:dcm2bids.sidecar:No Pairing ← 005_Mri_Brain_Wo-W_Contrast_20200228_AXL_FBIRN_T2_20200228194619

INFO:dcm2bids.sidecar:No Pairing ← 006_Mri_Brain_Wo-W_Contrast_20200228_AXL_fbirn_field_mapping_20200228194619_e1

INFO:dcm2bids.sidecar:No Pairing ← 006_Mri_Brain_Wo-W_Contrast_20200228_AXL_fbirn_field_mapping_20200228194619_e2

INFO:dcm2bids.sidecar:No Pairing ← 007_Mri_Brain_Wo-W_Contrast_20200228_AXL_fbirn_field_mapping_20200228194619_e2_ph

INFO:dcm2bids.sidecar:No Pairing ← 008_Mri_Brain_Wo-W_Contrast_20200228_AXL_REST_lowres_3.4x3.4x3.0_AXL_(GRAPPA=0)_20200228194619

INFO:dcm2bids.sidecar:No Pairing ← 009_Mri_Brain_Wo-W_Contrast_20200228_OBL_COR_REST_hires_1.7x1.7x2.4_(180-frames;_GRAPPA=2)_20200228194619

INFO:dcm2bids.sidecar:No Pairing ← 010_Mri_Brain_Wo-W_Contrast_20200228_OBL_COR_FBIRN_T1w_TR2000_Pre_20200228194619

INFO:dcm2bids.sidecar:No Pairing ← 011_Mri_Brain_Wo-W_Contrast_20200228_OBL_COR_REST_hires_1.7x1.7x2.4_(110-frames;_GRAPPA=2)_20200228194619

INFO:dcm2bids.sidecar:No Pairing ← 012_Mri_Brain_Wo-W_Contrast_20200228_OBL_COR_FBIRN_T1w_TR2000_Post_20200228194619

INFO:dcm2bids.dcm2bids:moving acquisitions into BIDS folder

::::::::::::::

sub-201949700004_ses-1_2021-03-01T095139.121568.log

::::::::::::::

INFO:dcm2bids.dcm2bids:— dcm2bids start —

INFO:dcm2bids.dcm2bids:OS:version: Linux-3.10.0-1127.19.1.el7.x86_64-x86_64-with-glibc2.17

INFO:dcm2bids.dcm2bids:python:version: 3.9.2 | packaged by conda-forge | (default, Feb 21 2021, 05:02:46) [GCC 9.3.0]

INFO:dcm2bids.dcm2bids:dcm2bids:version: 2.1.6

INFO:dcm2bids.dcm2bids:dcm2niix:version: v1.0.20201224

INFO:dcm2bids.dcm2bids:participant: sub-201949700004

INFO:dcm2bids.dcm2bids:session: ses-1

INFO:dcm2bids.dcm2bids:config: /dfs4/som/tvanerp_col/r21cbv/scripts/bids/config_HippoCBV.json

INFO:dcm2bids.dcm2bids:BIDS directory: /dfs4/som/tvanerp_col/r21cbv/dicom/201949700004_DOS_02-28-20/bids

WARNING:dcm2bids.dcm2niix:Previous dcm2niix directory output found:

WARNING:dcm2bids.dcm2niix:…/…/dicom/201949700004_DOS_02-28-20/bids/tmp_dcm2bids/sub-201949700004_ses-1

WARNING:dcm2bids.dcm2niix:‘force’ argument is set to True

WARNING:dcm2bids.dcm2niix:Cleaning the previous directory and running dcm2niix

INFO:dcm2bids.utils:Running dcm2niix -b y -ba y -z y -f ‘%3s_%f_%p_%t’ -o …/…/dicom/201949700004_DOS_02-28-20/bids/tmp_dcm2bids/sub-201949700004_ses-1 …/…/dicom/201949700004_DOS_02-28-20/Mri_Brain_Wo-W_Contrast_20200228

INFO:dcm2bids.dcm2niix:Check log file for dcm2niix output

INFO:dcm2bids.sidecar:Sidecars pairing:

INFO:dcm2bids.sidecar:No Pairing ← 001_Mri_Brain_Wo-W_Contrast_20200228_LOCALIZER_20200228194619_i00001

INFO:dcm2bids.sidecar:No Pairing ← 001_Mri_Brain_Wo-W_Contrast_20200228_LOCALIZER_20200228194619_i00002

INFO:dcm2bids.sidecar:No Pairing ← 001_Mri_Brain_Wo-W_Contrast_20200228_LOCALIZER_20200228194619_i00003

INFO:dcm2bids.sidecar:No Pairing ← 002_Mri_Brain_Wo-W_Contrast_20200228_2D_Midline_Sagittal_20200228194619

INFO:dcm2bids.sidecar:T1w ← 003_Mri_Brain_Wo-W_Contrast_20200228_SAG_FBIRN_T1w_TR2300.9iso_20200228194619

INFO:dcm2bids.sidecar:_acq-TSEcor_T2w ← 004_Mri_Brain_Wo-W_Contrast_20200228_OBL_COR_t2_tse_NEW_ZOOM_3mm_thck_20200228194619

INFO:dcm2bids.sidecar:_T2w ← 005_Mri_Brain_Wo-W_Contrast_20200228_AXL_FBIRN_T2_20200228194619

INFO:dcm2bids.sidecar:_magnitude1 ← 006_Mri_Brain_Wo-W_Contrast_20200228_AXL_fbirn_field_mapping_20200228194619_e1

INFO:dcm2bids.sidecar:_magnitude1 ← 006_Mri_Brain_Wo-W_Contrast_20200228_AXL_fbirn_field_mapping_20200228194619_e2

INFO:dcm2bids.sidecar:_magnitude1 ← 007_Mri_Brain_Wo-W_Contrast_20200228_AXL_fbirn_field_mapping_20200228194619_e2_ph

INFO:dcm2bids.sidecar:task-restWB_bold ← 008_Mri_Brain_Wo-W_Contrast_20200228_AXL_REST_lowres_3.4x3.4x3.0_AXL(GRAPPA=0)_20200228194619

INFO:dcm2bids.sidecar:task-restHIP_bold ← 009_Mri_Brain_Wo-W_Contrast_20200228_OBL_COR_REST_hires_1.7x1.7x2.4(180-frames;_GRAPPA=2)_20200228194619

INFO:dcm2bids.sidecar:_acq-T1wPre_T1w ← 010_Mri_Brain_Wo-W_Contrast_20200228_OBL_COR_FBIRN_T1w_TR2000_Pre_20200228194619

INFO:dcm2bids.sidecar:task-restHIP_bold ← 011_Mri_Brain_Wo-W_Contrast_20200228_OBL_COR_REST_hires_1.7x1.7x2.4(110-frames;_GRAPPA=2)_20200228194619

INFO:dcm2bids.sidecar:_acq-T1wPost_T1w ← 012_Mri_Brain_Wo-W_Contrast_20200228_OBL_COR_FBIRN_T1w_TR2000_Post_20200228194619

INFO:dcm2bids.sidecar:sub-201949700004/ses-1/fmap/sub-201949700004_ses-1_magnitude1 has 3 runs

INFO:dcm2bids.sidecar:Adding ‘run’ information to the acquisition

INFO:dcm2bids.sidecar:sub-201949700004/ses-1/func/sub-201949700004_ses-1_task-restHIP_bold has 2 runs

INFO:dcm2bids.sidecar:Adding ‘run’ information to the acquisition

INFO:dcm2bids.dcm2bids:moving acquisitions into BIDS folder

2

:wave: @Theo_van_Erp, welcome to Neurostars :slight_smile: !

I have never noticed or encountered this behavior before so I have a couple of questions that I hope will help us resolve your problem:

  1. I’m assuming you have never noticed this random behavior before (on the older cluster) with the same data and same configuration file, is this correct?
  2. Can you provide your whole configuration file (either attached to your reply here or in a repository somewhere).
  3. Probably not related, but did you install dcm2bids with pip vs conda? Both should work just fine since it is the same source.
  4. Do you run a single subject or multiple at once?
  5. Do you submit jobs on your cluster through a SLURM workload manager or similar, or you just run scripts on it?
  6. When you say “It randomly pairs or does not pair with the .json configuration side car.”, do you mean for different subjects or for the same subject?

That’s what I can think off the top of my head right now. Let’s start from there and see where this will take us.

Thanks,

Sam

3

Thanks Sam, really appreciate the help.

Responses inline:

  1. I’m assuming you have never noticed this random behavior before (on the older cluster) with the same data and same configuration file, is this correct?

Yes, never noticed this behavior before on the old cluster with the same .json sidecar configuration file.

  1. Can you provide your whole configuration file (either attached to your reply here or in a repository somewhere).

cat config_HippoCBV.json
{
“descriptions”: [
{
“dataType”: “anat”,
“modalityLabel”: “T1w”,
“criteria”: {
“SeriesDescription”: “SAG_FBIRN_T1w”
}
},
{
“dataType”: “anat”,
“modalityLabel”: “T2w”,
“customLabels”: “acq-TSEcor”,
“criteria”: {
“SeriesDescription”: “tse_NEW_ZOOM”
}
},
{
“dataType”: “anat”,
“modalityLabel”: “T1w”,
“customLabels”: “acq-T1wPre”,
“criteria”: {
“SeriesDescription”: “FBIRN_T1w_TR2000_Pre”
}
},
{
“dataType”: “anat”,
“modalityLabel”: “T1w”,
“customLabels”: “acq-T1wPost”,
“criteria”: {
“SeriesDescription”: “FBIRN_T1w_TR2000_Post”
}
},
{
“dataType”: “anat”,
“modalityLabel”: “T2w”,
“criteria”: {
“SeriesDescription”: “FBIRN_T2”
}
},
{
“dataType”: “fmap”,
“modalityLabel”: “magnitude1”,
“criteria”: {
“SeriesDescription”: “fbirn_field_mapping”
}
},
{
“dataType”: “func”,
“modalityLabel”: “bold”,
“customLabels”: “task-restWB”,
“criteria”: {
“SeriesDescription”: “REST_lowres”
}
},
{
“dataType”: “func”,
“modalityLabel”: “bold”,
“customLabels”: “task-restHIP”,
“criteria”: {
“SeriesDescription”: “REST_hires”
}
}
]
}

  1. Probably not related, but did you install dcm2bids with pip vs conda? Both should work just fine since it is the same source.

I think this may be my problem. Previously I did a conda install and it looks like I mixed conda and pip this time based on install notes (duh).

I did a “conda env create -f environment.yml” with the following contents in the yml file:

name: dcm2bids
channels:

  • conda-forge
    dependencies:
  • dcm2niix
  • dcm2bids
  • pip

Then a “pip install dcm2bids” and probably should have done a “conda install dcm2bids”.

I am going to try a “pip uninstall dcm2bids” followed by a “conda install dcm2bids” to see if that will solve the issue.

  1. Do you run a single subject or multiple at once?

I have tried both, single subject and multiple.

  1. Do you submit jobs on your cluster through a SLURM workload manager or similar, or you just run scripts on it?

I have tried both and both have the same issue.

  1. When you say “It randomly pairs or does not pair with the .json configuration side car.”, do you mean for different subjects or for the same subject?

Each configuration has occurred.

One subject does not pair during 3 runs, then fully pairs on the 4th run.
One subject pairs for a few scans but not others.

4

Unfortunately, the “pip uninstall dcm2bids” followed by the “conda install dcm2bids” did not solve the issue.

5

I just want to report that my pairing issue is solved.

Apparently, I had previously installed dcm2bids with pip, not conda and the conda install was giving me all the No pairing issues.

The following worked, no more pairing issues

this installed dcm2bids in .local/bin

In my bash script, I just added

export PATH=${PATH}:~/bin:~/.local/bin

Now dcm2bids is running like before with consistent pairing for each data set.

Your comment related to pip vs. conda install put me on the right path. I am not sure why the conda install did no work for me but the No pairing issue is gone and I am back in business.

6

I guess I was too optimistic, if did 18 correct pairings in a row, then failed 2, the correctly paired the last one. I may revert to dcm2bids 2.1.4 next to see if his issue is only with 2.1.6:

INFO:dcm2bids.sidecar:No Pairing ← 001_Mri_Brain_Wo-W_Contrast_20210219_LOCALIZER_20210219122445_i00001
INFO:dcm2bids.sidecar:No Pairing ← 001_Mri_Brain_Wo-W_Contrast_20210219_LOCALIZER_20210219122445_i00002
INFO:dcm2bids.sidecar:No Pairing ← 001_Mri_Brain_Wo-W_Contrast_20210219_LOCALIZER_20210219122445_i00003
INFO:dcm2bids.sidecar:No Pairing ← 002_Mri_Brain_Wo-W_Contrast_20210219_2D_Midline_Sagittal_20210219122445
INFO:dcm2bids.sidecar:No Pairing ← 003_Mri_Brain_Wo-W_Contrast_20210219_SAG_FBIRN_T1w_TR2300_.9iso_20210219122445
INFO:dcm2bids.sidecar:No Pairing ← 004_Mri_Brain_Wo-W_Contrast_20210219_OBL_COR_t2_tse_NEW_ZOOM_3mm_thck_20210219122445
INFO:dcm2bids.sidecar:No Pairing ← 005_Mri_Brain_Wo-W_Contrast_20210219_AXL_FBIRN_T2_20210219122445
INFO:dcm2bids.sidecar:No Pairing ← 006_Mri_Brain_Wo-W_Contrast_20210219_AXL_fbirn_field_mapping_20210219122445_e1
INFO:dcm2bids.sidecar:No Pairing ← 006_Mri_Brain_Wo-W_Contrast_20210219_AXL_fbirn_field_mapping_20210219122445_e2
INFO:dcm2bids.sidecar:No Pairing ← 007_Mri_Brain_Wo-W_Contrast_20210219_AXL_fbirn_field_mapping_20210219122445_e2_ph
INFO:dcm2bids.sidecar:No Pairing ← 008_Mri_Brain_Wo-W_Contrast_20210219_AXL_REST_lowres_3.4x3.4x3.0_AXL_(GRAPPA=0)20210219122445
INFO:dcm2bids.sidecar:No Pairing ← 009_Mri_Brain_Wo-W_Contrast_20210219_OBL_COR_REST_hires_1.7x1.7x2.4(180-frames;_GRAPPA=2)20210219122445
INFO:dcm2bids.sidecar:No Pairing ← 010_Mri_Brain_Wo-W_Contrast_20210219_OBL_COR_FBIRN_T1w_TR2000_Pre_20210219122445
INFO:dcm2bids.sidecar:No Pairing ← 011_Mri_Brain_Wo-W_Contrast_20210219_OBL_COR_REST_hires_1.7x1.7x2.4(110-frames;_GRAPPA=2)_20210219122445
INFO:dcm2bids.sidecar:No Pairing ← 012_Mri_Brain_Wo-W_Contrast_20210219_OBL_COR_FBIRN_T1w_TR2000_Post_20210219122445

7

Glad to know I was able to put you on the right path :slight_smile:.

Maybe if both packages are installed at once within the same environment it creates problem. However, I recently added dcm2bids to conda-forge to simplify the installation since dcm2niix is there too. Since you got an issue, I’ll make sure everything works.

In fact, I just looked at the documentation and realized I forgot to erase the pip install dcm2bids :grimacing:.

To make sure it works fine, can you try to create a new virtual env by creating a new environment.yml with these lines:

name: dcm2bids-test
channels:
    - conda-forge
dependencies:
    - dcm2niix
    - dcm2bids

Then create the environment by typing:

conda env create -f environment.yml

and activate it by

source activate dcm2bids-test

Let me know how that goes.

8

I just saw this comment after having posted my reply. We are never too optimistic :+1: !

I’ll try to replicate the error on my end to see if 2.1.6 is problematic or it is something else.

9

I was able to replicate the same subjects not pairing using dcm2bids 2.1.4 and tracked down why.

Apparently, during data transfer, the sticky bits for these directories did not get set correctly resulting in nii.gz files that were 0 in size and No pairing.

Fixed the sticky bits for those two directories and all subjects are converted to bids with correct pairing.

1 Like
10

Yes, that must be it (-;

instead of:

conda env create -f environment.yml
source activate dcm2bids
pip install dcm2bids

Have a separate section for the conda install:

conda env create -f environment.yml
source activate dcm2bids

and pip install:

pip install dcm2bids

for people like me who do without thinking (-;

11

I’ll re-run with a proper conda install and fresh 2.1.6 pip install but I am 99.9% sure the issue was with the incorrect inheritance of the sticky bit during data transfer.

I’ll just add a sticky bit fix to my code to avoid that from happening again.

12

Good! Let us know if you still have issues, it helps us prioritize what to improve next.

Rest assured that I had already planned to make 2 separate installation sections (1 for conda; 1 for pip) in the documentation and you just confirmed to me it will be useful :grinning_face_with_smiling_eyes:.

By the way, if you have any feature or idea you’d like to see implemented in dcm2bids, feel free to submit them here.