Bidscoin 4.6.2 does not find all files

Software Support
1

Summary of what happened:

I am using the bidsmapper command from bidscoin 4.6.2 to find unique provenances of my sourcedata. My sourcedata is organized in a flat DICOM format:

/path/to/sourcedata
├── sub-INDI132M2/
│   ├── ses-01/
│   │   ├── PSY04757.MR._.0001.0001.2022.01.14.15.56.16.173653.15355014.IMA
│   │   ├── PSY04757.MR._.0001.0002.2022.01.14.15.56.16.173653.15355052.IMA
│   │   ├── PSY04757.MR._.0001.0003.2022.01.14.15.56.16.173653.15355037.IMA
│   │   ├── PSY04757.MR._.0002.0001.2022.01.14.15.56.16.173653.15369352.IMA
│   │   └── PSY04757.MR._.0002.0002.2022.01.14.15.56.16.173653.15369370.IMA
│   └── ses-02/
│       ├── PSY04757.MR._.0001.0001.2022.01.14.15.56.16.173653.15355014.IMA
│       ├── PSY04757.MR._.0001.0001.2022.08.05.14.21.56.313214.31865948.IMA
│       ├── PSY04757.MR._.0001.0002.2022.01.14.15.56.16.173653.15355052.IMA
│       ├── PSY04757.MR._.0001.0002.2022.08.05.14.21.56.313214.31865986.IMA
│       └── PSY04757.MR._.0001.0003.2022.01.14.15.56.16.173653.15355037.IMA
├── sub-INDI1W934/
│   ├── ses-01/
│   │   ├── PSY04707.MR._.0001.0001.2021.10.21.15.41.07.724833.39866486.IMA
│   │   ├── PSY04707.MR._.0001.0002.2021.10.21.15.41.07.724833.39866524.IMA
│   │   ├── PSY04707.MR._.0001.0003.2021.10.21.15.41.07.724833.39866509.IMA
│   │   ├── PSY04707.MR._.0002.0001.2021.10.21.15.41.07.724833.39869560.IMA
│   │   └── PSY04707.MR._.0002.0002.2021.10.21.15.41.07.724833.39869578.IMA
│   └── ses-02/
│       ├── PSY04707.MR._.0001.0001.2022.04.07.11.41.06.780808.19422288.IMA
│       ├── PSY04707.MR._.0001.0002.2022.04.07.11.41.06.780808.19422326.IMA
│       ├── PSY04707.MR._.0001.0003.2022.04.07.11.41.06.780808.19422311.IMA
....

Command used (and if a helper script was used, a link to the helper script or the command generated):

bidsmapper /path/to/sourcedata /path/to/rawdata/ -s

Version:

4.6.2

Relevant log outputs (up to 20 lines):

bidsmapper only finds files for one particular subject and session (leaving lots of other modalities, etc. out):

INFO     Mapping: /zi/flstorage/group_snip/INDICATE/MRI/sourcedata/ANONIMA/sub-INDI6P7ZL/ses-01 (subject 25/154)                                                                                                                                                                                                            
INFO     No dcm2niix2bids sourcedata found in: /zi/flstorage/group_snip/INDICATE/MRI/sourcedata/ANONIMA/sub-INDI6P7ZL/ses-01                                                                                                                                                                                                
INFO     Mapping: /zi/flstorage/group_snip/INDICATE/MRI/sourcedata/ANONIMA/sub-INDI6P7ZL/ses-02 (subject 25/154)                                                                                                                                                                                                            
INFO     No dcm2niix2bids sourcedata found in: /zi/flstorage/group_snip/INDICATE/MRI/sourcedata/ANONIMA/sub-INDI6P7ZL/ses-02                                                                                                                                                                                                
INFO     Mapping: /zi/flstorage/group_snip/INDICATE/MRI/sourcedata/ANONIMA/sub-INDI6TEYX/ses-01 (subject 26/154)                                                                                                                                                                                                            
INFO     No dcm2niix2bids sourcedata found in: /zi/flstorage/group_snip/INDICATE/MRI/sourcedata/ANONIMA/sub-INDI6TEYX/ses-01                                                                                                                                                                                                
INFO     Mapping: /zi/flstorage/group_snip/INDICATE/MRI/sourcedata/ANONIMA/sub-INDI71FGT/ses-01 (subject 27/154)                                                                                                                                                                                                            
INFO     No dcm2niix2bids sourcedata found in: /zi/flstorage/group_snip/INDICATE/MRI/sourcedata/ANONIMA/sub-INDI71FGT/ses-01                                                                                                                                                                                                
INFO     Mapping: /zi/flstorage/group_snip/INDICATE/MRI/sourcedata/ANONIMA/sub-INDI71FGT/ses-02 (subject 27/154)                                                                                                                                                                                                            
INFO     Detected a flat data-structure in: /zi/flstorage/group_snip/INDICATE/MRI/sourcedata/ANONIMA/sub-INDI71FGT/ses-02                                                                                                                                                                                                   
INFO     Making temporary copy: /zi/flstorage/group_snip/INDICATE/MRI/sourcedata/ANONIMA/sub-INDI71FGT/ses-02 -> /tmp/tmpv6ltpo84/zi/flstorage/group_snip/INDICATE/MRI/sourcedata/ANONIMA/sub-INDI71FGT/ses-02                                                                                                              
INFO     >> Sorting: /tmp/tmpv6ltpo84/zi/flstorage/group_snip/INDICATE/MRI/sourcedata/ANONIMA/sub-INDI71FGT/ses-02 (2006 files)                                                                                                                                                                                             
INFO     Discovered sample: [DICOM] /tmp/tmpv6ltpo84/zi/flstorage/group_snip/INDICATE/MRI/sourcedata/ANONIMA/sub-INDI71FGT/ses-02/002-new_t1_7min0.8iso_mprage_sag_p2_iso/PSY04725.MR._.0002.0224.2022.06.03.14.30.52.355462.62813228.IMA                                                                                   
INFO     Discovered sample: [DICOM] /tmp/tmpv6ltpo84/zi/flstorage/group_snip/INDICATE/MRI/sourcedata/ANONIMA/sub-INDI71FGT/ses-02/007-restingstate_PA_cmrr_mbep2d_bold_AF6/PSY04725.MR._.0007.0036.2022.06.03.14.30.52.355462.62822186.IMA                                                                                  
INFO     Discovered sample: [DICOM] /tmp/tmpv6ltpo84/zi/flstorage/group_snip/INDICATE/MRI/sourcedata/ANONIMA/sub-INDI71FGT/ses-02/009-nback_PA_cmrr_mbep2d_bold_AF6/PSY04725.MR._.0009.0001.2022.06.03.14.30.52.355462.62867586.IMA                                                                                         
INFO     Discovered sample: [DICOM] /tmp/tmpv6ltpo84/zi/flstorage/group_snip/INDICATE/MRI/sourcedata/ANONIMA/sub-INDI71FGT/ses-02/010-nback_PA_cmrr_mbep2d_bold_AF6_PhysioLog/PSY04725.MR._.0010.0001.2022.06.03.14.30.52.355462.62894641.IMA                                                                               
INFO     Discovered sample: [DICOM] /tmp/tmpv6ltpo84/zi/flstorage/group_snip/INDICATE/MRI/sourcedata/ANONIMA/sub-INDI71FGT/ses-02/011-faces_PA_cmrr_mbep2d_bold_AF6/PSY04725.MR._.0011.0001.2022.06.03.14.30.52.355462.62896260.IMA                                                                                         
INFO     Discovered sample: [DICOM] /tmp/tmpv6ltpo84/zi/flstorage/group_snip/INDICATE/MRI/sourcedata/ANONIMA/sub-INDI71FGT/ses-02/012-faces_PA_cmrr_mbep2d_bold_AF6_PhysioLog/PSY04725.MR._.0012.0001.2022.06.03.14.30.52.355462.62924485.IMA                                                                               
INFO     Discovered sample: [DICOM] /tmp/tmpv6ltpo84/zi/flstorage/group_snip/INDICATE/MRI/sourcedata/ANONIMA/sub-INDI71FGT/ses-02/013-mid_PA_cmrr_mbep2d_bold_AF6/PSY04725.MR._.0013.0001.2022.06.03.14.30.52.355462.62924934.IMA                                                                                           
INFO     Discovered sample: [DICOM] /tmp/tmpv6ltpo84/zi/flstorage/group_snip/INDICATE/MRI/sourcedata/ANONIMA/sub-INDI71FGT/ses-02/014-mid_PA_cmrr_mbep2d_bold_AF6_PhysioLog/PSY04725.MR._.0014.0001.2022.06.03.14.30.52.355462.62957543.IMA                                                                                 
INFO     Discovered sample: [DICOM] /tmp/tmpv6ltpo84/zi/flstorage/group_snip/INDICATE/MRI/sourcedata/ANONIMA/sub-INDI71FGT/ses-02/015-B0diff_PA_64ch_DCCN_1.6mm_120b_2500_1250_SBRef/PSY04725.MR._.0015.0001.2022.06.03.14.30.52.355462.62957704.IMA                                                                        
INFO     Discovered sample: [DICOM] /tmp/tmpv6ltpo84/zi/flstorage/group_snip/INDICATE/MRI/sourcedata/ANONIMA/sub-INDI71FGT/ses-02/016-B0diff_PA_64ch_DCCN_1.6mm_120b_2500_1250/PSY04725.MR._.0016.0001.2022.06.03.14.30.52.355462.62957832.IMA                                                                              
INFO     Discovered sample: [DICOM] /tmp/tmpv6ltpo84/zi/flstorage/group_snip/INDICATE/MRI/sourcedata/ANONIMA/sub-INDI71FGT/ses-02/017-B0diff_PAropepolarity_64ch_DCCN_1.6mm_120b_2500_1250_SBRef/PSY04725.MR._.0017.0001.2022.06.03.14.30.52.355462.62958730.IMA                                                            
INFO     Discovered sample: [DICOM] /tmp/tmpv6ltpo84/zi/flstorage/group_snip/INDICATE/MRI/sourcedata/ANONIMA/sub-INDI71FGT/ses-02/018-B0diff_PAropepolarity_64ch_DCCN_1.6mm_120b_2500_1250/PSY04725.MR._.0018.0001.2022.06.03.14.30.52.355462.62958858.IMA                                                                  
INFO     Discovered sample: [DICOM] /tmp/tmpv6ltpo84/zi/flstorage/group_snip/INDICATE/MRI/sourcedata/ANONIMA/sub-INDI71FGT/ses-02/019-diff_PA_64ch_DCCN_1.6mm_120b_2500_1250_SBRef/PSY04725.MR._.0019.0001.2022.06.03.14.30.52.355462.62959756.IMA                                                                          
INFO     Discovered sample: [DICOM] /tmp/tmpv6ltpo84/zi/flstorage/group_snip/INDICATE/MRI/sourcedata/ANONIMA/sub-INDI71FGT/ses-02/020-diff_PA_64ch_DCCN_1.6mm_120b_2500_1250/PSY04725.MR._.0020.0001.2022.06.03.14.30.52.355462.62959883.IMA                                                                                
INFO     Discovered sample: [DICOM] /tmp/tmpv6ltpo84/zi/flstorage/group_snip/INDICATE/MRI/sourcedata/ANONIMA/sub-INDI71FGT/ses-02/021-diff_PA_64ch_DCCN_1.6mm_120b_2500_1250_TRACEW/PSY04725.MR._.0021.0001.2022.06.03.14.30.52.355462.62989480.IMA                                                                         
INFO     Mapping: /zi/flstorage/group_snip/INDICATE/MRI/sourcedata/ANONIMA/sub-INDI7FWFU/ses-01 (subject 28/154)                                                                                                                                                                                                            
INFO     No dcm2niix2bids sourcedata found in: /zi/flstorage/group_snip/INDICATE/MRI/sourcedata/ANONIMA/sub-INDI7FWFU/ses-01                                                                                                                                                                                                
INFO     Mapping: /zi/flstorage/group_snip/INDICATE/MRI/sourcedata/ANONIMA/sub-INDI819BX/ses-01 (subject 29/154)                                                                                                                                                                                                            
INFO     No dcm2niix2bids sourcedata found in: /zi/flstorage/group_snip/INDICATE/MRI/sourcedata/ANONIMA/sub-INDI819BX/ses-01                                                                                                                                                                                                
INFO     Mapping: /zi/flstorage/group_snip/INDICATE/MRI/sourcedata/ANONIMA/sub-INDI87DVT/ses-01 (subject 30/154)                                                                                                                                                                                                            
INFO     No dcm2niix2bids sourcedata found in: /zi/flstorage/group_snip/INDICATE/MRI/sourcedata/ANONIMA/sub-INDI87DVT/ses-01

Screenshots / relevant information:

Something must be special about this subject / be missing in all other subjects so that bidsmapper oversees lots of files.

2

I ran dicomsort on my /path/to/sourcedata and then ran bidsmapper /path/to/sourcedata /path/to/rawdata/ -s again. This time, all unique instances were picked up correctly. Don’t know why this made a difference…

3

Hi Johannes,

Yes, I can see that sorting your DICOM directly can help. The bidsmapper tries to detect your DICOM organization, and then decides whether running dicomsort is needed or not. There must have been something in your folder that caused on issue, such as a DICOMDIR file or some zipped files?

4

Good question. I will try to find out what caused the issue (the directory is quite large so I cannot really check manually, takes long to open folders, search for files, etc.)