BIDS validator error

, ,
1

Hi,

I’m running fmriprep version 1.3.0 and recently ran into a validation error when trying to run multiple subjects. Currently, the structure of my bids directory is as follows (before running fmriprep, mriqc, etc):

.
   |-code
   |-derivatives
   |-sourcedata
   |-sub-000
   |---anat
   |---fmap
   |---func
   |-sub-001
   |---anat
   |---fmap
   |---func
   |-sub-002
   |---anat
   |---fmap
   |---func
   |-sub-003
   |---anat
   |---fmap
   |---func
   |-sub-004
   |---anat
   |---fmap
   |---func
   |-sub-005
   |---anat
   |---fmap
   |---func

When I run fmriprep on a single subject, it creates a fmriprep_wf directory, as well as derivatives/fmriprep, etc. The error log when I try to run a new subject is:

1: [ERR] Files with such naming scheme are not part of BIDS specification. This error is most commonly caused by typos in file names that make them not BIDS compatible. Please consult the specification and make sure your files are named correctly. If this is not a file naming issue (for example when including files not yet covered by the BIDS specification) you should include a ".bidsignore" file in your dataset (see https://github.com/bids-standard/bids-validator#bidsignore for details). Please note that derived (processed) data should be placed in /derivatives folder and source data (such as DICOMS or behavioural logs in proprietary formats) should be placed in the /sourcedata folder. (code: 1 - NOT_INCLUDED)
		./fmriprep_wf/d3.js
			Evidence: d3.js
		./fmriprep_wf/fsdir_run_20190222_163801_3cd723ac_5254_4dcf_b642_458834c2c147/_0xf406549eff9ccf9d957106c885fc93cb.json
			Evidence: _0xf406549eff9ccf9d957106c885fc93cb.json
		./fmriprep_wf/fsdir_run_20190222_163801_3cd723ac_5254_4dcf_b642_458834c2c147/_inputs.pklz
			Evidence: _inputs.pklz
		./fmriprep_wf/fsdir_run_20190222_163801_3cd723ac_5254_4dcf_b642_458834c2c147/_node.pklz
			Evidence: _node.pklz
		./fmriprep_wf/fsdir_run_20190222_163801_3cd723ac_5254_4dcf_b642_458834c2c147/_report/report.rst
			Evidence: report.rst
		./fmriprep_wf/fsdir_run_20190222_163801_3cd723ac_5254_4dcf_b642_458834c2c147/result_fsdir_run_20190222_163801_3cd723ac_5254_4dcf_b642_458834c2c147.pklz
			Evidence: result_fsdir_run_20190222_163801_3cd723ac_5254_4dcf_b642_458834c2c147.pklz
		./fmriprep_wf/graph.dot
			Evidence: graph.dot
		./fmriprep_wf/graph.json
			Evidence: graph.json
		./fmriprep_wf/graph.svg
			Evidence: graph.svg
		./fmriprep_wf/graph1.json
			Evidence: graph1.json
		... and 41294 more files having this issue (Use --verbose to see them all).

	2: [ERR] You have to define 'PhaseEncodingDirection' for this file. (code: 18 - PHASE_ENCODING_DIRECTION_MUST_DEFINE)
		./sub-001/fmap/sub-001_run-01_epi.nii.gz

	3: [ERR] You have to define 'TotalReadoutTime' for this file. (code: 19 - TOTAL_READOUT_TIME_MUST_DEFINE)
		./sub-001/fmap/sub-001_run-01_epi.nii.gz

	4: [ERR] Sub label contain an Illegal Character hyphen or underscore. Please edit the filename as per BIDS spec. (code: 62 - SUBJECT_VALUE_CONTAINS_ILLEGAL_CHARECTER)
		./reportlets/fmriprep/sub-000/anat/sub-000_seg_brainmask.svg
			Evidence: sub name contains illegal character:/reportlets/fmriprep/sub-000/anat/sub-000_seg_brainmask.svg
		./reportlets/fmriprep/sub-000/anat/sub-000_t1_2_mni.svg
			Evidence: sub name contains illegal character:/reportlets/fmriprep/sub-000/anat/sub-000_t1_2_mni.svg

	5: [ERR] Subject label in the filename doesn't match with the path of the file. File seems to be saved in incorrect subject directory. (code: 64 - SUBJECT_LABEL_IN_FILENAME_DOESNOT_MATCH_DIRECTORY)
		./fmriprep_wf/single_subject_000_wf/func_preproc_task_std_run_01_wf/bold_hmc_wf/mcflirt/sub-000_task-std_run-01_bold_mcf.nii.gz.par
			Evidence: File: /fmriprep_wf/single_subject_000_wf/func_preproc_task_std_run_01_wf/bold_hmc_wf/mcflirt/sub-000_task-std_run-01_bold_mcf.nii.gz.par is saved in incorrect subject directory as per sub-id in filename.
		./fmriprep_wf/single_subject_000_wf/func_preproc_task_std_run_01_wf/bold_stc_wf/copy_xform/sub-000_task-std_run-01_bold_tshift_xform.nii.gz
			Evidence: File: /fmriprep_wf/single_subject_000_wf/func_preproc_task_std_run_01_wf/bold_stc_wf/copy_xform/sub-000_task-std_run-01_bold_tshift_xform.nii.gz is saved in incorrect subject directory as per sub-id in filename.
		./fmriprep_wf/single_subject_000_wf/func_preproc_task_std_run_01_wf/bold_stc_wf/slice_timing_correction/sub-000_task-std_run-01_bold_tshift.nii.gz
			Evidence: File: /fmriprep_wf/single_subject_000_wf/func_preproc_task_std_run_01_wf/bold_stc_wf/slice_timing_correction/sub-000_task-std_run-01_bold_tshift.nii.gz is saved in incorrect subject directory as per sub-id in filename.
		./fmriprep_wf/single_subject_000_wf/func_preproc_task_std_run_02_wf/bold_hmc_wf/mcflirt/sub-000_task-std_run-02_bold_mcf.nii.gz.par
			Evidence: File: /fmriprep_wf/single_subject_000_wf/func_preproc_task_std_run_02_wf/bold_hmc_wf/mcflirt/sub-000_task-std_run-02_bold_mcf.nii.gz.par is saved in incorrect subject directory as per sub-id in filename.
		./fmriprep_wf/single_subject_000_wf/func_preproc_task_std_run_02_wf/bold_stc_wf/copy_xform/sub-000_task-std_run-02_bold_tshift_xform.nii.gz
			Evidence: File: /fmriprep_wf/single_subject_000_wf/func_preproc_task_std_run_02_wf/bold_stc_wf/copy_xform/sub-000_task-std_run-02_bold_tshift_xform.nii.gz is saved in incorrect subject directory as per sub-id in filename.
		./fmriprep_wf/single_subject_000_wf/func_preproc_task_std_run_02_wf/bold_stc_wf/slice_timing_correction/sub-000_task-std_run-02_bold_tshift.nii.gz
			Evidence: File: /fmriprep_wf/single_subject_000_wf/func_preproc_task_std_run_02_wf/bold_stc_wf/slice_timing_correction/sub-000_task-std_run-02_bold_tshift.nii.gz is saved in incorrect subject directory as per sub-id in filename.
		./fmriprep_wf/single_subject_000_wf/func_preproc_task_std_run_03_wf/bold_hmc_wf/mcflirt/sub-000_task-std_run-03_bold_mcf.nii.gz.par
			Evidence: File: /fmriprep_wf/single_subject_000_wf/func_preproc_task_std_run_03_wf/bold_hmc_wf/mcflirt/sub-000_task-std_run-03_bold_mcf.nii.gz.par is saved in incorrect subject directory as per sub-id in filename.
		./fmriprep_wf/single_subject_000_wf/func_preproc_task_std_run_03_wf/bold_stc_wf/copy_xform/sub-000_task-std_run-03_bold_tshift_xform.nii.gz
			Evidence: File: /fmriprep_wf/single_subject_000_wf/func_preproc_task_std_run_03_wf/bold_stc_wf/copy_xform/sub-000_task-std_run-03_bold_tshift_xform.nii.gz is saved in incorrect subject directory as per sub-id in filename.
		./fmriprep_wf/single_subject_000_wf/func_preproc_task_std_run_03_wf/bold_stc_wf/slice_timing_correction/sub-000_task-std_run-03_bold_tshift.nii.gz
			Evidence: File: /fmriprep_wf/single_subject_000_wf/func_preproc_task_std_run_03_wf/bold_stc_wf/slice_timing_correction/sub-000_task-std_run-03_bold_tshift.nii.gz is saved in incorrect subject directory as per sub-id in filename.
		./fmriprep_wf/single_subject_000_wf/func_preproc_task_std_run_04_wf/bold_hmc_wf/mcflirt/sub-000_task-std_run-04_bold_mcf.nii.gz.par
			Evidence: File: /fmriprep_wf/single_subject_000_wf/func_preproc_task_std_run_04_wf/bold_hmc_wf/mcflirt/sub-000_task-std_run-04_bold_mcf.nii.gz.par is saved in incorrect subject directory as per sub-id in filename.
		... and 56 more files having this issue (Use --verbose to see them all).

	6: [ERR] No BIDS compatible data found for at least one subject. (code: 67 - NO_VALID_DATA_FOUND_FOR_SUBJECT)
		./sub-000_task-std_run-01_bold_mcf.nii.gz.mat
		./sub-000_task-std_run-02_bold_mcf.nii.gz.mat
		./sub-000_task-std_run-03_bold_mcf.nii.gz.mat
		./sub-000_task-std_run-04_bold_mcf.nii.gz.mat
		./sub-000_task-std_run-05_bold_mcf.nii.gz.mat
		./sub-000_task-std_run-06_bold_mcf.nii.gz.mat

	7: [ERR] Empty files not allowed. (code: 99 - EMPTY_FILE)
		./fmriprep_wf/single_subject_000_wf/func_preproc_task_std_run_01_wf/bold_bold_trans_wf/bold_reference_wf/validate/report.html
		./fmriprep_wf/single_subject_000_wf/func_preproc_task_std_run_01_wf/bold_mni_trans_wf/bold_reference_wf/validate/report.html
		./fmriprep_wf/single_subject_000_wf/func_preproc_task_std_run_01_wf/bold_reference_wf/validate/report.html
		./fmriprep_wf/single_subject_000_wf/func_preproc_task_std_run_01_wf/bold_t1_trans_wf/bold_reference_wf/validate/report.html
		./fmriprep_wf/single_subject_000_wf/func_preproc_task_std_run_01_wf/ica_aroma_wf/bold_mni_trans_wf/bold_reference_wf/validate/report.html
		./fmriprep_wf/single_subject_000_wf/func_preproc_task_std_run_02_wf/bold_bold_trans_wf/bold_reference_wf/validate/report.html
		./fmriprep_wf/single_subject_000_wf/func_preproc_task_std_run_02_wf/bold_mni_trans_wf/bold_reference_wf/validate/report.html
		./fmriprep_wf/single_subject_000_wf/func_preproc_task_std_run_02_wf/bold_reference_wf/validate/report.html
		./fmriprep_wf/single_subject_000_wf/func_preproc_task_std_run_02_wf/bold_t1_trans_wf/bold_reference_wf/validate/report.html
		./fmriprep_wf/single_subject_000_wf/func_preproc_task_std_run_02_wf/ica_aroma_wf/bold_mni_trans_wf/bold_reference_wf/validate/report.html
		... and 27 more files having this issue (Use --verbose to see them all).

        Summary:                     Available Tasks:        Available Modalities:                  
        43030 Files, 116.03GB        std                     joined                                 
        6 - Subjects                                         wf/acompcor/acompcor                   
        1 - Session                                          motion                                 
                                                             wf/concat/confounds                    
                                                             nstd                                   
                                                             std                                    
                                                             wf/signals/signals                     
                                                             filter                                 
                                                             wf/tcompcor/tcompcor                   
                                                             extraction/AROMAAggrCompAROMAConfounds 
                                                             extraction/MELODICmix                  
                                                             /participants                          
                                                             T1w                                    
                                                             T2w                                    
                                                             bold                                   
                                                             fieldmap                               

If you have any questions please post on https://neurostars.org/tags/bids

Traceback (most recent call last):
  File "/usr/local/miniconda/bin/fmriprep", line 11, in <module>
    sys.exit(main())
  File "/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/cli/run.py", line 354, in main
    validate_input_dir(exec_env, opts.bids_dir, opts.participant_label)
  File "/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/cli/run.py", line 535, in validate_input_dir
    subprocess.check_call(['bids-validator', bids_dir, '-c', temp.name])
  File "/usr/local/miniconda/lib/python3.7/subprocess.py", line 341, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['bids-validator', '/N/dc2/scratch/dlevitas/fMRI_data/std/bids', '-c', '/tmp/tmpb6i75q59']' returned non-zero exit status 1.
Sentry is attempting to send 1 pending error messages
Waiting up to 2.0 seconds
Press Ctrl-C to quit

I realize the error is due to the fmriprep files created from the first subject, but I thought my bids directory was structured correctly according to the spec

Do I need to reorganize my bids directory in any way? I see that I can specify --skip_bids_validation; however, I appreciate that there’s a validation check at the start, and would like to avoid skipping that if possible.

Thanks for the help.

2

@dlevitas what is the full command that you are calling to run run fmriprep? It looks like the working directory and not just the output directory for fmriprep might be outputting into your bids dataset.

3

For error 1, when you run fmriprep for each of the subjects, you can either specify the work directory (with the -w flag) to be derivatives, or have a .bidsingnore file with fmriprep_wf in it. I would recommend the first option.

For errors 2 and 3, you will have to enter the required fields by hand in the corresponding json file (./sub-001/fmap/sub-001_run-01_epi.json)

Errors 4, 5 and 7 will clear once you fix error 1.

For error 6, I’m not sure where those files were generated. Where they generated by the scanner, or by running some other tool (e.g., mcflirt) on your data? Either way, they need to be moved from the top level folder.

4

Thanks @rwblair @pvelasco

The main issue was indeed that I needed to set -w $data_dir/derivatives

I forgot to mention that I do have a tmp_dcm2bids directory which has also been causing issues in the validation. I tried creating a .bidsignore file to ignore this directory, but it doesn’t seem to be working:

{
  "ignore": [
    "/N/dc2/scratch/dlevitas/fMRI_data/std/bids/tmp_dcm2bids"
  ]
}

where data_dir=/N/dc2/scratch/dlevitas/fMRI_data/std/bids

Is there a different way that the .bidsignore file needs to be formatted?

5

Hi @dlevitas

The .bidsignore are a bit different than a json file. May you try giving it the path from your root directory. Assuming your bids directory is your root, try just inputting tmp_dcm2bids/ in your .bidsignore file. That will be the only line in it.

6

Hi @franklin

I gave that a shot, but still ran into the same error. I tried providing the absolute path as well as using ignoredFiles, also to no avail.

7

Hi @dlevitas

Interesting, can you attach your .bidsignore file here so I can take a look?

8

The layout of the file is:

{
  "ignore": [
    "/N/dc2/scratch/dlevitas/fMRI_data/std/bids/tmp_dcm2bids"
  ]
}

bidsignore.txt (84 Bytes)

9

may you please try this one (attached). Please change the filename to .bidsignore. For the bidsignore file you can specify the directory you would like to skip, the structure of “ignore” is not needed.

bidsignore.txt (14 Bytes)

10

Ah, that seems to have fixed it, thanks! I think the quotation marks were causing some issues.

1 Like