BIDS-validator Error 1: [Code 0] INTERNAL ERROR

egor.levchenko · August 10, 2020, 4:19pm

Hello everyone,

I tried to validate OASIS3 dataset and faced with the problem. Below the error output of the online bids-validator:

Error 1: [Code 0] INTERNAL ERROR

Internal error. SOME VALIDATION STEPS MAY NOT HAVE OCCURRED1 file

NaN KB |

Location:Reason:

Maximum call stack size exceeded; please help the BIDS team and community by opening an issue >at (https://github.com/bids-standard/bids-validator/issues) with the evidence here.

Evidence:

RangeError: Maximum call stack size exceeded at Oi (https://bids-standard.github.io/bids->validator/_next/static/lcAFkzjYJB3Y9y9R5kJls/pages/index.js:1:1384079) at https://bids->standard.github.io/bids-validator/_next/static/lcAFkzjYJB3Y9y9R5kJls/pages/index.js:1:1396153

Looks like that the dataset is too big for the validation, am I right? The dataset description:

Summary

8792 Files, 61.31GB
351 - Subjects
565 - Sessions

Available Tasks

rest

Available Modalities

bold
dwi
T1w

You can find the output log file attached.controls_errors.txt (418.9 KB)

BIDS-validation is important for me because I am preprocessing this dataset with FMRIPrep. And during bids validation in FMRIPrep, a different error appears:

You are using fMRIPrep-20.0.7, and a newer version of fMRIPrep is available: 20.1.1. Please check out our documentation about how and when to upgrade:
https://fmriprep.readthedocs.io/en/latest/faq.html#upgrading
Making sure the input data is BIDS compliant (warnings can be ignored in most cases).
bids-validator@1.4.0

Unhandled rejection (
reason: TypeError: Cannot read property ‘split’ of undefined
at Minimatch.match (/usr/lib/node_modules/bids-validator/node_modules/minimatch/minimatch.js:717:9)
at minimatch (/usr/lib/node_modules/bids-validator/node_modules/minimatch/minimatch.js:107:42)
at Object.ignoredFile (/usr/lib/node_modules/bids-validator/utils/config.js:11:13)
at Function.format (/usr/lib/node_modules/bids-validator/utils/issues/index.js:65:16)
at Function.exceptionHandler (/usr/lib/node_modules/bids-validator/utils/issues/index.js:191:25)
at validateMisc.then.then.then.then.then.then.then.then.catch.err (/usr/lib/node_modules/bids-validator/validators/bids/fullTest.js:213:35)
at process._tickCallback (internal/process/next_tick.js:68:7)
).

Traceback (most recent call last):
File “/usr/local/miniconda/bin/fmriprep”, line 10, in
sys.exit(main())
File “/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/cli/run.py”, line 319, in main
validate_input_dir(exec_env, opts.bids_dir, opts.participant_label)
File “/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/utils/bids.py”, line 146, in validate_input_dir
subprocess.check_call([‘bids-validator’, bids_dir, ‘-c’, temp.name])
File “/usr/local/miniconda/lib/python3.7/subprocess.py”, line 341, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command ‘[‘bids-validator’, PosixPath(’/home/elevchenko/OASIS-3/raw_data/controls’), ‘-c’, ‘/tmp/tmpfulb8q86’]’ returned non-zero exit status 3.
Sentry is attempting to send 0 pending error messages

Do you have any ideas why such errors appear?

rwblair · August 10, 2020, 6:10pm

So the second is a bug with the validator. I have made an issue for it and will update here when a fix has been made and a new version published:

First one is a problem at large that will take a bit longer to figure out. We are seeing the validator run into a variety of limits depending on system as datasets grow in size.

As a heads up the validator called inside fMRIPrep ignores a wide variety of issues during bids-validation:

github.com

poldracklab/fmriprep/blob/master/fmriprep/utils/bids.py#L62


        desc['License'] = orig_desc['License']

    with (deriv_dir / 'dataset_description.json').open('w') as fobj:
        json.dump(desc, fobj, indent=4)


def validate_input_dir(exec_env, bids_dir, participant_label):
    # Ignore issues and warnings that should not influence FMRIPREP
    import tempfile
    import subprocess
    validator_config_dict = {
        "ignore": [
            "EVENTS_COLUMN_ONSET",
            "EVENTS_COLUMN_DURATION",
            "TSV_EQUAL_ROWS",
            "TSV_EMPTY_CELL",
            "TSV_IMPROPER_NA",
            "VOLUME_COUNT_MISMATCH",
            "BVAL_MULTIPLE_ROWS",
            "BVEC_NUMBER_ROWS",
            "DWI_MISSING_BVAL",

If its urgent to try and process this through fmriprep before a fix is implemented there is a flag in fmriprep to skip the validator:
--skip_bids_validation

egor.levchenko · August 11, 2020, 12:11pm

Thank you very much!

Yes, I think I am gonna use --skip_bids_validation flag because I am pretty sure that is is bids valid.

egor.levchenko · August 25, 2020, 5:32pm

I don’t know are these problems dependent or related somehow to each other, but I faced with a problem during fMRIPrep usage on this dataset. Detailed description is here: