Hi all - we are trying to process the unprocessed HCP data through fmriprep. We have converted a sample subject using hcp2bids. Here is the tree for the sample dataset. As you’ll see, some of the files are symlinks and some aren’t. This was built into hcp2bids.
hcp_test_bids
├── dataset_description.json
├── sub-132118
│ ├── anat
│ │ ├── sub-132118_run-01_T1w.nii.gz -> /gpfs/gsfs6/users/HCP/hcp_test/132118/unprocessed/3T/T1w_MPR2/132118_3T_T1w_MPR2.nii.gz
│ │ ├── sub-132118_run-01_T2w.nii.gz -> /gpfs/gsfs6/users/HCP/hcp_test/132118/unprocessed/3T/T2w_SPC1/132118_3T_T2w_SPC1.nii.gz
│ │ ├── sub-132118_run-02_T1w.nii.gz -> /gpfs/gsfs6/users/HCP/hcp_test/132118/unprocessed/3T/T1w_MPR1/132118_3T_T1w_MPR1.nii.gz
│ │ └── sub-132118_run-02_T2w.nii.gz -> /gpfs/gsfs6/users/HCP/hcp_test/132118/unprocessed/3T/T2w_SPC2/132118_3T_T2w_SPC2.nii.gz
│ ├── fmap
│ │ ├── sub-132118_dir-1_epi.json
│ │ ├── sub-132118_dir-1_epi.nii.gz
│ │ ├── sub-132118_dir-2_epi.json
│ │ ├── sub-132118_dir-2_epi.nii.gz
│ │ ├── sub-132118_dir-3_epi.json
│ │ ├── sub-132118_dir-3_epi.nii.gz
│ │ ├── sub-132118_dir-4_epi.json
│ │ ├── sub-132118_dir-4_epi.nii.gz
│ │ ├── sub-132118_dir-5_epi.json
│ │ ├── sub-132118_dir-5_epi.nii.gz
│ │ ├── sub-132118_dir-6_epi.json
│ │ ├── sub-132118_dir-6_epi.nii.gz
│ │ ├── sub-132118_dir-7_epi.json
│ │ ├── sub-132118_dir-7_epi.nii.gz
│ │ ├── sub-132118_dir-8_epi.json
│ │ ├── sub-132118_dir-8_epi.nii.gz
│ │ ├── sub-132118_run-01_magnitude1.nii.gz
│ │ ├── sub-132118_run-01_magnitude2.nii.gz
│ │ ├── sub-132118_run-01_phasediff.json
│ │ ├── sub-132118_run-01_phasediff.nii.gz
│ │ ├── sub-132118_run-02_magnitude1.nii.gz
│ │ ├── sub-132118_run-02_magnitude2.nii.gz
│ │ ├── sub-132118_run-02_phasediff.json
│ │ ├── sub-132118_run-02_phasediff.nii.gz
│ │ ├── sub-132118_run-03_magnitude1.nii.gz
│ │ ├── sub-132118_run-03_magnitude2.nii.gz
│ │ ├── sub-132118_run-03_phasediff.json
│ │ ├── sub-132118_run-03_phasediff.nii.gz
│ │ ├── sub-132118_run-04_magnitude1.nii.gz
│ │ ├── sub-132118_run-04_magnitude2.nii.gz
│ │ ├── sub-132118_run-04_phasediff.json
│ │ └── sub-132118_run-04_phasediff.nii.gz
│ └── func
│ ├── sub-132118_task-rest_acq-lr_run-01_bold.nii.gz -> /gpfs/gsfs6/users/HCP/hcp_test/132118/unprocessed/3T/rfMRI_REST1_LR/132118_3T_rfMRI_REST1_LR.nii.gz
│ ├── sub-132118_task-rest_acq-lr_run-01_sbref.nii.gz -> /gpfs/gsfs6/users/HCP/hcp_test/132118/unprocessed/3T/rfMRI_REST1_LR/132118_3T_rfMRI_REST1_LR_SBRef.nii.gz
│ ├── sub-132118_task-rest_acq-lr_run-02_bold.nii.gz -> /gpfs/gsfs6/users/HCP/hcp_test/132118/unprocessed/3T/rfMRI_REST2_LR/132118_3T_rfMRI_REST2_LR.nii.gz
│ ├── sub-132118_task-rest_acq-lr_run-02_sbref.nii.gz -> /gpfs/gsfs6/users/HCP/hcp_test/132118/unprocessed/3T/rfMRI_REST2_LR/132118_3T_rfMRI_REST2_LR_SBRef.nii.gz
│ ├── sub-132118_task-rest_acq-rl_run-01_bold.nii.gz -> /gpfs/gsfs6/users/HCP/hcp_test/132118/unprocessed/3T/rfMRI_REST1_RL/132118_3T_rfMRI_REST1_RL.nii.gz
│ ├── sub-132118_task-rest_acq-rl_run-01_sbref.nii.gz -> /gpfs/gsfs6/users/HCP/hcp_test/132118/unprocessed/3T/rfMRI_REST1_RL/132118_3T_rfMRI_REST1_RL_SBRef.nii.gz
│ ├── sub-132118_task-rest_acq-rl_run-02_bold.nii.gz -> /gpfs/gsfs6/users/HCP/hcp_test/132118/unprocessed/3T/rfMRI_REST2_RL/132118_3T_rfMRI_REST2_RL.nii.gz
│ └── sub-132118_task-rest_acq-rl_run-02_sbref.nii.gz -> /gpfs/gsfs6/users/HCP/hcp_test/132118/unprocessed/3T/rfMRI_REST2_RL/132118_3T_rfMRI_REST2_RL_SBRef.nii.gz
├── task-rest_acq-lr_bold.json
├── task-rest_acq-lr_sbref.json
├── task-rest_acq-rl_bold.json
└── task-rest_acq-rl_sbref.json
4 directories, 49 files
The sample dataset passes the validator in the browser but fails when using a singularity image (v1.6.0) on our HPC:
1: [ERR] Internal error. SOME VALIDATION STEPS MAY NOT HAVE OCCURRED (code: 0 - INTERNAL ERROR)
.undefined
Evidence: Error: Could not load HED schema version "Latest" from remote repository - "Error: getaddrinfo EAI_AGAIN raw.githubusercontent.com".
at /src/node_modules/hed-validator/utils/schema.js:40:13
at tryCallOne (/src/node_modules/promise/lib/core.js:37:12)
at /src/node_modules/promise/lib/core.js:123:15
at flush (/src/node_modules/asap/raw.js:50:29)
at processTicksAndRejections (internal/process/task_queues.js:76:11)
Please visit https://neurostars.org/search?q=INTERNAL ERROR for existing conversations about this issue.
2: [ERR] 'IntendedFor' field needs to point to an existing file. (code: 37 - INTENDED_FOR)
./sub-132118/fmap/sub-132118_dir-1_epi.nii.gz
Evidence: func/sub-132118_task-rest_acq-rl_run-01_bold.nii.gz
./sub-132118/fmap/sub-132118_dir-2_epi.nii.gz
Evidence: func/sub-132118_task-rest_acq-rl_run-01_bold.nii.gz
./sub-132118/fmap/sub-132118_dir-3_epi.nii.gz
Evidence: func/sub-132118_task-rest_acq-lr_run-01_bold.nii.gz
./sub-132118/fmap/sub-132118_dir-4_epi.nii.gz
Evidence: func/sub-132118_task-rest_acq-lr_run-02_bold.nii.gz
./sub-132118/fmap/sub-132118_dir-5_epi.nii.gz
Evidence: func/sub-132118_task-rest_acq-lr_run-01_bold.nii.gz
./sub-132118/fmap/sub-132118_dir-6_epi.nii.gz
Evidence: func/sub-132118_task-rest_acq-rl_run-02_bold.nii.gz
./sub-132118/fmap/sub-132118_dir-7_epi.nii.gz
Evidence: func/sub-132118_task-rest_acq-lr_run-02_bold.nii.gz
./sub-132118/fmap/sub-132118_dir-8_epi.nii.gz
Evidence: func/sub-132118_task-rest_acq-rl_run-02_bold.nii.gz
./sub-132118/fmap/sub-132118_run-01_phasediff.nii.gz
Evidence: anat/sub-132118_run-01_T1w.nii.gz
./sub-132118/fmap/sub-132118_run-02_phasediff.nii.gz
Evidence: anat/sub-132118_run-02_T2w.nii.gz
... and 2 more files having this issue (Use --verbose to see them all).
Please visit https://neurostars.org/search?q=INTENDED_FOR for existing conversations about this issue.
1: [WARN] The recommended file /README is missing. See Section 03 (Modality agnostic files) of the BIDS specification. (code: 101 - README_FILE_MISSING)
Please visit https://neurostars.org/search?q=README_FILE_MISSING for existing conversations about this issue.
2: [WARN] The Authors field of dataset_description.json should contain an array of fields - with one author per field. This was triggered because there are no authors, which will make DOI registration from dataset metadata impossible. (code: 113 - NO_AUTHORS)
Please visit https://neurostars.org/search?q=NO_AUTHORS for existing conversations about this issue.
Summary: Available Tasks: Available Modalities:
37 Files, 42.25MB rest epi
1 - Subject magnitude1
1 - Session magnitude2
phasediff
If you have any questions, please post on https://neurostars.org/tags/bids.
We decided to push through to get fmriprep running and are hitting an error. Below is the fmriprep command, version 20.2.0. I have doubled check that we have all the available resources needed for this processing.
singularity run --cleanenv \
-B $HOME/.cache/templateflow:/templateflow \
-B /data/HCP/hcp_test_bids:/in \
-B /data/HCP/fmriprep_test:/out \
-B /data/HCP/fmriprep_test/wrk:/wrk \
/data/MLDSST/containers/fmriprep-20.2.0.simg /in /out participant \
--participant_label sub-132118 \
-w /wrk \
--nthreads $SLURM_CPUS_PER_TASK \
--mem_mb $SLURM_MEM_PER_NODE \
--fs-license-file /data/BrainBlocks/code/license.txt \
--output-spaces fsaverage6 fsLR MNI152NLin2009cAsym \
--cifti-output \
--skip-bids-validation \
--notrack \
--omp-nthreads 8 \
--use-aroma \
--error-on-aroma-warnings
And here is the error:
/usr/local/miniconda/lib/python3.7/site-packages/bids/layout/validation.py:46: UserWarning: The ability to pass arguments to BIDSLayout that control indexing is likely to be removed in future; possibly as early as PyBIDS 0.14. This includes the `config_filename`, `ignore`, `force_index`, and `index_metadata` arguments. The recommended usage pattern is to initialize a new BIDSLayoutIndexer with these arguments, and pass it to the BIDSLayout via the `indexer` argument.
warnings.warn("The ability to pass arguments to BIDSLayout that control "
You are using fMRIPrep-20.2.0, and a newer version of fMRIPrep is available: 20.2.1.
Please check out our documentation about how and when to upgrade:
https://fmriprep.readthedocs.io/en/latest/faq.html#upgrading
210422-15:18:28,290 cli WARNING:
Per-process threads (--omp-nthreads=8) exceed total threads (--nthreads/--n_cpus=2)
210422-15:18:35,116 nipype.workflow WARNING:
Previous output generated by version 0+unknown found.
210422-15:18:35,163 nipype.workflow IMPORTANT:
Running fMRIPREP version 20.2.0:
* BIDS dataset path: /in.
* Participant list: ['132118'].
* Run identifier: 20210422-151824_4c3074cb-2378-4557-92a9-41776d33eca6.
* Output spaces: fsaverage:den-41k fsLR MNI152NLin2009cAsym:res-native.
* Pre-run FreeSurfer's SUBJECTS_DIR: /out/freesurfer.
210422-15:18:36,196 nipype.workflow INFO:
Using single-band reference file(s) sub-132118_task-rest_acq-lr_run-01_sbref.nii.gz.
210422-15:18:37,156 nipype.workflow INFO:
Using single-band reference file(s) sub-132118_task-rest_acq-lr_run-02_sbref.nii.gz.
210422-15:18:37,782 nipype.workflow INFO:
Using single-band reference file(s) sub-132118_task-rest_acq-rl_run-01_sbref.nii.gz.
210422-15:18:38,434 nipype.workflow INFO:
Using single-band reference file(s) sub-132118_task-rest_acq-rl_run-02_sbref.nii.gz.
210422-15:18:42,544 nipype.workflow INFO:
fMRIPrep workflow graph with 1413 nodes built successfully.
210422-15:18:55,955 nipype.workflow IMPORTANT:
fMRIPrep started!
210422-15:19:23,2 nipype.workflow CRITICAL:
fMRIPrep failed: The 'header_source' trait of a _MergeInputSpec instance must be a pathlike object or string representing an existing file, but a value of '/in/sub-132118/func/sub-132118_task-rest_acq-lr_run-01_bold.nii.gz' <class 'str'> was specified.
- Do you have any idea why the validator might be giving different results between the browser and the singularity image? The browser is running from the same machine as the container.
- What issues might cause the fmriprep error?
Thanks in advance for your help!