Summary of what happened:
I am currently trying to upload dataset ds005444 to OpenNeuro via the online GUI. It is a large dataset of over 300 GB. The upload started out well and was listing the files being uploaded from the computer - however, it appears to have gotten stuck very close to the end of the progress bar (for ~3 hours when a previous attempt to upload took a bit over 1 hour for the same dataset), and the “View Details” window does not list any files that are currently being uploaded. Furthermore, the dataset page to which it is being uploaded appears empty with no files.
When I check the console logs on the OpenNeuro website → inspect element, I see a 502 bad gateway HTTP error (the same one I get when I try uploading anything via the CLI). I previously tried uploading using openneuro-cli with the same dataset and a similar result occurred with no files apparently being uploaded. Any help would be greatly appreciated, thanks!
Command used:
- Uploading via GUI, from a folder on a remote HPC cluster mounted on Ubuntu 22.04 via
sshfswith full permissions. - Uploading via CLI directly from an HPC cluster running Rocky Linux 9, using the commands:
openneuro upload /arc/project/st-toddrebe-1/SENSIGEN/BIDS_dataset_aug25_24 --ignoreSubjectConsistency --ignoreWarnings
and then:
openneuro upload --dataset ds005441 /arc/project/st-toddrebe-1/SENSIGEN/BIDS_dataset_aug25_24 --ignoreSubjectConsistency --ignoreWarnings
Version:
- Browser upload (Firefox on Ubuntu 22.04)
- openneuro-cli v4.27.0
Data formatted according to a validatable standard? Please provide the output of the validator:
- Data validates with known warnings:
bids-validator@1.14.8
1: [WARN] You should define 'SliceTiming' for this file. If you don't provide this information slice time correction will not be possible. 'Slice Timing' is the time at which each slice was acquired within each volume (frame) of the acquisition. Slice timing is not slice order -- rather, it is a list of times containing the time (in seconds) of each slice acquisition in relation to the beginning of volume acquisition. (code: 13 - SLICE_TIMING_NOT_DEFINED)
./sub-001/func/sub-001_task-NA_run-08_echo-1_bold.nii.gz
./sub-001/func/sub-001_task-NA_run-08_echo-2_bold.nii.gz
./sub-001/func/sub-001_task-NA_run-08_echo-3_bold.nii.gz
./sub-001/func/sub-001_task-morph_run-10_echo-1_bold.nii.gz
./sub-001/func/sub-001_task-morph_run-10_echo-2_bold.nii.gz
./sub-001/func/sub-001_task-morph_run-10_echo-3_bold.nii.gz
./sub-001/func/sub-001_task-rest_run-05_echo-1_bold.nii.gz
./sub-001/func/sub-001_task-rest_run-05_echo-2_bold.nii.gz
./sub-001/func/sub-001_task-rest_run-05_echo-3_bold.nii.gz
./sub-002/func/sub-002_task-morph_run-09_echo-1_bold.nii.gz
... and 1820 more files having this issue (Use --verbose to see them all).
Please visit https://neurostars.org/search?q=SLICE_TIMING_NOT_DEFINED for existing conversations about this issue.
2: [WARN] Task scans should have a corresponding events.tsv file. If this is a resting state scan you can ignore this warning or rename the task to include the word "rest". (code: 25 - EVENTS_TSV_MISSING)
./sub-001/func/sub-001_task-NA_run-08_echo-1_bold.nii.gz
./sub-001/func/sub-001_task-NA_run-08_echo-2_bold.nii.gz
./sub-001/func/sub-001_task-NA_run-08_echo-3_bold.nii.gz
./sub-001/func/sub-001_task-morph_run-10_echo-1_bold.nii.gz
./sub-001/func/sub-001_task-morph_run-10_echo-2_bold.nii.gz
./sub-001/func/sub-001_task-morph_run-10_echo-3_bold.nii.gz
./sub-002/func/sub-002_task-morph_run-09_echo-1_bold.nii.gz
./sub-002/func/sub-002_task-morph_run-09_echo-2_bold.nii.gz
./sub-002/func/sub-002_task-morph_run-09_echo-3_bold.nii.gz
./sub-003/func/sub-003_task-NA_run-07_echo-1_bold.nii.gz
... and 854 more files having this issue (Use --verbose to see them all).
Please visit https://neurostars.org/search?q=EVENTS_TSV_MISSING for existing conversations about this issue.
3: [WARN] Not all subjects contain the same files. Each subject should contain the same number of files with the same naming unless some files are known to be missing. (code: 38 - INCONSISTENT_SUBJECTS)
./sub-001/func/sub-001_task-NA_run-01_events.json
Evidence: Subject: sub-001; Missing file: sub-001_task-NA_run-01_events.json
./sub-001/func/sub-001_task-NA_run-01_events.tsv
Evidence: Subject: sub-001; Missing file: sub-001_task-NA_run-01_events.tsv
./sub-001/func/sub-001_task-NA_run-04_echo-1_bold.json
Evidence: Subject: sub-001; Missing file: sub-001_task-NA_run-04_echo-1_bold.json
./sub-001/func/sub-001_task-NA_run-04_echo-1_bold.nii.gz
Evidence: Subject: sub-001; Missing file: sub-001_task-NA_run-04_echo-1_bold.nii.gz
./sub-001/func/sub-001_task-NA_run-04_echo-2_bold.json
Evidence: Subject: sub-001; Missing file: sub-001_task-NA_run-04_echo-2_bold.json
./sub-001/func/sub-001_task-NA_run-04_echo-2_bold.nii.gz
Evidence: Subject: sub-001; Missing file: sub-001_task-NA_run-04_echo-2_bold.nii.gz
./sub-001/func/sub-001_task-NA_run-04_echo-3_bold.json
Evidence: Subject: sub-001; Missing file: sub-001_task-NA_run-04_echo-3_bold.json
./sub-001/func/sub-001_task-NA_run-04_echo-3_bold.nii.gz
Evidence: Subject: sub-001; Missing file: sub-001_task-NA_run-04_echo-3_bold.nii.gz
./sub-001/func/sub-001_task-NA_run-06_echo-1_bold.json
Evidence: Subject: sub-001; Missing file: sub-001_task-NA_run-06_echo-1_bold.json
./sub-001/func/sub-001_task-NA_run-06_echo-1_bold.nii.gz
Evidence: Subject: sub-001; Missing file: sub-001_task-NA_run-06_echo-1_bold.nii.gz
... and 35810 more files having this issue (Use --verbose to see them all).
Please visit https://neurostars.org/search?q=INCONSISTENT_SUBJECTS for existing conversations about this issue.
4: [WARN] Not all subjects/sessions/runs have the same scanning parameters. (code: 39 - INCONSISTENT_PARAMETERS)
./sub-001/func/sub-001_task-NA_run-08_echo-1_bold.nii.gz
./sub-001/func/sub-001_task-NA_run-08_echo-2_bold.nii.gz
./sub-001/func/sub-001_task-NA_run-08_echo-3_bold.nii.gz
./sub-001/func/sub-001_task-morph_run-10_echo-1_bold.nii.gz
./sub-001/func/sub-001_task-morph_run-10_echo-2_bold.nii.gz
./sub-001/func/sub-001_task-morph_run-10_echo-3_bold.nii.gz
./sub-002/func/sub-002_task-morph_run-09_echo-1_bold.nii.gz
./sub-002/func/sub-002_task-morph_run-09_echo-2_bold.nii.gz
./sub-002/func/sub-002_task-morph_run-09_echo-3_bold.nii.gz
./sub-003/func/sub-003_task-NA_run-07_echo-1_bold.nii.gz
... and 128 more files having this issue (Use --verbose to see them all).
Please visit https://neurostars.org/search?q=INCONSISTENT_PARAMETERS for existing conversations about this issue.
5: [WARN] The onset of the last event is after the total duration of the corresponding scan. This design is suspiciously long. (code: 85 - SUSPICIOUSLY_LONG_EVENT_DESIGN)
./sub-049/func/sub-049_task-brush_run-07_events.tsv
./sub-049/func/sub-049_task-brush_run-07_events.tsv
./sub-049/func/sub-049_task-brush_run-07_events.tsv
Please visit https://neurostars.org/search?q=SUSPICIOUSLY_LONG_EVENT_DESIGN for existing conversations about this issue.