Thanks for all the hard work @franklin! Just to clarify, it does not seem in my experience that the upload gets farther each time; sometimes it gets to sub-050, next time it’ll break at sub-022, and I haven’t noticed any pattern when it breaks. Regarding the dataset rendering, as far as I can tell, my existing ds002345 does not render even if, e.g., I open it in an incognito window with no login. This is troublesome because as far as I can tell, not only does my draft fail to render (and doesn’t have my updates), but other folks also can’t access the pre-existing snapshots.
Hi @lukassnoek and @snastase
Thank you for the additional information!
@lukassnoek - in your case the dataset may have been uploaded with the error message produced from a server timeout. The dataset may take a bit of time to render if this is the case. We are looking into the separate regression seen with the uploader not noticing uploaded files. (side note: appears the render time may have been it? https://openneuro.org/datasets/ds002547/versions/1.0.0)
@snastase - regrading the dataset rendering, it is associated with server serving up the information and perhaps the server giving a timeout (on the user side shows the spinner continuing to spin). A resolution is refreshing the page or reconnecting to the wifi. We are working on enhancing load times for larger datasets.
Thank you,
Franklin
Hi Franklin,
Actually, I managed to upload this particular dataset with the web interface (resulting in ds002547). But I’d love to use the CLI tool to upload three very large (± 600 GB) datasets (AOMIC … Let me know if I can give you more information to help you debug the (existing?) issues!
Best,
Lukas
Hi @lukassnoek
Great to hear that your particular dataset is up! We patched over the bug: not skipping uploaded files. We are focusing on enhancing the upload processes. We aim at the root causes that take a bit more time to resolve. I think perhaps trying and seeing if you get the same ServerParseErrors (timeouts) and this give more cases for the development team to hold onto. (looks like a cool dataset!)
Thank you,
Franklin
Hi @franklin,
Great that you are working on resolving the CLI issues! I tried to upload another dataset to see whether the ServerParseErrors would come up again, but I got another login error this time (for which I filed an issue on Github).
Best,
Lukas
Hi @lukassnoek,
Please refer to your issue on github for the fix.
Quick update: we have patched the bug related to the cli not skipping files. We are working through the root issue of enhancing our upload processes.
Thank you,
Franklin
1 Like
Hi @franklin,
Do you have any news concerning the upload issue?
I tried again from OSX 10.14.6 (18G3020), openneuro 3.13.0 installed via yarn. Here is my terminal output: https://gist.github.com/alexfoias/c2b2ea63faa6ce47dfff9532f27928c5
thanks,
Alexandru
Hi @alexfoias
Thank you for your follow up and sending over the terminal output! We have this issue documented and investigating the root cause of this issue. We recently had internal discussions on approaches to resolve the root causes of the issues related to uploading data you and others are experiencing. We apologize for the inconveniences these have caused. This is crucial for us to resolve and has our full attention.
By chance, was this error seen at the end of your dataset?
Thank you,
Franklin
Hi @franklin,
Unfortunately, I have some bad news. We’ve managed to upload and do a snapshot using openneuro-cli 3.14.2 on 2020-05-21.
I was able to modify today (2020-06-05) the authors using the webpage before doing the upload using the CLI. After that I tried to upload more data using with the same version (3.14.2) but I got this error message:
SyntaxError [ServerParseError]: Unexpected token < in JSON at position 0
at JSON.parse (<anonymous>)
at /usr/lib/node_modules/openneuro-cli/node_modules/apollo-link-http-common/lib/index.js:35:25
at processTicksAndRejections (internal/process/task_queues.js:93:5) {
name: 'ServerParseError',
response: Response {
size: 0,
timeout: 0,
[Symbol(Body internals)]: { body: [PassThrough], disturbed: true, error: null },
[Symbol(Response internals)]: {
url: 'https://openneuro.org/crn/graphql',
status: 504,
statusText: 'Gateway Time-out',
headers: [Headers],
counter: 0
}
},
statusCode: 504,
bodyText: '<html>\r\n' +
'<head><title>504 Gateway Time-out</title></head>\r\n' +
'<body bgcolor="white">\r\n' +
'<center><h1>504 Gateway Time-out</h1></center>\r\n' +
'</body>\r\n' +
'</html>\r\n'
}
I upgraded to openneuro 3.15.0-alpha.0 but the error message was the same. After that I tried to generate a new API key (in case the new version would require a new one), but the error message remained the same.
At this point I cannot access my datasets in my dashboard and neither on the link.
Do you have any suggestions for us ?
Thanks,
Alexandru
Hi @alexfoias
Darn! It appears you ran through the soft fixes we had put up previously. These soft fixes were deployed to help resolve the publication submission. We are working to enhance our underlying storage configuration. These enhancements are being developed and are posed to help alleviate the timeout issues (the case here). I apologize for this inconvenience.
It appears the dataset rendering has resolved.
Thank you,
Franklin
Hello @franklin,
I don’t know if it’s the right thread to report this, but I’m having issues with both browser and CLI upload.
I’m trying to upload a new dataset (it’s still a draft). It took me approx 18 hours for the browser upload to finish.
When I checked the dataset, I saw that only one subject was partially uploaded, and I had an error message saying that the dataset wasn’t valid any more due to files missing.
Assuming the upload went wrong (wasn’t at the screen when it finished), I tried to use the CLI to finish the upload. I installed it using npm (node v10.19.0, npm v6.14.4, openneuro v3.20.0).
I did the login with the API key and didn’t get any message from it (I think it went ok though, cause I tried to download a draft dataset of mine and it worked). Just in case, I tried another coupe of time with new API keys, once even as a super user.
When I start the upload, it gets stuck at 0 for a very long time.
This is what I get from my terminal:
$ openneuro upload --dataset ds003192 -i .
Adding files to "ds003192"
bids-validator@1.5.4
Summary: Available Tasks: Available Modalities:
1880 Files, 74.66GB TODO: full task name for breathhold T2w
7 - Subjects T1w
10 - Sessions bold
physio
sbref
fieldmap
If you have any questions, please post on https://neurostars.org/tags/bids.
=======================================================================
1686 files to be uploaded with a total size of 67.3 GB
? Begin upload? Yes
=======================================================================
Starting a new upload (6e385bc8-8254-42ca-87da-26a085ce6352) to dataset: 'ds003192'
ds003192 [----------------------------------------] 0% | ETA: 0s | 0/1686(node:2495066) UnhandledPromisen
at Proxy.uploadParallel (/usr/local/lib/node_modules/openneuro-cli/node_modules/openneuro-client/src)
at uploadFiles (/usr/local/lib/node_modules/openneuro-cli/src/upload.js:161:17)
at uploadDataset (/usr/local/lib/node_modules/openneuro-cli/src/actions.js:74:11)
at process._tickCallback (internal/process/next_tick.js:68:7)
(node:2495066) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated eith)
(node:2495066) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future,.
ds003192 [----------------------------------------] 0% | ETA: 508s | 11/1686
Am I missing something?
Hi @smoia
Thank you for your message! This may be caused by an out of date node package. May you please try with Node 12?
Thank you,
Franklin
Hi @franklin,
thank you so much, the node update (v12.18.4) solved that issue!
However, now I do see this issue quite often:
Error [ERR_STREAM_PREMATURE_CLOSE]: Premature close
at ClientHttp2Stream.onclose (internal/streams/end-of-stream.js:80:15)
at ClientHttp2Stream.emit (events.js:327:22)
at ClientHttp2Stream.EventEmitter.emit (domain.js:483:12)
at closeStream (internal/http2/core.js:1727:14)
at Http2Stream.onStreamClose (internal/http2/core.js:505:5) {
code: 'ERR_STREAM_PREMATURE_CLOSE'
}
Are my files too big?
Hi @smoia
Good to hear we were able to resolve that one!
May you please clarify when this message is received? Was this error received during the upload process? Is there an error log associated with this (e.g. error code)
Thank you,
Franklin
Hello @franklin
This error was (repeatedly) received during the upload, that finished about three hours ago. No error log that I can see though - I looked in the dataset folder, in the folder I run the process from, and in the home folder.
The website is not showing the updated dataset yet though. I imagine it’s due to the delay that was discuss in earlier posts in this topic, so I’ll wait a couple of days to see if my files were correctly uploaded.
Cheers,
Stefano
Hi @smoia
If I may clarify - while the uploading is happening do you see the file is being uploaded but then this error message is populated, but the upload continues? I am piecing together how this message comes up.
It looks like I was able to get through a stuck commit. There is another subject there along with a session folder (I presume done through the upload directory button).
Thank you,
Franklin
Hi @franklin
Indeed, while the uploading is happening I see the file is being uploaded but then this error message is populated AND the upload continues.
Technically, this dataset should have seven subjects with ten sessions each. There should not be “free” session folders.
Here’s some ls commands from the folder I’m uploading.
~/Data/EuskalIBUR$ ls *
CHANGES dataset_description.json participants.tsv task-breathhold_bold.json
sub-001:
ses-01 ses-02 ses-03 ses-04 ses-05 ses-06 ses-07 ses-08 ses-09 ses-10
sub-002:
ses-01 ses-02 ses-03 ses-04 ses-05 ses-06 ses-07 ses-08 ses-09 ses-10
sub-003:
ses-01 ses-02 ses-03 ses-04 ses-05 ses-06 ses-07 ses-08 ses-09 ses-10
sub-004:
ses-01 ses-02 ses-03 ses-04 ses-05 ses-06 ses-07 ses-08 ses-09 ses-10
sub-007:
ses-01 ses-02 ses-03 ses-04 ses-05 ses-06 ses-07 ses-08 ses-09 ses-10
sub-008:
ses-01 ses-02 ses-03 ses-04 ses-05 ses-06 ses-07 ses-08 ses-09 ses-10
sub-009:
ses-01 ses-02 ses-03 ses-04 ses-05 ses-06 ses-07 ses-08 ses-09 ses-10
~/Data/EuskalIBUR$ ls */*
sub-001/ses-01:
anat fmap func
sub-001/ses-02:
fmap func
[...]
~/Data/EuskalIBUR$ ls sub-001/ses-01/*
sub-001/ses-01/anat:
sub-001_ses-01_acq-inv1_T1w.json sub-001_ses-01_acq-inv2_T1w.json sub-001_ses-01_acq-uni_T1w.json sub-001_ses-01_T2w.json
sub-001_ses-01_acq-inv1_T1w.nii.gz sub-001_ses-01_acq-inv2_T1w.nii.gz sub-001_ses-01_acq-uni_T1w.nii.gz sub-001_ses-01_T2w.nii.gz
sub-001/ses-01/fmap:
sub-001_ses-01_acq-breathhold_dir-AP_epi.json sub-001_ses-01_acq-breathhold_dir-AP_epi.nii.gz sub-001_ses-01_acq-breathhold_dir-PA_epi.json sub-001_ses-01_acq-breathhold_dir-PA_epi.nii.gz
sub-001/ses-01/func:
sub-001_ses-01_task-breathhold_echo-1_bold.json sub-001_ses-01_task-breathhold_echo-5_bold.json sub-001_ses-01_task-breathhold_rec-magnitude_echo-3_sbref.json
sub-001_ses-01_task-breathhold_echo-1_bold.nii.gz sub-001_ses-01_task-breathhold_echo-5_bold.nii.gz sub-001_ses-01_task-breathhold_rec-magnitude_echo-3_sbref.nii.gz
sub-001_ses-01_task-breathhold_echo-2_bold.json sub-001_ses-01_task-breathhold_physio.json sub-001_ses-01_task-breathhold_rec-magnitude_echo-4_sbref.json
sub-001_ses-01_task-breathhold_echo-2_bold.nii.gz sub-001_ses-01_task-breathhold_physio.tsv.gz sub-001_ses-01_task-breathhold_rec-magnitude_echo-4_sbref.nii.gz
sub-001_ses-01_task-breathhold_echo-3_bold.json sub-001_ses-01_task-breathhold_rec-magnitude_echo-1_sbref.json sub-001_ses-01_task-breathhold_rec-magnitude_echo-5_sbref.json
sub-001_ses-01_task-breathhold_echo-3_bold.nii.gz sub-001_ses-01_task-breathhold_rec-magnitude_echo-1_sbref.nii.gz sub-001_ses-01_task-breathhold_rec-magnitude_echo-5_sbref.nii.gz
sub-001_ses-01_task-breathhold_echo-4_bold.json sub-001_ses-01_task-breathhold_rec-magnitude_echo-2_sbref.json
sub-001_ses-01_task-breathhold_echo-4_bold.nii.gz sub-001_ses-01_task-breathhold_rec-magnitude_echo-2_sbref.nii.gz
[...]
Should I try the upload again?
Hi @smoia
Thank you for sharing the directory structure and additional information! Perhaps may you please update to our latest version of the cli and try to upload? It may be something particular to that version.
Thank you,
Franklin
Hi @franklin,
I don’t want to jinx it, but with openneuro 3.21.0 the upload seems to be working. If I encounter further issues I’ll let you know.
Just one question: the website front-end might take some time to show the updated dataset, right?