Openneuro-cli problem

Hi,

I’m trying to update a dataset on openneuro using openneuro-cli.

I tried with version 3.5.5. Here is the terminal output:

 django:~ $ sudo npm install -g openneuro-cli@3.5.5
/usr/local/bin/openneuro -> /usr/local/lib/node_modules/openneuro-cli/src/index.js
+ openneuro-cli@3.5.5
removed 3 packages and updated 3 packages in 13.637s
django:~ $ openneuro upload -i --dataset ds001919 /Volumes/sct_testing/bids/spineGeneric_multiSubjects
Adding files to "ds001919"
(node:86769) UnhandledPromiseRejectionWarning: Error: Network error: Unexpected token < in JSON at position 0
    at new ApolloError (/usr/local/lib/node_modules/openneuro-cli/node_modules/apollo-client/bundle.umd.js:92:26)
    at /usr/local/lib/node_modules/openneuro-cli/node_modules/apollo-client/bundle.umd.js:1587:34
    at /usr/local/lib/node_modules/openneuro-cli/node_modules/apollo-client/bundle.umd.js:2007:15
    at Set.forEach (<anonymous>)
    at /usr/local/lib/node_modules/openneuro-cli/node_modules/apollo-client/bundle.umd.js:2005:26
    at Map.forEach (<anonymous>)
    at QueryManager.broadcastQueries (/usr/local/lib/node_modules/openneuro-cli/node_modules/apollo-client/bundle.umd.js:2003:20)
    at /usr/local/lib/node_modules/openneuro-cli/node_modules/apollo-client/bundle.umd.js:1482:29
    at process._tickCallback (internal/process/next_tick.js:68:7)
(node:86769) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 2)
(node:86769) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

I also tried 3.6.0-rc.7. Here is the terminal output:

django:~ $ sudo npm install -g openneuro-cli
/usr/local/bin/openneuro -> /usr/local/lib/node_modules/openneuro-cli/src/index.js
+ openneuro-cli@3.6.0-rc.7
added 3 packages from 8 contributors and updated 3 packages in 12.185s
django:~ $ openneuro upload -i --dataset ds001919 /Volumes/sct_testing/bids/spineGeneric_multiSubjects
Adding files to "ds001919"
(node:86805) UnhandledPromiseRejectionWarning: Error: Network error: Unexpected token < in JSON at position 0
   at new ApolloError (/usr/local/lib/node_modules/openneuro-cli/node_modules/apollo-client/bundle.umd.js:92:26)
   at /usr/local/lib/node_modules/openneuro-cli/node_modules/apollo-client/bundle.umd.js:1587:34
   at /usr/local/lib/node_modules/openneuro-cli/node_modules/apollo-client/bundle.umd.js:2007:15
   at Set.forEach (<anonymous>)
   at /usr/local/lib/node_modules/openneuro-cli/node_modules/apollo-client/bundle.umd.js:2005:26
   at Map.forEach (<anonymous>)
   at QueryManager.broadcastQueries (/usr/local/lib/node_modules/openneuro-cli/node_modules/apollo-client/bundle.umd.js:2003:20)
   at /usr/local/lib/node_modules/openneuro-cli/node_modules/apollo-client/bundle.umd.js:1482:29
   at process._tickCallback (internal/process/next_tick.js:68:7)
(node:86805) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 2)
(node:86805) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

I don’t think is a firewall problem, because I was able to upload data using the cli from the same computer.

I checked the .openneuro contents and the api key seems to be correct.

Thank you for your help,
Alexandru

Hi @alexfoias

Thank you for your message! To clarify - you have an API key added at login (it may be good to regenerate to confirm this is not the issue, we just released a new version this afternoon). I am assuming that you have access to the email that is the administrator over the dataset.

Thank you,
Franklin

Hi @franklin,

Yes I have access to the email that is the administrator.
I reconfigured the login with the new API key.

I updated npm and openneuro.

This time it seems to upload the data, but it still gives a timeout. On the openneuro page it’s mentioned Incomplete Upload or Edit. Could it be that the dataset generates warnings or errors while running the bids-validator ?

Here is my terminal output:

Thank you for you help,
Alexandru

Hi @alexfoias

Thank you for the CLI output! It appears that the first issue has been resolved and now we are trying to upload new data onto this dataset? The permissions are correct and the validator passes (the validator would error out before checking for existing files).

By chance are the files you are uploading larger than the typical Nifti files? We have had loading limits that this may be hitting. Interesting that the new data is not appearing after the existing data has been established

I am also going to open an issue at OpenNeuro to flag it there too as this issue.

Thank you,
Franklin

Hi @franklin,

Yes, I’m trying to upload more data to our dataset.
Some nifti files are ~120Mb. Is there a way to bypass the file size limitation while uploading ?

Thanks,
Alexandru

Hi @alexfoias

Thank you for your message! Those should be okay.

Have you tried the web browser too? The browser is good too for updating datasets (performed a few last week and were successful)

The issue is open so will raise to the development team to discover the error

Thank you,
Franklin

Hi @franklin,

I tried this morning the web browser (chrome & firefox). First I tried to update a file. It shows that it’s uploading the file, but when I download it, it grabs the old version.
In addition I tried to upload new data. As previously, it shows it’s uploading, but the data doesn’t show up.
Is it normal that on the draft it’s indicated Incomplete upload or edit ?

Thanks,
Alexandru

Hi @alexfoias

This appears to be a delay between the server updating and rendering on your side. I have found that our rendering has been slightly delayed in a few other instances. So these updates may have been uptaken, but have not shown up yet (it took a day or two for others to appear). In this attempt did it complete the upload or have the same error previously shown?

The message seen on the draft page is from an unfinished attempt, this will cleared on a successful upload or edit (potentially from the attempts you did this morning that are uptaking to render)

I apologize for this inconvenience, thank you for your patience as we work through this!

Thank you,
Franklin

Hi @franklin,

Do you have any updates concerning our issue ?

I retried today using cli 3.7.0 but the problem seems to persist. I also tried the website upload but it seems like the data is not updated.

Is this issue related to our dataset or is it a common issue ? Is a recommended web browser/configuration required ?

thanks,
Alexandru

Hi @alexfoias

Thank you for your message! We have deployed a several fixes over the past few weeks that may resolve these issues!

Regarding the website website upload - did the upload complete with the last step having the files rendered? If so, this leads me to believe that the server is uptaking your uploaded data and preparing to render back.

There are a few datasets are in these errors and am diagnosing what potential is happening and collecting information. We recommend the latest versions of your browser - we are using advancing technology that becomes challenging on older browsers.

Thank you,
Franklin

The site didn’t manage to render the files. I tried with the latest version of Chrome & Firefox.

Hi @alexfoias

Thank you for checking! I am assuming it is the same error messages you have seen? These messages will help us diagnose and deploy fixes for your dataset (and OpenNeuro!) I’m sorry to hear you are running into these issues!

Thank you,
Franklin

Hi @alexfoias,

Following up - may you please check what version of node.js you are using? This will help continue tracking down what may be the cause of the issues you are experiencing.

Thank you,
Franklin

Hi @franklin,

I’m running node.js v10.16.3 .

Hi @alexfoias

Thank you for the additional information! We are working to track this down. This issue has become much more complex as we discover the root causes for the issues you are experiencing.

Thank you,
Franklin

MANAGED BY INCF