Openneuro-cli problem

alexfoias · September 25, 2019, 8:33pm

Hi,

I’m trying to update a dataset on openneuro using openneuro-cli.

I tried with version 3.5.5. Here is the terminal output:

 django:~ $ sudo npm install -g openneuro-cli@3.5.5
/usr/local/bin/openneuro -> /usr/local/lib/node_modules/openneuro-cli/src/index.js
+ openneuro-cli@3.5.5
removed 3 packages and updated 3 packages in 13.637s
django:~ $ openneuro upload -i --dataset ds001919 /Volumes/sct_testing/bids/spineGeneric_multiSubjects
Adding files to "ds001919"
(node:86769) UnhandledPromiseRejectionWarning: Error: Network error: Unexpected token < in JSON at position 0
    at new ApolloError (/usr/local/lib/node_modules/openneuro-cli/node_modules/apollo-client/bundle.umd.js:92:26)
    at /usr/local/lib/node_modules/openneuro-cli/node_modules/apollo-client/bundle.umd.js:1587:34
    at /usr/local/lib/node_modules/openneuro-cli/node_modules/apollo-client/bundle.umd.js:2007:15
    at Set.forEach (<anonymous>)
    at /usr/local/lib/node_modules/openneuro-cli/node_modules/apollo-client/bundle.umd.js:2005:26
    at Map.forEach (<anonymous>)
    at QueryManager.broadcastQueries (/usr/local/lib/node_modules/openneuro-cli/node_modules/apollo-client/bundle.umd.js:2003:20)
    at /usr/local/lib/node_modules/openneuro-cli/node_modules/apollo-client/bundle.umd.js:1482:29
    at process._tickCallback (internal/process/next_tick.js:68:7)
(node:86769) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 2)
(node:86769) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

I also tried 3.6.0-rc.7. Here is the terminal output:

django:~ $ sudo npm install -g openneuro-cli
/usr/local/bin/openneuro -> /usr/local/lib/node_modules/openneuro-cli/src/index.js
+ openneuro-cli@3.6.0-rc.7
added 3 packages from 8 contributors and updated 3 packages in 12.185s
django:~ $ openneuro upload -i --dataset ds001919 /Volumes/sct_testing/bids/spineGeneric_multiSubjects
Adding files to "ds001919"
(node:86805) UnhandledPromiseRejectionWarning: Error: Network error: Unexpected token < in JSON at position 0
   at new ApolloError (/usr/local/lib/node_modules/openneuro-cli/node_modules/apollo-client/bundle.umd.js:92:26)
   at /usr/local/lib/node_modules/openneuro-cli/node_modules/apollo-client/bundle.umd.js:1587:34
   at /usr/local/lib/node_modules/openneuro-cli/node_modules/apollo-client/bundle.umd.js:2007:15
   at Set.forEach (<anonymous>)
   at /usr/local/lib/node_modules/openneuro-cli/node_modules/apollo-client/bundle.umd.js:2005:26
   at Map.forEach (<anonymous>)
   at QueryManager.broadcastQueries (/usr/local/lib/node_modules/openneuro-cli/node_modules/apollo-client/bundle.umd.js:2003:20)
   at /usr/local/lib/node_modules/openneuro-cli/node_modules/apollo-client/bundle.umd.js:1482:29
   at process._tickCallback (internal/process/next_tick.js:68:7)
(node:86805) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 2)
(node:86805) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

I don’t think is a firewall problem, because I was able to upload data using the cli from the same computer.

I checked the .openneuro contents and the api key seems to be correct.

Thank you for your help,
Alexandru

franklin · September 25, 2019, 10:20pm

Hi @alexfoias

Thank you for your message! To clarify - you have an API key added at login (it may be good to regenerate to confirm this is not the issue, we just released a new version this afternoon). I am assuming that you have access to the email that is the administrator over the dataset.

Thank you,
Franklin

alexfoias · September 26, 2019, 3:14pm

Hi @franklin,

Yes I have access to the email that is the administrator.
I reconfigured the login with the new API key.

I updated npm and openneuro.

This time it seems to upload the data, but it still gives a timeout. On the openneuro page it’s mentioned Incomplete Upload or Edit. Could it be that the dataset generates warnings or errors while running the bids-validator ?

Here is my terminal output:

gist.github.com

https://gist.github.com/alexfoias/32fbf4496a14f6267630e3adc94378f4

openneuro


(base) alex-macbook:~ $ openneuro -V
3.6.0
(base) alex-macbook:~ $ npm -v
6.9.0
(base) alex-macbook:~ $ openneuro upload -i --dataset ds001919 /Volumes/sct_testing/bids/spineGeneric_multiSubjects
Adding files to "ds001919"
bids-validator@1.3.6

(node:6019) ExperimentalWarning: The fs.promises API is experimental

This file has been truncated. show original

Thank you for you help,
Alexandru

franklin · September 27, 2019, 12:04am

Hi @alexfoias

Thank you for the CLI output! It appears that the first issue has been resolved and now we are trying to upload new data onto this dataset? The permissions are correct and the validator passes (the validator would error out before checking for existing files).

By chance are the files you are uploading larger than the typical Nifti files? We have had loading limits that this may be hitting. Interesting that the new data is not appearing after the existing data has been established

I am also going to open an issue at OpenNeuro to flag it there too as this issue.

Thank you,
Franklin

alexfoias · September 30, 2019, 6:45pm

Hi @franklin,

Yes, I’m trying to upload more data to our dataset.
Some nifti files are ~120Mb. Is there a way to bypass the file size limitation while uploading ?

Thanks,
Alexandru

franklin · October 1, 2019, 12:10am

Hi @alexfoias

Thank you for your message! Those should be okay.

Have you tried the web browser too? The browser is good too for updating datasets (performed a few last week and were successful)

The issue is open so will raise to the development team to discover the error

Thank you,
Franklin

alexfoias · October 1, 2019, 2:14pm

Hi @franklin,

I tried this morning the web browser (chrome & firefox). First I tried to update a file. It shows that it’s uploading the file, but when I download it, it grabs the old version.
In addition I tried to upload new data. As previously, it shows it’s uploading, but the data doesn’t show up.
Is it normal that on the draft it’s indicated Incomplete upload or edit ?

Thanks,
Alexandru

franklin · October 1, 2019, 8:39pm

Hi @alexfoias

This appears to be a delay between the server updating and rendering on your side. I have found that our rendering has been slightly delayed in a few other instances. So these updates may have been uptaken, but have not shown up yet (it took a day or two for others to appear). In this attempt did it complete the upload or have the same error previously shown?

The message seen on the draft page is from an unfinished attempt, this will cleared on a successful upload or edit (potentially from the attempts you did this morning that are uptaking to render)

I apologize for this inconvenience, thank you for your patience as we work through this!

Thank you,
Franklin

alexfoias · October 24, 2019, 6:06pm

Hi @franklin,

Do you have any updates concerning our issue ?

I retried today using cli 3.7.0 but the problem seems to persist. I also tried the website upload but it seems like the data is not updated.

Is this issue related to our dataset or is it a common issue ? Is a recommended web browser/configuration required ?

thanks,
Alexandru

franklin · October 24, 2019, 11:46pm

Hi @alexfoias

Thank you for your message! We have deployed a several fixes over the past few weeks that may resolve these issues!

Regarding the website website upload - did the upload complete with the last step having the files rendered? If so, this leads me to believe that the server is uptaking your uploaded data and preparing to render back.

There are a few datasets are in these errors and am diagnosing what potential is happening and collecting information. We recommend the latest versions of your browser - we are using advancing technology that becomes challenging on older browsers.

Thank you,
Franklin

alexfoias · October 29, 2019, 5:48pm

The site didn’t manage to render the files. I tried with the latest version of Chrome & Firefox.

franklin · October 29, 2019, 10:55pm

Hi @alexfoias

Thank you for checking! I am assuming it is the same error messages you have seen? These messages will help us diagnose and deploy fixes for your dataset (and OpenNeuro!) I’m sorry to hear you are running into these issues!

Thank you,
Franklin

franklin · November 5, 2019, 9:58pm

Hi @alexfoias,

Following up - may you please check what version of node.js you are using? This will help continue tracking down what may be the cause of the issues you are experiencing.

Thank you,
Franklin

alexfoias · November 6, 2019, 2:15pm

Hi @franklin,

I’m running node.js v10.16.3 .

franklin · November 7, 2019, 1:22am

Hi @alexfoias

Thank you for the additional information! We are working to track this down. This issue has become much more complex as we discover the root causes for the issues you are experiencing.

Thank you,
Franklin

alexfoias · December 9, 2019, 3:17pm

Hi @franklin,

Do you have any updates concerning the issue ?

I tried the web insterface but I don’t arrive to load the dataset at the address: https://openneuro.org/datasets/ds001919 . Here is the error code : Sentry ID: 46255553657541899820459415564ccd — TypeError: Cannot read property ‘dataset’ of undefined

I tried again with openneuro-cli 3.8.0 and npm 6.12.1. I also regenerated the API key from openeneuro.org.

Here is my error message:

(node:99349) UnhandledPromiseRejectionWarning: Error: Network error: Unexpected token < in JSON at position 0
    at new ApolloError (/usr/local/lib/node_modules/openneuro-cli/node_modules/apollo-client/bundle.umd.js:92:26)
    at Object.error (/usr/local/lib/node_modules/openneuro-cli/node_modules/apollo-client/bundle.umd.js:1329:28)
    at notifySubscription (/usr/local/lib/node_modules/openneuro-cli/node_modules/zen-observable/lib/Observable.js:140:18)
    at onNotify (/usr/local/lib/node_modules/openneuro-cli/node_modules/zen-observable/lib/Observable.js:179:3)
    at SubscriptionObserver.error (/usr/local/lib/node_modules/openneuro-cli/node_modules/zen-observable/lib/Observable.js:240:7)
    at /usr/local/lib/node_modules/openneuro-cli/node_modules/apollo-client/bundle.umd.js:1106:37
    at Set.forEach (<anonymous>)
    at Object.error (/usr/local/lib/node_modules/openneuro-cli/node_modules/apollo-client/bundle.umd.js:1105:21)
    at notifySubscription (/usr/local/lib/node_modules/openneuro-cli/node_modules/zen-observable/lib/Observable.js:140:18)
    at onNotify (/usr/local/lib/node_modules/openneuro-cli/node_modules/zen-observable/lib/Observable.js:179:3)
(node:99349) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 2)
(node:99349) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

Thanks for your help

franklin · December 9, 2019, 9:45pm

Hi @alexfoias

Thank you for following up and information! We are actively working on developing a solution for these issues you are running into! We apologize for the inconvenience while we are developing the solution. This issue and solution has gotten complex.

The TypeError you are seeing is regarding loading/rendering the dataset from the server. I have found waiting a few minutes and refreshing the page will resolve this (more likely to occur if you tried to update right before accessing)

Thank you,
Franklin

alexfoias · December 18, 2019, 7:24pm

Hi,

I tried with the latest version. Output here: https://gist.github.com/alexfoias/003ad6610cddb04d7802cd0ca54e6f41

It seems like the problem persists.

Is there any other way to force the upload ?

thanks,
Alexandru

franklin · January 9, 2020, 3:43am

Hi @alexfoias

Thank you for sharing the output! I apologize for the delayed response due to the holidays. I have this documented and continue to dive more into this issue. Thank you for your patience as we work through this complex issue!

Thank you,
Franklin

alexfoias · January 30, 2020, 2:30pm

Hi @franklin,

I tried the latest release of openneuro-cli, but I still have an error message.

alex-macbook:~ $ openneuro upload -i --dataset ds001919 /Volumes/sct_testing/bids/spineGeneric_multiSubjects
Adding files to "ds001919"
SyntaxError [ServerParseError]: Unexpected token < in JSON at position 0
    at JSON.parse (<anonymous>)
    at /Users/alfoi/.config/yarn/global/node_modules/apollo-link-http-common/lib/index.js:35:25
    at processTicksAndRejections (internal/process/task_queues.js:97:5) {
  name: 'ServerParseError',
  response: Response {
    size: 0,
    timeout: 0,
    [Symbol(Body internals)]: { body: [PassThrough], disturbed: true, error: null },
    [Symbol(Response internals)]: {
      url: 'https://openneuro.org/crn/graphql',
      status: 504,
      statusText: 'Gateway Time-out',
      headers: [Headers],
      counter: 0
    }
  },
  statusCode: 504,
  bodyText: '<html>\r\n' +
    '<head><title>504 Gateway Time-out</title></head>\r\n' +
    '<body bgcolor="white">\r\n' +
    '<center><h1>504 Gateway Time-out</h1></center>\r\n' +
    '</body>\r\n' +
    '</html>\r\n'
}

In addition, in my dashboard on the website, the dataset doesn’t seem to show all the info.

Also, when you try to access the page https://openneuro.org/datasets/ds001919, it just says loading dataset.
06%20AM

Thanks.