Openneuro-cli problem

Hello,
I’m afraid I’ve hit a similar issue.

bt206016@is151225:/neurospin/tmp/bthirion/ibc_release/sourcedata$ openneuro upload -i --dataset ds000244  .
Adding files to "ds000244"
(node:24898) UnhandledPromiseRejectionWarning: Error: Network error: Unexpected token < in JSON at position 0
    at new ApolloError (/home/bt206016/.nvm/versions/node/v13.7.0/lib/node_modules/openneuro-cli/node_modules/apollo-client/bundle.umd.js:92:26)
    at /home/bt206016/.nvm/versions/node/v13.7.0/lib/node_modules/openneuro-cli/node_modules/apollo-client/bundle.umd.js:1588:34
    at /home/bt206016/.nvm/versions/node/v13.7.0/lib/node_modules/openneuro-cli/node_modules/apollo-client/bundle.umd.js:2008:15
    at Set.forEach (<anonymous>)
    at /home/bt206016/.nvm/versions/node/v13.7.0/lib/node_modules/openneuro-cli/node_modules/apollo-client/bundle.umd.js:2006:26
    at Map.forEach (<anonymous>)
    at QueryManager.broadcastQueries (/home/bt206016/.nvm/versions/node/v13.7.0/lib/node_modules/openneuro-cli/node_modules/apollo-client/bundle.umd.js:2004:20)
    at /home/bt206016/.nvm/versions/node/v13.7.0/lib/node_modules/openneuro-cli/node_modules/apollo-client/bundle.umd.js:1483:29
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
(node:24898) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 2)
(node:24898) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

The problem is that I’m upload biggish data (600 GB, with files of about 1GB), so that the GUI is not a reasonable option.
What should I do ?
Best,
Bertrand

Hi @alexfoias and @bthirion,

Thank you for your messages and information regarding the errors you are experiencing! We have been discovering and actively working on solutions to address this issue. We found that in the process of doing this led to several blocking issues that were also complex. We are expanding some of our memory constraints which may help alleviate or push ahead past some of these issues as an intermediate remedy.

Alex, these issues are different than the previous issues (parser vs network like what Bertrand is seeing). The dataset does not show the information because the validator cannot serve that information. The loading may be linked to the memory issues that we are expanding.

In the intermediate when the memory expansion has been deployed I will follow up and can test.

Thank you,
Franklin

Thx for your support !
Meanwhile, is there any possibility to upload the data with a simple rsync of scp command to a server ?

Bertrand

Hi @bthirion

Unfortunately we do not - the system does a number of operations on the data while it is being input and stored. We deployed this memory expansion perhaps trying again? It may be good to track how it progresses (the upload command may need to be to rerun a few times if a network error is found)

Thank you,
Franklin

Hi @franklin,

I’m hitting the same parser issue here trying to use openneuro-cli (v3.11.2, Node.js v12.14.1) to upload more subjects to an existing dataset:

snastase$ openneuro upload --dataset ds002345 -i -n --verbose $PWD
Adding files to "ds002345"
SyntaxError [ServerParseError]: Unexpected token < in JSON at position 0
    at JSON.parse (<anonymous>)
    at /usr/local/lib/node_modules/openneuro-cli/node_modules/apollo-link-http-common/lib/index.js:35:25
    at processTicksAndRejections (internal/process/task_queues.js:94:5) {
  name: 'ServerParseError',
  response: Response {
    size: 0,
    timeout: 0,
    [Symbol(Body internals)]: { body: [PassThrough], disturbed: true, error: null },
    [Symbol(Response internals)]: {
      url: 'https://openneuro.org/crn/graphql',
      status: 504,
      statusText: 'Gateway Time-out',
      headers: [Headers],
      counter: 0
    }
  },
  statusCode: 504,
  bodyText: '<html>\r\n' +
    '<head><title>504 Gateway Time-out</title></head>\r\n' +
    '<body bgcolor="white">\r\n' +
    '<center><h1>504 Gateway Time-out</h1></center>\r\n' +
    '</body>\r\n' +
    '</html>\r\n'
}

Presumably the <anonymous> passed to JSON.parse isn’t interpreted correctly or points to HTML—no idea how to fix something like this, though. This dataset is relatively large (4k+ files, ~140 GB), but nothing extreme. I tried upping e.g. ulimit -n, but no improvement.

Hi @snastase

Thank you for sending this error over! We are working through this server parse error. To confirm - it appears this is off your local machine (thank you for providing version numbers!) Perhaps for testing and confirmation by simplifying the command a bit by removing the -n and --verbose (unless skipping nifti headers is needed?) this may give us the same server parse but worth a try

Regarding previous issues of uploading - this appears to be working it self out. If the dataset reaches the end it may throw a network error even though the dataset was successfully uploaded (tracing this one back to reduce confusion)

Thank you,
Franklin

Hi @franklin,

I made another try with openneuro 3.11.2 installed via yarn. Unfortunately the problem persists.

Here is my output:

alex-macbook:~ $ openneuro upload -i --dataset ds001919 /Volumes/sct_testing/bids/spineGeneric_multiSubjects
Adding files to "ds001919"
SyntaxError [ServerParseError]: Unexpected token < in JSON at position 0
    at JSON.parse (<anonymous>)
    at /Users/alfoi/.config/yarn/global/node_modules/apollo-link-http-common/lib/index.js:35:25
    at processTicksAndRejections (internal/process/task_queues.js:97:5) {
  name: 'ServerParseError',
  response: Response {
    size: 0,
    timeout: 0,
    [Symbol(Body internals)]: { body: [PassThrough], disturbed: true, error: null },
    [Symbol(Response internals)]: {
      url: 'https://openneuro.org/crn/graphql',
      status: 504,
      statusText: 'Gateway Time-out',
      headers: [Headers],
      counter: 0
    }
  },
  statusCode: 504,
  bodyText: '<html>\r\n' +
    '<head><title>504 Gateway Time-out</title></head>\r\n' +
    '<body bgcolor="white">\r\n' +
    '<center><h1>504 Gateway Time-out</h1></center>\r\n' +
    '</body>\r\n' +
    '</html>\r\n'
}
alex-macbook:~ $ openneuro -V
3.11.2

I retried the upload, but again the error comes up after uploading a few GBs of data, far from the whole dataset.
Rerunning the upload command several times does not help: The upload is redone from the beginning, so we never get far enough…
Best,
Bertrand

Yeah, I’m currently in the same boat as @bthirion where it doesn’t seem to recognize the existing content already on OpenNeuro (excluding the very small dataset_description.json etc files), proceeds to try to re-upload everything, then eventually breaks at some arbitrary point with the SyntaxError [ServerParseError] reported above. The first part is odd because in my past experience I’ve had the openneuro-cli (not sure what older version) smoothly recognize the existing content and only upload the new content. I’m only trying to add ~30 new subjects, but it wants to re-upload all the stimuli, all the subjects, takes forever :sweat:

In the browser, my dataset page (https://openneuro.org/datasets/ds002345/) doesn’t typically render—just hangs on a spinning wheel indefinitely. But when it occasionally does load, I’ve tried the “resume” button, but it just seems to hang (3+ hrs) with the progress bar at zero.

Hi @alexfoias, @bthirion and @snastase

Thank you for sending this error over and confirming that you are all seeing similar errors! I want to apologize for these issues you are all facing and we are working to discover and fix the root cause of these issues. As we go after the root cause this is the cause for the time taken to resolve this. Our goal is to fix these root cause issues so we can enhance OpenNeuro overall. I have documented this particular ServerParseError on our github so if there is conversation there you will see it. These messages are helpful for me to understand further and thank you for your patience with us as we continue working to resolve this.

When this error occurs the upload process may appear to start from the beginning but do you find that the upload gets further at each iteration? The bug of OpenNeuro-CLI not noticing the uploaded files may be a regression and have opened an issue to track this. When this happens are you seeing progress in each iteration?

@snastase - the additional dataset rendering is due to the server updating from the information that is being sent to the server. This will take a bit of time (variable depending on the different characteristics of your dataset) for the update to occur resulting in this hanging rendering and then the dataset being able to render.

Thank you,
Franklin

Hi @franklin,

I’m running into similar problems as documented in this thread already, and I know you are already working on a fix, but I thought perhaps additional information about experience might be helpful.

Basically, I’m getting the same ServerParseError (Unexpected toekn < in JSON at position 0). This happens right after all the “transferring” is done (below, the Transferring "sub-26/ses-2/func/sub-26_ses-2_task-self_run-2_physio.tsv.gz" - 100% complete is the “end” of my dataset).

Transferring "sub-26/ses-2/func/sub-26_ses-2_task-self_run-2_events.tsv" - 100% complete 
Transferring "sub-26/ses-2/func/sub-26_ses-2_task-self_run-2_physio.json" - 100% complete 
Transferring "sub-26/ses-2/func/sub-26_ses-2_task-self_run-2_physio.tsv.gz" - 100% complete 
{ SyntaxError [ServerParseError]: Unexpected token < in JSON at position 0
    **strong text**at JSON.parse (<anonymous>)
    at /usr/lib/node_modules/openneuro-cli/node_modules/apollo-link-http-common/lib/index.js:35:25
    at processTicksAndRejections (internal/process/task_queues.js:86:5)
  name: 'ServerParseError',
  response:
   Response {
     size: 0,
     timeout: 0,
     [Symbol(Body internals)]: { body: [PassThrough], disturbed: true, error: null },
     [Symbol(Response internals)]:
      { url: 'https://openneuro.org/crn/graphql',
        status: 504,
        statusText: 'Gateway Time-out',
        headers: [Headers],
        counter: 0 } },
**[**  statusCode: 504,
  bodyText:
   '<html>\r\n<head><title>504 Gateway Time-out</title></head>\r\n<body bgcolor="white">\r\n<center><h1>504 Gateway Time-out</h1></center>\r\n</body>\r\n</html>\r\n' }

During this process, however, it did create a new (Unnamed) dataset on openneuro (ds002545), yet without any files/information/metadata. Now, when I retry the upload (openneuro upload --dataset ds002545 -i my_bids_dir), it starts reuploading again from the start, with the exception of some small files (README, dataset_description.json, participants.json, participants.tsv; see below). Then, it crashes with the same ServerParseError again, yet this time not at the end (at sub-24 this time).

Adding files to "ds002545"                                                                                                                                                                                                                                                      
bids-validator@1.4.0                                                                                                                                                                                                                                                            
                                                                                                                                                                                                                                                                                
        Summary:                 Available Tasks:        Available Modalities:                                                                                                                                                                                                  
        621 Files, 9.78GB        other                   T1w                                                                                                                                                                                                                    
        26 - Subjects            self                    bold                                                                                                                                                                                                                   
        2 - Sessions                                     events                                                                                                                                                                                                                 
                                                         physio                                                                                                                                                                                                                 
                                                                                                                                                                                                                                                                                
                                                                                                                                                                                                                                                                                
        If you have any questions, please post on https://neurostars.org/tags/bids.                                                                                                                                                                                             
                                                                                  
Skipping existing file - "README"
Skipping existing file - "dataset_description.json"
Skipping existing file - "participants.json"
Skipping existing file - "participants.tsv"
Transferring "code/convert_logfiles.py" - 100% complete
Transferring "code/convert_phys2bids.py" - 100% complete
Transferring "derivatives/physiology/sub-01/figures/sub-01_ses-1_task-other_physio_alignment.png" - 100% complete
...
Transferring "sub-24/ses-1/func/sub-24_ses-1_task-self_run-1_bold.nii.gz" - 37% complete (52.3 MB remaining)
{ SyntaxError [ServerParseError]: Unexpected token < in JSON at position 0
    at JSON.parse (<anonymous>)
    at /usr/lib/node_modules/openneuro-cli/node_modules/apollo-link-http-common/lib/index.js:35:25
    at processTicksAndRejections (internal/process/task_queues.js:86:5)
  name: 'ServerParseError',
  response:
   Response {
     size: 0,
     timeout: 0,
     [Symbol(Body internals)]: { body: [PassThrough], disturbed: true, error: null },
     [Symbol(Response internals)]:
      { url: 'https://openneuro.org/crn/graphql',
        status: 502,
        statusText: 'Bad Gateway',
        headers: [Headers],
        counter: 0 } },
  statusCode: 502,
  bodyText:
   '<html>\r\n<head><title>502 Bad Gateway</title></head>\r\n<body bgcolor="white">\r\n<center><h1>502 Bad Gateway</h1></center>\r\n</body>\r\n</html>\r\n' }

Every time I retry uploading after the second time, It immediately crashes (after `Adding files to “ds002545”):

Adding files to "ds002545"
{ SyntaxError [ServerParseError]: Unexpected token < in JSON at position 0
    at JSON.parse (<anonymous>)
    at /usr/lib/node_modules/openneuro-cli/node_modules/apollo-link-http-common/lib/index.js:35:25
    at processTicksAndRejections (internal/process/task_queues.js:86:5)
  name: 'ServerParseError',
  response:
   Response {
     size: 0,
     timeout: 0,
     [Symbol(Body internals)]: { body: [PassThrough], disturbed: true, error: null },
     [Symbol(Response internals)]:
      { url: 'https://openneuro.org/crn/graphql',
        status: 504,
        statusText: 'Gateway Time-out',
        headers: [Headers],
        counter: 0 } },
  statusCode: 504,
  bodyText:
   '<html>\r\n<head><title>504 Gateway Time-out</title></head>\r\n<body bgcolor="white">\r\n<center><h1>504 Gateway Time-out</h1></center>\r\n</body>\r\n</html>\r\n' }

At this point, the “My dashboard” page on openneuro.org shows the (Unnamed) dataset, but when I click on it, it just shows “Loading dataset” …

Some more info: I’m using openneuro version 3.11.2 and node version v11.14.0. I don’t have any very large nifti files in my dataset (the largest file is 165 MB).

Hope this helps you in debugging this issue!
If you need more info, let me know.

Best,
Lukas

Thanks for all the hard work @franklin! Just to clarify, it does not seem in my experience that the upload gets farther each time; sometimes it gets to sub-050, next time it’ll break at sub-022, and I haven’t noticed any pattern when it breaks. Regarding the dataset rendering, as far as I can tell, my existing ds002345 does not render even if, e.g., I open it in an incognito window with no login. This is troublesome because as far as I can tell, not only does my draft fail to render (and doesn’t have my updates), but other folks also can’t access the pre-existing snapshots.

Hi @lukassnoek and @snastase

Thank you for the additional information!

@lukassnoek - in your case the dataset may have been uploaded with the error message produced from a server timeout. The dataset may take a bit of time to render if this is the case. We are looking into the separate regression seen with the uploader not noticing uploaded files. (side note: appears the render time may have been it? https://openneuro.org/datasets/ds002547/versions/1.0.0)

@snastase - regrading the dataset rendering, it is associated with server serving up the information and perhaps the server giving a timeout (on the user side shows the spinner continuing to spin). A resolution is refreshing the page or reconnecting to the wifi. We are working on enhancing load times for larger datasets.

Thank you,
Franklin

Hi Franklin,

Actually, I managed to upload this particular dataset with the web interface (resulting in ds002547). But I’d love to use the CLI tool to upload three very large (± 600 GB) datasets (AOMIC … Let me know if I can give you more information to help you debug the (existing?) issues!

Best,
Lukas

Hi @lukassnoek

Great to hear that your particular dataset is up! We patched over the bug: not skipping uploaded files. We are focusing on enhancing the upload processes. We aim at the root causes that take a bit more time to resolve. I think perhaps trying and seeing if you get the same ServerParseErrors (timeouts) and this give more cases for the development team to hold onto. (looks like a cool dataset!)

Thank you,
Franklin

Hi @franklin,

Great that you are working on resolving the CLI issues! I tried to upload another dataset to see whether the ServerParseErrors would come up again, but I got another login error this time (for which I filed an issue on Github).

Best,
Lukas

Hi @lukassnoek,

Please refer to your issue on github for the fix.

Quick update: we have patched the bug related to the cli not skipping files. We are working through the root issue of enhancing our upload processes.

Thank you,
Franklin

1 Like

Hi @franklin,

Do you have any news concerning the upload issue?

I tried again from OSX 10.14.6 (18G3020), openneuro 3.13.0 installed via yarn. Here is my terminal output: https://gist.github.com/alexfoias/c2b2ea63faa6ce47dfff9532f27928c5

thanks,
Alexandru

Hi @alexfoias

Thank you for your follow up and sending over the terminal output! We have this issue documented and investigating the root cause of this issue. We recently had internal discussions on approaches to resolve the root causes of the issues related to uploading data you and others are experiencing. We apologize for the inconveniences these have caused. This is crucial for us to resolve and has our full attention.

By chance, was this error seen at the end of your dataset?

Thank you,
Franklin

Hi @franklin,

Unfortunately, I have some bad news. We’ve managed to upload and do a snapshot using openneuro-cli 3.14.2 on 2020-05-21.

I was able to modify today (2020-06-05) the authors using the webpage before doing the upload using the CLI. After that I tried to upload more data using with the same version (3.14.2) but I got this error message:

     SyntaxError [ServerParseError]: Unexpected token < in JSON at position 0
        at JSON.parse (<anonymous>)
        at /usr/lib/node_modules/openneuro-cli/node_modules/apollo-link-http-common/lib/index.js:35:25
        at processTicksAndRejections (internal/process/task_queues.js:93:5) {
      name: 'ServerParseError',
      response: Response {
        size: 0,
        timeout: 0,
        [Symbol(Body internals)]: { body: [PassThrough], disturbed: true, error: null },
        [Symbol(Response internals)]: {
          url: 'https://openneuro.org/crn/graphql',
          status: 504,
          statusText: 'Gateway Time-out',
          headers: [Headers],
          counter: 0
        }
      },
      statusCode: 504,
      bodyText: '<html>\r\n' +
        '<head><title>504 Gateway Time-out</title></head>\r\n' +
        '<body bgcolor="white">\r\n' +
        '<center><h1>504 Gateway Time-out</h1></center>\r\n' +
        '</body>\r\n' +
        '</html>\r\n'
    }

I upgraded to openneuro 3.15.0-alpha.0 but the error message was the same. After that I tried to generate a new API key (in case the new version would require a new one), but the error message remained the same.

At this point I cannot access my datasets in my dashboard and neither on the link.

Do you have any suggestions for us ?

Thanks,
Alexandru