Hello @franklin!
Unluckily, I still have issues with the dataset and the CLI. This time, it uploaded one subject out of five.
Anything else I can do to try and get my dataset online?
Should I try to delete all the files from the dataset, or make a new one?
Hi @smoia
Darn - thank you for testing! We have been addressing related cli issues - may you please try this suggestion?
Another approach can also be to try to upload as a new dataset and evaluate how the cli responds.
Thank you,
Franklin
Hello Franklin,
I tried again with the upload following the suggestions you linked.
It seems to have worked (Who-hooooo!), but I also got this message:
./node_modules/.bin/openneuro upload -d ds003192 -i -v ~/Data/EuskalIBUR
Adding files to "ds003192"
bids-validator@1.5.7
Summary: Available Tasks: Available Modalities:
1880 Files, 74.66GB Breath-holds T2w
7 - Subjects T1w
10 - Sessions bold
physio
sbref
fieldmap
If you have any questions, please post on https://neurostars.org/tags/bids.
=======================================================================
1879 files to be uploaded with a total size of 74.7 GB
? Begin upload? Yes
=======================================================================
Starting a new upload (439373bc) to dataset: 'ds003192'
ds003192 [========================================] 100% | ETA: 0s | 1879/1879
(node:2297945) UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'end' of null
at uploadDataset (/home/nemo/Data/openneuro/node_modules/openneuro-cli/src/actions.js:95:49)
at runMicrotasks (<anonymous>)
at processTicksAndRejections (internal/process/task_queues.js:93:5)
(Use `node --trace-warnings ...` to show where the warning was created)
(node:2297945) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
(node:2297945) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
Thank you for your help!
Cheers,
Stefano
Hi @smoia
Thank you for your message and sending over the error! Perhaps one thing to clarify - the data appear to have been uploaded successfully? This error may be due to a readout message rather than an issue with the upload.
Thank you,
Franklin
Hi @franklin,
The data appear to be uploaded. I didn’t try to download the dataset, but at least the validator is passing - and that’s a first. All subjects and sessions seems to be there, and the size of the dataset is similar to my local copy.
For future reference, I deleted all files and folders from the uploaded dataset (through the web interface), followed the suggestion you indicated earlier, and re-uploaded the dataset after being sure I wasn’t having any error running the bids validator.
Thank you again for your help!
Hello,
Thank you for your work on Openneuro-cli! I’ve previously been able to use the command to update my dataset, but recently I got an error after passing bids-validation (which I can’t reproduce). Therefore, I tried to update some of the relevant software and tried to follow suggestions in this and other threads, but I’m still unable to upload anything. I’m unsure where the problem is, and how to update properly. I pasted my error trail below. Please let me know if you have any suggestions of how to get things working again and if you need any further information that would help to pin down the problem.
Many thanks,
Nicole
$ node -v
v14.15.1
$ npm -v
6.14.9
# uninstall openneuro-cli to start from scratch
$ sudo npm uninstall -g openneuro-cli
removed 926 packages in 8.676s
# check that uninstalled
$ openneuro -V
-bash: /usr/local/bin/openneuro: No such file or directory
# attempt to re-install
$ sudo npm install -g openneuro-cli
npm WARN deprecated resolve-url@0.2.1: https://github.com/lydell/resolve-url#deprecated
npm WARN deprecated urix@0.1.0: Please see https://github.com/lydell/urix#deprecated
npm WARN deprecated request@2.88.2: request has been deprecated, see https://github.com/request/request/issues/3142
npm WARN deprecated har-validator@5.1.5: this library is no longer supported
npm WARN deprecated mkdirp-promise@5.0.1: This package is broken and no longer maintained. 'mkdirp' itself supports promises now, please switch to that.
/usr/local/bin/openneuro -> /usr/local/lib/node_modules/openneuro-cli/src/index.js
npm WARN @octokit/plugin-request-log@1.0.2 requires a peer of @octokit/core@>=3 but none is installed. You must install peer dependencies yourself.
+ openneuro-cli@3.25.3
added 1009 packages from 552 contributors and updated 4 packages in 42.846s
# seems like something has gone wrong
$ openneuro -V
/usr/local/lib/node_modules/openneuro-cli/src/cli.js:1
#!/usr/bin/env node
SyntaxError: Invalid or unexpected token
at new Script (vm.js:100:7)Hello again,
just an update from my side. I manually uninstalled openneuro-cli by deleting the folder /usr/local/lib/node_modules/openneuro-cli/ rather than using the npm command. Re-installation via npm was then successful. However, now I run again into my initial upload error. I’ve pasted the error trail below. Please let me know about any suggestions of how to fix this problem. I can open a separate issue, if my post is not suitable in this thread.
Many thanks,
Nicole
$ node -v
v14.15.3
$ npm -v
6.14.9
$ openneuro -V
3.26.1
$ openneuro upload --dataset ds002634 -i .
Adding files to "ds002634"
(node:65785) UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'addLabels' of undefined
at uploadDataset (/usr/local/lib/node_modules/openneuro-cli/src/actions.js:65:18)
at Command.upload (/usr/local/lib/node_modules/openneuro-cli/src/actions.js:169:7)
at Proxy.listener (/usr/local/lib/node_modules/openneuro-cli/node_modules/commander/index.js:315:8)
at Proxy.emit (events.js:315:20)
at Proxy.EventEmitter.emit (domain.js:467:12)
at Proxy.Command.parseArgs (/usr/local/lib/node_modules/openneuro-cli/node_modules/commander/index.js:651:12)
at Proxy.Command.parse (/usr/local/lib/node_modules/openneuro-cli/node_modules/commander/index.js:474:21)
at Object.<anonymous> (/usr/local/lib/node_modules/openneuro-cli/src/cli.js:66:11)
at Generator.next (<anonymous>)
at bl (/usr/local/lib/node_modules/openneuro-cli/node_modules/esm/esm.js:1:245412)
(Use `node --trace-warnings ...` to show where the warning was created)
(node:65785) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 2)
(node:65785) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.hi @Nicole
Thank you for your message. The versions all look good. Perhaps can you try to upload a test dataset to ensure the cli is uploading properly? The test can be removed after this has been performed. I presume you used the same sudo npm install -g openneuro-cli command on the retry?
Also, can you please share the directory you intended to upload? (checking the directory level the upload originated at)
Thank you,
Franklin
Dear Franklin,
thank you for your reply. In the meantime I have updated the dataset manually on the OpenNeuro website, but let’s find out what is going wrong with the cli tool.
See below the info about the directory I’m intending to upload. Tool version numbers are the same as posted above. The same error occurs when I ‘cd’ into the directory and use ‘.’ for the dataset argument, as posted above.
$ ls project_larynx/
README sub-07 sub-16
dataset_description.json sub-08 sub-17
derivatives sub-09 sub-18
sub-01 sub-10 sub-19
sub-02 sub-11 sub-20
sub-03 sub-12 task-ArtVoc_events.json
sub-04 sub-13 task-Factorial_events.json
sub-05 sub-14
sub-06 sub-15
$ openneuro upload --dataset ds002634 -i project_larynx
Adding files to "ds002634"
(node:21295) UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'addLabels' of undefined
at uploadDataset (/usr/local/lib/node_modules/openneuro-cli/src/actions.js:65:18)
at Command.upload (/usr/local/lib/node_modules/openneuro-cli/src/actions.js:169:7)
at Proxy.listener (/usr/local/lib/node_modules/openneuro-cli/node_modules/commander/index.js:315:8)
at Proxy.emit (events.js:315:20)
at Proxy.EventEmitter.emit (domain.js:467:12)
at Proxy.Command.parseArgs (/usr/local/lib/node_modules/openneuro-cli/node_modules/commander/index.js:651:12)
at Proxy.Command.parse (/usr/local/lib/node_modules/openneuro-cli/node_modules/commander/index.js:474:21)
at Object.<anonymous> (/usr/local/lib/node_modules/openneuro-cli/src/cli.js:66:11)
at Generator.next (<anonymous>)
at bl (/usr/local/lib/node_modules/openneuro-cli/node_modules/esm/esm.js:1:245412)
(Use `node --trace-warnings ...` to show where the warning was created)
(node:21295) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 2)
(node:21295) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
Here is what happens when I try to upload a test dataset (tried with and without sudo):
$ mkdir test_upload
$ cd test_upload/
$ touch test.nii.gz
$ ls
test.nii.gz
$ sudo openneuro upload .
Password:
? **This will create a new dataset, continue?** Yes
TypeError: Cannot read property 'addLabels' of undefined
at uploadDataset (/usr/local/lib/node_modules/openneuro-cli/src/actions.js:65:18)
at /usr/local/lib/node_modules/openneuro-cli/src/actions.js:184:20
at processTicksAndRejections (internal/process/task_queues.js:93:5)
"." may not exist or is inaccessible
Please let me know if you require any further info from my side for trouble-shooting.
Best,
Nicole
Hi @Nicole
Thank you for sharing this information! Perhaps can you try to upload a subset of this dataset (say a subject or two) as a new dataset using a relative path rather than . to evaluate the upload process? This will help drill down why this error is popping up. I presume logging in was successful?
Thank you,
Franklin
Hello,
I appear to be having a similar issue to Nicole above. After having gone through the discussion, I haven’t been able to get around the issue.
I have attempted to upload data using the web interface and openneuro-cli to no avail.
When using the web interface, my data passed the bids validator step and started to upload. However after > 16 hours, it was still uploading, with several messages about failed attempts to upload files. I suspected it had to do with the size of the data, so instead tried with 10 participants and it was successful. However, it seems I can only upload one additional directory at a time on the web interface (but please correct me if I’m wrong!), which is not ideal.
I decided to stop and switch over to CLI upload and landed here.
Dataset description: ~300 participants, 2 rsfmri runs (multiecho data, so 6 nii/participant), T1, and some FLAIR.
node -v
v14.16.0
npm -v
6.14.11
./node_modules/.bin/openneuro --version
3.31.0
I have tried uploading the entire dataset (terminal output below) as well as one additional subject, but neither work.
~/openneurocli_dir$ ./node_modules/.bin/openneuro upload -d ds003588 -i /home/lbcspreng/data/openneuro/derivatives/
Adding files to "ds003588"
(node:5204) UnhandledPromiseRejectionWarning: TypeError: Cannot destructure property 'affirmedDefaced' of 'undefined' as it is undefined.
at uploadDataset (/home/lbcspreng/openneurocli_dir/node_modules/openneuro-cli/src/actions.js:59:5)
at Command.upload (/home/lbcspreng/openneurocli_dir/node_modules/openneuro-cli/src/actions.js:169:7)
at Proxy.listener (/home/lbcspreng/openneurocli_dir/node_modules/commander/index.js:315:8)
at Proxy.emit (events.js:315:20)
at Proxy.EventEmitter.emit (domain.js:467:12)
at Proxy.Command.parseArgs (/home/lbcspreng/openneurocli_dir/node_modules/commander/index.js:651:12)
at Proxy.Command.parse (/home/lbcspreng/openneurocli_dir/node_modules/commander/index.js:474:21)
at Object.<anonymous> (/home/lbcspreng/openneurocli_dir/node_modules/openneuro-cli/src/cli.js:84:11)
at Generator.next (<anonymous>)
at bl (/home/lbcspreng/openneurocli_dir/node_modules/esm/esm.js:1:245412)
at kl (/home/lbcspreng/openneurocli_dir/node_modules/esm/esm.js:1:247659)
at Object.u (/home/lbcspreng/openneurocli_dir/node_modules/esm/esm.js:1:287740)
at Object.o (/home/lbcspreng/openneurocli_dir/node_modules/esm/esm.js:1:287137)
at Object.<anonymous> (/home/lbcspreng/openneurocli_dir/node_modules/esm/esm.js:1:284879)
at Object.apply (/home/lbcspreng/openneurocli_dir/node_modules/esm/esm.js:1:199341)
at /home/lbcspreng/openneurocli_dir/node_modules/esm/esm.js:1:231006
at /home/lbcspreng/openneurocli_dir/node_modules/esm/esm.js:1:231041
at /home/lbcspreng/openneurocli_dir/node_modules/esm/esm.js:1:231254
at Su (/home/lbcspreng/openneurocli_dir/node_modules/esm/esm.js:1:228587)
at ku (/home/lbcspreng/openneurocli_dir/node_modules/esm/esm.js:1:230341)
at _c (/home/lbcspreng/openneurocli_dir/node_modules/esm/esm.js:1:258239)
at Module.<anonymous> (/home/lbcspreng/openneurocli_dir/node_modules/esm/esm.js:1:296060)
at n (/home/lbcspreng/openneurocli_dir/node_modules/esm/esm.js:1:279589)
at Object.<anonymous> (/home/lbcspreng/openneurocli_dir/node_modules/openneuro-cli/src/index.js:5:18)
at Module._compile (internal/modules/cjs/loader.js:1063:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:1092:10)
at Module.load (internal/modules/cjs/loader.js:928:32)
at Function.Module._load (internal/modules/cjs/loader.js:769:14)
at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:72:12)
at internal/main/run_main_module.js:17:47
(Use `node --trace-warnings ...` to show where the warning was created)
(node:5204) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 2)
(node:5204) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
Any help would be greatly appreciated!
Thanks,
Roni
hi @rsetton
thank you for your message and sharing details of your configurations. May you please try upgrading npm to version 7 and reuploading? This’ll help us further determine what could be causing this. The cli may be the more reliable option for uploading larger datasets like yours.
thank you,
Franklin
Hi Franklin,
Thanks for the speedy response! Upgraded to npm v7.7.5, but output is the same.
Roni
hi @rsetton
thank you for your message and trying that approach - perhaps to check a other couple things:
- can you try outputting the help page to double-check the tool is configured properly:
openneuro -help - perhaps resetting your login token could do the trick:
openneuro login
this will help us further identify what could be the reason
thank you,
Franklin
Franklin,
openneuro -help outputs normally. I have also reset the login token.
It is worth mentioning that when I install openneuro-cli (as recommended here), I get:
pm WARN deprecated mkdirp-promise@5.0.1: This package is broken and no longer maintained. 'mkdirp' itself supports promises now, please switch to that.
npm WARN deprecated urix@0.1.0: Please see https://github.com/lydell/urix#deprecated
npm WARN deprecated har-validator@5.1.5: this library is no longer supported
npm WARN deprecated resolve-url@0.2.1: https://github.com/lydell/resolve-url#deprecated
npm WARN deprecated request@2.88.2: request has been deprecated, see https://github.com/request/request/issues/3142
added 1073 packages, and audited 1074 packages in 35s
46 packages are looking for funding
run `npm fund` for details
6 low severity vulnerabilities
To address all issues, run:
npm audit fix
Run `npm audit` for details.
I then run npm audit fix and get the following:
up to date, audited 1074 packages in 5s
46 packages are looking for funding
run `npm fund` for details
# npm audit report
node-fetch <=2.6.0 || 3.0.0-beta.1 - 3.0.0-beta.8
Denial of Service - https://npmjs.com/advisories/1556
fix available via `npm audit fix --force`
Will install openneuro-cli@2.4.3, which is a breaking change
node_modules/bids-validator/node_modules/node-fetch
cross-fetch <=3.0.5
Depends on vulnerable versions of node-fetch
node_modules/bids-validator/node_modules/cross-fetch
bids-validator >=0.21.0
Depends on vulnerable versions of cross-fetch
Depends on vulnerable versions of yargs
node_modules/bids-validator
openneuro-cli >=2.5.3
Depends on vulnerable versions of bids-validator
node_modules/openneuro-cli
yargs-parser <=13.1.1 || 14.0.0 - 15.0.0 || 16.0.0 - 18.1.1
Prototype Pollution - https://npmjs.com/advisories/1500
fix available via `npm audit fix --force`
Will install openneuro-cli@2.4.3, which is a breaking change
node_modules/yargs/node_modules/yargs-parser
yargs 4.0.0-alpha1 - 12.0.5 || 14.1.0 || 15.0.0 - 15.2.0
Depends on vulnerable versions of yargs-parser
node_modules/yargs
bids-validator >=0.21.0
Depends on vulnerable versions of cross-fetch
Depends on vulnerable versions of yargs
node_modules/bids-validator
openneuro-cli >=2.5.3
Depends on vulnerable versions of bids-validator
node_modules/openneuro-cli
6 low severity vulnerabilities
To address all issues (including breaking changes), run:
npm audit fix --force
Might this be contributing to the problem?
Roni
hi @rsetton
thank you for the additional information! This appears to be potentially a bit deeper-rooted - I have surfaced this on our GitHub page to get more thoughts on what may be happening: openneuro-cli: TypeError: Cannot destructure property 'affirmedDefaced' of 'undefined' as it is undefined. · Issue #2023 · OpenNeuroOrg/openneuro · GitHub . Perhaps if there may be more information I did not include may add it over there.
thank you,
Franklin
Got it. Thank you!
A small update: it appears to have started working when I uploaded as a new data set instead of trying to update an existing one, albeit with warnings for every file that looked like this:
}
[Error: EMFILE: too many open files, open '/home/lbcspreng/data/openneuro/derivatives/sub-96/ses-1/anat/sub-96_ses-1_T1w.nii.gz'] {
errno: -24,
code: 'EMFILE',
syscall: 'open',
path: '/home/lbcspreng/data/openneuro/derivatives/sub-96/ses-1/anat/sub-96_ses-1_T1w.nii.gz'
}
But, it was working! It got to ~29% and then overnight, it seems like the session timed out, with the terminal being populated with:
Error [ERR_HTTP2_INVALID_SESSION]: The session has been destroyed
Error [ERR_STREAM_PREMATURE_CLOSE] [ERR_STREAM_PREMATURE_CLOSE]: Premature close
When I try to update this dataset, I get the same previous messages. I realize this might be a separate issue (sorry!). I’ll wait for some development before tinkering around some more 
Thanks again!
hi @rsetton
good to hear uploading a new dataset started to work! it could have been a specific configuration on the previous one. These are likely slightly different root issues. The premature_close is likely associated with a timeout or connection issue. As the upload gets much larger (say with raw plus derivatives) this could happen. One approach could be to split up the upload between the raw and derivative sections of the dataset. That would help address the dataset size.
(minor note: development work has started to resolve the issue surfaced yesterday)
thank you,
Franklin
Hi @franklin
I’m trying to use node version 12.19.0 to download a dataset from openneuro using this command:
/nafs/apps/openneuro-cli/bin/openneuro download --snapshot 1.0.2 ds003604 ds003604-download/
And I am getting this error message:
/nafs/apps/openneuro-cli/lib/node_modules/openneuro-cli/src/actions.js:1
TypeError: Cannot read property ‘startTransaction’ of undefined
at Command.download (/nafs/apps/openneuro-cli/lib/node_modules/openneuro-cli/src/actions.js:232:30)
at Proxy.listener (/nafs/apps/openneuro-cli/lib/node_modules/openneuro-cli/node_modules/commander/index.js:315:8)
at Proxy.emit (events.js:314:20)
at Proxy.EventEmitter.emit (domain.js:483:12)
at Proxy.Command.parseArgs (/nafs/apps/openneuro-cli/lib/node_modules/openneuro-cli/node_modules/commander/index.js:651:12)
at Proxy.Command.parse (/nafs/apps/openneuro-cli/lib/node_modules/openneuro-cli/node_modules/commander/index.js:474:21)
at Object. (/nafs/apps/openneuro-cli/lib/node_modules/openneuro-cli/src/cli.js:66:11)
at Generator.next ()
Any advice on how to fix this issue would be appreciated!
Thank you,
Becky