just an update from my side. I manually uninstalled openneuro-cli by deleting the folder /usr/local/lib/node_modules/openneuro-cli/ rather than using the npm command. Re-installation via npm was then successful. However, now I run again into my initial upload error. I’ve pasted the error trail below. Please let me know about any suggestions of how to fix this problem. I can open a separate issue, if my post is not suitable in this thread.
Many thanks,
Nicole
$ node -v
v14.15.3
$ npm -v
6.14.9
$ openneuro -V
3.26.1
$ openneuro upload --dataset ds002634 -i .
Adding files to "ds002634"
(node:65785) UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'addLabels' of undefined
at uploadDataset (/usr/local/lib/node_modules/openneuro-cli/src/actions.js:65:18)
at Command.upload (/usr/local/lib/node_modules/openneuro-cli/src/actions.js:169:7)
at Proxy.listener (/usr/local/lib/node_modules/openneuro-cli/node_modules/commander/index.js:315:8)
at Proxy.emit (events.js:315:20)
at Proxy.EventEmitter.emit (domain.js:467:12)
at Proxy.Command.parseArgs (/usr/local/lib/node_modules/openneuro-cli/node_modules/commander/index.js:651:12)
at Proxy.Command.parse (/usr/local/lib/node_modules/openneuro-cli/node_modules/commander/index.js:474:21)
at Object.<anonymous> (/usr/local/lib/node_modules/openneuro-cli/src/cli.js:66:11)
at Generator.next (<anonymous>)
at bl (/usr/local/lib/node_modules/openneuro-cli/node_modules/esm/esm.js:1:245412)
(Use `node --trace-warnings ...` to show where the warning was created)
(node:65785) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 2)
(node:65785) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
Thank you for your message. The versions all look good. Perhaps can you try to upload a test dataset to ensure the cli is uploading properly? The test can be removed after this has been performed. I presume you used the same sudo npm install -g openneuro-cli command on the retry?
Also, can you please share the directory you intended to upload? (checking the directory level the upload originated at)
thank you for your reply. In the meantime I have updated the dataset manually on the OpenNeuro website, but let’s find out what is going wrong with the cli tool.
See below the info about the directory I’m intending to upload. Tool version numbers are the same as posted above. The same error occurs when I ‘cd’ into the directory and use ‘.’ for the dataset argument, as posted above.
$ ls project_larynx/
README sub-07 sub-16
dataset_description.json sub-08 sub-17
derivatives sub-09 sub-18
sub-01 sub-10 sub-19
sub-02 sub-11 sub-20
sub-03 sub-12 task-ArtVoc_events.json
sub-04 sub-13 task-Factorial_events.json
sub-05 sub-14
sub-06 sub-15
$ openneuro upload --dataset ds002634 -i project_larynx
Adding files to "ds002634"
(node:21295) UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'addLabels' of undefined
at uploadDataset (/usr/local/lib/node_modules/openneuro-cli/src/actions.js:65:18)
at Command.upload (/usr/local/lib/node_modules/openneuro-cli/src/actions.js:169:7)
at Proxy.listener (/usr/local/lib/node_modules/openneuro-cli/node_modules/commander/index.js:315:8)
at Proxy.emit (events.js:315:20)
at Proxy.EventEmitter.emit (domain.js:467:12)
at Proxy.Command.parseArgs (/usr/local/lib/node_modules/openneuro-cli/node_modules/commander/index.js:651:12)
at Proxy.Command.parse (/usr/local/lib/node_modules/openneuro-cli/node_modules/commander/index.js:474:21)
at Object.<anonymous> (/usr/local/lib/node_modules/openneuro-cli/src/cli.js:66:11)
at Generator.next (<anonymous>)
at bl (/usr/local/lib/node_modules/openneuro-cli/node_modules/esm/esm.js:1:245412)
(Use `node --trace-warnings ...` to show where the warning was created)
(node:21295) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 2)
(node:21295) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
Here is what happens when I try to upload a test dataset (tried with and without sudo):
$ mkdir test_upload
$ cd test_upload/
$ touch test.nii.gz
$ ls
test.nii.gz
$ sudo openneuro upload .
Password:
? **This will create a new dataset, continue?** Yes
TypeError: Cannot read property 'addLabels' of undefined
at uploadDataset (/usr/local/lib/node_modules/openneuro-cli/src/actions.js:65:18)
at /usr/local/lib/node_modules/openneuro-cli/src/actions.js:184:20
at processTicksAndRejections (internal/process/task_queues.js:93:5)
"." may not exist or is inaccessible
Please let me know if you require any further info from my side for trouble-shooting.
Thank you for sharing this information! Perhaps can you try to upload a subset of this dataset (say a subject or two) as a new dataset using a relative path rather than . to evaluate the upload process? This will help drill down why this error is popping up. I presume logging in was successful?
I appear to be having a similar issue to Nicole above. After having gone through the discussion, I haven’t been able to get around the issue.
I have attempted to upload data using the web interface and openneuro-cli to no avail.
When using the web interface, my data passed the bids validator step and started to upload. However after > 16 hours, it was still uploading, with several messages about failed attempts to upload files. I suspected it had to do with the size of the data, so instead tried with 10 participants and it was successful. However, it seems I can only upload one additional directory at a time on the web interface (but please correct me if I’m wrong!), which is not ideal.
I decided to stop and switch over to CLI upload and landed here.
Dataset description: ~300 participants, 2 rsfmri runs (multiecho data, so 6 nii/participant), T1, and some FLAIR.
node -v
v14.16.0
npm -v
6.14.11
./node_modules/.bin/openneuro --version
3.31.0
I have tried uploading the entire dataset (terminal output below) as well as one additional subject, but neither work.
~/openneurocli_dir$ ./node_modules/.bin/openneuro upload -d ds003588 -i /home/lbcspreng/data/openneuro/derivatives/
Adding files to "ds003588"
(node:5204) UnhandledPromiseRejectionWarning: TypeError: Cannot destructure property 'affirmedDefaced' of 'undefined' as it is undefined.
at uploadDataset (/home/lbcspreng/openneurocli_dir/node_modules/openneuro-cli/src/actions.js:59:5)
at Command.upload (/home/lbcspreng/openneurocli_dir/node_modules/openneuro-cli/src/actions.js:169:7)
at Proxy.listener (/home/lbcspreng/openneurocli_dir/node_modules/commander/index.js:315:8)
at Proxy.emit (events.js:315:20)
at Proxy.EventEmitter.emit (domain.js:467:12)
at Proxy.Command.parseArgs (/home/lbcspreng/openneurocli_dir/node_modules/commander/index.js:651:12)
at Proxy.Command.parse (/home/lbcspreng/openneurocli_dir/node_modules/commander/index.js:474:21)
at Object.<anonymous> (/home/lbcspreng/openneurocli_dir/node_modules/openneuro-cli/src/cli.js:84:11)
at Generator.next (<anonymous>)
at bl (/home/lbcspreng/openneurocli_dir/node_modules/esm/esm.js:1:245412)
at kl (/home/lbcspreng/openneurocli_dir/node_modules/esm/esm.js:1:247659)
at Object.u (/home/lbcspreng/openneurocli_dir/node_modules/esm/esm.js:1:287740)
at Object.o (/home/lbcspreng/openneurocli_dir/node_modules/esm/esm.js:1:287137)
at Object.<anonymous> (/home/lbcspreng/openneurocli_dir/node_modules/esm/esm.js:1:284879)
at Object.apply (/home/lbcspreng/openneurocli_dir/node_modules/esm/esm.js:1:199341)
at /home/lbcspreng/openneurocli_dir/node_modules/esm/esm.js:1:231006
at /home/lbcspreng/openneurocli_dir/node_modules/esm/esm.js:1:231041
at /home/lbcspreng/openneurocli_dir/node_modules/esm/esm.js:1:231254
at Su (/home/lbcspreng/openneurocli_dir/node_modules/esm/esm.js:1:228587)
at ku (/home/lbcspreng/openneurocli_dir/node_modules/esm/esm.js:1:230341)
at _c (/home/lbcspreng/openneurocli_dir/node_modules/esm/esm.js:1:258239)
at Module.<anonymous> (/home/lbcspreng/openneurocli_dir/node_modules/esm/esm.js:1:296060)
at n (/home/lbcspreng/openneurocli_dir/node_modules/esm/esm.js:1:279589)
at Object.<anonymous> (/home/lbcspreng/openneurocli_dir/node_modules/openneuro-cli/src/index.js:5:18)
at Module._compile (internal/modules/cjs/loader.js:1063:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:1092:10)
at Module.load (internal/modules/cjs/loader.js:928:32)
at Function.Module._load (internal/modules/cjs/loader.js:769:14)
at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:72:12)
at internal/main/run_main_module.js:17:47
(Use `node --trace-warnings ...` to show where the warning was created)
(node:5204) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 2)
(node:5204) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
Any help would be greatly appreciated!
Thanks,
Roni
thank you for your message and sharing details of your configurations. May you please try upgrading npm to version 7 and reuploading? This’ll help us further determine what could be causing this. The cli may be the more reliable option for uploading larger datasets like yours.
openneuro -help outputs normally. I have also reset the login token.
It is worth mentioning that when I install openneuro-cli (as recommended here), I get:
pm WARN deprecated mkdirp-promise@5.0.1: This package is broken and no longer maintained. 'mkdirp' itself supports promises now, please switch to that.
npm WARN deprecated urix@0.1.0: Please see https://github.com/lydell/urix#deprecated
npm WARN deprecated har-validator@5.1.5: this library is no longer supported
npm WARN deprecated resolve-url@0.2.1: https://github.com/lydell/resolve-url#deprecated
npm WARN deprecated request@2.88.2: request has been deprecated, see https://github.com/request/request/issues/3142
added 1073 packages, and audited 1074 packages in 35s
46 packages are looking for funding
run `npm fund` for details
6 low severity vulnerabilities
To address all issues, run:
npm audit fix
Run `npm audit` for details.
I then run npm audit fix and get the following:
up to date, audited 1074 packages in 5s
46 packages are looking for funding
run `npm fund` for details
# npm audit report
node-fetch <=2.6.0 || 3.0.0-beta.1 - 3.0.0-beta.8
Denial of Service - https://npmjs.com/advisories/1556
fix available via `npm audit fix --force`
Will install openneuro-cli@2.4.3, which is a breaking change
node_modules/bids-validator/node_modules/node-fetch
cross-fetch <=3.0.5
Depends on vulnerable versions of node-fetch
node_modules/bids-validator/node_modules/cross-fetch
bids-validator >=0.21.0
Depends on vulnerable versions of cross-fetch
Depends on vulnerable versions of yargs
node_modules/bids-validator
openneuro-cli >=2.5.3
Depends on vulnerable versions of bids-validator
node_modules/openneuro-cli
yargs-parser <=13.1.1 || 14.0.0 - 15.0.0 || 16.0.0 - 18.1.1
Prototype Pollution - https://npmjs.com/advisories/1500
fix available via `npm audit fix --force`
Will install openneuro-cli@2.4.3, which is a breaking change
node_modules/yargs/node_modules/yargs-parser
yargs 4.0.0-alpha1 - 12.0.5 || 14.1.0 || 15.0.0 - 15.2.0
Depends on vulnerable versions of yargs-parser
node_modules/yargs
bids-validator >=0.21.0
Depends on vulnerable versions of cross-fetch
Depends on vulnerable versions of yargs
node_modules/bids-validator
openneuro-cli >=2.5.3
Depends on vulnerable versions of bids-validator
node_modules/openneuro-cli
6 low severity vulnerabilities
To address all issues (including breaking changes), run:
npm audit fix --force
A small update: it appears to have started working when I uploaded as a new data set instead of trying to update an existing one, albeit with warnings for every file that looked like this:
}
[Error: EMFILE: too many open files, open '/home/lbcspreng/data/openneuro/derivatives/sub-96/ses-1/anat/sub-96_ses-1_T1w.nii.gz'] {
errno: -24,
code: 'EMFILE',
syscall: 'open',
path: '/home/lbcspreng/data/openneuro/derivatives/sub-96/ses-1/anat/sub-96_ses-1_T1w.nii.gz'
}
But, it was working! It got to ~29% and then overnight, it seems like the session timed out, with the terminal being populated with:
Error [ERR_HTTP2_INVALID_SESSION]: The session has been destroyed
Error [ERR_STREAM_PREMATURE_CLOSE] [ERR_STREAM_PREMATURE_CLOSE]: Premature close
When I try to update this dataset, I get the same previous messages. I realize this might be a separate issue (sorry!). I’ll wait for some development before tinkering around some more
good to hear uploading a new dataset started to work! it could have been a specific configuration on the previous one. These are likely slightly different root issues. The premature_close is likely associated with a timeout or connection issue. As the upload gets much larger (say with raw plus derivatives) this could happen. One approach could be to split up the upload between the raw and derivative sections of the dataset. That would help address the dataset size.
(minor note: development work has started to resolve the issue surfaced yesterday)
I’m trying to use node version 12.19.0 to download a dataset from openneuro using this command:
/nafs/apps/openneuro-cli/bin/openneuro download --snapshot 1.0.2 ds003604 ds003604-download/
And I am getting this error message:
/nafs/apps/openneuro-cli/lib/node_modules/openneuro-cli/src/actions.js:1
TypeError: Cannot read property ‘startTransaction’ of undefined
at Command.download (/nafs/apps/openneuro-cli/lib/node_modules/openneuro-cli/src/actions.js:232:30)
at Proxy.listener (/nafs/apps/openneuro-cli/lib/node_modules/openneuro-cli/node_modules/commander/index.js:315:8)
at Proxy.emit (events.js:314:20)
at Proxy.EventEmitter.emit (domain.js:483:12)
at Proxy.Command.parseArgs (/nafs/apps/openneuro-cli/lib/node_modules/openneuro-cli/node_modules/commander/index.js:651:12)
at Proxy.Command.parse (/nafs/apps/openneuro-cli/lib/node_modules/openneuro-cli/node_modules/commander/index.js:474:21)
at Object. (/nafs/apps/openneuro-cli/lib/node_modules/openneuro-cli/src/cli.js:66:11)
at Generator.next ()
Any advice on how to fix this issue would be appreciated!
Thank you for your message. By chance, can you please confirm you are using the latest version of the cli? Is node a recent download as well? It can sometimes be finicky.
Thank you for your message and sharing the versions. It appears the openneuro version is a bit outdated - may you please update to the latest version? We have been doing quite a bit of enhancements so there may be an internal API call that was improved.
I presume you were logged in as well prior to trying to download?
thank you for your message! there could be an oddity with your server. we also offer the AWS-cli for downloading OpenNeuro data - perhaps trying out that tool? On every dataset we provide the command for downloading that dataset
