Cannot find latest version when updating #datalad datasets

openneuro
datalad
#1

Hi,

this might be related to this thread, but I have a dataset on OpenNeuro, and the URL currently points to the latest version 1.0.4.

When installing it with datalad, it only fetches version 1.0.3 (not 1.0.4):

datalad install https://github.com/OpenNeuroDatasets/ds001919.git
cd ds001919/
datalad get .
cat CHANGES

Output:

1.0.0 2019-05-14
- Initial release

1.0.1 2019-05-21
- Updated: sub-mniS04 (not the correct subject)

1.0.2 2019-05-22
- Updated: sub-mniS04 (fixed error introduced in 1.0.1) 
- Updated: sub-tehranS04 (not the correct subject)

1.0.3 2019-07-08
  - Added: sub-pavia01, cmrra, cmrrb, ubc, tokyoSkyra(sub-07), tokyo750w(sub-07), tokyoIngenia(sub-07)
  - Fixed participants.tsv for tokyoSkyra

Whereas the latest CHANGES contains this.

Thank you for your help,
Julien

#2

Hi @jcohenadad

Thank you for your message! I have raised an issue with our developers. By chance, have you tried the AWS S3 download method as well?

Thank you,
Franklin

#3

have you tried the AWS S3 download method as well?

I have not

Thank you for the quick response!

#4

Hi,

I would like to make a follow-up concerning the issue reported by @jcohenadad. The issue seemed to have been adressed on GH-issue 1200 .

I ran the following commands:

datalad install https://github.com/OpenNeuroDatasets/ds001919.git
cd ds001919/
datalad get .

The result is :

get (error: 35, impossible: 1, notneeded: 1, ok: 1626)

You can find the complete output of the terminal here .

The command seems to grab the latest snapshot, but it cannot retrieve the latest .nii.gz & .json.

Thank you very much for your help,
Alexandru

#5

Hi @alexfoias

Thank you for your raising this! Looking at your output it appears to be associated with this issue. We are working on enhancing our GitHub remote audits to ensure that the latest version is able to be downloaded using Datalad (I have added this dataset to the list of confirming the GitHub remotes are the most up to date snapshot). We also have capabilities of downloading using S3 - aws s3 sync --no-sign-request s3://openneuro.org/ds001919 ds001919-download/

Thank you,
Franklin

#6

Hi,

I’ve tried with aws and the downloaded dataset is not the latest.

Experiment done on 2019-09-19 at 14:59:24: from OSX 10.14.6

aws s3 sync --no-sign-request s3://openneuro.org/ds001919 ds001919-aws
cd ds001919-aws
more CHANGES

Problem: this is only version 1.0.3

1.0.0 2019-05-14

- Initial release

1.0.1 2019-05-21

- Updated: sub-mniS04 (not the correct subject)

1.0.2 2019-05-22

- Updated: sub-mniS04 (fixed error introduced in 1.0.1) 

- Updated: sub-tehranS04 (not the correct subject)

1.0.3 2019-07-08

- Added: sub-pavia01, cmrra, cmrrb, ubc, tokyoSkyra(sub-07), tokyo750w(sub-07), tokyoIngenia(sub-07)

- Fixed participants.tsv for tokyoSkyra

Whereas on https://openneuro.org/datasets/ds001919/versions/1.0.4, the same file gives:

1.0.0 2019-05-14
- Initial release

1.0.1 2019-05-21
- Updated: sub-mniS04 (not the correct subject)

1.0.2 2019-05-22
- Updated: sub-mniS04 (fixed error introduced in 1.0.1) 
- Updated: sub-tehranS04 (not the correct subject)

1.0.3 2019-07-08
- Added: sub-pavia01, cmrra, cmrrb, ubc, tokyoSkyra(sub-07), tokyo750w(sub-07), tokyoIngenia(sub-07)
- Fixed participants.tsv for tokyoSkyra

1.0.4 2019-07-17
  - - Added: sub-pavia02, sub-pavia03, sub-pavia04, sub-pavia05, sub-pavia06

Also, is there any update on this issue regarding datalad download?

Many thanks,
Julien

#7

Hi @jcohenadad

Thank you for your message and additional information! I apologize for this inconvenience! This issue may be linked to the validation pending (we can retrigger the validation on your dataset). We have been working on enhancing our validation process on larger datasets.

Regarding the datalad download issue - we have this on our log to enhance the GitHub auditing functionality to make sure that datalad can function properly with our system.

Thank you,
Franklin