Fail to download ds003382 from openneuro using aws

,
1

Hi everyone,

From ds003382 of openneuro,
I failed to download the *_listmode.bf files and *_pet.nii.gz files using following command line:

aws s3 sync --no-sign-request s3://openneuro.org/ds003382 E:/

with error messages:

An error occurred (AccessDenied) when calling the GetObject operation: Access Denied"

How to solve this problem?

BTW, I can directly download each of these files using the browser, but always ended up with wrong file size.

Thanks!

2

hi @Shimmering

Thank you for surfacing this. Perhaps to clarify - it appears these two files are primarily in sourcedata or sub-XY/pet directories respectively?

May you please clarify what you mean by always ended up with the wrong file size? Wrong file size relative to what?

We are also checking the configurations on our side.

Thank you,
Franklin

3

Hi franklin,
Thanks for your reply!

Yes, the files are in sourcedata and sub-xy/pet directories respectively.

I guess the file size of each of the *_pet.nii.gz will be near 8GB, as described in OpenNeuro.
When I directly downloaded these files using the browser, Chrome to be specific, I only got file size of 1~2GB and the connection was closed. I checked the content of the downloaded file, it was obvious that many of the content were lost. Maybe the connection was unstable when I download files using browsers?

Thanks,
Bin

4

Hi @Shimmering

Thank you for your message and clarifications. Regarding the size of each *_pet.nii.gz file - perhaps may you please point me (or take a screenshot) to the location conveying the file sizes are 8GB? On the server, I am seeing around 5GB. The README mentions total sizes of the dataset (and imaging components).

We are running a dataset check to evaluate the configurations.

Thank you,
Franklin

5

There indeed were permission issues with some of the .bf and _pet.nii.gz files. I have updated these, please let me know if downloading them with the aws-cli works.

6

image
image912×399 32.8 KB

Hi @franklin

Here’s the information of file size. I did a rough estimation. 10 subject for ~81GB of pet files.
But since the pet files in folder sourcedata account for ~1.5GB, it is fair enough that each *_pet.nii.gz in folder sub-xy/pet will be ~5-6GB.

The downloading process starts now, with the update from @rwblair.
So for now everything’s fine.
I’ll keep you informed for further updates.

Bin

7

Hi @rwblair ,

Thanks for the update!
It starts downloading now, with the aws-cli.
I’ll keep you informed when I finish downloading.

Bin

8

Hi @rwblair

I’ve finished downloading related pet files.

The aws-cli works well.

Thanks!

Bin

9

Hi @franklin ,

I’ve successfully downloaded the related pet files.

Unfortunately, I got several files (the ones named *_pet.nii.gz in the subj-xy/pet folder) with size <5GB.
I’m pretty sure these files are NOT intact.

I took a screenshot of file ‘sub-16_task-vis_run-001_pet.nii.gz’:

image
image902×340 16.7 KB

This file is supposed to contain 94 channels (indicated by yellow arrow).
But from the 10th channel on, images are filled with zeros (circled in green).

Similar phenomenon can be observed in whichever file with size <5GB.
You can recheck this on the server.

Hope this can be fixed soon,
Thanks!

Bin

10

hi @Shimmering

Good to hear the download worked!

Regarding the file - it appears the smaller file size reflects what we have on the server. Perhaps reach out to the dataset owners to convey what you are seeing and if it reflects what they have locally

Thank you,
Franklin

11

Hi @franklin @rwblair

I’ve reached out to the dataset owner for re-uploading the dataset.
They’ve corrected the error of file size.

But when I tried to download the dataset, the error of “AccessDenied” mentioned previously occurred again.

Can you fix this issue?

Best,
Bin

12

@Shimmering I ran the script to fix permissions on ds003382 again. Still trying to figure out why they weren’t corrected automatically. If you run into any more issues please let me know.

13

Thanks, @rwblair . It starts downloading now.

Perhaps you can have a look at dataset ds002814, too.
It’s the same issue.

Thanks!

14

hi @Shimmering

Good to hear regarding ds003382!

Regarding ds002814 - may you please point us at the files that are having trouble?

Thank you,
Franklin

15

Hi @franklin

As for ds002814, all the files showed the “AccessDenied” issue.

BTW, I used aws-cli for downloading.

Bin

16

Permission fixes took a bit longer to run on ds002814 but finally completed. Let me know if you run into any more of these.