Fail to download ds003007 derivatives from openneuro using aws or datalad

Software Support
,
1

Dear NeuroStars expert,
I am trying to download mriqc and fmriprep derivatives from ds003007. However, I am getting the following errors:

-aws scenario:

$ aws s3 sync --no-sign-request s3://openneuro-derivatives/mriqc/ds003007-mriqc ds003007-mriqc
fatal error: An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied

-datalad scenario:

datalad install GitHub - OpenNeuroDerivatives/ds003007-mriqc

In this case, the folder gets downloaded, however all files inside seem to be much smaller than what they should, and I cannot open them

Could you please give me a hint on how to solve this?
Thanks a lot for your help and support!

2

Hi @FedeliD, and welcome to neurostars!

The files you are likely seeing are just links to what is stored in the cloud. To retrieve them, use datalad get. You can go inside the directory at the highest level and just run datalad get * to retrieve everything (or you can get more specific folders by inputting specific folders or files to the command).

Best,
Steven

1 Like
3

Thank you @Steven, it worked perfectly!

However when trying to open the fmriprep derivatives of the same dataset I get the following error:

datalad install GitHub - OpenNeuroDerivatives/ds003007-fmriprep
CommandError: ‘git -c diff.ignoreSubmodules=none for-each-ref ‘–format=%(refname:strip=2)’ refs/heads refs/remotes’ failed with exitcode 128
fatal: detected dubious ownership in repository at ‘/mnt/226AE1226AE0F407/scans_temp/ds003007-fmriprep’
To add an exception for this directory, call:

  git config --global --add safe.directory /mnt/226AE1226AE0F407/scans_temp/ds003007-fmriprep

I have added the exception with git config but when trying to retrieve one subject (e.g., sub-21.html) I get:

datalad get sub-21.html
[WARNING] Failed to get ‘sourcedata/templateflow/tpl-Fischer344.path’, skipping this submodule
[WARNING] Failed to get ‘sourcedata/templateflow/tpl-fsaverage.path’, skipping this submodule
[WARNING] Failed to get ‘sourcedata/templateflow/tpl-fsLR.path’, skipping this submodule
[WARNING] Failed to get ‘sourcedata/templateflow/tpl-MNI152Lin.path’, skipping this submodule
[WARNING] Failed to get ‘sourcedata/templateflow/tpl-MNI152NLin2009aAsym.path’, skipping this submodule
[WARNING] Failed to get ‘sourcedata/templateflow/tpl-MNI152NLin2009aSym.path’, skipping this submodule
[WARNING] Failed to get ‘sourcedata/templateflow/tpl-MNI152NLin2009bAsym.path’, skipping this submodule
[WARNING] Failed to get ‘sourcedata/templateflow/tpl-MNI152NLin2009bSym.path’, skipping this submodule
[WARNING] Failed to get ‘sourcedata/templateflow/tpl-MNI152NLin2009cAsym.path’, skipping this submodule
[WARNING] Failed to get ‘sourcedata/templateflow/tpl-MNI152NLin2009cSym.path’, skipping this submodule
[WARNING] Failed to get ‘sourcedata/templateflow/tpl-MNI152NLin6Asym.path’, skipping this submodule
[WARNING] Failed to get ‘sourcedata/templateflow/tpl-MNI152NLin6Sym.path’, skipping this submodule
[WARNING] Failed to get ‘sourcedata/templateflow/tpl-MNI305.path’, skipping this submodule
[WARNING] Failed to get ‘sourcedata/templateflow/tpl-MNIColin27.path’, skipping this submodule
[WARNING] Failed to get ‘sourcedata/templateflow/tpl-MNIInfant.path’, skipping this submodule
[WARNING] Failed to get ‘sourcedata/templateflow/tpl-MNIPediatricAsym.path’, skipping this submodule
[WARNING] Failed to get ‘sourcedata/templateflow/tpl-MouseIn.path’, skipping this submodule
[WARNING] Failed to get ‘sourcedata/templateflow/tpl-NKI.path’, skipping this submodule
[WARNING] Failed to get ‘sourcedata/templateflow/tpl-NMT31Sym.path’, skipping this submodule
[WARNING] Failed to get ‘sourcedata/templateflow/tpl-OASIS30ANTs.path’, skipping this submodule
[WARNING] Failed to get ‘sourcedata/templateflow/tpl-PNC.path’, skipping this submodule
[WARNING] Failed to get ‘sourcedata/templateflow/tpl-RESILIENT.path’, skipping this submodule
[WARNING] Failed to get ‘sourcedata/templateflow/tpl-UNCInfant.path’, skipping this submodule
[WARNING] Failed to get ‘sourcedata/templateflow/tpl-VALiDATe29.path’, skipping this submodule
[WARNING] Failed to get ‘sourcedata/templateflow/tpl-WHS.path’, skipping this submodule
get(error): sub-21.html (file) [not available; (Note that these git remotes have annex-ignore set: origin)]

On a different machine I have tried the same procedure and I get:

datalad get sub-21.html
[ERROR ] expected str, bytes or os.PathLike object, not tuple [pathlib.py:_parse_args:667] (TypeError)

I am wonderig why it works perfectly fine with the mriqc derivatives but fails with the fmriprep ones

thank you again for all the help
Davide

4

please share output of datalad wtf -D html_details which should include details of the filesystem and versions:

  • something funny with that filesystem you use which lead git itself to puke fatal: detected dubious ownership
  • may be that somehow triggered subsequent problems – not sure why it even considered those locations for that straightforwad get. May be do directly git annex get sub-21.html but also at git annex whereis sub-21.html – it should show that it is available from openneuro-derivatives which should have already been enabled during install/clone, but may be due to the prior error – it didn’t . So you could enable manually git annex enableremote openneuro-derivatives and try geting it again.

then on that other machine error, I searched up only this ConfigManager.get() not aligned with `git config <var>` behavior · Issue #4919 · datalad/datalad · GitHub for hitting the same error message. Would be interested to know wtf output for it as well – what is special about it too.

1 Like
5

Thank you @yarikoptic, I’ve tried with git annex get and it worked perfectly fine!

Thanks for all the help and support

6

Did you do

first? it would still be nice to figure out what is wrong/going on @FedeliD which caused all those odd behaviors.