Steven
August 28, 2022, 5:09pm
21
Is sbp_env your datalad environment? Because in the screenshot above it indicated you were in base
While downloading I thought I would try using both, the miniconda environment as they suggest on the website (which is base) as well as sbp_env (which is created using: conda create -n sbp_env python=3.7 anaconda
) but neither of them seem to work.
Steven
August 28, 2022, 5:23pm
23
Do you get the same error in the other environment?
Yes, all the file sizes are too small to be nii.gz files.
Steven
August 28, 2022, 5:27pm
25
I was more asking if the error message the same in both environments?
No error message here but just the fact that it hasn’t really downloaded
Steven
August 28, 2022, 5:32pm
27
I wonder if it has to do with git not being configured, per the first screenshot.
If I run it the first time the output looks like this
(Right now I’ve done it for another file because I cleared the output)
Running it nth time:
But the file size is still 4K
Steven
August 28, 2022, 5:54pm
29
Perhaps this is something you should ping the Templateflow and/or Datalad GitHub issues about.
Just so that I and everyone with the same issue can keep track, I have submitted two issues:
opened 06:04PM - 28 Aug 22 UTC
Most of the issues is covered in this link: https://neurostars.org/t/fmriprep-pr… epocessing-not-successful-file-not-gzip/23353/28
I have tried downloading templateflow using:
1. Datalad installation: http://handbook.datalad.org/en/latest/intro/installation.html#linux-machines-with-no-root-access-e-g-hpc-systems and then running:
```bash
datalad install -r ///templateflow
cd templateflow/
datalad siblings -d "REDACTED/sbp/templateflow_install/templateflow" enable -s public-s3 -r
datalad get -r tpl-MNI152NLin2009cAsym
datalad get -r tpl-MNI152Lin
datalad get -r tpl-MNI152NLin2009cSym
```
2. Downloading using a python script: https://pastebin.com/NAZmHFMA
But in both cases, the file sizes are just a couple of kilobytes.
opened 06:14PM - 28 Aug 22 UTC
### What is the problem?
I am unable to download complete files using datalad o… n an HPC. I am trying to download templateflow archive (https://www.templateflow.org/usage/archive/), but the files don't get downloaded completely.
### What steps will reproduce the problem?
I have tried downloading templateflow on an HPC using:
1. Datalad installation: http://handbook.datalad.org/en/latest/intro/installation.html#linux-machines-with-no-root-access-e-g-hpc-systems and then running:
```bash
datalad install -r ///templateflow
cd templateflow/
datalad siblings -d "REDACTED/sbp/templateflow_install/templateflow" enable -s public-s3 -r
datalad get -r tpl-MNI152NLin2009cAsym
datalad get -r tpl-MNI152Lin
datalad get -r tpl-MNI152NLin2009cSym
```
2. Downloading using a python script: https://pastebin.com/NAZmHFMA
But in both cases, the file sizes are just a couple of kilobytes.
### DataLad information
Datalad version: datalad 0.17.3
Output from `datalad wtf`:
```bash
# WTF
## configuration <SENSITIVE, report disabled by configuration>
## credentials
- keyring:
- active_backends:
- PlaintextKeyring with no encyption v.1.0 at /home/mj606/.local/share/python_keyring/keyring_pass.cfg
- config_file: /home/mj606/.config/python_keyring/keyringrc.cfg
- data_root: /home/mj606/.local/share/python_keyring
## datalad
- version: 0.17.3
## dependencies
- annexremote: 1.5.0
- boto: 2.49.0
- cmd:7z: 16.02
- cmd:annex: 10.20220724-g05cae5801
- cmd:bundled-git: 2.36.1
- cmd:git: 2.36.1
- cmd:ssh: 8.0p1
- cmd:system-git: 2.29.2
- cmd:system-ssh: 8.0p1
- exifread: 3.0.0
- humanize: 4.3.0
- iso8601: 1.0.2
- keyring: 22.3.0
- keyrings.alt: 4.0.2
- msgpack: 1.0.2
- mutagen: 1.45.1
- platformdirs: 2.5.2
- requests: 2.25.1
## environment
- LANG: C.UTF-8
- PATH: /home/mj606/.conda/envs/sbp_env/bin:/home/mj606/miniconda3/condabin:/usr/local/software/spack/spack-views/._rocky8-icelake-20220710/pbtmwfe3tbtsyrxpj5lm4rfm7cw3jcz6/intel-oneapi-mpi-2021.6.0/intel-2021.6.0/guxuvcpmykplbrr2e3af2yd7njqhau5e/mpi/2021.6.0/libfabric/bin:/usr/local/software/spack/spack-views/._rocky8-icelake-20220710/pbtmwfe3tbtsyrxpj5lm4rfm7cw3jcz6/intel-oneapi-mpi-2021.6.0/intel-2021.6.0/guxuvcpmykplbrr2e3af2yd7njqhau5e/mpi/2021.6.0/bin:/usr/local/software/spack/spack-views/._rocky8-icelake-20220710/pbtmwfe3tbtsyrxpj5lm4rfm7cw3jcz6/intel-oneapi-compilers-2022.1.0/gcc-11.3.0/b6zld2mz7cid27yloxznoidymd7vywwz/compiler/2022.1.0/linux/lib/oclfpga/bin:/usr/local/software/spack/spack-views/._rocky8-icelake-20220710/pbtmwfe3tbtsyrxpj5lm4rfm7cw3jcz6/intel-oneapi-compilers-2022.1.0/gcc-11.3.0/b6zld2mz7cid27yloxznoidymd7vywwz/compiler/2022.1.0/linux/bin/intel64:/usr/local/software/spack/spack-views/._rocky8-icelake-20220710/pbtmwfe3tbtsyrxpj5lm4rfm7cw3jcz6/intel-oneapi-compilers-2022.1.0/gcc-11.3.0/b6zld2mz7cid27yloxznoidymd7vywwz/compiler/2022.1.0/linux/bin:/usr/local/software/cuda/11.4/bin:/usr/local/software/cuda/11.4/samples/bin/x86_64/linux/release:/usr/local/software/cuda/11.4/nvvm/bin:/usr/local/software/cuda/11.4/libnvvp:/usr/local/software/global-rhel8/bin:/opt/singularity/bin:/usr/local/Cluster-Apps/singularity/images:/usr/local/software/slurm/current-rhel8/sbin:/usr/local/software/slurm/current-rhel8/bin:/usr/share/Modules/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/dell/srvadmin/bin:.:/home/mj606/bin
- PYTHONDONTWRITEBYTECODE: 1
## extensions
## git-annex
- build flags:
- Assistant
- Webapp
- Pairing
- Inotify
- DBus
- DesktopNotify
- TorrentParser
- MagicMime
- DebugLocks
- Feeds
- Testsuite
- S3
- WebDAV
- dependency versions:
- aws-0.22
- bloomfilter-2.0.1.0
- cryptonite-0.26
- DAV-1.3.4
- feed-1.3.0.1
- ghc-8.8.4
- http-client-0.6.4.1
- persistent-sqlite-2.10.6.2
- torrent-10000.1.1
- uuid-1.3.13
- yesod-1.6.1.0
- key/value backends:
- SHA256E
- SHA256
- SHA512E
- SHA512
- SHA224E
- SHA224
- SHA384E
- SHA384
- SHA3_256E
- SHA3_256
- SHA3_512E
- SHA3_512
- SHA3_224E
- SHA3_224
- SHA3_384E
- SHA3_384
- SKEIN256E
- SKEIN256
- SKEIN512E
- SKEIN512
- BLAKE2B256E
- BLAKE2B256
- BLAKE2B512E
- BLAKE2B512
- BLAKE2B160E
- BLAKE2B160
- BLAKE2B224E
- BLAKE2B224
- BLAKE2B384E
- BLAKE2B384
- BLAKE2BP512E
- BLAKE2BP512
- BLAKE2S256E
- BLAKE2S256
- BLAKE2S160E
- BLAKE2S160
- BLAKE2S224E
- BLAKE2S224
- BLAKE2SP256E
- BLAKE2SP256
- BLAKE2SP224E
- BLAKE2SP224
- SHA1E
- SHA1
- MD5E
- MD5
- WORM
- URL
- X*
- operating system: linux x86_64
- remote types:
- git
- gcrypt
- p2p
- S3
- bup
- directory
- rsync
- web
- bittorrent
- webdav
- adb
- tahoe
- glacier
- ddar
- git-lfs
- httpalso
- borg
- hook
- external
- supported repository versions:
- 8
- 9
- 10
- upgrade supported from repository versions:
- 0
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- version: 10.20220724-g05cae5801
## location
- path: /home/mj606
- type: directory
## metadata_extractors
- annex:
- distribution: datalad 0.17.3
- load_error: None
- module: datalad.metadata.extractors.annex
- audio:
- distribution: datalad 0.17.3
- load_error: None
- module: datalad.metadata.extractors.audio
- datacite:
- distribution: datalad 0.17.3
- load_error: None
- module: datalad.metadata.extractors.datacite
- datalad_core:
- distribution: datalad 0.17.3
- load_error: None
- module: datalad.metadata.extractors.datalad_core
- datalad_rfc822:
- distribution: datalad 0.17.3
- load_error: None
- module: datalad.metadata.extractors.datalad_rfc822
- exif:
- distribution: datalad 0.17.3
- load_error: None
- module: datalad.metadata.extractors.exif
- frictionless_datapackage:
- distribution: datalad 0.17.3
- load_error: None
- module: datalad.metadata.extractors.frictionless_datapackage
- image:
- distribution: datalad 0.17.3
- load_error: None
- module: datalad.metadata.extractors.image
- xmp:
- distribution: datalad 0.17.3
- load_error: ModuleNotFoundError(No module named 'libxmp')
- module: datalad.metadata.extractors.xmp
## metadata_indexers
## python
- implementation: CPython
- version: 3.7.10
## system
- distribution: Rocky Linux/8.6/Green Obsidian
- encoding:
- default: utf-8
- filesystem: utf-8
- locale.prefered: UTF-8
- filesystem:
- CWD:
- max_pathlength: 4096
- mount_opts: rw,relatime
- path: /home/mj606
- type: ext4
- HOME:
- max_pathlength: 4096
- mount_opts: rw,relatime
- path: /home/mj606
- type: ext4
- TMP:
- max_pathlength: 4096
- mount_opts: rw,relatime
- path: /tmp
- type: ext4
- max_path_length: 267
- name: Linux
- release: 4.18.0-372.19.1.el8_6.x86_64
- type: posix
- version: #1 SMP Tue Aug 2 16:19:42 UTC 2022
```
### Additional context
Most of the issues is covered in this link: https://neurostars.org/t/fmriprep-prepocessing-not-successful-file-not-gzip/23353/28
And another issue has been submitted: https://github.com/templateflow/templateflow/issues/144
### Have you had any success using DataLad before?
_No response_
1 Like
psymbio
September 5, 2022, 6:06am
31
The issue has been resolved for me. Here’s what I ran:
mkdir templateflow_installation/
cd templateflow_installation/
datalad clone https://github.com/templateflow/templateflow
cd templateflow/
datalad get -r tpl-Fischer344
datalad get -r tpl-MNI152Lin
datalad get -r tpl-MNI152NLin2009aAsym
datalad get -r tpl-MNI152NLin2009aSym
datalad get -r tpl-MNI152NLin2009bAsym
datalad get -r tpl-MNI152NLin2009bSym
datalad get -r tpl-MNI152NLin2009cAsym
datalad get -r tpl-MNI152NLin2009cSym
datalad get -r tpl-MNI152NLin6Asym
datalad get -r tpl-MNI152NLin6Sym
datalad get -r tpl-MNI305
datalad get -r tpl-MNIColin27
datalad get -r tpl-MNIInfant
datalad get -r tpl-MNIPediatricAsym
datalad get -r tpl-MouseIn
datalad get -r tpl-NKI
datalad get -r tpl-NMT31Sym
datalad get -r tpl-OASIS30ANTs
datalad get -r tpl-PNC
datalad get -r tpl-RESILIENT
datalad get -r tpl-UNCInfant
datalad get -r tpl-VALiDATe29
datalad get -r tpl-WHS
datalad get -r tpl-fsLR
datalad get -r tpl-fsaverage
cd ..
mv templateflow/ $HOME/.cache/
And in my slurm script I mentioned:
export SINGULARITYENV_TEMPLATEFLOW_HOME=$HOME/.cache/templateflow
The size of the files still seem the same (4K, 24K etc.) if you use ls -sh
inside tpl-MNI152NLin6Asym
.
I suggest you just run
cd tpl-MNI152NLin6Asym
du -hs
to see the size of the entire folder:
And it seems fine so, just roll with it I guess.