Datalad: install datalad-containers extension in conjunction with Singularity datalad container?

1

I am trying to use datalad on an hpc (running CentOS 7), and would like to use the datalad-containers extension to track changes to my dataset made using containerized versions heudiconv and fmriprep.

I pulled the singularity container for datalad (a la https://www.datalad.org/get_datalad.html) and exported the path to the directory where I put the datalad image (export PATH="/path/to/singularity_images_directory:$PATH"), and can use much of the datalad functionality without a problem. However, I am uncertain if it is possible to install extensions when using the containerized version of datalad.

Alternatively, I have tried to set up a virtual environment for datalad according to the instructions in the DataLad Hanbook for for Linux-machines with no root access (e.g. HPC systems) (a la http://handbook.datalad.org/en/latest/intro/installation.html#initial-configuration), but it seems that the version of datalad available via Miniconda is 0.11.8, which is missing much of the functionality of the current release.

Any advice on how to add extension functionality to a containerized datalad instance or set up a virtual environment with a more recent version of datalad would be much appreciated!

2

I would recommend using the conda-based approach. If you have installed datalad 0.11 already, you should be able to upgrade to the most recent 0.12 release candidate via:

pip install datalad==0.12.0rc5
3

Ahh, I see. Thanks very much!

1 Like
4

FWIW – now there is an rc channel with release candidates. handbook was updated to reflect that, see http://handbook.datalad.org/en/latest/intro/installation.html#linux-machines-with-no-root-access-e-g-hpc-systems which currently provides recipe:

$ wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
$ bash Miniconda3-latest-Linux-x86_64.sh
# acknowledge license, keep everything at default
$ conda install -c conda-forge datalad
# upgrade to the latest release candidate to match the requires of the book
$ conda install -c conda-forge/label/rc datalad

you can add -b to miniconda script installation for batch installation (no interactions)

1 Like
5

Do I just add -b to the miniconda script when I call it, like:

$ bash -b Miniconda3-latest-Linux-x86_64.sh

or do I need to edit the script itself?

6

add it as an option to script invocation:

bash Miniconda3-latest-Linux-x86_64.sh -b

here is my current version of the script to (re)install miniconda:

#!/bin/bash

set -eu

if [ ! -e $HOME/miniconda ]; then
	wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
	bash Miniconda3-latest-Linux-x86_64.sh -b -p $HOME/miniconda
fi

export PATH=~/miniconda/bin/:$PATH
conda install -c conda-forge datalad
conda install -c conda-forge/label/rc datalad

Note that in -b mode it doesn’t modify your ~/.bash* files, so you need to adjust PATH env variable to place conda into your PATH. I like it that way anyways so I can then still switch between different conda installations etc

1 Like