Problem using neurodocker and singularity-hub

Hi all,

I’m trying to use neurodocker via singularity to build a singularity recipe which I would like to build on singularity-hub and then download. The issue I am running into is that after building on singularity-hub, the container is missing my conda packages and I see log lines as follows:

Will remove the following tarballs:
idna-2.8-py37_0.conda 85 KB
openssl-1.1.1e-h7b6447c_0.conda 2.5 MB
Total: 76.6 MB
Proceed ([y]/n)? Invalid choice: conda create -y -q --name neuro
Proceed ([y]/n)? Invalid choice: conda install -y -q --name neuro \
Proceed ([y]/n)? Invalid choice: "numpy" \

This suggests to me that it is not answering this prompt and thus not installing any Conda packages.

I’ve tried to modify my recipe to echo “y” there but that did not work. This is my first time using all of these programs, am I going about this the wrong way or has anyone run into similar issues? Any help or input would be appreciated, thanks!

More detailed info:

I use the following neurodocker command (using singularity):

singularity run neurodocker.simg generate singularity \
--base centos:7 \
--pkg-manager yum \
--miniconda create_env=neuro \
conda_install="numpy pandas scipy scikit-learn matplotlib seaborn networkx ipython joblib nibabel datalad" \
pip_install="nibabel ply neurosynth nilearn pliers" \
--install emacs > Singularity

This seems to work and gives me a recipe:

# Your version: 0.6.0 Latest version: 0.6.0
# Generated by Neurodocker version 0.6.0
# Timestamp: 2020-03-20 20:25:56 UTC
# Thank you for using Neurodocker. If you discover any issues
# or ways to improve this software, please submit an issue or
# pull request on our GitHub repository:

Bootstrap: docker
From: centos:7

export ND_ENTRYPOINT="/neurodocker/"
yum install -y -q \
    bzip2 \
    ca-certificates \
    curl \
    localedef \
yum clean packages
rm -rf /var/cache/yum/*
localedef -i en_US -f UTF-8 en_US.UTF-8
chmod 777 /opt && chmod a+s /opt
mkdir -p /neurodocker
if [ ! -f "$ND_ENTRYPOINT" ]; then
  echo '#!/usr/bin/env bash' >> "$ND_ENTRYPOINT"
  echo 'set -e' >> "$ND_ENTRYPOINT"
  echo 'export USER="${USER:=`whoami`}"' >> "$ND_ENTRYPOINT"
  echo 'if [ -n "$1" ]; then "$@"; else /usr/bin/env bash; fi' >> "$ND_ENTRYPOINT";
chmod -R 777 /neurodocker && chmod a+s /neurodocker

export PATH="/opt/miniconda-latest/bin:$PATH"
echo "Downloading Miniconda installer ..."
curl -fsSL --retry 5 -o "$conda_installer"
bash "$conda_installer" -b -p /opt/miniconda-latest
rm -f "$conda_installer"
conda update -yq -nbase conda
conda config --system --prepend channels conda-forge
conda config --system --set auto_update_conda false
conda config --system --set show_channel_urls true
sync && conda clean --all && sync
echo 'y'
conda create -y -q --name neuro
conda install -y -q --name neuro \
    "numpy" \
    "pandas" \
    "scipy" \
    "scikit-learn" \
    "matplotlib" \
    "seaborn" \
    "networkx" \
    "ipython" \
    "joblib" \
    "nibabel" \
sync && conda clean --all && sync
bash -c "source activate neuro
  pip install --no-cache-dir  \
      "nibabel" \
      "ply" \
      "neurosynth" \
      "nilearn" \
rm -rf ~/.cache/pip/*

yum install -y None \
yum clean packages
rm -rf /var/cache/yum/*

echo '{
\n  "pkg_manager": "yum",
\n  "instructions": [
\n    [
\n      "base",
\n      "centos:7"
\n    ],
\n    [
\n      "_header",
\n      {
\n        "version": "generic",
\n        "method": "custom"
\n      }
\n    ],
\n    [
\n      "miniconda",
\n      {
\n        "create_env": "neuro",
\n        "conda_install": [
\n          "numpy",
\n          "pandas",
\n          "scipy",
\n          "scikit-learn",
\n          "matplotlib",
\n          "seaborn",
\n          "networkx",
\n          "ipython",
\n          "joblib",
\n          "nibabel",
\n          "datalad"
\n        ],
\n        "pip_install": [
\n          "nibabel",
\n          "ply",
\n          "neurosynth",
\n          "nilearn",
\n          "pliers"
\n        ]
\n      }
\n    ],
\n    [
\n      "install",
\n      [
\n        "emacs"
\n      ]
\n    ]
\n  ]
\n}' > /neurodocker/neurodocker_specs.json

export LANG="en_US.UTF-8"
export LC_ALL="en_US.UTF-8"
export ND_ENTRYPOINT="/neurodocker/"
export CONDA_DIR="/opt/miniconda-latest"
export PATH="/opt/miniconda-latest/bin:$PATH"

/neurodocker/ "$@"

I’m not sure right now why this is not working. But it seems this is specific to centos and singularity. I was able to generate a Dockerfile using a neurodocker specification you provided and build a docker container without any issue.

For the singularity image, I’ve checked that I’m able to create an image if the base one is debian. Not sure if that helps… We can search more, or perhaps someone else knows the answer

Ah thanks for checking! I tried this also earlier with debian stretch as the base OS and it seemed to build fine. The issue I then had was that I could not use any modules still. Upon starting python I could not load numpy for example.

It could be a problem with activating the conda env:

Singularity> conda info --envs
# conda environments:
base                  *  /opt/miniconda-latest
neuro                    /opt/miniconda-latest/envs/neuro

Singularity> activate neuro
Singularity> conda activate neuro

CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'.
To initialize your shell, run

    $ conda init <SHELL_NAME>

Currently supported shells are:
  - bash
  - fish
  - tcsh
  - xonsh
  - zsh
  - powershell

See 'conda init --help' for more information and options.

IMPORTANT: You may need to close and restart your shell after running 'conda init'.

Though from the documentation I thought this would be activated by default. Haven’t yet been able to figure this out.

perhaps something is out of date in neurodocker, maybe you can open an issue, to remember that we should check this.

if you want to activate once you are inside, you can run source activate neuro.

Or try to run conda init bash (or other shells), you will have to read from the output which files has been modified by conda and run source ..../.bashrc (or something similar)

1 Like

Thanks, that did it! I will open an issue on neurodocker re: centos / singularity problem