Installing Datalad DS over ssh

1

I am experiencing unstable behavior with

datalad install -s ssh:user@host:/path/to/dataset /path/to/installed/dataset

on a bare git repository.

Sometimes it works fine, other times it doesn’t. Here is the log output from my most recent fail:

     datalad -l 2 install -s ssh:admin@Voxel-Forge.local:/Users/admin/Anvil/sync/projects/central-repository/community-life-adolescent-development.git -D "SuperDataset for Community Life & Adolescent Development Project on Voxel-Forge" community-life-adolescent-development
    [Level 9] Running: ['git', 'config', '-z', '-l', '--show-origin']
    [Level 8] Finished running ['git', 'config', '-z', '-l', '--show-origin'] with status 0
    [DEBUG  ] Determined class of decorated function: <class 'datalad.distribution.install.Install'>
    [Level 5] Parsed ri community-life-adolescent-development into fields {'username': '', 'scheme': '', 'fragment': '', 'hostname': '', 'query': '', 'path': 'community-life-adolescent-development', 'password': '', 'port': ''}
    [Level 5] Detected file ri
    [DEBUG  ] Determined class of decorated function: <class 'datalad.distribution.clone.Clone'>
    [Level 5] Parsed ri ssh:admin@Voxel-Forge.local:/Users/admin/Anvil/sync/projects/central-repository/community-life-adolescent-development.git into fields {'username': '', 'scheme': 'ssh', 'fragment': '', 'hostname': '', 'query': '', 'path': 'admin@Voxel-Forge.local:/Users/admin/Anvil/sync/projects/central-repository/community-life-adolescent-development.git', 'password': '', 'port': ''}
    [Level 5] Assuming ssh style ri, adjusted: {'username': '', 'scheme': 'ssh', 'fragment': '', 'hostname': '', 'query': '', 'path': 'admin@Voxel-Forge.local:/Users/admin/Anvil/sync/projects/central-repository/community-life-adolescent-development.git', 'password': '', 'port': ''}
    [Level 5] Detected ssh ri
    [DEBUG  ] Resolved clone source from 'ssh:admin@Voxel-Forge.local:/Users/admin/Anvil/sync/projects/central-repository/community-life-adolescent-development.git' to 'ssh:admin@Voxel-Forge.local:/Users/admin/Anvil/sync/projects/central-repository/community-life-adolescent-development.git'
    [DEBUG  ] Resolved clone target path to: '/Users/admin/Anvil/sync/projects/community-life-adolescent-development'
    [Level 5] Parsed ri /Users/admin/Anvil/sync/projects/community-life-adolescent-development into fields {'username': '', 'scheme': '', 'fragment': '', 'hostname': '', 'query': '', 'path': '/Users/admin/Anvil/sync/projects/community-life-adolescent-development', 'password': '', 'port': ''}
    [Level 5] Detected file ri
    [INFO   ] Cloning ssh:admin@Voxel-Forge.local:/Users/admin/Anvil/sync/projects/central-repository/community-life-adolescent-development.git to '/Users/admin/Anvil/sync/projects/community-life-adolescent-development'
    [DEBUG  ] Attempting to clone ssh:admin@Voxel-Forge.local:/Users/admin/Anvil/sync/projects/central-repository/community-life-adolescent-development.git (1 out of 1 candidates) to '/Users/admin/Anvil/sync/projects/community-life-adolescent-development'
    [Level 9] Running: ['git', 'version']
    [Level 8] Finished running ['git', 'version'] with status 0
    [Level 9] Running: ['git', 'config', '-z', '-l', '--show-origin']
    [Level 8] Finished running ['git', 'config', '-z', '-l', '--show-origin'] with status 0
    [Level 5] Found 0 previous connections
    [Level 5] Not opening SSHConnection(ctrl_path=<<'/Users/admin/Library/...>>, sshri=<<SSHRI(hostname='Voxel-...>>) for checking since /Users/admin/Library/Caches/datalad/sockets/6eed3025 does not exist
    [DEBUG  ] Opening SSHConnection(ctrl_path=<<'/Users/admin/Library/...>>, sshri=<<SSHRI(hostname='Voxel-...>>) by calling ['ssh', '-fN', '-o', 'ControlMaster=auto', '-o', 'ControlPersist=15m', '-o', 'ControlPath="/Users/admin/Library/Caches/datalad/sockets/6eed3025"', 'ssh:admin@Voxel-Forge.local']
2

And that is the last line of the log?
Did it halt? Did it exit (what exit code)? Please provide output of datalad wtf command so we know versions etc .

3

Yes, this is the last line of the log. The command would hang at this step. If I entered CTRL + C to interrupt the call, I get a password prompt. The password prompt would not take my password (100% it is correct).

datalad wtf doesn’t seem to be a proper system call on my install

datalad wtf
    usage: datalad [-h] [-l LEVEL] [--pbs-runner {condor}] [-C PATH] [--version]
                   [--dbg] [--idbg] [-c KEY=VALUE]
                   [--output-format {default,json,json_pp,tailored,'<template>']
                   [--report-status {success,failure,ok,notneeded,impossible,error}]
                   [--report-type {dataset,file}]
                   [--on-failure {ignore,continue,stop}]
                   [--run-before PLUGINSPEC [PLUGINSPEC ...]]
                   [--run-after PLUGINSPEC [PLUGINSPEC ...]] [--cmd]
                   {create,install,get,add,publish,uninstall,drop,remove,update,create-sibling,create-sibling-github,unlock,save,plugin,search,metadata,aggregate-metadata,test,crawl,crawl-init,ls,clean,add-archive-content,download-url,run,rerun,annotate-paths,clone,create-test-dataset,diff,siblings,sshrun,subdatasets}
                   ...
    datalad: error: invalid choice: 'wtf' (choose from 'create', 'install', 'get', 'add', 'publish', 'uninstall', 'drop', 'remove', 'update', 'create-sibling', 'create-sibling-github', 'unlock', 'save', 'plugin', 'search', 'metadata', 'aggregate-metadata', 'test', 'crawl', 'crawl-init', 'ls', 'clean', 'add-archive-content', 'download-url', 'run', 'rerun', 'annotate-paths', 'clone', 'create-test-dataset', 'diff', 'siblings', 'sshrun', 'subdatasets')

The version I’m running is

datalad --version
datalad 0.9.3

Copyright (c) 2013-2018 DataLad developers

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
4

Would it be too much to ask to update it to current 0.10.2 – there were a number of fixes since 0.9.3 in it, also relating to SSH connections IIRC.

5

Ok, upgraded to most recent version with no problems.

I tried running the same exact command again and was prompted to input a password. My password was rejected each time I attempted to put it in. I’m wondering now if this problem is related to my ssh settings. Here is my ssh config file:

Host *
AddKeysToAgent yes
UseKeychain yes
IdentityFile /Users/admin/.ssh/id_rsa
ServerAliveInterval 240

Oh and here is output of wtf

DataLad

Version : 0.10.2
Full version: 0.10.2

System

OS : posix Darwin 17.4.0 Darwin Kernel Version 17.4.0: Sun Dec 17 09:19:54 PST 2017; root:xnu-4570.41.2~1/RELEASE_X86_64
Distribution: 10.13.3/x86_64

Locale/Encoding

default : ascii
filesystem : utf-8
locale.prefered: UTF-8

Environment

LC_CTYPE=‘en_US.UTF-8’
PATH=’/usr/local/opt/qt/bin:/usr/local/bin:/Applications/NEURON-7.5/iv/x86_64/bin:/Applications/NEURON-7.5/nrn/x86_64/bin:/Users/admin/local/bin:/Users/admin/Anvil/opt/mrtrix3/bin:/usr/local/opt/qt5/bin:/Users/admin/Anvil/opt/freesurfer/bin:/Users/admin/Anvil/opt/freesurfer/fsfast/bin:/Users/admin/Anvil/opt/freesurfer/tktools:/Users/admin/Anvil/opt/fsl/bin:/Users/admin/Anvil/opt/freesurfer/mni/bin:/Users/admin/Anvil/opt/fsl/bin:/Users/admin/Anvil/opt/mICA_Toolbox:/Users/admin/Anvil/opt/ants/bin:/Users/admin/Anvil/opt/MATLAB_R2017b.app/bin:/Users/admin/miniconda2/bin:/usr/local/opt/coreutils/libexec/gnubin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/opt/sge//bin/darwin-x64:/Users/admin/Anvil/opt/abin:/Users/admin/java:/Volumes/Anvil/opt/abin:/Users/admin/anvilopt’
LANG=‘en_US.UTF-8’
PYTHONPATH=’/Users/admin/Anvil/opt/mrtrix3/lib:/usr/local/lib/mrtrix3:/Users/admin/Anvil/opt/freesurfer/bin:/Users/admin/Anvil/opt/freesurfer/fsfast/bin:/Users/admin/Anvil/opt/freesurfer/tktools:/Users/admin/Anvil/opt/fsl/bin:/Users/admin/Anvil/opt/freesurfer/mni/bin:/Users/admin/Anvil/opt/fsl/bin:/Users/admin/Anvil/opt/mICA_Toolbox:/Users/admin/Anvil/opt/ants/bin:/Users/admin/Anvil/opt/MATLAB_R2017b.app/bin:/Users/admin/miniconda2/bin:/usr/local/opt/coreutils/libexec/gnubin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/opt/sge//bin/darwin-x64:/Users/admin/Anvil/opt/abin’
LC_ALL=‘en_US.UTF-8’

Externals

appdirs=1.4.3
boto=2.48.0
cmd:annex=6.20180316
cmd:git=2.16.1
cmd:system-git=2.16.1
cmd:system-ssh=7.6p1
git=2.1.9
gitdb=2.0.3
humanize=0.5.1
iso8601=0.1.12
msgpack=0.5.6
requests=2.18.4
six=1.11.0
tqdm=4.22.0
wrapt=1.10.11

Installed extensions

Known metadata extractors

exif: BROKEN (No module named exifread [exif.py::16])
annex: OK
datacite: OK
image: OK
datalad_core: OK
frictionless_datapackage: OK
datalad_rfc822: OK
audio: BROKEN (No module named mutagen [audio.py::17])
xmp: BROKEN (No module named libxmp.utils [xmp.py::20])

Configuration

<Sensitive data, use -s=some or -s=all to show>

6

ah – sorry, I’ve missed a misspecification in the original report:

 ssh:admin@Voxel-Forge.local:/Users/admin/Anvil/sync/projects/central-repository/community-life-adolescent-development.git

should be either

 ssh://admin@Voxel-Forge.local:/Users/admin/Anvil/sync/projects/central-repository/community-life-adolescent-development.git

or just

admin@Voxel-Forge.local:/Users/admin/Anvil/sync/projects/central-repository/community-life-adolescent-development.git
1 Like
7

That’ll do it! Installs seem to work fine now.