'Transfer failed': error during datalad get from remote ssh server

Hi everyone!

My problem:
I’ve been trying to download data from my remote repo using datalad get /path/to/file and obtained the following uninformative output:

get(error): sub-00/ses-post/anat/sub-00_ses-post_T1w.nii.gz (file) [Transfer failed Transfer failed Transfer failed]

where sub-00/ses-post/anat/sub-00_ses-post_T1w.nii.gz is a file, which I’ve been trying to download.

“Remote” repo is situated on my local machine, and i run datalad get after connecting to my server through ssh (where i have the reverce access to my local machine).
So at the following this computational server will be mentioned as “client” (machine where I need data from datalad repo), and local machine (where dataset is physically situated) will be mentioned as “server”.

How to reproduce this problem?
On client i’ve run the following commands:

#datalad install -s <username>@<ip>:path/to/repodir
#cd repodir
#datalad get sub-00/ses-post/anat/sub-00_ses-post_T1w.nii.gz

What version of DataLad am I using (run datalad --version)? On what operating system (consider running datalad wtf)?

Currently I am using DataLad version 0.15.4, and git-annex version 8.20211118-g23ee48898 on client. The client OS is CentOS 7.9.2009. I’ve also tryied Ubuntu 20.04.2 with the same result.

UPD: On server I have Datalad 0.15.1, git-annex 8.20211028-g581d4e9 and Ubuntu 18.04.6

Current results of my attempts to find the source of the problem

Using flag --log-level debug I’ve found out that datalad called git annex get as it should be. Than git-annex have managed with the connection to my server (it asked me password, and accepted it) and then bring back to datalad another useless json with the following content:

  "command": "get",
  "wanted": [],
  "note": "from origin...\nUnable to access these remotes: origin",
  "success": false,
  "input": [
  "key": "MD5E-s9600993--bb02ba4212e654054b53491b77e997ac.nii.gz",
  "error-messages": [
    "  Transfer failed",
    "  Transfer failed",
    "  Transfer failed"
  "file": "sub-00/ses-post/anat/sub-00_ses-post_T1w.nii.gz"

After moving further and using git annex get --debug sub-00/ses-post/anat/sub-00_ses-post_T1w.nii.gz I’ve seen following message about the process (1071278) with the failure return code (the last two lines):

[2022-01-27 23:26:15.27735512] (Utility.Process) process [1071274] read: ssh ["-S",".git/annex/ssh/dmitriy_mikhailov@","-o","ControlMaster=auto","-o","ControlPersist=yes","-n","-T","dmitriy_mikhailov@","true"]
dmitriy_mikhailov@'s password: 
[2022-01-27 23:26:21.769245319] (Utility.Process) process [1071274] done ExitSuccess
[2022-01-27 23:26:21.770827896] (Utility.Process) process [1071278] chat: ssh ["dmitriy_mikhailov@","-S",".git/annex/ssh/dmitriy_mikhailov@","-o","ControlMaster=auto","-o","ControlPersist=yes","-T","git-annex-shell 'p2pstdio' '/home-local/dmitriy_mikhailov/fMRI_data/pirogov_aphazy_data' '--debug' '847f16f5-d7d4-4d17-8323-cf6d72c3102a' --uuid 261a954c-c87f-4f92-9df8-436cc412819f"]
[2022-01-27 23:26:21.801630603] (Utility.Process) process [1071278] done ExitFailure 1

And at this point I’ve got lost. I have no idea where the problem came from. I’am not so experienced ssh user, and I don’t understand the underlying logic of git-annex get working. Could anyone here help me?

P.S. I can send the full versions of datalad get, git annex get debug output, and output of datalad wtf, but I can’t attach these files to this question because neurostars doesn’t allow to do this by newcomer like me.

Nice troubleshooting! Quick one: is git annex installed on server side so it is available to non login session? Ie does ssh git-annex-shell works?

Thanks, I’ve just spent… some time with this)

Sorry, I don’t really understand how I can force ssh to run script in nologin session… Command ssh <script-name> in my case try to connect with using user name I’m currently logged in (on client machine). Server can’t find any user with client’s username and return permission denied.

But if I use correct username and password it works as if I were running git-annex-shell on server directly. Git, git-annex and datalad - all of them works on the server.

Hmm… Could the problem be related to datalad (or git-annex) version difference between server and client? I have datalad 0.15.1 and git-annex 8.20211028-g581d4e9 on server…

You mention that server is CentOS, what is client OS?

Oh, sorry, I meant client machine.
As a client (machine where I want to get data) I’ve used CentOS 7.9.2009 and Ubuntu 20.04.2
And as a server (machine where data is physically situated) I’ve used Ubuntu 18.04.6

UPD: Corrected in question’s text and fixed client-server terminology.

I was asking because recalled we had similarish problem on windows recently: get: no passphrase ask for the 2nd ssh invocation

I guess I would have first tried to run those two commands manually and see if 2nd one gives more output to “digest”:

  • establish socket connection
ssh "-S" ".git/annex/ssh/dmitriy_mikhailov@" "-o" "ControlMaster=auto" "-o" "ControlPersist=yes" "-n" "-T" "dmitriy_mikhailov@" "true"
  • connect to it using that socket
ssh "dmitriy_mikhailov@" "-S" ".git/annex/ssh/dmitriy_mikhailov@" "-o" "ControlMaster=auto" "-o" "ControlPersist=yes" "-T" "git-annex-shell 'p2pstdio' '/home-local/dmitriy_mikhailov/fMRI_data/pirogov_aphazy_data' '--debug' '847f16f5-d7d4-4d17-8323-cf6d72c3102a' --uuid 261a954c-c87f-4f92-9df8-436cc412819f"

Also what is the file system for the repo, i.e. where that .git/annex/ssh/dmitriy_mikhailov@ is located – I wonder if may be socket is non functional somehow… (although probably there would have been some other error or a password prompt)

Yes, thanks) the answer is here:

First of all there is another little thing I’ve overlooked: git annex get make three ssh invocation: in addition to invokation which establish socket connection there was another attempt to do this was distinguished by an additional parameter -o BatchMode=true.

I runned all three commands with a verbose (-v) flag:

ssh -v -o BatchMode=true -S .git/annex/ssh/dmitriy_mikhailov@ -o ControlMaster=auto -o ControlPersist=yes -n -T dmitriy_mikhailov@ true

ssh -v -S .git/annex/ssh/dmitriy_mikhailov@ -o ControlMaster=auto -o ControlPersist=yes -n -T dmitriy_mikhailov@ true

ssh -v dmitriy_mikhailov@ -S .git/annex/ssh/dmitriy_mikhailov@ -o ControlMaster=auto -o ControlPersist=yes -T 'git-annex-shell p2pstdio /home-local/dmitriy_mikhailov/fMRI_data/pirogov_aphazy_data --debug 847f16f5-d7d4-4d17-8323-cf6d72c3102a --uuid 261a954c-c87f-4f92-9df8-436cc412819f'

I’ve saved all outputs to the repo: https://github.com/MihailovDmitriy/git_annex_logs

What interesting have I see.

  1. First invocation was failed with permission denied without even asking for a password.

  2. Second invocation ask me for the password, and then finished correctly with a creation of the socket file at the repodir on client (/repodir/.git/annex/ssh/dmitriy_mikhailov@

  3. Third command send the command to the server and get an error:

    debug1: Sending command: git-annex-shell p2pstdio /home-local/dmitriy_mikhailov/fMRI_data/pirogov_aphazy_data --debug 847f16f5-d7d4-4d17-8323-cf6d72c3102a --uuid 261a954c-c87f-4f92-9df8-436cc412819f
    fatal: Run with no arguments or with -c cmd
    git-annex-shell: git-shell failed

At the other hand command

git-annex-shell p2pstdio /home-local/dmitriy_mikhailov/fMRI_data/pirogov_aphazy_data --debug 847f16f5-d7d4-4d17-8323-cf6d72c3102a --uuid 261a954c-c87f-4f92-9df8-436cc412819f

works perfectly on the server with the following stdout:

[2022-01-28 17:49:21.524797] (P2P.IO) [ThreadId 4] P2P > AUTH-SUCCESS 261a954c-c87f-4f92-9df8-436cc412819f
AUTH-SUCCESS 261a954c-c87f-4f92-9df8-436cc412819f

probably the culprit here… Found get over ssh fails with fd:19: hClose: resource vanished which seems to boil down back then to outdated git-annex on server side (not your case) and prompt printed by server upon non-interactive session (didn’t spot anything in your logs). Can you confirm the version of git annex you get whenever you do non-interactive login? something like

ssh -v dmitriy_mikhailov@ -S .git/annex/ssh/dmitriy_mikhailov@ -o ControlMaster=auto -o ControlPersist=yes git annex version


Hmm… and really, the version is different from the one that can be seen by calling git annex version on the server interactivly:

debug1: Sending command: git annex version
git-annex version: 6.20180227
build flags: Assistant Webapp Pairing S3(multipartupload)(storageclasses) WebDAV Inotify DBus DesktopNotify ConcurrentOutput TorrentParser MagicMime Feeds Testsuite
dependency versions: aws-0.16 bloomfilter- cryptonite-0.23 DAV-1.3.1 feed- ghc-8.0.2 http-client- persistent-sqlite-2.6.3 torrent-10000.1.1 uuid-1.3.13 yesod-1.4.5
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 SHA1E SHA1 MD5E MD5 WORM URL

Do you know why this happens and how can I correct this?

bingo! so in your interactive session if you use which -a git-annex you will see that there is some other one which is available due to your custom PATH in that session…
the easiest way – place tune up of PATH into your ~/.bashrc so it points to the desired location of git-annex (if you use bash as your login shell)

longer term – please ask that elderly system wide git-annex to be updated.

Oh, god…) That was on surface… Thanks a lot for your help, I’ve add proper git-annex path into /etc/enviroment and datalad get start working.