Datalad push --to gin errors

Hi Datalad Team,

I recently updated to Datalad v0.17.9, and used the very useful datalad-create-sibling-gin command to set up gin repos as siblings for my superdataset and its subdatasets.

I succesfully datalad pushed one of my subdatasets, but when trying another one (containing mriqc output) using the command datalad push --to gin I get the following error message, regardless of whether I include the --data anything or -f checkdatapresent options.

I also noted that during the pushing, I sometimes get the Connection to gin.g-node.org closed by remote host message, after which pushing resumes automatically (sometimes I need to re-enter my GIN ssh password though).

Any suggestions on how to fix this would be welcome!

Thanks,

Lukas

CommandError: ‘git -c diff.ignoreSubmodules=none push --progress --porcelain gin master:master git-annex:git-annex’ failed with exitcode 128 under /data/proj_erythritol/proj_erythritol_4a/mriqc
Delta compression using up to 64 threads
CommandError: ‘ssh -o ControlPath=/home/luna.kuleuven.be/u0027997/.cache/datalad/sockets/53dce49f git@gin.g-node.org ‘git-receive-pack ‘"’"’/labgas/proj_erythritol_4a-mriqc.git’"’"’’’ failed with exitcode 255
send-pack: unexpected disconnect while reading sideband packet
fatal: the remote end hung up unexpectedly

Just adding here that in another subdataset, I got a different error when executing datalad push --to gin --data anything -f checkdatapresent:

CommandError: ‘git -c diff.ignoreSubmodules=none annex copy --batch -z --to gin --json --json-error-messages --json-progress -c annex.dotfiles=true’ failed with exitcode 1 under /data/proj_erythritol/proj_erythritol_4a/secondlevel [info keys: stdout_json]
to gin…
Transfer failed
Transfer failed [14 times]
verification of content failed
verification of content failed
verification of content failed
verification of content failed
verification of content failed
verification of content failed
verification of content failed
verification of content failed
verification of content failed
verification of content failed
verification of content failed
verification of content failed
verification of content failed
verification of content failed
verification of content failed
verification of content failed
verification of content failed
verification of content failed
verification of content failed
verification of content failed
verification of content failed
verification of content failed
verification of content failed
verification of content failed
verification of content failed
verification of content failed
verification of content failed
verification of content failed
copy: 14 failed

apparently I saw that error before: FTR: git push error - send-pack: unexpected disconnect while reading sideband packet · Issue #6130 · datalad/datalad · GitHub but it remained a mystery .

What is also your git-annex version? there were recent tune ups (like initremote type=git is not working for unkn reason) which might relate.

What do you see if you do manually

git annex copy --to=gin SOMESAMPLEFILE
git push gin master:master

Thanks a lot for the quick response Yarik!

Here is the version info of git-annex (output of git annex version)

git-annex version: 10.20220822-1~ndall+1
build flags: Assistant Webapp Pairing Inotify DBus DesktopNotify TorrentParser MagicMime Feeds Testsuite S3 WebDAV
dependency versions: aws-0.22 bloomfilter-2.0.1.0 cryptonite-0.26 DAV-1.3.4 feed-1.3.0.1 ghc-8.8.4 http-client-0.6.4.1 persistent-sqlite-2.10.6.2 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.1.0
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL X*
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs httpalso borg hook external
operating system: linux x86_64
supported repository versions: 8 9 10
upgrade supported from repository versions: 0 1 2 3 4 5 6 7 8 9 10

The repo is indeed private, but repos for the subdatasets I managed to push without problems are private too, so this is probably not the main issue?

Manually running git annex copy and git push resulted in a similar error

u0027997@gbw-s-labgas01:/data/proj_erythritol/proj_erythritol_4a/mriqc$ git annex copy --to=gin sub-003_T1w.html
u0027997@gbw-s-labgas01:/data/proj_erythritol/proj_erythritol_4a/mriqc$ git push gin master:master
Enter passphrase for key ‘/home/luna.kuleuven.be/u0027997/.ssh/id_ed25519’:
Enumerating objects: 660, done.
Counting objects: 100% (660/660), done.
Delta compression using up to 64 threads
Connection to gin.g-node.org closed by remote host.
fatal: the remote end hung up unexpectedly
Compressing objects: 100% (624/624), done.
fatal: the remote end hung up unexpectedly

Any idea what the cause of the other error may be? That one looks different…

Thanks a ton,

Lukas

On another note, I also get the following error message when running several datalad commands, without however the commands being prevented from exiting correctly (at least so it seems)

[ERROR ] Internal error, cannot import interface ‘datalad_hirni.commands.import_dicoms’: ImportError(cannot import name ‘AnnotatePaths’ from ‘datalad.interface.annotate_paths’ (/opt/anaconda3/lib/python3.8/site-packages/datalad/interface/annotate_paths.py))
[ERROR ] Skipping unusable command interface ‘datalad_hirni.commands.import_dicoms.ImportDicoms’ from extension ‘hirni’
[ERROR ] Internal error, cannot import interface ‘datalad_hirni.commands.spec4anything’: ImportError(cannot import name ‘AnnotatePaths’ from ‘datalad.interface.annotate_paths’ (/opt/anaconda3/lib/python3.8/site-packages/datalad/interface/annotate_paths.py))
[ERROR ] Skipping unusable command interface ‘datalad_hirni.commands.spec4anything.Spec4Anything’ from extension ‘hirni’
[ERROR ] Internal error, cannot import interface ‘datalad_hirni.commands.dicom2spec’: ImportError(cannot import name ‘AnnotatePaths’ from ‘datalad.interface.annotate_paths’ (/opt/anaconda3/lib/python3.8/site-packages/datalad/interface/annotate_paths.py))
[ERROR ] Skipping unusable command interface ‘datalad_hirni.commands.dicom2spec.Dicom2Spec’ from extension ‘hirni’

Any idea whether I should do something about this, and what can be done if yes?

Thanks again and happy Thanksgiving,

Lukas