I’d like to use datalad to manage a large neuroimaging dataset stored on a NAS device, but the shared filesystem doesn’t offer fine-grained permissions which causes an error in datalad (below). Is there any way to address this? Thanks a bunch!
$ datalad create -f dl_test
[INFO ] Creating a new annex repo at /some/network/path/dl_test
[ERROR ] Cmd('/usr/lib/git-annex.linux/git') failed due to: exit code(128)
| cmdline: /usr/lib/git-annex.linux/git init
| stderr: 'error: chmod on /mnt/warren_lab/user_docs/dewarren/scratch/dl_test/.git/config.lock failed: Permission denied
| fatal: could not set 'core.filemode' to 'false'' [cmd.py:execute:821]
[ERROR ] Cmd('/usr/lib/git-annex.linux/git') failed due to: exit code(128)
| cmdline: /usr/lib/git-annex.linux/git init
| stderr: 'error: chmod on /mnt/warren_lab/user_docs/dewarren/scratch/dl_test/.git/config.lock failed: Permission denied
| fatal: could not set 'core.filemode' to 'false'' [cmd.py:execute:821] (GitCommandError)
Not sure if datalad could work on such type of a mount - it seems that even git can’t operate there for a basic “git init”.
Can you “git init” a new repository there without an error?
what is the exit code of “git init” command? (you could discover it by “echo $?” right after). If non-0, it means that git errored out. I wouldn’t rely then on it operating correctly (even if seemingly full tree is under .git there)
well – in the original case it was 128 so “go figure”
my point is that such NAS mount isn’t “fully supported” to say the least. Could you provide more information - for that path - what is the file system, mount options. What is the OS, datalad version, output of datalad plugin wtf could be of help
I guess details about the remote location is what would be interesting. But again - if even git fails to init there cleanly, I am not sure if we could do anything about it, and we shouldn’t just ignore… so on remote location, where you are trying to create-sibling on
what is the version of git?
does plain git init work and exits with 0 status code?
if that remote location also has standalone build of annex installed, could you check what is git version shipped with it, and either it fails the same way?
details about that remote file system would be of interest as well
It would be informative to see the output of the mount command for the shared drive on the local system. In order to get a handle on this issue, we need to figure out why git init seemingly fails for datalad (code 128), but not when run manually. Was the directory confirmed empty (no hidden content) in both cases? It would be good to have all tests above include the actual creation of the repo directory. Likewise informative would be to see the output of ls -la on the parent directory.
That being said a local cifs could look all kinds of ways on the local system and (based on server-side) configuration behave in seemingly unexpected way. The more we get to know about both ends, the easier it will be to get a handle on this.
Here are a few more pointers. All done under a cifs mount with the following parameters (minus banking passwords that I redacted, etc…):
//remote*** on /mnt/smb/fritz type cifs (rw,relatime,vers=1.0,cache=strict,username=***,domain=,uid=1000,forceuid,gid=1000,forcegid,addr=***,soft,unix,posixpaths,serverino,mapposix,acl,noperm,rsize=61440,wsize=65536,echo_interval=60,actimeo=1)
And here is what git/datalad have to say:
mih@meiner /mnt/smb/fritz % mkdir democifs
mih@meiner /mnt/smb/fritz % cd democifs
mih@meiner /mnt/smb/fritz/democifs % git init
Initialized empty Git repository in /mnt/smb/fritz/democifs/.git/
mih@meiner /mnt/smb/fritz/democifs (git)-[master] % cd ..
mih@meiner /mnt/smb/fritz % rm -rf democifs
mih@meiner /mnt/smb/fritz % . ~/env/datalad3-dev/bin/activate
(datalad3-dev) mih@meiner /mnt/smb/fritz % datalad create democifs
[INFO ] Creating a new annex repo at /mnt/smb/fritz/democifs
[INFO ] Detected a crippled filesystem.
[INFO ] Disabling core.symlinks.
[INFO ] Enabling direct mode.
create(ok): /mnt/smb/fritz/democifs (dataset)
(datalad3-dev) mih@meiner /mnt/smb/fritz % rm -rf democifs
(datalad3-dev) mih@meiner /mnt/smb/fritz % datalad create democifs --annex-version 6
[INFO ] Creating a new annex repo at /mnt/smb/fritz/democifs
[INFO ] Detected a crippled filesystem.
[INFO ] Disabling core.symlinks.
[INFO ] Entering an adjusted branch where files are unlocked as this filesystem does not support locked files.
[INFO ] Switched to branch 'adjusted/master(unlocked)'
create(ok): /mnt/smb/fritz/democifs (dataset)
It looks like my problem was with permissions in the local folder.
I had created a new unix user specifically for datalad to ssh-key into the remote (during create-sibling) and had incorrectly set the local folder permissions.
Thanks for the help!
Trying this with a new, UNIX-flavored CIFS share on our FreeNAS system helped a bit — git seems to be working without issue — but datalad still throws an error (see below). Of note, my system’s mount.cifs command will take most of the parameters @eknahm mentioned, but not unix or relatime. If anyone has suggestions, I’d be happy to try again after Thanksgiving.
[ dewarren@hostname ] /mnt/dataladtest
$ mkdir democifs
[ dewarren@hostname ] /mnt/dataladtest
$ cd democifs
[ dewarren@hostname ] /mnt/dataladtest/democifs
$ git init
Initialized empty Git repository in /mnt/dataladtest/democifs/.git/
[ dewarren@hostname ] /mnt/dataladtest/democifs
$ cd ..
[ dewarren@hostname ] /mnt/dataladtest
$ rm -rf democifs/
[ dewarren@hostname ] /mnt/dataladtest
$ datalad create democifs
[INFO ] Creating a new annex repo at /mnt/dataladtest/democifs
[INFO ] warning: unable to unlink '.git/objects/4b/tmp_obj_BwSAEc': Permission denied
[INFO ] warning: unable to unlink '.git/objects/fc/tmp_obj_ET5qUd': Permission denied
[INFO ] git-annex: .git/annex/misctmp/gaprobe: removeLink: invalid argument (Invalid argument)
[INFO ] git-annex: init: 1 failed
Failed to run ['git', '-c', 'receive.autogc=0', '-c', 'gc.auto=0', 'annex', 'init'] under '/mnt/dataladtest/democifs'. Exit code=1.
init
failed
so “go figure”