Permission issues with datalad remove and datalad drop

lukasvo76 · April 20, 2023, 11:24am

Hi datalad team,

I run into the following two possibly related problems when trying to drop/remove subdatasets from a superdataset

1. datalad remove -d . firstlevel error
The first step seems to be datalad drop, which works fine, but the actual removal in the next step generates [ERROR ] [Errno 1] Operation not permitted: '/data/proj_bitter-reward/firstlevel' (PermissionError) for me (member of group) as well as the owner of the subdataset, even though permissions for the subdataset/dir look similar to other subdatasets for which this operation did work

2. datalad drop --reckless availability
When running this command from a subdataset, nothing happens, and when I try dropping one file/dir,
either nothing happens, or I get notneeded messages even though files are not dropped.

Any help would be appreciated!

Thanks,

Lukas

adina · April 21, 2023, 8:09am

Hi @lukasvo76,

Thanks for the report. Here’s how I understand the setup, please let me know if I got anything wrong :

you have a hierarchy of datasets in some central place (/data/proj_bitter-reward)
you and other team members perform operations in this central dataset (i.e., not in user-specific clones)
there are subdatasets in this hierarchy that you can drop, and some/one subdataset where this shows the PermissionError

For the “permission denied” error, one thing that comes to my mind are potential problems if multiple people work in the same dataset, but it isn’t configured as a shared Git repository. Do you know if (all/any of) the datasets have been configured (later or at creation) specifically to be shared, e.g., with --shared=group? I would also be curious if these permission denied errors only happen for drop/remove operations, or if they also show up for, for example, save.

The drop --reckless availability error could indeed be related, but please try running the drop command with datalad -f json_pp <drop and the parameters you used>? There are not many cases where drop would say notneeded, and with -f json_pp you should get an additional message with some indication of its reasoning. In addition, please also run your drop command with debug output (datalad -l debug <drop and the parameters you used>) - maybe this has additional insights as well.

Lastly, could you report the datalad and git-annex versions? Ideally, run datalad wtf inside of the affected superdataset.

lukasvo76 · April 21, 2023, 8:30am

Hi @adina ,

Thanks a lot!

You understand the setup correctly

The permission denied error seems to be specific to the drop/remove operation, we have not experienced saving problems.

When creating super- and subdatasets, we typically always run git config core.sharedrepository world, but the team member who created this particular subdataset may have forgotten, I am not sure. Interestingly, when trying to run that command in this particular subdataset now, I unexpectedly get a fatal: not in a git directory error, contrary to running this in the superdataset. Hence, something may be wrong here, but there is a .git and .datalad dir inside.

The superdataset and its subdataset have also been succesfully pushed to GIN before, but I already removed the gin sibling for this sub-f dataset since I wanted to remove it from the local superdataset.

Here is the output from the first suggestion for drop

u0027997@gbw-s-labgas01:/data/proj_bitter-reward/firstlevel$ datalad -f json_pp drop --reckless availability
{
“action”: “drop”,
“message”: [
“nothing to drop from %s”,
“/data/proj_bitter-reward/firstlevel”
],
“path”: “/data/proj_bitter-reward/firstlevel”,
“refds”: “/data/proj_bitter-reward/firstlevel”,
“status”: “notneeded”,
“type”: “directory”
}

…and here the second one

u0027997@gbw-s-labgas01:/data/proj_bitter-reward/firstlevel$ datalad -l debug drop --reckless availability
[DEBUG ] Command line args 1st pass for DataLad 0.17.9. Parsed: Namespace() Unparsed: [‘drop’, ‘–reckless’, ‘availability’]
[DEBUG ] Building doc for <class ‘datalad.core.local.status.Status’>
[DEBUG ] Building doc for <class ‘datalad.distributed.drop.Drop’>
[DEBUG ] Parsing known args among [’/opt/anaconda3/bin/datalad’, ‘-l’, ‘debug’, ‘drop’, ‘–reckless’, ‘availability’]
[DEBUG ] Determined class of decorated function: <class ‘datalad.distributed.drop.Drop’>
[DEBUG ] Resolved dataset to drop: /data/proj_bitter-reward/firstlevel
[DEBUG ] Run [‘git’, ‘config’, ‘-z’, ‘-l’, ‘–show-origin’] (protocol_class=StdOutErrCapture) (cwd=/data/proj_bitter-reward/firstlevel)
[DEBUG ] Finished [‘git’, ‘config’, ‘-z’, ‘-l’, ‘–show-origin’] with status 0
[DEBUG ] Run [‘git’, ‘config’, ‘-z’, ‘-l’, ‘–show-origin’, ‘–file’, ‘/data/proj_bitter-reward/firstlevel/.datalad/config’] (protocol_class=StdOutErrCapture) (cwd=/data/proj_bitter-reward/firstlevel)
[DEBUG ] Finished [‘git’, ‘config’, ‘-z’, ‘-l’, ‘–show-origin’, ‘–file’, ‘/data/proj_bitter-reward/firstlevel/.datalad/config’] with status 0
[DEBUG ] Discovered 1 datasets to drop (from)
[DEBUG ] Starting to drop filecontent at Dataset(/data/proj_bitter-reward/firstlevel)
[DEBUG ] Start dropping for Dataset(/data/proj_bitter-reward/firstlevel)
[DEBUG ] Determined class of decorated function: <class ‘datalad.core.local.status.Status’>
[DEBUG ] Resolved dataset to report status: /data/proj_bitter-reward/firstlevel
[DEBUG ] Querying AnnexRepo(/data/proj_bitter-reward/firstlevel).diffstatus() for paths: None
[DEBUG ] Run [‘git’, ‘-c’, ‘diff.ignoreSubmodules=none’, ‘rev-parse’, ‘–quiet’, ‘–verify’, ‘HEAD^{commit}’] (protocol_class=GeneratorStdOutErrCapture) (cwd=/data/proj_bitter-reward/firstlevel)
[DEBUG ] AnnexRepo(/data/proj_bitter-reward/firstlevel).get_content_info(…)
[DEBUG ] Query repo: [‘ls-files’, ‘–stage’, ‘-z’, ‘–exclude-standard’, ‘-o’, ‘–directory’, ‘–no-empty-directory’]
[DEBUG ] Run [‘git’, ‘-c’, ‘diff.ignoreSubmodules=none’, ‘ls-files’, ‘–stage’, ‘-z’, ‘–exclude-standard’, ‘-o’, ‘–directory’, ‘–no-empty-directory’] (protocol_class=GeneratorStdOutErrCapture) (cwd=/data/proj_bitter-reward/firstlevel)
[DEBUG ] Done query repo: [‘ls-files’, ‘–stage’, ‘-z’, ‘–exclude-standard’, ‘-o’, ‘–directory’, ‘–no-empty-directory’]
[DEBUG ] Done AnnexRepo(/data/proj_bitter-reward/firstlevel).get_content_info(…)
[DEBUG ] Run [‘git’, ‘-c’, ‘diff.ignoreSubmodules=none’, ‘ls-files’, ‘-z’, ‘-m’, ‘-d’] (protocol_class=GeneratorStdOutErrCapture) (cwd=/data/proj_bitter-reward/firstlevel)
[DEBUG ] AnnexRepo(/data/proj_bitter-reward/firstlevel).get_content_info(…)
[DEBUG ] Query repo: [‘ls-tree’, ‘HEAD’, ‘-z’, ‘-r’, ‘–full-tree’, ‘-l’]
[DEBUG ] Run [‘git’, ‘-c’, ‘diff.ignoreSubmodules=none’, ‘ls-tree’, ‘HEAD’, ‘-z’, ‘-r’, ‘–full-tree’, ‘-l’] (protocol_class=GeneratorStdOutErrCapture) (cwd=/data/proj_bitter-reward/firstlevel)
[DEBUG ] Done query repo: [‘ls-tree’, ‘HEAD’, ‘-z’, ‘-r’, ‘–full-tree’, ‘-l’]
[DEBUG ] Done AnnexRepo(/data/proj_bitter-reward/firstlevel).get_content_info(…)
[DEBUG ] Run [‘git’, ‘annex’, ‘version’, ‘–raw’] (protocol_class=StdOutErrCapture) (cwd=None)
[DEBUG ] Finished [‘git’, ‘annex’, ‘version’, ‘–raw’] with status 0
[DEBUG ] Run [‘git’, ‘-c’, ‘diff.ignoreSubmodules=none’, ‘annex’, ‘drop’, ‘–force’, ‘–json’, ‘–json-error-messages’, ‘-c’, ‘annex.dotfiles=true’, ‘–’, ‘.’] (protocol_class=GeneratorAnnexJsonNoStderrProtocol) (cwd=/data/proj_bitter-reward/firstlevel)
[DEBUG ] nothing to drop from /data/proj_bitter-reward/firstlevel [drop(/data/proj_bitter-reward/firstlevel)]
[DEBUG ] Finished dropping filecontent at Dataset(/data/proj_bitter-reward/firstlevel)

Finally, here is some relevant output from datalad wtf in the superdataset, lmk if you need more

datalad

version: 0.17.9

dataset

branches:

git-annex@cc502fe

master@f0a881d

id: 9a242ace-8b4d-4405-8d8d-0b195590b30c

metadata: <SENSITIVE, report disabled by configuration>

path: /data/proj_bitter-reward

repo: AnnexRepo

dependencies

annexremote: 1.5.0

boto: 2.49.0

cmd:7z: 16.02

cmd:annex: 10.20220822-1~ndall+1

cmd:bundled-git: 2.30.2

cmd:git: 2.30.2

cmd:ssh: 8.2p1

cmd:system-git: 2.39.0

cmd:system-ssh: 8.2p1

exifread: 2.3.2

humanize: 3.11.0

iso8601: 0.1.16

keyring: 23.2.1

keyrings.alt: 4.1.0

msgpack: 1.0.0

mutagen: 1.45.1

platformdirs: 2.4.0

requests: 2.26.0

Thanks a lot in advance!

Best wishes,

Lukas

adina · April 21, 2023, 8:59am

When creating super- and subdatasets, we typically always run git config core.sharedrepository world, but the team member who created this particular subdataset may have forgotten, I am not sure.

You should be able to check, can you take a look in the subdatasets .git/config file for the core.sharedRepository key?

Interestingly, when trying to run that command in this particular subdataset now, I unexpectedly get a fatal: not in a git directory error, contrary to running this in the superdataset. Hence, something may be wrong here, but there is a .git and .datalad dir inside.

This looks like its a Git safeguard, I have seen this before when calling Git in repositories owned by another user that haven’t been declared as “safe directories”. A colleague recently wrote up the details about this: KBI0001: Security considerations for accessing shared datasets on multi-user machines — PsyInf Knowledge Base documentation

I would suggest checking these two things first, as it likely explains the problems you’re describing.

With regard to the drop output: Whoops, sorry, I misled you. Your commands don’t include the path you’d like to drop, so the command reports “nothing to drop”. Can you include the paths to the files that you were trying to drop as well in the command (ie. datalad -f json_pp drop --reckless availability <filepath>)? Just one file is enough for demo purposes.

With regard to wtf, what I was specifically interested was file system information, because I recall an old issue were NFS shares did not drop file content even though they were requested to. I think we have enough leads already to make that info not a top priority, but in case the leads from above don’t help, can you include the info what file system you’re running this on?

lukasvo76 · April 21, 2023, 11:42am

Thanks a ton again @adina!

Here is the content of .git/config, hence sharedrepository key looks fine

[core]
repositoryformatversion = 0
filemode = true
bare = false
logallrefupdates = true
sharedrepository = world
[annex]
uuid = 902a8fee-fe0f-4401-8a5f-1fbfc07d3b0e
version = 10
[filter “annex”]
smudge = git-annex smudge – %f
clean = git-annex smudge --clean – %f
process = git-annex filter-process
[remote “gin”]
annex-ignore = false
url = https://gin.g-node.org/labgas/proj_bitter-reward-firstlevel
fetch = +refs/heads/:refs/remotes/gin/
annexurl = git@gin.g-node.org:/labgas/proj_bitter-reward-firstlevel.git
pushurl = git@gin.g-node.org:/labgas/proj_bitter-reward-firstlevel.git
annex-uuid = adf72665-9fcd-4983-b9da-d92f6f455f9e

I added the subdataset as a safe directory, but still getting the same error when trying to datalad remove it unfortunately.

When adding a path to a single file to the drop command, I still get notneeded, so it looks like things did get dropped at some point?

u0027997@gbw-s-labgas01:/data/proj_bitter-reward/firstlevel$ datalad -f json_pp drop model_1_food_images/sub-001/beta_0001.nii --reckless availability
{
“action”: “drop”,
“message”: “no annex’ed content”,
“path”: “/data/proj_bitter-reward/firstlevel/model_1_food_images/sub-001/beta_0001.nii”,
“refds”: “/data/proj_bitter-reward/firstlevel”,
“status”: “notneeded”,
“type”: “file”
}

Here is the system info from datalad wtf

system

distribution: ubuntu/20.04/focal

encoding:

default: utf-8

filesystem: utf-8

locale.prefered: UTF-8

filesystem:

CWD:

mount_opts: rw,relatime

path: /data/proj_bitter-reward/firstlevel

type: ext4

HOME:

mount_opts: rw,relatime

path: /home/luna.kuleuven.be/u0027997

type: ext4

TMP:

mount_opts: rw,relatime

path: /tmp

type: ext4

max_path_length: 291

name: Linux

release: 5.4.0-146-generic

type: posix

version: #163-Ubuntu SMP Fri Mar 17 18:26:02 UTC 2023

Thanks again!

Lukas

lukasvo76 · May 3, 2023, 8:03am

Hi @adina, just wanted to remind you of this issue!

Wondering what would happen if I would just use rm -r subdataset from the superdataset and then datalad save the superdataset?

I guess the entire firstlevel subdataset would remain present in git (annex) of the superdataset (and continue to use disk space there)?

Also wondering whether this approach would cause trouble when I want to recreate the firstlevel subdataset later (which I do want to do)?

Thanks,

Lukas

cmo · May 4, 2023, 1:20pm

Hi @lukasvo76 , we have created a support issue in our knowledge base: Permission issues with datalad remove and datalad drop - #5 by lukasvo76. That is a good thing, because it exposes your question to the complete datalad team. That is also one of the reasons, why I am writing you today instead of @adina. Having said that, I am looking into the issue right now, and will hopefully soon get back to you.

cmo · May 4, 2023, 1:56pm

Hi again. I am not sure that this will help us to fix the issue, but could you provide owner-ids and permission flags, e.g. with (ls -l) for the superset’s and the subdataset’s .git and .git/config, as well as for the subdataset’s directory, i.e. ls -ld /data/proj_bitter-reward/firstlevel? (You wrote they look similar to other permission flags, but I am not sure what that means)

Could you also give an example of the owner, group, and permission flags of directories that you can drop? Feel free to obfuscate the IDs, e.g. use numeric IDs.

lukasvo76 · May 4, 2023, 2:21pm

Thanks @cmo!

Here is the ls -lah output for the different dirs you requested

1. superdataset root

0027997@gbw-s-labgas01:/data/proj_bitter-reward$ ls -lah
total 68K
drwxrws—+ 12 u0027997 domain users 4.0K Apr 21 10:27 .
drwxrws—+ 17 root domain users 4.0K Apr 19 15:50 …
drwxrws—+ 26 u0027997 domain users 4.0K Jan 9 12:41 BIDS
drwxrws—+ 6 u0027997 domain users 4.0K Mar 2 13:14 code
drwxrws—+ 2 u0027997 domain users 4.0K Nov 24 15:07 .datalad
drwxrws—+ 5 u0139539 domain users 4.0K Jan 8 09:04 derivatives
drwxrws—+ 5 r0963545 domain users 4.0K Apr 21 13:42 firstlevel
drwxrws—+ 9 u0027997 domain users 4.0K May 4 12:25 .git
-rw-rw---- 1 u0027997 domain users 191 Jan 8 09:01 .gitattributes
-rw-rw---- 1 u0027997 domain users 17 Nov 24 15:16 .gitignore
-rw-rw---- 1 u0027997 domain users 1.1K Mar 24 14:37 .gitmodules
drwxrws—+ 26 u0139539 domain users 12K Mar 27 11:21 mriqc
drwxrws—+ 4 u0139539 domain users 4.0K Dec 1 15:46 pipeline
drwxrws—+ 27 u0027997 domain users 4.0K Jan 8 09:02 sourcedata
drwxrws—+ 39 u0139539 domain users 4.0K Mar 27 11:09 work

2. superdataset .git

u0027997@gbw-s-labgas01:/data/proj_bitter-reward/.git$ ls -lah
total 64K
drwxrws—+ 9 u0027997 domain users 4.0K May 4 12:25 .
drwxrws—+ 12 u0027997 domain users 4.0K Apr 21 10:27 …
drwxrwsrwx+ 5 u0027997 domain users 4.0K Apr 12 10:14 annex
drwxrwsrwx+ 2 u0027997 domain users 4.0K Nov 24 15:07 branches
-rw-rw-rw- 1 u0027997 domain users 46 May 4 12:25 COMMIT_EDITMSG
-rw-rw-rw- 1 u0027997 domain users 1004 Apr 25 17:52 config
-rw-rw-rw- 1 u0027997 domain users 0 Nov 24 15:07 config.dataladlock
-rw-rw-rw- 1 u0027997 domain users 73 Nov 24 15:07 description
-rw-rw---- 1 u0027997 domain users 106 Apr 25 17:52 FETCH_HEAD
-rw-rw-rw- 1 u0027997 domain users 23 Nov 24 15:07 HEAD
drwxrwsrwx+ 2 u0027997 domain users 4.0K Nov 24 15:07 hooks
-rw-rw-r-- 1 r0963545 domain users 1.1K May 4 12:25 index
drwxrwsrwx+ 2 u0027997 domain users 4.0K Nov 24 15:07 info
drwxrwsrwx+ 3 u0027997 domain users 4.0K Nov 24 15:07 logs
drwxrwsrwx+ 175 u0027997 domain users 4.0K May 4 12:25 objects
-rw-rw-r-- 1 u0027997 domain users 46 Nov 24 16:22 packed-refs
drwxrwsrwx+ 6 u0027997 domain users 4.0K Nov 24 15:51 refs

3. subdataset root

u0027997@gbw-s-labgas01:/data/proj_bitter-reward/firstlevel$ ls -lah
total 24K
drwxrws—+ 5 r0963545 domain users 4.0K Apr 21 13:42 .
drwxrws—+ 12 u0027997 domain users 4.0K Apr 21 10:27 …
drwxrws—+ 2 r0963545 domain users 4.0K Mar 2 11:53 .datalad
drwxrws—+ 9 r0963545 domain users 4.0K Apr 20 11:15 .git
-rw-rw---- 1 r0963545 domain users 155 Mar 2 11:56 .gitattributes
drwxrws—+ 24 u0027997 domain users 4.0K Mar 27 16:46 model_1_food_images

4. subdataset .git

u0027997@gbw-s-labgas01:/data/proj_bitter-reward/firstlevel/.git$ ls -lah
total 444K
drwxrws—+ 9 r0963545 domain users 4.0K Apr 20 11:15 .
drwxrws—+ 5 r0963545 domain users 4.0K Apr 21 13:42 …
drwxrws—+ 7 r0963545 domain users 4.0K Apr 20 11:18 annex
drwxrws—+ 2 r0963545 domain users 4.0K Mar 2 11:53 branches
-rw-rw---- 1 r0963545 domain users 66 Mar 27 17:54 COMMIT_EDITMSG
-rw-rw---- 1 u0027997 domain users 653 Apr 20 11:15 config
-rw-rw---- 1 r0963545 domain users 0 Mar 2 11:53 config.dataladlock
-rw-rw---- 1 r0963545 domain users 73 Mar 2 11:53 description
-rw-rw---- 1 u0027997 domain users 0 Apr 25 17:50 FETCH_HEAD
-rw-rw---- 1 r0963545 domain users 21 Mar 2 12:16 .gitignore
-rw-rw---- 1 r0963545 domain users 23 Mar 2 11:53 HEAD
drwxrws—+ 2 r0963545 domain users 4.0K Mar 2 11:53 hooks
-rw-rw-r-- 1 u0027997 domain users 381K Mar 27 17:54 index
drwxrws—+ 2 r0963545 domain users 4.0K Apr 12 10:29 info
drwxrws—+ 3 r0963545 domain users 4.0K Apr 12 10:29 logs
drwxrws—+ 260 r0963545 domain users 4.0K Apr 20 11:18 objects
-rw-rw-r-- 1 u0027997 domain users 363 Apr 12 10:29 packed-refs
drwxrws—+ 6 r0963545 domain users 4.0K Mar 24 14:38 refs

I don’t immediately have another subdataset I would like to drop at hand, maybe this helps already?

Cheers,

Lukas

eknahm · May 8, 2023, 11:06am

Hey, I tried to figure out what is happening here. I think this platform may not be very suitable for this kind of problem, because providing a solution is not possible before understanding the actual issue. Would it be possible for you to pop into the DataLad office hour to get this debugged and sorted out in a live session?

One concern I have re the description of your issue and the subsequent demos and logs is that you are trying to

drop/remove subdatasets from a superdataset

but the shown commands do not show usage of --what.

The parameter determines what is to be dropped, and it defaults to filecontent. Without something like --what datasets or --what all, no attempt would be made to drop a subdataset. This would explain the notneeded results that you observed. Running drop with default parameters in a plain Git repo will not find any file content to drop, for example.

But again, it might be best to join a live session.

lukasvo76 · May 8, 2023, 3:46pm

Thanks @eknahm!

I tried adding --what all to my datalad drop command, but got the same permission error.

More than happy to join the office hour for live debugging, do I understand correctly that it is at 16:00 every Tuesday?

cmo · May 11, 2023, 2:11pm

Hi Lukas, yes the office hour is at 16:00 (Berlin time) on Tuesdays. Looking forward to see you there.

lukasvo76 · May 11, 2023, 3:02pm

Thanks a lot @cmo!

Do I need to book an appointment or can I just walk in?

Thanks,

Lukas

StephanHeunis · May 23, 2023, 8:50am

Hey @lukasvo76, the office hour chatroom has a pinned link to the weekly call (also happening today). You can just pop in, no need to make an appointment.

lukasvo76 · August 28, 2023, 8:15am

Hey @cmo @StephanHeunis,

I am still struggling with this, and wanted to check whether there is a Datalad office hour I can pop into tomorrow Tue Aug 29th at 16 Berlin time?

Thanks a lot and hopefully see you tomorrow

Cheers,

Lukas

StephanHeunis · August 29, 2023, 8:27am

Hi @lukasvo76. Unfortunately, the office hour starts again next week, since many of our core developers and users are on leave during the month of August. Please check the office hour chatroom for updates.

lukasvo76 · August 29, 2023, 8:29am

Thanks a ton, will try to join next week!

lukasvo76 · September 5, 2023, 3:29pm

Thanks a lot to the entire team for a very helpful office hour!

Here is a summary of what finally solved the problem, feel free to add/correct where needed!

1. Run datalad remove command in debug mode to identify exact location of permission error

datalad --dbg remove -d firstlevel --drop all --reckless kill

Output indicates that the permission error is present at the top level subdataset dir

Traceback (most recent call last):
File “/opt/anaconda3/bin/datalad”, line 8, in
sys.exit(main())
File “/opt/anaconda3/lib/python3.8/site-packages/datalad/cli/main.py”, line 143, in main
_run(cmdlineargs)
File “/opt/anaconda3/lib/python3.8/site-packages/datalad/cli/main.py”, line 167, in _run
ret = _run_with_debugger(namespace)
File “/opt/anaconda3/lib/python3.8/site-packages/datalad/cli/main.py”, line 181, in _run_with_debugger
return cmdlineargs.func(cmdlineargs)
File “/opt/anaconda3/lib/python3.8/site-packages/datalad/cli/exec.py”, line 107, in call_from_parser
ret = list(ret)
File “/opt/anaconda3/lib/python3.8/site-packages/datalad/interface/utils.py”, line 440, in execute_command
for r in _process_results(
File “/opt/anaconda3/lib/python3.8/site-packages/datalad/interface/utils.py”, line 619, in _process_results
for res in results:
File “/opt/anaconda3/lib/python3.8/site-packages/datalad/local/remove.py”, line 212, in call
for res in Drop.call(
File “/opt/anaconda3/lib/python3.8/site-packages/datalad/interface/utils.py”, line 440, in execute_command
for r in _process_results(
File “/opt/anaconda3/lib/python3.8/site-packages/datalad/interface/utils.py”, line 619, in _process_results
for res in results:
File “/opt/anaconda3/lib/python3.8/site-packages/datalad/distributed/drop.py”, line 268, in call
for res in _drop_dataset(
File “/opt/anaconda3/lib/python3.8/site-packages/datalad/distributed/drop.py”, line 432, in _drop_dataset
yield from _kill_dataset(ds)
File “/opt/anaconda3/lib/python3.8/site-packages/datalad/distributed/drop.py”, line 711, in _kill_dataset
rmtree(ds.path)
File “/opt/anaconda3/lib/python3.8/site-packages/datalad/utils.py”, line 500, in rmtree
rotree(path, ro=False, chmod_files=chmod_files)
File “/opt/anaconda3/lib/python3.8/site-packages/datalad/utils.py”, line 458, in rotree
chmod(root)
File “/opt/anaconda3/lib/python3.8/site-packages/datalad/utils.py”, line 449, in
chmod = lambda f: os.chmod(f, os.stat(f).st_mode | stat.S_IWRITE | stat.S_IREAD)
PermissionError: [Errno 1] Operation not permitted: ‘/data/proj_bitter-reward/firstlevel’

/opt/anaconda3/lib/python3.8/site-packages/datalad/utils.py(449)()
→ chmod = lambda f: os.chmod(f, os.stat(f).st_mode | stat.S_IWRITE | stat.S_IREAD)
(Pdb) p f
‘/data/proj_bitter-reward/firstlevel’

2. Trying to change permission/ownership of firstlevel subdataset fails

chmod u+rw firstlevel

Output

chmod: changing permissions of ‘firstlevel’: Operation not permitted

setfacl -m "u:u0027997:rwx" firstlevel

Output

setfacl: firstlevel: Operation not permitted

3. Changing ownership of subdataset recursively with root user privileges works

sudo chown -R u0027997 firstlevel

4. Datalad remove command can now succesfully be executed

datalad remove -d firstlevel --drop all --reckless kill

Output

uninstall(ok): . (dataset)

5. A simple datalad save at the level of the superdataset succesfully removes the subdataset from its subdatasets list/.gitmodules file

yarikoptic · September 5, 2023, 4:40pm

Thanks for sharing all the details!

So you are using/relying on ACLs on that system?
Then you might [*] need to devise custom hooks for freezing/thawing context by annex: see lockdown hooks . E.g. in our case we had to do that on our HPC which relies on NFS4 ACLs… still fighting it, heh.

[*] given that you ran into it only for drop may be somehow it all just works fine and only during drop some operation is not allowed, since ACLs have this more detailed sets of permissions. Would be interesting to see if git annex test passes fine while running under one of the folders on such a system. And then may be mkdir temp; TMPDIR=$PWD/tmp python -m pytest -v datalad to see how well datalad copes with it.

lukasvo76 · September 6, 2023, 7:53am

Hi Yarik,

My pleasure!

Following your suggestion, I ran git annex test in another subdataset in the same superdataset of which my student and not myself is the owner (but which I do not want to remove). It did pass all tests.

I then ran your second test, of which I share the output here

============================================================================================================ test session starts ============================================================================================================
platform linux – Python 3.8.3, pytest-5.3.5, py-1.9.0, pluggy-0.13.1 – /opt/anaconda3/bin/python
cachedir: .pytest_cache
rootdir: /data/proj_bitter-reward/mriqc
plugins: console-scripts-0.2.0, cov-2.12.1
collected 0 items

=========================================================================================================== no tests ran in 0.01s ===========================================================================================================
ERROR: file not found: datalad

Not sure why I get that error? Datalad is installed via pip.

Thanks again,

Lukas