Unfortunately, I am encountering some problems after I ran fmriprep preprocessing on a version-controlled dataset by datalad. The dataset itself came from openneuro and is installed locally on my computer. Having a first look, t seems that the preprocessing worked.
However, I received multiple error-messages (some hundreds or maybe even more) after running fmriprep, stating that some of the files generated by fmriprep cannot be saved/added in the output.
The error states: […] SetFileMode: permission denied (operation not permitted). So those files can’t be added, leaving my dataset unclean and I can’t not proceed with further preprocessing. A datalad save unfortunately throws errors as well, so the untracked files can’t be added.
So is there a “datalad-way” to solve this or do I need to go through each file and somehow change permission?
I apologize if I overlooked something maybe obvious, I am quite new to the field. ^^
Thanks for your help in advance!
Best regards,
Anja
P.S.: When I performed a test-run with one subject earlier, I did not receive any errors at all and so I am wondering where it comes from.
The datalad version I am using is 0.13.6 on Debian 10 (buster), the fmriprep version is 20.2.1, running in a docker-container.
I believe the solution is to delete the existing files first from the tree under datalad’s control. The reason is that datalad is backed by git-annex, and git-annex does not keep the actual files on the tree, but softlinks pointing to files in the annex.
The way of making datalad aware that a file has changed requires you to first delete these softlinks. Otherwise, the file that is attempted to be overwritten is the blob in the annex, and you don’t want to do that. In order to make sure you don’t do that, git-annex disallows you from doing so (and hence the error you get).
That all said:
I invite experts of datalad to correct my description as it has high likelihood of being inaccurate. cc/ @yarikoptic@eknahm
We should/could consider making fMRIPrep’s DataSinks (or even Nipype’s) datalad-friendly by making sure they delete existing targets before writing. cc/ @mgxd@effigies
If you’re not using datalad run (or datalad containers-run), then you should probably be manually datalad unlocking anything that might get overwritten.
My first question, under assumption that we are talking about raw BIDS dataset would be – how do you invoke fmriprep that it tries to overwrite already datalad’ified files? Ideally output of fmriprep should be placed into a separate dataset, ideally following YODA nesting principles – see e.g. https://github.com/ReproNim/containers/#a-typical-workflow for an example with mriqc .
+1 on @oesteban 's 2. – would be nice if datasink first removed target paths right before placement. (do not remember if the outputs are going to some temp location first, since if not and output file == input file , this wouldn’t work)
Thank you @effigies - indeed if output locations are specified in --output argument to run or containers-run - datalad would first unlock them, thus making them write-able again.
Another approach could be a git-annex “adjusted branch” with all content unlocked, but IMHO it better be avoided (unless mandated by OS or filesystem which has no symlinks support etc).