Q: how can I use local bash variables inside 'datalad run'?

1

Hi to everyone!

I have one stupid question faced to me when I try to make some simple thing with files in my dataset (like renaming, or making another soft link to the files) and I don’t want to create special bash script for this, but still want to keep reproducibility.

So, the question is: how can I use var ‘file’ (see below) inside this script:

datalad run -i 'sub-*' 'for file in {inputs}; do echo ${file}; done

How to reproduce:

datalad create test && cd test
for num in 1 2 3 4 5; do touch sub-$num; done
datalad run -i 'sub-*' 'for file in {inputs}; do echo ${file}; done

I have an error: run(impossible): /home-local/dmitriy_mikhailov/test (dataset) [command has an unrecognized placeholder: 'file']

I understand that I can just remove brackets: ${file} → $file, and this will be work. But what if I want to use bash substitution possibilities like ${file/pattern/word}?

I have tried “${file}” “’${file}’”, “${{file}}” (the last one worked, but with ${{file/sub/SUB}} i’ve got ‘bad substitution’ from bash)

2 
$> datalad run --help | grep '{{'
To escape a brace character, double it (i.e., "{{" or "}}").

so

(git-annex)lena:/tmp/testds[master]git
$> datalad save 
add(ok): 1 (file)                                                                                         
add(ok): 2 (file)                                                                                         
add(ok): 3 (file)                                                                                         
save(ok): . (dataset)                                                                                     
(dev3) (datalad-py3.8) 1 11620 [1].....................................:Tue 08 Feb 2022 08:48:10 AM EST:.
(git-annex)lena:/tmp/testds[master]git
$> datalad run -i '*' 'for file in {inputs}; do echo ${{file}}; done'
[INFO   ] Making sure inputs are available (this may take some time) 
[INFO   ] == Command start (output follows) ===== 
1
2
3
[INFO   ] == Command exit (modification check follows) =====

so your ${{file}} attempt was the right way. But as for substitution – it is a “bashism”, so /bin/sh (even if points to bash) would not do it:

(git-annex)lena:/tmp/testds[master]git
$> /bin/sh -c 'for file in 1 2 3; do echo ${file/1/XXX}; done'
/bin/sh: 1: Bad substitution

$> /bin/bash -c 'for file in 1 2 3; do echo ${file/1/XXX}; done'
XXX
2
3

you can explicitly run bash though:

$> datalad run -i '*' /bin/bash -c 'for file in {inputs}; do echo ${{file/1/XXX}}; done' 
[INFO   ] Making sure inputs are available (this may take some time) 
[INFO   ] == Command start (output follows) ===== 
XXX
2
3
[INFO   ] == Command exit (modification check follows) =====