Datalad Error in 1.3.0.post2, libc/kernel mismatch

I’m using the recent 1.3.0.post2 singularity image built with singularity build fmriprep-1.3.0.simg docker://poldracklab/fmriprep:latest, and still getting an error related to datalad and git configuration (below). Never seen this with previous versions of fmriprep, I just pulled a new container today. Mounting the TemplateFlow folder with -B $HOME/templateflow:/opt/templateflow doesn’t help. Git version inside the container is 2.7.4, and the kernel release in the container is 2.6.32-696.18.7.el6.x86_64.

@effigies suggested that it was a libc/kernel mismatch (just quoting, not sure I understand the cogs of this problem that well).

Full traceback:

/data/jux/mackey_group/public_data/ABCD/bids_fast_track_site14/
This dataset appears to be BIDS compatible.
        Summary:                     Available Tasks:        Available Modalities: 
        36430 Files, 296.59GB        rest                    /participants         
        139 - Subjects                                       T1w                   
        1 - Session                                          bold                  
                                                             events                
                                                             scans                 
                                                             fieldmap              

If you have any questions please post on https://neurostars.org/tags/bids

Making sure the input data is BIDS compliant (warnings can be ignored in most cases).
/usr/local/miniconda/lib/python3.7/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
  return f(*args, **kwds)
/usr/local/miniconda/lib/python3.7/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
  return f(*args, **kwds)
/usr/local/miniconda/lib/python3.7/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
  return f(*args, **kwds)
/usr/local/miniconda/lib/python3.7/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
  return f(*args, **kwds)
/usr/local/miniconda/lib/python3.7/site-packages/nilearn/datasets/neurovault.py:16: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
  from collections import Container
/usr/local/miniconda/lib/python3.7/site-packages/datalad/utils.py:71: DeprecationWarning: dist() and linux_distribution() functions are deprecated in Python 3.5
  = platform.linux_distribution()[:2]
Process Process-2:
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/usr/local/miniconda/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/cli/run.py", line 560, in build_workflow
    from ..workflows.base import init_fmriprep_wf
  File "/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/workflows/base.py", line 24, in <module>
    from niworkflows.interfaces.bids import (
  File "/usr/local/miniconda/lib/python3.7/site-packages/niworkflows/interfaces/__init__.py", line 8, in <module>
    from .registration import (FLIRTRPT as FLIRT,
  File "/usr/local/miniconda/lib/python3.7/site-packages/niworkflows/interfaces/registration.py", line 25, in <module>
    from .mni import (
  File "/usr/local/miniconda/lib/python3.7/site-packages/niworkflows/interfaces/mni.py", line 20, in <module>
    from templateflow.api import get as get_template
  File "/usr/local/miniconda/lib/python3.7/site-packages/templateflow/api.py", line 5, in <module>
    from datalad import api
  File "/usr/local/miniconda/lib/python3.7/site-packages/datalad/__init__.py", line 31, in <module>
    cfg = ConfigManager()
  File "/usr/local/miniconda/lib/python3.7/site-packages/datalad/config.py", line 204, in __init__
    self.reload(force=True)
  File "/usr/local/miniconda/lib/python3.7/site-packages/datalad/config.py", line 258, in reload
    stdout, stderr = self._run(run_args, log_stderr=True)
  File "/usr/local/miniconda/lib/python3.7/site-packages/datalad/config.py", line 526, in _run
    out = self._runner.run(['git', 'config'] + args, **kwargs)
  File "/usr/local/miniconda/lib/python3.7/site-packages/datalad/cmd.py", line 670, in run
    cmd, env=self.get_git_environ_adjusted(env), *args, **kwargs)
  File "/usr/local/miniconda/lib/python3.7/site-packages/datalad/cmd.py", line 530, in run
    raise CommandError(str(cmd), msg, status, out[0], out[1])
datalad.support.exceptions.CommandError: CommandError: command '['git', 'config', '-z', '-l']' failed with exitcode 127
Failed to run ['git', 'config', '-z', '-l'] under None. Exit code=127. out= err=FATAL: kernel too old

In the Docker container at least, the libc version is 2.23:

$ /lib/x86_64-linux-gnu/libc.so.6
GNU C Library (Ubuntu GLIBC 2.23-0ubuntu10) stable release version 2.23, by Roland McGrath et al.
Copyright (C) 2016 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Compiled by GNU CC version 5.4.0 20160609.
Available extensions:
	crypt add-on version 2.1 by Michael Glad and others
	GNU Libidn by Simon Josefsson
	Native POSIX Threads Library by Ulrich Drepper et al
	BIND-8.2.3-T5B
libc ABIs: UNIQUE IFUNC
For bug reporting instructions, please see:
<https://bugs.launchpad.net/ubuntu/+source/glibc/+bugs>.

I don’t know the compatibility limits, but just for the information of anybody who knows more about this than I.

heh heh. I’ve not ran into this one for awhile… We are building git-annex-standalone package using Debian testing, currently buster. And I guess the libc6 in there is indeed too fresh to support kernels from a decade ago (initial 2.6.32 seems was released 14-Dec-2009).

$> docker run -it debian:buster
Unable to find image 'debian:buster' locally
buster: Pulling from library/debian
53d9d89325e4: Pull complete 
Digest: sha256:9646b0ee6d68448e09cdee7ac8deb336e519113e5717ec0856d38ca813912930
Status: Downloaded newer image for debian:buster
root@91ae471f5a96:/# /lib/x86_64-linux-gnu/libc.so.6
GNU C Library (Debian GLIBC 2.28-5) stable release version 2.28.
Copyright (C) 2018 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Compiled by GNU CC version 8.2.0.
libc ABIs: UNIQUE IFUNC ABSOLUTE
For bug reporting instructions, please see:
<http://www.debian.org/Bugs/>.

not sure how easy it would be for us to fix/workaround for this since IIRC building git-annex on debian stable (jessie) was not possible easily for a while by now. The best would be for templateflow to become a bit more forgiving to datalad quirks, especially since here nothing is really needed from datalad :-/ @oesteban - what about wrapping datalad import into try/except and then skipping datalad get inquiry with a warning if datalad import has failed?
sorry for the inconvenience

I think this is the right thread to post on (but please correct me if not!), I’m using the recent 1.3.0.post2 singularity image built with singularity build fmriprep-1.3.0.simg docker://poldracklab/fmriprep:latest, and still getting an error related to datalad and git configuration (below). Never seen this with previous versions of fmriprep, I just pulled a new container today.

/data/jux/mackey_group/public_data/ABCD/bids_fast_track_site14/
This dataset appears to be BIDS compatible.
        Summary:                     Available Tasks:        Available Modalities: 
        36430 Files, 296.59GB        rest                    /participants         
        139 - Subjects                                       T1w                   
        1 - Session                                          bold                  
                                                             events                
                                                             scans                 
                                                             fieldmap              

If you have any questions please post on https://neurostars.org/tags/bids

Making sure the input data is BIDS compliant (warnings can be ignored in most cases).
/usr/local/miniconda/lib/python3.7/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
  return f(*args, **kwds)
/usr/local/miniconda/lib/python3.7/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
  return f(*args, **kwds)
/usr/local/miniconda/lib/python3.7/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
  return f(*args, **kwds)
/usr/local/miniconda/lib/python3.7/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
  return f(*args, **kwds)
/usr/local/miniconda/lib/python3.7/site-packages/nilearn/datasets/neurovault.py:16: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
  from collections import Container
/usr/local/miniconda/lib/python3.7/site-packages/datalad/utils.py:71: DeprecationWarning: dist() and linux_distribution() functions are deprecated in Python 3.5
  = platform.linux_distribution()[:2]
Process Process-2:
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/usr/local/miniconda/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/cli/run.py", line 560, in build_workflow
    from ..workflows.base import init_fmriprep_wf
  File "/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/workflows/base.py", line 24, in <module>
    from niworkflows.interfaces.bids import (
  File "/usr/local/miniconda/lib/python3.7/site-packages/niworkflows/interfaces/__init__.py", line 8, in <module>
    from .registration import (FLIRTRPT as FLIRT,
  File "/usr/local/miniconda/lib/python3.7/site-packages/niworkflows/interfaces/registration.py", line 25, in <module>
    from .mni import (
  File "/usr/local/miniconda/lib/python3.7/site-packages/niworkflows/interfaces/mni.py", line 20, in <module>
    from templateflow.api import get as get_template
  File "/usr/local/miniconda/lib/python3.7/site-packages/templateflow/api.py", line 5, in <module>
    from datalad import api
  File "/usr/local/miniconda/lib/python3.7/site-packages/datalad/__init__.py", line 31, in <module>
    cfg = ConfigManager()
  File "/usr/local/miniconda/lib/python3.7/site-packages/datalad/config.py", line 204, in __init__
    self.reload(force=True)
  File "/usr/local/miniconda/lib/python3.7/site-packages/datalad/config.py", line 258, in reload
    stdout, stderr = self._run(run_args, log_stderr=True)
  File "/usr/local/miniconda/lib/python3.7/site-packages/datalad/config.py", line 526, in _run
    out = self._runner.run(['git', 'config'] + args, **kwargs)
  File "/usr/local/miniconda/lib/python3.7/site-packages/datalad/cmd.py", line 670, in run
    cmd, env=self.get_git_environ_adjusted(env), *args, **kwargs)
  File "/usr/local/miniconda/lib/python3.7/site-packages/datalad/cmd.py", line 530, in run
    raise CommandError(str(cmd), msg, status, out[0], out[1])
datalad.support.exceptions.CommandError: CommandError: command '['git', 'config', '-z', '-l']' failed with exitcode 127
Failed to run ['git', 'config', '-z', '-l'] under None. Exit code=127. out= err=FATAL: kernel too old

And the singularity container:

{
    "org.label-schema.version": "1.3.0.post2",
    "org.label-schema.schema-version": "1.0",
    "org.label-schema.vcs-ref": "af86dc6f",
    "org.label-schema.url": "http://fmriprep.org",
    "org.label-schema.vcs-url": "https://github.com/poldracklab/fmriprep",
    "org.label-schema.build-date": "2019-02-14T17:21:15Z",
    "org.label-schema.name": "fMRIPrep",
    "org.label-schema.description": "fMRIPrep - robust fMRI preprocessing tool"
}

Uggh that does look bad (and new).

Although I don’t think it will solve the issue, could you give @rwblair’s tip (FMRIPREP from Singularity: UnboundLocalError) a try? I’m referring to mounting the TemplateFlow folder with -B $HOME/templateflow:/opt/templateflow. I think this error comes from a different source, but just in case.

@yarikoptic, @eknahm any ideas about what is happening here? - looking at:

datalad.support.exceptions.CommandError: CommandError: command '['git', 'config', '-z', '-l']' failed with exitcode 127
Failed to run ['git', 'config', '-z', '-l'] under None. Exit code=127. out= err=FATAL: kernel too old

Just gave the TemplateFlow fix a try, with the command below after making that directory–still got the same error.

singularity run --cleanenv -B $HOME/templateflow:/opt/templateflow,${BIDS_folder}:/mnt ${tools_dir}/fmriprep-1.3.0.simg \
/mnt/ /mnt/derivatives \
participant \
-w /$TMPDIR \
--participant-label ${sub} \
--fs-license-file $HOME/license.txt \
--output-space T1w template fsnative fsaverage5 \
--ignore slicetiming \
--cifti-output \
--nthreads 1 \

Thanks for the prompt reply. I’ll focus on this issue later today.

1 Like

And how old is the kernel here we are talking about?

Is this something I have the answer to? I’m not sure, but if it helps, if I run git version I get 2.9.5. I’m on an HPC where I use their apps and this is the version that’s being used. If that’s an issue, I can definitely bug them (we’re going to transition to a newer system soon, hopefully, and things like this would be good to know).

There should be a git packaged in the Singularity container, which should take precedence, especially if using --cleanenv. Can you check:

$ which git
$ git version
$ singularity shell ${tools_dir}/fmriprep-1.3.0.simg
> which git
> git version

Makes total sense–I figured local git shouldn’t affect it, but looks like git inside the container is older…

/share/apps/git/2.9.5/bin/git
git version
git version 2.9.5
singularity shell ${tools_dir}/fmriprep-1.3.0.simg
Singularity fmriprep-1.3.0.simg:~> which git
/usr/bin/git
Singularity fmriprep-1.3.0.simg:~> git version
git version 2.7.4

Are we talking about the Linux kernel? If so, that would be 4.15.0-45-generic

@Ursula_Tooley, could you also run?

singularity exec ${tools_dir}/fmriprep-1.3.0.simg /bin/uname -r 

oookay, singularity of course is talking to the host’s kernel, right?

So this looks like a libc/kernel mismatch, which is sufficiently different from the original topic of this thread that I would suggest we create another thread.

@oesteban that gives:

 singularity exec ${tools_dir}/fmriprep-1.3.0.simg /bin/uname -r
2.6.32-696.18.7.el6.x86_64

Hopefully that’s helpful…This has gotten a bit deeper than my knowledge of troubleshooting covers, but @ me if there is other info I can provide!

I can also repost a summary to a new thread or open an issue if needed.

3 posts were merged into an existing topic: fMRIPrep 1.3.x on Singularity - git-annex permission errors

Hi @Ursula_Tooley,

The only workaround to this issue is to avoid git-annex. That will be possible with the upcoming 1.3.1 version of fMRIPrep (hopefully, it’ll be rolled out tonight). Stay tuned!

PS. Sorry for the messed up ordering of posts, I brought the comments from the other conversation where this problem was first reported.

1 Like

Hi, the latest release 1.3.1 is out. Please let us know if that version resolves this problem!

1 Like