Datalad Error in 1.3.0.post2, libc/kernel mismatch

fmriprep

#1

I’m using the recent 1.3.0.post2 singularity image built with singularity build fmriprep-1.3.0.simg docker://poldracklab/fmriprep:latest, and still getting an error related to datalad and git configuration (below). Never seen this with previous versions of fmriprep, I just pulled a new container today. Mounting the TemplateFlow folder with -B $HOME/templateflow:/opt/templateflow doesn’t help. Git version inside the container is 2.7.4, and the kernel release in the container is 2.6.32-696.18.7.el6.x86_64.

@effigies suggested that it was a libc/kernel mismatch (just quoting, not sure I understand the cogs of this problem that well).

Full traceback:

/data/jux/mackey_group/public_data/ABCD/bids_fast_track_site14/
This dataset appears to be BIDS compatible.
        Summary:                     Available Tasks:        Available Modalities: 
        36430 Files, 296.59GB        rest                    /participants         
        139 - Subjects                                       T1w                   
        1 - Session                                          bold                  
                                                             events                
                                                             scans                 
                                                             fieldmap              

If you have any questions please post on https://neurostars.org/tags/bids

Making sure the input data is BIDS compliant (warnings can be ignored in most cases).
/usr/local/miniconda/lib/python3.7/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
  return f(*args, **kwds)
/usr/local/miniconda/lib/python3.7/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
  return f(*args, **kwds)
/usr/local/miniconda/lib/python3.7/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
  return f(*args, **kwds)
/usr/local/miniconda/lib/python3.7/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
  return f(*args, **kwds)
/usr/local/miniconda/lib/python3.7/site-packages/nilearn/datasets/neurovault.py:16: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
  from collections import Container
/usr/local/miniconda/lib/python3.7/site-packages/datalad/utils.py:71: DeprecationWarning: dist() and linux_distribution() functions are deprecated in Python 3.5
  = platform.linux_distribution()[:2]
Process Process-2:
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/usr/local/miniconda/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/cli/run.py", line 560, in build_workflow
    from ..workflows.base import init_fmriprep_wf
  File "/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/workflows/base.py", line 24, in <module>
    from niworkflows.interfaces.bids import (
  File "/usr/local/miniconda/lib/python3.7/site-packages/niworkflows/interfaces/__init__.py", line 8, in <module>
    from .registration import (FLIRTRPT as FLIRT,
  File "/usr/local/miniconda/lib/python3.7/site-packages/niworkflows/interfaces/registration.py", line 25, in <module>
    from .mni import (
  File "/usr/local/miniconda/lib/python3.7/site-packages/niworkflows/interfaces/mni.py", line 20, in <module>
    from templateflow.api import get as get_template
  File "/usr/local/miniconda/lib/python3.7/site-packages/templateflow/api.py", line 5, in <module>
    from datalad import api
  File "/usr/local/miniconda/lib/python3.7/site-packages/datalad/__init__.py", line 31, in <module>
    cfg = ConfigManager()
  File "/usr/local/miniconda/lib/python3.7/site-packages/datalad/config.py", line 204, in __init__
    self.reload(force=True)
  File "/usr/local/miniconda/lib/python3.7/site-packages/datalad/config.py", line 258, in reload
    stdout, stderr = self._run(run_args, log_stderr=True)
  File "/usr/local/miniconda/lib/python3.7/site-packages/datalad/config.py", line 526, in _run
    out = self._runner.run(['git', 'config'] + args, **kwargs)
  File "/usr/local/miniconda/lib/python3.7/site-packages/datalad/cmd.py", line 670, in run
    cmd, env=self.get_git_environ_adjusted(env), *args, **kwargs)
  File "/usr/local/miniconda/lib/python3.7/site-packages/datalad/cmd.py", line 530, in run
    raise CommandError(str(cmd), msg, status, out[0], out[1])
datalad.support.exceptions.CommandError: CommandError: command '['git', 'config', '-z', '-l']' failed with exitcode 127
Failed to run ['git', 'config', '-z', '-l'] under None. Exit code=127. out= err=FATAL: kernel too old

Fmriprep 1.3.0.post2 crash
#2

In the Docker container at least, the libc version is 2.23:

$ /lib/x86_64-linux-gnu/libc.so.6
GNU C Library (Ubuntu GLIBC 2.23-0ubuntu10) stable release version 2.23, by Roland McGrath et al.
Copyright (C) 2016 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Compiled by GNU CC version 5.4.0 20160609.
Available extensions:
	crypt add-on version 2.1 by Michael Glad and others
	GNU Libidn by Simon Josefsson
	Native POSIX Threads Library by Ulrich Drepper et al
	BIND-8.2.3-T5B
libc ABIs: UNIQUE IFUNC
For bug reporting instructions, please see:
<https://bugs.launchpad.net/ubuntu/+source/glibc/+bugs>.

I don’t know the compatibility limits, but just for the information of anybody who knows more about this than I.


#3

heh heh. I’ve not ran into this one for awhile… We are building git-annex-standalone package using Debian testing, currently buster. And I guess the libc6 in there is indeed too fresh to support kernels from a decade ago (initial 2.6.32 seems was released 14-Dec-2009).

$> docker run -it debian:buster
Unable to find image 'debian:buster' locally
buster: Pulling from library/debian
53d9d89325e4: Pull complete 
Digest: sha256:9646b0ee6d68448e09cdee7ac8deb336e519113e5717ec0856d38ca813912930
Status: Downloaded newer image for debian:buster
root@91ae471f5a96:/# /lib/x86_64-linux-gnu/libc.so.6
GNU C Library (Debian GLIBC 2.28-5) stable release version 2.28.
Copyright (C) 2018 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Compiled by GNU CC version 8.2.0.
libc ABIs: UNIQUE IFUNC ABSOLUTE
For bug reporting instructions, please see:
<http://www.debian.org/Bugs/>.

not sure how easy it would be for us to fix/workaround for this since IIRC building git-annex on debian stable (jessie) was not possible easily for a while by now. The best would be for templateflow to become a bit more forgiving to datalad quirks, especially since here nothing is really needed from datalad :-/ @oesteban - what about wrapping datalad import into try/except and then skipping datalad get inquiry with a warning if datalad import has failed?
sorry for the inconvenience


#4

I think this is the right thread to post on (but please correct me if not!), I’m using the recent 1.3.0.post2 singularity image built with singularity build fmriprep-1.3.0.simg docker://poldracklab/fmriprep:latest, and still getting an error related to datalad and git configuration (below). Never seen this with previous versions of fmriprep, I just pulled a new container today.

/data/jux/mackey_group/public_data/ABCD/bids_fast_track_site14/
This dataset appears to be BIDS compatible.
        Summary:                     Available Tasks:        Available Modalities: 
        36430 Files, 296.59GB        rest                    /participants         
        139 - Subjects                                       T1w                   
        1 - Session                                          bold                  
                                                             events                
                                                             scans                 
                                                             fieldmap              

If you have any questions please post on https://neurostars.org/tags/bids

Making sure the input data is BIDS compliant (warnings can be ignored in most cases).
/usr/local/miniconda/lib/python3.7/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
  return f(*args, **kwds)
/usr/local/miniconda/lib/python3.7/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
  return f(*args, **kwds)
/usr/local/miniconda/lib/python3.7/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
  return f(*args, **kwds)
/usr/local/miniconda/lib/python3.7/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
  return f(*args, **kwds)
/usr/local/miniconda/lib/python3.7/site-packages/nilearn/datasets/neurovault.py:16: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
  from collections import Container
/usr/local/miniconda/lib/python3.7/site-packages/datalad/utils.py:71: DeprecationWarning: dist() and linux_distribution() functions are deprecated in Python 3.5
  = platform.linux_distribution()[:2]
Process Process-2:
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/usr/local/miniconda/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/cli/run.py", line 560, in build_workflow
    from ..workflows.base import init_fmriprep_wf
  File "/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/workflows/base.py", line 24, in <module>
    from niworkflows.interfaces.bids import (
  File "/usr/local/miniconda/lib/python3.7/site-packages/niworkflows/interfaces/__init__.py", line 8, in <module>
    from .registration import (FLIRTRPT as FLIRT,
  File "/usr/local/miniconda/lib/python3.7/site-packages/niworkflows/interfaces/registration.py", line 25, in <module>
    from .mni import (
  File "/usr/local/miniconda/lib/python3.7/site-packages/niworkflows/interfaces/mni.py", line 20, in <module>
    from templateflow.api import get as get_template
  File "/usr/local/miniconda/lib/python3.7/site-packages/templateflow/api.py", line 5, in <module>
    from datalad import api
  File "/usr/local/miniconda/lib/python3.7/site-packages/datalad/__init__.py", line 31, in <module>
    cfg = ConfigManager()
  File "/usr/local/miniconda/lib/python3.7/site-packages/datalad/config.py", line 204, in __init__
    self.reload(force=True)
  File "/usr/local/miniconda/lib/python3.7/site-packages/datalad/config.py", line 258, in reload
    stdout, stderr = self._run(run_args, log_stderr=True)
  File "/usr/local/miniconda/lib/python3.7/site-packages/datalad/config.py", line 526, in _run
    out = self._runner.run(['git', 'config'] + args, **kwargs)
  File "/usr/local/miniconda/lib/python3.7/site-packages/datalad/cmd.py", line 670, in run
    cmd, env=self.get_git_environ_adjusted(env), *args, **kwargs)
  File "/usr/local/miniconda/lib/python3.7/site-packages/datalad/cmd.py", line 530, in run
    raise CommandError(str(cmd), msg, status, out[0], out[1])
datalad.support.exceptions.CommandError: CommandError: command '['git', 'config', '-z', '-l']' failed with exitcode 127
Failed to run ['git', 'config', '-z', '-l'] under None. Exit code=127. out= err=FATAL: kernel too old

And the singularity container:

{
    "org.label-schema.version": "1.3.0.post2",
    "org.label-schema.schema-version": "1.0",
    "org.label-schema.vcs-ref": "af86dc6f",
    "org.label-schema.url": "http://fmriprep.org",
    "org.label-schema.vcs-url": "https://github.com/poldracklab/fmriprep",
    "org.label-schema.build-date": "2019-02-14T17:21:15Z",
    "org.label-schema.name": "fMRIPrep",
    "org.label-schema.description": "fMRIPrep - robust fMRI preprocessing tool"
}

FMRIPREP from Singularity: UnboundLocalError
#5

Uggh that does look bad (and new).

Although I don’t think it will solve the issue, could you give @rwblair’s tip (FMRIPREP from Singularity: UnboundLocalError) a try? I’m referring to mounting the TemplateFlow folder with -B $HOME/templateflow:/opt/templateflow. I think this error comes from a different source, but just in case.

@yarikoptic, @eknahm any ideas about what is happening here? - looking at:

datalad.support.exceptions.CommandError: CommandError: command '['git', 'config', '-z', '-l']' failed with exitcode 127
Failed to run ['git', 'config', '-z', '-l'] under None. Exit code=127. out= err=FATAL: kernel too old

#6

Just gave the TemplateFlow fix a try, with the command below after making that directory–still got the same error.

singularity run --cleanenv -B $HOME/templateflow:/opt/templateflow,${BIDS_folder}:/mnt ${tools_dir}/fmriprep-1.3.0.simg \
/mnt/ /mnt/derivatives \
participant \
-w /$TMPDIR \
--participant-label ${sub} \
--fs-license-file $HOME/license.txt \
--output-space T1w template fsnative fsaverage5 \
--ignore slicetiming \
--cifti-output \
--nthreads 1 \

#7

Thanks for the prompt reply. I’ll focus on this issue later today.


#8

And how old is the kernel here we are talking about?


#9

Is this something I have the answer to? I’m not sure, but if it helps, if I run git version I get 2.9.5. I’m on an HPC where I use their apps and this is the version that’s being used. If that’s an issue, I can definitely bug them (we’re going to transition to a newer system soon, hopefully, and things like this would be good to know).


#10

There should be a git packaged in the Singularity container, which should take precedence, especially if using --cleanenv. Can you check:

$ which git
$ git version
$ singularity shell ${tools_dir}/fmriprep-1.3.0.simg
> which git
> git version

#11

Makes total sense–I figured local git shouldn’t affect it, but looks like git inside the container is older…

/share/apps/git/2.9.5/bin/git
git version
git version 2.9.5
singularity shell ${tools_dir}/fmriprep-1.3.0.simg
Singularity fmriprep-1.3.0.simg:~> which git
/usr/bin/git
Singularity fmriprep-1.3.0.simg:~> git version
git version 2.7.4

#12

Are we talking about the Linux kernel? If so, that would be 4.15.0-45-generic

@Ursula_Tooley, could you also run?

singularity exec ${tools_dir}/fmriprep-1.3.0.simg /bin/uname -r 

#13

oookay, singularity of course is talking to the host’s kernel, right?


#14

So this looks like a libc/kernel mismatch, which is sufficiently different from the original topic of this thread that I would suggest we create another thread.


#15

@oesteban that gives:

 singularity exec ${tools_dir}/fmriprep-1.3.0.simg /bin/uname -r
2.6.32-696.18.7.el6.x86_64

Hopefully that’s helpful…This has gotten a bit deeper than my knowledge of troubleshooting covers, but @ me if there is other info I can provide!

I can also repost a summary to a new thread or open an issue if needed.


#17

3 posts were merged into an existing topic: fMRIPrep 1.3.x on Singularity - git-annex permission errors


#18

Hi @Ursula_Tooley,

The only workaround to this issue is to avoid git-annex. That will be possible with the upcoming 1.3.1 version of fMRIPrep (hopefully, it’ll be rolled out tonight). Stay tuned!

PS. Sorry for the messed up ordering of posts, I brought the comments from the other conversation where this problem was first reported.


#19

Hi, the latest release 1.3.1 is out. Please let us know if that version resolves this problem!