Intermittent git-annex SQLite3 ErrorNotFound during datalad get on BeeGFS

Gandom · August 10, 2026, 5:35pm

I am using DataLad/git-annex to download fMRIPrep derivative data on an HPC system.

The DataLad datasets are stored on a BeeGFS filesystem. During datalad get, I repeatedly encounter the following error:

git-annex: sqlite worker thread crashed: user error (SQLite3 returned ErrorNotFound while attempting to perform step. (while opening database connection))

The problem is intermittent within the same download operation: some files download successfully (get(ok)), while other files fail with this SQLite error (get(error)).

This happens across multiple participants and different file types, rather than being limited to one particular file. Rerunning the download sometimes results in additional files being downloaded successfully, but the SQLite error continues to occur for other files.

I am downloading a relatively large dataset, so I would like to understand whether this is related to git-annex’s local SQLite database, the BeeGFS filesystem, interrupted downloads, or another issue, and whether there is a safe way to repair/check the database without losing data that has already been downloaded.

PAST#!/bin/bash
# Download the minimal fMRIPrep inputs needed for XCP-D (linc / CIFTI)
# for the final 183 participants, split into parts of 5.
#
# Parts 1-36 = 5 participants each
# Part 37 = 3 participants
#
# Usage:
#   bash download_183_xcpd_in_parts_of_5.sh PART_NUMBER
#
# Example:
#   bash download_183_xcpd_in_parts_of_5.sh 1
#
# IMPORTANT:
# - This script downloads data with DataLad.
# - It does NOT use "set -e", because DataLad/git-annex may sometimes
#   return non-zero even when some requested content was obtained.
# - Anatomical and functional derivatives are stored in separate RBC clones.

set -u

ANAT_DATASET="/mnt/beegfs/wrkdir/u111428/RBC_data/HBN_fMRIPrep-Anat"
FUNC_DATASET="/mnt/beegfs/wrkdir/u111428/RBC_data/HBN_fMRIPrep-Func"

PART="${1:-}"
TOTAL_PARTS=37
PART_SIZE=5

if [[ -z "$PART" || ! "$PART" =~ ^[0-9]+$ || "$PART" -lt 1 || "$PART" -gt "$TOTAL_PARTS" ]]; then
    echo "Usage: bash $0 PART_NUMBER"
    echo "PART_NUMBER must be 1..$TOTAL_PARTS"
    exit 1
fi

IDS=(
    "NDARAL606CW5"
    "NDARAP049KXJ"
    "NDARAP283ZBW"
    "NDARBD050LT3"
    "NDARBE287EFC"
    "NDARBK032VNE"
    "NDARBL990KHP"
    "NDARBN365EV3"
    "NDARCH795JD4"
    "NDARCL927JHR"
    "NDARCW932EU6"
    "NDARCW946WNE"
    "NDARDE769PHW"
    "NDARDL033XRG"
    "NDAREF074ZAQ"
    "NDAREV593YN4"
    "NDARFA358RJJ"
    "NDARFJ546HXE"
    "NDARGD919DHK"
    "NDARGF445UFB"
    "NDARGM159ZL9"
    "NDARGN499DJ5"
    "NDARGV434RTP"
    "NDARGV455JV1"
    "NDARGX161LRM"
    "NDARHN206XY3"
    "NDARHT403JJJ"
    "NDARHZ476MJP"
    "NDARKC420XZF"
    "NDARKE254HKM"
    "NDARKN139APF"
    "NDARKP414AZ5"
    "NDARKX640YF6"
    "NDARLF102RJF"
    "NDARLF484WJL"
    "NDARLN749FML"
    "NDARLP160GHJ"
    "NDARME100GK1"
    "NDARMK825WAX"
    "NDARML681XZJ"
    "NDARMT939UCK"
    "NDARNK005BRN"
    "NDARNN493WMD"
    "NDARNP254RWE"
    "NDARNP956FPR"
    "NDARNW188JTF"
    "NDARNZ141GNH"
    "NDARPC745LTD"
    "NDARRB901DWV"
    "NDARRC295CHW"
    "NDARRD943ZWU"
    "NDARRM073JKA"
    "NDARRP384BVX"
    "NDARRR595LRJ"
    "NDARRV837BZQ"
    "NDARUP132ADW"
    "NDARUR153ZWB"
    "NDARUX070CWZ"
    "NDARVA930UA3"
    "NDARVC874KGR"
    "NDARWF306VWJ"
    "NDARWR888KKT"
    "NDARWY566RM"
    "NDARXG961UAB"
    "NDARXT196WPG"
    "NDARXW330ZJT"
    "NDARYE017HZA"
    "NDARYG988PUQ"
    "NDARYJ293XH4"
    "NDARYK236YUD"
    "NDARYM586MYN"
    "NDARYU345GZJ"
    "NDARYZ693DE2"
    "NDARYZ761TA3"
    "NDARZL239PGW"
    "NDARZU615HUG"
    "NDARZW930MF2"
    "NDARAB653ZXP"
    "NDARAC904DMU"
    "NDARAP522AFK"
    "NDARAV610EY3"
    "NDARAX283MAK"
    "NDARBF998MBA"
    "NDARBT436PMT"
    "NDARCD182XT1"
    "NDARCG808HDJ"
    "NDARCH514JCT"
    "NDARCN766ED4"
    "NDARCP698AT5"
    "NDARCW963FP9"
    "NDARDC814WW2"
    "NDARDH086ZKK"
    "NDARDH753GUF"
    "NDARDJ825GBP"
    "NDARDL511UND"
    "NDARDN489EXJ"
    "NDAREG013BLG"
    "NDAREU551GPC"
    "NDAREW661NZJ"
    "NDAREX065KJU"
    "NDARFK610GY5"
    "NDARFL411AT1"
    "NDARFW292PBD"
    "NDARGF543PM2"
    "NDARGG205WVN"
    "NDARGG547JN2"
    "NDARGT022BEW"
    "NDARGT682ZWN"
    "NDARHB764VZ2"
    "NDARHM475DC5"
    "NDARHX754KYY"
    "NDARJ257ZU2"
    "NDARJH910CGX"
    "NDARJJ345VHW"
    "NDARJM708VGE"
    "NDARJW989EM6"
    "NDARJX121GKP"
    "NDARJZ089HVP"
    "NDARKD168VHJ"
    "NDARKF615JNZ"
    "NDARKV482FU9"
    "NDARLF446MT5"
    "NDARLL846UYQ"
    "NDARLM196YRG"
    "NDARLU111UYF"
    "NDARMB290GPL"
    "NDARMC759CX3"
    "NDARME930DE7"
    "NDARMJ333WJM"
    "NDARML514ZFM"
    "NDARMM486AJU"
    "NDARNC834UF7"
    "NDARNF362NU0"
    "NDARNG689AAP"
    "NDARNH342XLK"
    "NDARNK489GNR"
    "NDARNT939YMG"
    "NDARPH022TRY"
    "NDARPW482TVE"
    "NDARPZ621ZLE"
    "NDARRD326KB9"
    "NDARRE333EKT"
    "NDARRG199RU4"
    "NDARRH908NZD"
    "NDARRK528GFZ"
    "NDARRK882CLT"
    "NDARRV002MA8"
    "NDARRZ640EG2"
    "NDARTF566PYH"
    "NDARTH473LF8"
    "NDARTH610GMK"
    "NDARTK185PBH"
    "NDARTU768MY1"
    "NDARUC356FCD"
    "NDARUD764NFJ"
    "NDARUM569EV1"
    "NDARUP249AMD"
    "NDARUR987CDM"
    "NDARUX616RC1"
    "NDARVF852TZ4"
    "NDARVG461LA2"
    "NDARVG958MHH"
    "NDARWA276AW0"
    "NDARWB591KLW"
    "NDARWR628EFJ"
    "NDARXF203DCD"
    "NDARXH597ML1"
    "NDARXK462WRZ"
    "NDARXW455MJ1"
    "NDARXX895FNJ"
    "NDARXZ902NFM"
    "NDARYA503XVQ"
    "NDARYH480GTD"
    "NDARYL272HDW"
    "NDARYW789GNP"
    "NDARZC028AEC"
    "NDARZG263HRK"
    "NDARZG690NHH"
    "NDARZM903TNL"
    "NDARZR529EJ1"
    "NDARZR567HWG"
    "NDARZV421TCZ"
    "NDARZW363UGM"
)

# Safety check: the embedded list must contain exactly 183 IDs.
if [[ "${#IDS[@]}" -ne 183 ]]; then
    echo "ERROR: expected 183 IDs, found ${#IDS[@]}"
    exit 1
fi

# ----------------------------------------------------------------------
# Helper: download every file in DATASET/ROOT matching PATTERN.
# It first searches the clone layout, then asks DataLad for the exact paths.
# ----------------------------------------------------------------------
get_matches() {
    local dataset="$1"
    local root="$2"
    local label="$3"
    local pattern="$4"

    local -a matches=()

    if [[ ! -d "$dataset/$root" ]]; then
        echo "  [MISSING ROOT] $label : $dataset/$root"
        return 2
    fi

    while IFS= read -r -d '' f; do
        matches+=("$f")
    done < <(
        cd "$dataset" &&
        find "$root" -name "$pattern" -print0 2>/dev/null
    )

    if [[ "${#matches[@]}" -eq 0 ]]; then
        echo "  [NO MATCH] $label"
        return 3
    fi

    echo "  [$label] ${#matches[@]} file(s)"

    (
        cd "$dataset" || exit 1
        datalad get "${matches[@]}"
    )

    return $?
}

# ----------------------------------------------------------------------
# Helper for native-space T1w.
# We explicitly exclude any filename containing "space-".
# ----------------------------------------------------------------------
get_native_t1() {
    local dataset="$1"
    local root="$2"
    local extension="$3"
    local label="$4"

    local -a matches=()

    if [[ ! -d "$dataset/$root" ]]; then
        echo "  [MISSING ROOT] $label : $dataset/$root"
        return 2
    fi

    while IFS= read -r -d '' f; do
        matches+=("$f")
    done < <(
        cd "$dataset" &&
        find "$root"             -name "*desc-preproc_T1w.${extension}"             ! -name "*space-*"             -print0 2>/dev/null
    )

    if [[ "${#matches[@]}" -eq 0 ]]; then
        echo "  [NO MATCH] $label"
        return 3
    fi

    echo "  [$label] ${#matches[@]} file(s)"

    (
        cd "$dataset" || exit 1
        datalad get "${matches[@]}"
    )

    return $?
}

# ----------------------------------------------------------------------
# Dataset-level metadata.
# XCP-D expects a dataset_description.json in the input derivative dataset.
# These are tiny, so requesting them on every part is harmless.
# ----------------------------------------------------------------------
echo "============================================================"
echo "Getting dataset-level metadata"
echo "============================================================"

(
    cd "$ANAT_DATASET" || exit 1
    [[ -e dataset_description.json || -L dataset_description.json ]] &&
        datalad get dataset_description.json
)

(
    cd "$FUNC_DATASET" || exit 1
    [[ -e dataset_description.json || -L dataset_description.json ]] &&
        datalad get dataset_description.json
)

# ----------------------------------------------------------------------
# Determine which 5 participants belong to the requested part.
# ----------------------------------------------------------------------
START=$(( (PART - 1) * PART_SIZE ))
END=$(( START + PART_SIZE ))
(( END > ${#IDS[@]} )) && END=${#IDS[@]}

echo
echo "============================================================"
echo "Part $PART / $TOTAL_PARTS"
echo "Participants $((START + 1))..$END of ${#IDS[@]}"
echo "Started: $(date)"
echo "============================================================"

# ----------------------------------------------------------------------
# Download each participant.
# ----------------------------------------------------------------------
for ((i=START; i<END; i++)); do

    ID="${IDS[$i]}"

    echo
    echo "############################################################"
    echo "Participant: sub-${ID}"
    echo "Started: $(date)"
    echo "############################################################"

    ANAT_ROOT="sub-${ID}"
    FUNC_ROOT="sub-${ID}"

    # ================================================================
    # A) ANATOMICAL MINIMAL INPUTS
    # ================================================================
    echo
    echo "----- ANATOMICAL -----"

    # 1) Native-space preprocessed T1w
    get_native_t1 "$ANAT_DATASET" "$ANAT_ROOT" "nii.gz"         "native preproc T1w NIfTI"

    # 2) Native-space T1w JSON
    get_native_t1 "$ANAT_DATASET" "$ANAT_ROOT" "json"         "native preproc T1w JSON"

    # 3) MNI152NLin6Asym anatomical brain mask
    get_matches "$ANAT_DATASET" "$ANAT_ROOT"         "MNI anatomical brain mask NIfTI"         "*space-MNI152NLin6Asym*_desc-brain_mask.nii.gz"

    # 4) MNI152NLin6Asym anatomical brain-mask JSON
    get_matches "$ANAT_DATASET" "$ANAT_ROOT"         "MNI anatomical brain mask JSON"         "*space-MNI152NLin6Asym*_desc-brain_mask.json"

    # 5) MNI152NLin6Asym -> T1w transform
    get_matches "$ANAT_DATASET" "$ANAT_ROOT"         "MNI-to-T1w transform"         "*from-MNI152NLin6Asym_to-T1w_mode-image_xfm.h5"

    # 6) T1w -> MNI152NLin6Asym transform
    get_matches "$ANAT_DATASET" "$ANAT_ROOT"         "T1w-to-MNI transform"         "*from-T1w_to-MNI152NLin6Asym_mode-image_xfm.h5"


    # ================================================================
    # B) RESTING-STATE FUNCTIONAL MINIMAL INPUTS
    # ================================================================
    echo
    echo "----- REST FUNCTIONAL -----"

    # 1) Confounds TSV for every resting-state run
    get_matches "$FUNC_DATASET" "$FUNC_ROOT"         "rest confounds TSV"         "*task-rest*desc-confounds_timeseries.tsv"

    # 2) Confounds JSON
    get_matches "$FUNC_DATASET" "$FUNC_ROOT"         "rest confounds JSON"         "*task-rest*desc-confounds_timeseries.json"

    # 3) CIFTI BOLD fsLR 91k
    get_matches "$FUNC_DATASET" "$FUNC_ROOT"         "rest fsLR 91k CIFTI"         "*task-rest*space-fsLR_den-91k_bold.dtseries.nii"

    # 4) CIFTI BOLD JSON
    get_matches "$FUNC_DATASET" "$FUNC_ROOT"         "rest fsLR 91k CIFTI JSON"         "*task-rest*space-fsLR_den-91k_bold.json"

    # 5) Volumetric MNI BOLD (used by XCP-D QC)
    get_matches "$FUNC_DATASET" "$FUNC_ROOT"         "rest MNI preproc BOLD"         "*task-rest*space-MNI152NLin6Asym*_desc-preproc_bold.nii.gz"

    # 6) MNI boldref
    get_matches "$FUNC_DATASET" "$FUNC_ROOT"         "rest MNI boldref"         "*task-rest*space-MNI152NLin6Asym*_boldref.nii.gz"

    # 7) MNI boldref JSON
    get_matches "$FUNC_DATASET" "$FUNC_ROOT"         "rest MNI boldref JSON"         "*task-rest*space-MNI152NLin6Asym*_boldref.json"


    echo
    echo "Finished participant: sub-${ID} | $(date)"

done

echo
echo "============================================================"
echo "PART $PART FINISHED: $(date)"
echo "============================================================"
E CODE HERE




The helper script runs `datalad get` on selected fMRIPrep anatomical and resting-state functional files for small batches of participants. I intentionally download participants in small batches because the dataset is large.

The underlying operations are standard `datalad get` calls on files within the cloned DataLad datasets.

Version:

DataLad: 1.6.1
git-annex: 10.20250721

Environment (Docker, Singularity / Apptainer, custom installation):

The download is being run directly from the shell on an HPC system over SSH; DataLad/git-annex are not being run inside a Docker, Singularity, or Apptainer container.
The datasets are located on a BeeGFS distributed filesystem:
/mnt/beegfs/wrkdir/u111428/RBC_data/
The two relevant DataLad datasets are separate fMRIPrep derivative datasets for anatomical and functional data.

Data formatted according to a validatable standard? Please provide the output of the validator:

N/A. This issue occurs during retrieval of annexed file content with DataLad/git-annex and does not appear to be related to BIDS validation.

N/A — this is a DataLad/git-annex retrieval issue, not a BIDS validation issue.

Relevant log outputs (up to 20 lines):

get(error): sub-NDARBF998MBA/ses-1/anat/sub-NDARBF998MBA_ses-1_acq-VNav_rec-Norm_hemi-R_sulc.shape.gii (file) [git-annex: sqlite worker thread crashed: user error (SQLite3 returned ErrorNotFound while attempting to perform step.(while opening database connection))]
get(error): sub-NDARBF998MBA/ses-1/anat/sub-NDARBF998MBA_ses-1_acq-VNav_rec-Norm_space-fsLR_den-91k_thickness.dscalar.nii (file) [git-annex: sqlite worker thread crashed: user error (SQLite3 returned ErrorNotFound while attempting to perform step.(while opening database connection))]
get(error): sub-NDARBF998MBA/ses-1/anat/sub-NDARBF998MBA_ses-1_acq-VNav_rec-Norm_hemi-R_thickness.shape.gii (file) [git-annex: sqlite worker thread crashed: user error (SQLite3 returned ErrorNotFound while attempting to perform step.(while opening database connection))]
get(error): sub-NDARBF998MBA/ses-1/anat/sub-NDARBF998MBA_ses-1_acq-VNav_rec-Norm_from-T1w_to-MNI152NLin2009cAsym_mode-image_xfm.h5 (file) [git-annex: sqlite worker thread crashed: user error (SQLite3 returned ErrorNotFound while attempting to perform step.(while opening database connection))]
get(error): sub-NDARBF998MBA/ses-1/anat/sub-NDARBF998MBA_ses-1_acq-VNav_rec-Norm_hemi-L_space-fsLR_den-32k_midthickness.surf.gii (file) [git-annex: sqlite worker thread crashed: user error (SQLite3 returned ErrorNotFound while attempting to perform step.(while opening database connection))]
get(error): sub-NDARBF998MBA/ses-1/anat/sub-NDARBF998MBA_ses-1_acq-VNav_rec-Norm_hemi-R_pial.surf.gii (file) [git-annex: sqlite worker thread crashed: user error (SQLite3 returned ErrorNotFound while attempting to perform step.(while opening database connection))]
get(error): sub-NDARBF998MBA/ses-1/anat/sub-NDARBF998MBA_ses-1_acq-VNav_rec-Norm_hemi-R_sphere.surf.gii (file) [git-annex: sqlite worker thread crashed: user error (SQLite3 returned ErrorNotFound while attempting to perform step.(while opening database connection))]
get(error): sub-NDARBF998MBA/ses-1/anat/sub-NDARBF998MBA_ses-1_acq-VNav_rec-Norm_from-MNI152NLin2009cAsym_to-T1w_mode-image_xfm.h5 (file) [git-annex: sqlite worker thread crashed: user error (SQLite3 returned ErrorNotFound while attempting to perform step.(while opening database connection))]
get(error): sub-NDARBF998MBA/ses-1/anat/sub-NDARBF998MBA_ses-1_acq-VNav_rec-Norm_hemi-L_pial.surf.gii (file) [git-annex: sqlite worker thread crashed: user error (SQLite3 returned ErrorNotFound while attempting to perform step.(while opening database connection))]
get(error): sub-NDARBF998MBA/ses-1/anat/sub-NDARBF998MBA_ses-1_acq-VNav_rec-Norm_hemi-L_space-fsLR_den-32k_pial.surf.gii (file) [git-annex: sqlite worker thread crashed: user error (SQLite3 returned ErrorNotFound while attempting to perform step.(while opening database connection))]
get(error): sub-NDARBF998MBA/ses-1/anat/sub-NDARBF998MBA_ses-1_acq-VNav_rec-Norm_space-MNI152NLin6Asym_res-2_desc-brain_mask.nii.gz (file) [git-annex: sqlite worker thread crashed: user error (SQLite3 returned ErrorNotFound while attempting to perform step.(while opening database connection))]
get(error): sub-NDARBF998MBA/ses-1/anat/sub-NDARBF998MBA_ses-1_acq-VNav_rec-Norm_label-GM_probseg.nii.gz (file) [git-annex: sqlite worker thread crashed: user error (SQLite3 returned ErrorNotFound while attempting to perform step.(while opening database connection))]

Screenshots / relevant information:

yarikoptic · August 10, 2026, 7:40pm

I would recommend to upgrade to the most recent release since the one you had I believe some fixes related to beegfs were introduced, see e.g. 35 failed tests on beegfs . Please get back whether newer git-annex addressed your issues.

Gandom · August 11, 2026, 10:20am

Thank you for the suggestion. I upgraded git-annex to a newer version. However, I am still having difficulties with the downloads. At the moment, I am not completely sure what the current issue is, as I have asked our technical support team to help with the download process.

The main concern for me now is time. Due to the current situation in the region, Internet access in Iran is quite unstable, and there is a possibility of unexpected Internet disruptions. For this reason, I am trying to retrieve the data I need as soon as possible while access is still available.

For my project, I need data for approximately 280 participants. Would it be possible to obtain direct download links for the required files, so that I could download them without relying on DataLad/git-annex?

If direct HTTP/HTTPS or other download URLs are available, or if there is a way to generate a list of direct URLs for a specified set of participants, that would be extremely helpful.

Thank you very much for your help.

yarikoptic · August 11, 2026, 1:27pm

what is exactly the source dataset URL?

the overall answer is “it depends”. Try git annex whereis (there is --json for machine readable output). If there are URLs per files of interest – there you have them. If not – might need special remotes access where no direct URLs are associated.

tientong · August 11, 2026, 2:18pm

All the binary contents are on AWS.

As @yarikoptic mentioned, you can use git annex whereis to get the s3 link. Below is an example with a file you had trouble downloading:

git annex whereis sub-NDARBF998MBA/ses-1/anat/sub-NDARBF998MBA_ses-1_acq-VNav_rec-Norm_hemi-R_sulc.shape.gii       
whereis sub-NDARBF998MBA/ses-1/anat/sub-NDARBF998MBA_ses-1_acq-VNav_rec-Norm_hemi-R_sulc.shape.gii (1 copy) 
  	c88d7f7e-2980-4610-9367-f2dd28cd6894 -- [fcp-indi]

  fcp-indi: https://fcp-indi.s3.amazonaws.com/data/Projects/RBC/HBN_fMRIPrep-Func/SHA256E-s676879--e01efec2b061e9fc3bd5ad44128523880555cf20b4def12a7f159a0e5f945fc5.gii
ok

Then you can download with

aws s3 cp \
  s3://fcp-indi/data/Projects/RBC/HBN_fMRIPrep-Func/SHA256E-s676879--e01efec2b061e9fc3bd5ad44128523880555cf20b4def12a7f159a0e5f945fc5.gii \
  sub-NDARBF998MBA/ses-1/anat/sub-NDARBF998MBA_ses-1_acq-VNav_rec-Norm_hemi-R_sulc.shape.gii \
  --no-sign-request