Unexpected error of `database or disk is full` using xcp_d

psychelzh · November 28, 2023, 4:00am

Summary of what happened:

A simple use case of xcp_d reported database or disk is full error 3 minutes after it started.

Command used (and if a helper script was used, a link to the helper script or the command generated):

singularity run -c -e -B /seastor:/seastor -B /home/zhangliang:/home/zhangliang /opt/fmritools/containers/xcp_d-0.5.2.sif \
  /seastor/CAMP/derivatives/fmriprep /seastor/CAMP/derivatives/example \
  `# filtering bids queries` \
  --participant_label SICNU001 \
  --cifti \
  `# performance options` \
  --nthreads 8 \
  --omp-nthreads 8 \
  --mem_gb 12 \
  `# other options` \
  --work-dir /seastor/CAMP/tmp \
  --notrack \
  `# freesurfere options` \
  --fs-license-file /seastor/zhangliang/license.txt

Version:

xcp_d v0.5.2

Environment (Docker, Singularity, custom installation):

Singularity

Data formatted according to a validatable standard? Please provide the output of the validator:

Our dataset was generated by a previous run of fmriprep.

Relevant log outputs (up to 20 lines):

Process Process-2:
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1276, in _execute_context
    self.dialect.do_execute(
  File "/usr/local/miniconda/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 608, in do_execute
    cursor.execute(statement, parameters)
sqlite3.OperationalError: database or disk is full

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/local/miniconda/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/miniconda/lib/python3.8/site-packages/xcp_d/cli/run.py", line 901, in build_workflow
    layout = BIDSLayout(str(opts.fmri_dir), validate=False, derivatives=True)
  File "/usr/local/miniconda/lib/python3.8/site-packages/bids/layout/layout.py", line 152, in __init__
    indexer(self)
  File "/usr/local/miniconda/lib/python3.8/site-packages/bids/layout/index.py", line 153, in __call__
    self._index_metadata()
  File "/usr/local/miniconda/lib/python3.8/site-packages/bids/layout/index.py", line 285, in _index_metadata
    all_files = self._layout.get(absolute_paths=True, **filters)
  File "/usr/local/miniconda/lib/python3.8/site-packages/bids/layout/layout.py", line 653, in get
    results.extend(query.all())
  File "/usr/local/miniconda/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3373, in all
    return list(self)
  File "/usr/local/miniconda/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3535, in __iter__
    return self._execute_and_instances(context)
  File "/usr/local/miniconda/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3560, in _execute_and_instances
    result = conn.execute(querycontext.statement, self._params)
  File "/usr/local/miniconda/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1011, in execute
    return meth(self, multiparams, params)
  File "/usr/local/miniconda/lib/python3.8/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/usr/local/miniconda/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1124, in _execute_clauseelement
    ret = self._execute_context(
  File "/usr/local/miniconda/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1316, in _execute_context
    self._handle_dbapi_exception(
  File "/usr/local/miniconda/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1510, in _handle_dbapi_exception
    util.raise_(
  File "/usr/local/miniconda/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
    raise exception
  File "/usr/local/miniconda/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1276, in _execute_context
    self.dialect.do_execute(
  File "/usr/local/miniconda/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 608, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) database or disk is full
[SQL: SELECT files.path AS files_path, files.filename AS files_filename, files.dirname AS files_dirname, files.is_dir AS files_is_dir, files.class_ AS files_class_ 
FROM files 
WHERE files.is_dir = 0 GROUP BY files.path]
(Background on this error at: http://sqlalche.me/e/13/e3q8)

Screenshots / relevant information:

OS:

No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 12 (bookworm)
Release:        12
Codename:       bookworm

Steven · November 28, 2023, 4:05am

Hi @psychelzh,

What kind of storage do you have in /seastor? Also…

For simplicity you can just write -B /seastor,/home/zhangliang. Can you also do -C instead of -c in the singularity arguments?

Best,
Steven

psychelzh · November 28, 2023, 4:15am

Thanks! I have stored all my data in /seastor sector. Specifically, the BIDS data structure is in the CAMP subdirectory. Here are screenshots of the dataset:

Unfortunately, changing -c to -C does not help.

Steven · November 28, 2023, 1:35pm

Hi @psychelzh,

I was referring to how much space you have left in seastor. Is there any chance you’re out of storage?

Best,
Steven

psychelzh · November 28, 2023, 5:01pm

Gotcha. It turns out that I have enough space and quota, about 1.5TB. So I tend to think it is not because the disk is full.

Steven · November 28, 2023, 6:04pm

Your working directory (/seastor/CAMP/tmp) appears to be in your BIDS directory (/seastor/CAMP/), try moving it out of there.

psychelzh · November 29, 2023, 8:19am

It does not help. I also searched help from sqlite side, but nothing really helps.

psychelzh · December 1, 2023, 3:23am

Interestingly, the following command succeeded:

singularity exec -C -e -B /seastor,/home/zhangliang /opt/fmritools/containers/xcp_d-0.5.2.sif pybids layout /seastor/CAMP/derivatives/fmriprep /seastor/CAMP/test --no-validate
# Successfully generated database index at /seastor/CAMP/test

So, I am wondering why workflow will fail. See also this cross-reference issue Unexpected error database or disk is full · Issue #1001 · PennLINC/xcp_d (github.com) and the related issue in fmiprep sqlite3.OperationalError: database or disk is full · Issue #2525 · nipreps/fmriprep (github.com). Can anyone help me about this?

psychelzh · December 1, 2023, 12:16pm

Finally found the real cause. I cannot add -C or -c option to the call of singularity, i.e., the following code works:

singularity run -e -B /seastor,/home/zhangliang /opt/fmritools/containers/xcp_d-0.5.2.sif \
  /seastor/CAMP/derivatives/fmriprep /seastor/CAMP/derivatives/example \
  `# filtering bids queries` \
  --participant_label SICNU001 \
  --cifti \
  `# performance options` \
  --nthreads 8 \
  --omp-nthreads 8 \
  --mem_gb 12 \
  `# other options` \
  --work-dir /seastor/CAMP/tmp \
  --notrack \
  `# freesurfere options` \
  --fs-license-file /seastor/zhangliang/license.txt

However, it is kind of strange because fmriprep will work if -C or -c is added.