Load preset paired t-test with CPAC

luisagulleiro · April 8, 2025, 3:59pm

Hello everyone!

I’m using C-PAC (v1.8.7.dev1) on a HPC (singularity v3.11.5). I want to run an analysis comparing two groups pre-post, for which I’m trying to get the paired-two preset from the command line.

These are the arguments that cpac asks for:

Usage: cpac group feat load-preset paired-two [OPTIONS] PIPELINE_DIR Z_THRESH
                                              P_THRESH CONDITIONS
                                              CONDITION_TYPE MODEL_NAME

So I’m entering:

singularity exec c-pac.sif \
cpac group feat load-preset paired-two /path/to/pipeline_cpac-ost-fix 2.3 0.05 [pre, post] session --output-dir /path/to/output

For which I get this error message:

INFO:    Converting SIF file to temporary sandbox...
/usr/share/fsl/6.0/bin/cpac:4: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
  __import__('pkg_resources').require('CPAC==1.8.7.dev1')
Traceback (most recent call last):
  File "/code/CPAC/utils/create_fsl_flame_preset.py", line 998, in run
    keys = pd.read_csv(keys_csv)
  File "/usr/share/fsl/6.0/lib/python3.10/site-packages/pandas/io/parsers/readers.py", line 912, in read_csv
    return _read(filepath_or_buffer, kwds)
  File "/usr/share/fsl/6.0/lib/python3.10/site-packages/pandas/io/parsers/readers.py", line 577, in _read
    parser = TextFileReader(filepath_or_buffer, **kwds)
  File "/usr/share/fsl/6.0/lib/python3.10/site-packages/pandas/io/parsers/readers.py", line 1407, in __init__
    self._engine = self._make_engine(f, self.engine)
  File "/usr/share/fsl/6.0/lib/python3.10/site-packages/pandas/io/parsers/readers.py", line 1661, in _make_engine
    self.handles = get_handle(
  File "/usr/share/fsl/6.0/lib/python3.10/site-packages/pandas/io/common.py", line 859, in get_handle
    handle = open(
FileNotFoundError: [Errno 2] No such file or directory: '/code/CPAC/resources/cpac_outputs.csv'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/share/fsl/6.0/bin/cpac", line 7, in <module>
    exec(compile(f.read(), __file__, 'exec'))
  File "/code/scripts/cpac", line 5, in <module>
    main()
  File "/usr/share/fsl/6.0/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/usr/share/fsl/6.0/lib/python3.10/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/usr/share/fsl/6.0/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/share/fsl/6.0/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/share/fsl/6.0/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  [Previous line repeated 1 more time]
  File "/usr/share/fsl/6.0/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/share/fsl/6.0/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/code/CPAC/__main__.py", line 258, in paired_two
    create_fsl_flame_preset.run(pipeline_dir, 'all', z_thresh, p_thresh,
  File "/code/CPAC/utils/create_fsl_flame_preset.py", line 1002, in run
    raise Exception(err)
Exception: 
[!] Could not access or read the cpac_outputs.csv resource file:
/code/CPAC/resources/cpac_outputs.csv

Error details [Errno 2] No such file or directory: '/code/CPAC/resources/cpac_outputs.csv'

I have tried changing PIPELINE_DIR to the group_analysis_participants.txt file, different combinations of singularity run and singularity exec, as well as different inputs for pre and post with the same result.
I was checking C-PAC’s GitHub and it seems that cpac_outputs is a .tsv file (~/CPAC/resources/cpac_outputs.tsv), which I’m not sure if could be the issue?

Thanks in advance!!

~L

tamsinrogers · April 10, 2025, 7:35pm

Hi @luisagulleiro, thanks for reaching out!

That is a strange error.

I know you said you’re using v1.8.7.dev1, but could you triple-check that this is actually the version that’s running? Will be in log/.../pypeline.log next to C-PAC version: .

luisagulleiro · April 10, 2025, 8:19pm

Hi @tamsinrogers, thanks for your reply!

I’ve checked log/.../pypeline.log from the previous preprocessing run and it says C-PAC version: 1.8.7.dev1.

Just to confirm, I ran singularity exec c-pac.sif cpac version, which returned:

Environment
===========
AFNI: 23.3.09:linux_universal
ldd: (Ubuntu GLIBC 2.35-0ubuntu3.1) 2.35
Python: 3.10.6 | packaged by conda-forge | (main, Aug 22 2022, 20:36:39) [GCC 10.4.0]
  Python packages
  ---------------

[...long list of packages...]

C-PAC version: 1.8.7.dev1

tamsinrogers · April 11, 2025, 3:56pm

Hi @luisagulleiro,

Thanks for checking! As it turns out, cpac_outputs.csv was in fact changed to cpac_outputs.tsv in v1.8.1, but it was not updated in some places - the group-level builder hasn’t been updated/supported for quite some time.

Our team is working on a fix and I’ll get back to you shortly.

Thank you for bringing this to our attention!

tamsinrogers · April 14, 2025, 3:14pm

Hi @luisagulleiro,

This update has just been merged to dev - should work as expected now when using the nightly image. Please let me know if you run into any additional issues!

luisagulleiro · April 18, 2025, 3:03pm

Hi @tamsinrogers,

Thanks for your help! I’ve pulled the nightly image and now the command is able to run, but I found a new error message.

I ran the following:

singularity exec --bind /path/to/ost_fix:/path/to/ost_fix c-pac:nightly.sif \
cpac group feat load-preset paired-two \
/path/to/ost_fix/output/pipeline_cpac-ost-fix 2.3 0.05 \
[ses-pre,ses-post] session grp_analysis \
--output-dir /path/to/ost_fix/output

Which returns this error:

INFO:    Converting SIF file to temporary sandbox...
/usr/share/fsl/6.0/bin/cpac:4: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
  __import__('pkg_resources').require('CPAC==1.8.8.dev1')
Traceback (most recent call last):
  File "/usr/share/fsl/6.0/bin/cpac", line 7, in <module>
    exec(compile(f.read(), __file__, 'exec'))
  File "/code/scripts/cpac", line 5, in <module>
    main()
  File "/usr/share/fsl/6.0/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/usr/share/fsl/6.0/lib/python3.10/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/usr/share/fsl/6.0/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/share/fsl/6.0/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/share/fsl/6.0/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  [Previous line repeated 1 more time]
  File "/usr/share/fsl/6.0/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/share/fsl/6.0/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/code/CPAC/__main__.py", line 318, in paired_two
    create_fsl_flame_preset.run(
  File "/code/CPAC/utils/create_fsl_flame_preset.py", line 1291, in run
    design_df, contrasts_df, group_config_update = preset_paired_two_group(
  File "/code/CPAC/utils/create_fsl_flame_preset.py", line 695, in preset_paired_two_group
    design_df = create_design_matrix_df(group_list, ses_id=sesflag)
  File "/code/CPAC/utils/create_fsl_flame_preset.py", line 224, in create_design_matrix_df
    sess = part_ses.split("_")[1]
IndexError: list index out of range
INFO:    Cleaning up image...

I modified the names of the directories to not include underscores (ie, ost-fix, pipeline-cpac-ost-fix), but it returns the same error. I also tried a variation of the session string as [pre,post] only, with the same result.

tamsinrogers · April 18, 2025, 8:05pm

Hi @luisagulleiro,

Looking at /utils/create_fsl_flame_preset.py/create_design_matrix, group_list needs to look something like this:

["sub-01_ses-pre", "sub-01_ses-post", ...]

I know you said you tried without underscores, and looks like in your initial run command you’re giving it ses-pre,ses-post. Can you try rerunning with your group list in the above format? If that doesn’t work I can dig deeper.

luisagulleiro · April 21, 2025, 3:27pm

Hi @tamsinrogers,

I’ve tried the following command:

singularity exec --bind /path/to/ost_fix:/path/to/ost_fix c-pac:nightly.sif \
cpac group feat load-preset paired-two \
/path/to/ost_fix/output/pipeline_cpac-ost-fix \ #This is PIPELINE_DIR, where all the preprocessed files and derivatives are. Structure looks like ~/pipeline_cpac-ost-fix/678301/ses-pre/func/
2.3 0.05 \ #Z_THRESH and P_THRESH
["sub-678301_ses-post","sub-678301_ses-pre","sub-678311_ses-post","sub-678311_ses-pre", ...] \ # I've also tried "678301_pre" (this is how the group_analysis_participants.txt names them) and "678301_ses-pre" (to match the naming of PIPELINE_DIR) with the same result
session grp_analysis \
--output-dir /path/to/ost_fix/output

But I get this error message:

/usr/share/fsl/6.0/bin/cpac:4: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
  __import__('pkg_resources').require('CPAC==1.8.8.dev1')
Traceback (most recent call last):
  File "/usr/share/fsl/6.0/bin/cpac", line 7, in <module>
    exec(compile(f.read(), __file__, 'exec'))
  File "/code/scripts/cpac", line 5, in <module>
    main()
  File "/usr/share/fsl/6.0/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/usr/share/fsl/6.0/lib/python3.10/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/usr/share/fsl/6.0/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/share/fsl/6.0/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/share/fsl/6.0/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  [Previous line repeated 1 more time]
  File "/usr/share/fsl/6.0/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/share/fsl/6.0/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/code/CPAC/__main__.py", line 318, in paired_two
    create_fsl_flame_preset.run(
  File "/code/CPAC/utils/create_fsl_flame_preset.py", line 1291, in run
    design_df, contrasts_df, group_config_update = preset_paired_two_group(
  File "/code/CPAC/utils/create_fsl_flame_preset.py", line 686, in preset_paired_two_group
    raise Exception
Exception

Thanks for your help!!