Run cwas pipeline with cpac with already preprocessed fMRI data

Hi, neurostarts experts :slight_smile:

I have a question on running cwas (one of group-level analysis pipelines) thru cpac with already preprocessed fMRI data (by fMRIprep).

I tried to figure out what I should do to run CWAS pipeline but having a hard time understanding overall steps (what to do first, second and so on).

I have created data_settings.yml and data_config.yml files.

So, I guess it’s time to select my pipeline that will run cwas without any preprocessing steps.

(this page) told me to add the following lines to my file with appropriate paths.

runCWAS = [0] cwas_roi_file = ‘/path/to/cwas_mask_file’ cwas_regressor_file = ‘/path/to/cwas_regressor_file’ cwas_regressor_participant_column = 'ID cwas_regressor_columns = [‘FIQ’, ‘VIQ’, ‘PIQ’] cwas_permutations = 500 cwas_parallel_nodes = 3

I want to make sure where exactly should I add these lines since I don’t need any preprocesssing. I thought cpac gui will make my head clear but the page somehow reports an error (Something went wrong!) everytime I try to acccess.

Is it okay to manually create and run pipeline.yml that only contains the upper lines (since I don’t need any preprocessing steps) to run cwas?

And I found MDMR workflow developer documentation) where there are 2 relevant source codes (CPAC.cwas.pipeline & CPAC.cwas.cwas).

I want to check if I should run CPAC.cwas.cwas first and then run CPAC.cwas.pipeline (seems like the latter is run with input from the former) and that if that’s all for cwas analysis.

So, I wonder if there’s anyone who can explain sequential steps to run cwas in this case…

I’ll be so grateful for any of your help.

Thank you in advance and hope you have a great day!!!

Susan.

Hello Susan!

My name is Diego and I have been working with C-PAC to update the deprecated group analysis parts in C-PAC. Specifically I have been working on the changes to have CWAS/MDMR function with the BIDS preprocessed participant data. I can share with you my fixes for the code (GitHub - diegoaper/C-PAC_MDMR-Group_Runner: Configurable Pipeline for the Analysis of Connectomes) however use this with caution as we are still testing and trying to make sure all the bugs are removed before committing the changes. If you are still interested in working with my changes I am more than happy to help troubleshoot as you go. We are also updating the documentation as we go along.

Hello Diego,

I have one question about CWAS. Does the mask used here refer to which region the analysis is restricted to? If I want a whole brain analysis, then I need to input a gray matter mask. If I wanna restrict my analysis within a local brain region/regions, then I need a mask of the region/regions. Is that right?

runCWAS = [0]
cwas_roi_file = ‘/path/to/cwas_mask_file’
cwas_regressor_file = ‘/path/to/cwas_regressor_file’
cwas_regressor_participant_column = 'ID
cwas_regressor_columns = [‘FIQ’, ‘VIQ’, ‘PIQ’]
cwas_permutations = 500
cwas_parallel_nodes = 3

Best regards,
Elenita

Hello Elenita,

Yes that is correct.

Best,

Diego

1 Like

Hi Diego,

Thank you for the timely reply!
I added these lines at the end of my config yaml file as follows,

runCWAS:
  run: on
  cwas_roi_file: /home/mask_file/GM_mask.nii.gz
  cwas_regressor_file: /home/cwas_regressor_file.tsv
  cwas_regressor_participant_column: ID
  cwas_regressor_columns:
    - IQ
    - group
  cwas_permutations: 5000
  cwas_parallel_nodes: 4

However error occurred:

voluptuous.error.MultipleInvalid: extra keys not allowed @ data[‘runCWAS’]

Could you please tell me what went wrong.

BTW, the voxel size of the mask file in cwas should be the same as the func data (4mm), right?

Thank you!

Best,
Elenita

Hello,

Are you using the CPAC branch that I linked in the beginning of this thread? It has the updated scripts that you can call that have these changes added. The issue you are running into is that the compiler does not have a module called “runCWAS” its actually called “mdmr” in the config file. These scripts have the update work that should allow you to run CWAS/MDMR:

  • main/CPAC/pipeline/cpac_group_runner.py
  • main/CPAC/resources/cpac_outputs.tsv
  • main/CPAC/cwas/cwas.py
  • configs/group_config_template.yml
  • main/CPAC/cwas/mdmr.py
  • main/CPAC/cwas/pipeline.py

Yes, the mask needs to be the same dimensions as the data. If you don’t have CPAC installed in your computer you can run CPAC on docker (1.8.4) and pipe these scripts from a local copy. I can help you more once you have reached this step!

Best,

Diego

1 Like

Hello Diego!

Thank you so much for the help! Sorry for not clarifying the context of my use. I am running CPAC(1.8.4.dev) on docker. I tried to edit the cpac_pipeline_config yaml file as I mentioned above to run CWAS/MDMR. So how could I pipe these scripts from a local copy in this case? Thank you!

Best,

Elenita

Hello!

Sure, this is an example of what I used to run from my terminal:

docker run -i -v [Local Copy Directory]/cwas.py:/code/CPAC/cwas/cwas.py \
-v [Local Copy Directory]/cpac_group_runner.py:/code/CPAC/pipeline/cpac_group_runner.py \
-v [Local Copy Directory]/pipeline.py:/code/CPAC/cwas/pipeline.py \
-v [Local Copy Directory]/mdmr.py:/code/CPAC/cwas/mdmr.py \
-v [Local Copy Directory]/cpac_outputs.tsv:/code/CPAC/resources/cpac_outputs.tsv \
-v /output:/mdmr_output \
-v /configs:/configs \
-v /working:/working \
-v /log:/log \
-v /crash:/crash \
fcpindi/c-pac:release-v1.8.4 /preproc /mdmr_output group --group_file /mdmr_output/group_config.yml --save_working_dir --tracking_opt-out

With [Local Copy Directory] where you are storing the scripts from the branch. It follows the idea of -v location of script : to the script you want to replace. If that makes sense. Let me know if you have more questions!

Best,

Diego

Hi Diego!

Thank you for your patient guidance!
I followed this example command and edited the group_config.yml with appropriate substitutions for paths. I am not sure whether it is because I am still preprocessing the data, and there is an error:

Traceback (most recent call last):
  File "/code/run.py", line 807, in <module>
    run_main()
  File "/code/run.py", line 420, in run_main
    cgr.run(args.group_file)
  File "/code/CPAC/pipeline/cpac_group_runner.py", line 2068, in run
    if 1 in c["mdmr"]["run"]:
KeyError: 'mdmr'

Best regards,

Elenita

Hello,

If you are still preprocessing it would give an error that it couldn’t find all the data so no problems there!

It seems like the group_config.yml that you used didn’t have the following arguments:

# Multivariate Distance Matrix Regression (MDMR)
mdmr:

  # Used to determine if Multivariate Distance Matrix Regression (MDMR) will be added to the pipeline or not.
  run:  [0]

  # Inclusion list text file listing the participant IDs you wish to include in the MDMR analysis. If left as None, will include all subjects.
  inclusion_list : None

  # Path to a mask file. Voxels outside of the mask will be excluded from MDMR.
  roi_file: /path

  # Path to a CSV file containing the phenotypic regressor.
  regressor_file:

  # Name of the participants column in your regressor file.
  regressor_participant_column: ''

  # Columns from the CSV file indicating factor variables. Other columns will be handled as covariates. Separated by commas.
  regressor_columns: ''

  # Number of permutation tests to run on the Pseudo-F statistics.
  permutations:  15000

  # Number of Nipype nodes created while computing MDMR. Dependent upon computing resources.
  parallel_nodes:  10

  # If you want to create zstat maps
  zscore: [1]

Make sure this is the config file you are using and it is in the output folder so that the --group_file /mdmr_output/group_config.yml works.

Best,

Diego

Hi Diego!

I was using the
group_config.txt (14.2 KB)
of the version 1.8.4.dev (fcpindi/c-pac:latest), and there are a few difference between that of the release of 1.8.4. I switched to the latter, but it seems like it cannot find the output directory of my individual-level analysis pipeline run, which I have finished using 1.8.4.dev.

Gathering the output file paths from /home/cpac_cwas/output/cpac_cpac-default-pipeline...
Traceback (most recent call last):
  File "/code/run.py", line 807, in <module>
    run_main()
  File "/code/run.py", line 420, in run_main
    cgr.run(args.group_file)
  File "/code/CPAC/pipeline/cpac_group_runner.py", line 2069, in run
    run_cwas(config_file)
  File "/code/CPAC/pipeline/cpac_group_runner.py", line 1347, in run_cwas
    inclusion=inclusion)
  File "/code/CPAC/pipeline/cpac_group_runner.py", line 1267, in run_cwas_group
    get_func=True)
  File "/code/CPAC/pipeline/cpac_group_runner.py", line 489, in gather_outputs
    get_func
  File "/code/CPAC/pipeline/cpac_group_runner.py", line 195, in gather_nifti_globs
    raise Exception(err)
Exception:

[!] No output filepaths found in the pipeline output directory provided for the derivatives selected!

Pipeline output directory provided: /home/cpac_cwas/output/cpac_cpac-default-pipeline
Derivatives selected:['space-template_desc-preproc_bold']

I have checked my output path. Is it because of different versions? Or maybe it is because I have two results: space-template_desc-preproc-1_bold (with GSR) and space-template_desc-preproc-2_bold(without GSR)? I’ve tried. I don’t think that’s the reason. Do I need to unify to version 1.8.4? It seems that the two versions are the same image on docker.

Best,

Elenita

Hello,

I see, does the docker container have access to your /home directory? You will have to pipe in your home directory as -v /home:/home so that the container can access that directory and all the paths should work from there. Try this and see if the images are found!

Best,

Diego

I tried but it seems like it didn’t work…

Hello,

Sorry I am out of office for the rest of the week. But I asked a coworker to look into it. I think it might be because of the names to your files but I’d have to check the output directory. Did you include an “inclusion_list”? The path to the images should be just output file directory. Then the subfolders are the cpac pipeline name then all the subjects like this:

output
> cpac_pipeline
>> sub-####
   sub-####
   .
   .
   .

I think how the code works as well you cannot run the subjects in the same folder unless you have the inclusion list. I believe if you specify the in the subject list or create a separate folder for the GSR vs w/o GSR that might work better. I’ll have to look into this when I return. Sorry for the inconvenience.

Best,

Diego

Hello Diego!

I’m honored that my issue is getting attention.

I included the “inclusion_list” text file. And I changed the format of the name of output file by deleting -1 in sub-xxx_ses-1_task-rest_space-template_desc-preproc-1_bold.nii.gz. I test for three subjects and this time it works. While there was no outputs in the mdmr_output folder, and only in the temporary folder. There was no error reproted so I don’t know what the problem is. I also want to know what kind of files mdmr_cwas would output and what they represent.

Thank you so much!

Best,
Elenita