Hi, neurostarts experts 
I have a question on running cwas (one of group-level analysis pipelines) thru cpac with already preprocessed fMRI data (by fMRIprep).
I tried to figure out what I should do to run CWAS pipeline but having a hard time understanding overall steps (what to do first, second and so on).
I have created data_settings.yml and data_config.yml files.
So, I guess itās time to select my pipeline that will run cwas without any preprocessing steps.
(this page) told me to add the following lines to my file with appropriate paths.
runCWAS = [0] cwas_roi_file = ā/path/to/cwas_mask_fileā cwas_regressor_file = ā/path/to/cwas_regressor_fileā cwas_regressor_participant_column = 'ID cwas_regressor_columns = [āFIQā, āVIQā, āPIQā] cwas_permutations = 500 cwas_parallel_nodes = 3
I want to make sure where exactly should I add these lines since I donāt need any preprocesssing. I thought cpac gui will make my head clear but the page somehow reports an error (Something went wrong!) everytime I try to acccess.
Is it okay to manually create and run pipeline.yml that only contains the upper lines (since I donāt need any preprocessing steps) to run cwas?
And I found MDMR workflow developer documentation) where there are 2 relevant source codes (CPAC.cwas.pipeline & CPAC.cwas.cwas).
I want to check if I should run CPAC.cwas.cwas first and then run CPAC.cwas.pipeline (seems like the latter is run with input from the former) and that if thatās all for cwas analysis.
So, I wonder if thereās anyone who can explain sequential steps to run cwas in this caseā¦
Iāll be so grateful for any of your help.
Thank you in advance and hope you have a great day!!!
Susan.
Hello Susan!
My name is Diego and I have been working with C-PAC to update the deprecated group analysis parts in C-PAC. Specifically I have been working on the changes to have CWAS/MDMR function with the BIDS preprocessed participant data. I can share with you my fixes for the code (GitHub - diegoaper/C-PAC_MDMR-Group_Runner: Configurable Pipeline for the Analysis of Connectomes) however use this with caution as we are still testing and trying to make sure all the bugs are removed before committing the changes. If you are still interested in working with my changes I am more than happy to help troubleshoot as you go. We are also updating the documentation as we go along.
Hello Diego,
I have one question about CWAS. Does the mask used here refer to which region the analysis is restricted to? If I want a whole brain analysis, then I need to input a gray matter mask. If I wanna restrict my analysis within a local brain region/regions, then I need a mask of the region/regions. Is that right?
runCWAS = [0]
cwas_roi_file = ā/path/to/cwas_mask_fileā
cwas_regressor_file = ā/path/to/cwas_regressor_fileā
cwas_regressor_participant_column = 'ID
cwas_regressor_columns = [āFIQā, āVIQā, āPIQā]
cwas_permutations = 500
cwas_parallel_nodes = 3
Best regards,
Elenita
Hi Diego,
Thank you for the timely reply!
I added these lines at the end of my config yaml file as follows,
runCWAS:
run: on
cwas_roi_file: /home/mask_file/GM_mask.nii.gz
cwas_regressor_file: /home/cwas_regressor_file.tsv
cwas_regressor_participant_column: ID
cwas_regressor_columns:
- IQ
- group
cwas_permutations: 5000
cwas_parallel_nodes: 4
However error occurred:
voluptuous.error.MultipleInvalid: extra keys not allowed @ data[ārunCWASā]
Could you please tell me what went wrong.
BTW, the voxel size of the mask file in cwas should be the same as the func data (4mm), right?
Thank you!
Best,
Elenita
Hello,
Are you using the CPAC branch that I linked in the beginning of this thread? It has the updated scripts that you can call that have these changes added. The issue you are running into is that the compiler does not have a module called ārunCWASā its actually called āmdmrā in the config file. These scripts have the update work that should allow you to run CWAS/MDMR:
- main/CPAC/pipeline/cpac_group_runner.py
- main/CPAC/resources/cpac_outputs.tsv
- main/CPAC/cwas/cwas.py
- configs/group_config_template.yml
- main/CPAC/cwas/mdmr.py
- main/CPAC/cwas/pipeline.py
Yes, the mask needs to be the same dimensions as the data. If you donāt have CPAC installed in your computer you can run CPAC on docker (1.8.4) and pipe these scripts from a local copy. I can help you more once you have reached this step!
Best,
Diego
1 Like
Hello Diego!
Thank you so much for the help! Sorry for not clarifying the context of my use. I am running CPAC(1.8.4.dev) on docker. I tried to edit the cpac_pipeline_config yaml file as I mentioned above to run CWAS/MDMR. So how could I pipe these scripts from a local copy in this case? Thank you!
Best,
Elenita
Hello!
Sure, this is an example of what I used to run from my terminal:
docker run -i -v [Local Copy Directory]/cwas.py:/code/CPAC/cwas/cwas.py \
-v [Local Copy Directory]/cpac_group_runner.py:/code/CPAC/pipeline/cpac_group_runner.py \
-v [Local Copy Directory]/pipeline.py:/code/CPAC/cwas/pipeline.py \
-v [Local Copy Directory]/mdmr.py:/code/CPAC/cwas/mdmr.py \
-v [Local Copy Directory]/cpac_outputs.tsv:/code/CPAC/resources/cpac_outputs.tsv \
-v /output:/mdmr_output \
-v /configs:/configs \
-v /working:/working \
-v /log:/log \
-v /crash:/crash \
fcpindi/c-pac:release-v1.8.4 /preproc /mdmr_output group --group_file /mdmr_output/group_config.yml --save_working_dir --tracking_opt-out
With [Local Copy Directory] where you are storing the scripts from the branch. It follows the idea of -v location of script : to the script you want to replace. If that makes sense. Let me know if you have more questions!
Best,
Diego
Hi Diego!
Thank you for your patient guidance!
I followed this example command and edited the group_config.yml with appropriate substitutions for paths. I am not sure whether it is because I am still preprocessing the data, and there is an error:
Traceback (most recent call last):
File "/code/run.py", line 807, in <module>
run_main()
File "/code/run.py", line 420, in run_main
cgr.run(args.group_file)
File "/code/CPAC/pipeline/cpac_group_runner.py", line 2068, in run
if 1 in c["mdmr"]["run"]:
KeyError: 'mdmr'
Best regards,
Elenita
Hello,
If you are still preprocessing it would give an error that it couldnāt find all the data so no problems there!
It seems like the group_config.yml that you used didnāt have the following arguments:
# Multivariate Distance Matrix Regression (MDMR)
mdmr:
# Used to determine if Multivariate Distance Matrix Regression (MDMR) will be added to the pipeline or not.
run: [0]
# Inclusion list text file listing the participant IDs you wish to include in the MDMR analysis. If left as None, will include all subjects.
inclusion_list : None
# Path to a mask file. Voxels outside of the mask will be excluded from MDMR.
roi_file: /path
# Path to a CSV file containing the phenotypic regressor.
regressor_file:
# Name of the participants column in your regressor file.
regressor_participant_column: ''
# Columns from the CSV file indicating factor variables. Other columns will be handled as covariates. Separated by commas.
regressor_columns: ''
# Number of permutation tests to run on the Pseudo-F statistics.
permutations: 15000
# Number of Nipype nodes created while computing MDMR. Dependent upon computing resources.
parallel_nodes: 10
# If you want to create zstat maps
zscore: [1]
Make sure this is the config file you are using and it is in the output folder so that the --group_file /mdmr_output/group_config.yml works.
Best,
Diego
Hi Diego!
I was using the
group_config.txt (14.2 KB)
of the version 1.8.4.dev (fcpindi/c-pac:latest), and there are a few difference between that of the release of 1.8.4. I switched to the latter, but it seems like it cannot find the output directory of my individual-level analysis pipeline run, which I have finished using 1.8.4.dev.
Gathering the output file paths from /home/cpac_cwas/output/cpac_cpac-default-pipeline...
Traceback (most recent call last):
File "/code/run.py", line 807, in <module>
run_main()
File "/code/run.py", line 420, in run_main
cgr.run(args.group_file)
File "/code/CPAC/pipeline/cpac_group_runner.py", line 2069, in run
run_cwas(config_file)
File "/code/CPAC/pipeline/cpac_group_runner.py", line 1347, in run_cwas
inclusion=inclusion)
File "/code/CPAC/pipeline/cpac_group_runner.py", line 1267, in run_cwas_group
get_func=True)
File "/code/CPAC/pipeline/cpac_group_runner.py", line 489, in gather_outputs
get_func
File "/code/CPAC/pipeline/cpac_group_runner.py", line 195, in gather_nifti_globs
raise Exception(err)
Exception:
[!] No output filepaths found in the pipeline output directory provided for the derivatives selected!
Pipeline output directory provided: /home/cpac_cwas/output/cpac_cpac-default-pipeline
Derivatives selected:['space-template_desc-preproc_bold']
I have checked my output path. Is it because of different versions? Or maybe it is because I have two results: space-template_desc-preproc-1_bold (with GSR) and space-template_desc-preproc-2_bold(without GSR)? Iāve tried. I donāt think thatās the reason. Do I need to unify to version 1.8.4? It seems that the two versions are the same image on docker.
Best,
Elenita
Hello,
I see, does the docker container have access to your /home directory? You will have to pipe in your home directory as -v /home:/home so that the container can access that directory and all the paths should work from there. Try this and see if the images are found!
Best,
Diego
I tried but it seems like it didnāt workā¦
Hello,
Sorry I am out of office for the rest of the week. But I asked a coworker to look into it. I think it might be because of the names to your files but Iād have to check the output directory. Did you include an āinclusion_listā? The path to the images should be just output file directory. Then the subfolders are the cpac pipeline name then all the subjects like this:
output
> cpac_pipeline
>> sub-####
sub-####
.
.
.
I think how the code works as well you cannot run the subjects in the same folder unless you have the inclusion list. I believe if you specify the in the subject list or create a separate folder for the GSR vs w/o GSR that might work better. Iāll have to look into this when I return. Sorry for the inconvenience.
Best,
Diego
Hello Diego!
Iām honored that my issue is getting attention.
I included the āinclusion_listā text file. And I changed the format of the name of output file by deleting -1 in sub-xxx_ses-1_task-rest_space-template_desc-preproc-1_bold.nii.gz. I test for three subjects and this time it works. While there was no outputs in the mdmr_output folder, and only in the temporary folder. There was no error reproted so I donāt know what the problem is. I also want to know what kind of files mdmr_cwas would output and what they represent.
Thank you so much!
Best,
Elenita
Hello,
Iām a C-PAC engineer and noticed this issue you raised. Are you still seeing this issue or has it been resolved for you?
Hi Amy!
I was busy doing something else and put it down for a while. I am so glad to hear from you! So I really want to get more information about MDMR-CWAS!
Best,
Elenita
Hi Elenita,
What kind of information are you looking for? Weād be happy to help point you in the right direction or find resources if possible.
Best,
Steve
Hello!
Currently, I have followed Diegoās example and ensured that the CPAC output folder contains only one pipe of results. As a result, the CWAS output is now exclusively stored in the tmp folder. I would like to confirm if these files in the tmp folder are indeed the final outputs and also gain an understanding of the specific content within each result file.
Thank you!
The intended behavior of CPAC (or any of our pipelines) is to have the final results written to an output directory - not sure why itās going to /tmp, but I can help find out. You said you pulled from fcpindi/c-pac:latest, right? And in your runs, or in the logs, are you seeing that the version is v1.8.4?
Also- could you post a list of the CWAS output files in /tmp that you are seeing and would like more info on?