Missing runs from reports

Shotgunosine · August 5, 2019, 2:29pm

I’m running fmriprep on some longitudinal data our lab has collected and I’m missing lots of runs from the reports. Out of 668 functional runs across 148 subjects that didn’t fail for some other reason, I’ve got 124 runs on 27 subjects that are not appearing on the subject level reports and the figures are not in the figures directory. As far as I can tell, there’s not anything else missing for those subjects and all the other expected outputs are present. Most of these subjects have OSError(“handle is closed”) and No contour levels were found in the output that’s going to standard error, but those messages are present for most of the subjects that aren’t missing runs from their reports as well. I’ve attached the standard out and standard error outputs for one of the subjects that’s missing 7 runs from their reports, along with their HTML report (extension changed to .txt so I can upload it) and the results from running tree on their output directory.

I’m running these subjects in a singularity container created from fmriprep version 1.4.1 on dockerhub and each subject is being run in their own job with 32 gigs of ram, 24 threads, and 200 gigs of local scratch space for the tmp and working directories.

Just to save some digging, here’s the command we ran for this subject:
export TMPDIR=/lscratch/$SLURM_JOB_ID && export SINGULARITY_BINDPATH=/gs3,/gs4,/gs5,/gs6,/gs7,/gs8,/gs9,/gs10,/gs11,/spin1,/scratch,/fdb,/data,/lscratch && mkdir -p $TMPDIR/out && mkdir -p $TMPDIR/wrk && singularity run --cleanenv /data/MBDU/singularity_images/fmriprep_1.4.1.simg /data/MBDU/midla/data/bids $TMPDIR/out participant --participant_label 23649 -w $TMPDIR/wrk --nthreads $SLURM_CPUS_PER_TASK --mem_mb $SLURM_MEM_PER_NODE --fs-license-file /data/MBDU/singularity_images/license.txt --output-spaces MNI152NLin2009cAsym fsaverage6 func --dummy-scans 4 --longitudinal --use-aroma --error-on-aroma-warnings --t2s-coreg && mv $TMPDIR/out /data/MBDU/midla/data/derivatives/fmriprep/run_aroma/sub-23649

sub-23649_html.txt (95.8 KB)
swarm_32696834_59.e.txt (24.1 KB)
swarm_32696834_59.o.txt (2.6 MB)
sub-23649_tree.txt (28.0 KB)

rwblair · August 5, 2019, 3:24pm

I wonder if this bug in python 3.7 is related at all:
https://bugs.python.org/issue34073

Shotgunosine · August 5, 2019, 3:29pm

I don’t know, a lot of the subjects that did have reports generate successfully also have the OSError, so I think that may be a red herring.

rwblair · August 5, 2019, 3:54pm

How short a memory I have from reading your post to reading the logs…

oesteban · August 5, 2019, 6:16pm

@rwblair are we indirectly using asyncio?

rwblair · August 5, 2019, 6:41pm

Not positive if it is being used but jinja2 can make use of it in a limited way.

Shotgunosine · August 6, 2019, 12:17pm

I ran another subject that failed to generate reports for all runs with -vvv. The standard out and standard error outputs for that subject, as well as their report are in a zip file on my google drive: https://drive.google.com/open?id=1R9XqCyvGIsOjw_8TkZ-i4gCHHkJy0a9W
One of the things that’s weird is that the report correctly lists all of the runs at the top, but then only has information on one of them. If we can get this diagnosed, it seems like there might be an opportunity to catch the report mismatch within fmriprep.

Shotgunosine · August 7, 2019, 1:58pm

So I’ve rerun 9 of the subjects that are missing runs from their reports and downloaded the working directories from the compute nodes. All of the svg and html reportlets appear to be generated and present in the working directory, but don’t have corresponding figures directories in the output directory. Looking through the messages printed to standard out, it looks like only the runs being written to reports are getting copied over. For all but one of those 9 subjects, only a single run was written to the report and in one case two reports were written.

I’ve got all the logs and all of the working directories for these subjects. What can I do to continue debugging this?

oesteban · August 7, 2019, 3:56pm

I wonder whether this is a problem on the BIDS structure of the output runs. Can you post a tree of your original dataset (one subject showing this issue)?

oesteban · August 7, 2019, 4:11pm

Additionally, if you want to run the extra mile, my hypothesis is that there is some change in pyBIDS 0.9.2 we’ve missed in the latest update and the selection of reportlets is not working.

The process is as follows:

Given a subject, generate a Report object.
Initialize a BIDS Layout, which is based on this config file.
Run queries scripted in the report template.

Shotgunosine · August 7, 2019, 4:12pm

├── ses-01
│   ├── anat
│   │   ├── sub-23017_ses-01_acq-mprage_rec-ORIG_T1w.json
│   │   ├── sub-23017_ses-01_acq-mprage_rec-ORIG_T1w.nii.gz -> ../../../../edb/subj_internal_id/7/40008/subj_internal_id_7_40008_00001.nii.gz
│   │   ├── sub-23017_ses-01_rec-ORIG_T2w.json
│   │   └── sub-23017_ses-01_rec-ORIG_T2w.nii.gz -> ../../../../edb/subj_internal_id/7/40006/subj_internal_id_7_40006_00001.nii.gz
│   ├── fmap
│   │   ├── sub-23017_ses-01_acq-rest_dir-match_epi.json
│   │   ├── sub-23017_ses-01_acq-rest_dir-match_epi.nii.gz -> ../../../../edb/subj_internal_id/7/11/subj_internal_id_7_11_00001.nii.gz
│   │   ├── sub-23017_ses-01_acq-rest_dir-opp_epi.json
│   │   ├── sub-23017_ses-01_acq-rest_dir-opp_epi.nii.gz -> ../../../../edb/subj_internal_id/7/10/subj_internal_id_7_10_00001.nii.gz
│   │   ├── sub-23017_ses-01_acq-task_dir-match_epi.json
│   │   ├── sub-23017_ses-01_acq-task_dir-match_epi.nii.gz -> ../../../../edb/subj_internal_id/7/14/subj_internal_id_7_14_00001.nii.gz
│   │   ├── sub-23017_ses-01_acq-task_dir-opp_epi.json
│   │   └── sub-23017_ses-01_acq-task_dir-opp_epi.nii.gz -> ../../../../edb/subj_internal_id/7/13/subj_internal_id_7_13_00001.nii.gz
│   └── func
│       ├── sub-23017_ses-01_task-mid_bold.json
│       ├── sub-23017_ses-01_task-mid_bold.nii.gz -> ../../../../edb/subj_internal_id/7/12/subj_internal_id_7_12_00001.nii.gz
│       ├── sub-23017_ses-01_task-mid_events.json
│       ├── sub-23017_ses-01_task-mid_events.tsv
│       ├── sub-23017_ses-01_task-rest_echo-1_bold.json
│       ├── sub-23017_ses-01_task-rest_echo-1_bold.nii.gz -> ../../../../edb/subj_internal_id/7/9/subj_internal_id_7_9_00001.nii.gz
│       ├── sub-23017_ses-01_task-rest_echo-2_bold.json
│       ├── sub-23017_ses-01_task-rest_echo-2_bold.nii.gz -> ../../../../edb/subj_internal_id/7/9/subj_internal_id_7_9_00002.nii.gz
│       ├── sub-23017_ses-01_task-rest_echo-3_bold.json
│       ├── sub-23017_ses-01_task-rest_echo-3_bold.nii.gz -> ../../../../edb/subj_internal_id/7/9/subj_internal_id_7_9_00003.nii.gz
│       ├── sub-23017_ses-01_task-rest_echo-4_bold.json
│       ├── sub-23017_ses-01_task-rest_echo-4_bold.nii.gz -> ../../../../edb/subj_internal_id/7/9/subj_internal_id_7_9_00004.nii.gz
│       ├── sub-23017_ses-01_task-rest_physio.json
│       └── sub-23017_ses-01_task-rest_physio.tsv.gz
├── ses-02
│   ├── anat
│   │   ├── sub-23017_ses-02_acq-mprage_rec-ORIG_T1w.json
│   │   └── sub-23017_ses-02_acq-mprage_rec-ORIG_T1w.nii.gz -> ../../../../edb/subj_internal_id/10/40009/subj_internal_id_10_40009_00001.nii.gz
│   ├── fmap
│   │   ├── sub-23017_ses-02_acq-rest_dir-match_epi.json
│   │   ├── sub-23017_ses-02_acq-rest_dir-match_epi.nii.gz -> ../../../../edb/subj_internal_id/10/4/subj_internal_id_10_4_00001.nii.gz
│   │   ├── sub-23017_ses-02_acq-rest_dir-opp_epi.json
│   │   ├── sub-23017_ses-02_acq-rest_dir-opp_epi.nii.gz -> ../../../../edb/subj_internal_id/10/3/subj_internal_id_10_3_00001.nii.gz
│   │   ├── sub-23017_ses-02_acq-task_dir-match_epi.json
│   │   ├── sub-23017_ses-02_acq-task_dir-match_epi.nii.gz -> ../../../../edb/subj_internal_id/10/7/subj_internal_id_10_7_00001.nii.gz
│   │   ├── sub-23017_ses-02_acq-task_dir-opp_epi.json
│   │   └── sub-23017_ses-02_acq-task_dir-opp_epi.nii.gz -> ../../../../edb/subj_internal_id/10/6/subj_internal_id_10_6_00001.nii.gz
│   └── func
│       ├── sub-23017_ses-02_task-mid_bold.json
│       ├── sub-23017_ses-02_task-mid_bold.nii.gz -> ../../../../edb/subj_internal_id/10/8/subj_internal_id_10_8_00001.nii.gz
│       ├── sub-23017_ses-02_task-mid_events.json
│       ├── sub-23017_ses-02_task-mid_events.tsv
│       ├── sub-23017_ses-02_task-rest_echo-1_bold.json
│       ├── sub-23017_ses-02_task-rest_echo-1_bold.nii.gz -> ../../../../edb/subj_internal_id/10/5/subj_internal_id_10_5_00001.nii.gz
│       ├── sub-23017_ses-02_task-rest_echo-2_bold.json
│       ├── sub-23017_ses-02_task-rest_echo-2_bold.nii.gz -> ../../../../edb/subj_internal_id/10/5/subj_internal_id_10_5_00002.nii.gz
│       ├── sub-23017_ses-02_task-rest_echo-3_bold.json
│       ├── sub-23017_ses-02_task-rest_echo-3_bold.nii.gz -> ../../../../edb/subj_internal_id/10/5/subj_internal_id_10_5_00003.nii.gz
│       ├── sub-23017_ses-02_task-rest_echo-4_bold.json
│       ├── sub-23017_ses-02_task-rest_echo-4_bold.nii.gz -> ../../../../edb/subj_internal_id/10/5/subj_internal_id_10_5_00004.nii.gz
│       ├── sub-23017_ses-02_task-rest_physio.json
│       └── sub-23017_ses-02_task-rest_physio.tsv.gz
├── ses-03
│   ├── anat
│   │   ├── sub-23017_ses-03_acq-mprage_rec-ORIG_T1w.json
│   │   ├── sub-23017_ses-03_acq-mprage_rec-ORIG_T1w.nii.gz -> ../../../../edb/subj_internal_id/11/40005/subj_internal_id_11_40005_00001.nii.gz
│   │   ├── sub-23017_ses-03_rec-ORIG_T2w.json
│   │   └── sub-23017_ses-03_rec-ORIG_T2w.nii.gz -> ../../../../edb/subj_internal_id/11/40003/subj_internal_id_11_40003_00001.nii.gz
│   ├── fmap
│   │   ├── sub-23017_ses-03_acq-rest_dir-match_epi.json
│   │   ├── sub-23017_ses-03_acq-rest_dir-match_epi.nii.gz -> ../../../../edb/subj_internal_id/11/8/subj_internal_id_11_8_00001.nii.gz
│   │   ├── sub-23017_ses-03_acq-rest_dir-opp_epi.json
│   │   ├── sub-23017_ses-03_acq-rest_dir-opp_epi.nii.gz -> ../../../../edb/subj_internal_id/11/7/subj_internal_id_11_7_00001.nii.gz
│   │   ├── sub-23017_ses-03_acq-task_dir-match_epi.json
│   │   ├── sub-23017_ses-03_acq-task_dir-match_epi.nii.gz -> ../../../../edb/subj_internal_id/11/12/subj_internal_id_11_12_00001.nii.gz
│   │   ├── sub-23017_ses-03_acq-task_dir-opp_epi.json
│   │   └── sub-23017_ses-03_acq-task_dir-opp_epi.nii.gz -> ../../../../edb/subj_internal_id/11/11/subj_internal_id_11_11_00001.nii.gz
│   └── func
│       ├── sub-23017_ses-03_task-mid_run-2_bold.json
│       ├── sub-23017_ses-03_task-mid_run-2_bold.nii.gz -> ../../../../edb/subj_internal_id/11/10/subj_internal_id_11_10_00001.nii.gz
│       ├── sub-23017_ses-03_task-mid_run-2_events.json
│       ├── sub-23017_ses-03_task-mid_run-2_events.tsv
│       ├── sub-23017_ses-03_task-rest_echo-1_bold.json
│       ├── sub-23017_ses-03_task-rest_echo-1_bold.nii.gz -> ../../../../edb/subj_internal_id/11/6/subj_internal_id_11_6_00001.nii.gz
│       ├── sub-23017_ses-03_task-rest_echo-2_bold.json
│       ├── sub-23017_ses-03_task-rest_echo-2_bold.nii.gz -> ../../../../edb/subj_internal_id/11/6/subj_internal_id_11_6_00002.nii.gz
│       ├── sub-23017_ses-03_task-rest_echo-3_bold.json
│       ├── sub-23017_ses-03_task-rest_echo-3_bold.nii.gz -> ../../../../edb/subj_internal_id/11/6/subj_internal_id_11_6_00003.nii.gz
│       ├── sub-23017_ses-03_task-rest_echo-4_bold.json
│       ├── sub-23017_ses-03_task-rest_echo-4_bold.nii.gz -> ../../../../edb/subj_internal_id/11/6/subj_internal_id_11_6_00004.nii.gz
│       ├── sub-23017_ses-03_task-rest_physio.json
│       └── sub-23017_ses-03_task-rest_physio.tsv.gz
├── sub-23017_sessions.json
└── sub-23017_sessions.tsv

oesteban · August 7, 2019, 4:16pm

I’m not seeing any echos on your outputs (above). Also the run indexes don’t seem consistent between subjects (not sure that could be a problem, just to note).

Are you missing only the echos in the reports?

Shotgunosine · August 7, 2019, 4:21pm

For that particular subject, the only functional run in the report is session 3 task mid run 2. In other reports the multiecho rest does seem to have worked fine.

oesteban · August 7, 2019, 4:26pm

BTW - this other thread could be related Fmriprep html summary report does not include functional figures unless fieldmaps are ignored

How the IntendedFor metadata of the fieldmaps look like?

Shotgunosine · August 7, 2019, 4:46pm

Here’s sub-23017_ses-01_acq-rest_dir-match_epi.json’s indented for field (this coreesponds to a functional run that doesn’t show up in the report):
“IntendedFor”: [
“ses-01/func/sub-23017_ses-01_task-rest_echo-1_bold.nii.gz”,
“ses-01/func/sub-23017_ses-01_task-rest_echo-2_bold.nii.gz”,
“ses-01/func/sub-23017_ses-01_task-rest_echo-3_bold.nii.gz”,
“ses-01/func/sub-23017_ses-01_task-rest_echo-4_bold.nii.gz”
]

sub-23017_ses-03_acq-task_dir-match_epi.json (corresponding to the one that does show up in the report:
“IntendedFor”: [
“ses-03/func/sub-23017_ses-03_task-mid_run-2_bold.nii.gz”
]

Shotgunosine · August 7, 2019, 4:59pm

@effigies suggested I look at the Report object from niworkflows. There’s definitely something screwy with pybids here:

broke_rplt_dir = Path('/data/MBDU/midla/data/derivatives/fmriprep/rerun_aroma_2/sub-23017/wrk/wrk/reportlets')
rpt = reports.core.Report(broke_rplt_dir, '/data/MBDU/midla/notebooks/test', '1', subject_id='sub-23017', packagename='fmriprep')
frpt = [rs for rs in rpt.sections if rs.name == 'Functional'][0]
[frpt_rplt.name for frpt_rplt in frpt.reportlets]
> [frpt_rplt.name for frpt_rplt in frpt.reportlets]

['datatype-func_desc-summary_suffix-bold_session-03_task-mid_run-2',
 'datatype-func_desc-validation_suffix-bold_session-03_task-mid_run-2',
 'datatype-func_desc-sdc_suffix-bold_session-03_task-mid_run-2',
 'datatype-func_desc-bbregister_suffix-bold_session-03_task-mid_run-2',
 'datatype-func_desc-rois_suffix-bold_session-03_task-mid_run-2',
 'datatype-func_desc-compcorvar_suffix-bold_session-03_task-mid_run-2',
 'datatype-func_desc-carpetplot_suffix-bold_session-03_task-mid_run-2',
 'datatype-func_desc-confoundcorr_suffix-bold_session-03_task-mid_run-2',
 'datatype-func_desc-aroma_suffix-bold_session-03_task-mid_run-2']

rpt.layout
> BIDS Layout: .../reportlets/fmriprep/sub-23017 | Subjects: 1 | Sessions: 3 | Runs: 1

rpt.layout.get()
> [<BIDSFile filename='/data/MBDU/midla/data/derivatives/fmriprep/rerun_aroma_2/sub-23017/wrk/wrk/reportlets/fmriprep/sub-23017/anat/sub-23017_desc-about_T1w.html'>,
 <BIDSFile filename='/data/MBDU/midla/data/derivatives/fmriprep/rerun_aroma_2/sub-23017/wrk/wrk/reportlets/fmriprep/sub-23017/anat/sub-23017_desc-conform_T1w.html'>,
 <BIDSFile filename='/data/MBDU/midla/data/derivatives/fmriprep/rerun_aroma_2/sub-23017/wrk/wrk/reportlets/fmriprep/sub-23017/anat/sub-23017_desc-reconall_T1w.svg'>,
 <BIDSFile filename='/data/MBDU/midla/data/derivatives/fmriprep/rerun_aroma_2/sub-23017/wrk/wrk/reportlets/fmriprep/sub-23017/anat/sub-23017_desc-summary_T1w.html'>,
 <BIDSFile filename='/data/MBDU/midla/data/derivatives/fmriprep/rerun_aroma_2/sub-23017/wrk/wrk/reportlets/fmriprep/sub-23017/anat/sub-23017_dseg.svg'>,
 <BIDSFile filename='/data/MBDU/midla/data/derivatives/fmriprep/rerun_aroma_2/sub-23017/wrk/wrk/reportlets/fmriprep/sub-23017/anat/sub-23017_space-MNI152NLin6Asym_T1w.svg'>,
 <BIDSFile filename='/data/MBDU/midla/data/derivatives/fmriprep/rerun_aroma_2/sub-23017/wrk/wrk/reportlets/fmriprep/sub-23017/anat/sub-23017_space-MNI152NLin2009cAsym_T1w.svg'>,
 <BIDSFile filename='/data/MBDU/midla/data/derivatives/fmriprep/rerun_aroma_2/sub-23017/wrk/wrk/reportlets/fmriprep/sub-23017/ses-01/func/sub-23017_ses-01_task-mid_desc-aroma_bold.svg'>,
 <BIDSFile filename='/data/MBDU/midla/data/derivatives/fmriprep/rerun_aroma_2/sub-23017/wrk/wrk/reportlets/fmriprep/sub-23017/ses-01/func/sub-23017_ses-01_task-mid_desc-bbregister_bold.svg'>,
 <BIDSFile filename='/data/MBDU/midla/data/derivatives/fmriprep/rerun_aroma_2/sub-23017/wrk/wrk/reportlets/fmriprep/sub-23017/ses-01/func/sub-23017_ses-01_task-mid_desc-carpetplot_bold.svg'>,
 <BIDSFile filename='/data/MBDU/midla/data/derivatives/fmriprep/rerun_aroma_2/sub-23017/wrk/wrk/reportlets/fmriprep/sub-23017/ses-01/func/sub-23017_ses-01_task-mid_desc-compcorvar_bold.svg'>,
 <BIDSFile filename='/data/MBDU/midla/data/derivatives/fmriprep/rerun_aroma_2/sub-23017/wrk/wrk/reportlets/fmriprep/sub-23017/ses-01/func/sub-23017_ses-01_task-mid_desc-confoundcorr_bold.svg'>,
 <BIDSFile filename='/data/MBDU/midla/data/derivatives/fmriprep/rerun_aroma_2/sub-23017/wrk/wrk/reportlets/fmriprep/sub-23017/ses-01/func/sub-23017_ses-01_task-mid_desc-rois_bold.svg'>,
 <BIDSFile filename='/data/MBDU/midla/data/derivatives/fmriprep/rerun_aroma_2/sub-23017/wrk/wrk/reportlets/fmriprep/sub-23017/ses-01/func/sub-23017_ses-01_task-mid_desc-sdc_bold.svg'>,
 <BIDSFile filename='/data/MBDU/midla/data/derivatives/fmriprep/rerun_aroma_2/sub-23017/wrk/wrk/reportlets/fmriprep/sub-23017/ses-01/func/sub-23017_ses-01_task-mid_desc-summary_bold.html'>,
 <BIDSFile filename='/data/MBDU/midla/data/derivatives/fmriprep/rerun_aroma_2/sub-23017/wrk/wrk/reportlets/fmriprep/sub-23017/ses-01/func/sub-23017_ses-01_task-mid_desc-validation_bold.html'>,
 <BIDSFile filename='/data/MBDU/midla/data/derivatives/fmriprep/rerun_aroma_2/sub-23017/wrk/wrk/reportlets/fmriprep/sub-23017/ses-01/func/sub-23017_ses-01_task-rest_echo-1_desc-aroma_bold.svg'>,
 <BIDSFile filename='/data/MBDU/midla/data/derivatives/fmriprep/rerun_aroma_2/sub-23017/wrk/wrk/reportlets/fmriprep/sub-23017/ses-01/func/sub-23017_ses-01_task-rest_echo-1_desc-bbregister_bold.svg'>,
 <BIDSFile filename='/data/MBDU/midla/data/derivatives/fmriprep/rerun_aroma_2/sub-23017/wrk/wrk/reportlets/fmriprep/sub-23017/ses-01/func/sub-23017_ses-01_task-rest_echo-1_desc-carpetplot_bold.svg'>,
 <BIDSFile filename='/data/MBDU/midla/data/derivatives/fmriprep/rerun_aroma_2/sub-23017/wrk/wrk/reportlets/fmriprep/sub-23017/ses-01/func/sub-23017_ses-01_task-rest_echo-1_desc-compcorvar_bold.svg'>,
 <BIDSFile filename='/data/MBDU/midla/data/derivatives/fmriprep/rerun_aroma_2/sub-23017/wrk/wrk/reportlets/fmriprep/sub-23017/ses-01/func/sub-23017_ses-01_task-rest_echo-1_desc-confoundcorr_bold.svg'>,
 <BIDSFile filename='/data/MBDU/midla/data/derivatives/fmriprep/rerun_aroma_2/sub-23017/wrk/wrk/reportlets/fmriprep/sub-23017/ses-01/func/sub-23017_ses-01_task-rest_echo-1_desc-rois_bold.svg'>,
 <BIDSFile filename='/data/MBDU/midla/data/derivatives/fmriprep/rerun_aroma_2/sub-23017/wrk/wrk/reportlets/fmriprep/sub-23017/ses-01/func/sub-23017_ses-01_task-rest_echo-1_desc-sdc_bold.svg'>,
 <BIDSFile filename='/data/MBDU/midla/data/derivatives/fmriprep/rerun_aroma_2/sub-23017/wrk/wrk/reportlets/fmriprep/sub-23017/ses-01/func/sub-23017_ses-01_task-rest_echo-1_desc-summary_bold.html'>,
 <BIDSFile filename='/data/MBDU/midla/data/derivatives/fmriprep/rerun_aroma_2/sub-23017/wrk/wrk/reportlets/fmriprep/sub-23017/ses-01/func/sub-23017_ses-01_task-rest_echo-1_desc-validation_bold.html'>,
 <BIDSFile filename='/data/MBDU/midla/data/derivatives/fmriprep/rerun_aroma_2/sub-23017/wrk/wrk/reportlets/fmriprep/sub-23017/ses-02/func/sub-23017_ses-02_task-mid_desc-aroma_bold.svg'>,
 <BIDSFile filename='/data/MBDU/midla/data/derivatives/fmriprep/rerun_aroma_2/sub-23017/wrk/wrk/reportlets/fmriprep/sub-23017/ses-02/func/sub-23017_ses-02_task-mid_desc-bbregister_bold.svg'>,
 <BIDSFile filename='/data/MBDU/midla/data/derivatives/fmriprep/rerun_aroma_2/sub-23017/wrk/wrk/reportlets/fmriprep/sub-23017/ses-02/func/sub-23017_ses-02_task-mid_desc-carpetplot_bold.svg'>,
 <BIDSFile filename='/data/MBDU/midla/data/derivatives/fmriprep/rerun_aroma_2/sub-23017/wrk/wrk/reportlets/fmriprep/sub-23017/ses-02/func/sub-23017_ses-02_task-mid_desc-compcorvar_bold.svg'>,
 <BIDSFile filename='/data/MBDU/midla/data/derivatives/fmriprep/rerun_aroma_2/sub-23017/wrk/wrk/reportlets/fmriprep/sub-23017/ses-02/func/sub-23017_ses-02_task-mid_desc-confoundcorr_bold.svg'>,
 <BIDSFile filename='/data/MBDU/midla/data/derivatives/fmriprep/rerun_aroma_2/sub-23017/wrk/wrk/reportlets/fmriprep/sub-23017/ses-02/func/sub-23017_ses-02_task-mid_desc-rois_bold.svg'>,
 <BIDSFile filename='/data/MBDU/midla/data/derivatives/fmriprep/rerun_aroma_2/sub-23017/wrk/wrk/reportlets/fmriprep/sub-23017/ses-02/func/sub-23017_ses-02_task-mid_desc-sdc_bold.svg'>,
 <BIDSFile filename='/data/MBDU/midla/data/derivatives/fmriprep/rerun_aroma_2/sub-23017/wrk/wrk/reportlets/fmriprep/sub-23017/ses-02/func/sub-23017_ses-02_task-mid_desc-summary_bold.html'>,
 <BIDSFile filename='/data/MBDU/midla/data/derivatives/fmriprep/rerun_aroma_2/sub-23017/wrk/wrk/reportlets/fmriprep/sub-23017/ses-02/func/sub-23017_ses-02_task-mid_desc-validation_bold.html'>,
 <BIDSFile filename='/data/MBDU/midla/data/derivatives/fmriprep/rerun_aroma_2/sub-23017/wrk/wrk/reportlets/fmriprep/sub-23017/ses-02/func/sub-23017_ses-02_task-rest_echo-1_desc-aroma_bold.svg'>,
 <BIDSFile filename='/data/MBDU/midla/data/derivatives/fmriprep/rerun_aroma_2/sub-23017/wrk/wrk/reportlets/fmriprep/sub-23017/ses-02/func/sub-23017_ses-02_task-rest_echo-1_desc-bbregister_bold.svg'>,
 <BIDSFile filename='/data/MBDU/midla/data/derivatives/fmriprep/rerun_aroma_2/sub-23017/wrk/wrk/reportlets/fmriprep/sub-23017/ses-02/func/sub-23017_ses-02_task-rest_echo-1_desc-carpetplot_bold.svg'>,
 <BIDSFile filename='/data/MBDU/midla/data/derivatives/fmriprep/rerun_aroma_2/sub-23017/wrk/wrk/reportlets/fmriprep/sub-23017/ses-02/func/sub-23017_ses-02_task-rest_echo-1_desc-compcorvar_bold.svg'>,
 <BIDSFile filename='/data/MBDU/midla/data/derivatives/fmriprep/rerun_aroma_2/sub-23017/wrk/wrk/reportlets/fmriprep/sub-23017/ses-02/func/sub-23017_ses-02_task-rest_echo-1_desc-confoundcorr_bold.svg'>,
 <BIDSFile filename='/data/MBDU/midla/data/derivatives/fmriprep/rerun_aroma_2/sub-23017/wrk/wrk/reportlets/fmriprep/sub-23017/ses-02/func/sub-23017_ses-02_task-rest_echo-1_desc-rois_bold.svg'>,
 <BIDSFile filename='/data/MBDU/midla/data/derivatives/fmriprep/rerun_aroma_2/sub-23017/wrk/wrk/reportlets/fmriprep/sub-23017/ses-02/func/sub-23017_ses-02_task-rest_echo-1_desc-sdc_bold.svg'>,
 <BIDSFile filename='/data/MBDU/midla/data/derivatives/fmriprep/rerun_aroma_2/sub-23017/wrk/wrk/reportlets/fmriprep/sub-23017/ses-02/func/sub-23017_ses-02_task-rest_echo-1_desc-summary_bold.html'>,
 <BIDSFile filename='/data/MBDU/midla/data/derivatives/fmriprep/rerun_aroma_2/sub-23017/wrk/wrk/reportlets/fmriprep/sub-23017/ses-02/func/sub-23017_ses-02_task-rest_echo-1_desc-validation_bold.html'>,
 <BIDSFile filename='/data/MBDU/midla/data/derivatives/fmriprep/rerun_aroma_2/sub-23017/wrk/wrk/reportlets/fmriprep/sub-23017/ses-03/func/._sub-23017_ses-03_task-mid_run-2_desc-summary_bold.html'>,
 <BIDSFile filename='/data/MBDU/midla/data/derivatives/fmriprep/rerun_aroma_2/sub-23017/wrk/wrk/reportlets/fmriprep/sub-23017/ses-03/func/sub-23017_ses-03_task-mid_run-2_desc-aroma_bold.svg'>,
 <BIDSFile filename='/data/MBDU/midla/data/derivatives/fmriprep/rerun_aroma_2/sub-23017/wrk/wrk/reportlets/fmriprep/sub-23017/ses-03/func/sub-23017_ses-03_task-mid_run-2_desc-bbregister_bold.svg'>,
 <BIDSFile filename='/data/MBDU/midla/data/derivatives/fmriprep/rerun_aroma_2/sub-23017/wrk/wrk/reportlets/fmriprep/sub-23017/ses-03/func/sub-23017_ses-03_task-mid_run-2_desc-carpetplot_bold.svg'>,
 <BIDSFile filename='/data/MBDU/midla/data/derivatives/fmriprep/rerun_aroma_2/sub-23017/wrk/wrk/reportlets/fmriprep/sub-23017/ses-03/func/sub-23017_ses-03_task-mid_run-2_desc-compcorvar_bold.svg'>,
 <BIDSFile filename='/data/MBDU/midla/data/derivatives/fmriprep/rerun_aroma_2/sub-23017/wrk/wrk/reportlets/fmriprep/sub-23017/ses-03/func/sub-23017_ses-03_task-mid_run-2_desc-confoundcorr_bold.svg'>,
 <BIDSFile filename='/data/MBDU/midla/data/derivatives/fmriprep/rerun_aroma_2/sub-23017/wrk/wrk/reportlets/fmriprep/sub-23017/ses-03/func/sub-23017_ses-03_task-mid_run-2_desc-rois_bold.svg'>,
 <BIDSFile filename='/data/MBDU/midla/data/derivatives/fmriprep/rerun_aroma_2/sub-23017/wrk/wrk/reportlets/fmriprep/sub-23017/ses-03/func/sub-23017_ses-03_task-mid_run-2_desc-sdc_bold.svg'>,
 <BIDSFile filename='/data/MBDU/midla/data/derivatives/fmriprep/rerun_aroma_2/sub-23017/wrk/wrk/reportlets/fmriprep/sub-23017/ses-03/func/sub-23017_ses-03_task-mid_run-2_desc-summary_bold.html'>,
 <BIDSFile filename='/data/MBDU/midla/data/derivatives/fmriprep/rerun_aroma_2/sub-23017/wrk/wrk/reportlets/fmriprep/sub-23017/ses-03/func/sub-23017_ses-03_task-mid_run-2_desc-validation_bold.html'>,
 <BIDSFile filename='/data/MBDU/midla/data/derivatives/fmriprep/rerun_aroma_2/sub-23017/wrk/wrk/reportlets/fmriprep/sub-23017/ses-03/func/sub-23017_ses-03_task-rest_echo-1_desc-aroma_bold.svg'>,
 <BIDSFile filename='/data/MBDU/midla/data/derivatives/fmriprep/rerun_aroma_2/sub-23017/wrk/wrk/reportlets/fmriprep/sub-23017/ses-03/func/sub-23017_ses-03_task-rest_echo-1_desc-bbregister_bold.svg'>,
 <BIDSFile filename='/data/MBDU/midla/data/derivatives/fmriprep/rerun_aroma_2/sub-23017/wrk/wrk/reportlets/fmriprep/sub-23017/ses-03/func/sub-23017_ses-03_task-rest_echo-1_desc-carpetplot_bold.svg'>,
 <BIDSFile filename='/data/MBDU/midla/data/derivatives/fmriprep/rerun_aroma_2/sub-23017/wrk/wrk/reportlets/fmriprep/sub-23017/ses-03/func/sub-23017_ses-03_task-rest_echo-1_desc-compcorvar_bold.svg'>,
 <BIDSFile filename='/data/MBDU/midla/data/derivatives/fmriprep/rerun_aroma_2/sub-23017/wrk/wrk/reportlets/fmriprep/sub-23017/ses-03/func/sub-23017_ses-03_task-rest_echo-1_desc-confoundcorr_bold.svg'>,
 <BIDSFile filename='/data/MBDU/midla/data/derivatives/fmriprep/rerun_aroma_2/sub-23017/wrk/wrk/reportlets/fmriprep/sub-23017/ses-03/func/sub-23017_ses-03_task-rest_echo-1_desc-rois_bold.svg'>,
 <BIDSFile filename='/data/MBDU/midla/data/derivatives/fmriprep/rerun_aroma_2/sub-23017/wrk/wrk/reportlets/fmriprep/sub-23017/ses-03/func/sub-23017_ses-03_task-rest_echo-1_desc-sdc_bold.svg'>,
 <BIDSFile filename='/data/MBDU/midla/data/derivatives/fmriprep/rerun_aroma_2/sub-23017/wrk/wrk/reportlets/fmriprep/sub-23017/ses-03/func/sub-23017_ses-03_task-rest_echo-1_desc-summary_bold.html'>,
 <BIDSFile filename='/data/MBDU/midla/data/derivatives/fmriprep/rerun_aroma_2/sub-23017/wrk/wrk/reportlets/fmriprep/sub-23017/ses-03/func/sub-23017_ses-03_task-rest_echo-1_desc-validation_bold.html'>]

!tree {rpt.root}

> /data/MBDU/midla/data/derivatives/fmriprep/rerun_aroma_2/sub-23017/wrk/wrk/reportlets/fmriprep/sub-23017
├── anat
│   ├── sub-23017_desc-about_T1w.html
│   ├── sub-23017_desc-conform_T1w.html
│   ├── sub-23017_desc-reconall_T1w.svg
│   ├── sub-23017_desc-summary_T1w.html
│   ├── sub-23017_dseg.svg
│   ├── sub-23017_space-MNI152NLin2009cAsym_T1w.svg
│   └── sub-23017_space-MNI152NLin6Asym_T1w.svg
├── ses-01
│   └── func
│       ├── sub-23017_ses-01_task-mid_desc-aroma_bold.svg
│       ├── sub-23017_ses-01_task-mid_desc-bbregister_bold.svg
│       ├── sub-23017_ses-01_task-mid_desc-carpetplot_bold.svg
│       ├── sub-23017_ses-01_task-mid_desc-compcorvar_bold.svg
│       ├── sub-23017_ses-01_task-mid_desc-confoundcorr_bold.svg
│       ├── sub-23017_ses-01_task-mid_desc-rois_bold.svg
│       ├── sub-23017_ses-01_task-mid_desc-sdc_bold.svg
│       ├── sub-23017_ses-01_task-mid_desc-summary_bold.html
│       ├── sub-23017_ses-01_task-mid_desc-validation_bold.html
│       ├── sub-23017_ses-01_task-rest_echo-1_desc-aroma_bold.svg
│       ├── sub-23017_ses-01_task-rest_echo-1_desc-bbregister_bold.svg
│       ├── sub-23017_ses-01_task-rest_echo-1_desc-carpetplot_bold.svg
│       ├── sub-23017_ses-01_task-rest_echo-1_desc-compcorvar_bold.svg
│       ├── sub-23017_ses-01_task-rest_echo-1_desc-confoundcorr_bold.svg
│       ├── sub-23017_ses-01_task-rest_echo-1_desc-rois_bold.svg
│       ├── sub-23017_ses-01_task-rest_echo-1_desc-sdc_bold.svg
│       ├── sub-23017_ses-01_task-rest_echo-1_desc-summary_bold.html
│       └── sub-23017_ses-01_task-rest_echo-1_desc-validation_bold.html
├── ses-02
│   └── func
│       ├── sub-23017_ses-02_task-mid_desc-aroma_bold.svg
│       ├── sub-23017_ses-02_task-mid_desc-bbregister_bold.svg
│       ├── sub-23017_ses-02_task-mid_desc-carpetplot_bold.svg
│       ├── sub-23017_ses-02_task-mid_desc-compcorvar_bold.svg
│       ├── sub-23017_ses-02_task-mid_desc-confoundcorr_bold.svg
│       ├── sub-23017_ses-02_task-mid_desc-rois_bold.svg
│       ├── sub-23017_ses-02_task-mid_desc-sdc_bold.svg
│       ├── sub-23017_ses-02_task-mid_desc-summary_bold.html
│       ├── sub-23017_ses-02_task-mid_desc-validation_bold.html
│       ├── sub-23017_ses-02_task-rest_echo-1_desc-aroma_bold.svg
│       ├── sub-23017_ses-02_task-rest_echo-1_desc-bbregister_bold.svg
│       ├── sub-23017_ses-02_task-rest_echo-1_desc-carpetplot_bold.svg
│       ├── sub-23017_ses-02_task-rest_echo-1_desc-compcorvar_bold.svg
│       ├── sub-23017_ses-02_task-rest_echo-1_desc-confoundcorr_bold.svg
│       ├── sub-23017_ses-02_task-rest_echo-1_desc-rois_bold.svg
│       ├── sub-23017_ses-02_task-rest_echo-1_desc-sdc_bold.svg
│       ├── sub-23017_ses-02_task-rest_echo-1_desc-summary_bold.html
│       └── sub-23017_ses-02_task-rest_echo-1_desc-validation_bold.html
└── ses-03
    └── func
        ├── sub-23017_ses-03_task-mid_run-2_desc-aroma_bold.svg
        ├── sub-23017_ses-03_task-mid_run-2_desc-bbregister_bold.svg
        ├── sub-23017_ses-03_task-mid_run-2_desc-carpetplot_bold.svg
        ├── sub-23017_ses-03_task-mid_run-2_desc-compcorvar_bold.svg
        ├── sub-23017_ses-03_task-mid_run-2_desc-confoundcorr_bold.svg
        ├── sub-23017_ses-03_task-mid_run-2_desc-rois_bold.svg
        ├── sub-23017_ses-03_task-mid_run-2_desc-sdc_bold.svg
        ├── sub-23017_ses-03_task-mid_run-2_desc-summary_bold.html
        ├── sub-23017_ses-03_task-mid_run-2_desc-validation_bold.html
        ├── sub-23017_ses-03_task-rest_echo-1_desc-aroma_bold.svg
        ├── sub-23017_ses-03_task-rest_echo-1_desc-bbregister_bold.svg
        ├── sub-23017_ses-03_task-rest_echo-1_desc-carpetplot_bold.svg
        ├── sub-23017_ses-03_task-rest_echo-1_desc-compcorvar_bold.svg
        ├── sub-23017_ses-03_task-rest_echo-1_desc-confoundcorr_bold.svg
        ├── sub-23017_ses-03_task-rest_echo-1_desc-rois_bold.svg
        ├── sub-23017_ses-03_task-rest_echo-1_desc-sdc_bold.svg
        ├── sub-23017_ses-03_task-rest_echo-1_desc-summary_bold.html
        └── sub-23017_ses-03_task-rest_echo-1_desc-validation_bold.html

7 directories, 61 files

Shotgunosine · August 7, 2019, 5:43pm

@tal Do you have any ideas why pybids isn’t correctly parsing out the number of runs here? Is there anything I can do to keep debugging this, or should I just zip up the directory and send it to you?

tal · August 7, 2019, 6:03pm

Can you describe the screwyness in more detail? Are files missing, or something else? And if the former, can you give examples of filenames that aren’t being returned from .get()? (May be better to open an issue on the PyBIDS repo, and also see this issue, which may be related).

Shotgunosine · August 7, 2019, 6:04pm

.get returns all the files, but .layout says there is only 1 run, and the reports generated only have reportlets from that run

oesteban · August 7, 2019, 6:34pm

To give some context, we are using PyBIDS here to query a “reportlets” root folder, which should be compliant with this PyBIDS confing - https://github.com/poldracklab/niworkflows/blob/master/niworkflows/reports/figures.json

So it is very likely there are errors on the config file.

@Shotgunosine, can you post a tree of $TMPDIR/wrk/reportlets so that @tal can see the root being indexed?