Hi @Rachel_S,
I mean using a different argument for -w or clearing out /scratch for each run.
I believe it has to be -d anat|func|fmap DIRECTORY, depending on what modality you want to use. What derivative do you want to use from mriqc? I am not sure I would not recommend using MRIQC derivatives, as the preprocessing for that pipeline are not as rigorous as fMRIPrep…
Best,
Steven
Hi Steve:
Thanks for your answers but I think we found another possible culprit. We noticed that the summary for each output gives an error reporting the pediatric template and the adult template while our code does not call for the pediatric template.
fmriprep-docker /Volumes/ITALY/raw/ /Volumes/ITALY/derivatives/ participant --skip_bids_validation --participant_label 006 --dummy-scans 0 --fs-license-file /Volumes/ITALY/license.txt --output-spaces MNI152NLin2009cAsym -w /Volumes/ITALY/tmp/
1- could this be the cause of our errors?
2- How can we remove it from the pipeline?
Thank you,
Ilaria
Did you clear the working directory? Also, fmriprep updates the html report, so if you didn’t delete the old html, you should expect the pediatric space from your old run to still be present.
Yes, we did clear the working directory. Tried again but the same error popped up again about 30 minutes into the run. We are running it all the way through to see if the final output looks complete (takes about 5 hours) but wonder why we can’t run it again without issues.
Is it possibly related to running it with 0 dummy scans? That is the one thing we changed from the original code - we had 6 dummy scans originally but then after getting warnings about 0-1 non-steady-state volumes automatically recognized we reviewed the information on Open Neuro and realized that they had already been removed from the data
TIA
Probably not.
But otherwise, I am stumped at the moment. Are final outputs created in the requested output space, and if so do they look okay?
Hi Steven, apologies for the delayed response.
We wiped our system and reinstalled everything. Again, the first time we ran a participant, it went through with no errors. We then ran a set of 3 participants over night and got similar errors, though each participant had a different number of crash reports, and for seemingly random runs (two of the three had crashes with the anatomy scan, all of them had at least one crash from various functional scans). I will include the final lines of the error message from one participant below identifying which file could not be found.
'/scratch/fmriprep_23_2_wf/sub_011_wf/anat_fit_wf/ds_template_wf/ds_t1w_ref_xfms/mapflow/_ds_t1w_ref_xfms1/result__ds_t1w_ref_xfms1.pklz'
'/scratch/fmriprep_23_2_wf/sub_011_wf/bold_ses_T1_task_Num_run_01_wf/bold_confounds_wf/acc_msk_brain/mapflow/_acc_msk_brain2/result__acc_msk_brain2.pklz'
'/scratch/fmriprep_23_2_wf/sub_011_wf/bold_ses_T2_task_Sub_run_01_wf/bold_confounds_wf/acc_msk_brain/mapflow/_acc_msk_brain2/result__acc_msk_brain2.pklz'
'/scratch/fmriprep_23_2_wf/sub_011_wf/bold_ses_T1_task_Rhyming_wf/bold_confounds_wf/acc_msk_brain/mapflow/_acc_msk_brain2/result__acc_msk_brain2.pklz'
'/scratch/fmriprep_23_2_wf/sub_011_wf/bold_ses_T1_task_Sub_run_02_wf/bold_confounds_wf/acc_msk_bin/mapflow/_acc_msk_bin1/result__acc_msk_bin1.pklz'
What is even more confusing is that I ran this same participant today (still running, 6.5 hours in, the first one that finished only took 5 hours) and got a similar crash report pop up but for a different functional run than any of the four runs that created crash reports the first time around:
'No such file or directory: '/scratch/fmriprep_23_2_wf/sub_011_wf/bold_ses_T1_task_Num_run_02_wf/bold_confounds_wf/acc_msk_bin/mapflow/_acc_msk_bin2/result__acc_msk_bin2.pklz'
We skimmed over the output files and they look normal/similar to the files from our runs with no errors, but none of us have a ton of experience with the fMRIprep outputs so if something was off, it would have to be pretty significant for us to catch it.
We set docker permissions for 28 GB (Mac Mini with 32 max) and it is the only thing running on the computer. I don’t know off hand exact CPUs but can look at it when I’m back in lab tomorrow.
28GB might not be enough for multiple subjects concurrently. Are you running in serial or parallel?
Serial (I assume, we just put the participant numbers in order in the participant_label argument (006 011 016), but we get the error even when just running one participant.
The one from yesterday finished after 7 hours and the error above was the only error that popped up.
CPU load was regularly above 90% when it was running and CPU% for fMRIprep was really high, I want to say 800-900%, but it might have gone higher than that at points.
If you put all participant labels in one fmriprep command it will run in parallel (to the best of its ability, based on the number of cpus).
8 runs could also be a lot of data (I’ve seen it push above 32 gb for single subject). As a test, can you try using a --bids-filter-file to just run a single run for a single subject, to see if it is memory related or something else?
Best,
Steven
Hi Steven,
Before seeing your comment we ran the same participant using the low-mem tag and without free surfer, but ran into the same anatomy error as above.
We then ran it with just a single run for the same subject using the --bids-filter-file tag you suggested (and --low-mem and without free surfer) and it ran without errors.
For the project I am working on, we only need 4 runs from each of the two time points, so we modified the filter file for just those 8 (json text below), but again got the same file not found anatomy type scan. One possible difference is that when I ran it for just the one run, I also only used the T1 anatomy scan versus when I ran it will all 8, we included the anatomy scans at both time points.
{
"t1w": {
"datatype": "anat",
"session": "*",
"acquisition": null,
"suffix": "T1w"
},
"bold": {
"datatype": "func",
"session": "*",
"task": ["Num", "Sub"],
"suffix": "bold"
}
}
terminal command:
fmriprep-docker /Volumes/ITALY/raw/ \
/Volumes/ITALY/derivatives6/ \
participant --skip_bids_validation \
--participant_label 011 \
--dummy-scans 0 \
--fs-license-file /Volumes/ITALY/license.txt \
--fs-no-reconall \
--low-mem \
--output-spaces MNI152NLin2009cAsym \
-w /Volumes/ITALY/tmp6/ \
--bids-filter-file /Volumes/ITALY/num_sub_only.json
We compared the final .html files for the single scan that ran without errors and the equivalent scan in one that ran with errors. It seems like the only differences are minor changes in the noise estimates (ex: 26 components explaining 50% of the variance versus 27 components). Do you think that these errors mean we should not use any of the outputs or should they be considered more of warnings that noise estimates might be slightly off?
We are going to run another subject with the filter for 8 runs to verify that it is not something specifically wrong with 011, but after that we are out of ideas.
I wouldn’t read too much into noise component differences. The ICA in compcor isn’t deterministic.
You can feel free to DM me a subject worth of raw data and I can see if I replicate error on my HPC. Do you have access to a more powerful computing cluster? You might also consider brainlife.io
The data is all on OpenNeuro if that would be easier for you to access - if not I will send you a subject.
And unfortunately we don’t have access to any computing clusters. I did briefly try brainlife.io at some point, I can’t remember what issue popped up, could look into it again if you think it would be worth it.
Thanks, I’ll give it a shot
Updating: the second participant (016) also produced errors, both the anat related one and one related to a functional scan (when we ran it in a batch of 3 participants a couple of days ago this participant only produced the functional scan error - though it was for the same task both times - see below - including the whole error just in case there is something I’m not noticing in the middle)
Node: fmriprep_23_2_wf.sub_016_wf.bold_ses_T2_task_Num_run_01_wf.bold_confounds_wf.acc_msk_brain
Working directory: /scratch/fmriprep_23_2_wf/sub_016_wf/bold_ses_T2_task_Num_run_01_wf/bold_confounds_wf/acc_msk_brain
Node inputs:
in_file = ['/scratch/fmriprep_23_2_wf/sub_016_wf/bold_ses_T2_task_Num_run_01_wf/bold_confounds_wf/acc_msk_tfm/mapflow/_acc_msk_tfm0/acompcor_csf_trans.nii.gz', '/scratch/fmriprep_23_2_wf/sub_016_wf/bold_ses_T2_task_Num_run_01_wf/bold_confounds_wf/acc_msk_tfm/mapflow/_acc_msk_tfm1/acompcor_wm_trans.nii.gz', '/scratch/fmriprep_23_2_wf/sub_016_wf/bold_ses_T2_task_Num_run_01_wf/bold_confounds_wf/acc_msk_tfm/mapflow/_acc_msk_tfm2/acompcor_wmcsf_trans.nii.gz']
in_mask = <undefined>
threshold = 0.5
Traceback (most recent call last):
File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/utils.py", line 94, in nodelist_runner
result = node.run(updatehash=updatehash)
File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 543, in run
write_node_report(self, result=result, is_mapnode=isinstance(self, MapNode))
File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/utils.py", line 208, in write_node_report
report_file.write_text("\n".join(lines), encoding='utf-8')
File "/opt/conda/envs/fmriprep/lib/python3.10/pathlib.py", line 1154, in write_text
with self.open(mode='w', encoding=encoding, errors=errors, newline=newline) as f:
File "/opt/conda/envs/fmriprep/lib/python3.10/pathlib.py", line 1119, in open
return self._accessor.open(self, mode, buffering, encoding, errors,
FileNotFoundError: [Errno 2] No such file or directory: '/scratch/fmriprep_23_2_wf/sub_016_wf/bold_ses_T2_task_Num_run_01_wf/bold_confounds_wf/acc_msk_brain/mapflow/_acc_msk_brain2/_report/report.rst'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
result["result"] = node.run(updatehash=updatehash)
File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
result = self._run_interface(execute=True)
File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 1380, in _run_interface
result = self._collate_results(
File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 1249, in _collate_results
for i, nresult, err in nodes:
File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/utils.py", line 99, in nodelist_runner
result = node.result
File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 223, in result
return _load_resultfile(
File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/utils.py", line 293, in load_resultfile
result = loadpkl(results_file)
File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/utils/filemanip.py", line 621, in loadpkl
with pklopen(str(infile), "rb") as pkl_file:
File "/opt/conda/envs/fmriprep/lib/python3.10/gzip.py", line 58, in open
binary_file = GzipFile(filename, gz_mode, compresslevel)
File "/opt/conda/envs/fmriprep/lib/python3.10/gzip.py", line 174, in __init__
fileobj = self.myfileobj = builtins.open(filename, mode or 'rb')
FileNotFoundError: [Errno 2] No such file or directory: '/scratch/fmriprep_23_2_wf/sub_016_wf/bold_ses_T2_task_Num_run_01_wf/bold_confounds_wf/acc_msk_brain/mapflow/_acc_msk_brain2/result__acc_msk_brain2.pklz'
When I run the same participant with only the task with the error (T2, Num, run 01), no errors are reported 
Hi @Rachel_S,
I ran subject 011 yesterday with no errors on my compute cluster. When using 32 cores (16 max per task), the job maxed out at 26.17GB used (out of 64 available total) and took 3 hours and 16 minutes. I used all default flags (related to fmriprep steps, at least) except for adding --longitudinal. I doubt that flag is relevant to the errors you get, which, at least to me, sounds like they’re due to not having enough resources.
Best,
Steven
Thank you so much for taking the time to run a participant and confirm for us that it was not a problem with the data itself!
We tested a few more things in lab and it seems like it might have actually been a problem with our external hard drive, it is large enough to hold all of our data (8T) but it is quite old and when we opened it on a different computer, we saw that the permissions for reading and writing were a bit odd (“custom permissions” but no ability to modify them). So we tested having fMRIprep read the data from the hard drive and then write the working directories to a separate drive. This seems to have solved the problem - we have successfully run 3 participants (without free surfer, under 1.5 hours per participant) with no errors thus far using this approach, so it seems like it might be the solution. We are running a batch of participants overnight and if any errors come up we will share them, but fingers crossed we are good to go!
Hi @Rachel_S,
You should test if it works with FreeSurfer, as it will improve the quality of your outputs.
Best,
Steven
Hi @Steven,
Confirmed it also ran successfully with FreeSurfer in about 5.5 hours using the new approach of writing to a different hard drive.
Thank you again for all of your help!
MANAGED BY INCF