Realized that the contents of the work folder that fmriprep produces, is taking up large amounts of storage- Can we delete the contents for subjects that have run successfully? Like the fmriprep_wf and reportlets subfolders under work.
Thanks!
Yes. These are intermediate results and can be safely deleted at any point. The only cost is if you need to rerun, you can’t reuse those results.
I was wondering if there is a built-in flag to do this? I tried including “–clean-workdir”, but that seems clean up the directory before running fmriprep, instead of after (unless I’m misusing it)?
No, there isn’t. Adding such a feature seems like an opportunity for logic failures leading to unintended removals; if you’re confident you won’t need it, you can add && rm -rf $WORKDIR to the end of your command.
1 Like
Apologies if this has been answered elsewhere, but can I check that --clean-workdir doesn’t clear the entirety of $WORKDIR, just the temporary storage used by the current process? We’re running fmriprep in a parallel environment, so clearing/removing $WORKDIR would be bad if e.g. it just deletes “work”.
Best, Jon
Hello @effigies
Can I check the logic here - what is the concern about deleting a temporary data folder after processing has completed? Currently the --clean-workdir option seems redundant as what’s the point of clearing a folder that has no data in it? Apologies if I’m completely missing the point!
I was looking at other user’s scripts for submitting batch jobs via slurm, and they explicitly specify a WORKDIR variable to be equal to something like:
/some/path/to/work/$subjectID
such that each one could be deleted safely. Is this the preferred method and then use the && rm -rf $WORKDIR as you describe?
Yours bemusedly,
Jon
Hi @Jon_Brooks,
If you want to rerun anything (e.g. after changing a single fmriprep parameter), or restart a process that might have crashed along the way, keeping the workdir is valuable.
There is a lot of content in the work dir that does is not output in the final folder that can be useful for debugging or learning the internal mechanisms of fmriprep.
It should clear workdir of all contents. You can see here (fmriprep/fmriprep/cli/parser.py at 391a0c2b082bb3377878864d62ab9209b3eddf85 · nipreps/fmriprep · GitHub) that fmriprep is calling this function (niworkflows/niworkflows/utils/misc.py at a9f00d56d50272fb2414dcebcd2f8e871b0ac046 · nipreps/niworkflows · GitHub).
This would depend on specifics about the file system/computing system and how the jobs are being submitted (e.g., one big job vs a job array or something else).
Best,
Steven