I am hoping to implement a workflow using fmri- and qsiprep singularity containers on my institution’s HPC clusters. Currently the clusters use Sun Grid Engine to manage submitted jobs.
Does SGE tend to work well with singularity containers? Would I need to explore an option such as Kubernetes instead for controlling singularity containers?
Singularity and the container-based software you are looking at should work fine with SGE systems.
Thank you.
If I submit a script that calls either a qsi- or fmriprep singularity container to the cluster via qsub, will SGE properly manage the container’s resources?
Finally, qsi- and fmriprep let you specify the max amount of memory and number of OMP threads available to the program. Since SGE will be managing the container resources, should I set these options to whatever amount I set when submitting using qsub (i.e. ‘mem_free’ and ‘h_vmem’)?
SGE should manage resources properly. In regards to the second question, I tend to set the “mem_mb” argument of qsi/fmriprep to be half a GB below the memory limits of the job, but you don’t need to do that.