HI, I’m working on getting the tractoflow pipeline running on the UK Biobank dataset. Nextflow docs say that any option you can use with singularity engine or runtime can be called from the nextflow.config with the scope settings ‘engineOptions’ and ‘runOptions’. I can’t find an example of the syntax the options expect. The UKB dataset is stored in a series of squashfs images that can be overlay-ed on a singularity container, which works fine when I run singularity shell with the tractoflow container, but stuffing the same “-overlay” options into the nextflow.config file doesn’t seem to work, it might be my syntax is wrong, but also to make matters even more confusing I am pointing the BIDS option in the pipeline at an overlayed squashfs-ed symlink tree that points at the actual BIDS directory This is to be able to control which subjects are run, since there are a lot. I’m pretty sure I may be the first poor soul to attempt such a convolution, but I sincerely hope someone here has attempted something at least a little similar and might be able to point me in a direction away from my flailings. Cheers
Do you have a link to the configuration you are using? As best I can tell you’ll want to use runOptions when trying to set overlay for singularity. From the nextflow tests it looks like engineOptions are flags that are supposed to be inserted after singularity but before exec.
Hi, Thanks for responding. The --bind /home,/scratch… mounts work, At the risk of polluting the topic, here’s the profile I’m using(sorry for the long line): fully_reproducible_ukb { params.processes_brain_extraction_t1=1 params.processes_denoise_dwi=4 params.processes_denoise_t1=4 params.processes_eddy=1 params.processes_fodf=4 params.processes_registration=1 singularity.runOptions="-B /home,/project,/scratch,/localscratch --overlay /lustre03/project/6008063/neurohub/ukbb/imaging/neurohub_ukbb_dwi_ses2_0_bids.squashfs /lustre03/project/6008063/neurohub/ukbb/imaging/neurohub_ukbb_dwi_ses2_1_bids.squashfs /lustre03/project/6008063/neurohub/ukbb/imaging/neurohub_ukbb_dwi_ses2_2_bids.squashfs /lustre03/project/6008063/neurohub/ukbb/imaging/neurohub_ukbb_dwi_ses2_3_bids.squashfs /lustre03/project/6008063/neurohub/ukbb/imaging/neurohub_ukbb_dwi_ses2_4_bids.squashfs /lustre03/project/6008063/neurohub/ukbb/imaging/neurohub_ukbb_dwi_ses2_5_bids.squashfs /lustre03/project/6008063/neurohub/ukbb/imaging/neurohub_ukbb_dwi_ses2_6_bids.squashfs /lustre03/project/6008063/neurohub/ukbb/imaging/neurohub_ukbb_dwi_ses2_7_bids.squashfs /lustre03/project/6008063/neurohub/ukbb/imaging/neurohub_ukbb_t1_ses2_0_bids.squashfs /lustre03/project/6008063/neurohub/ukbb/imaging/neurohub_ukbb_t1_ses2_0_jsonpatch.squashfs /lustre04/scratch/atrefo/sherbrooke/squash_tractoflow/subs_66.squashfs" }
I just spotted an obvious error, [D’OH!] /lustre03 and lustre04 aren't bind mounted, subdirs /projectand/scratch` are… could it be that simple? I’m changing that and testing.
Sadly, that did not fix it. It is not mounting the root directory of the squashfs images.
Dang. Config seems reasonable to me. Hopefully someone with more direct nextflow experience can chime in.
Saw something interesting related to Lustre and Singularity not positive its related/relevant to your issue:
Thanks! It’s currently stumping us. And thanks for finding that bugreport, but I can use the overlays when I rung the container outside of nextflow. I’ll be sure to post here if/when we find a solution.