Summary of what happened:
Hello I have a nipype workflow that uses MATLAB MCR and SPM12 standalone. I have access to a HPC that uses SLURM and have a batch script that kicks off my nipype workflow inside of a conda environment I created for running this workflow and processing neuroimaging data. I noticed that sometimes hours would go by before the python script containing my nipype workflow would seem to start. After doing some tests I ran the small test on my command line
import nipype.interfaces.spm as spm
matlab_cmd = 'my/path/to/spm12/standalone/run_spm12.sh my/path/to/MCR/v910/ script'
spm.SPMCommand.set_mlab_paths(matlab_cmd=matlab_cmd, use_mcr=True)
and I ended up killing it after 45 minutes or so. Since I am using a HPC I know it has a matlab module so I made sure it wasn’t loaded and had the same issues. I also tried giving the job more memory and that didn’t have any effect either. I would imagine maybe there are some environment variables could be in conflict with the nipype workflow and the bash batch script used to kick it off? I am not sure and am not very knowledgeable about those details. My python script containing the workflow is around 400 lines long so I’ll just include everything up until the line that I think is hanging that I posted above along with my batch script.
Any help with this would be greatly appreciated.
Command used (and if a helper script was used, a link to the helper script or the command generated):
#!/bin/bash
#SBATCH --job-name=SPM_L1_L2 # Name job to make it easier to see in the queue
#SBATCH --mail-user=myemail@something.edu # Email to send notifications about the job
#SBATCH --mail-type=FAIL,END # What emails to send. Here only email when job finishes or fails.
#SBATCH --nodes=1 # Number of nodes to use
#SBATCH --ntasks=1 # Number of tasks
#SBATCH --cpus-per-task=8 # Number of CPU cores to use
#SBATCH --mem=64gb # Total memory limit for the job.
#SBATCH --time=72:00:00 # Job run time in hours
#SBATCH --output=../logs/nipype_wf/spm_l1_l2_wf_%j.log # Standard output and error log
#SBATCH --account=hpc_group_account_name
#SBATCH --qos=hpc_group_account_qos
date
ml graphviz
ml fsl
ml conda
conda activate path/to/conda/envs/my_env_name
# First positional input sets the analysis level (i.e., level 1 or 2)
analysis_level=$1
if [ $analysis_level -eq 1 ]; then
echo "Running level 1 analysis"
python3 spm_l1_wf.py
elif [ $analysis_level -eq 2 ]; then
echo "Running level 2 analysis"
python3 spm_l2_wf.py
else
echo "Please input a 1 or 2 to indicate which level analysis you would like to run"
fi
date
import nipype.interfaces.spm as spm # spm
from nipype.pipeline import Node, Workflow, MapNode
from nipype.interfaces.utility import Function
from nipype import IdentityInterface
import os # system function
from nipype import SelectFiles
import nipype.interfaces.fsl as fsl
from nipype import config
import yaml
from yaml.loader import SafeLoader
from pathlib import Path
from datetime import datetime
# Create timestamp
start_time = datetime.now()
# Import configuration settings
config_path = Path("path").joinpath("to", "spm_l1_and_l2_wf.yml")
with open(config_path) as config_file:
my_config = yaml.load(config_file, Loader=SafeLoader)
# Based on documentation, config and logging need to be imported and
# set before importing workflow
# Set up base and results directory, i.e., where my workflow will look for files
# and where it will write output files.
l2_base_dir = Path(my_config['output_data_dir']).joinpath(
my_config["analysis_name"], "l1_analyses")
results_dir = Path(my_config['output_data_dir']).joinpath(
my_config['analysis_name'])
# Create results directory if it doesn't already exist
results_dir.mkdir(parents=True, exist_ok=True)
# Create crash and log file directories
crashdump_dir = results_dir.joinpath('crash')
crashdump_dir.mkdir(parents=True, exist_ok=True)
log_dir = results_dir.joinpath('logs')
log_dir.mkdir(parents=True, exist_ok=True)
# Set log and crash directories globally
config_dict = {'execution': {'crashdump_dir': str(crashdump_dir),
'crashfile_format': 'txt'},
'logging': {'log_directory': str(log_dir),
'log_to_file': True}}
config.update_config(config_dict)
# Set matlab command which really is a standalone, compiled version of SPM12. Use mcr. Both are version 2021a.
# Documentation: https://nipype.readthedocs.io/en/latest/api/generated/nipype.interfaces.spm.base.html#module-nipype.interfaces.spm.base
matlab_cmd = 'path/to/my/spm12_standalone/run_spm12.sh path/to/my/MCR/v910/ script'
spm.SPMCommand.set_mlab_paths(matlab_cmd=matlab_cmd, use_mcr=True)
print(f"Running SPM version: {spm.SPMCommand().version}")
Version:
- conda: 24.3.0
- graphviz: 2.46.1 (This module was needed to generated workflow graph diagrams)
- Python: 3.10.14
- Nipype: 1.8.6
- SPM: 12.7771
- FSL: 6.0.6 (I am using the
fslmodule on my HPC) - MCR: 9.10.0.1739362 R2021a Update 5 Aug 09 2021
Environment (Docker, Singularity / Apptainer, custom installation):
see above. Also I am not sure what info you would need about my environment but I ran the command uname -a which returned
Linux c0709a-s30.ufhpc 4.18.0-477.27.1.el8_8.x86_64 #1 SMP Thu Aug 31 10:29:22 EDT 2023 x86_64 x86_64 x86_64 GNU/Linux
lsb_release -a returned
LSB Version: :core-4.1-amd64:core-4.1-noarch:cxx-4.1-amd64:cxx-4.1-noarch:desktop-4.1-amd64:desktop-4.1-noarch:languages-4.1-amd64:languages-4.1-noarch:printing-4.1-amd64:printing-4.1-noarch Distributor ID: RedHatEnterprise Description: Red Hat Enterprise Linux release 8.8 (Ootpa) Release: 8.8 Codename: Ootpa
Data formatted according to a validatable standard? Please provide the output of the validator:
NA
Relevant log outputs (up to 20 lines):
NA
Screenshots / relevant information:
NA