ValueError: Fieldmap-less (SyN) estimation was requested, but PhaseEncodingDirection information appears to be absent. fMRIprep v. 22.0.2

Nuttaon · October 26, 2022, 3:17pm

Hi Steven,

I used Singularity.

Here is the script to call fMRIprep.

#!/bin/bash

#Read the input file list
while read -r input; do
subj=$(basename “$input”)
echo $input
#Place each job in the queue
cat > temp.sh <<EOF
#!/bin/bash
#SBATCH --job-name=Task_$subj
#SBATCH --mail-type=ALL
#SBATCH --mail-user=Email
#SBATCH --ntasks=2
#SBATCH --cpus-per-task=12
#SBATCH --mem-per-cpu=7gbs
#SBATCH --time=01-00:00:00
#SBATCH --output=Out_files/%x-%j.out
#SBATCH --account=GR

settup env

module load singularity
module load fsl/6.0.4
module load python/3.9.1
module load fmriprep/22.0.2
source ~/myPython39venv/bin/activate
python --version
cd /scratch/g/GR1/GR2/Task/PROJECT
chmod 777 ./fmriprep.sh
./fmriprep.sh subj >> Log_files/{subj}.log
EOF

chmod a+wrx temp.sh
sbatch temp.sh
done <"$1"

rm temp.sh

echo ‘All jobs send’

#Show that the job is queued
squeue -u whoami

Here is the fMRIprep.sh
#settup env

#User inputs:
bids_root_dir=/scratch/g/GR1/GR2/Task/PROJECT/bidstest5/
#subj=3172101920
subj=$1
nthreads=4
mem=40 #gb
container=singularity

#Begin:

#Convert virtual memory from gb to mb
mem=echo "${mem//[!0-9]/}" #remove gb at end
mem_mb=echo $(((mem*1000)-5000)) #reduce some memory for buffer space during pre-processing

export FS_LICENSE=license.txt \

fmriprep $bids_root_dir $bids_root_dir/derivatives
participant
–participant-label $subj
–md-only-boilerplate
–fs-license-file /license.txt
–fs-no-reconall
–output-spaces MNI152NLin2009cAsym:res-2
–nthreads $nthreads
–mem_mb $mem_mb
–ignore slicetiming
–ignore fieldmaps
–use-syn-sdc
-w fmriprep_work_MID

The content in the subj_list.txt is
subj-ID

Below is the log output.

==========================================
SLURM_JOB_ID = 478028
SLURM_NODELIST = cn59

Python 3.9.1
Process Process-2:
Traceback (most recent call last):
File “/opt/conda/lib/python3.9/multiprocessing/process.py”, line 315, in _bootstrap
self.run()
File “/opt/conda/lib/python3.9/multiprocessing/process.py”, line 108, in run
self._target(*self._args, **self._kwargs)
File “/opt/conda/lib/python3.9/site-packages/fmriprep/cli/workflow.py”, line 114, in build_workflow
retval[“workflow”] = init_fmriprep_wf()
File “/opt/conda/lib/python3.9/site-packages/fmriprep/workflows/base.py”, line 91, in init_fmriprep_wf
single_subject_wf = init_single_subject_wf(subject_id)
File “/opt/conda/lib/python3.9/site-packages/fmriprep/workflows/base.py”, line 381, in init_single_subject_wf
raise ValueError(message)
ValueError: Fieldmap-less (SyN) estimation was requested, but PhaseEncodingDirection information appears to be absent.

Thanks for your help!
Nuttaon

Steven · October 26, 2022, 4:07pm

Hi,

This does not appear to be a singularity command to me. A singularity fmriprep invocation would begin with something like singularity run fmriprep_22.0.2.img $bids_dir $output_dir participant [ARGS]

Nuttaon · October 26, 2022, 7:49pm

Hi,

I have set up the singularity but still got the same error.

Here is the new script:
singularity run /scratch/g/GR1/GR2/Task/fmriprep-22.0.2.simg $bids_root_dir $bids_root_dir/derivatives
participant
–participant-label $subj
–skip-bids-validation
–md-only-boilerplate
–fs-license-file /license.txt
–fs-no-reconall
–output-spaces MNI152NLin2009cAsym:res-2
–nthreads $nthreads
–mem_mb $mem_mb
–ignore slicetiming
–ignore fieldmaps
–use-syn-sdc
-w /scratch/g/GR1/GR2/Task/TOPUP_test1

Here is the log output:

SLURM_JOB_ID = 478116
SLURM_NODELIST = cn04

Python 3.9.1
Process Process-2:
Traceback (most recent call last):
File “/opt/conda/lib/python3.9/multiprocessing/process.py”, line 315, in _bootstrap
self.run()
File “/opt/conda/lib/python3.9/multiprocessing/process.py”, line 108, in run
self._target(*self._args, **self._kwargs)
File “/opt/conda/lib/python3.9/site-packages/fmriprep/cli/workflow.py”, line 114, in build_workflow
retval[“workflow”] = init_fmriprep_wf()
File “/opt/conda/lib/python3.9/site-packages/fmriprep/workflows/base.py”, line 91, in init_fmriprep_wf
single_subject_wf = init_single_subject_wf(subject_id)
File “/opt/conda/lib/python3.9/site-packages/fmriprep/workflows/base.py”, line 381, in init_single_subject_wf
raise ValueError(message)
ValueError: Fieldmap-less (SyN) estimation was requested, but PhaseEncodingDirection information appears to be absent.

Thank you.

jsein · October 26, 2022, 8:00pm

Hi again,

In your script, it looks like your singularity call does not bound any directory to the singularity container using the -B option). Here is an example of a call I use :

singularity run -B /scratch/jsein/BIDS:/work,$HOME/.templateflow:/opt/templateflow --cleanenv /scratch/jsein/my_images/fmriprep-22.0.2.simg \
		 --fs-license-file /work/freesurfer/license.txt /work/$study /work/$study/derivatives/fmriprep  \
		 participant --participant-label $sub \
		 -w /work/temp_data_${study}\
		 --mem-mb 50000 --omp-nthreads 10 --nthreads 12  \
		 --fd-spike-threshold 0.5 --dvars-spike-threshold 2.0 --bold2t1w-dof 9 --cifti-output 91k   \
		 --output-spaces MNI152NLin2009cAsym MNI152NLin6Asym T1w --ignore slicetiming sbref \
		 --fs-subjects-dir /work/$study/derivatives/fmriprep/sourcedata/freesurfer --debug compcor \
		 --bids-filter-file /work/$study/code/fmriprep/bids-filter.json

I am not sure if it is a problem in your case but other than that everything looks there and I don’t see why this error is still popping up.

Nuttaon · October 27, 2022, 2:52pm

Hi,

I tried adding the -B option. The fMRIprep did run successfully but there were no results. If you could please explain it a little bit more of -B function, it would be great.

Thank you!

Steven · October 27, 2022, 4:09pm

When singularity begins a container, no data from your machine goes inside of it by default (except for your home drive). So if you want fMRIPrep to be able to read from / write to a drive, you need to mount it with the -B argument. All relevant drives should be mounted (some people may have their bids root and scratch directory on different file system). You can mount multiple drives with -B /drive1,/drive2 etc.

Best,
Steven

Nuttaon · October 27, 2022, 10:09pm

Hi,

I added the -B arguments. The script ran successfully but it still showed “susceptibility distortion correction: none” in the report.

Here is the script

#User inputs:
bids_root_dir=/scratch/g/jgoveas/R01_nii/baseline_mid/PROJECT/bidstest5/
subj=3172101920
nthreads=4
mem=40 #gb
container=singularity #docker or singularity

#Begin:

#Convert virtual memory from gb to mb
mem=echo "${mem//[!0-9]/}"
mem_mb=echo $(((mem*1000)-5000))

export FS_LICENSE=/scratch/g/jgoveas/R01_nii/baseline_mid/bids/license.txt \
singularity run
-B /scratch/g/jgoveas/R01_nii/baseline_mid
-B /scratch/g/jgoveas/R01_nii/baseline_mid/PROJECT
-B /scratch/g/jgoveas/R01_nii/baseline_mid/PROJECT/bidstest5
–cleanenv /scratch/g/jgoveas/R01_nii/baseline_mid/fmriprep-22.0.2.simg $bids_root_dir $bids_root_dir/derivatives
participant
–participant-label $subj
–skip-bids-validation
–md-only-boilerplate
–fs-license-file /scratch/g/jgoveas/R01_nii/baseline_mid/bids/license.txt
–fs-no-reconall
–output-spaces MNI152NLin2009cAsym:res-2
–output-spaces func:res-native
–output-spaces MNI152NLin2009cAsym:res-native
–nthreads $nthreads
–mem_mb mem_mb \ --ignore slicetiming \ -w /scratch/g/jgoveas/R01_nii/baseline_mid/TOPUP_{subj}

Here is the tree hierarchy:
sub-3172101920
└── ses-1
├── anat
│ ├── sub-3172101920_ses-1_FLAIR.json
│ ├── sub-3172101920_ses-1_FLAIR.nii
│ ├── sub-3172101920_ses-1_T1w.json
│ └── sub-3172101920_ses-1_T1w.nii
├── fmap
│ ├── sub-3172101920_ses-1_acq-mid1_dir-AP_epi.json
│ ├── sub-3172101920_ses-1_acq-mid1_dir-AP_epi.nii
│ ├── sub-3172101920_ses-1_acq-mid1_dir-PA_epi.json
│ ├── sub-3172101920_ses-1_acq-mid1_dir-PA_epi.nii
│ ├── sub-3172101920_ses-1_acq-mid2_dir-AP_epi.json
│ ├── sub-3172101920_ses-1_acq-mid2_dir-AP_epi.nii
│ ├── sub-3172101920_ses-1_acq-mid2_dir-PA_epi.json
│ └── sub-3172101920_ses-1_acq-mid2_dir-PA_epi.nii
└── func
├── sub-3172101920_ses-1_task-mid1_bold.json
├── sub-3172101920_ses-1_task-mid1_bold.nii
├── sub-3172101920_ses-1_task-mid2_bold.json
└── sub-3172101920_ses-1_task-mid2_bold.nii

Here is one of the fmap.json
{
“Modality”: “MR”,
“MagneticFieldStrength”: 3,
“ImagingFrequency”: 127.772,
“Manufacturer”: “GE”,
“PulseSequenceName”: “epi”,
“InternalPulseSequenceName”: “EPI”,
“ManufacturersModelName”: “SIGNA Premier”,
“InstitutionName”: “MCW Premier”,
“DeviceSerialNumber”: “00000000FLORIO1X”,
“StationName”: “MCWPREM”,
“BodyPartExamined”: “BRAIN”,
“PatientPosition”: “HFS”,
“ProcedureStepDescription”: “GovGriefO1”,
“SoftwareVersions”: “27\LX\MR Software release:RX27.0_R03_1850.a”,
“MRAcquisitionType”: “2D”,
“SeriesDescription”: “SE Map PE2”,
“ProtocolName”: “Goveas_GovGriefR01_10052”,
“ScanningSequence”: “EP\RM”,
“SequenceVariant”: “NONE”,
“ScanOptions”: “SAT_GEMS\MP_GEMS\EPI_GEMS\FS”,
“ImageType”: [“ORIGINAL”, “PRIMARY”, “OTHER”],
“SeriesNumber”: 26,
“AcquisitionTime”: “10:20:1.000000”,
“AcquisitionNumber”: 1,
“TriggerDelayTime”: 14677,
“SliceThickness”: 2.4,
“SpacingBetweenSlices”: 2.4,
“SAR”: 0.20231,
“EchoTime”: 0.08,
“RepetitionTime”: 7.4,
“FlipAngle”: 90,
“PhaseEncodingPolarityGE”: “Unflipped”,
“CoilString”: “RM:32NovaHead2”,
“PercentPhaseFOV”: 100,
“PercentSampling”: 100,
“AcquisitionMatrixPE”: 90,
“ReconMatrixPE”: 128,
“EffectiveEchoSpacing”: 0.000403654,
“TotalReadoutTime”: 0.051264,
“IntendedFor”: “ses-1/func/sub-3172101920_ses-1_task-mid1_bold.nii”,
“PixelBandwidth”: 3906.25,
“PhaseEncodingDirection”: “j-”,
“SliceTiming”: [
0,
3.7,
0.123333,
3.82333,
0.246667,
3.94667,
0.37,
4.07,
0.493333,
4.19333,
0.616667,
4.31667,
0.74,
4.44,
0.863333,
4.56333,
0.986667,
4.68667,
1.11,
4.81,
1.23333,
4.93333,
1.35667,
5.05667,
1.48,
5.18,
1.60333,
5.30333,
1.72667,
5.42667,
1.85,
5.55,
1.97333,
5.67333,
2.09667,
5.79667,
2.22,
5.92,
2.34333,
6.04333,
2.46667,
6.16667,
2.59,
6.29,
2.71333,
6.41333,
2.83667,
6.53667,
2.96,
6.66,
3.08333,
6.78333,
3.20667,
6.90667,
3.33,
7.03,
3.45333,
7.15333,
3.57667,
7.27667 ],
“ImageOrientationPatientDICOM”: [
1,
-0,
0,
-0,
1,
0 ],
“InPlanePhaseEncodingDirectionDICOM”: “COL”,
“ConversionSoftware”: “dcm2niix”,
“ConversionSoftwareVersion”: “v1.0.20210317”
}

Please let me know what is missing.

Thank you for your help.

Steven · October 27, 2022, 11:44pm

You need the phase encoding direction set in your BOLD json file to use SYN-SDC.

Nuttaon · October 28, 2022, 12:30am

Hi,

Thank you for your help. I have “PhaseEncodingDirection”: “j-”, in the BOLD.json. Is this ok or do I need to modify it?

Thanks again.

jsein · October 28, 2022, 1:24pm

If you want to use your fmap/_epi images for Fieldmap correction (since in your last version of your script you don’t use --ignore fieldmaps), you need to provide the "IntendedFor" tag in the fmap/epi.json files or "B0FieldSource"/"B0FieldIdentifier" tags in func.json and fmap/epi.json files respectively.

Nuttaon · October 29, 2022, 2:23pm

Hi,

I added "B0FieldSource" /"B0FieldIdentifier" tags in func.json and fmap/epi.json files respectively and still have "IntendedFor" tag in the fmap/epi.json files. However, the susceptibility distortion correction is still not run.

Note: Bids validated. I cleaned the work folder every time. The data was collected using GE scanner.

Here is the fmap.json:

"EffectiveEchoSpacing": 0.000403654,
"TotalReadoutTime": 0.051264,
    "IntendedFor": "ses-1/func/sub-3172101920_ses-1_task-mid1_bold.nii",
"PixelBandwidth": 3906.25,
"PhaseEncodingDirection": "j-",
    "B0FieldIdentifier": "pepolar_fmap0",

Here is the corresponding Bold.json:

"EffectiveEchoSpacing": 0.000588944,
"TotalReadoutTime": 0.052416,
"PixelBandwidth": 5555.56,
"PhaseEncodingDirection": "j-",
    "B0FieldSource": "pepolar_fmap0",

Here is the fmriprep command:

singularity run \
-B /scratch/g/jgoveas/R01_nii/baseline_mid \
-B /scratch/g/jgoveas/R01_nii/baseline_mid/PROJECT \
-B /scratch/g/jgoveas/R01_nii/baseline_mid/PROJECT/bidstest5 \
--cleanenv /scratch/g/jgoveas/R01_nii/baseline_mid/fmriprep-22.0.2.simg \
$bids_root_dir $bids_root_dir/derivatives \
participant \
--participant-label $subj \
--skip-bids-validation \
--md-only-boilerplate \
--fs-license-file /scratch/g/jgoveas/R01_nii/baseline_mid/bids/license.txt \
--fs-no-reconall \
--output-spaces MNI152NLin2009cAsym:res-2 \
--nthreads $nthreads \
--mem_mb $mem_mb \
--ignore slicetiming \
-w /scratch/g/jgoveas/R01_nii/baseline_mid/TOPUP_${subj}

Thank you for your help.

jsein · October 29, 2022, 9:29pm

Thank you for providing those details. We must be still missing something here.

I guess you correctly wrote the B0FiledIdentifier and ÌntendedFor fields in both AP and PA _epi.json files in the fmap/ folder?
Did you check that dir-AP_epi.json and dir-PA_epi.json files have actually opposite phase encoding direction (i.e "-j"and "j"respectively)?
Why do you keep --skip-bids-validationoption in your command? Would you get an error if you remove it?
Would you mind sharing the beginning of the output log of FMRIPREP, where details are thrown about the detection of fieldmaps and the identification of functional runs to be corrected for SDC?

Nuttaon · October 30, 2022, 8:07pm

Thank you for being patient with me.

Yes, I believed that I have the B0FiledIdentifier and ÌntendedFor fields in both AP and PA _epi.json files in the fmap/ folder; and the AP and PA were also correct.

I reran with --skip-bids-validation. There were some warning regarding missing README file. The fmriprep ran successfully otherwise.

Below is the beginning of the output log. There were nothing about sdc or fieldmaps. (nothing showed up when using the find function for those two keywords either).

221030-09:34:17,968 nipype.workflow IMPORTANT:
Building fMRIPrep’s workflow:
* BIDS dataset path: /scratch/g/jgoveas/R01_nii/baseline_mid/PROJECT/bidstest5.
* Participant list: [‘3172101920’].
* Run identifier: 20221030-093359_ba671b65-2560-4ecb-afdc-1dbd47b3ca7a.
* Output spaces: MNI152NLin2009cAsym:res-2.
* Pre-run FreeSurfer’s SUBJECTS_DIR: /scratch/g/jgoveas/R01_nii/baseline_mid/PROJECT/bidstest5/derivatives/sourcedata/freesurfer.
221030-09:34:19,758 nipype.workflow INFO:
No single-band-reference found for sub-3172101920_ses-1_task-mid1_bold.nii.
221030-09:34:19,977 nipype.workflow INFO:
No single-band-reference found for sub-3172101920_ses-1_task-mid2_bold.nii.
221030-09:34:23,182 nipype.workflow INFO:
fMRIPrep workflow graph with 522 nodes built successfully.
221030-09:34:36,142 nipype.workflow IMPORTANT:
fMRIPrep started!
221030-09:34:39,390 nipype.workflow INFO:
[Node] Setting-up “fmriprep_22_0_wf.single_subject_3172101920_wf.anat_preproc_wf.brain_extraction_wf.full_wm” in “/scratch/g/jgoveas/R01_nii/baseline_mid/TOPUP2_3172101920/fmriprep_22_0_wf/single_subject_3172101920_wf/anat_preproc_wf/brain_extraction_wf/full_wm”.
221030-09:34:39,413 nipype.workflow INFO:
[Node] Executing “full_wm” <nipype.interfaces.utility.wrappers.Function>
221030-09:34:39,570 nipype.workflow INFO:
[Node] Setting-up “fmriprep_22_0_wf.single_subject_3172101920_wf.anat_preproc_wf.brain_extraction_wf.lap_tmpl” in “/scratch/g/jgoveas/R01_nii/baseline_mid/TOPUP2_3172101920/fmriprep_22_0_wf/single_subject_3172101920_wf/anat_preproc_wf/brain_extraction_wf/lap_tmpl”.
221030-09:34:39,573 nipype.workflow INFO:
[Node] Setting-up “fmriprep_22_0_wf.single_subject_3172101920_wf.anat_preproc_wf.brain_extraction_wf.res_tmpl” in “/scratch/g/jgoveas/R01_nii/baseline_mid/TOPUP2_3172101920/fmriprep_22_0_wf/single_subject_3172101920_wf/anat_preproc_wf/brain_extraction_wf/res_tmpl”.
221030-09:34:39,592 nipype.workflow INFO:
[Node] Executing “res_tmpl” <niworkflows.interfaces.nibabel.RegridToZooms>
221030-09:34:39,592 nipype.workflow INFO:
[Node] Executing “lap_tmpl” <nipype.interfaces.ants.utils.ImageMath>
221030-09:34:40,113 nipype.workflow INFO:
[Node] Finished “full_wm”, elapsed time 0.690009s.

Thank you.

jsein · October 30, 2022, 9:45pm

Being only a regular user of FMRIPREP, I am running out of suggestions to offer. One last thing, which should be unrelated, is the presence of --fs-no-reconall which was not advised to use earlier by @Steven.

When you run the bids validator, does fmap shows up as one available modality?

It looks like the fmap folder is not even seen by FMRIPREP!

I would be curious to try a minimal sample of your dataset on my side!

Nuttaon · October 30, 2022, 10:31pm

Yep, the bids validator does see fmap folder. Yes, I totally agree. It seems that fMRIprep doesn’t see it at all.

When I used the previous version (20.2.3) to run with fmap or without (–use-syn-sdc), both worked.

I still wonder what “ValueError: Fieldmap-less (SyN) estimation was requested, but PhaseEncodingDirection information appears to be absent.” mean when using --use-syn-sdc.

That would be great if you can try running it. Thanks!

Steven · November 1, 2022, 8:05pm

Are you absolutely sure you are using fMRIPrep 22.0.2? Because as you can see in the changelog (What’s new — fmriprep version documentation) there was a bug with using SYN and --ignore-fieldmaps that was not patched until 22.0.2. If you run fmriprep -v using your 22.0.2 container, does the terminal return 22.0.2?

Nuttaon · November 3, 2022, 8:36pm

Hi,

Yep, it shows 22.0.2.

Thank you.

jsein · November 4, 2022, 12:11pm

Hi @Nuttaon , I played with your dataset and indeed, you may have uncovered a bug!

I run your dataset with either FMRIPREP v21.02 or v22.02, and in one case (v21.0.2) SDC correction is run, and in the other case (v22.0.2) , it is not run!

I used exactly the same dataset, and the same script, just changing the singularity image and the name of the working directory, the output directory and the freesurfer directory.

singularity run -B /scratch/jsein/BIDS:/work,$HOME/.templateflow:/opt/templateflow --cleanenv /scratch/jsein/my_images/fmriprep-22.0.2.simg \
		 --fs-license-file /work/freesurfer/license.txt /work/$study /work/$study/derivatives/fmriprep22  \
		 participant --participant-label $sub \
		 -w /work/temp_data22_${study}\
		 --mem-mb 50000 --omp-nthreads 10 --nthreads 12  \
		 --fd-spike-threshold 0.5 --dvars-spike-threshold 2.0 --bold2t1w-dof 6 --debug fieldmaps --write-graph  \
		 --output-spaces T1w MNI152NLin2009cAsym --ignore slicetiming  --fs-subjects-dir /work/$study/derivatives/fmriprep22/sourcedata/freesurfer \

or

singularity run -B /scratch/jsein/BIDS:/work,$HOME/.templateflow:/opt/templateflow --cleanenv /scratch/jsein/my_images/fmriprep-21.0.2.simg \
		 --fs-license-file /work/freesurfer/license.txt /work/$study /work/$study/derivatives/fmriprep21  \
		 participant --participant-label $sub \
		 -w /work/temp_data21_${study}\
		 --mem-mb 50000 --omp-nthreads 10 --nthreads 12  \
		 --fd-spike-threshold 0.5 --dvars-spike-threshold 2.0 --bold2t1w-dof 6 --debug fieldmaps --write-graph  \
		 --output-spaces T1w MNI152NLin2009cAsym --ignore slicetiming  --fs-subjects-dir /work/$study/derivatives/fmriprep21/sourcedata/freesurfer \

jsein · November 4, 2022, 3:02pm

It is very strange, I don’t understand what is going on.

I can’t wrap my mind around the problem right now:

What is sure is that FMRIPREP v21.0.2 systematically “detects” the fieldmaps.
I had previously used FMRIPREP 22.0.2 where the fieldmaps were correctly detected. But If I create a new directory in which I copy-paste a subject from the project that worked, and I re-run FMRIPREP v22.0.2 (same code that worked previously) on this new project (but the same subject!), the field maps are not detected anymore!
-Crazier: if now I copy the subject from @Nuttaon into the BIDS directory for which FMRIPREP v22.0.2 was detecting the fieldmaps, it does detect the fieldmap!!!

I don’t know FMRIPREP deep enough to know what is the heuristic for the selection of the workflows, to debug further.

Nuttaon · November 4, 2022, 7:31pm

Hi @jsein ,

Thank you so much for testing this! I still have no luck. My experience is that FMRIPREP version 20.2.3 consistently detects the fieldmaps. While, version 22.0.2 never detected them (I tried singularity, docker, and python on two different platforms: OS and Linux, and on different task and different subjects). I also tried using the updated dcm2niix.

This is a silly question. Can it be dataset_description.json?

ValueError: Fieldmap-less (SyN) estimation was requested, but PhaseEncodingDirection information appears to be absent. fMRIprep v. 22.0.2

settup env

========================================== SLURM_JOB_ID = 478028 SLURM_NODELIST = cn59

Here is the log output:

SLURM_JOB_ID = 478116 SLURM_NODELIST = cn04

==========================================
SLURM_JOB_ID = 478028
SLURM_NODELIST = cn59

SLURM_JOB_ID = 478116
SLURM_NODELIST = cn04