'dcm2niix' could not be found, using heudiconv on cluster


#1

Hello,

I thought I would ask a question to see if I’m missing something obvious.

I built a singularity image to run heudiconv on our cluster, using the latest release. I seem to be running into an error where the image begins running, but quickly errors out. saying that it cannot find the command dcm2niix. It also throws a warning about ‘Input terminal_output in interface Dcm2niix is deprecated.’, which may also be part of the problem.

I’ve copied the commands below, from the job request on the cluster through to the error. Any insight at all is appreciated!

-Jordan

qsub -X -I -l nodes=1:ppn=3,vmem=21gb,walltime=4:00:00
# move subject to scratch
setenv TEST sub-001 
setenv SOURCE /autofs/cluster/<lab_data_directory>
setenv WORK /autofs/cluster/<my_account>
mkdir /scratch/raw_dicom && cp -r $SOURCE/dicom/$TEST /scratch/raw_dicom/
# move heuristic over to scratch.
mkdir /scratch/heuristic && cp -r $WORK/heudiconv/heuristics/*proto1*.py /scratch/heuristic/ 

#Move to singularity directory
cd /usr/local/bin

And here is the actual call to singularity:

compute-0-65:jq86[92] ./singularity -v --debug run \
?     --bind '/scratch:/scratch' \
?     $WORK/singularity_images/heudiconv/2018-01-18/nipy_heudiconv_latest*.img \
?     -d '/scratch/raw_dicom/{subject}/ses-01/*/MR.*' \
?     -s $TEST \
?     -f /scratch/heuristic/FSMAP_convert_proto1_ses01.py \
?     -c dcm2niix -b --minmeta \
?     -o /scratch/$TEST

The will run for a minute or so, then report this crash:

compute-0-65:jq86[96] /opt/conda/envs/neuro/lib/python2.7/site-packages/nipype/interfaces/base.py:451: UserWarning: Input terminal_output in interface Dcm2niix is deprecated. Will be removed or raise an error as of release 1.0.0 
  warn(msg)
180202-17:22:36,371 workflow INFO:
	 Executing node convert in dir: /tmp/dcm2niixpCBVMC/convert
180202-17:22:36,531 workflow INFO:
	 Running node "convert" ("nipype.interfaces.dcm2nii.Dcm2niix"), a CommandLine Interface with command:
dcm2niix -b y -z i -x n -t n -m n -f func -o /tmp/dcm2niixpCBVMC/convert -s n -v n /tmp/dcm2niixpCBVMC/convert/MR.1.3.12.2.1107.5.2.34.18001.2016101014450484033594013.
Traceback (most recent call last):
  File "/opt/conda/envs/neuro/bin/heudiconv", line 11, in <module>
    load_entry_point('heudiconv==0.4', 'console_scripts', 'heudiconv')()
  File "/opt/conda/envs/neuro/lib/python2.7/site-packages/heudiconv/cli/run.py", line 99, in main
    process_args(args)
  File "/opt/conda/envs/neuro/lib/python2.7/site-packages/heudiconv/cli/run.py", line 289, in process_args
    overwrite=args.overwrite,)
  File "/opt/conda/envs/neuro/lib/python2.7/site-packages/heudiconv/convert.py", line 164, in prep_conversion
    overwrite=overwrite,)
  File "/opt/conda/envs/neuro/lib/python2.7/site-packages/heudiconv/convert.py", line 242, in convert
    tmpdir)
  File "/opt/conda/envs/neuro/lib/python2.7/site-packages/heudiconv/convert.py", line 349, in nipype_convert
    return convertnode.run()
  File "/opt/conda/envs/neuro/lib/python2.7/site-packages/nipype/pipeline/engine/nodes.py", line 407, in run
    self._run_interface()
  File "/opt/conda/envs/neuro/lib/python2.7/site-packages/nipype/pipeline/engine/nodes.py", line 517, in _run_interface
    self._result = self._run_command(execute)
  File "/opt/conda/envs/neuro/lib/python2.7/site-packages/nipype/pipeline/engine/nodes.py", line 650, in _run_command
    result = self._interface.run()
  File "/opt/conda/envs/neuro/lib/python2.7/site-packages/nipype/interfaces/base.py", line 1089, in run
    runtime = self._run_interface(runtime)
  File "/opt/conda/envs/neuro/lib/python2.7/site-packages/nipype/interfaces/dcm2nii.py", line 287, in _run_interface
    new_runtime = super(Dcm2niix, self)._run_interface(runtime)
  File "/opt/conda/envs/neuro/lib/python2.7/site-packages/nipype/interfaces/base.py", line 1688, in _run_interface
    (self.cmd.split()[0], runtime.hostname))
IOError: command 'dcm2niix' could not be found on host compute-0-65.nmr.mgh.harvard.edu

I can also give the verbose output from Singularity if it would be helpful, but it’s too verbose for the board to allow it in the post!


#2

@jordan_theriault

did you install dcm2niix in your singularity image? It is not included within the heudiconv package.

Alternatively, we have a docker image available on dockerhub. You should be able to convert this into a singularity image fairly easily with the command:

singularity build heudiconv.simg docker://nipy/heudiconv:latest

HTH,
Mathias


#3

Hmm, the singularity image was already built from the latest docker image, using the following command:

docker run --privileged -t --rm \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v ~/singularity_images/heudiconv/:/output \
    singularityware/docker2singularity \
    nipy/heudiconv:latest

But I can try it again using your command and let you know what happens!

-Jordan


#4

@jordan_theriault

one thing to note - this is how I call the singularity image to ensure the correct environment is activated.

singularity exec [singularity-options] $IMAGE \
/neurodocker/startup.sh heudiconv [heudiconv-options]

#5

I’m still getting the error… Used singularity exec after constructing a new singularity image from the docker image.

Error text is below. The errors seem to be the same as before.

compute-0-56:jq86[102] ./singularity exec --bind '/scratch:/scratch' $IMAGE \
? /neurodocker/startup.sh heudiconv \
? -d '/scratch/raw_dicom/{subject}/ses-01/*/MR.*' \
? -s sub-001 \
? -f /scratch/heuristic/FSMAP_convert_proto1_ses01.py \
? -c dcm2niix -b --minmeta \
? -o /scratch/sub-001
/opt/conda/envs/neuro/lib/python2.7/site-packages/nipype/interfaces/base/specs.py:145: UserWarning: Input terminal_output in interface Dcm2niix is deprecated. Will be removed or raise an error as of release 1.0.0 
  warn(msg)
180202-20:37:44,659 workflow INFO:
	 [Node] Setting-up "convert" in "/tmp/dcm2niixZYWKsV/convert".
180202-20:37:44,827 workflow INFO:
	 [Node] Running "convert" ("nipype.interfaces.dcm2nii.Dcm2niix"), a CommandLine Interface with command:
dcm2niix -b y -z i -x n -t n -m n -f func -o /usr/local/bin -s n -v n /tmp/dcm2niixZYWKsV/convert/MR.1.3.12.2.1107.5.2.34.18001.2016101014450484033594013
180202-20:37:44,854 workflow WARNING:
	 [Node] Error on "convert" (/tmp/dcm2niixZYWKsV/convert)
Traceback (most recent call last):
  File "/opt/conda/envs/neuro/bin/heudiconv", line 11, in <module>
    load_entry_point('heudiconv==0.4', 'console_scripts', 'heudiconv')()
  File "/opt/conda/envs/neuro/lib/python2.7/site-packages/heudiconv/cli/run.py", line 99, in main
    process_args(args)
  File "/opt/conda/envs/neuro/lib/python2.7/site-packages/heudiconv/cli/run.py", line 289, in process_args
    overwrite=args.overwrite,)
  File "/opt/conda/envs/neuro/lib/python2.7/site-packages/heudiconv/convert.py", line 164, in prep_conversion
    overwrite=overwrite,)
  File "/opt/conda/envs/neuro/lib/python2.7/site-packages/heudiconv/convert.py", line 242, in convert
    tmpdir)
  File "/opt/conda/envs/neuro/lib/python2.7/site-packages/heudiconv/convert.py", line 350, in nipype_convert
    return convertnode.run()
  File "/opt/conda/envs/neuro/lib/python2.7/site-packages/nipype/pipeline/engine/nodes.py", line 487, in run
    result = self._run_interface(execute=True)
  File "/opt/conda/envs/neuro/lib/python2.7/site-packages/nipype/pipeline/engine/nodes.py", line 571, in _run_interface
    return self._run_command(execute)
  File "/opt/conda/envs/neuro/lib/python2.7/site-packages/nipype/pipeline/engine/nodes.py", line 650, in _run_command
    result = self._interface.run(cwd=outdir)
  File "/opt/conda/envs/neuro/lib/python2.7/site-packages/nipype/interfaces/base/core.py", line 515, in run
    runtime = self._run_interface(runtime)
  File "/opt/conda/envs/neuro/lib/python2.7/site-packages/nipype/interfaces/dcm2nii.py", line 347, in _run_interface
    new_runtime = super(Dcm2niix, self)._run_interface(runtime)
  File "/opt/conda/envs/neuro/lib/python2.7/site-packages/nipype/interfaces/base/core.py", line 1013, in _run_interface
    runtime.hostname))
IOError: No command "dcm2niix" found on host compute-0-56.nmr.mgh.harvard.edu. Please check that the corresponding package is installed.

Could there be some problem with a part of Dcm2nix being deprecated? Thanks for the help!


#6

@jordan_theriault

that’s strange, what version of singularity are you running?

I just tried the following with singularity 2.4.2-HEAD.g91881f7

singularity build heudiconv.simg docker://nipy/heudiconv:latest
singularity shell heudiconv.simg
Singularity: Invoking an interactive shell within container...

Singularity heudiconv.simg:~> which dcm2niix
/usr/local/bin/dcm2niix
Singularity heudiconv.simg:~> dcm2niix
Chris Rorden's dcm2niiX version v1.0.20171017 GCC6.3.0 (64-bit Linux)
...

#7

We’re using singularity 2.4-dist.

Could it be that Singularity is mounting a drive in the same place where the internal /usr/local/bin drive should be? And that’s making it impossible to access dcm2niix?

One strange feature of the call I’m making is that the cluster does not allow me to call singularity as a command. I need to navigate to /usr/local/bin, then call singularity directly via ./singularity. Maybe it’s mounting the current directory by default, and overwriting what’s internal to heudiconv?

Just a guess, as I’m not very familiar with singularity commands. Does it automatically mount the current directory though? And is there a way to override the default if so?


#8

Yup, turns out that was the problem. Singularity mounts your current drive, so be careful about it or else you might overwrite an existing folder within the container.

An easy solution is to just call singularity from outside of the folder, if the node isn’t set up to load it as a module. This code works:

setenv TEST sub-001
setenv DATA <data path>
setenv WORK <my account path>
setenv IMAGE <path to image file>
setenv SINGULARITY /usr/local/bin/singularity
mkdir /scratch/raw_dicom && cp -r $DATA/dicom/$TEST /scratch/raw_dicom/
mkdir /scratch/heuristic && cp -r $WORK/heudiconv/heuristics/*proto1*.py /scratch/heuristic/

$SINGULARITY exec --bind '/scratch:/scratch' $IMAGE \
/neurodocker/startup.sh heudiconv \
-d '/scratch/raw_dicom/{subject}/ses-01/*/MR.*' \
-s $TEST \
-f /scratch/heuristic/FSMAP_convert_proto1_ses01.py \
-c dcm2niix -b --minmeta \
-o /scratch/$TEST

Thanks for the help!