Termination when running the dhcp-structural-pipeline

1

Hi experts,

I originally posted on the MIRTK mailing list, but this seems to be the appropriate location for support on the dHCP structural pipeline.

I want to use the dhcp structural pipeline on neonatal data. Following the instruction on dHCP structural pipelineI have executed a simple run (using docker) on a T2w image as:

$ docker run --rm -t -u $(id -u):$(id -g) -v $PWD/data:/data biomedia/dhcp-structural-pipeline:latest pk323 mr1 40 -T2 data/T2w.nii.gz -t 6
dHCP pipeline v1.1
Subject: pk323
Session: mr1
Age: 40
T1: -
T2: T2w.nii.gz
Directory: /data
Threads: 6
Minimal: 1
Warning: Number of threads>1: This may result in minor reproducibility
differences

/usr/src/structural-pipeline/dhcp-pipeline.sh pk323 mr1 40 -T2
data/T2w.nii.gz -t 6

running segmentation pipeline
/usr/src/structural-pipeline/scripts/segmentation/pipeline.sh
/data/workdir/pk323-mr1/T2/pk323-mr1.nii.gz pk323-mr1 40 -d
/data/workdir/pk323-mr1 -t 6
Pipeline failed: see log files /data/logs/pk323-mr1.segmentation.log
/data/logs/pk323-mr1.segmentation.err for details

I keep getting “Pipeline failed” with the same termination. From the err-file, the problem seems to be related to copying the T2w to a location where it already exists.

The err-file reads

$ cat pk323-mr1.segmentation.err
cp: ‘/data/workdir/pk323-mr1/T2/pk323-mr1.nii.gz’ and ‘/data/workdir/pk323-mr1/T2/pk323-mr1.nii.gz’ are the same file

which in the log-file is related to the executions

/usr/src/structural-pipeline/build/MIRTK/Packages/DrawEM/scripts/tissue-priors.sh pk323-mr1 40 non-rigid-v2 6 : failed
/usr/src/structural-pipeline/build/MIRTK/Packages/DrawEM/pipelines/neonatal-pipeline.sh /data/workdir/pk323-mr1/T2/pk323-mr1.nii.gz 40 -t 6 -c 1 -p 1 -v 1 -atlas non-rigid-v2 : failed

I guess there an obvious solution, but I cannot find it. Advice/Solution appreciated!

A related question. Have previously used the neonatal segmentation pipeline (DrawEM/pipelines/neonatal-pipeline-v1.1.sh). I am primarily interested in the segmentation step of the dHCP pipeline, can I execute this specific step via docker? I guess it translates to how I can execute sh-scripts inside docker?

Cheers
Finn

2

Hi Finn,

just some notes on your last question regarding Docker and running custom scripts or only a subset of the dHCP pipeline within a Docker container using the same environment as provided by dHCP.

For running custom scripts within a Docker container based on the biomedia/dhcp-structural-pipeline image, you have two main options:

a) Create a Docker container from this image by overriding the entry point to a shell, e.g., Bash (option --entrypoint of docker run). You can either copy your scripts into the already mapped /data volume, or use the -v option to mount the location containing your scripts as well into the container (when first creating it with docker run). You probably could also open up the ssh ports with the -p option of docker run and use scp to copy files from host to Docker container.

b) Create your own Docker image based on biomedia/dhcp-structural-pipeline. You use the FROM statement to declare the base image in a “Dockerfile”, and COPY/ADD to copy files from your machine into the Docker image. Then you can either specify your custom script as ENTRYPOINT of the resulting image, or simply use a shell as entry point to run commands within a Docker container created from your custom image. To build the Docker image, you write the build instructions in said “Dockerfile” which is read by “docker build” to create a Docker image that you can then use in place of biomedia/dhcp-structural-pipeline. I suggest you have a look at the Dockerfile used to build the biomedia/dhcp-structural-pipeline image to get an idea (https://github.com/BioMedIA/dhcp-structural-pipeline/blob/527f8357046256651399f92241c1942eaee51c98/Dockerfile) and the Docker documentation (https://docs.docker.com/engine/reference/builder/).

If the step you want to run is an already existing script in the Docker image, you can use “docker run --entrypoint /usr/src/structural-pipeline/scripts/…”. For example, to run the segmentation pipeline you may just need to add the following option to the “docker run” command you mentioned in your post with possibly adjusted arguments: --entrypoint /usr/src/structural-pipeline/scripts/segmentation/pipeline.sh.

Hope that helps,
Andreas

1 Like
3

Hi Andreas,

Thanks for your reply! I am mainly interested in running already existing scripts within the docker.

Defining the docker entry point as docker run -entrypoint /usr/src/structural-pipeline/scripts/segmentation/pipeline.sh ... allows my to run that script. However, when I execute the scripts/segmentation/pipeline.sh as of above:

$ docker run --entrypoint /usr/src/structural-pipeline/scripts/segmentation/pipeline.sh --rm -t -u $(id -u):$(id -g) -v $PWD:/data biomedia/dhcp-structural-pipeline:latest 3dt2_bias_in_meanb0.nii.gz pk323-mr1 40
dHCP Segmentation pipeline
T2:         3dt2_bias_in_meanb0.nii.gz 
Subject:    pk323-mr1 
Age:        40 
Directory:  /data 
Threads:    1

/usr/src/structural-pipeline/scripts/segmentation/pipeline.sh 3dt2_bias_in_meanb0.nii.gz pk323-mr1 40

----------------------------
/usr/src/structural-pipeline/build/MIRTK/Packages/DrawEM/pipelines/neonatal-pipeline.sh 3dt2_bias_in_meanb0.nii.gz 40 -t 1 -c 1 -p 1 -v 1 -atlas non-rigid-v2
DrawEM multi atlas  (HEAD (branch version: 7971b8f1c55ea5137b4006057708c55e11442b27)
Subject:    3dt2_bias_in_meanb0 
Age:        40
Directory:  /data 
Posteriors: 1 
Cleanup:    1 
Threads:    1

/usr/src/structural-pipeline/build/MIRTK/Packages/DrawEM/pipelines/neonatal-pipeline.sh 3dt2_bias_in_meanb0.nii.gz 40 -t 1 -c 1 -p 1 -v 1 -atlas non-rigid-v2

----------------------------
preprocess.sh 3dt2_bias_in_meanb0
FSLDIR environment variable not set!
/usr/src/structural-pipeline/build/MIRTK/Packages/DrawEM/scripts/preprocess.sh 3dt2_bias_in_meanb0 : failed
/usr/src/structural-pipeline/build/MIRTK/Packages/DrawEM/pipelines/neonatal-pipeline.sh 3dt2_bias_in_meanb0.nii.gz 40 -t 1 -c 1 -p 1 -v 1 -atlas non-rigid-v2 : failed

the script preprocess.sh complains the $FSLDIR is not set. But this should have been done in the Dockerfile in , shouldn’t it? It seems all very tangled. I await John’s reply on the original question.

Would an alternative be to use the DrawEM/tools/neonatal-segmentation.sh in MIRTK (the dhcp branch - above is the master branch but they seem to be the same?)? Which would be the entry point to this when run via Docker? Possible to run via Bintray?

Cheers
Finn

4

Hi Finn,

Have you tried using the -e option of docker run to set FSLDIR=/usr/local/fsl? The Dockerfile only creates a file /etc/fsl/fsl.sh that is sourced by the main dhcp-pipeline.sh script if FSLDIR is not set. Hence, you would either have to source this file before invoking the segmentation pipeline or simply set the environment variable as part of the docker run command.

I’m not sure to which extend Draw-EM’s neonatal-segmentation script differs from this official dHCP segmentation pipeline. Best person to ask would be Antonis himself. Maybe you could reach out to him via GitHub issue on the MIRTK/DrawEM project or email.

Andreas

5

Hello Finn,

As Andreas says, your shell needs to be set up for FSL. You need to run:

. /etc/fsl/fsl.sh

Before running anything that uses FSL. The top-level dhcp pipeline script does this as it starts if FSLDIR is not set.

If you set the entrypoint to bash, you can run anything, eg.:

docker run --rm -t \
    -u $(id -u):$(id -g) \
    -v $PWD/data:/data \
    --entrypoint /bin/bash \
    biomedia/dhcp-structural-pipeline:latest \
    -c ". /etc/fsl/fsl.sh && fslmaths"

So to run a single stage:

docker run --rm -t \
    -u $(id -u):$(id -g) \
    -v $PWD/data:/data \
    --entrypoint /bin/bash \
    biomedia/dhcp-structural-pipeline:latest \
    -c ". /etc/fsl/fsl.sh && /usr/src/structural-pipeline/scripts/segmentation/pipeline.sh ..."
6

Regarding your original question:

cp: ‘/data/workdir/pk323-mr1/T2/pk323-mr1.nii.gz’ and ‘/data/workdir/pk323-mr1/T2/pk323-mr1.nii.gz’ are the same file

I’ve been bitten by this annoying message too, but I can’t remember now how I fixed it, sorry. I’ll investigate.

7

Thanks John and Andreas.

Will try to do as you suggest John to run a the segmentation stage only.

Regarding my original question. While awaiting your answer John, I am currently re-running the dhcp-structural-pipeline for the same data and with the same command as in the original post, but on my laptop which has fewer cores but more RAM. Now this troublesome cp-failing instance is passed and pipeline is running, albeit slow (-t 3 now compared to -t 8 before). Inspecting the log-file, there seems to be the same branch of DrawEM:
DrawEM multi atlas (HEAD (branch version: 7971b8f1c55ea5137b4006057708c55e11442b27)
However, I forgot that I have the put the Bintray AppImage of MIRTK in my $PATH. I don’t know if this complicates things?? Many commands are simple MIRTK commands according to the log-file, executed as mirtk "command" ..., and without any path.

Will try to re-run again on the desktop, to see if thing is intermittent or related to the MIRTK Bintray AppImage.

/Finn

8

If you’re running in docker there should be no possibility of mixups like that — the stuff in the container is completely isolated.

9

Yes, thought so the docker container is isolated.

Well, sourcing fsl.sh in docker like you suggest, and then start the segment pipeline works. However, there seems to be non-correct usage description in the scripts/segmentation/pipeline.sh, since it should take more arguments. I can successfully run the segmentation step as:

$ docker run --rm -t -u $(id -u):$(id -g) -v $PWD:/data --entrypoint /bin/bash biomedia/dhcp-structural-pipeline:latest -c ". /etc/fsl/fsl.sh && /usr/src/structural-pipeline/scripts/segmentation/pipeline.sh 3dt2_bias_in_meanb0.nii.gz pk323-mri 40 -t 1

10

Is there a " missing at the end?

11

Yes! Problems when copying from the terminal. Should be:
$ docker run --rm -t -u $(id -u):$(id -g) -v $PWD:/data --entrypoint /bin/bash biomedia/dhcp-structural-pipeline:latest -c ". /etc/fsl/fsl.sh && /usr/src/structural-pipeline/scripts/segmentation/pipeline.sh 3dt2_bias_in_meanb0.nii.gz pk323-mri 40 -t 1"

12

Thanks! Fixed with:

13

Hi,

The message is anoying, but if I’m correct it’s only a warning, it doesn’t affect to the final outputs, am I right? Thanks in advance.

Best regards,

Manuel

14

Clarification on the termination issue. As I wrote originally, dhcp-structrural pipeline segmentation.err-file had only the entry:

$ cat pk323-mr1.segmentation.err
cp: ‘/data/workdir/pk323-mr1/T2/pk323-mr1.nii.gz’ and ‘/data/workdir/pk323-mr1/T2/pk323-mr1.nii.gz’ are the same file

Which, like @mblesac points out, seems to be present in every segmentation err-file.

However, inspecting the logs in the /workdir, the termination was actually related to the an instance when running mirtk draw-em, and seems to be related to a probable memory issue in my case. With the same data, both the segmentation pipeline and the additional pipeline completes on my other computer. However, the surface pipeline now fails early on:

mirtk recon-neonatal-cortex -v --threads=4 --config=/usr/src/structural-pipeline/parameters/recon-neonatal-cortex.cfg --sessions=pk323-mr1 --prefix=surfaces/pk323-mr1/vtk/pk323-mr1 --temp=surfaces/pk323-mr1/vtk/temp-recon/pk323-mr1 --white --pial : failed

With the err-file:

cat logs/pk323-mr1.surface.err
Error: Surface outside bounds of input image: /data/workdir/pk323-mr1/surfaces/pk323-mr1/vtk/temp-recon/pk323-mr1/corpus-callosum-mask.nii.gz
Traceback (most recent call last):
  File "/usr/src/structural-pipeline/build/MIRTK/build/lib/tools/recon-neonatal-cortex", line 938, in <module>
    check=args.check)
  File "/usr/src/structural-pipeline/build/MIRTK/build/lib/tools/recon-neonatal-cortex", line 570, in recon_neonatal_cortex
    corpus_callosum_mask=corpus_callosum_mask, temp=temp_dir)
  File "/usr/src/structural-pipeline/build/MIRTK/build/lib/python/mirtk/deformable/neonatal_cortex.py", line 1102, in recon_cortical_surface
    out1 = push_output(stack, add_corpus_callosum_mask(hull, mask=corpus_callosum_mask, oname=out1))
  File "/usr/src/structural-pipeline/build/MIRTK/build/lib/python/mirtk/deformable/neonatal_cortex.py", line 578, in add_corpus_callosum_mask
    project_mask(iname, oname, mask, name='ImplicitSurfaceFillMask', dilation=20, invert=True)
  File "/usr/src/structural-pipeline/build/MIRTK/build/lib/python/mirtk/deformable/neonatal_cortex.py", line 467, in project_mask
    opts={'labels': mask, 'fill': fill, 'dilation-radius': dilation, 'name': name})
  File "/usr/src/structural-pipeline/build/MIRTK/build/lib/python/mirtk/deformable/neonatal_cortex.py", line 183, in run
    _run(tool, args=args, opts=opts, verbose=showcmd, threads=threads)
  File "/usr/src/structural-pipeline/build/MIRTK/build/lib/python/mirtk/subprocess.py", line 162, in run
    check_call(argv, verbose=verbose)
  File "/usr/src/structural-pipeline/build/MIRTK/build/lib/python/mirtk/subprocess.py", line 124, in check_call
    _call(argv, verbose=verbose, execfunc=subprocess.check_call)
  File "/usr/src/structural-pipeline/build/MIRTK/build/lib/python/mirtk/subprocess.py", line 114, in _call
    return execfunc(argv)
  File "/usr/lib/python2.7/subprocess.py", line 541, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '[u'/usr/src/structural-pipeline/build/MIRTK/build/lib/tools/project-onto-surface', '/data/workdir/pk323-mr1/surfaces/pk323-mr1/vtk/temp-recon/pk323-mr1/cerebrum-rh-hull.vtp', '/data/workdir/pk323-mr1/surfaces/pk323-mr1/vtk/temp-recon/pk323-mr1/cerebrum-rh-1.vtp', '-labels', '/data/workdir/pk323-mr1/surfaces/pk323-mr1/vtk/temp-recon/pk323-mr1/corpus-callosum-mask.nii.gz', '-name', 'ImplicitSurfaceFillMask', '-dilation-radius', '20', '-fill', 'False']' returned non-zero exit status 1
Error: recon-neonatal-cortex command returned non-zero exit status 1

Inspecting the surface folder I cannot find the input image: /data/workdir/pk323-mr1/surfaces/pk323-mr1/vtk/temp-recon/pk323-mr1/corpus-callosum-mask.nii.gz. This folder only contains a one T2w image:

$ ls workdir/pk323-mr1/surfaces/pk323-mr1/vtk/temp-recon/pk323-mr1/
t2w-image.nii.gz

which, indeed, has the right temporal brain surface cut out. Is this the problem? How come it is too truncated?

t2_in_surface_folder0000
t2_in_surface_folder0000.png454×512 79.1 KB

I don’t use dHCP, but I have high-resolution and high-quality sMRI data. In this case I only provided a T2 as input (the T1 was motion-corrupted), but the pipeline only uses T1 for estimation of myelination and (if available) refining the WM mesh, according to the pipeline paper.

Cheers
Finn

15

Hi,

I’ve been trying your orders and I’ve got an error coming up and I don’t understand where the problem is coming from. If you have had the same thing I am taking solution or idea that can help me.

docker run --rm \

-u (id -u):(id -g) \

-v $PWD/data:/data \

–entrypoint /usr/src/structural-pipeline/scripts/segmentation/pipeline.sh T2w.nii.gz subject1 session2 29 \

biomedia/dhcp-structural-pipeline:latest \

-c ". /etc/fsl/fsl.sh && /usr/src/structural-pipeline/scripts/segmentation/pipeline.sh subject1 session2 29 -T2 data/T2w.nii.gz "

docker: Error parsing reference: “T2w.nii.gz” is not a valid repository/tag: repository name must be lowercase.

See ‘docker run --help’.

Thank you in advance,
Inès

16

Hello Finn,

I’m facing a similar issue as yours.
How did you solved it?

Thank you in advance.
Tânia

17

Hi @tfvaz,

Since this is an old issue, may you please open a new issue under the Software Support category, making sure to fill in all the information requested by the prepopulated template?

Thanks,
Steven

18

Hi Tânia,

I run this line of code to just run the segmentation part.

In my code you set beforehand the parameters $outputFolder, the BIDS-like subject and session IDs couple $sID and $ssID, $age and $threads.

Note that segmentation.sh will give you the files in the “working directory”-tree.

docker run --rm -t -u $(id -u):$(id -g) \
-v $outputFolder:/data --entrypoint /bin/bash biomedia/dhcp-structural-pipeline:latest \
-c ". /etc/fsl/fsl.sh && /usr/src/structural-pipeline/scripts/segmentation/pipeline.sh $T2.nii.gz sub-${sID}_ses-${ssID} $age -t $threads"

Hope this can help!

Otherwise, follow the instructions from @Steven.
Finn

19

Hi Finn,
Thanks for your reply, but unfortunately it didn’t work.
I’ll open a new issue.
Thanks,
Tânia

20

Hi Steven,
I’ll open a new issue.
Thanks,
Tânia