Issues with GNU libc, freesurfer license in fmriprep

1

Just following up from the conversation stared on twitter.

I’m running fmriprep 1.2.6 on the Midnight Scan Club dataset. I’m using fmriprep-docker to launch fmriprep using the following command:

sudo -E env "PATH=$PATH" fmriprep-docker /Raid6/raw/midnight_scan_club /Raid6/raw/users/dan/Documents/Projects/midnight/data/ participant --fs-license-file ~/Documents/licenses/freesurfer/license.txt -w /Raid6/users/dan/Documents/Projects/midnight/fmriprep_working_dir/

The env "PATH=$PATH" is there to allow me to run fmriprep-docker (installed in my user) as root. This produces the following command:

RUNNING: docker run --rm -it -e DOCKER_VERSION_8395080871=18.06.1-ce -v /Raid6/users/dan/Documents/licenses/freesurfer/license.txt:/opt/freesurfer/license.txt:ro -v /Raid6/raw/midnight_scan_club:/data:ro -v /Raid6/raw/users/dan/Documents/Projects/midnight/data:/out -v /Raid6/users/dan/Documents/Projects/midnight/fmriprep_working_dir:/scratch poldracklab/fmriprep:1.2.6-1 /data /out participant -w /scratch

It will usually run for ~2 days until the following error is reached for each participant:

reading source '/scratch/fmriprep_wf/single_subject_MSC10_wf/anat_preproc_wf/anat_template_wf/n4_correct/mapflow/_n4_correct0/sub-MSC10_ses-struct01_run-01_T1w_corrected.nii.gz'...
--------------------------------------------------------------------------
GNU libc version: 2.23
ERROR: Systems running GNU glibc version greater than 2.15
  require a newly formatted license file (it's free). Please
  download a new one from the following page:
  http://surfer.nmr.mgh.harvard.edu/registration.html
--------------------------------------------------------------------------
Standard error:

Return code: 255

This ultimately produces the following message:

190207-11:46:41,681 nipype.workflow INFO:
	 [Node] Finished "fmriprep_wf.single_subject_MSC10_wf.func_preproc_ses_func11_task_motor_run_01_wf.bold_confounds_wf.dvars".
190207-11:46:42,219 nipype.workflow INFO:
	 [Node] Finished "fmriprep_wf.single_subject_MSC10_wf.func_preproc_ses_func11_task_motor_run_02_wf.bold_confounds_wf.dvars".
190207-11:48:20,257 nipype.workflow ERROR:
	 could not run node: fmriprep_wf.single_subject_MSC01_wf.anat_preproc_wf.anat_template_wf.t1_merge
190207-11:48:20,271 nipype.workflow ERROR:
	 could not run node: fmriprep_wf.single_subject_MSC02_wf.anat_preproc_wf.anat_template_wf.t1_merge
190207-11:48:20,284 nipype.workflow ERROR:
	 could not run node: fmriprep_wf.single_subject_MSC03_wf.anat_preproc_wf.anat_template_wf.t1_merge
190207-11:48:20,296 nipype.workflow ERROR:
	 could not run node: fmriprep_wf.single_subject_MSC04_wf.anat_preproc_wf.anat_template_wf.t1_merge
190207-11:48:20,309 nipype.workflow ERROR:
	 could not run node: fmriprep_wf.single_subject_MSC05_wf.anat_preproc_wf.anat_template_wf.t1_merge
190207-11:48:20,322 nipype.workflow ERROR:
	 could not run node: fmriprep_wf.single_subject_MSC06_wf.anat_preproc_wf.anat_template_wf.t1_merge
190207-11:48:20,335 nipype.workflow ERROR:
	 could not run node: fmriprep_wf.single_subject_MSC07_wf.anat_preproc_wf.anat_template_wf.t1_merge
190207-11:48:20,348 nipype.workflow ERROR:
	 could not run node: fmriprep_wf.single_subject_MSC08_wf.anat_preproc_wf.anat_template_wf.t1_merge
190207-11:48:20,361 nipype.workflow ERROR:
	 could not run node: fmriprep_wf.single_subject_MSC09_wf.anat_preproc_wf.anat_template_wf.t1_merge
190207-11:48:20,375 nipype.workflow ERROR:
	 could not run node: fmriprep_wf.single_subject_MSC10_wf.anat_preproc_wf.anat_template_wf.t1_merge
Preprocessing did not finish successfully. Errors occurred while processing data from participants: MSC01 (1), MSC02 (1), MSC03 (1), MSC04 (1), MSC05 (1), MSC06 (1), MSC07 (1), MSC08 (1), MSC09 (1), MSC10 (1). Check the HTML reports for details.
Sentry is attempting to send 1 pending error messages
Waiting up to 2.0 seconds
Press Ctrl-C to quit
Captured warning (<class 'ResourceWarning'>): unclosed <socket.socket fd=7, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.17.0.3', 57825), raddr=('35.188.42.15', 443)>
fMRIPrep: Please report errors to https://github.com/poldracklab/fmriprep/issues

I’ve downloaded new linux Freesurfer licences and still continue to get the error, despite the message. I’m aware that this is a known, somewhat unresolved, issue discussed on here (e.g., 1, 2) and on the freesurfer mailing list (e.g., 1, 2). Despite the error, I am able to use the --shell command and run mri_convert and recon-all on an example .nii image within the container itself.

Thoughts??

Thanks!

2

Okay, this is definitely strange, given that you can use --shell and run it manually.

Could you try running a single subject and --anat-only? e.g.

fmriprep-docker /Raid6/raw/midnight_scan_club \
                /Raid6/raw/users/dan/Documents/Projects/midnight/data/ \
                participant \
                --fs-license-file ~/Documents/licenses/freesurfer/license.txt \
                -w /Raid6/users/dan/Documents/Projects/midnight/fmriprep_working_dir/ \
                --participant-label MSC01 --anat-only

If you could post the entire log as a gist, it might be easier to spot other issues that are slipping by.

Just out of curiosity, why do you need to run as root? If you have sudo, you should be able to set yourself up to run Docker containers from your user. I don’t know that this would cause problems, but if this is a fiddly permissions issue, that might be part of it.

3

Thanks for the reply. I’m running the --anat-only command you provided now and will report back.

Regarding running docker from my user, I’m a little unclear for that step. Are you referring to something along these lines, https://docs.docker.com/install/linux/linux-postinstall/ , or are you referring to running docker with a -u option?

4

Yup, that. As long as your user is allowed to start containers, there’s no real need to put yourself into root. sudo can distort the environment in some non-obvious ways, so when I see something that might be a permissions issue, I get suspicious.

You can also pass -u $UID to fmriprep-docker, which is useful if you don’t have sudo, and don’t like how your output directories are all root-owned. But that can backfire inside the container sometimes, if for some reason the program tries to write to a root-owned file/directory as a non-root user.

5

Okay, so I ran the command you provided ( --anat-only), and it produced the following error:

repositioning pial surface locations using  ../mri/T2.mgz
No such file or directory
mris_make_surfaces: could not load flair volume ../mri/T2.mgz
No such file or directory
Linux d97344b9823e 3.13.0-147-generic #196-Ubuntu SMP Wed May 2 15:51:34 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

recon-all -s sub-MSC01 exited with ERRORS at Fri Feb  8 23:45:16 UTC 2019

For more details, see the log file /out/freesurfer/sub-MSC01/scripts/recon-all.log
To report a problem, see http://surfer.nmr.mgh.harvard.edu/fswiki/BugReporting

Standard error:

Return code: 1

190209-01:35:59,480 nipype.workflow INFO:
	 [Node] Finished "_autorecon30".
190209-01:36:03,410 nipype.workflow ERROR:
	 could not run node: fmriprep_wf.single_subject_MSC01_wf.anat_preproc_wf.surface_recon_wf.autorecon_resume_wf.autorecon3
Preprocessing did not finish successfully. Errors occurred while processing data from participants: MSC01 (1). Check the HTML reports for details.
Sentry is attempting to send 2 pending error messages
Waiting up to 2.0 seconds
Press Ctrl-C to quit
Captured warning (<class 'ResourceWarning'>): unclosed <socket.socket fd=7, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.17.0.3', 58592), raddr=('35.188.42.15', 443)>
fMRIPrep: Please report errors to https://github.com/poldracklab/fmriprep/issues

Not sure how to get at the log file inside the container (that is now exited). Any time I try to re-start the container to re-enter it, fmriprep requires the bids_dir, output_dir, and participant. Not sure how to get around that here… Is there another way I can access the log file specified in the error?

6

Update: I ran the same command as my previous post (--anat-only) with docker properly set up for my user, and I did not get the same error as above. However, I did get:

Sentry is attempting to send 1 pending error messages

Not sure if that’s anything significant. I also noticed there was no output in my output folder (data/). Nevertheless, adjusting docker for my user does make a difference…

7

Thank you for the update! Was there an error message outputted or just that Sentry is attempting to send 1 pending error message?

Interesting there was no data in the output folder. Does it appear fMRIPrep created something (like the folder structure)?

8

May you please send us the full log of processing (the command line readouts) and the crash log files? The crash log files would be found within your specified output folder/fmriprep/sub-<sub_id>/log

With this message - we are not certain there is an actual error

9

No actual error was reported, just the Sentry message. Nothing was created in the output folder (logs included) either. I did save off the working directory however; is there something in there that I can provide?

The command line output can be found here: https://gist.github.com/danjgale/c39afc7a6abf1128311f26ad7abeddcb

10

Don’t you have a /Raid6/raw/users/dan/Documents/Projects/midnight_test/data/fmriprep/sub-MSC01/anat/ folder with the preprocessed anatomical images and a /Raid6/raw/users/dan/Documents/Projects/midnight_test/data/freesurfer/sub-MSC01 with the results of FreeSurfer recon-all?

11

Weirdly, no. /Raid6/raw/users/dan/Documents/Projects/midnight_test/data/ is completely empty. Could this be a docker configuration thing?

12

What about /Raid6/users/dan/Documents/Projects/midnight_test/fmriprep_working_dir is it also empty?

I also noticed the following warning issued by Docker: WARNING: Error loading config file: /Raid6/users/dan/.docker/config.json: stat /Raid6/users/dan/.docker/config.json: permission denied. See https://askubuntu.com/questions/747778/docker-warning-config-json-permission-denied for a potential solution.

13

In fmriprep_working_dir/ I do have stuff, reportlets/ and fmriprep_wf/. Is there anything in there that would be useful to share?

As far as your second point, I caught that yesterday and already made the fix (same info you provided) which didn’t change anything. Thanks for catching that as well.

14

Ok. Some followup questions:

  1. Does the folder /Raid6/raw/users/dan/Documents/Projects/midnight_test/data exist on the host (not inside docker)? Please mind the exact spelling (I noticed you sometimes use /Raid6/raw/users/dan and sometimes /Raid6/users/dan)

  2. Could you rerun this (with clean working directory) but with poldracklab/fmriprep:1.3.0 image and a flag --notrack (passed to FMRIPREP not Docker)

2 Likes
15

Oh geez, this is embarrassing… you’re right, I’ve been erroneously including raw in my output directory but not my working directory. Hence why I get stuff in my working directory but nothing in what was supposed to be my output directory. Thanks for catching that!

/Raid6/raw/users/dan/Documents/Projects/midnight_test/data does exist on the host, and does contain everything – no crash logs, correct output with preprocessed data and all. So, we know now that it was in fact working. Note, I ran this analysis with docker properly configured for my user.

Going back to the original issue that brought me here, I am taking a look at the output directory (/Raid6/raw/users/dan/Documents/Projects/midnight/data, note that it is just midnight). This attempt included the full dataset and I ran this without setting up docker for my user. The original crash logs can be found there, hence my confusion earlier. I can share one if need be.

I’m guessing what was causing the problem was the docker user configuration.

I’ll fix my output path and run the full dataset again. I would expect it to work this time now that the docker issue is solved, and I’ll report back once completed.

2 Likes
16

Hi @danjgale, it would also be great to hear from you when you run fmriprep without the --anat-only flag :+1:

17

@oesteban I’ve been running the full preprocessing on the full dataset and it’s still ongoing (which is expected given the size of the dataset) with no issues yet. Everything seems to be working and some subjects are mostly complete. :tada: I don’t foresee any further issues

2 Likes
18

3 posts were split to a new topic: Memory issues running fMRIPrep on the Midnight Scan Club dataset