I have tried using different dofs in the flirt, different applyxfm, using the MNI as ref in step 7, with and without the robustfov and so and so but nothing seems to work.
What would be interesting would be to see at which step the command fails to achieve its goal: Has the T1w normalization to MNI worked well? Is the brain mask well extracted? Is step 4 ok? What about step 5? Is the PET image well registered to the T1w image?
My guess is that it is the step 7 that fails in your case.
One reason would be that the PET image is to far from the T1w and the minimisation falls into a local minima which is not the correct solution
One other reason would be that the pet image contrast is very different than the MRI T1w contrast and you will need to choose the appropriate cost function to account for that. The default of flirt is corratio , perhaps mutual info would work better?
A recommended strategy would be to use “betted” images of both the MNI and the T1w image in the initialization process with flirt, and then the full versions of T1w and MNI images are used for the fnirt command. (see what is written in the --aff section of the page linked above)
If you have CT to go along with the PET, then you will most likely get better results. If you also have a subject-specific MRI, that would be useful too. If you only have this Tau PET image, then an NMI based alignment cost might work well. If the brain could be roughly extracted first, that might help too.