FreeSurfer-generated GIFTIs have two affines. One is the GIFTI-specified one, and the other is a GIFTI-metadata encoding of the FreeSurfer volume geometry (VolGeom{X,Y,Z,C}_{R,A,S} off of memory; this might not be exactly right). FreeSurfer knows to apply the VolGeom affine, pretty much nobody else does.
You can provide the --to-scanner option to mris_convert to have FreeSurfer update the coordinates with the VolGeom, but this will also leave in the VolGeom affine, so everybody but FreeSurfer will handle the file correctly.
The way to get a generic GIFTI from a FreeSurfer surface appears to be to use mris_convert --to-scanner and then remove the VolGeom metadata from the file yourself.
I just read your gist, @alexisthual. You’re doing the same thing that HCP Pipelines and fMRIPrep have done, which is just applying the translation encoded in C_RAS. If the image is oblique, you can get a non-identity XYZ_RAS matrix, in which case this will fail.
FWIW as of fMRIPrep 22.0, we now use the method I describe above: FIX: Use `mris_convert --to-scanner`, and update normalization step by effigies · Pull Request #295 · nipreps/smriprep · GitHub