For anyone who uses GE scanner and has such direct B0 fieldmap, and meets this same problem.
I did more tests yesterday and it seems it’s a matter of orientation and its relation to phase encoding direction. I have some functional data with A>P phase encoding, and some functional data with L>R, all converted in the LAS orientation (default orientation from dcm2niix for my data). If I just flip the sign of the spiral fieldmap (in LPS orientation), very interestingly, images after correction only look good for images with A>P phase encoding, while images after correction look worse for images with L>R phase encoding.
So what happened? According to FSL TOPUP, the phase encoding direction (j- or j or y- or y) is relative to the image orientation (LPS or LAS in our case), and given the B0 offset is calculated in the LPS coordinates – unfortunately, neuroimaging packages do not check this, so, if our fmri data is converted in an opposite direction in one axis comparing to the spiral fieldmap and the axis happens to be the phase encoding axis, we need to flip the sign of the fieldmap (the A>P phase encoding images in this case). Otherwise, we don’t need to if the opposite axis does not happen to be in the phase encoding direction (the L>R phase encoding images).
Now to solve the problem for all functional images using this spiral map, I just re-orient them into LPS and the results now look correct to me.