Using surface space images for nilearn's Ward parcellation

We have been successfully using Nilearn’s Ward parcellation function for our volumetric data. Now we want to try parcellations with surface space images. We have images rendered in surface space via freesurfer (created en route during the standard fmriprep preprocessing pipeline). It looks like you can input a spatial connectivity sparse matrix in scikit-learn’s version of parcellation (sklearn.cluster.Ward), which nilearn’s version is based on. However, I am stumped for a straight-forward way to get this sparse matrix of neighboring connections. I found some files that define the coordinate points in surface space, but nothing that defines which nodes are contiguous. Does such a file exist or any ideas on how to produce this? My only idea so far is to calculate the distance between coordinate points as a proxy for contiguity, but I am hoping there might be a more direct method.