The spectrum of a Gelfand pair $(K⋉N,K)$, where $N$ is a nilpotent group, can be embedded in a Euclidean space. We prove that in general, the Schwartz functions on the spectrum are the Gelfand transforms of Schwartz $K$-invariant functions on $N$. We also show the converse in the case of the Gelfand pair $(SO\left(3\right)⋉N_{3,2},SO\left(3\right))$, where $N_{3,2}$ is the free two-step nilpotent Lie group with three generators. This extends recent results for the Heisenberg group.