The Pauli Z gate inverts the phase of $\lvert1\rangle $ while leaving $\lvert0\rangle$ unaffected.
When I think about how $\lvert1\rangle $ and $\lvert0\rangle$ are physically realized, however, as in the Physical Implementations section here, there tends to be a physical symmetry between the physical realizations of the two.
It would seem, then, that a Z gate would need to be realized as some experiment which selectively delays the tuning of only spin-up electrons or nuclei, for example. Given the physical symmetry I expect between physical realizations of $\lvert1\rangle$ and $\lvert0\rangle$, this seems counterintuitive.
Is the fact that a physical $\lvert0\rangle$ cannot have phase delay, and a physical $\lvert1\rangle$ can, a matter of convention? If not, why can't I describe a single $\lvert0\rangle$ in a multi-qubit system as phase delayed relative to the others? E.g., $(I \otimes Z)\lvert10\rangle$ seems to be a perfectly reasonable experiment that delays my right qubit relative to my left, why is only the converse experiment allowed?
Is it, in fact, the case that the $Z$ gate is implemented as an experiment that only affects, e.g., spin-up nuclei but not spin-down nuclei? As background, I'd find an overview of how phase is physically implemented very helpful.