Exchanging the two qubits swaps the basis states $|01\rangle \leftrightarrow |10\rangle$, but keeps $|00\rangle$ and $|11\rangle$ unchanged. Suppose you have a gate whose action on the computational basis is

$$
|00\rangle \to a|00\rangle \\
|01\rangle \to b|01\rangle \\
|10\rangle \to c|10\rangle \\
|11\rangle \to d|11\rangle.
$$

If you swap the inputs you obtain the gate whose action on the computational basis is

$$
|00\rangle \to a|00\rangle \\
|01\rangle \to \color{red}{c}|01\rangle \\
|10\rangle \to \color{red}{b}|10\rangle \\
|11\rangle \to d|11\rangle.
$$

Thus, all such gates are unchanged under exchange of qubits if and only if $b=c$.

Controlled-$Z$ is just such a gate with $a=b=c=1$ and $d=-1$. In fact, all controlled rotations around the $Z$ axis such as the controlled-$S$ gate have $b=c=1$ and are therefore symmetric under qubit exchange and so we do not generally label their inputs as control and target.

A releted answer.

– Davit Khachatryan – 2020-08-31T17:39:46.7832CZ(control=i,target=j)=CZ(control=j,target=i)...maybe a bit surprising at first...so you can pick either bit for control/target – unknown – 2020-08-31T17:50:06.020