From a geometric point of view, if all your data are unitary, $\forall x, ||x||^2 = \langle x,x \rangle = 1$, then the scalar product of two vectors defines an angle $\phi$, $\langle x,y \rangle = \cos \phi$, and you have a distance $\phi = \arccos \langle x,y \rangle$.

Visually, all your data live on a unit sphere. Using a dot product as a distance will give you a chordal distance, but if you use this cosine distance, it corresponds to the length of the path between the two points on the sphere. That means, if you want an average of the two points, you should take the point in-between on this path (geodesic) rather than the mid-point obtained from the 'arithmetic average/dot product/euclidean geometry' since this point does not live on the sphere (hence essentially not the same object)!

Note that neither of these are proper distance metrics, even if you transform them to be a value that is small when points are "similar". It may or may not matter for your use case. – Sean Owen – 2014-07-18T11:34:09.417