How does a rectilinear filter used for QKD-BB84 detect both horizontal and vertical polarizations?

2

When using the BB84 algorithm for QKD, you arbitrarily choose which of two filters (bases) to use when detecting a photon: rectilinear or diagonal. If you choose rectilinear, you can detect polarization with certainty if the photon is rectilinear, i.e., either vertically- or horizontally-polarized.

But how? A polarization filter that allows transmission of vertically-polarized photons will block horizontally-polarized photons and vice-versa, no? Perhaps you use a vertically-polarized filter as the rectilinear filter and then the complete lack of a photon indicates horizontal polarization, but how would you distinguish that from, say, absorption of the photon into the fiber?