First things first, theoretically infinite dimensional Hilbert space is natural in quantum mechanics due to the relation $[x,p]=ih$. This produces the Heisenberg algebra. This algebra is solvable and then invoking the Lie-Kolchin theorem, if the dimension of representation is finite, then it is necessarily one dimensional which is not useful as it would make everything commute and nothing would happen. So we consider the other alternative which is that our Hilbert space is infinite dimensional.
Fair enough, the mathematical ground is solid. But in reality, since we use only a finite space and work for finitely small time scales, we generally approximate the operators in our infinite Hilbert space to a finite subspace of it. Classic examples are this
The spectrum and eigenstates of any field quadrature operator restricted to a finite
number N of photons are studied, in terms of the Hermite polynomials. By (naturally)
defining approximate eigenstates, which represent highly localized wavefunctions with up
to N photons, one can arrive at an appropriate notion of limit for the spectrum of the
quadrature as N goes to infinity, in the sense that the limit coincides with the spectrum of
the infinite-dimensional quadrature operator. In particular, this notion allows the spectra
of truncated phase operators to tend to the complete unit circle, as one would expect. A
regular structure for the zeros of the Christoffel-Darboux kernel is also shown.
We present several new techniques for approximating spectra of linear operators (not necessarily bounded) on an infinite-dimensional, separable Hilbert space. Our approach is to take well-known techniques from finite-dimensional matrix analysis and show how they can be generalized to an infinite-dimensional setting to provide approximations of spectra of elements in a large class of operators. We conclude by proposing a solution to the general problem of approximating the spectrum of an arbitrary bounded operator by introducing the n-pseudospectrum and argue how that can be used as an approximation to the spectrum.
But does that mean that infinite Hilbert spaces are relegated to the corners of theoretical physics? The answer is, not surprisingly, no. The experimenters did find application of this, and is technically known as continuous-variable quantum information.
Quoting Wikipedia :
One approach to implementing continuous-variable quantum information protocols in the laboratory is through the techniques of quantum optics. By modeling each mode of the electromagnetic field as a quantum harmonic oscillator with its associated creation and annihilation operators, one defines a canonically conjugate pair of variables for each mode, the so-called "quadratures", which play the role of position and momentum observables. These observables establish a phase space on which Wigner quasiprobability distributions can be defined. Quantum measurements on such a system can be performed using homodyne and heterodyne detectors.
But remember what we had mentioned in the beginning, about why theoretically our Hilbert space needs to be infinite dimensional? Well, even that exact formulation has been proposed as the basis of a quantum computer.
Another proposal is to modify the ion-trap quantum computer: instead of storing a single qubit in the internal energy levels of an ion, one could in principle use the position and momentum of the ion as continuous quantum variables.
And all these have been formalised. For example, one can look here:
The science of quantum information has arisen over the last two decades centered on the manipulation of individual quanta of information, known as quantum bits or qubits. Quantum computers, quantum cryptography and
quantum teleportation are among the most celebrated ideas that have emerged from this new field. It was realized
later on that using continuous-variable quantum information carriers, instead of qubits, constitutes an extremely
powerful alternative approach to quantum information processing. This review focuses on continuous-variable
quantum information processes that rely on any combination of Gaussian states, Gaussian operations, and Gaussian measurements. Interestingly, such a restriction to the Gaussian realm comes with various benefits, since on
the theoretical side, simple analytical tools are available and, on the experimental side, optical components effecting Gaussian processes are readily available in the laboratory. Yet, Gaussian quantum information processing
opens the way to a wide variety of tasks and applications, including quantum communication, quantum cryptography, quantum computation, quantum teleportation, and quantum state and channel discrimination. This review
reports on the state of the art in this field, ranging from the basic theoretical tools and landmark experimental
realizations to the most recent successful developments.
Quantum error-correcting codes are constructed that embed a finite-dimensional code space in the infinite-dimensional
Hilbert space of a system described by continuous quantum
variables. These codes exploit the noncommutative geometr
of phase space to protect against errors that shift the value
of the canonical variables
p. In the setting of quantum
optics, fault-tolerant universal quantum computation can be
executed on the protected code subspace using linear optical operations, squeezing, homodyne detection, and photon
counting; however, nonlinear mode coupling is required for
the preparation of the encoded states. Finite-dimensional
versions of these codes can be constructed that protect encoded quantum information against shifts in the amplitude
or phase of a
d-state system. Continuous-variable codes can
be invoked to establish lower bounds on the quantum capacity
of Gaussian quantum channels.
and finally here:
We consider the quantum processor based on a chain of trapped ions to propose an architecture wherein the motional
degrees of freedom of trapped ions (position and momentum) could be exploited as the computational Hilbert space. We
adopt a continuous-variables approach to develop a toolbox of quantum operations to manipulate one or two vibrational
modes at a time. Together with the intrinsic non-linearity of the qubit degree of freedom, employed to mediate the
interaction between modes, arbitrary manipulation and readout of the ionic wave function could be achieved.
Thus, there are several proposals for using infinite dimensional Hilbert spaces, and the same is used in optics sometimes. These are just a few ways in which this can be utilised, and more are expected to open as research progresses.