You cannot always find such a Kraus decomposition. Notice that any CPTP map $\mathcal E$ which does have a decomposition as a probabilistic mixture unitaries is *unital*, which is to say that it maps the identity to the identity, and in particular it maps the maximally mixed state to the maximally mixed state:
$$ \mathcal E(\tfrac{1}{d} \mathbf 1) = \tfrac{1}{d} \mathbf 1 . $$
This is true because $U \cdot \mathbf 1 \cdot U^\dagger = \mathbf 1$ for each unitary $U$, and taking a mixture over several unitaries $U$ does not change the result for the operator $\mathbf 1$.
But it is easy to find maps that don't have this property — so for such maps, there is no interpretation as a probabilistic mixture of unitary processes.
For instance, the map $\mathcal R$ that resets a qubit to $\lvert 0 \rangle$ does not preserve the maximally mixed state on a qubit:
$$ \mathcal R(\rho) \,=\, \mathrm{tr}(\rho) \cdot \lvert 0 \rangle\!\langle 0 \rvert . $$
Of course, this map can be expressed using Kraus operators: for instance, you could take $K_0 = \lvert 0 \rangle\!\langle 0 \rvert$ and $K_1 = \lvert 0 \rangle\!\langle 1 \rvert$, which would suffice for $\mathcal R(\rho) = K_0^{\phantom\dagger} \rho K_0^\dagger + K_1^{\phantom\dagger} \rho K_1^\dagger$.

*Edited to add:* Following up on Norbert Schuh's comment below, not all *unital* channels are mixtures of unitaries either. An example for qutrits can be found in John Watrous' *Theory of Quantum Information* (which is freely available online here), in Example 4.3:
$$
\begin{gathered}
\Phi(\rho) = A_1^{\phantom\dagger} \rho A_1^\dagger + A_2^{\phantom\dagger} \rho A_2^\dagger + A_3^{\phantom\dagger} \rho A_3^\dagger,
\\[1ex]
A_1 = \tfrac{1}{\sqrt 2}\mbox{$\scriptstyle\begin{bmatrix}
0 \!&\! 0 \!&\! 0
\\ 0 \!&\! 0 \!&\! 1
\\ 0 \!&\! -1 \!&\! 0
\end{bmatrix}$}\,,
\quad
A_2 = \tfrac{1}{\sqrt 2}\mbox{$\scriptstyle\begin{bmatrix}
0 \!&\! 0 \!&\! 1
\\ 0 \!&\! 0 \!&\! 0
\\ -1 \!&\! 0 \!&\! 0
\end{bmatrix}$}\,,
\quad
A_3 = \tfrac{1}{\sqrt 2}\mbox{$\scriptstyle\begin{bmatrix}
0 \!&\! 1 \!&\! 0
\\ -1 \!&\! 0 \!&\! 0
\\ 0 \!&\! 0 \!&\! 0
\end{bmatrix}$}\,.
\end{gathered}
$$
As Watrous observes, it is possible to show that this is an extreme point among the channels on a qutrit (*i.e.,* not a convex combination of *any* other channels) by showing that the operators $A_j^\dagger A_k^{\phantom\dagger}$ are a linearly independent set of operators; then as $\Phi$ is not itself unitary, it isn't a mixture of unitaries either.

Watrous also notes the following characterisation of channels which *are* mixtures of unitaries:

**Theorem 4.8** (paraphrased) —
Let $\Phi$ be a channel on a Hilbert space $\mathcal X$, characterised by a Stinespring dilation $$\Phi(\rho) = \mathrm{Tr}_{\mathcal Z}(A \rho A^\dagger)$$ for some unitary embedding $A: \mathcal X \to \mathcal X \otimes \mathcal Z$. Then $\Phi$ is a mixture of unitaries if and only if there exists a POVM $\{E_k\}_{k \in K}$ on $\mathcal Z$ with outcomes labeled by elements of some set $K$, and some channels $\Psi_k$ on $\mathcal X$ for $k \in K$, such that
$$ \rho = \sum_{k \in K} \Psi_k\bigl(\tilde \rho_k\bigr), $$
where $\tilde \rho_k = \mathrm{Tr}_{\mathcal Z}\bigl(\bigl[\mathbf 1 \!\otimes\! E_k\bigr]\bigl[A \rho A^\dagger\bigr]\bigr)$.

That is to say: if (and only if) $\Phi$ can be characterised by an interaction of an input system $\mathcal X$ with an environment $\mathcal Z$, in such a way that there is a way in principle that you could measure the environment and use the outcome to control a 'correction' operation to recover the original input, then $\Phi$ is a mixture of unitaries.

It might be worth noting that even for unital channels, this is not always possible. – Norbert Schuch – 2019-08-07T00:42:28.920

@NorbertSchuch Does it says anything about the channel whether this can be found or not? (A part from the obvious "it is not a statistical mixture of unitary evolutions"). It seemed like a very natural idea and it strikes me as if a (unital) quantum channel cannot be represented in such a way, it should imply something is different about that channel – user2723984 – 2019-08-07T09:37:33.397

2@user2723984 This is (was?) known as the "Quantum Birkhoff conjecture". You might want to check the corresponding papers. – Norbert Schuch – 2019-08-07T09:48:13.197