The main two problems are security-related.
Auditability (that is, the ability to recount votes when results get challenged) is the most important problem by far for elections that matter.
With physical ballots, you can always recount the results; and, in theory at least, the ballots get stored for years. With electronic votes, any amount of tampering could theoretically happen to the votes. Or, as happened recently in the US, the votes could get deleted.
You can use techs like blockchain to reduce the risk of tampering with individual bulletins, but even that offers little protection if bulletins get tampered with wholesale - e.g. on the fly as they are cast, or by overwriting all votes before the counting.
In practice the only real protection is printing the electronic ballot and having the voter vet and physically cast it. You can then use the electronic votes to compute the results. And if the result gets challenged, you can always resort to counting the physical ballots manually.
Anonymity has multiple underlying problems that relate to preventing people from voting more than once. The latter is easy enough in theory: maintain a list of people who already voted, and prevent them from voting twice.
In practice it's less simple. At the physical level, timestamps can get in the way of perfect anonymity if the user-related tokens themselves aren't anonymous as well - but then you also need to figure out how to securely and anonymously deliver the token.
Also, you might want to allow voters to be able to change their votes just in case. Because they might want to correct a mistake, for instance. Or, like in Estonia, because you might want a voter to be able to verify that what they've voted is what's registered in the system.
In theory you could use a cryptographically secure one-way hashing algorithm to guarantee anonymity here, but that still leaves the door open to tampering - e.g. by hacking voting devices so it confirms you've voted on X while actually casting a vote on Y.
Note that, in practice, physical votes aren't a panacea either. Horror stories include:
- Ballot stuffing, e.g. when a ballot box is left behind during a (fake) fire emergency.
- Ballot theft, e.g. by intercepting votes sent by snail mail, by making a ballot box disappear, etc.
- Irregular votes, e.g. when someone votes for you using a fake ID (or indeed no ID at all, in countries where there's no such a thing), or when dead voters are mysteriously casting votes.
It's arguably difficult to scale any of this when physical ballots are involved, but the point here is that physical ballots aren't a silver bullet. Electronic voting, for all its problems, is not without merits in this sense.
As one who worked a bit in the field, I expect that what will triumph in the end is a combination of the two as hinted further up: electronic votes backed with physical ballots. It gives you the best of both worlds: quick counts and auditability. (It might simply take the shape of a sophisticated enough ballot counting machine.)