I guess it depends on what you want to do with it. If for example you're using those 20 bytes as keys in a lookup table, then you should be fine as long as you remember you might get an occasional false positive so you always double check if it really is the transaction you were looking for. For that purpose less than 20 bytes is probably already enough.
If 20 bytes is everything that you store while throwing away everything else, and you make important decisions based only on those 20 bytes, then you might run into problems at some point.
Remember that it's not mere chance. If your application plays an important role and an attacker knows about your 20 bytes limitation, they might try to forge 2 conflicting transactions IDs to and crash your system or scam you out of some money.