First of all, part of the premise of your question is mistaken:
In other countries like the USA, that weren't responsible for the Nazi crimes, Nazi symbols are not banned as well, but there they are talking about banning the southern states flag.
The First Amendment to the US Constitution gives very strong protection to freedom of speech. For the purposes of this discussion, displaying a flag or banner is a form of "speech". So the USA will not place a general ban on displaying a Confederate flag, swastika, or any other political symbol.
More generally, this comes down to a distinction between democratic and authoritarian forms of government.
A democratic government will typically allow all forms of political speech, unless there are specific reasons for them to be restricted. In some cases, such as the USA, there is little or no restriction. Other governments may be more restrictive, most often if speech is associated with violence. Examples include:
Any such restrictions will be specific to that country's circumstances; so usually the ban relates to a violent political movement which is or has been active in that country.
An authoritarian government is likely to disallow all forms of political speech, except those which are specifically permitted. In many countries, displaying any kind of political symbol without prior permission is likely to get you in serious trouble. Where the symbol originated from would be a secondary issue.