(Assuming you are talking about supervised learning)
Correlated features will not always worsen your model, but they will not always improve it either.
There are three main reasons why you would remove correlated features:
- Make the learning algorithm faster
Due to the curse of dimensionality, less features usually mean high improvement in terms of speed.
If speed is not an issue, perhaps don't remove these features right away (see next point)
The keyword being harmful. If you have correlated features but they are also correlated to the target, you want to keep them. You can view features as hints to make a good guess, if you have two hints that are essentially the same, but they are good hints, it may be wise to keep them.
Some algorithms like Naive Bayes actually directly benefit from "positive" correlated features. And others like random forest may indirectly benefit from them.
Imagine having 3 features A, B, and C. A and B are highly correlated to the target and to each other, and C isn't at all. If you sample out of the 3 features, you have 2/3 chance to get a "good" feature, whereas if you remove B for instance, this chance drops to 1/2
Of course, if the features that are correlated are not super informative in the first place, the algorithm may not suffer much.
So moral of the story, removing these features might be necessary due to speed, but remember that you might make your algorithm worse in the process. Also, some algorithms like decision trees have feature selection embedded in them.
A good way to deal with this is to use a wrapper method for feature selection. It will remove redundant features only if they do not contribute directly to the performance. If they are useful like in naive bayes, they will be kept. (Though remember that wrapper methods are expensive and may lead to overfitting)
- Interpretability of your model
If your model needs to be interpretable, you might be forced to make it simpler. Make sure to also remember Occam's razor. If your model is not "that much" worse with less features, then you should probably use less features.