Here is an alternate opinion for you that you can consider.
He drives not so much fast, as he drives reckless.
He drives not as much fast as reckless.
He drives quickly, not as much as recklessly.
Though it is not an absolute rule, I think what is affecting me is that frequently the -ly adverb follows the verb and not a quantifier so for me it is just awkward. However, others feel differently than I do.
When I think of driving quickly, I think of someone getting into a care and putting it into motion as soon as possible, which could be a slow motion. Quickly usually implies doing something in a short time.
I ate quickly, so I could leave for the movies. I am not sure that driving quickly is related to being reckless. Driving fast may or may not be wreckless.