The other answers do an excellent job explaining too adjective to infinitive, so I won't address that. I'll try to explain a different aspect of this, which might be what you're finding so surprising: specifically, how it could possibly be that removing an adverb could render a sentence ungrammatical? As you said, removing "slowly" from "He slowly walked down the street" changes the meaning a little but it doesn't make it ungrammatical. An adverb is just an optional modifier, so what could be wrong with leaving it out?
It's a phrasal thing
The reason removing an adverb can cause ungrammaticality is that in English, often a phrase is the unit of meaning, not the individual words. Sometimes, the meaning of a phrase doesn't derive from combining the meanings of the individual words like when you combine slowly with walked to mean slow walking. The phrase too adjective to infinitive is actually an indivisible unit of meaning. The word to doesn't express a relationship between one part of the sentence and the object of a prepositional phrase the way prepositions usually do. Instead, too and to together supply the distinctive signature of this phrase. If either of those exact words is missing, the phrase is gone. The phrase is itself like a word in the language. It just happens to consist of two words plus two slots to be filled in by other words (the adjective and the infinitive).
This drives people learning English as a foreign language crazy. In most languages, the individual words combine their meanings according to standard rules for combining meanings. English mostly does that, too, but English also makes heavy use of phrasal verbs and similar phrasal templates that are themselves indivisible units of meaning. (Actually, I think nearly all languages do this at least a tiny bit, just not to the extent that English does.)
Sometimes up does not mean up
I once spent most of a plane flight talking with a woman from Brazil. When the plane landed, I asked her, "Do you have someone to pick you up from the airport?" She said, "Why is everything always 'up' in English? Why don't you just say 'pick me from the airport'?"
It's because pick is actually a completely different verb than pick up. To pick someone means to choose them, like when children "pick" who they want on their team before playing a game. To pick someone up from the airport means to give them a ride.
Prepositions often serve as the distinctive word in the signature of phrasal verbs. Here are some more phrasal verbs that include pick:
pick on someone = bully someone, or single someone out for unfair, harsh treatment
pick up after someone = clean up someone else's mess
pick at something = repeatedly scratch or poke something with a sharp object; for example, "pick at a scab"
pick a person up = meet a person you've never met before (not a planned meeting) and go on an impromptu romantic date
pick a call up = answer a telephone call
In phrasal verbs, one of the words is always a preposition. But it doesn't function like a preposition. The up in pick up doesn't work like up in walk up the stairs, where up introduces a prepositional phrase that modifies walk. In pick up, the word up changes pick into a completely different verb! Some linguists call up a particle rather than a preposition when it plays this role, since it doesn't introduce a prepositional phrase.
It's got to be too
Here is a consequence of all this that might be surprising: you can't replace too with a synonym. "I'm excessively tired to drive" is also ungrammatical! The problem isn't that an adverb is missing from I'm tired to drive, it's that specifically the word too is missing.
It's just like pick up: you can't replace up with a synonym or approximation like above or upward without radically changing the meaning. Pick you above from the airport is actually ungrammatical. There's no phrasal verb pick above.
When I hear I'm tired to drive, I feel disoriented. The word tired seems to be leading somewhere, but then I can't recognize a familiar phrase that starts with tired, like tired of gerund. I also can't find a way to join to into a familiar phrase with anything earlier in the sentence. Is it like in order to (for the purpose of)? No, because nobody would get tired in order to drive. Is it like toward? No, because to drive is not a prepositional phrase. It's a to-infinitive. The word to doesn't mean a direction here.
The sentence is ungrammatical because the parts don't join together. No familiar phrase starting with tired concludes in to infinitive. And no familiar kind of phrase ending in to infinitive starts with tired and gets a reasonable meaning. Normal, one-word-at-a-time combining of meanings doesn't work, either.
Related: Why does “I was happy to do my homework” work, but “I was tired to do my homework” doesn't?
– CowperKettle – 2016-12-16T14:25:43.510@F.E., re your first comment -- specifically 'The infinitival "to drive" is licensed by the adverb "too"' -- Is this a special case (on accout of the construction "too X to Y" being a fixed phrase), or do adverbs (and any other parts of speech?) commonly license various types of complements? I'm asking because I have only heard of verbs and adjectives licensing all types of complements in English. – HeWhoMustBeNamed – 2020-02-16T13:19:19.780
7The infinitival "to drive" is licensed by the adverb "too". I can be tired or sleepy and still be able to drive. But if the amount of tiredness or amount of sleepiness becomes a lot, i.e. too much, so that I won't be able to drive, then I am *too* tired or *too* sleepy to be able to drive. That is, I cannot drive, and the reason why is that I'm too tired or too sleepy. The adverb "too" is necessary to allow the presence of the infinitival "to drive" in that sentence. – F.E. – 2014-12-11T07:50:16.563
1In your 2nd link is this: *Laying out his targets for next year, he said: "To win. I am unhappy because this year we have not got since the beginning a competitive car. I am tired to lose the championship at the last race; it has happened too many times in the last years. "You remember what happened to Felipe in 2008, not 20 years ago, what happened to Fernando last year in Abu Dhabi. . . . So I hope that we can have clear rules and to make a condition to win again the championship now."* – F.E. – 2014-12-11T08:06:01.000
2Your 1st link seems to be Indian English, and in it is: *I am blessed that I can sleep at any time for any duration of time. . . . My wife keeps telling me someday I will collapse, but I don't feel physically tired. Sometimes, I tell them I am tired to please them, though I am actually not.* – F.E. – 2014-12-11T08:12:07.497
1And for that example, the adverb "too" should be in there, so that it should read: "*Sometimes, I tell them I am too tired to please them, though I am actually not."* For he is telling his family that he cannot please them because he is too tired to be able to (though he is actually lying when he says that). – F.E. – 2014-12-11T08:20:42.027
Luca DiM's comment really should read "I am tired of losing the championship…" He is not renowned for his English grammar. – gone fishin' again. – 2014-12-11T11:58:49.137
3@F.E.: No, the word "too" does not belong in the 2nd one. From the context, its meaning seems to be Sometimes, I tell them [that] I am tired [, telling them this fiction in order] to please them. – shoover – 2014-12-11T16:04:45.603
1(Native AmE speaker here) When I see the phrase "I am tired to drive", at first it doesn't suggest a meaning to me. After I think about it a little, I think maybe it means "I am tired in order to drive" by analogy with "I am well-rested in order to drive" or "I am on amphetamines in order to drive" or maybe "I am licensed to drive". But of course, you can't get tired in order to drive, unless maybe you want to fall asleep at the wheel. I am able to find a conceivable meaning for "tired to drive", but I have to stretch reasonable semantics very far. – Ben Kovitz – 2014-12-11T18:14:39.900
@Maulik V Can you explain more of what you intend the phrase "tired to drive" to mean? – Ben Kovitz – 2014-12-11T18:17:30.870
1@shoover Er, no. The word "too" does need to be in there for the sentence to have the intended meaning. – F.E. – 2014-12-11T18:20:29.940
3@F.E. How do you figure? What is the meaning you think it has? It certainly should not have "too" for the meaning shoover mentions; in that reading, "to please them" doesn't attach to "I am tired" at all (the reading is "I tell them I am tired to please them", not "I tell them I am tired to please them"). – cpast – 2014-12-11T21:14:16.707
Supporting that reading, he says it in the context "other people think I'm tired even though I'm not; they think I'm putting on an act and pretending I'm not getting old. Sometimes I just tell them I'm tired (even though it's not true) to stop them badgering me about how I shouldn't pretend I don't get tired." – cpast – 2014-12-11T21:17:23.147
1@BenKovitz Sir, I just wondered why simply removing 'too' makes it ungrammatical. Especially when I want to say that I was *not* very much tired. I just don't want to emphasize by putting too over there. But then it turned out to be ungrammatical. Take any sentence removing adverb does not make it ungrammatical especially in such case. He slowly walked down the street over He walked down the street -I just removed adverb because I don't want to describe it! But in this case, it simply got invalid! :) – Maulik V – 2014-12-12T04:25:54.160
@MaulikV Ah, now I think I understand what you're getting at. I think I'll need more than 600 characters, so I'll post an answer. – Ben Kovitz – 2014-12-12T05:23:45.517
5The only sense I can make out of "I am tired to drive" is if the driver has just gotten new tires on his vehicle- his vehicle and by extension, he is now tired and ready to drive. – Jim – 2014-12-12T08:23:38.597
4@Jim And if the driver had just gotten two tires, he'd be two tired to drive? – Jim Reynolds – 2014-12-12T11:08:36.573
@MaulikV I just took the liberty of revising your question to include your point about removing adverbs and de-emphasize everything else. I'll bet that a lot of EFL learners find this baffling. It's hard to even think of how to phrase the question. Hopefully people who've reached similar frustrations will find this page on Google. – Ben Kovitz – 2014-12-14T18:36:52.707