I think part of the answer is that these phrases aren't just filler. They resemble other filler words, like "um" or "uh", in that people say them unconsciously. But "sort of/kind of" seem to carry some real meaning, too. They serve as hedge phrases, to make people sound less blunt. In this sense, they're related to the "like" of teen-speak, which really, like, drives the grownups crazy*. More than that, though, these phrases seem to be unconscious affectations or signals, which say "I am speaking the way thoughtful, educated people do". "Sort of" and "kind of" serve as indicators that the speaker is part of a certain subculture (more or less the NPR-listening, college-educated, self-consciously-cultured subculture), and can speak the way their peers in that subculture do.
Which is really sort of mildly annoying to me. I'm very attuned, for some reason, to people's unconscious badges that mark their subcultural identity. I notice how skateboarders walk differently than hipsters, and hipsters smoke their cigarettes differently than construction workers. But when the subculture I'm observing is one that I'm (loosely) associated with, like that bookish, NPR-listening crowd, those subcultural badges irritate me. I'm part of that subculture that tries not to be part of a subculture (we're probably very predictable). I know I probably wear my share of these badges, but I do think there's a good reason to notice when you're unconsciously mimicking your peers. As soon as you start automatically speaking and dressing like a particular group of people, you run the risk of automatically thinking like them. And if you're automatically thinking like any group of people, well, that really sort of means you're not thinking at all. It's not that I think people should immediately cease and desist with the "kind of" and "sort of". It's just that it wouldn't be a bad idea for them to notice it, and stop to think about what else they're doing, and thinking, without really thinking about it.
* Wikipedia says that people have been using "like" this way for a long time, but it became popular with the beatnik character Maynard G. Krebs, on the TV show Dobie Gillis in the early '60's. Of course, we all know Shaggy said it a lot on Scooby Doo. So, like, people that are grandparents now used to talk like this. In fact, some of them still do. Wikipedia also says words like "like", "um", "uh", "well", etc, are called discourse markers or discourse particles. Apparently, linguists used to think of these things as mostly being filler, but now see them as carrying real meaning. I think I'll look into this some more, once I find some readable sources on this stuff beyond Wikipedia. Here's a couple of interesting ones:
