Apparently, to those not all that familiar with it, allAnime and Manga are either geared towards children like Pokémon or Hentai. Which tends to be why fans of anime and manga are Acceptable Hobby Targets. Or more broadly, all anime, as opposed to "cartoons", which anime technically is, is for adults (and thus not appropriate for children), a stigma that exists among both anime fans and mainstream society. It's just the opposite, in fact. Because of cultural and societal differences, what Japan considers "family-friendly" is similar, but not entirely the same as the western definition. Often, those same people use the words "anime" and "manga" interchangeably, or not even realise that the word "manga" exists. (They're used interchangeably in Japan. Japanese people often use the word "manga" in much the same way English speaking people would use the word "cartoon", which refers to both comics, animation, or any other kind of drawing.)
Don't you know? All anime is sci-fi with 50 foot tall neural-interfacing robots.
Many people who aren't big anime fans have only heard of Pokémon, Yu-Gi-Oh!, Dragon Ball Z, Naruto, One Piece, and Sailor Moon. And most of these people believe that Dragon Ball Z refers to the entire Dragon Ball franchise, when it's really just the second of three series, after the original Dragon Ball but before Dragon Ball GT.
Speaking of Pokémon: Even though there are at least 150 basic Pokémon species, the great majority of non-anime fans have only heard of Pikachu (most likely because of his mascot status). Squirtle, Bulbasaur and Charmander might also be mentioned. It's also not widely known that Pokémon can actually mutate, and that Pikachu's "upgraded" form is Raichu, who is orange instead of yellow.
To a number of anime fans, all Japanese "anime" is AKIRA, Ghost in the Shell, Ranma ˝, or any other show, movie or OVA that has graphic violence, swearing (though that's more of an issue with dubbed anime; it's a long story), nudity, and/or other "adult content" that one (usually) wouldn't see in a western "cartoon".
Amongst Japanese viewers, Eiken (not the anime OVA, but the studio) tends to be known more for Sazae-san these days (it IS Japan's longest-running anime TV series). Back when they were called TCJ, they made once show that is still very well remembered in Japan to day: Tetsujin 28-go (AKA: Gigantor).
Aversions and notable exceptions of this trope from Anime and Manga:
Averted in Haruhi Suzumiya, mostly in the novels. We have a story which uses Euler's Planar Graph Formula as a plot device. Jean-Jacques Rousseau is mentioned in another short story extremely casually, and half the historical references are of Japanese history. Best of all, Yuki's books always refer to the current plot, like when she reads Hyperion in Melancholy. Koizumi, especially in Melancholy, peppers his words with philosophy, like the Anthropic Principle and the Omphalos Hypothesis. Even the title sequence for the first season isn't spared. Read up about it in the Genius Bonus and the Viewers Are Geniuses page.
Princess Tutu, an anime based around a Magical Girl Ballerina, smashed this trope. Classical music serves as almost all of the background music in the show, and while a number of famous works are included (for example, Pyotr Ilyich Tchaikovsky's The Nutcracker and Swan Lake), both more obscure composers (Smetana, Mussorgsky, Satie) and less-popular works from famous composers (Beethoven's Egmont Overture). And it features a lot of ballets, from Giselle to the aforementioned Cinderella to Coppelia.
Nodame Cantabile naturally also uses works not by Beethoven & Mozart. The animators love "Veni, creator spiritus" from Mahler's 8th, for example, a fact that escapes the Other Wiki's notice. And Purcell's Abedlazar...
The Gag Dub of Crayon Shin-chan includes references to many obscure things, all the way to making a reference to MOTHER. An interview by one of the writers said they deliberately tried to avoid this.
Hunter × Hunter features cameos and references to well known Japanese celebrities, but also much more obscure ones (one of the sadistic antagonists reading Trevor Brown probably takes the cake).
The fairy tale anthology anime Grimm's Fairy Tale Classics included many obscure fairy tales such as "The Iron Stove" and "Jorinde and Joringel", in addition to well-known ones like "Cinderella" and "Snow White."
Meta example: The Japanese surname "Yagami" is spelt with the kanji for "eight" and "god" — so, 八神, "eight gods"; it turns out to derive from a placename. However, most Western anime fans first encounter it through Light Yagami, who spells it with the kanji for "night god." This has resulted in at least two similarly-named characters on this very wiki being written up with incorrect name meanings of "night god." Whoops.