Technically a Smile is when the narration (or narrator, in the case of first-person) notes that the mere fact of baring one's teeth doesn't necessarily make a "smile", in the sense of an expression designed to convey friendliness, happiness or harmlessness.
Humans are one of the few creatures to bare their teeth as anything other than a threat; this trope comes into play during those times when a "smile" conveys something more... animalistic. Look for the phrase "didn't reach his/her eyes" to appear.
Obviously, this is far more common in print media. In comics, film and TV, the expression doesn't have to be described because the audience can see it right there.
This is probably the trope that inspired a classic series of humorous posters a few decades back (often seen in libraries and such) showing scowling animals, each one with the caption "I am smiling."
Related to the Cheshire Cat Grin, Slasher Smile, The Unsmile and Psychotic Smirk.
Kiyo in Zatch Bell! frequently smiles when he's not at all happy with a situation; it's more of an indication of altogether irritation and/or embarrassment, and is usually accompanied with Mind-Control Eyes. Other characters, like Folgore and the mangaka himself, are almost always drawn with that kind of smile, though.
In Bleach, Kensei trying to comfort Hisagi in the Turn Back The Pendulum arc.
Balalaika does this occasionally. It is on those occasions that the audience is reminded that, despite her affability, she's still a ruthless mass murderer.
It is noted about Sesshoumaru from InuYasha that if he ever smiles, it is generally not a good thing.
The Millennium Earl from D.Gray-Man always smiles, which is creepy enough. When he gets really, really angry, his smile gets very frightening.
There is also this other parody of the same moment. Notice the AT (Absolute Terror) Field.
Sunako Nakahara from The Wallflower, when she is asked to smile, smiles like this.
Izaya Orihara of Durarara!!. It is noted occasionally in the graphic novel that his smile 'doesn't reach his eyes.'
When Akane first meets Shizuo, she gives a smile that's compared to the sort a child would make while stomping on ants. Then she attempts to kill him.
When Ren of Skip Beat! gets angry, he doesn't shout or frown. Instead, he smiles. He smiles so hard that the Bishie Sparkle can blind onlookers. Kyoko and Yashiro find this to be incredibly terrifying.
Once when Garfield was on a diet and he was grumpy about it, Jon kept urging him to smile. Garfield finally agreed...and grabbed the corners of his mouth and twisted them upward into a "smile", while still flashing his "angry eyes."
Another had fun with this trope, when Garfield demonstrated two smiles; one with rectangular teeth, and one with triangular, serrated teeth. Naturally, Jon becomes noticeably alarmed at the second one.
In an issue of G.I. Joe Special Missions, the Joes go to a rural house to rescue a family from terrorists. They see one guy who always has a crazy-looking grin on his face. They initially assume that he is one of the terrorists. Turns out he's one of the hostages. He has a rictus, a facial spasm that causes a permanent teeth-baring grin.
If The Joker's grin isn't depicted (in comics or animation) as being permanently stuck on his face, it's usually depicted as this trope.
In Disney's Beauty and the Beast, Lumiere attempts to coach the Beast in order to help him win Belle over, "Give her a dashing, debonair smile. Come, come, show me the smile." The Beast tries, but the result mostly involves baring a mouthful of very sharp fangs and is not remotely dashing or debonair.
A scene late in The Jungle Book has Mowgli running away in disgust after both Bagheera and Baloo tell him it's time to leave the jungle. He eventually finds himself wandering through a bleak wasteland that's dangerously close to Shere Khan's territory. A quartet of vultures who look a lot like The Beatles (and sound like them, too) befriend Mowgli and try to cheer him up - but Mowgli is so miserable that when one of the vultures says: "Give us a smile," all he can do is halfheartedly curl up one corner of his mouth.
In The Pagemaster, Horror "smiles" to show he's actually very friendly. Unfortunately, he has bad teeth, thin lips, and an ugly face to boot, so it comes off as frightening/hideous anyway.
In Beverly Hills Chihuahua 2, Pedro the pitbull makes a snarling-like face that many say is a smile. Many characters in the film either show disgust, fright, or exasperation towards this, especially since this got him kicked out of the dog show. Later on, he uses it against robbers, saying "This ISN'T a smile!"
Invoked verbatim when describing Duv Galeni when pushed to his breaking point in the Vorkosigan Saga.
Seems to occur about once a chapter throughout The Wheel of Time series, with the phrase "didn't reach his/her eyes" attached about 80% of the time. In the latest book, one of the Wise Ones makes a comment about Aes Sedai and smiles to show she's joking. The narration notes that she only succeeds in baring her teeth.
The Discworld books have this from time to time, usually with werewolves and vampires, but sometimes with Vetinari, and once even with Vimes.
"Vimes' smile was as funny as the one that moves very fast towards drowning sailors. And has a fin on top."
Rincewind does it at least once too— in his case, it's a rictus grin of sheer terror.
In White Night, Marcone is described as making a facial expression where the corners of his mouth turn up and he shows his teeth, but doesn't smile at all.
Harry himself is also prone to a rather shark-like Slasher Smile when he's about to set someone on fire...
Kincaid is prone to this as well. At one point he's described as having "the kind of smile that would make dogs break into nervous howls."
Anne McCaffrey's Acorna Series eventually has the titular Petting ZooUnicorn reunite with her species. There's a hiccup in the road because she, having been raised by humans, will show big toothy grins which are threat displays to the unicorn people. It's really the only notable instance of culture shock.
"[Angleton] smiles, and despite all the horrible sights I've seen today so far, I hope most of all that I never see it again before the day I die."
In Isaac Asimov's The Caves of Steel, human Lije Baley asks humanoid robot Daneel Olivaw, "Can you smile?" Daneel can, technically, but it doesn't reach his eyes and Baley finds it more unnerving than reassuring.
Perhaps it was that he smiled slightly too broadly and gave people the unnerving impression that he was about to go for their neck.
X-Wing Series: Wedge Antilles sometimes wears a smile that's pure predator. As he gets older it becomes even more so - in Legacy of the Force he once flashes a smile at someone "like he was a rancor, and they were made of meat."
A good deal of the works in Star Wars Expanded Universe, particularly that which took place during her political career, depict some variant of the sentence "Leia grinned, showing teeth," giving the notion that the gesture is not particularly friendly.
In Johannes Cabal the Necromancer, the title character is described at one point as a "tallish, blond pale man who only ever seemed to deploy a smile as an offensive weapon."
Quite commonplace in the Honor Harrington universe, with Exact Words eventually showing up when things get really nasty. See the quotes page.
"When most people smile their lips go side-to-side. This lady's went up-and-down, showing lots of her teeth and gums."
In Animorphs: When some Jerk Jocks make the mistake of picking on Marco while he's having a rough week, he loses his temper so badly that he starts going into the preliminary stages of shark morph. Since he's in a pool at the time, the only part of the morph that they can really see is his new mouth full of very sharp teeth. Put that together with the way he starts staring at one guy's jugular vein, and they understandably back off.
Live Action TV
Michael in Burn Notice often laughs and smiles, but rarely conveys actual warmth, having great difficulty showing emotion thanks to his abusive childhood and training as a spy. Notably, his fake smiles when dealing with his family are much more obvious than the ones he uses to deal with criminals, due to their somewhat strained relationship. Some of his antagonists have similar mannerisms. Take a good look at Victor's smiles sometime; he looks like he wants to bite out your throat.
Hilary Briss is always doing this on The League of Gentlemen. The bottom half of his face is split in an enormous Slasher Smile, but his eyes remain the same. Actor Mark Gatiss must have spent a lot of time in front of the mirror practicing, because it's creepy.
Buffy the Vampire Slayer: Buffy meets her mother's surprise romantic interest Ted, and says she's okay with it with a pasted-on smile - Gilligan Cut to her whaling on a vampire viciously enough to make Giles cringe.
In one CSI: Crime Scene Investigation episode, Sara Sidle is investigating a boxing death and has to examine the dead boxer's spit bucket which is full of bloody saliva. Saliva really grosses her out, but when Warrick Brown notices she's smiling, Sara explains that smiling suppresses the gag reflex.
In Xena's second appearance on Hercules: The Legendary Journeys, while she's still a cold-blooded warlord leading a band of marauders, Salmoneus briefly takes over as her P.R. manager and suggests that she smile more. For a brief moment, you can see her trying it as a smile passes over her face. It looks very out of place.
TNA Impact had a genuinely terrifying example in 2011, when Kurt Angle was forced to serve as the best man at the wedding of his ex-wife Karen and his archenemy Jeff Jarrett. After smashing Karen's face in the wedding cake just before the ceremony, he appeared to have gotten all the anger out of his system and came to the ring for the ceremony with a big smile on his face - only it was obviously faked, and seemed to be creepily frozen on his face. Once the ceremony had concluded and Karen and Jeff had left the ring, Kurt's rage erupted again and he destroyed the entire set erected for the wedding. (Incidentally, this was one of the rare times in Professional Wrestling history where the violence occurred after the newlyweds had left the ring.)
In Final Fantasy X, Kimahri attempts to smile once. Tidus' response is "Sad."
In Fire Emblem 7, Raven attempts to smile during one of his support conversations. It hurts his face to do so.
Civilization IV: In the event that Montezuma is furious with you (a very common occurrence unless you happen to follow the same religion he does), he'll spend his diplomatic meetings baring a grin that could crack a sink, teeth grinding the whole way until he loses patience and screams in frustration.
This was implied in Sonic Unleashed where one of the NPCs asked Sonic why he doesn't smile in his Werehog form and wants him to do it. Sonic does so (albeit offscreen) and the NPC was startled by the discovery, regretting that he asked.
There are about three instances where Atsuki Saijo smiles in Lux-Pain. The first is when he fights with Mika over the strawberry shortcake. The second one is forced on him by Akira in archive 1. The third one, which is the most disturbing of them all is Atsuki loses it when Edward calls him a hypocrite. Not only does he smile, but he laughs similar to Light...
Inverted in Freefall, where Florence (an anthropomorphic wolf) has to remember not to show her fangs when she smiles in case she scares people off.
Although she has discovered that many arguments can be averted by calmly stating her case and then flashing a big, toothy smile.
Her smile looks forced because itis. The same as with the Werewolf example above.
It's worth noting that even the producers and writers thought it was probably more horrifying than simply killing someone with a gunshot.
Octus gives his best go at a full grin on Sym-Bionic Titan. It's quite off, but he really does mean it.
How the Grinch Stole Christmas!: The Grinch smiles in a sinister fashion when he gets his "wonderful, awful idea" to impersonate Santa Claus and steal Christmas.
In one episode of Hey Arnold!, Arnold is forced to pair up with Willie the Jolly Olly Man, an ice-cream man who happens to be an intense Child Hater and a Talkative Loon for Career Day. To help him keep his job, Arnold tries to make him smile to appear friendlier to his young customers, but his grimacing teeth-baring grin just sends the kids running off in terror. By the end, he learns to do a proper smile.
The prehistoric episode of SpongeBob SquarePants has Patar give what might be a smile, but just looks disturbing.
We humans are the only ones who turned smiling into an expression of joy. If a chimpanzee (and possibly most apes) grins at you, it is more likely an expression of fear or aggression than friendliness. It's because when you grin, you show your teeth, as in "bare your weapons". Grinning is an expression of fear that became a show of submission in social primates. It's thought that's why it became a friendly gesture in humans - it's advertising that you don't want any trouble.
A friendly chimpanzee smile, if it shows any teeth at all, only shows the lower ones. Even then, humans might find it to look threatening (at least someone has noted this about friendly bonobos smiling), which would be a subversion.
There are several accounts of dog owners getting growled at, nipped, or even bitten by their frightened dogs after smiling.
This is because in "dog language", baring your teeth and/or snarling - the motions of which can be simulated by an enthusiastic smile - is considered to be a threat. In their minds, they're simply defending themselves after you've threatened them.
Dalmatians are the only dog breed that can approximate a smile, usually done so if they know they're in trouble or are just very happy. It looks like a snarl, but they do not show their back teeth, and they're usually wagging their tails.
Some Jack Russell terriers can do it too.
A mutt, half beagle and half schnauzer, may be capable of doing this.
Most of the retriever-type breeds are capable of smiling (it goes along with the "soft mouth" that's essential for returning a downed game bird to the dog's master undamaged). Labradors and Goldens are particularly known for it.
Horses can be trained to do this, based on a horse's natural response to help them identify a scent. Some humans however can find this a bit disconcerting.
Most Swedes smile without showing their teeth, and doing so is seen as a little intimidating. Apparently, "only crazy people or Americans smile a lot with their teeth."
According to legend, victims of cyanide poisoning will die with a horrifying grin on their faces, due to muscle spasms.
It's called rictus and it can happen to any victim of facial muscle spasms. It may be caused by paralysis as well.
The "Sardonic Herb" of legend, used in Ancient Rome and Greece to kill political prisoners, supposedly caused rictus upon death. "Rictus", for the record, comes from a Latin word meaning "to laugh".
Some people just can't stop grinning as an unconscious reaction to stress, hence why many scoldings include "wipe that smile off your face". Needless to say, this causes misunderstandings.
This is also the most common reason for Corpsing, which usually starts as an inappropriate for the situation grin and spirals into laughter.