This is a type of ensemble, in which the top authority figure rules in a more refined way and the under boss in a more straightforward way. That is, The Captain will be an Officer and a Gentleman and A Father to His Men. When he needs to intimidate he will use subtle means like a Death Glare. By contrast lower level bosses like a Father Neptune, a Sergeant Rock or, if the crew is unlucky, Drill Sergeant Nasty will be harsh and direct in their method of rule.
This is to some degree Truth in Television. Not only is it a holdover from class differences, but it reflects the fundamental difference between the two roles: The officer must concern himself with the big picture, providing direction to his unit in the form of plans and orders, while the NCO's business is in the details, enforcing discipline, maintaining the unit's proficiency, and personally directing his soldiers in battle. It can also be a useful psychological trick that bears relations with Good Cop/Bad Cop.
Readily capable of subversion, as that refined top figure is the boss for a reason and might be very dangerous if his full attention is called for.
Anime and Manga
Taki and Klaus of Hyakujitsu No Bara fit this trope, although their positions are more analogous to "colonel" and "captain" respectively. Although Klaus is shown to have a lot of concern for his subordinates in action, he is distastefully viewed by Taki's compatriots as wild and rough-around-the-edges, and he is very frank with orders. In contrast Taki is idolized by his troops who see him as someone who can never do wrong, and the Death Glare is the most he will exercise on his own men. At one point Taki does come closer to the "rough" side however, when he uses training to vent his own frustration, resulting in him ruthlessly beating soldiers (one of whom was already injured) foolish enough to try their luck against him.
Captain America occasionally features this during his WWII adventures, with the Sergeant usually being Sgt. Nick Fury, but his first mission featured Bucky as the Sergeant Rough as seen in Wolverine Origins.
The comic version of 300 had a King Smooth and Captain Rough. Leonidas was usually very calm and stoic while his Captain was brutish and violent.
In Big Human on Campus, The new captain and vice-captain of the Enforcers, Tsukune and Ranma respectively, tend to fall into this category.
Lieutenant Philip Holtack, a Britsh Army officer stranded in the Discworld with part of his unit in Slipping Between Worlds, has cause to reflect on this and is thankful his platoon sergeant has slipped with him. Holtack also muses on the various types of Universal Sergeants based on several familiar Discworld characters.
The titular Dusk Guard from The Dusk Guard Saga has an inversion. Captain Steel is a stern, no-nonsense officer, while his second in command, Lieutenant Hunter, is an easy-going individual.
In Fort Apache this trope is even somewhat lampshaded in a scene where the young lieutenant O'Rourke seems embarrassed as he is about to train a platoon of recruits, the sergeants comment that young O'Rourke is a gentleman and training recruits is not a job for a gentleman. And then they take it over.
A non-military example occurs in The Shawshank Redemption, with Warden Norton filling the role of Captain Smooth and Captain Hadley being the Sergeant Rough. These roles are apparent in the dressing-down of the new meat.
Although both are officers, Von Ryan's Express has Colonel Ryan and Major Fincham. Ryan is a self-declared "90 Day Wonder" who was drafted in to serve as an Army pilot, earning his rank of colonel due to age and education. Fincham is a major but has lived his whole life as a battlefield soldier, and obsesses over discipline and adhering to a code of behavior that Ryan can't fathom. They clash early and often over every step Ryan implements, with Ryan proved right some times and Fincham proved right at other times.
In The Good, the Bad and the Ugly Blondie(the Good) and Tuco (the Ugly) are captured by Union soldiers and brought to a harsh prison camp (they were both wearing Confederate uniforms at the time and Tuco had foolishly shouted some pro-Confederate remarks just before being captured). There are three officers shown to be running the camp, and the two most prominently shown are the extremely brutal Corporal Wallace and Angel Eyes (the Bad). The commandant is actually a decent guy who tries to get the two brutal officers to treat the prisoners fairly. Unfortunately, he's dying from an infected wound, and unable to stop the two officers from taking prisoners inside a building just so they can beat the crap out of them.
Gettysburg—and The Killer Angels, the book it was based on—features Col. Joshua Lawrence Chamberlain and his sergeant Buster Kilrain. Chamberlain is an academic and a professor, Kilrain is a rough, sarcastic, out-and-out career soldier. Between them they keep the 20th Maine up and running for its Crowning Moment Of Awesome at Little Round Top.
George MacDonald Fraser's McAuslan series about life in a Highland regiment circa 1947 has this in spades. Best-illustrated by Lieutenant MacKenzie, the son of a baronet and "politically somewhere to the right of Louis XIV", and his Communist sergeant McCaw, who run their platoon like a well-oiled machine despite a tendency to argue politics with each other in front of their men.
In one scene, a bosun beats up a recalcitrant sailor. Hornblower is grateful that he is a Captain and too exalted to do such things as he is probably not a good enough fighter.
Hornblower and his loyal first officer William Bush fit this trope quite well. Though Bush is in no way unusually rough for his time — it's rather that Hornblower has very modern views about discipline and punishment.
Inverted in the book Ender’s Game, and Ender even has a psychological reason for it: mercy and good consequences should come from his troops' immediate commanders, discipline and bad consequences from he, Ender, as the overall commander, mainly so it bonds his squads into tighter units willing to protect each other.
In Robin Hobb's Liveship Traders trilogy, it's mentioned a few times that the captain gives the orders and the first mate beats the men into line when necessary.
In first two novels of The Serpentwar Saga by Raymond Feist, The aloof half-elven Captain Calis with super-human strength and senses, and the rough, foul-mouthed Sergent DeLoungeville are an obvious fit.
In Watership Down Hazel is the Captain Smooth, and Bigwig is the Sergeant Rough. Inverted with their Efrafran counterparts: General Woundwart is a brute who rules by force and Captain Campion is a calm-headed strategist.
In the Night Watch, the roles are reversed. People tend to respect Commander Vimes, but everyone seems to genuinely like his second-in-command, Captain Carrot.
Averted in Eric, where the captain is The Neidermeyer (his training being in composing victory odes and heroic poses) and the sergeant is Sergeant Rock (his training consisted of 50+ years of fighting and not getting eaten by the various horrible creatures the Discworld has to offer).
In the Harry Potter series, headmaster Albus Dumbledore and deputy headmistress Minerva McGonagall fit this pretty well. Dumbledore is the Eccentric Mentor and generally doesn't seem to care much about school rules being enforced. McGonagall is a serious Stern Teacher, although she is occasionally Not So Stoic. In Philosopher's Stone, Dumbledore catches Harry out of bed after curfew and his response is to have a nice mentorly talk with him. Later on in the same book, McGonagall catches Harry and friends out after curfew and her response is to dock a shitload of points and give them all detentions.
Invoked as the difference between Kurt and Mendez:
Kurt: Chief, I'm sorry that order had to come from you. Mendez: I understand, sir. You're the CO. You have to inspire and command their respect. I'm their drill instructor. I get to be their worst nightmare.
Later when the Sentinels start to attack, the Spartans (on a training exercise) wonder if they're hearing artillery strikes. They conclude that while Kurt wouldn't use artillery against them, there's a good chance that Mendez would.
Inverted in Dad's Army where Captain Mainwaring is a oafish but brave amateur and Sergeant Wilson is a suave aristocrat with extensive military experience. Mainwaring's (and Wilson's) intense awareness of the class inversion was a comedy goldmine. Only in the last episode was it revealed that Wilson was indeed a genuine Regular Army Captain Smooth from the previous war. He was filling in as sergeant so that Mainwaring could be Captain if he wanted to; having been a real captain he had no intention of squabbling over who should be a Home Guard captain at his time of life.
In Andromeda, Hunt is the captain smooth, while Valentine bears some traces of sergeant rough.
In the tv series Boston Public, the principal is smooth (shaved head!) and tactful, while the VP is a strict disciplinarian with a knack for intimidating... and the funny thing is that the principal is just a scowl away from being a Scary Black Man, while the VP is a short, slim and normally unremarkable white guy.
In Ultimate Force, Col. Aidan Dempsey (Miles Anderson) is the Captain Smooth, while Ross Kemp's SSGT Henry 'Henno' Garvie fits the Sgt Rough trope to a T.
To a lesser extent, Gustavo and Griffin of Big Time Rush fit this trope. Griffin (who owns the record company) is smooth and collected, while Gustavo (who works under him) yells a lot.
Battlestar Galactica: In the reimagined series, Admiral Adama and Colonel Tigh. Adama knows the name and face of every single member of his crew by heart, has demonstrated that he will go to war over every last one of them, and has a Death Glare with its own page on the Battlestar Wiki. Tigh, on the other hand, openly states in the pilot miniseries "If the crew doesn't hate the XO, he's not doing his job."
Inverted in the first episode. Captain Sobel is the Drill Sergeant Nasty while Lieutenant Winters is the nice officer the men like.
Firefly inverts this: Malcolm Reynolds, who keeps his Sergeant Rough from his war days, calls himself a captain. His Lancer, Zoe, is calm, cool, and collected, even when she's about to end you.
Inverted in the remake of Hawaii Five-0. Despite being in charge, Steve is the biggest wildcard on the team and is known for his inventive interrogation techniques, while Danny is much more by-the-book and civilised.
Sharpe and Harper in Sharpe, though Sharpe is somewhat less refined than most officers.
Sharpe himself arguably fulfils this role for a couple of his commanders, notably Mc Candless, Lawford and Wellington himself.
The first two (chronological) novels also give us the Heel version in Captain Morris and Sgt. Hakeswill. They're both devious, conniving villains, but Morris is a suave social climber who uses Hakeswill, a horrible thug, to do his dirty work. Hakeswill later returns as a sidekick for Wyndham, although Wyndham is a genuinely decent officer who only sees Hakeswill as a model sergeant and isn't aware of what he's up to.
Captain Sisko and Major Kira of Star Trek: Deep Space Nine. Sisko is rather more polite and refined about his orders. Kira is not refined, nor does she pretend to be. Her preferred manner of dealing with problems is to yell at them until they go away and/or resolve themselves. If this doesn't work, she resorts to fists.
The Circle Tower in Dragon Age: Origins is shared by the Circle of Magi and the Templars, with a First Enchanter and a Knight-Commander usually sharing the duties. In the game, Knight-Commander Gregoir is "Rough" and First Enchanter Irving is "Smooth," and the two of them butt heads much more often than one would usually see in this situation. Of course, the fact that the Templars' duties include killing rogue Mages might have something to do with it...
Inverted in Dragon Age II, Meredith is a lot more hard-line and brutal than her more diplomatic and practical second-in-command Cullen.
In Pokémon Mystery Dungeon Explorers of Time/Darkness/Sky, Guild Leader Wigglytuff and his second-in-command Chatot work as this!
The Cavalry in Final Fantasy XIII are led by the gentlemanly Brigadier General Cid Raines, and his more rough-and-tumble subordinate Rygdea.
Lightning, being an actual sergeant, literalizes this trope. Her solution to everything is to crack heads open, while her cooler-headed superior officer Amodar suggests she try to stay out of the business of Cocoon's leaders.
From Valkyria Chronicles, we have Captain Varrot (who is Welkin's direct superior and overall commander of the militia) and Largo (whose rank is unspecified but presumably that of an NCO). Formerly squad mates in the previous war, they still maintain a rather tenuous relationship, due to Varrot's inability to get over the death of her lover and Largo's unrequited feelings for her. In the end, she gets over it and settles down with Largo after the war.
Inverted in Starcraft II, where Raynor is a Cowboy and Matt is a boyscout. Matt technically is a Captain, but he's also the subordinate of Raynor.
Lieutenant J.T. Marsh and his second-in-command Sergeant Rita Torres in Exo Squad probably qualify, though they are less extreme than the usual description. Also, Nara Burns and Torres after Marsh is promoted.
As noted above, this trope largely originates from the time when most of the officers were from aristocratic families, while NCOs were generally low-born grunts who had survived previous campaigns long enough to learn the "tricks of the trade" through personal experience. Today, officers and NCOs go through completely separate training programmes and promotion from one group to the other is rare (as they do different jobs, it's more of a career change than a promotion). In the modern era, it's also fairly common for sergeants to have a higher pay grade than their commanding officer, for this reason.
Yet it's still an effective chemistry, sometimes invoked a bit, but always lampshaded.
Most American high schools have the vice-principal as the chief enforcer and disciplinarian.
Before conscription was eliminated in Romania, all newly drafted individuals with a university degree would be enlisted into an academy that would train them into officers. While high ranking officers oversaw the education, the sergeants were still responsible for keeping them disciplined, leading to an inversion of this trope. On graduation day however, the trope would end up being played straight, with the students getting promoted, and outranking the sergeants who had been bossing them around for six months. The sergeants, of course, never participated in such events to avoid the humiliation.
In various parliamentary systems (as in the UK and the US), each party has a leader and a "whip" among its representatives, where the latter's job is to enforce party discipline by (almost) all means.