Applied Phlebotinum - You need some form of explanation for why they're using giant humanoid robots/golems/biomechs instead of tanks and planes. It can range from "Getter Rays need a human form!" to "They're magical" and even more 'realistic' explanations such as Gundam's "Humanoid robots can turn around in space without wasting fuel". Careful with making these mecha more specialized - they may slide out of your series' focus.
Rule of Cool - Even most Real Robot shows operates on this, because if it were any more real, it'd just be using tanks instead.
First of all, choose the media. Are you working on print, animation, a Mecha Game, Toku? Each media has their own strengths and limitations; Toku mecha series, for example, often require a human to fit in the mecha costume as to make it move.
Real Robot or Super Robot? - Do remember that Real and Super are adjectives, and not strict delineations of genre; nor are they Power Levels. Super robots can have numerous weaknesses and drawbacks to usage, and real robots can be extremely powerful (ever hear of nuclear weapons?). They are best defined as settings.
Real Robot: Gundam is perhaps the archetypal example of the Real Robot setting, with Armored Trooper VOTOMS taking it to the max. Real Robots settings typically have the mecha as mass-produced weapons of war, and oftentimes they're ignominiously blown away, much like soldiers in war films.
If you really wanted to, you could forgo the crew/pilots and have sentient humungous mecha, such as in Transformers. It's possible to mix them with pilots, as seen in Zoids and various Super Sentai series. Maybe the mecha are Mechanical Lifeforms with intelligence only equal to that of beasts, and require pilots to fight more effectively. Or maybe they're an alien race who had to transfer their brains (or souls or whatever) into humongous mecha in order to survive a great disaster (maybe the same evil alien race threatening the protagonists?).
2D Visuals, 3D Effects: CGI will help you get the most out of your budget and can produce some impressive visual effects, but screw it up and the entire project will look halfassed. Less of a problem nowadays because Technology Marches On and in fact, it can be a Necessary Weasel at times due to a scarcity of mecha animators nowadays, but do be aware you might be alienating animation purists if you go this route.
Rule of Cool - What if the ultimate attacks are Awesome, but Impractical? What if the heroes can win by being sensible and practical, reigning in their Hot Blood? What if the bad guys are the ones who do things just 'cause they're cool, while the heroes take these life-or-death battles more seriously?
Grand Theft Prototype / Black Box - what if whoever makes the Super Prototype / advanced technology in question had the foresight to take out a patent on the design, making it worthless for reverse engineering purposes? Sure, it's just silly in war story (why would people on the opposite side of a war care about patent laws, especially if one side are aliens?), but can be done on smaller scale, like Mega Corps conflict.
Gambit Pileup - Nothing says suspense like loading on a huge convoluted plot to keep the audience guessing. Don't get too over-the-top with it, however, as it can sometimes end up contrived, confusing, or just plain silly.
When it comes to mecha design, if you're trying to pull off Rule of Cool in a Super Robot Genre, call in Masami Obari, he can design awesome mechas for you. However, try to ask him not to design female characters in your show, because he'll usually go overboard and make them extremely big-boobed no matter the age, which doesn't always translate to 'better design' (some people might laugh at the design)
Any mecha show can be made exponentially cooler with moar Techno Babble.
Take into account the possibility of your work appearing in a Super Robot Wars game - don't go out of your way to include stuff for the writers of those games to fix, since it's not a sure thingnote live action works such as Super Sentai have only recently made their debut, and Vandread, despite having been out since 2000, has not been included in any game as of yet, but do at least consider elements that could work with other staples in the franchise (Mazinger Z, Getter Robo, and Gundam) if you think your work has a shot.
While Super Robot Wars is known to fix many problematic series, even some of those in the Epic Fails section below, do not make this an excuse to slack off and make your creation fail and hope SRW will salvage it in the future. Just consider that option to be some sort of last resort, in case you already did your best and somehow still failed.
Zambot 3 - The first anime to invoke the idea of Kill 'Em All, this was also the first work of a then-unknown Yoshi Yuki Tominonote although he did direct the first 26 episodes of hereinafter Raideen, this was his first solo effort.
Space Runaway Ideon - Directed by "Kill 'Em All" Tomino, who lives up to his nickname here, it is about the Ideon, a Super Robot with a rather scary power source, the Ide. Was one of the inspirations for Evangelion.
Armored Trooper VOTOMS - Votoms is the leader of what defines 'Real Robot'. The mecha are basically glorified humanoid tanks, and their main propulsion most of the time are wheels built into their feet. Instead of a kid, the main character, Chirico, is a hardened elite soldier who knows what he's doing.
Dancougar - A deconstruction of Hot-Blooded heroes. The heroes have trouble getting along with their superiors and each other, often drop the ball and cause collateral damage, and they can't even successfully combine into the titular robot until halfway through.
G Gundam - A great example of how Humongous Mecha can be combined with any genre. A Chinese martial arts adventure with giant robots? Awesome!
Giant Robo - Both an old live-action series (with Giant Robo being a guy in a suit) and later an OVA series, which was directed by the same director who did G Gundam.
Neon Genesis Evangelion - Love it or hate it, Evangelion changed the genre forever. It created a whole new school of more organic mecha design, and subverted so many Super Robot tropes. It is the Watchmen of mecha anime.
Gundam Wing - The first Gundam series to gain a major fanbase in America, and in both countries, it introduced a new viewer demographic: Girls.
Eureka Seven - While this series took time to find its footing, the series became a classic for its badass action, excellent character development, and for having one of the most best handled anime love stories of all time.
RahXephon - At first glance, it seems to be an Evangelion imitator, but it's actually a Spiritual Successor to Brave Raideen. It combines a musical motif with mesoamerican mythology to create a rich, alien atmosphere.
Tengen Toppa Gurren Lagann - in case you don't want your Super Robot to fight one Monster of the Week after another, you can turn to these for good examples of lighthearted Super Robot adventure series. Gets darker later on, though also more awesome.
Gundam SEED - Say what you will about its sequel (see below), Seed repopularised the Gundam franchise and brought in a whole new generation of fans to Gundam. It also adapted parts of the original Mobile Suit Gundam storyline, while still managing to tell an original story with likeable, sympathetic, and above-all, well-developed characters.
Macross Frontier - Took Macross to a whole new level. Just make sure you don't get your Humongous Mecha series in a sticky legal situation with American distributors though. Harmony Gold is notoriously responsible for preventing this series from reaching American shores.
Danball Senki Wars - A good example of how an anime can retain the dark themes associated with this genre while still being aimed towards kids. It has well delivered political message, good character development, kickass battles and epic OST.
Gundam SEED Destiny: This installment is notorious for its huge hatedom. If anything, it's greatest failure(s) seem to lie in the fact that all too often, personal conflicts between cast and crew would spill over into the story itself. Whether you like this series or not, don't let this happen.
Gundam AGE: While having an ambitious plot can help your series stand out, this show proves that ambition can become a wash if you do not have competent writers in your writing staff. Thanks to its rushed and/or poorly-developed plot-lines and characters, AGE became the worst received (and rated) Gundam installment yet. Moreover, Kio Asuno is a prime example of how not to write a Technical Pacifist into your story.
Martian Successor Nadesico: The Prince of Darkness shows how not to do a movie sequel. Its problems lie mainly in an overly abrupt shift in tone and characterization and poor pacing throughout. Much of this stems from the fact that it was a planned trilogy cut to one movie, so if you're planning on multiple movies make sure you have a good sense of what plot points are most important, just in case. Aside from that, know why people like your story and characters and don't lose that.
Super Dimension Cavalry Southern Cross is not really spectacularly bad so much as uninteresting to the point of being a cure for insomnia. Reportedly, the Japanese reaction to the "Southern Cross" segment of Robotech was along the lines of "How the hell did they make Southern Crosswatchable?"
BattleTech's animated cartoon was an attempt to take a popular video game and tabletop wargame franchise and bring it to television. Unfortunately, an astounding lack of research and clumsily rendered animations led to mediocre results at best. Be mindful of your setting and prior fan expectations if you're going to do an Animated Adaptation or similar derivative work.
Gundam: Reconguista in G spent so much time layering complications and additional factions into the setting that it forgot to tell the story. By the time all the players got on the board, there were only about three or four episodes left.