Interestingly enough, the game as released is what could have been. It's common knowledge that the game is a localized version of Doki Doki Panic released as a replacement for the Japanese Mario sequel, which was deemed both too similar to the original game and too borderline-Platform Hell to be commercially viable elsewhere. What's not as well-known is that Doki Doki Panic was originally conceived as a more vertically-oriented title to begin with. Whether this version of Doki Doki Panic was a Mario game or not is still up for debate.
Prototypes of the game had the classic underground theme used for underground/inside areas instead of the remixed Doki Doki Panic music used in the final versions.
Originally released as a launch title for the system, the game was a starting point for the the team behind Super Mario Bros. 3 to adapt their work to a new system (and indeed, the game had the subtitle "Super Mario Bros. 4" up until very late in the development process, as evidenced by television commercials promoting the game). The game was originally intended to be released on the NES with a graphics enhancing chip in the cartridge (as the SNES later had with the Super FX in Star Fox 1 and the Super FX 2 in Yoshi's Island), before ultimately moving onto the new system. Other scrapped elements:
The game would have originally taken place in a much smaller world (two islands, which looked to be similar versions of Donut Plains and Chocolate Island), and Lakitu and/or a blue sparrow would appear periodically and fly around Mario (though their purpose is still unknown). Lakitu and the bird are still programmed into the final game, but Dummied Out.
Toad Houses like those in Super Mario Bros. 3 can be seen in some of the earliest screenshots of the beta. These were cut from the final build, and as a result, Toad is absent from a main Mario platformer for the first and only time.
There would have been several areas where the player would have to dismount Yoshi (or have the dinosaur disappear), indicated by a sign with a crossed-out Yoshi image; in the final version Mario only dismounts Yoshi before going through a ghost house, fortress, or castle, and Yoshi returns for the next normal level after that. Again, the sign sprite is still programmed into the game, but unused.
At least three of the Koopalings were intended to be present on the overworld map (in very different locations than any corresponding level in the final game), and would "pull" Mario into stages (much like the hands in World 8 of Super Mario Bros. 3). Although their access methods were removed, their original positions can be viewed in Lunar Magic.
There were many sprites from Mario Bros. 3 that were intended to carry over into World, including the original Pirahna Plant, Venus Fire Trap and Raccoon Mario (as seen in beta screenshots and the SNES Test Program cartridge).
Yoshi itself went through several changes in production — starting off as a frog, then a western dragon, an ostrich, and finally the Yoshi we know and love — who incorporates aspects of the first three creatures.
Yoshi was also supposed to have been a Koopa Troopa (or a variant of the species) that would have joined Mario on his adventure. The Yoshi Koopa concept can still be seen on the final character design, where the saddle looks like a small Koopa shell.
According to Word of God, the original localized name for Punchinello in Super Mario RPG was James Bomb. This name was rejected, although he still introduces himself as "Nello... Punchinello".
There were at least two other Donkey Kong games planned for the NES — one was a simply musical spinoff, while the other was known as Return of Donkey Kong. While there is little word on this one, common speculation is that the game was ported to the Game Boy. Regardless, Donkey Kong '94 was considered one of the better original titles often overlooked. The game was going to get remade again as Donkey Kong Plus with GBA-GCN connectivity, but was dropped in favor of Mario vs. Donkey Kong (developed by the Redmond-based Nintendo Software Technology Corporation).
Super Mario's Wacky Worlds was originally meant to be a CD-i exclusive sequel of sorts to Super Mario World, where Mario would travel through various Earth-based locations. However, it got canceled due to various reasons, and ended up replaced by Hotel Mario, also known as one of the worst games in the series, which ended up in Canon Discontinuity.
In addition to Wacky Worlds, there was another CD-i Mario game titled Mario Takes America, which was going to be an educational game about American culture using live-action Full Motion Video. There was also apparently a Donkey Kong game planned, though nothing is known about it beyond a guy listing it on his Linkedin resume.
Mario Clash, a pseudo remake of Mario Bros. for the Virtual Boy, was originally conceived as a sequel to the Super Mario Land games for the Game Boy. View a beta video here (skip to 3:12).
Mario & Sonic at the Olympic Games was originally going to feature Silver the Hedgehog, Jet the Hawk, Birdo, Kiki, and Donkey Kong as additional characters, as discovered here. In Mario and Sonic at the Olympic Winter Games, Silver and DK were elevated to playable characters, but Kiki is still MIA. However, you do race against Jet in the Festival, and Birdo is in crowd shots and eventually becomes an opponent in 2012 London.
Super Mario 64 was originally going to be made entirely of more linear levels in the style of the Bowser stages.
Nintendo Power at one point mentioned a sequel to Super Mario 64 for the ill-fated 64DD that would have had Luigi as a playable character, which didn't happen until the DS version of the original game (and the first time he was playable in an original 3D game was the Super Mario Galaxy games, over a decade after the release of Super Mario 64).
There were supposed to be 32 worlds rather than the 15 that were shipped. Players were also supposed to be able to take control of both Mario and Luigi and complete different parts of the castle simultaneously, which obviously means the game was originally a multiplayer game. Yoshi was also supposed to have had a much larger role in the game rather than just hiding on the roof of the castle.
As revealed by Miyamoto in an interview with Famitsu, the game was going to have more power ups and features that were left out due to time constraints. They were saved for the sequel.
The sequel to Galaxy was supposed to have a more developed story and expand upon Rosalina's history. However, Shigeru Miyamoto decided against the idea and wanted the game to focus more on the game play itself as Super Mario Bros. had done. This explains why Rosalina only appears in the ending.
According to a Super Mario Galaxy preview on G4 (circa 2006), it was initially going to be a Nintendo GameCube game.
Some members of the "The Cutting Room Floor" wiki found some leftovers in the Japanese Super Mario Sunshine version which indicate that you got to the levels via train stations. You buy tickets with Sol Coins. Another interesting thing is that the station list mentions levels that were cut out or even never created at all. These are "Hotel Lacrima" (another hotel!), "Battleship Island", "Erto Rock", "Lighthouse Island", and "Flame Temple".
Going all the way back to the beginning: Shigeru Miyamoto wanted to have the game that would become the original Donkey Kong feature Popeye, Bluto, and Olive Oyl (which Nintendo had publishing rights to in Japan). When King Features balked, Miyamoto reworked the characters to include Donkey Kong, Pauline, and "Jumpman".
Miyamoto got his chance to make a Popeye game shortly after making Donkey Kong. The Popeye game was a modest arcade hit and got ported to several platforms while "Jumpman" went on to be the core character of the Super Mario mega-franchise.
For the originalSuper Mario Bros., Mario originally was meant to shoot bullets rather than fireballs. Also, Miyamoto originally planned the controls to be A to shoot, B to run, and up on the control pad to jump, but it was decided that being able to run and shoot at the same time was too much of a Game Breaker, so it was changed to put both shooting and running on the B button, meaning the jump could be moved to A.
A later Iwata Asks interview gave a few more details about this- apparently Mario could have had a rifle, a beam gun, or a kicking move. There were also plans for Mario to be able to fly on a rocket (presumably at least somewhat similar to the plane and submarine in Super Mario Land), and later a cloud (which would later actually happen in the American Super Mario Bros. 2, and Super Mario World respectively.)
Retro Studios wanted to do a Mario football game. As the company was formed mainly for games viewing mature audiences (its founder and many employees worked on Turok), Nintendo instead suggested to make a realistic football sim. That game had some development, but got canned as development of Metroid Prime progressed (and that game itself earns its own entry elsewhere on this site).
Rosalina was intended to be playable in Mario Tennis Open at some point in development, but since designing a model for her would have taken some time, they took the easy way out and added Luma instead.
Around the time that Waluigi was created for Mario Tennis, there were ideas for a "Wapeach" to also be introduced, but thankfully, that idea was scrapped.
Luigi's Mansion was planned to be in stereoscopic 3D using an add-on, but when the production of said add-on turned out to be more expensive than the system itself, it was canned. In hindsight, it makes sense that when Luigi's Mansion: Dark Moon was announced, it was on the Nintendo 3DS, which has 3D without the need for an add-on.
The game originally had Mario in a starring role and the house was more of a Japanese ninja house. The house was later re-tooled into a dollhouse and, after the developers began to experiment with lighting effects, decided to turn it into a haunted house.
The game originally had RPG elements, but the developers wanted more of a focus on action-based gameplay.
Mahito Yokota originally composed up to 28 musical pieces for Super Mario Galaxy that had Latin beats. Koji Kondo said they were no good, so Yokota eventually came up with the sweeping orchestral pieces that made it into the final game.
For Super Mario 3D Land, director Koichi Hayashida suggested fairly early on in development to include the Goomba's Shoe as a power-up. This idea was nixed fairly quickly, though.
There were also suggestions of a cockroach-like enemy that can only be squashed by folding up the 3DS. (While no Mario game had this function, there were games for the original DS with similar functions.)
Paper Mario 64 was originally entitle "Super Mario RPG 2". After copyright conflicts rose with Squaresoft, the game was renamed "Mario RPG 64". After one of the developers showed an idea for a new style of gameplay with paper characters in cardboard environments, the game was re-named "Mario Story" in Japan and got the name "Paper Mario" in North America.
Mario Kart 64 was, according to the previews, going to have a Magikoopa in the character roster, but he was later replaced by Donkey Kong.
The game Croc was originally pitched by developer Argonaut to Nintendo as a 3D Yoshi game, though Nintendo turned it down. This upset the developers so much that Croc was only released on rival platforms, such as the Playstation, as revenge.
Next Level Games, the developers of the Mario Strikers games, were planning to make another Mario sports game for the Wii. Originally called Mario Volleyball, it was changed to Super Mario Spikers as the developers decided to add wrestling and game show elements, inspired by the cancellation of one of their earlier WWE games, WWE Titans: Parts Unknown. While Nintendo was originally open for the idea, they eventually decided not to greenlight the project, as the game's more realistic violence (compared to the more slapstick "violence" of the Mario Strikers or Super Smash Bros. games) clashed with their code of honor, and the game was cancelled in 2007.
Super Mario Maker was originally intended to be a sequel to Mario Paint but along the line they decided to add the ability to create new levels. Also, it was originally just called "Mario Maker", and the graphics style for the Super Mario Bros. theme was more stylized, with a beveled edge on the ground blocks.
Infamously, Paper Mario: Sticker Star was originally closer in design to the first two games, boasting such things as a Chain Chomp partner, a crowned Monty Mole boss, and a unique card level. When Shigeru Miyamoto got involved, however, he declared the game was too similar to "the Gamecube game", forcing the developers to go in a different direction. The end result was a very polarizing game that is widely seen as the worst installment in the Paper Mario series, and tainted Miyamoto's reputation considerably among many fans.
According to MarioWiki, Archie Comics proposed the idea of a Mario comic series to Nintendo, but was eventually rejected.