Simply put: when a 2D original's transition to a 3D turns out to be difficult, usually because a Video Game 3D Leap often requires more skill and talent than some developers have.
There are two aspects of dimensionality when it comes to games. A game can be rendered in 2D or 3D, and the gameplay can be 2D or 3D. All 4 combinations have been seen.
Switching from one of these combinations to another, especially going from 2D/2D to 3D/3D is fraught with peril. Particularly in the early days of 3D rendering, art styles that were painstakingly developed in 2D could be lost in the transition to 3D rendering due to lack of hardware capable of bringing it to life in 3D. And of course, there are some art styles that simply don't work in 3 dimensions at all.
Gameplay offers some very perplexing challenges too. A direct adaptation of 2D gameplay into 3D gamespaces can cause things like the dreaded Camera Screw. 3D gameplay, by necessity of both viewpoint and larger gamespace, needs to take things a bit slower than their 2D cousins.
Then, there's the version where developers forgot about trying to port the 2D gameplay into a 3D world, and just use 3D gameplay that is not entirely unlike what the 2D gameplay had. Freedom overload can ensue, where developers become so enamored with building a gigantic world that they forget to actually put interesting things into it. Puzzles that would have been simpler in a 2D game can become exceedingly complicated because of the changed viewpoint.
Oddly enough, for RPGs and adventure games, 3D rendering once was much more limiting than 2D. Consider a set of shelves with miscellaneous bottles on it. In 2D, this is part of the background; it costs comparatively little. In 3D, each bottle must have polygons, which means the quality of that bookshelf goes way down. You only get so many polygons per frame, so they should be spent on actual characters. Some games tried to cheat by using 3D backgrounds but computationally cheap sprites for the actual objects; this seldom resulted in a pleasant experience to the eye.
Add to this the lack of tilemapping: a common 2D technique for reusing images. Through tilemapping, it was possible for designers to create large terrain, with stuff in it, fairly easily. The time to develop one area would be pretty much the same as any other. You couldn't do that with 3D in the early days; every room had to be hand-built from scratch. You could reuse textures, maybe certain decorations (chairs, tables, etc), but that's about it: the basic blocking of each area had to be done from scratch.
What you get is that some games that had large 2D worlds seemed to get compressed in their 3D outings. This isn't as much a problem nowadays, but in the early days of 3D rendering, it was pretty widespread.
Be aware though, sometimes good 3D video-game adaptations are accused of this, mainly because the fans don't like to see their original product change. Notice that very similar games with no 2D predecessors were often well received.
On the other hand, when this trope was common, not being in 3D could also invoke It's the Same, Now It Sucks among fans, so some game producers didn't have a real choice but to try.
Do not confuse with They Changed It, Now It Sucks, which is about the reception of such games.
Bomberman hit the Polygon Ceiling a bit harshly with the 64 series. They later tried again when the Gamecube rolled around and broke through the ceiling just fine with Bomberman Generation and Bomberman Jetters, having made the smart move of giving only the single-player mode 3D gameplay while keeping the multiplayer the same as with the 2D games. It's a pity, though... they could have paved the way for a lot of remarks about why he's called Bomberman.
Adventure Games, generally speaking, have not dealt with the transition to the 3D era well (Telltale Games' games being an exception), as they usually require a lot more attention to detail than your average FPS, platformer or what have you. Rarely will players need to explore too much or pay attention to everything in a run-and-gun shooter, whereas an adventure can easily require one to, for instance, look behind furniture, under objects, or to search everywhere for that one missing piece of the puzzle. The difference in developing time and resources was rather significant - though it's gotten less so nowadays, due to procedural generation making creation easier and a much higher detail requirement in most other game types equalizing things.
After an extremely successful run with Gabriel Knight, the series went through two separate clashes with this trope. The first sequel had full-motion-video gameplay in a 2D environment, while Gabriel Knight 3 went into 3D. While both sequels had strong storylines, they were very difficult to play.
Jane Jensen later said that they did not foresee the amount of details (and work) a full 3D game required. For her future projects she would prefer a 2D game on pre-rendered 3D backgrounds — it would look good enough while being much cheaper.
On the other hand, it was also a pretty sharp Genre Shift, from Adventure Game to Action, wherein large parts of the game were spent hitting things with swords and keeping track of hit points.
The Legend of Kyrandia 1 and 2 were widely praised, while the 3rd got a mixed reception. Transition from painted to rendered backgrounds and objects was not the main peeve, but it definitely affected the game atmosphere. The primitivism of models also did not help.
Dialogs in Star Control 2 used bright-colored low-resolutionnote about half of a 320x200 screen still pictures with a bit of animation. That seemed adequate. Star Control 3 upgraded that to dim higher-resolutionnote up to 640x480 videos with compression artifacts. The animatronic puppets were intended to look like a more realistic 3D, but instead Spathi looked like a flayed carcass, Pkunk looked like a sock puppet and humanoids went to Uncanny Valley.
There are two key aspects to the Polygon Ceiling here. First, the 3-D gameplay is usually slower than its 2-D counterpart, making the games less dynamic (something required for a fighting game). Second, the projectiles, which are a key element of 2-D fighters, hardly ever work effectively in 3-D.
Mortal Kombat is particularly guilty of this, with the gameplay and fatality systems being rebuilt with almost every new game.
Mortal Kombat 4 was a weird case. Even when it was fully 3-D, the gameplay was not greatly altered from the previous games (with only one limited way to move on the Z-Axis), making it very faithful to the original 2-D games.
The following games Deadly Alliance, Deception and Armageddon varied in quality but were said to be somewhat decent.
For Special Forces, it probably doesn't help that the creative team behind it (including series co-creator John Tobias) quit Midway midway through development, thereafter the remaining brood rushed it to development.
With the release of Mortal Kombat 9, they've gone back to the series' roots with 2.5D gameplay a la Street Fighter IV.
One Must Fall Battlegrounds attempted to do jump to 3D but failed due to many functionality problems. It was one of the most promising games ever made, but the bugs, lack of pilot/robot progress, inability to go through the arenas fast and awkward controls made for a big disappointment.
The King of Fighters got around this with a "2.5D" subseries a la SFIV (in fact, predating it), the Maximum Impact line. For those who still aren't comfortable with the idea, it is explicitly an Alternate Continuity; the main games still use sprites. In a strange inversion, KOF XII, which is sprite-based, proceeded to bomb. One of the reasons is how they scaled back on everything - half of the roster had been cut, and the main gameplay mode was nothing more than a glorified time trial.
Came back with a vengeance with KOF XIII which rectified the previous games shortcomings and added some new modes.
Blood hit the ceiling hard with the with release of Blood II: The Chosen, which traded the first game's sprite-based 3D Build Engine for the full-3D LithTech engine. The game was rushed out to compete with the likes of Half-Life and Unreal, resulting in an Obvious Beta. On top of that, many felt the second game lacked the feel that made the first game fun.
Before that, Sonic Colors (Wii version) refined the Sonic Unleashed gameplay up to where most can agree that it was what finally pushed the hedgehog through the eponymous ceiling where it was stuck seemingly ever since they failed to recapture the success of Sonic Adventure and Sonic Adventure 2.
Earthworm Jim fell victim to this trope heavily, as it abandoned almost all of the gameplay elements (not to mention many of the series' characters) in its jump to 3-D. It also didn't help that the series creator, Doug Tennapel, was completely excluded from its development.
Turned into a Franchise Killer for Bubsy with Bubsy 3D, where the entire game looked entirely unfinished along with a very restrictive control scheme. This was especially awkward considering Super Mario 64 was released months before.
While Prince of Persia 3D was not as well received as its predecessors, the Sands of Time reboot was widely regarded as a classic.
The Mega Man franchise has made two runs at the Polygon Ceiling so far. The Mega Man Legends series was a Base Breaker, looking clunky and featuring gameplay that was Mega Man In Name Only. Mega Man X7, on the other hand, looked gorgeous and played closer to Mega Man games, but screwed up the gameplay; realizing this, Capcom dropped half a dimension and went with 2.5D for X8, leaving X7 an Oddball in the Series.
Bionic Commando ran into this with the 2009 sequel. The following game, Rearmed 2, returned to 2D gameplay.
The first eight Might and Magic games used sprites for NPC's, monsters, trees, etc (in a 3D world in 6 thru 8) and were quite good. M&M 9 was fully rendered. It was also full of bugs, had ridiculous looking characters with flat faces, trees with a few 'blades' of leaves, and a UI with a fraction of the functions of its predecessors.
After a licensing dispute, Acclaim grabbed the NBA Jam name from Midway Games and made NBA Jam Extreme. Extreme suffered from long load times, blocky character models, and painful overuse of the word "extreme." It added two changes to the formula established by the original Jam: First of all, players no longer caught fire, but became "Smokin'", which could be achieved either by scoring three straight baskets without the opponent scoring or blocking three straight baskets. If the player did both, he became "Unstoppable." The second change was the addition of an "Extreme" button, which was even stronger than Turbo, but drained the Turbo meter twice as fast. Neither did much to change the gameplay. Now, Extreme is barely remembered, if at all.