This is when the broadcast networks have their evening network programming on their affiliates. It's when most of the possible demographics are off work, out of school, and awake—and thus, it's the time with the largest possible general audience. It usually is when the most people are actually watching as well, especially early in the week. (Later in the week, people often go out, and since the networks are all tied in to movie studios these days, they don't want to discourage the impulse completely.)
In America, Prime Time is 8-11 p.m. Eastern, 7-10 p.m. Central weekdays, and 7-11 p.m. Eastern, 6-10 p.m. Central on Sundays. (It's been a while since the American broadcast networks have taken Saturday seriously.) FOX, The CW (and its predecessors), and MyNetworkTV don't use the last hour.
The first hour or two of prime time used to be called the Family Hour during The '70s and The '80s, and was supposed to air family-friendly programming. Innuendo, Bloodless Carnage, and Technical Pacifists could be aired, but not outright sex or bloody violence. This has changed for many reasons, including changes in the Federal Communications Commission and the advent of the v-chip. Few people use the v-chip directly, but the parental guideline icons that tell you which "block" range contains a program work as a ratings system, and they are aired publicly every half hour. If you know to avoid, or can program your TV to avoid, all TV-14 or TV-MA programming, there's no need to force it all after 10 p.m., is there?
Well into The '90s, the first hour of prime time was always two half-hour Sit Coms, followed by two hour-long drama/action shows in the 9-10 and 10-11 ET slots, with the only variations being two hours of sitcoms (8-10 p.m.) on Fridays and Saturdays (when more kids were watching) and maybe a newsmagazine or two later in the week—see ANSI Standard Broadcast TV Schedule. There's a lot more variation now; networks will air more or fewer sitcoms or dramas depending on audience taste and how strong their options are for each, Reality TV has become a virtual peer of scripted programming that can be plugged in at any time (though usually at 8 o'clock), and it's not unheard of for a network to run an hour of sitcoms at 9 following an 8 PM reality or even drama/action show, though sitcoms in the 10 PM hour remained completely verboten until very recently. Cable networks, which may not air sitcoms or dramas at all, have even more variation.
In Germany, prime time starts at 20:15 (8:15 p.m.). The main reason is that the Tagesschau, the oldest newscast on German TV, has been aired from 20:00 until 20:15 since The '50s. Other stations use the same timing, and an experiment by a few private stations in the 1990s to start their prime-time programs on the full hour was not popular.
In the United Kingdom, prime time lasts from roughly 6:30 until 10:30. This is because it is thought that the average British family will eat at 6:00, finish tea at 6:30 and then settle down for the night's telly. The 7 o'clock news comes half an hour into prime time, the 10 o'clock half an hour before its end. Between 6:30 and 9:00, family-friendly shows are displayed Doctor Who usually airs between 7:00 and 8:00, and Top Gear is on between 8:00 and 9:00. Talent shows/bear-baiting like Britain's Got Talent, Strictly Come Dancing, and The X Factor UK usually sit across this time period. After the 9 p.m. "watershed", children are sent to bed, and comedy shows and harder-hitting dramas are broadcast, like Sherlock, Spooks, and Rome, alongside panel games like HIGNFY and racier Reality TV like Big Brother UK and I'm a Celebrity... Get Me Out of Here!. Advertisement breaks during this time are slightly shortened, not only to allow more room in the schedules but also to allow British viewers to nip off and make a cup of tea in the interval. Volumes are similarly enhanced, so the ads can be heard in the kitchen, away from the TV.