Useful Notes / Beta Test
In the software development world, progress is spread out into various milestones. The Beta Test
is the most often heard one of the public, which represents software that is mostly done, but needs to be tested beyond its developers and internal testers to catch bugs. The idea is to give the software another set of eyes to try different things, after all, a developer who's been working on one part knows exactly what to do and wouldn't normally try to do what not to do or has the time to do it.
Software Release Cycle
The development stages of software generally go as follows, this being the road map that IBM laid out
in the 1950s:
Pre-Alpha stages of development narrow down the requirements of the software and the design to meet those requirements. It answers the questions of "what should this software do?" and "how should this software do it?" Test plans are also made so that these features can be tested empirically. Sometimes a Tech Demo
is developed in this phase to give a proof-of-concept to show off at trade shows and meet ups like Game Jams.
If we were developing a game, this might be the early stages of a game engine. We don't know what we'll do with it, but it'll show off some nice things that we could do.
Implementation of the software has started. Requirements can be adjusted to address issues brought up during implementation. For example, a requirement may say "this game must support two players", but in reality the game and hardware can support four players. Software starts out crude with many issues in the mix, but as time goes on, most of the components start meshing well together. Testing is usually done by the developers and only on the components they're working on. They're also at best, spot checked. This means if the developer is working on, for example, the UI system, they only test the UI part and make sure the correct actions works. They may not discover that clicking the same button 50 times results in the UI breaking.
In our game development scenario, we've decided on a racing game. At the start, it may not have many maps, the AI is not there, the physics may be crude and rudimentary. But features will continually be added in.
Implementation of individual pieces slow down and bringing those pieces together, known as integration, starts. Requirements are even more rigid, limited to minor adjustments to avoid major changes. At this point, the software is stable enough for others to test features and find bugs. Graphical assets though may or may not be ready. However, these can to be dropped in since what the development team is worried about at this point is getting the software to work.
In a racing game, the AI will at least follow the track, the car's handling behaves believably, but something silly like driving backwards across the finish line will result in a win may slip through the cracks. Textures may also be basic or not there, models will be boxes, and the sound may as well be one of the audio engineers making engine noises.
Later in the beta phase all major and most or all minor features as well as graphical and audio assets are in the game and it looks like a releasable product. They may still be incomplete or buggy. Requirements are frozen at this point.
The software is handed off to system testers to play with. For games, this could be an internal team of play testers or made public as an open beta. This is intended to find more obscure bugs, balance or tune features, and otherwise ensure the software works as intended. In a racing game, the AI may still be goaded into stupid decisions, a certain part of the wall may not have collision, but almost all tracks exist and the cars (mostly) stay on the track. Major features that are troublesome will be cut or delayed at this point. If a cosmetic feature to decorate the cars in a custom color crashes the game and the schedule is closing in, it may be cut for good or added later in a patch once the bug is figured out.
Note that according to the original IBM test specification, this is the stage at which all
features are to be complete with no room to add anything else, although modern software developers are sometimes a bit more flexible with this, particularly when a feature can demonstratively improve the final product. If someone claims a product is in "beta" with major features missing, however (using the previous example, a racing game that's supposed to feature multi-level tracks but only one clipping plane works) then it isn't really in beta. This happens with depressing regularity.
of the features have been implemented. This is, theoretically, a product that you could sell immediately, thus the term "release candidate". Testing that goes on in here is system wide and higher level. If there are any new bugs found, only show stopping bugs will be addressed as changing anything
in the software means everything
will have to be retested. Minor bugs will be left in, either to be fixed with a patch, later release, or a workaround.
Certain members of the public may get copies. In games, journalists and some publications, like Prima and Brady Games, will get preview copies. In other software, large corporate clients may receive advance copies so they can ensure their custom software still works with the newer version.
In our racing game, all tracks and cars are in game and look like they will when released. The minor clipping through a barrel will not be fixed, but the game crashing due to rolling over the timer will be. If the crash happens because the car takes a nose dive at a particular out-of-the-way location, it won't be fixed and may be silently patched later.
A trend that started in the late 2000's was publishers putting a stop to the use of the word "Demo", instead changing to the use of "Public Beta". Publishers and creators realized that by changing the name, people who would otherwise have refused to purchase a game with a bad "Demo", would now forgive the developer for a "Beta" under the mistaken impression that the "Beta" would be used to improve the game, or that the player could choose to believe that it was "just a beta" and that it would be magically fixed upon release.
Release to Manufacturer/Gold
The development state of the software is "tagged", or backed up and frozen. Any changes from hereon out has to be made to a new version. Because of the lead time needed to get the software to distributors, this may happen weeks before the actual launch date.
A particular note in the game development industry are "day one patches." These are patches that are released on the same day as the game. One might wonder if there's already a patch out, why wasn't it included in the final game? As the game needs lead time to get to the distributors and for those distributors to set things up, those working on the game can continue working on fixes rather than twiddle their thumbs or work on other projects. Some minor bugs can be fixed and tested in that time time frame.
See Perpetual Beta
and Obvious Beta