Useful Notes / UNIX
"It's a UNIX system! I know this!"
Quirky, seemingly counterintuitive, but incredibly flexible, UNIX has gone from a little-known research operating system in the 1970s to an entire design philosophy. In its experimental days, UNIX stood in the background, influencing OSes but not making much noise on its own; that changed starting in the mid-1980s, when the first commercial UNIX products appeared, and exploded in the mid-1990s, as OSes either based directly on UNIX or following its principles came to the forefront via the World Wide Web. As of 2014, UNIX-like operating systems have finally managed to outrank Microsoft Windows in terms of usage share, with UNIX-like systems spanning pretty much all the scales and sizes of computing — from uCLinux for microcontrollers and embedded devices, to Apple iOS and Android for smartphones, to Linuxnote
(formerly OS X) for desktop computers, Oracle Solaris, HP-UX and IBM AIX for servers and mainframes.
UNIX was once considered unfriendly, terse and somewhat elitist, requiring expensive licensing and large, expensive minicomputers to run; now, almost all of UNIX's source code is freely licensedmuch longer explanation
and UNIX derivatives can be found on things as small as a smart phone note
, or as large as an IBM mainframe. It's also probably the only OS that came into existence specifically because a bored programmer wanted to play a game.
Science-fiction writer Neal Stephenson
wrote of Unix, in his essay "In The Beginning Was The Command Line"
in January 1999:
What made old epics like Gilgamesh so powerful and so long-lived was that they were living bodies of narrative that many people knew by heart, and told over and over again—making their own personal embellishments
whenever it struck their fancy. The bad embellishments were shouted down, the good ones picked up by others, polished, improved, and, over time, incorporated into the story
. Likewise, Unix is known, loved, and understood by so many hackers that it can be re-created from scratch whenever someone needs it."
Multics for the rest of us
Before diving into what UNIX is, it'd make sense to mention the OS that inspired it: Multics
. Multics was itself a research OS; it was highly ambitious, was highly secure, required expensive IBM-ish mainframes to run, and was also stuck in Development Hell
. (Eventually Honeywell-Bull would commercialize it, but that was several years off.) One of the research partners was Bell Labs, then the experimental division of the Bell System, and in 1969, they decided to leave the project. This left one programmer there, Ken Thompson, with not much else to do.
Thompson had a pet project going on Bell's Multics machine, a game called Space Travel
, but at several dollars (in 1969 money) per run, using it was not economically sound. Looking for a cheaper alternative (as well as a new project to work on), he spotted a spare DEC PDP-7
in his lab, and began porting Space Travel
to it. As he did, he found himself recreating various parts of Multics around it; eventually, he had a file system, kernel and utilities ready to go. Since it was kind of like Multics, but "neutered", he decided to call the new system "Unics". Eventually, after getting a more powerful system (a DEC PDP-11) to improve Unics and work on the first real application for it (a typesetting package that eventually became nroff
), Ken and his development partner Dennis Ritchie started setting the name in small caps, at which point it mutated into "UNIX," which would have been displayed as "Unix
" with Ritchie's formatting.
UNIX used to be trademarked by Bell Labs itself, but through several historical accidents it's now controlled by the Open Group, who publishes a "Single UNIX Specification" and grants use of the trademark to OSes that pass a verification suite based on the SUS. Many UNIX-like OSes or even descendants of UNIX are not officially UNIX: for example, BSD, while being descended from UNIX, is not "UNIX"(tm) because the developers did not pay for a trademark certification. Linux, BSD, and the other UNIX-alikes all try to comply with the POSIX standard, however.
A lot of tropes associated with UNIX from the start began here; Multics had long, self-explanatory
commands with much shorter aliases, and UNIX generally just used the shorter forms (like ls
, etc.) directly. Back in those days, your terminal was very likely going to be a clunky old Teletype 33 printing terminal, and when the terminal was capable of about 10 characters
per second, brevity counted. Also, the PDP-7 in particular didn't have much memory, and Multics' way of handling communications between running programs would have been massive overkill, so Doug McIlroy came up with "pipes", a feature even non-UNIX OSes like Windows NT use now.
Portability, C and the Berkeley connection
Not long after the first edition of UNIX was published inside Bell Labs, Ken and Dennis started work on making the system portable. In the early 1970s, this was a big deal, since almost all OSes up to that point had been written specifically for the machine they were going to run on. The idea was to write most of the kernel in a higher-level language that could be "compiled" into code for different machines, then add small bits of machine-dependent code where needed to handle things like interrupts and memory. The language Dennis invented for the project was called "C"note
, and after he and Brian Kernighan published a book on it, it became a hit outside Bell Labs and even outside the UNIX community (in fact, it's highly likely the Web browser you're reading this in has C or C++note
code in it).
Once Unix was ported to C, Bell Labs started allowing researchers at universities to study its insides. Since the Bell System was still a regulated monopoly at the time, and thus couldn't sell computers or OSes, Bell Labs would give university computer science departments access to the UNIX source code for the cost of duplication once they signed a non-disclosure agreement. UNIX became very popular in operating systems classes after this. The lecture notes of one Australian computer science professor, John Lions of the University of New South Wales, were compiled into a book (Lions' Commentary on UNIX 6th Edition, with Source Code
, aka the Lions Book
) and became a widely bootlegged underground publication and circulated
for years this way due to both the Bell System's licensing strictures and the later UNIX Wars (see below);note
it was, finally, legally published in book form 20 years later, in 1996.
The college with the most influence by far, though, was the University of California at Berkeley ("Berkeley" for short). Berkeley's Computer Science Research Group added many new features to the UNIX system, eventually creating their own version of UNIX called the Berkeley Software Distribution
, or BSD. Among the many things they did were porting UNIX from the PDP-11 to the VAX (in the process, modernizing the way it handled virtual memory), adding networking support, improving the user input/output functions significantly, and generally cleaning things up. The icing on the cake was the licensing — Berkeley put their changes under a license that allowed free distribution and modification. This meant that anyone with a Bell UNIX license could use BSD, and this also had bigger implications later on.
In the meantime, Bell Labs released what would be the last important version of "classic" UNIX, the 7th Edition (otherwise known as "Version 7" or "V7" to fans). V7 was considered by many the last true version of UNIX ever
, and through ports like Xenix found its way onto many machines not powerful enough to run BSD UNIX. There was also a full rewrite called "MINIX", aimed at teaching UNIX concepts (and, later, advanced concepts such as microkernels) to students and curious hackers that couldn't get a V7 license from Bell, and as a legal substitute for the Lions Book. This was many people's first introduction to UNIX concepts, and like BSD, it would have a huge influence later on. (MINIX 3, whose first version was released in 2005, dropped the license fees in favor of being licensed similarly to BSD. This was eventually followed by the OS borrowing code from NetBSD.)
UNIX goes commercial; GNU is born (with the Free Software Movement)
In the early 80s, it looked like things would keep going as they had, with UNIX being primarily an academic research OS, for the foreseeable future. But as usually happens in history, a major change was triggered by something that at first seemed completely unrelated. In the case of UNIX, the seemingly unrelated trigger was American antitrust law.
In 1982, AT&T (the Bell System's parent company) lost a long-running lawsuit with the US Department of Justice, and was forced to divest itself of its local telephone service companies (the "baby Bells"). In return, they were finally allowed to enter the computer industry, and commercializing UNIX was at the top of their priorities. They changed the licensing such that various parts of the system were "unbundled" or a la carte
, making a usable UNIX system much more expensive. This annoyed many people, but there was little they could do about it at the time. In the meantime, the UNIX group at Bell continued working on the original UNIX tree, which eventually became UNIX System V
, the basis of all of AT&T and its partners' commercial UNIX offerings. Most vendors ended up merging it with their own code bases (often based on Version 7) and creating their own UNIX variants such as HP-UX (Hewlett-Packard) and AIX (IBM). Direct ports of System V were available from outside porting houses like Interactive Systems.
At around the same time, a researcher at MIT's AI Lab named Richard Stallman decided he was fed up with companies founded by ex-AI-lab-members (such as Symbolics), which often built upon AI lab software and were expected to be used by the AI lab, "hoarding" their innovations and preventing sharing of code.note
His main concerns were that, once the source to various parts of an OS were "nonfree" or made unavailable, fixing or improving on them was impossible (or, at least, extremely difficult), and that this could lead to monopoly behavior and rip communities apart (such as Symbolics and the AI lab, which had lost many talented personnel). This was hastened by the death of the AI lab's own operating system, ITS (a sort-of cousin to UNIX—ITS and Multics were both derived from the early operating system CTSS, but seen as more elegant yet less portable), the abandonment of the AI lab by his friends, and the "locking up" of their successors, the Lisp Machines. He envisioned a complete operating system where all of the parts were "free software" (in analogy with "free speech")note
and could be modified and/or shared with others at will, but with the caveat that the programming code for the changes had to be shared as well. note
He called his vision "GNU", a Recursive Acronym
for "GNU's Not UNIX!", posted a manifesto describing his intentions to Usenet in 1983, and created the Free Software Foundation to oversee the effort. By 1990, the FSF would have most of the parts of GNU ready, including a compiler, utilities, libraries and such, but no kernel to run the system under.
Thus, Hurd was born. Hurd was to be a completely free and open kernel, a hybrid between a Microkernel and a Monolithic kernel based on the Mach kernel layout of BSD. However, development was slow, and even as of 2013, it is still not considered suitable for everyday use. Development slumped even further when it appeared that the Linux kernel (which we will get to later in the article) had progressed much faster to the point where, by the mid-90s, it was stable enough for production use and under the right license to integrate with the rest of the GNU parts.
On the other side of the US, in Silicon Valley, some of the Berkeley researchers, along with hardware designers from Stanford, got together to make the first computers designed specifically to run UNIX, called workstations
. The most famous of these was Sun Microsystems, named after the Stanford University Network, a part of the ARPANET, the ancestor of the Internet. Other companies, like Silicon Graphics, soon followed, and even companies that historically had ignored UNIX before (HP and IBM) got into the fold with the HP 9000 (running HP/UX) and the IBM RT-PC (running either AIX or AOS, a BSD derivative).
The UNIX Wars; Linux and the unencumbered BSDs appear
Starting around 1988, Sun and AT&T entered into an agreement to develop UNIX-based software together, and called the joint venture "UNIX International". As part of the agreement, Sun (who had been using a custom BSD variant called SunOS
up to this point) and AT&T agreed to make a "merged" UNIX that would combine the best parts of BSD and System V; this was released as System V Release 4
in 1990, with Sun's version being released under the name Solaris 2note
. Other vendors, specifically DEC, IBM, and HP, felt snubbed and formed the Open Software Foundation in protest, and began work on a UNIX derivative called OSF/1
, which (like NeXTStep and, much later, Mac OS X
), was based on the Mach microkernel and parts from 4.3 BSD.
In the meantime, the IBM Personal Computer
and its clones were taking the commercial market by storm, and the Mac
was introducing people to a new way of working that many felt no one could touch. UNIX was in a state of disarray; most vendors had proprietary changes to their UNIX builds, none of which were compatible with the others, and porting software was becoming more difficult. This inspired some of the first standards for UNIX, promulgated by the POSIX working groups and by X/Open. Getting everyone to agree would be difficult, but as the PC became more and more powerful and Microsoft began talking about making a version of Windows to compete with UNIX, the impetus to cooperate grew. By the mid-1990s, UNIX International had been disbanded, and OSF stopped development on OSF/1 (leaving DEC to maintain their own branch, renamed "Tru 64
", for their Alpha machines); OSF merged with X/Open to form the aforementioned Open Group.
At Berkeley, a programmer by the name of Keith Bostic, inspired by Stallman's GNU Project (and Berkeley's own separation of their TCP/IP code in "Networking Release 1"), came up with a large project of his own: purge the BSD codebase of proprietary AT&T code to allow Berkeley-based startups to market it without licenses from AT&T. Involving almost all of the UCB Computer Science Research Group, the project was almost completely successful, with the results being released as "Networking Release 2" or "Net/2", and two "forks" soon emerged; "BSD/OS" from startup BSDi, and "386/BSD", a free software product from Berkeley hacker Bill Jolitz.
Eventually, AT&T decided to wash their hands of the whole thing, and spun off the UNIX group as UNIX System Laboratories (USL). In one final salvo before concluding the war, USL sued Berkeley and BSDi, claiming copyright infringement and restraint of trade. The possibility of a free UNIX appearing in the near future appeared dim.
In 1991, a programmer in Helsinki, Finland, named Linus Torvalds, who was inspired by MINIX note
posted his intent to build a "little" UNIX clone (which he intended to be "just a hobby" and "nothing big like GNU"
) to Usenet, with some of the work already done. Others agreed to help, and by 1992 Linux
, as it was dubbed by one of Linus's helpers,note
was maturing quickly.
As the complete system itself incorporated GNU software, with Linus's kernel being the main original contribution, Linus made the important decision to put the Linux kernel under the same license as GNU's tools and utilities, making it attractive to developers who appreciated GNU's stance. With commercial UNIX still expensive, and BSD's future unclear, Linux grew quickly, and by 1995 had reached version 1.0; already, there were several vendors offering "distributions" ("distros" for short), or fully usable OSes incorporating the Linux kernel and GNU software at that time.note
Some of them, like Linspire, SLS and AV Linux, came and went, others such as Slackware, Red Hat (later splitting into Fedora and the commercial RHEL), SuSE, and Debian lived on, and new distros appear on the scene periodically, more on this later.
While all this was going on, Jolitz suddenly dropped support for 386BSD, resulting in a set of unofficial patches to the last version appearing. Eventually, these patches were cleaned up, improved, and became the subject of two forks: NetBSD, which focused on porting the Net/2 tapes to as many platforms as possible, and FreeBSD, which sought to continue where 386BSD left off, and make a quality BSD release intended specifically for PC compatibles. The USL lawsuit still loomed large, though, and progress was slow compared to Linux (which wasn't party to the lawsuit and could move at full speed).
Finally, USL was bought by Novell, and Novell decided to call a truce. It was decided that, with the exception of small parts of code inside the kernel, that BSD was solely under Berkeley's license, and that the UNIX trademark would be given over to X/Open. The decision meant that NetBSD and FreeBSD could return to full development, and by the late 1990s, development speed was on par with Linux. NetBSD itself would later be forked into OpenBSD, which has an emphasis on security, and FreeBSD 4.0 was forked into DragonFlyBSD, due to a disagreement over an architectural change in FreeBSD 5.0.
The Open Source Movement
Not long after this, another development in the software world had a huge impact on UNIX and the Internet. The World Wide Web was finally gaining in popularity after years of development, and the vehicle most people used to access web content was a program called "Netscape Navigator". The company behind it, also called Netscape, had placed the software under a license that allowed free usage for individuals and non-commercial organizations, but required payment for use in corporations; it was hoped that corporations would also invest in their server software, which was top-of-the-line for the time and quite expensive. The problems started when it turned out few people were registering their copies of Netscape; despite what the license said, registration was never mandatory and the program was effectively free to anyone interested. To add to this, Netscape's server software was being severely threatened by the Apache HTTP Server, which was not only free but gaining steadily in market share (and to this day is still the most popular Web server on the Internet by a wide margin, even beating Microsoft's own Internet Information Services). Finally, Microsoft introduced Windows 98 with an "Active Desktop" feature that integrated Web-browser functionality with the Windows desktop.
Strangely, Netscape was actually more upset over what Microsoft did (and the loss of affiliate revenue it would have cost them) than the fact that their server business was dying, and decided to sue Microsoft in a landmark antitrust suit. But that wasn't the only strange thing Netscape did in 1998.
Inspired by the success of Linux and the BSDs, and no doubt knowing that the company itself was in bad shape, Netscape decided to release the sources to the latest version of Communicator (amounting to Navigator plus an email client and an HTML editor) under a free license. Out of the meetings for this came a term called "open source", which attempted to provide the gist of the free software movement's goals without having to go into licensing details, making it easier for people not up on their legalese to understand what was going on.
This was a controversial move. GNU partisans have said that it draws attention away from the ideological purpose, "missing the point" as it were, whereas others have argued that the English word "free" is too often used to mean "free-of-charge" to be understood correctly, and that said misunderstanding would give the impression that the software is worthless from a business perspective.note
Either way, the term stuck, and "open source" now indicates any software license that conforms to the Open Source Definition, a set of rules based on the Debian Free Software Guidelines (Debian being a highly influential and somewhat conservative distribution of Linux).
As for Netscape, most of the company migrated to Mozilla Foundation (originally known as the Mozilla Organization),note
a non-profit founded to handle the development of a new browser based on the Communicator code. After a few false starts (including a decision to rewrite most of the client from scratch, a move that rankled some of the Netscape veterans), the new browser was in usable shape by 2003. A slimmed-down version corresponding to Navigator only (without the built-in mail/news reader or the Web page editor) was developed alongside it, and eventually became Mozilla Firefox.note
A slimmed-down version consisting of only the email/news reader eventually became Mozilla Thunderbird. note
The full-blown internet suite was originally planned to be left for dead, but said group of Netscape veterans saved it by bringing it under their wing, rechristened it Seamonkey
, and brought it back in shape. And yes, it has a cult following to this day, again mostly by Netscape veterans who prefer the idea of heaving everything under one roof instead of in separate applications.
Now, in 2013, UNIX and its clones and derivatives are more popular than ever. Apple decided to open-source the kernel and user tools for Darwin (the basis of macOS
(formerly OS X) and iOS), and even Solaris (one of the last bastions of old-school commercial UNIX) has since been opened up- just to close again after Sun was bought up by Oracle, which caused much of the community to fork the last open release into the illumos
project (and the official distro OpenIndiana
Various PC-centric distributions of Linux are giving Microsoft a run for its money for the first time in years, especially on low-end PCs that can't run Vista or 7, and a resurgent Mac (running UNIX itself) is competing with Microsoft on the high end. The BSDs are popular in server, networking and security-related applications.
Ubuntu, a distribution of Linux and spin-off from Debian, is quite popular with computer-savvy people who appreciate its ease of use and its excellent hardware support, and is even starting to dent the average userdom, despite rankling some users with some recent unpopular moves. Ubuntu was #1 for a few years after it was introduced, however it slipped from the #1 pedestal after making some decisions that were unpopular with the Linux community (the Unity desktop manager and integrated Amazon search being the two most cited offenders, though the decision to push Mir instead of Wayland as an X11 replacement was a big one with developers) and earning the ire of the Free Software Foundation for "becoming too proprietary" (though they did eventually come to an agreement
over previous problems with the relationship between Ubuntu's EULA and the OS's free components).
On the other hand, those users offended by Ubuntu's "overcommercialization" remained loyal to Linux, with most moving on to Linux Mint (which is based on Ubuntu), or on to other distros at worst. As of 2013, Mint is the most popular distro.
Yet another spin on the Ubuntu/Debian architecture is LXLE
. It is currently lesser-known than the aforementioned Ubuntu, but is gaining fame for its light-weight system requirements, and therefore gaining position on distrowatch
. At the same time, it is intended to be highly functional right out of the box, designed to surpass even Lubuntu (another light-weight variant Ubuntu). LXLE is developed with dusty-old PC machines in mind.
There are also several forks of the official Linux kernel, and other UNIX-style systems, being used on high-end electronic devices and smartphones, as well as in networking gear like Ethernet switches and routers, and in many embedded devices. For example: Apple's iOS, used on the iPhone, iPod Touch, and iPad, contains code from the Darwin base of MacOS
(formerly OS X), which itself contains code forked from FreeBSD and NeXTSTEP, which the latter itself is based on 4.3BSD. Google's Android operating system, iOS's main competitor, contains a fork of the Linux kernel at its core. Also, Sony has confirmed that the PlayStation 4
's OS is in fact a customized version of FreeBSD while its predecessor, PlayStation 3
, used a customized FreeBSD kernel alongside various proprietary libraries. Meanwhile Nintendo coily admits that the Nintendo Switch
's kernel is a customized version of FreeBSD as well, while it's API consists of a mishmash of proprietary and Android libraries. Several home routers also run Linux: either by default, or modded by way of DD-WRT, OpenWRT or Tomato. Linux even powers a number of Midway Games
' slot and gambling machines, and some arcade game manufacturers (ie Bandai Namco Entertainment
with their Wangan Midnight
series starting from WMMT4) prefer Linux to Windows Embedded due to the practically nonexistent licensing costs.
The development of "netbooks"—small, simple, low-cost, low-power laptop computers geared towards Internet functionality at the expense of gaming and multimedia—also worked in UNIX's favor. Many discarded the traditional x86 architecture, due to power consumption and cost concerns, in favor of the ARM (Advanced RISC Machine) architecture used by the vast majority of embedded devices such as cell phones. The only current version of Microsoft Windows for ARM processors is the largely incompatible "Windows RT" (a modified version of Windows 8), and the vast majority of ARM netbooks run UNIX-style OSs, usually Linux OSs. For example, Chromebooks run Chrome OS, a specialized Gentoo Linux variant.
The biggest bombshell of it all, however, is probably when Valve Software
decided to create a spin of Debian
in 2013, calling it SteamOS
and announcing that the OS will be powering its line of Steam Machine consoles, but will also be free to download for anyone interested in building their own custom Steam Machine. This after porting Steam and several games to Linux, as well as encouraging developers who're using the Steam platform to roll out Linux support as well. Feral Interactive
in particular, has taken this to heart, and has ported quite a few Square Enix titles
to SteamOS and Linux. The move is seen as encouraging, and many fans of Linux believe this to be finally signalling the rise of Linux as a serious competitor of Windows in the gaming arena.