Tales From the Xenix Crypt

By Michal Necasek

What does Xenix have to do with the Enigma machine? Perhaps surprisingly, there is a clear connection…

When reconstructing 386 Xenix 2.2.3, the libmdep.a library proved to be a particularly tough nut to crack. That is because it’s one of the three files in a standard Xenix install (/xenix, /etc/getty, /usr/sys/mdep/libmdep.a) which need to be “branded”. Let’s back up a bit and describe the SCO Xenix copy protection.

The Xenix disks did not use any kind of physical copy protection. The media could be backed up and copied without any technical difficulties. The copy protection consisted of a serial number and an activation key which needed to be used in order to “brand” a Xenix installation before it could be used. The branding process wrote a scrambled copy of the serial number into the files, and for files other than the kernel, also decrypted the data which was shipped encrypted on the installation disks.

The installation disks were no good without the serial number and activation key, and if files installed on hard disk were copied, they would contain the serial number and could thus be traced to the source. The mechanism was just complex enough that it was not trivial to defeat, although from a modern perspective the cryptography used was laughably weak.

The Xenix kernel was special in that it was branded but not encrypted, but without a valid serial number severely restricted the number of concurrent processes. It was just good enough to install the OS from floppies but not enough to run a multi-user system.

Now back to libmdep.a. On the restored disks, one sector near the beginning of the library was missing. It turned out that reconstructing the data was not that hard: The missing 512 bytes covered part of the symbol directory and part of the OMF header of the first object file. The OMF header was recreated by copying it from another file and tweaking slightly, and ar had no trouble recreating the symbol directory. So we had a working libmdep.a but we needed an encrypted library which the installation process can decrypt.

Now, Xenix comes with /etc/brand which does what the name suggests. It writes the serial number into files and decrypts files which are shipped encrypted. But when renamed to debrand, the exact same binary turns around and encrypts files (as it happens, the encryption is symmetric). So obviously the debrand utility can be used to encrypt the reconstructed library for placement on the installation media. But just as obviously, debrand needs some kind of key or password, and that is not the serial number or activation key. Rather, it is a separate 3-character code. Which we don’t have.

By the way: Astute readers may be wondering how the serial number branded into the kernel gets propagated when the kernel is rebuilt (using the so-called Link Kit). That’s precisely where the libmdep.a library comes in. As it turns out, libmdep.a contains an object module which has the serial number branded into it as well, and that’s how a freshly built kernel still contains a serial number.

Brute Force

So, how hard can it be to break that 3-character code by brute force? Not that hard. 3 characters is really not much. The approach was very simple. Run debrand with successive iterations of the code on a an encrypted file (essentially libmdep.a or getty). Check if the first kilobyte or so matches the encrypted file on the distribution disk. Automate with a shell script.

After a long while of writing the script (courtesy of Michael) and a short while of running it in a VM, bingo! The 3-character code is ‘bkb’. We can encrypt the reconstructed library and copy the missing sector to the N4 disk.

It is worth noting that any given set of Xenix floppies is encrypted with a given 3-character code. There is a large number of serial number + activation key combinations which are tied to the same 3-character code and can unlock the installation media. Obviously SCO didn’t want to produce different disks to go with each serial number. This way many serial numbers could be handed out for a given product, yet keys for one product could not be used to unlock a different one.

Digging Deeper

But how does the encryption really work? It was fairly clear that it processes 256-byte blocks and that it wasn’t the DES encryption traditionally used for Unix passwords. Did SCO invent something of their own? That would be very un-Unix-like, and of course they didn’t. What they did was take the crypt command and adapt the encryption algorithm slightly. Note that crypt was not shipped with Xenix due to U.S. export regulations, but was available on request, and SCO obviously had the source code.

One clue was that at least some versions of /etc/brand were built from source files that among others included crypt.c. And after some digging, this turned up. “A one-rotor machine designed along the lines of Enigma but considerably trivialized”, with a 256-byte “rotor”.

Xenix uses the same algorithm with two noteworthy modifications. It always works on 256-byte blocks, and it uses the 3-character code (extended to 5 characters with two constant characters) directly in place of ‘buf’ in the setup code.

With the algorithm in hand, we can construct a much better key finder. When dealing with encrypted files like libmdep.a, how does /etc/brand know the serial number + activation key combination is the right one? That is actually a two-step process.

First, the given serial number and activation key are slightly transformed and a checksum is used to verify that they form a valid pair. The 3-character code is effectively part of the activation key.

The second step is decrypting the first 256-byte data block, calculating its checksum, and comparing it with the checksum stored in the encrypted file. If the checksum matches, the code is considered valid and the rest of the file is decrypted.

Now it’s trivial to try various 3-character codes (lowercase only!), decrypt 256 bytes, calculate the checksum, and see if it matches. On a modern PC, it takes less than a second to go through the entire 3-character key space. And interestingly, there always seem to be two valid 3-character codes which can unlock a given file. That may be a weakness in the algorithm used.

Final Touch

The last hurdle was generating an activation key given a serial number and a 3-character code. As it turns out, there are multiple valid activation keys. The first three characters of the activation key correspond to the 3-character code, the second three are usually unused, and the last two are derived from a checksum which covers the serial number and the first 6 characters of the activation key. The activation key is then scrambled using a relatively simple (but non-obvious) algorithm which nicely obscures the 3-character code.

With an activation key calculator at hand, it is possible to decrypt those 386 PS/2 Xenix disks, for example, even though the kind person who uploaded the images didn’t provide the serial number and activation key. By the way, the 3-character code for those disks is, very fittingly, ‘ibm’.

An unusual Xenix serial number

It is also possible to produce personalized license plates, as seen above.

What Have We Learned?

If there’s any lesson to be learned, it’s probably that 30-year old copy protection is relatively easy to break using tools and computing power that did not exist 30 years ago.