IBM and its large mainframe computers ruled the computer industry for decades. But during the 1980s, mainframes faced increasing competition from microprocessors, workstations, and super-minicomputers. To meet this challenge, IBM pushed technology to the limit to create the ES/9000 in 1991, a family of powerful mainframes with a price tag to match, from $70,500 up to $22 million. The processor of the ES/9000 wasn't a single chip, but a metal and ceramic package called a Thermal Conduction Module (TCM) that held 121 chips. Recently, Dave Jones of EEVBlog created a popular teardown video of a TCM, showing its complex construction. After disassembling the module, he kindly sent me some of these cutting-edge chips to analyze. In this blog post, I examine the circuitry inside one of these logic chips from the ES/9000.
Detail of a bipolar logic chip from the ES/9000 computer. This closeup of the die shows the four layers of metal and the transistors underneath. Click this photo (or any other) for a larger version.
The ES/9000 family of computers consisted of three lines with performance spanning two orders of magnitude: small entry-level systems for an office, mid-range air-cooled systems (below), and high-end water-cooled systems that could fill a room. The technology of the ES/9000 was very advanced for its time in many ways. Along with the ceramic thermal conduction modules, IBM created new high-speed integrated circuits with state-of-the-art transistors. At the system level, IBM introduced new operating systems as well as ESCON (Enterprise Systems Connection), a high-speed fiber-optic connection between the mainframe and peripherals. An optional cryptographic feature provided high-speed encryption in tamper-resistant hardware. Even the power supplies were innovative; the water-cooled power supplies could be swapped while the computer was running. The innovations of the ES/9000 generated numerous journal articles and patents.1
In this article, I'm focusing on the mid-range systems, known as the 9121 processors.2 This system (above) was packaged in a drab frame the size of a large refrigerator.3 It used 7.4 KVA of power, occupied 14.7 square feet of floor space, and weighed 2000 pounds. It could hold up to 1 gigabyte of memory, a large capacity at a time when personal computers typically had 1 to 4 megabytes of RAM. A typical 9121 system cost $1.5 million and had about twice the performance of a contemporary Intel 80486 computer that cost $10,000. This is a bit of an apples-and-oranges comparison, since the mainframe gave you high-speed I/O channels, fast memory access, and an advanced operating system, but it shows the dramatic price/performance advantage of microprocessors.
The TCM (Thermal Conduction Module)
One of the most interesting features of the ES/9000 was the Thermal Conduction Module (TCM) that held the integrated circuits. The high-performance bipolar chips generated a lot of heat, so IBM developed new cooling mechanisms so this computer could function without water cooling. The cut-away photo below shows a TCM with its large heat sink attached. At the bottom, some of the integrated circuit dies are visible along with the copper cooling pistons. The computer's main circuitry consists of five different TCMs.4
The TCM is surprisingly small, 5 inches (127.5mm) on a side, yet it holds 121 integrated circuits. Each integrated circuit has a spring-loaded copper piston on it to remove the heat. These pistons transfer the heat into the TCM's metal case, where the heat passes into the heat sink and then the air flow. The pistons are precision-machined to maximize contact and thus heat transfer. The module is filled with oil (visible below), which also increases heat transfer. The design of the TCM allows it to dissipate 600 watts of heat—imagine holding six 100-watt light bulbs in your hand.
The integrated circuits in the TCM are not packaged like regular integrated circuits, but consist of a silicon die soldered upside-down to the ceramic substrate, flip-chip style. This ceramic substrate is an incredible feat of engineering. It's essentially a printed-circuit board made out of ceramic, with 63 layers of wiring inside. It has over 80,000 connections on the top to the integrated circuits, 2 million vias, 400 meters of internal wiring, and 2772 pins on the bottom.
The manufacturing process for the ceramic substrate was very complex. Each ceramic sheet, the thickness of two sheets of paper (0.2mm), has tens of thousands of via holes punched in it. Next, the wiring was applied in the form of a molybdenum metal paste, forming wires just 100µm wide. The stack of 63 sheets was then laminated under heat and pressure. Next, the stack was sintered at 600°C to decompose the polymer binder, followed by hydrogen treatment at 1560°C for densification. During this process, the substrate shrank by 17%, but the millions of vias must remain aligned. After trimming and polishing, two layers of thin-film wiring were placed on top of the substrate. (The thin-film wiring allowed wiring changes to be made to the module for bug fixes.)5 Finally, the module was protected with a layer of polyimide film, with thousands of openings burned in it with a laser for the chip's connections.
The bipolar logic chip
Most of the chips on the TCM are bipolar logic chips; these are the square black chips in the previous photo. The die photo below shows one of these logic chips, 6.5mm on a side.8 This chip has an unusual appearance because it was connected directly to the substrate instead of the typical approach of putting pads around the perimeter with bond wires attached. The black circles are the 549 solder balls in a 27×27 grid that connect the chip to the substrate. Of these connections, 228 of these are used for signals, while 321 are used for power. The chip is covered with metal conductors that connect the solder balls to the circuitry underneath.
Die photo of a bipolar logic chip, showing the solder balls. (Click for a larger version.)
The chip is built from a type of transistor called the bipolar transistor, an older type of transistor than the MOS transistors in modern processors. The transistors in this chip used a cutting-edge design with a complex internal structure.6 IBM used bipolar transistors because they provided higher performance at the time, but they had the disadvantages of using higher power and taking up more area on a chip. (This is why the chip needed 321 connections for power and why the ES/9000 required multi-chip modules with a complex cooling system.) The chip contains approximately 85,000 transistors, 40,000 resistors, 10,000 capacitors, and 1000 Schottky diodes. While this may seem like a large number, contemporary CMOS microprocessors (such as the Intel 486) contained over a million transistors, illustrating the much higher density of MOS transistors.7
As shown in the closeup photo below, the chip has four layers of metal wiring on top of the silicon, a lot of layers for the time. The metal layer on top of the chip (called M4) provides power and signal distribution from the solder bumps. Underneath, layer M3 provides horizontal wiring: thick lines to distribute power across the chip and thin lines for signals. Layer M2 provides vertical wiring for both power and signals. The bottom layer (M1) implements the local wiring of the gate circuitry, connecting the transistors and resistors together. The narrowest metal lines are 1.6µm wide. Power distribution uses a hierarchy: the numerous solder balls feed power into the very wide power lines in the top metal layer. These are interconnected with the wide horizontal lines, which connect to the thinner vertical lines, which connect to the circuitry. This hierarchy ensures that voltage drop is minimized across the chip, while providing the multi-amp current it requires.
The chip has four layers of metal. The silicon circuitry is visible underneath, somewhat obscured by the multiple layers of insulating silicon dioxide and silicon nitride on top.
The architecture of the chip is IBM's "master slice" approach, building the chip from a gate array of identical cells. To avoid the expense of creating fully-custom chips, IBM built the various logic chips from a common grid of cells that was customized by the wiring on top. In the photo above, you can see some of these cells underneath the metal. The master cell approach has the disadvantage of being less dense than a custom chip. It turns out that roughly half of the cells in each logic chip went unused because the number of I/O pins on the chip was too small.12 You can see that most of the cells are unused in the photo above; while the transistors and resistors are present, they aren't connected to anything.
The chip contains 5240 cells, capable of implementing 2620 DCS logic gates. The structure of a cell is shown below. The cells are very flexible: each cell can implement one gate in the ECL (Emitter-Coupled Logic) family,9 two gates in the NTL (Non-Threshold Logic) family,10 or half a gate in the DCS (Differential Current Switch) family (which this chip uses). The key components are the transistors, which I've colored blue. The resistors are colored yellow.11 At the top are two large capacitors (red). The capacitors are unused in this DCS circuitry, but can be used to speed up ECL gates.
The image below shows six of the chip's 5240 cells after removing the metal layers from the chip. You can see how the layout matches the diagram above. (The cells in the middle are upside down.)
Closeup of the logic cells. I stacked multiple photos after removing the metal layers to get this image.
The logic chips are fabricated with a special technique that allows hundreds of different types of logic chips to be produced from a single set of masks. The transistors and other components in the silicon "master slice" are constructed using masks and photolithography as in most integrated circuits. However, the metal layers are patterned using direct-write electron beam lithography, rather than masks. This electron beam is steered to "write" the desired metal layer patterns on the die to produce the desired type of chip. In other words, the basic pattern of the chip is created using masks, but then the different chip types are manufactured directly from the design files, providing flexibility.
The photo below shows the entire die after dissolving the metal layers. This image shows the grid of cells, as well as three vertical rows holding 360 I/O cells.13 The grid pattern is most clear in the upper-right corner, where I sanded the die down. (Due to the difficulty of removing four layers of metal as well as layers of silicon nitride, I couldn't get the die as clean as I like.)
Die after removing the metal. The rounded corners are from my mechanical planarization processing (by which I mean sanding with 600-grit sandpaper). The original die was not rounded.
Differential Current Switch logic (DCS)
The chip is built with an uncommon logic family called DCS (Differential Current Switch).15 As the name suggests, DCS operates on differential signals: each input signal is expressed by two wires carrying both the signal and its complement. The voltage difference between the two wires represents a 0 or 1. Thus, a three-input logic gate will have six input wires, as well as two output wires.
Most logic families implement a NAND or NOR gate as their basic gate.
The basic DCS gate, however, is the SELECT operation:
it outputs either input A or input B, selected by the S input.
In other words, SELECT implements the function
if S then A else B, or in Boolean logic,
The SELECT operation is surprisingly flexible; with appropriate inputs, it can implement AND, XOR, or even a latch.14
A SELECT gate is shown below at the conceptual level. Three toggle switches are controlled by the S, A, and B inputs. These switches will pull one output to ground, while the other output will be pulled high by a resistor. Starting at the bottom, the S switch will direct the ground current to either the "A" side or the "B" side. With the switches in the indicated positions, the output will be pulled to ground, while the complemented output remains high. But if input A is set to 1, the output levels reverse, with the output pulled high. Now, suppose input S is set to 0, so the current is directed to the B side. In this case, the output is controlled by switch B. You can verify that the output matches A if S is 1 and matches B if S is 0. In other words, the circuit selects between inputs A and B, depending on the value of S. Note that this circuit generates differential outputs: both the output and its complement.
Conceptually, a DCS gate consists of toggle switches that pull one output high and the other low.
Next, I'll describe how the current switch is implemented with a pair of transistors. At the bottom, a current sink generates a fixed current, which can be switched to either the left side or the right side of the circuit. The idea is that the transistor with a higher input voltage will direct the current to that side, pulling that output low. Thus, the circuit acts like a toggle switch. An important feature of the circuit is that it provides a high degree of amplification: a slight difference in voltages is enough to switch most of the current to one side. (This circuit is essentially the same as the differential amplifier used in an op-amp.) As a result, a voltage swing of just 200 millivolts is enough to distinguish a logical 0 and 1, reducing power consumption. Another important feature of this circuit is that it is activated by the difference between the input voltages, so it is relatively insensitive to electrical noise. In other words, a voltage fluctuation that affects both inputs will cancel out, rather than causing an erroneous 0 or 1.
A 1 input switches the current through the transistor on the left. A 0 input switches the current through the transistor on the right.
The schematic below shows the implementation of a DCS gate. The three green boxes are current switches, using transistor pairs as described above. The yellow boxes are buffer circuits, called emitter followers. Two emitter followers buffer the outputs, while two more are used on the select inputs. Finally, the blue box is the current sink circuit, providing the fixed current that gets switched by the circuit.
Components of a DCS gate.
The diagram below shows this circuit in action. Starting at the bottom, the S input switches the current to the left. The A input then switches the current to the right. This current pulls the complemented output low, while the pull-up resistor pulls the output high. Note that a 0 input on A would switch the current to the other side, and thus switch the output. The B input has no effect since the current bypasses the B side of the circuit. Pulling the S input low, however, would switch the current to the B side, causing the B input to control the output. Thus, this circuit implements the SELECT operation.
Schematic of a SELECT gate, showing how the current is steered.
Reverse-engineering a DCS gate
In this section, I'll look at how a SELECT gate is implemented on the chip. The diagram below zooms in on a corner of the die, and then zooms again on one logic gate, the rectangle at the bottom. As you can see, each logic gate is very small on the die. Because this gate is at the edge of the die, it has less wiring over it so it is easier to see. Even so, the wiring layers on top partially obscure it. A DCS gate is created from four half-cells; I've highlighted the one I will discuss.
Starting from the die, zooming in on a corner and then a cell logic gate.
The components on the die can be matched against the diagram below. As before, the transistors are colored blue, the resistors yellow, and the unused capacitor red.
Below, I've indicated some of the components in the previously-highlighted half-cell. The wiring on the bottom metal layer customizes this cell for a particular function. Looking at this wiring, you can see that the emitters (E) of transistors T-5 and T-6 are connected, as are the emitters of transistors T-7 and T-8. The collectors (C) of transistors T-6 and T-8 are connected to the base of the output transistor T-12. The collector of transistor T-7 is connected to resistor R3. The wiring in the upper metal layers is shadowy and less clear. The vertical wiring along the sides provides power to the circuit. Other faint vertical wires are connected to the bases of transistors T-7 and T-8.
A half-cell as it appears on the die, with components labeled. "B" is a transistor base, "E" emitter, and "C" collector.
By studying the die closely, I traced out the circuitry for the gate and found it was a SELECT gate. The schematic below is from the patent; I modified it to match the gate I traced out. Note that IBM used its own symbol for a transistor as I've indicated at the bottom. I've marked the transistors and resistors from the photo above in red. The circuit has six transistors for testing, in the blue box.16 As you can see, one DCS gate takes a lot of components: 17 transistors and 18 resistors. This is one reason the density of the bipolar logic chips is so low.
Schematic of the DCS logic gate, as implemented on the chip. Vcc and Vee are the power supplies for the collector and emitter respectively. Vx controls the current sink. Vt is the pull-down voltage for the emitter-followers, but I'm not sure what Vt stands for. The original schematic was for an AND gate; I modified it to show a SELECT gate.
This shows the circuitry of one logic gate. Larger functional blocks such as adders were constructed by combining multiple gates. The full computer contains hundreds of thousands of these gates, implementing the processor and its control circuitry.
This bipolar logic chip illustrates the advanced technology of the ES/9000 mainframe.17 IBM pushed the limits of technology in everything from integrated circuit construction to ceramic modules to cooling systems. After all this effort, however, sales of the ES/9000 were underwhelming and couldn't slow the advance of microcomputers. Two years after the announcement, IBM had installed about 3600 of them, largely the lower-end models.18 In comparison, about 20 million personal computers were being sold per year, about 10,000 times the volume. Mainframes were 21.6% of computer industry revenue and dropping, less than half of personal computer revenue (44.5% of the industry). In 1997, IBM's bipolar processors reached the end of the road as IBM fully moved to CMOS processors.
If you're interested in the TCM, you should definitely watch Dave Jones' teardown video below, as well as the videos where he attempts to remove the chips with hot air and a heating plate before finally succeeding. Thanks to Dave for sending me the chips as well as letting me use his photos.