A few things have changed since, and so an update is in order.
The sum of the sizes reported by
go tool nm does not add up to the final size of the Go executable.
For example, in the CockroachDB 20.2.7 binary:
- the file occuppies 211694984 bytes (202MiB) on disk;
- however, the sum of symbol sizes adds up to 118928245 bytes (113MB).
- there is a gap of 92766739 bytes (88MiB) missing, or ~44% unaccounted for.
At first I suspected that this size was occupied by the symbol table itself, or the debugging information. To check this, we can use
strip to remove the symbol table and observe the difference. Alas:
- the stripped executable size is 190680384 bytes (182MiB) on disk;
- so there is still a gap of ~68MiB, or ~34% non-symtable data that is unaccounted for.
At this time, I do not have a satisfying explanation for this “dark” file usage.
We can see how this dark file usage has evolved throughout the growth of CockroachDB:
|CockroachDB version||Go||Exec. size||(MiB)||Stripped||Sum ||Symtable sz.||Dark bytes||% dark bytes|
In this table:
- “Exec. size” is the raw size of the executable file, in bytes.
- “Stripped” is the size of the executable after the
stripcommand was applied; i.e. after the symbol table and debugging information was removed.
nm -size” is the sum of the advertised sizes of the entries in the symbol table.
- “Symtable sz.” is the estimated size of the symbol table itself, as deducted by taking the difference between the first two sizes. We can see that the v1.0.7 to v2.0.7 executables, as well as v20.1.0, were released pre-stripped.
- “Dark bytes” is the gap between the raw file size and the combined sum of the advertised symbol sizes and the symbol table’s size, in bytes.
- “% dark bytes” is the percentage of the dark bytes relative to the raw file size.
We can see that the dark size percentage was lower than 20% prior to CockroachDB v19.1, and has then been oscillating around 33% of the file size until v21.1. With the upcoming v21.1 release, using Go 1.15, the dark size is reduced to 15% again.
As explained in the previous analysis, up to and including Go 1.15
the compiler would generate a special table called
inside the executable.
The purpose of this data structure is to enable the Go runtime system
to produce descriptive stack traces upon a crash or upon
internal requests via the
We can see how this table grows across Go versions until v1.13, and then decreases in v1.15:
|CockroachDB version||Go||Exec. size||(MiB)||(MiB)||% |
The large size of the
pclntab was due to a choice by the Go team
to store the mapping of program counters to function names uncompressed.
- prior to 1.2, the Go linker was emitting a compressed line table, and the program would decompress it upon initialization at run-time.
- in Go 1.2, a decision was made to pre-expand the line table in the executable file into its final format suitable for direct use at run-time, without an additional decompression step.
In other words, the Go team decided to make executable files larger to save up on initialization time.
As we discussed back then, this choice was not well warranted for network servers like CockroachDB which are executed rarely, and where the size of the program on disk matters more than the start-up time.
The publication of my article in 2019, together with the community outcry that it triggered, were actually noticed by the Go team.
The Go team subsequently decided to change course and start working on
We can see this change in the table above:
- starting in Go 1.15, the
pclntabis compressed again.
- starting in Go 1.16, the
pclntabis not present any more, and instead is re-computed from other data in the executable file. How exactly? Read on.
Using the source code for CockroachDB v21.1-alpha-geb1aa69bc4, we can produce custom builds across Linux and FreeBSD, with both the 1.15 and 1.16 compilers.
|Platform||Go||Build mode||Exec. sz.||pclntab sz.||Dark sz.|
|amd64-linux||1.15||release||183075488||30763345 (17%)||27559966 (15%)|
|amd64-freebsd||1.15||release (no geos)||305452856||30824594 (10%)||27052709 (9%)|
|amd64-freebsd||1.16||release (no geos)||289463288||0||64733620 (22%)|
|amd64-linux||1.15||dev||182679320||30811805 (17%)||27445431 (15%)|
|amd64-freebsd||1.15||dev (no geos)||305452912||30824594 (10%)||27052769 (9%)|
|amd64-freebsd||1.16||dev (no geos)||289463280||0||64733616 (22%)|
What do we see here?
The bytes previously occupied by pclntab are now part of the dark bytes, which are not in the symbol table.
Sure, the Go team can be proud that “
pclntab has been reduced to
zero”, but the net effect on the excutable size is not so clearly visible!
An interesting way to think about the results above is that we now have three parts of a Go executable file that do not really contribute to making a program “work”:
- The symbol table itself (which can be stripped via
strip, but is not stripped by default).
pclntab, when generated.
- The “dark bytes”, which is byte usage in the raw executable file not accounted for in the symbol table.
How many bytes remain that are “useful”? We can compute this as follows:
- take the stripped executable, to exclude the symbol table and debugging information.
- remove the dark size (stripped exec size, minus size of all the symbols accounted for).
- remove the size of
- see how many bytes remain.
This gives us:
|Go||Raw size||Stripped||Dark size||Remainder||% remainder||% non-useful|
To summarize, early on (in Go 1.8 and before) the non-code, non-data part of an executable was less than 40% of the total executable size.
Over time, it has grown to more than 70% of the total executable size.
Even with the new
pclntab replacement in Go 1.16, where
pclntab is computed at run-time, there is no gain: the data used
as input for the computation is stored somewhere in the executable and
its total size is larger than the original
pclntab even was.
These Go executable files are rather… bloated.
In our original analysis in 2019, we looked at the output of
nm -size and drew a tree map representation for it. This helped us
detect an anomaly, in the size of a special data structure called
runtime.pclntab, which was growing excessively large for no good reason.
After that article was published, the Go team decided to change
course and reduce the size of
pclntab (by computing it at run-time); so
that it is finally absent from binaries produced by Go 1.16.
This year, we revisited the analysis and discovered that the symbol table is not complete. There are many bytes in the binary executable that are not accounted for, neither by the announced size of objects in the symbol table, nor by the size of the symbol table itself.
We can call this the “dark file usage” of Go binaries, and it occupies between 15% and 33% of the total file size inside CockroachDB.
Sadly, the removal of
pclntab in Go 1.16 actually transferred the
payload to the “dark” bytes.
Moreover, if we take a step back, we realize that neither the symbol
pclntab, nor the dark file usage really contribute to
the functionality of a program: really, the functionality of a program
comes from code+data objects, which are properly listed in the symbol
table. We can call these the “non-useful bits” of the binary file.
Even more sadly, the non-useful bits have grown over the course of Go versions. In CockroachDB, they were less than 40% of a raw executable back in v1.0, compiled with Go 1.8, and have grown beyond 70% in v21.1, compiled with Go 1.16.
That’s right! More than two thirds of the file on disk is taken by… bits of dubious value to the software product.
Moreover, consider that these executable files fly around as container images, and/or are copied between VMs in the cloud, thousands of times per day! Every time, 70% of a couple hundred megabytes are copied around for no good reason and someone needs to pay ingress/egress networking costs for these file copies. That is quite some money being burned for no good reason!
Copyright © 2021, Raphael ‘kena’ Poss. Permission is granted to distribute, reuse and modify this document according to the terms of the Creative Commons Attribution-ShareAlike 4.0 International License. The original article can be found here. To view a copy of this license, visit http://creativecommons.org/licenses/by-sa/4.0/.