My Go Executable Files Are Still Getting Larger (What's New in 2021 and Go 1.16)


Two years ago, my article"Why are my Go executable files so large?" showed how to utilize D3 and a tree map visualization to explore the size of executable files produced by the Go compiler.

A few things have changed since, and so an update is in order.

The sum of the sizes reported by go tool nm does not add up to the final size of the Go executable.

For example, in the CockroachDB 20.2.7 binary:

  • the file occuppies 211694984 bytes (202MiB) on disk;
  • however, the sum of symbol sizes adds up to 118928245 bytes (113MB).
  • there is a gap of 92766739 bytes (88MiB) missing, or ~44% unaccounted for.

At first I suspected that this size was occupied by the symbol table itself, or the debugging information. To check this, we can use strip to remove the symbol table and observe the difference. Alas:

  • the stripped executable size is 190680384 bytes (182MiB) on disk;
  • so there is still a gap of ~68MiB, or ~34% non-symtable data that is unaccounted for.

At this time, I do not have a satisfying explanation for this “dark” file usage.

We can see how this dark file usage has evolved throughout the growth of CockroachDB:

CockroachDB versionGoExec. size(MiB)StrippedSum nm -sizeSymtable sz.Dark bytes% dark bytes
v1.0.01.83983079238.0397996243221698431168758264019.0%
v1.0.71.83979962438.039830792323713950745939718.7%
v1.1.01.84344749641.443447496356024830784501318.1%
v1.1.91.84630020044.246300200378496420845055818.2%
v2.0.01.105438456851.954384576444632670992130918.2%
v2.0.71.105643282453.8564328324596926301046356918.5%
v2.1.01.10135223352129.06883590455212282663874481362362210.1%
v2.1.111.10136101520129.86942905654714649666724641471440710.8%
v19.1.01.11124365384118.611147012071166968128952644030315232.4%
v19.1.111.11124588560118.811168800871257435129005524043057332.4%
v19.2.01.12163978120156.414539809692535059185800245286303732.2%
v19.2.121.12165974336158.314730343293850056186709045345337632.2%
v20.1.01.13135223352129.01475944489378975105380469739.8%
v20.1.131.13167269624159.514825612094208103190135045404801732.3%
v20.2.01.13209352968199.7188618784117667098207341847095168633.9%
v20.2.71.13211694984201.9190680384118928245210146007175213933.9%
v21.1-alpha-geb1aa69bc41.15183075488174.6135352792107792826477226962755996615.1%

In this table:

  • “Exec. size” is the raw size of the executable file, in bytes.
  • “Stripped” is the size of the executable after the strip command was applied; i.e. after the symbol table and debugging information was removed.
  • “Sum nm -size” is the sum of the advertised sizes of the entries in the symbol table.
  • “Symtable sz.” is the estimated size of the symbol table itself, as deducted by taking the difference between the first two sizes. We can see that the v1.0.7 to v2.0.7 executables, as well as v20.1.0, were released pre-stripped.
  • “Dark bytes” is the gap between the raw file size and the combined sum of the advertised symbol sizes and the symbol table’s size, in bytes.
  • “% dark bytes” is the percentage of the dark bytes relative to the raw file size.

We can see that the dark size percentage was lower than 20% prior to CockroachDB v19.1, and has then been oscillating around 33% of the file size until v21.1. With the upcoming v21.1 release, using Go 1.15, the dark size is reduced to 15% again.

As explained in the previous analysis, up to and including Go 1.15 the compiler would generate a special table called runtime.pclntab inside the executable.

The purpose of this data structure is to enable the Go runtime system to produce descriptive stack traces upon a crash or upon internal requests via the runtime.GetStack API.

We can see how this table grows across Go versions until v1.13, and then decreases in v1.15:

CockroachDB versionGoExec. size(MiB)pclntab sz(MiB)% pclntab
v1.0.01.83983079238.073167267.018.4%
v1.0.71.83979962438.073180307.018.4%
v1.1.01.84344749641.481933977.818.9%
v1.1.91.84630020044.291033188.719.7%
v2.0.01.105438456851.91074541910.219.8%
v2.0.71.105643282453.81120581810.719.9%
v2.1.01.10135223352129.01436456413.710.6%
v2.1.111.10136101520129.81444535313.810.6%
v19.1.01.11124365384118.62505540323.920.1%
v19.1.111.11124588560118.82509507923.920.1%
v19.2.01.12163978120156.43361908132.120.5%
v19.2.121.12165974336158.33401091032.420.5%
v20.1.01.13135223352129.02992783328.522.1%
v20.1.131.13167269624159.53007312228.718.0%
v20.2.01.13209352968199.73613987634.517.3%
v20.2.71.13211694984201.93646796134.817.2%
v21.1-alpha-geb1aa69bc41.15183075488174.63076334529.316.8%

The large size of the pclntab was due to a choice by the Go team to store the mapping of program counters to function names uncompressed.

To paraphrase:

  • prior to 1.2, the Go linker was emitting a compressed line table, and the program would decompress it upon initialization at run-time.
  • in Go 1.2, a decision was made to pre-expand the line table in the executable file into its final format suitable for direct use at run-time, without an additional decompression step.

In other words, the Go team decided to make executable files larger to save up on initialization time.

As we discussed back then, this choice was not well warranted for network servers like CockroachDB which are executed rarely, and where the size of the program on disk matters more than the start-up time.

The publication of my article in 2019, together with the community outcry that it triggered, were actually noticed by the Go team.

The Go team subsequently decided to change course and start working on compressinig pclntab again.

We can see this change in the table above:

  • starting in Go 1.15, the pclntab is compressed again.
  • starting in Go 1.16, the pclntab is not present any more, and instead is re-computed from other data in the executable file. How exactly? Read on.

Using the source code for CockroachDB v21.1-alpha-geb1aa69bc4, we can produce custom builds across Linux and FreeBSD, with both the 1.15 and 1.16 compilers.

PlatformGoBuild modeExec. sz.pclntab sz.Dark sz.
amd64-linux1.15release18307548830763345 (17%)27559966 (15%)
amd64-freebsd1.15release (no geos)30545285630824594 (10%)27052709 (9%)
amd64-freebsd1.16release (no geos)289463288064733620 (22%)
amd64-linux1.15dev18267932030811805 (17%)27445431 (15%)
amd64-freebsd1.15dev (no geos)30545291230824594 (10%)27052769 (9%)
amd64-freebsd1.16dev (no geos)289463280064733616 (22%)

What do we see here?

The bytes previously occupied by pclntab are now part of the dark bytes, which are not in the symbol table.

Sure, the Go team can be proud that “pclntab has been reduced to zero”, but the net effect on the excutable size is not so clearly visible!

An interesting way to think about the results above is that we now have three parts of a Go executable file that do not really contribute to making a program “work”:

  • The symbol table itself (which can be stripped via strip, but is not stripped by default).
  • The pclntab, when generated.
  • The “dark bytes”, which is byte usage in the raw executable file not accounted for in the symbol table.

How many bytes remain that are “useful”? We can compute this as follows:

  • take the stripped executable, to exclude the symbol table and debugging information.
  • remove the dark size (stripped exec size, minus size of all the symbols accounted for).
  • remove the size of pclntab.
  • see how many bytes remain.

This gives us:

GoRaw sizeStrippedDark sizepclntabRemainder% remainder% non-useful
1.83983079239799624758264073167262490025862.5%37.5%
1.83979962439830792745939773180302505336562.9%37.1%
1.84344749643447496784501381933972740908663.1%36.9%
1.84630020046300200845055891033182874632462.1%37.9%
1.1054384568543845769921309107454193371784862.0%38.0%
1.10564328245643283210463569112058183476344561.6%38.4%
1.101352233526883590413623622143645644084771830.2%69.8%
1.101361015206942905614714407144453534026929629.6%70.4%
1.1112436538411147012040303152250554034611156537.1%62.9%
1.1112458856011168800840430573250950794616235637.1%62.9%
1.1216397812014539809652863037336190815891597835.9%64.1%
1.1216597433614730343253453376340109105983914636.1%63.9%
1.1313522335214759444853804697299278336386191847.2%52.8%
1.1316726962414825612054048017300731226413498138.3%61.7%
1.1320935296818861878470951686361398768152722238.9%61.1%
1.1321169498419068038471752139364679618246028439.0%61.0%
1.1518307548813535279227559966307633457702948142.1%57.9%
1.1518267932013527012027445431308118057701288442.2%57.8%
1.1530545285613346040827052709308245947558310524.7%75.3%
1.1530545291213346047227052769308245947558310924.7%75.3%
1.162894632881405138166473362007578019626.2%73.8%
1.162894632801405138166473361607578020026.2%73.8%

To summarize, early on (in Go 1.8 and before) the non-code, non-data part of an executable was less than 40% of the total executable size.

Over time, it has grown to more than 70% of the total executable size.

Even with the new pclntab replacement in Go 1.16, where pclntab is computed at run-time, there is no gain: the data used as input for the computation is stored somewhere in the executable and its total size is larger than the original pclntab even was.

These Go executable files are rather… bloated.

In our original analysis in 2019, we looked at the output of go tool nm -size and drew a tree map representation for it. This helped us detect an anomaly, in the size of a special data structure called runtime.pclntab, which was growing excessively large for no good reason.

After that article was published, the Go team decided to change course and reduce the size of pclntab (by computing it at run-time); so that it is finally absent from binaries produced by Go 1.16.

Alas!

This year, we revisited the analysis and discovered that the symbol table is not complete. There are many bytes in the binary executable that are not accounted for, neither by the announced size of objects in the symbol table, nor by the size of the symbol table itself.

We can call this the “dark file usage” of Go binaries, and it occupies between 15% and 33% of the total file size inside CockroachDB.

Sadly, the removal of pclntab in Go 1.16 actually transferred the payload to the “dark” bytes.

Moreover, if we take a step back, we realize that neither the symbol table, nor pclntab, nor the dark file usage really contribute to the functionality of a program: really, the functionality of a program comes from code+data objects, which are properly listed in the symbol table. We can call these the “non-useful bits” of the binary file.

Even more sadly, the non-useful bits have grown over the course of Go versions. In CockroachDB, they were less than 40% of a raw executable back in v1.0, compiled with Go 1.8, and have grown beyond 70% in v21.1, compiled with Go 1.16.

That’s right! More than two thirds of the file on disk is taken by… bits of dubious value to the software product.

Moreover, consider that these executable files fly around as container images, and/or are copied between VMs in the cloud, thousands of times per day! Every time, 70% of a couple hundred megabytes are copied around for no good reason and someone needs to pay ingress/egress networking costs for these file copies. That is quite some money being burned for no good reason!

Copyright © 2021, Raphael ‘kena’ Poss. Permission is granted to distribute, reuse and modify this document according to the terms of the Creative Commons Attribution-ShareAlike 4.0 International License. The original article can be found here. To view a copy of this license, visit http://creativecommons.org/licenses/by-sa/4.0/.