GMP and assert()

Please consider subscribing to LWN

Subscriptions are the lifeblood of If you appreciate this content and would like to see more of it, your subscription will help to ensure that LWN continues to thrive. Please visit this page to join up and keep LWN on the net.

By Jake Edge
February 27, 2019

A report of a potential security problem in the GNU Multiple Precision Arithmetic (GMP) library was met with a mixed reaction, from skepticism to responses verging on hostility, but the report ultimately raised a question worth pondering. What role should assertions (i.e. calls to the POSIX assert() macro) play in error handling? An assertion that fails leads to a process exit, which may not be what a developer calling into a library expects. Unexpected behavior is, of course, one step on a path that can lead to security holes.

Jeffrey Walton posted a message with "Asserts considered harmful (or GMP spills its sensitive information)" as its subject line to the oss-security mailing list right at the end of 2018. He noted that GMP uses assert() for error checking, which could lead to a number of undesirable effects. Software calling GMP might well be handling encryption keys or other sensitive information, which could be exposed via core dumps or other means (e.g. error-reporting services) due to the abort() call made in response to the failing assertion.

As the subject of the mail might indicate, the tone of the message was a tad combative and, perhaps, a bit condescending. That may help explain some of the reactions to it. Walton's message begins as follows:

The GMP library uses asserts to crash a program at runtime when presented with data it did not expect. The library also ignores user requests to remove asserts using Posix's -DNDEBUG. Posix asserts are a [debugging] aide intended for [development], and using them in production software ranges from questionable to insecure.

Many programs can safely use assert to crash a program at runtime. However, the [prerequisite] is, the program cannot handle sensitive information like user passwords, user keys or sensitive documents.

High integrity software, like GMP and Nettle, cannot safely use an assert to crash a program.

He goes on to describe what happens when an assertion fails and how that can lead to sensitive information leaking. He also includes an example that uses the Nettle cryptographic library (which calls into GMP) to trigger an assertion, crash, and core dump. It turns out that the assertion is actually caused by a bug in Nettle that has since been fixed. Walton built Nettle and GMP with -DNDEBUG, which is meant to disable assert(). As noted above, that did not disable assert() in GMP, but did work for Nettle. However, that tickled a bug in one of the Nettle test cases, which caused the GMP assertion to fail. Discovering that led to some dismissive responses, but the larger point still stands: should applications expect libraries to abort()?

Most in the thread argued that the danger of trying to continue past a failed assertion could be worse than the information leak. Security-sensitive programs should already be ensuring that no core dumps are produced and that any abort()-reporting services are disabled. In addition, as Vincent Lefèvre pointed out, there are plenty of other ways for a program to crash (e.g. segmentation fault) that could lead to the same information leak. So programs handling sensitive information need to be prepared for that in any case.

The crux of Walton's argument is that the assertions would be better handled by returning an error rather than crashing (or attempting to crash, since programs could be set up to handle the SIGABRT). That may be true but it would change the API and, perhaps, slow down the library, Lefèvre said.

Moreover, some asserts may come from the detection of an inconsistent state. In this case, it is better to abort. Otherwise letting the program continue may have worse consequences.

The developer of Nettle, Niels Möller, agreed: "Crashing in a controlled fashion may also be *more* secure that continuing execution with undefined results. Depending on circumstances, of course." He went on to say that Walton's perspective differed from his own:

I read the general statement "asserts considered harmful" as your personal [opinion], likely based on experience with very different development projects than I'm involved with.

In the final analysis, developers of programs handling sensitive information need to be more careful than other developers. One could argue that those developers should be familiar with the behavior of any libraries that they use, so that assertion failures are not unexpected. Beyond that, as GMP developer Torbjörn Granlund pointed out, the API of a library is determined by its developers; other projects are free to use the library—or not—at their discretion.

(Log in to post comments)