Generating Software Tests

by Andreas Zeller, Rahul Gopinath, Marcel Böhme, Gordon Fraser, and Christian Holler

Welcome to "Generating Software Tests"! Software has bugs, and catching bugs can involve lots of effort. This book addresses this problem by automating software testing, specifically by generating tests automatically. Recent years have seen the development of novel techniques that lead to dramatic improvements in test generation and software testing. They now are mature enough to be assembled in a book – even with executable code.

from fuzzingbook_utils import YouTubeVideo

This work is designed as a textbook for a course in software testing; as supplementary material in a software testing or software engineering course; and as a resource for software developers. We cover random fuzzing, mutation-based fuzzing, grammar-based test generation, symbolic testing, and much more, illustrating all techniques with code examples that you can try out yourself.

This book is written by Andreas Zeller, Rahul Gopinath, Marcel Böhme, Gordon Fraser, and Christian Holler. All of us are long-standing experts in software testing and test generation; and we have written or contributed to some of the most important test generators and fuzzers on the planet. As an example, if you are reading this in a Firefox, Chrome, or Edge Web browser, you can do so safely partly because of us, as the very techniques listed in this book have found more than 2,600 bugs in their JavaScript interpreters so far. We are happy to share our expertise and making it accessible to the public.

Frequently Asked Questions

The interactive notebook uses the service, which runs notebooks on their own servers. Starting Jupyter through normally takes about 30 seconds, depending on your Internet connection. If, however, you are the first to invoke binder after a book update, binder recreates its environment, which will take a few minutes. Reload the page occasionally. imposes a limit of 100 concurrent users for a repository. Also, as listed on the status and reliability page,

As is a research pilot project, the main goal for the project is to understand usage patterns and workloads for future project evolution. While we strive for site reliability and availability, we want our users to understand the intent of this service is research and we offer no guarantees of its performance in mission critical uses.

There are alternatives to; see below.

If does not work or match your needs, you have a number of alternatives:

  1. Download the Python code (using the menu at the top) and edit and run it in your favorite environment. This is easy to do and does not require lots of resources.

  2. Download the Jupyter Notebooks (using the menu at the top) and open them in Jupyter. Here's how to install jupyter notebook on your machine.

  3. If you want to use the book in a classroom, and depend on your users having access to the interactive notebooks, consider using or deploying a JupyterHub or BinderHub instance.

Can I run the code on my Windows machine?

We try to keep the code as general as possible, but occasionally, when we interact with the operating system, we assume a Unix-like environment (because that is what Binder provides). To run these examples on your own Windows machine, you can install a Linux VM or a Docker environment. Instructions on how to do this are welcome.

Technically, yes; but this would cost money and effort, which we'd rather spend on the book at this point. If you'd like to host a JupyterHub or BinderHub instance for the public, please do so and let us know.

The contents of this book will include topics such as:

  1. Introduction to Testing
  2. Basic Fuzzing
  3. Coverage
  4. Mutation-Based Fuzzing
  5. Fuzzing with Grammars
  6. Efficient Grammar Fuzzing
  7. Grammar Coverage
  8. Configuration Fuzzing
  9. Parsing and Mutating Inputs
  10. Probabilistic Testing
  11. Reducing Failure-Inducing Inputs
  12. Fuzzing Function Calls
  13. Fuzzing User Interfaces
  14. Search-Based Testing
  15. Symbolic Testing
  16. Mining Grammars
  17. Fuzzing and Invariants
  18. Protection and Repair

See the table of contents in the menu above for those chapters that are already done.

Thanks for referring to our work! Once the book is complete, you will be able to cite it in the traditional way. In the meantime, just click on the "cite" button at the bottom of the Web page for each chapter to get a citation entry.

We're always happy to get suggestions! If we missed an important reference, we will of course add it. If you'd like specific material to be covered, the best way is to write a notebook yourself; see our Guide for Authors for instructions on coding and writing. We can then refer to it or even host it.

Of course! Just respect the license (including attribution and share alike). If you want to use the material for commercial purposes, contact us.

Yes! Again, please see the license for details.

We have successfully used the material in various courses.

  • Initially, we used the slides and code and did live coding in lectures to illustrate how a technique works.

  • Now, the goal of the book is to be completely self-contained; that is, it should work without additional support. Hence, we now give out completed chapters to students in a flipped classroom setting, with the students working on the notebooks at their leisure. We would meet in the classroom to discuss experiences with past notebooks and discuss future notebooks.

When running a course, do not rely on – it will not provide sufficient resources for a larger group of students. Instead, install and run your own hub.

Download the Jupyter Notebooks (using the menu at the top) and adapt the notebooks at your leisure (see above), including "Slide Type" settings. Then,

  1. Download slides from Jupyter Notebook; or
  2. Use the RISE extension (instructions) to present your slides right out of Jupyter notebook.

At this point, we do not provide support for PDF versions. We will be producing PDF and print versions once the book is complete.

We prioritize issues as follows:

  1. Bugs in code published on
  2. Bugs in text published on
  3. Writing missing chapters
  4. Issues in yet unpublished code or text
  5. Issues related to development or construction
  6. Things marked as "beta"
  7. Everything else

We're glad you ask that. The development page has all sources and some supplementary material. Pull requests that fix issues are very welcome.

Again, we're glad you're here! We are happy to accept

  • Code fixes and improvements. Please place any code under the MIT license such that we can easily include it.
  • Additional text, chapters, and notebooks on specialized topics. We plan to set up a special folder for third-party contributions.

See our Guide for Authors for instructions on coding and writing.

Creative Commons License The content of this project is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License. The source code that is part of the content, as well as the source code used to format and display that content is licensed under the MIT License. Last change: 2018-10-29 14:49:10+01:00CiteImprint

Andreas Zeller, Rahul Gopinath, Marcel Böhme, Gordon Fraser, and Christian Holler: "Generating Software Tests". Retrieved 2018-10-29 14:49:10+01:00.

@incollection{fuzzingbook2018:index, author = {Andreas Zeller and Rahul Gopinath and Marcel Böhme and Gordon Fraser and Christian Holler}, booktitle = {Generating Software Tests}, title = {Generating Software Tests}, year = {2018}, publisher = {Saarland University}, howpublished = {\url{}}, note = {Retrieved 2018-10-29 14:49:10+01:00}, url = {}, urldate = {2018-10-29 14:49:10+01:00}