The complexities of microservices, as well as where things tend to go wrong, are today largely centered around integration points between services. This shift in risk profile prompted many to rethink their test strategies and the traditional testing pyramid. Because of this, the notion of a test honeycomb emerged: a few unit tests, lots of integration tests, and a few end-to-end tests.
As we move into the world of serverless apps, complexities continue to move further out of the code. Instead, much of the complexity lies within the configuration of functions—event source configuration, timeout, memory, IAM permissions, and so on. Your functions are stateless, relying on external services to manage their application state. So the number of integration points have continued to increase as well.
In the past, we have optimized our testing strategy toward a fast feedback loop, because the deployment process was slow and time-consuming. But this is no longer the case. Functions can now be deployed and tested in seconds, so the speed of the feedback loop should no longer be your primary concern. Instead, you need to look to your tests to provide strong confidence that your serverless applications will behave correctly once they are deployed in the wild.
These changing dynamics should prompt a change in your testing philosophy accordingly. In this post, we’ll discuss how you should think about testing a serverless application, from testing the functions locally to unit, integration, and acceptance testing. We’ll also cover how to (safely) test in production.
Stages of Testing
From the moment you start writing code to implement a new feature or fix a bug, there are many stages where you can test your code. Each stage of testing covers a different aspect of the application, and together they help you build up confidence that your serverless app will work the way it’s intended to.
One of the most frequently asked questions about serverless is “How do I test my code locally?”. There are several approaches to doing this:
These tools can be useful for quickly assessing whether your code is working. But they are also limited in terms of how much confidence you can draw from them. After all, they do not simulate IAM permissions nor API authentication and are always facing an uphill battle to keep up with the latest changes in the platform.
As complexity leaves your code, the value of unit tests diminishes. There has been less need for unit tests, as the bulk of complexities is usually clustered around how a function interacts with external services.
However, if you have a piece of complex business logic, you should encapsulate it into its own module so that you can test it as a unit. You can use the same testing frameworks that you are familiar with already, such as Mocha, Jest, and Jasmine. They all work just fine.
And then there are integration tests, where we test our code against external services we depend on, such as DynamoDB or S3. These tests help catch errors in the way your code interacts with these external services. Maybe there is a bug in your DynamoDB query expression? Or maybe your expectation of the response format is incorrect?
Far too often, these external services are mocked or stubbed during integration testing. These mocks and stubs reflect assumptions about the external service’s behavior—the same assumptions used when writing the code in the first place. Because of this, the code and tests form a self-fulfilling prophecy, and the tests fail in their primary purpose—to inform you of inaccuracies in your assumptions about these external services. The two examples mentioned above would not be caught by integration tests that rely on mocks or stubs.
Therefore, you should run your code against the real DynamoDB tables or S3 buckets for the happy path. Reserve the use of mocks and stubs for failure test cases, where it’s difficult to cause the external services to exhibit the desired failure behavior.
Ultimately, your function is still just a function, one where you can invoke locally using a stubbed event and context object. Take the following function as an example.
During integration testing, you would invoke the function locally by passing in a stubbed event and context objects. The important thing is that, if the function needs to integrate with external services, then the function should be configured to talk to the real thing.
Up until now, you have only executed your function code locally. The unit tests help you build confidence that your core business logic is correct. The integration tests help you build confidence that your code interacts with external services correctly. Together, these tests help you identify problems in your code.
What else can possibly go wrong? Well, your functions might not have the right IAM permissions set up. Or maybe it’s not permissioned to talk to the DynamoDB table. Maybe the function’s timeout setting is too short. Maybe it has not been allocated enough memory. Or maybe you forgot to set up the API Gateway event source altogether! The point is, there are a lot of opportunities for misconfiguration.
You need to exercise your functions after they have been deployed to be sure that everything works as expected end-to-end. Can you imagine a car dealer delivering a brand new car to his customer without first test-driving it on the road, in the environment it’s intended to be used in?
If you’re using API Gateway and Lambda, then you should make HTTP requests against the deployed API and validate against the responses in order to achieve an end-to-end test. This is where you will catch permissions and other configuration errors that will likely be missed by unit and integration tests.
If your serverless application is used by a UI client either directly or indirectly, then you would also want to make sure that your changes are compatible with the client.
These tests might be carried out by a QA team manually. You might also have automated tests, using frameworks such as Selenium. Maybe you are running these automated tests against different devices and platforms using services such as AWS Device Farm. Or you could even be running automated visual tests too!
Whatever the case, your UI tests have not changed with serverless.
Testing in Production
Great, your code is deployed to production, but testing doesn’t end there.
We have tested our serverless application every step of the way. But, unfortunately, a lot of things can still go wrong once it has been deployed to production. AWS can experience an outage which would impact your serverless application. Any of the external services your serverless application depends upon can suffer outages as well. There are also many bugs related to scale that would only manifest themselves when the system is under load.
This is why it’s so important for you to have robust monitoring and error reporting for your applications, and tools like Epsagon are doing a great job at filling this space.
Furthermore, there is a whole discipline of Chaos Engineering, which focuses on testing the application’s ability to withstand turbulent conditions in production. It does this through a series of controlled experiments that inject small doses of failures into the system. These experiments help uncover previously unknown failure modes, giving you a fighting chance to build resilience into your system before these failures happen in production and cause irreparable damage.
So there you have it. Changing complexities in the world of serverless applications have demanded a shift in strategy when it comes to testing. Your code has become simpler, and as such the value of unit tests has diminished. In its place, you need to prioritize the testing of integration points to external services via integration tests. As much as possible, you should test against the real services instead of mocks or stubs, as these often fail to inform you of inaccurate assumptions about the external services’ behaviors, resulting in false positives.
As to testing in production, this is a very broad topic on its own and is beyond the scope of this post. If you want to read more about why you should test in production and how to do it safely, then give these posts a read: