Injecting Chaos to AWS Lambda functions with Lambda Layers— RELOADED

By Adrian Hornsby

As in the previous version of the FailureInjection layer, we can create a python decorator and inject some delay (in Millisecond) when the Lambda function executes.

And indeed, invoking the Lambda function decorated with @corrupt_delay as follows

@corrupt_delaydef lambda_handler(event, context): return { 'statusCode': 200, 'body': 'Hello from Lambda!'

}

produces the following execution result

{ "statusCode": 200, "body": "Hello from Lambda!"

}

You should also notice the Log output:

START RequestId: 5295aa0b-...-50fcfbebea1f Version: $LATESTdelay: 400, rate: 1Added 400.61ms to lambda_handlerEND RequestId: 5295aa0b-...-50fcfbebea1f

REPORT RequestId: 5295aa0b-...-50fcfbebea1f Duration: 442.65 ms Billed Duration: 500 ms Memory Size: 128 MB Max Memory Used: 79 MB

You can see that the Duration is indeed approximately the value of delay.

As explained in AWS Lambda Function Errors, if your Lambda function raises an exception, Lambda recognizes it and serializes the exception information into JSON and returns it. Consider the following example:

Invoking this function will raise an exception and Lambda will return the following error message:

{ "errorMessage": "I failed!", "errorType": "Exception", "stackTrace": [ " File \"/var/task/lambda_function.py\", line 2, in lambda_handler\n raise Exception('I failed!')\n" ]

}

Keep in mind that depending on the event source, Lambda may retry the failed Lambda function. For example, if Kinesis is the event source, Lambda will retry the failed invocation until the Lambda function succeeds or the records in the stream expire.

Moreover, if the invocation type is asynchronous, Lambda will not return anything. Instead, it will log the error information to CloudWatch Logs.

We can use that exception behavior of Lambda to create a python decorator and inject failure when the Lambda function executes.

And indeed, invoking the Lambda function decorated with @corrupt_exception as follows

@corrupt_exceptiondef lambda_handler(event, context): return { 'statusCode': 200, 'body': 'Hello from Lambda!'

}

produces the following execution result:

{ "errorMessage": "I really failed seriously", "errorType": "Exception", "stackTrace": [ " File \"/opt/python/chaos_lib.py\", line 76, in wrapper\n raise Exception(exception_msg)\n" ]

}

Distributed systems and micro-services are most of the time vulnerable to unexpected failure from services they depend on. Being able to simulate different HTTP status code will most definitely help you build more resilient application.

I have found that injecting errors such as 400 Bad Request , 500 Internal Server Error and 503 Service Unavailable often help better understand how the application responds to various failure conditions.

To modify the HTTP status code of Lambda response, we can use the following python decorator:

And indeed, invoking the Lambda function decorated with @corrupt_statuscode as follows

@corrupt_statuscodedef lambda_handler(event, context): return { 'statusCode': 200, 'body': 'Hello from Lambda!'

}

produces the following execution result:

{ "statusCode": 404, "body": "Hello from Lambda!"

}