Why I wouldn’t choose S3 bucket notifications again

By Christian Bandowski

Christian Bandowski

We are building a data hub within AWS where we obviously use several S3 buckets. As the whole data hub is built upon serverless technologies we also use several Lambdas and sometimes they get triggered when new files arrive in one of our S3 buckets.

Bucket notifications

AWS offers S3 bucket notifications for sending events to Lambdas and also other services like SQS and SNS. In plenty of places, we are using this to invoke our Lambdas which then can act upon these new files.

But wait…

… when I am happy with it, why wouldn’t I choose S3 bucket notifications again? The main reason is exactly this:

Photo by Kind and Curious on Unsplash

Solution A: Use AWS SNS

One of the often-mentioned solutions is to use AWS SNS for any kind of event fanout. I will not go into details, because you can find this solution described in the following blog post:

  1. SNS will forward the S3 event as an SNS message, meaning that the target will receive an SNS message where the message attribute contains the whole S3 event as a JSON String payload. Meaning you first have to unwrap this to get the S3 event and your target must be aware of your infrastructure.

Solution B: Use AWS EventBridge

What do you want to forward? Exactly. Events! And AWS EventBridge even has “event” in its name, so that’s the way I recommend it!

  • CloudTrail requires a logging bucket, so every event will be written to a bucket even if you just need the event in EventBridge, of course, you can set up a retention policy, but I also think this is an avoidable overhead.
  • Pricing.. okay, both solutions are most likely really cheap so most likely the development costs are much higher. But in general, I expect CloudTrail + a logging bucket will be more expensive than just invoking a Lambda that will not need much memory and will finish work after a few milliseconds. But I didn’t yet calculate the costs, so if I am wrong with it let me know 😄.

Event filtering

AWS EventBridge allows filtering of the events based on event patterns, so you can easily set up a rule that will just invoke the target if certain conditions match. Compared to regular S3 bucket notification settings you can also create much more conditions, e.g. you can even restrict by object size or other metadata available in the event if you like, and also restricting on object key prefixes (=filter prefixes) works as well.

Why I love this approach

It’s easy to set up — really.. do it as described in the AWS article or as I proposed with a small Lambda in between that you can even reuse for all your buckets and you simply have your events available in your event bus.
Because it is so simple I would always prefer this small overhead than having the possibility open that I, later on, have to migrate to a solution like this when I want to add one more notification target to a bucket.

  • More targets: EventBridge allows routing your events to way more targets than the regular S3 bucket notifications do.
  • Cross-account: Our system is split over several AWS accounts and EventBridge lets you connect your event buses so that you can easily forward events to a different account and act on them in this one.
  • Hiding infrastructure: Event rules will not just decide which events should be forwarded, they also allow you to transform the event and forward just the input you want to forward. Meaning things like unwrapping events (as required if SNS is used) are not required and you can even reformat the whole event and e.g. just forward the object key and not the whole event. Thus your target doesn’t even have to know that this event was triggered by S3 itself.