We are going to find how you can track a memory leak using the most recent and performant tools. This article’s goal is to give an up-to-date and as-simple-as-it-can-be reference on the main steps towards tracking a memory leak. If you want to get the most out of it, I’ve added (IMHO) very useful links all along with the article.
If you still want to enjoy the read and do not have a leak, you can create one.
And if your app is still leaking, or if you found another way around your leak, comment below!
1. Use Jemalloc, or MALLOC_ARENA_MAX=2
Another quick note about Jemalloc, sometimes
LD_PRELOAD can be overridden, preventing Jemalloc from being used. We had that exact problem using
bin/qgtunnel. You can make sure that Jemalloc is running using
MALLOC_CONF=stats_print:true ruby -e '1 + 1' | grep jemalloc.
2. Is your app really leaking?
Ok, now let’s start talking about real memory leaks. Maybe. If you are reading this article, chances are that you are experiencing OOM issues, or that your app starts swapping at some point and you don’t understand why.
The first thing to do in that case should be to ask yourself: what does that memory evolution mean? Question to which I know 3 answers:
- My app is too big for my available RAM
- I have a memory bloat
- I have a memory leak
Fortunately, those 3 answers can be distinguished fairly easily. First, increase your available RAM and then take a look at memory evolution.
NOTE: if you cannot do that in production, or do not want to do that, you can use
derailed exec perf:mem_over_time. See below Using derailed benchmarks.
If it keeps growing slowly, you can be fairly sure you have a leak. this gives you an indication of how much your app really consumes, and if it is too much, you will have to drop useless gems (and maybe more). If memory spikes suddenly at some point, you have a bloat.
3. Let others find the leak for you
gem install bundler-leak
bundle leak update
That’s it, if the last command shows something other than No leaks found, you are lucky here’s the leak. Fix it, ship it and let’s go! See rubymem/bundler-leak for more information.
Unfortunately, this tool (and maybe others that I don’t know about) is not enough for two reasons:
- Not all leaks are referenced here (hence if you find one, please advise @memruby).
- Sorry to say this, but more often than not, the issue is in our codebase, not the lib we use.
4. Finding the leak by yourself
So, once you are sure your issue is a leak, you can start searching for it. This task is harder than searching for a memory bloat since memory slowly grows over time, and you cannot just benchmark before and after a method to see evolution, you wouldn’t know if it was just retained because no GC occurred yet, or because it was leaking.
Fortunately, there exists one really great ruby tool:
ObjectSpace.dump_all (and a bit more) which lets us analyze our application memory at a given time. And we can take advantage of that to take a few snapshots, and then compare them.
Using derailed benchmarks
Now is a good time to talk about
derailed_benchmarks. If you don't know about it yet, it is a tool for benchmarking rails (and rack) applications performances either for time or memory issues. You may dig through the README for more information, I'll stick with a simple and essential way here:
gem install derailed_benchmarks
RAILS_ENV=production RACK_ENV=production derailed_benchmarks exec <cmd>
Analyzing memory heaps
The idea here is to generate three consecutive dumps (A, B, and C for instance) separated by a few requests to your application, and then compare them to detect what is retained and what is not. I will not detail what is the content of those dumps here, just how to exploit it (here for content).
So what we are going to do is checking which objects are in B, and not in A. Meaning, objects that have been created during requests received between both memory snapshots.
Then we’ll keep only objects that are both in B and C. Meaning, objects that were present at B and retained afterward.
For instance, if you have three dumps with the next objects in memory:
We can say that our object at
0x01 is a long-term object due to our usual memory growth at boot. However
0x02 is ALLOCATED at B, hence if big it could be the cause of a bloat. However, it cannot be the cause of our leak since it has been garbage collected in C. Finally,
0x03 is RETAINED since it appears after some time (B) and stays there (C), it has great chances to be the reason of our leak.
To do that in your code, you just need to
derailed exec perf:heap_diff. This will save three dumps for you, and run a three-way diff against those. You can then just check which retained line concern your application.
$ TEST_COUNT=2000 RACK_ENV=production PATH_TO_HIT=/v1/status derailed exec perf:heap_diff
$ heapy diff tmp/2021-05-06T11:15:43+02:00-heap-0.ndjson tmp/2021-05-06T11:15:52+02:00-heap-1.ndjson tmp/2021-05-06T11:16:04+02:00-heap-2.ndjson \
| grep klaxit-via
Retained DATA 1984 objects of size 158720/13482787 (in bytes) at: /Users/ulysse/Dev/klaxit/klaxit-via/app.rb:56
Retained IMEMO 1982 objects of size 142704/13482787 (in bytes) at: /Users/ulysse/Dev/klaxit/klaxit-via/app.rb:56
If you get a few lines with a huge size, bingo! It is now a matter of understanding why the code is leaking, and fixing that. Otherwise, consider testing another endpoint, changing
PATH_TO_HIT. I suggest to always start with a simple endpoint as it will tell you if the leak lies in your whole stack or only in a single endpoint.
5. Wrapping up
It is now time to commit, run
derailed exec perf:mem_over_time on the former leaking endpoint, and see that nice plateau after a few requests, indicating that you've solved the leak (congrats!)