Why is memory profiling in ruby so hard?

Or rather, why aren't there better tools for profiling memory in ruby, specifically rails apps?

Recently our rails app (hosted on heroku) has started seeing lots of R14 errors in the worker dynos. This means we're running out of memory. Bumping the dynos to 2x (512mb -> 1GB) only alleviated the problem temporarily, leading me to believe there is a memory leak somewhere. Naturally, my next step was to find a good profiling gem that can help me discover the source of the leak.

Maybe I'm just ignorant of the tools available, or maybe I just don't know how to use the ones I have. My wish is that I could install a gem and then run reports on the memory usage statistics. Hitting an endpoint to get a report is not really viable as my memory issues are isolated to worker dynos running delayed jobs.

I've looked at [memprof][1], but it's 1.8 only.

I've looked at [ruby-prof][2] (awesome), but the memory profiling requires a patched ruby intepreter.

I've looked at [GC::Profiler][3], but I don't understand how to find memory leaks with it.

So, is it just plain difficult to find memory leaks in ruby? Or am I missing the point somehow?

[1]:

[To see links please register here]

[2]:

[To see links please register here]

[3]:

[To see links please register here]

There is a nice way if your application is running on OS with dtrace or similar technology like SystemTap. In my case, we use RHEL/CentOS which has the latter:

[To see links please register here]

You can easily connect to production application and "inject" profiling code for a moment and track calls, memory, CPU time or I/O and then "disconnect" at any time. It's very efficient so you will not likely notice any drastice slowdown (unless you screw your script up).

Recently I've been having some success with [Skylight][1] to profile web and background worker methods and then hunt down opportunities for optimisation. It probably wasn't around when you posted this question. The downside is that it only really helps you debug in staging or production, not development environments, so the dev loop can be very slow.

Make sure you install both [skylight-ruby][2] and [sidekiq-skylight][3] to get profiling on both your web server and background workers if you're using Sidekiq.

Good luck!


[1]:

[To see links please register here]

[2]:

[To see links please register here]

[3]:

[To see links please register here]

Depending on your "type" of leak, You *can* run valgrind against ruby. Might require recompilation again though. In general it's hard because ruby does method allocation without firing any events, by default, so it's tricky to track. See also [perftools.rb project][1], which somewhat works around this limitation.


[1]:

[To see links please register here]

I disagree that memory profiling in Ruby is hard. The JVM has some of the best memory profiling tools on the planet, and you can run your Ruby programs on the JVM. Don't reinvent the wheel.

* [Browsing Memory the JRuby Way](

[To see links please register here]

)
* [Finding Leaks in Ruby Apps with Eclipse Memory Analyzer](

[To see links please register here]

)
* [Monitoring Memory with JRuby, Part 1: jhat and VisualVM](

[To see links please register here]

)
* [Monitoring Memory with JRuby, Part 2: Eclipse Memory Analyzer](

[To see links please register here]

)