Golang - Detecting Resource Leaks With Baseline Tests

Cleaning up resources is one fo the most difficult things in Go. If careful attention isn't paid to files, network connections, or goroutines then a special type of memory leaked, called a resource leak, can occur. One of the most effective way to identify resource leaks I've found are "baseline" tests. These tests establish a healthy baseline. Next some work is simulated. The final step is to verify that the baseline was returned to, once the work has completed.

Baseline tests are a reactionary way to detect resource leaks. This means that a resource leak must already be present in code. Contrast this with a proactive detection mechanism such as a static analysis. In static analysis a check may be performed to ensure that every io.Closer, has Close() called on it within the function scope the resource was opened. Another strategy to manage resource leaks is a strict adherence to convention. i.e. ensure that every goroutine that's started has a way to finish, or defering Close() right after a resource is successfully opened.

A baseline tests involves starting your application and then generating some short-lived test load. Below shows an example of a baseline test.

The graph shows the # of objects allocated by go. The flat sections of the graph display the "baseline". The spikes indicate load is being applied.

After the program is initialized it is a flat steady line. Then some load is applied to the program. Afterwards the runtime returns to the initial flat line. When resources leak the # of objects will increase after each application of load, and the flat "baseline" sections will slowly grow.

The next graph shows a base line tests affect on the # of goroutines. During steady state 11 goroutines are running.

If the # of go routines or the # of objects allocated never return after stopping load, it's a good indicator of a resource leak. Once a resource leak is detected its important to dig into the root cause. Go provides a tool called pprof which helps in detecting resources leaks.

Baseline tests are a really simple and easy way to detect resource leaks. Unfortunately, they only detect after a resource leak is already present in code, so they should be combined with static analysis and resource management conventions for the best coverage. Go makes it particularly easy to inspect the runtime and expose the # of allocated objects and goroutines through the runtime package.

References: