2.3x faster using the Go plugin to replace Lua virtual machine

Summary

Talaria, an open-source distributed time-series database developed at Grab, previously allowed users to run custom data transformation scripts during ingestion using a Lua virtual machine. Launching and executing Lua scripts caused significant performance overhead when processing large volumes of events. To resolve this bottleneck, the team replaced the Lua VM with Go plugins compiled as Linux shared libraries (.so files). Benchmarks revealed that calling Go plugins achieves performance on par with native Go functions, executing roughly 2.3 times faster and consuming 2.3 times less memory than cached Lua VMs. Both execution methods conform to a unified Handler interface to load and run custom transformations.

Takeaways

1. 1

   Go plugin calls perform almost identically to native Go functions, running approximately 2.3 times faster and using 2.3 times less memory than cached Lua VM executions.

2. 2

   Go plugins are compiled using the -buildmode=plugin flag, loaded via plugin.Open, and mapped to execution targets with the Lookup method.

3. 3

   The Go version used to compile a plugin must precisely match the Go version of the host application, which can be managed using Docker builds.