We also schedule a database commit for 30 seconds in the future whenever a Read State is updated. On cache key eviction, we commit your Read States to the database. There are hundreds of thousands of cache updates per second.įor persistence, we back the cache with a Cassandra database cluster. There are tens of millions of Read States in each cache. There are millions of Users in each cache. In order to get quick atomic counter updates, each Read States server has a Least Recently Used (LRU) cache of Read States. For example, one of the counters is how many you have in a channel. Each Read State has several counters that need to be updated atomically and often reset to 0. There is one Read State per User per Channel. The data structure we use to store read state information is conveniently called “Read State”. To explain why Go wasn’t meeting our performance targets, we first need to discuss the data structures, scale, access patterns, and architecture of the service. Why Go did not meet our performance targets After investigating, we determined the spikes were due to core Go features: its memory model and garbage collector (GC). It was fast most of the time, but every few minutes we saw large latency spikes that were bad for user experience.

With the Go implementation, the Read States service was not supporting its product requirements.

We want to make sure Discord feels super snappy all the time, so we need to make sure Read States is quick. In short, Read States is in the hot path. Read States is accessed every time you connect to Discord, every time a message is sent and every time a message is read. Its sole purpose is to keep track of which channels and messages you have read. The service we switched from Go to Rust is the “Read States” service. The Read States serviceĭiscord is a product focused company, so we’ll start with some product context. This post explains why it made sense for us to reimplement the service, how it was done, and the resulting performance improvements. Most recently, we drastically improved the performance of a service by switching its implementation from Go to Rust. For example, we use it on the client side for our video encoding pipeline for Go Live and on the server side for Elixir NIFs. At Discord, we’ve seen success with Rust on the client side and server side. Rust is becoming a first class language in a variety of domains.

Discord download canary