A Cache miss can get dangerous especially in a high throughput system where a lot of requests look for the same information.
Multiple requests come in looking for the same tenant/config/data The data is not available in the cache All of them go to fallback: your DB This could lead to DB getting overloaded, causing timeouts, retries and making the situation worse
Wouldn’t it be better if we could have queried the DB only once and shared the result to all the requests that have come in while our first call was in still in flight.
This is exactly what “Singleflight” does.
It takes in a key and function that loads data.
For the first request Singleflight runs the function. For the duplicate requests with the same key, it waits until the first function completes and shares the result with all of them
Internally Singleflight uses a map and a lock to group calls with same key together Once the original call is finished and results shared the key is deleted.
It is important to note that Singleflight does not do any caching. It only replaces duplicate work. It complements your caching strategy and is not a replacement to it.
One point to note, is in cases with high throughput and a lot of keys the lock can become a point of contention. This can be solved by sharding the map.