Skip to main content

Rate limiting APIs and Java services when operating at Scale to solve Thundering herd problem

When you are operating at scale and handling peak traffic of 1K+ request per sec on a jvm then no matter what you do you would get hit by a Thundering herd problem. There would be operations that happens once in a while but take more than 10 sec and if there are too many of them happening then you could choke backend services or worse cause a downtime. So you need to Rate limit these long running operations that only X can run at a time, this way you are leaving room for running lots of short lived transactions.

When you have millions of users then not all users are doing these long running operations and not all traffic is coming from online users. We are a cloud storage company and we give sync client to users so 80%+ traffic at a given time is coming from these clients that are trying to sync changes between cloud and local system behind the scenes.  Our application is written using REST apis and these clients  are using the same REST apis that our web ui is using.  Also some customers may have 10K users and some may have 10. So it may happen that a big customer may always starve the small customers.  So you need two things:
  1. Fair distribution of use of REST apis
  2. Separate priority for online vs bot traffic.
  3. Rate limit or protect a tier if a Thundering herd occurs
Most of these sync clients can exponential backoff, they handle 503 response and retry after some time with exponential delay with an upper threshold.

For fair distribution of use among customers I was using a pool of thread pools with hashing based on customerId and a salt, the technique is described here

for Rate limiting I was using ThreadPools. When requests would come I would create a callable for the action to be executed and submit it to threadpool and wait for the result. The Thread pool had a fixed processing capacity and a fixed queue length. After the queue is full we would send 503s.

But thread pools have many disadvantages:
  1. You already have a http thread and now that is idle as you delegated to a threadpool to do the job.
  2. New relic somehow goes nuts when you delegate to a threadpool and doesnt record any trace info, AppDynamics is smart and it recognizes it but I am not a big fan of AppDynamics.
  3. Logging context gets messed up and you now have to propagate it to the new thread.
  4. Exception handling is messed up as it would get wrapped in ExecutorService exceptions
  5. Under high load we ran into an issue where in tomcat if you wrote to response from 2 threads and sync clients abort a connection then it hangs the thread causing all threads to be gobbled up in a course of 6-12 hours.
I was looking for a RateLimiter that would give me a gate with a finite opening and a finite queue length, kinda like restaurant where you have a finite no of tables and a finite queue before they start accepting more guests. I didnt found anything so I cooked up a one in kitchen. 

The code is going live this weekend, we did perf test and the results looks promising. To be conservative I had to add a switch so that in case of issues you can revert to old way with the flip of a flag in config at runtime.

public class RateLimiter {
    private DiagnosticSemaphore allQueue;
    private DiagnosticSemaphore processingQueue;
    private volatile boolean shutdown;

    public RateLimiter(int numProcessing, int numWaiting) {
        this.allQueue = new DiagnosticSemaphore(numWaiting + numProcessing, true);
        this.processingQueue = new DiagnosticSemaphore(numProcessing, true);

    public T executeWithRateLimit(Callable callable) throws Exception {
        boolean allQueueAdded = false;
        boolean processingQueueAdded = false;
        try {
            allQueueAdded = allQueue.tryAcquire();
            if (!allQueueAdded) {
                throw new RejectedExecutionException();
            try {
                processingQueueAdded = true;
            } catch (InterruptedException e) {
                throw new ApplicationRuntimeException(e);
            } finally {
                if (processingQueueAdded) {
        } finally {
            if (allQueueAdded) {

    private void handleShutdown() {
        if (shutdown) {
            throw new RejectedExecutionException("Not accepting requests as shutting down");

    public void shutdownNow() {
        shutdown = true;
        for (Thread t : allQueue.getQueuedThreads()) {

    static final class DiagnosticSemaphore extends Semaphore {
        private static final long serialVersionUID = 1L;

        public DiagnosticSemaphore(int permits, boolean fair) {
            super(permits, fair);

        public Collection getQueuedThreads() {
            return super.getQueuedThreads();


Popular posts from this blog

RabbitMQ java clients for beginners

Here is a sample of a consumer and producer example for RabbitMQ. The steps are
Download ErlangDownload Rabbit MQ ServerDownload Rabbit MQ Java client jarsCompile and run the below two class and you are done.
This sample create a Durable Exchange, Queue and a Message. You will have to start the consumer first before you start the for the first time.

For more information on AMQP, Exchanges, Queues, read this excellent tutorial
import com.rabbitmq.client.Connection; import com.rabbitmq.client.Channel; import com.rabbitmq.client.*; public class RabbitMQProducer { public static void main(String []args) throws Exception { ConnectionFactory factory = new ConnectionFactory(); factory.setUsername("guest"); factory.setPassword("guest"); factory.setVirtualHost("/"); factory.setHost(""); factory.setPort(5672); Conne…

What a rocky start to labor day weekend

Woke up by earthquake at 7:00 AM in morning and then couldn't get to sleep. I took a bath, made my tea and started checking emails and saw that after last night deployment three storage node out of 100s of nodes were running into Full GC. What was special about the 3 nodes was that each one was in a different Data centre but it was named same app02.  This got me curious I asked the node to be taken out of rotation and take a heap dump.  Yesterday night a new release has happened and I had upgraded spymemcached library version as new relic now natively supports instrumentation on it so it was a suspect. And the hunch was a bullseye, the heap dump clearly showed it taking 1.3G and full GCs were taking 6 sec but not claiming anything.

I have a quartz job in each jvm that takes a thread dump every 5 minutes and saves last 300 of them, checking few of them quickly showed a common thread among all 3 data centres. It seems there was a long running job that was trying to replicate pending…

Logging to Graphite monitoring tool from java

We use Graphite as a tool for monitoring some stats and watch trends. A requirement is to monitor impact of new releases as build is deployed to app nodes to see if things like
1) Has the memcache usage increased.
2) Has the no of Java exceptions went up.
3) Is the app using more tomcat threads.
Here is a screenshot

We changed the installer to log a deploy event when a new build is deployed. I wrote a simple spring bean to log graphite events using java. Logging to graphite is easy, all you need to do is open a socket and send lines of events.
import org.slf4j.Logger;import org.slf4j.LoggerFactory; import; import; import; import java.util.HashMap; import java.util.Map; public class GraphiteLogger { private static final Logger logger = LoggerFactory.getLogger(GraphiteLogger.class); private String graphiteHost; private int graphitePort; public String getGraphiteHost() { return graphiteHost; } public void setGraphite…