Rate limiting APIs and Java services when operating at Scale to solve Thundering herd problem

When you are operating at scale and handling peak traffic of 1K+ request per sec on a jvm then no matter what you do you would get hit by a Thundering herd problem. There would be operations that happens once in a while but take more than 10 sec and if there are too many of them happening then you could choke backend services or worse cause a downtime. So you need to Rate limit these long running operations that only X can run at a time, this way you are leaving room for running lots of short lived transactions.

When you have millions of users then not all users are doing these long running operations and not all traffic is coming from online users. We are a cloud storage company and we give sync client to users so 80%+ traffic at a given time is coming from these clients that are trying to sync changes between cloud and local system behind the scenes.  Our application is written using REST apis and these clients  are using the same REST apis that our web ui is using.  Also some customers may have 10K users and some may have 10. So it may happen that a big customer may always starve the small customers.  So you need two things:

1. Fair distribution of use of REST apis
2. Separate priority for online vs bot traffic.
3. Rate limit or protect a tier if a Thundering herd occurs

Most of these sync clients can exponential backoff, they handle 503 response and retry after some time with exponential delay with an upper threshold.

For fair distribution of use among customers I was using a pool of thread pools with hashing based on customerId and a salt, the technique is described here http://neopatel.blogspot.com/2013/06/java-fair-share-threadpool.html.

for Rate limiting I was using ThreadPools. When requests would come I would create a callable for the action to be executed and submit it to threadpool and wait for the result. The Thread pool had a fixed processing capacity and a fixed queue length. After the queue is full we would send 503s.

But thread pools have many disadvantages:

1. You already have a http thread and now that is idle as you delegated to a threadpool to do the job.
2. New relic somehow goes nuts when you delegate to a threadpool and doesnt record any trace info, AppDynamics is smart and it recognizes it but I am not a big fan of AppDynamics.
3. Logging context gets messed up and you now have to propagate it to the new thread.
4. Exception handling is messed up as it would get wrapped in ExecutorService exceptions
5. Under high load we ran into an issue where in tomcat if you wrote to response from 2 threads and sync clients abort a connection then it hangs the thread causing all threads to be gobbled up in a course of 6-12 hours.

I was looking for a RateLimiter that would give me a gate with a finite opening and a finite queue length, kinda like restaurant where you have a finite no of tables and a finite queue before they start accepting more guests. I didnt found anything so I cooked up a one in kitchen. 

The code is going live this weekend, we did perf test and the results looks promising. To be conservative I had to add a switch so that in case of issues you can revert to old way with the flip of a flag in config at runtime.

public class RateLimiter {
    @Getter(AccessLevel.PACKAGE)
    private DiagnosticSemaphore allQueue;
    @Getter(AccessLevel.PACKAGE)
    private DiagnosticSemaphore processingQueue;
    private volatile boolean shutdown;

    public RateLimiter(int numProcessing, int numWaiting) {
        this.allQueue = new DiagnosticSemaphore(numWaiting + numProcessing, true);
        this.processingQueue = new DiagnosticSemaphore(numProcessing, true);
    }

    public T executeWithRateLimit(Callable callable) throws Exception {
        handleShutdown();
        boolean allQueueAdded = false;
        boolean processingQueueAdded = false;
        try {
            allQueueAdded = allQueue.tryAcquire();
            if (!allQueueAdded) {
                throw new RejectedExecutionException();
            }
            try {
                processingQueue.acquire();
                processingQueueAdded = true;
                handleShutdown();
                return callable.call();
            } catch (InterruptedException e) {
                throw new ApplicationRuntimeException(e);
            } finally {
                if (processingQueueAdded) {
                    processingQueue.release();
                }
            }
        } finally {
            if (allQueueAdded) {
                allQueue.release();
            }
        }
    }

    private void handleShutdown() {
        if (shutdown) {
            throw new RejectedExecutionException("Not accepting requests as shutting down");
        }
    }

    public void shutdownNow() {
        shutdown = true;
        for (Thread t : allQueue.getQueuedThreads()) {
            t.interrupt();
        }
    }

    static final class DiagnosticSemaphore extends Semaphore {
        private static final long serialVersionUID = 1L;

        public DiagnosticSemaphore(int permits, boolean fair) {
            super(permits, fair);
        }

        @Override
        public Collection getQueuedThreads() {
            return super.getQueuedThreads();
        }
    }