Wednesday, May 6, 2015

Have I started hating mysql and falling in love with distributed databases

It seems Mysql is rock solid if you want:
  1. Transactions
  2. ACID support
So I would still recommend mysql for any thing that is mission critical data and it should be the primary datastore for your transactions. But what about derived data, immutable data or analytical data?

In past I have built large scale cluster of mysql server storing metadata about billions of files and folders used by tens of thousands of customers daily and its scaling fine and working good, its still growing at a healthy rate and holding up.  But this requires a lot of baby sitting if you have 100s of nodes and you need to do
  1. replication
  2. add more nodes
  3. rebalancing data
  4. monitoring entire cluster
  5. Sharding
  6. Backup/restore
You have to write a lot of tooling and lot of monitoring/babysitting to scale the cluster. Plain stock Mysql will scale up to a limit but vertically scaling has its own issues. So +1 for Mysql but not everything should be stuffed there.

Recently me and my team built full text search/indexing on same dataset using elasticsearch and so far it hasn’t disappointed me. With just 1 engineer and 1 devops guy we are able to build a cluster per datacenter to store same data. The thing I liked most about Elasticsearch was half way through migration we started facing performance issues and we just added more nodes and the cluster rebalanced itself.  Also Elasticsearch has tools like kopf/HQ where I can monitor all nodes in one place.  For e.g. this is one of the smallest cluster that we just started migrating and as it grows if we see high load averages then we can add more data or client nodes.

I don’t need an lot of DBAs to manage the cluster as elasticsearch has built in support for
  1. replication
  2. adding more nodes
  3. rebalancing data
  4. monitoring entire cluster
  5. Sharding
I had earlier built an event store to store events on top of mysql but I was storing only few months of events into it. Now I have to build a store that can store events for 7 years and I dont want to use Mysql for it as I dont want to baby sit it. Our events data is way huge than the metadata. Because every change to a file generates an event and over 7 years this could be a huge no of records. I dont want to manage an army of mysql servers so researching for some database that has good querying support and can store long lived data with eventual consistency and rock solid durability. You could store them using kafka or scribe but they lack good querying support.  I am still not sure if Elasticsearch is a good store for this because I don’t need search, I need querying based on some paths and time based querying. Also most querying will be on recent data and past data would be queried rarely. Logstash uses Elasticsearch in same fashion so I need to find something else and then compare with Elasticsearch.

Also I see a trend emerging that in today's world DBAs need to get out of cocoon and know more than just relational databases. There are a lot of new tools like OpenTSDB, HBase, Mongo, Cassandra, ElasticSearch, BigQuery that are now getting used to store BigData so they need to catch up and catch up fast. Google recently released BigTable need to check it out. I had checked BigQuery but writes are cheap, its the reads that are costly when using things like BigQuery.

Rate limiting APIs and Java services when operating at Scale to solve Thundering herd problem

When you are operating at scale and handling peak traffic of 1K+ request per sec on a jvm then no matter what you do you would get hit by a Thundering herd problem. There would be operations that happens once in a while but take more than 10 sec and if there are too many of them happening then you could choke backend services or worse cause a downtime. So you need to Rate limit these long running operations that only X can run at a time, this way you are leaving room for running lots of short lived transactions.

When you have millions of users then not all users are doing these long running operations and not all traffic is coming from online users. We are a cloud storage company and we give sync client to users so 80%+ traffic at a given time is coming from these clients that are trying to sync changes between cloud and local system behind the scenes.  Our application is written using REST apis and these clients  are using the same REST apis that our web ui is using.  Also some customers may have 10K users and some may have 10. So it may happen that a big customer may always starve the small customers.  So you need two things:
  1. Fair distribution of use of REST apis
  2. Separate priority for online vs bot traffic.
  3. Rate limit or protect a tier if a Thundering herd occurs
Most of these sync clients can exponential backoff, they handle 503 response and retry after some time with exponential delay with an upper threshold.

For fair distribution of use among customers I was using a pool of thread pools with hashing based on customerId and a salt, the technique is described here

for Rate limiting I was using ThreadPools. When requests would come I would create a callable for the action to be executed and submit it to threadpool and wait for the result. The Thread pool had a fixed processing capacity and a fixed queue length. After the queue is full we would send 503s.

But thread pools have many disadvantages:
  1. You already have a http thread and now that is idle as you delegated to a threadpool to do the job.
  2. New relic somehow goes nuts when you delegate to a threadpool and doesnt record any trace info, AppDynamics is smart and it recognizes it but I am not a big fan of AppDynamics.
  3. Logging context gets messed up and you now have to propagate it to the new thread.
  4. Exception handling is messed up as it would get wrapped in ExecutorService exceptions
  5. Under high load we ran into an issue where in tomcat if you wrote to response from 2 threads and sync clients abort a connection then it hangs the thread causing all threads to be gobbled up in a course of 6-12 hours.
I was looking for a RateLimiter that would give me a gate with a finite opening and a finite queue length, kinda like restaurant where you have a finite no of tables and a finite queue before they start accepting more guests. I didnt found anything so I cooked up a one in kitchen. 

The code is going live this weekend, we did perf test and the results looks promising. To be conservative I had to add a switch so that in case of issues you can revert to old way with the flip of a flag in config at runtime.

public class RateLimiter {
    private DiagnosticSemaphore allQueue;
    private DiagnosticSemaphore processingQueue;
    private volatile boolean shutdown;

    public RateLimiter(int numProcessing, int numWaiting) {
        this.allQueue = new DiagnosticSemaphore(numWaiting + numProcessing, true);
        this.processingQueue = new DiagnosticSemaphore(numProcessing, true);

    public T executeWithRateLimit(Callable callable) throws Exception {
        boolean allQueueAdded = false;
        boolean processingQueueAdded = false;
        try {
            allQueueAdded = allQueue.tryAcquire();
            if (!allQueueAdded) {
                throw new RejectedExecutionException();
            try {
                processingQueueAdded = true;
            } catch (InterruptedException e) {
                throw new ApplicationRuntimeException(e);
            } finally {
                if (processingQueueAdded) {
        } finally {
            if (allQueueAdded) {

    private void handleShutdown() {
        if (shutdown) {
            throw new RejectedExecutionException("Not accepting requests as shutting down");

    public void shutdownNow() {
        shutdown = true;
        for (Thread t : allQueue.getQueuedThreads()) {

    static final class DiagnosticSemaphore extends Semaphore {
        private static final long serialVersionUID = 1L;

        public DiagnosticSemaphore(int permits, boolean fair) {
            super(permits, fair);

        public Collection getQueuedThreads() {
            return super.getQueuedThreads();

Friday, April 10, 2015

Cognitive overhead of email

Looks like these days bitching about email is my favourite topic. What can I do no matter what I do I just cant keep up with email flow.  The new kind of emails just keep increasing, I was doing a good job 2 years ago and close to inbox zero daily and was writing a lot of code but two things have increased the email flow to me recently:

  1. Internal reports about production montioring from Newrelic, haproxy, exception analysis by team. 
  2. Review requests.  This is a recent one. Most review requests are sent to Java review group and somehow I atleast need to spend 1-2 min on every email to first decide whether this is for me or not.  Even if there are 30 review request per day it consumes 30 min of my time just to make sense of it. Daily when I see 100 emails to be answered I feel disheartened that I cant write any code today.  
 Both above items are necessary evil, #1 has increased production stability so my weekends are less busy.  #2 I am hoping eventually will increase code quality and lead to more ownership and processes that would lead to a self correcting system  requiring less involvement from leads.

But I need to do a better job or come up with some other way to do this. Its almost 5:00  and I had two 1 hour meetings, 1 Elastic search node down that consumed 1.5 hours and 2 team calls 30 min each. And I cleared some 60 emails. The only code I wrote is removing some junk code as it was causing some exception when a penetration tester was testing that flow.

Feeling frustrated as need to do something about this.  This Slack killing email sounds interesting.