Wednesday, July 15, 2015

Data motivates you to do more

I always believed in "Trusting data" over human gut but lately I have been observing a simple fact that being data driven has a side effect, "it motivates you and  keeps you on track". Some recent examples are:

Fitbit: Last year I started the afternoon walk because by 2:00PM after doing calls and replying to a lot of emails the brain would be fried and I wont have energy left to code or think.  Doing these 30 min walk daily recharges the brain. I  had an Omron pedometer sitting around for almost an year and I seldom took it with me on walks. The problem with it was that it used to store last 30 days data of my steps and other things but it didn’t had a good way to graphically see the data. When it comes to data "less is more" but also one more important aspect is that you need to present your data in graphs so it doesn’t take a huge amount of cognitive effort to make sense of it.  Recently my employer gave a fitbit to everyone who participated in summer challenge and it isn’t a better  pedometer than my old Omron but immediately I saw that it can sync data stored on pedometer via bluetooth to my fitbit app and after a week I see this. Immediately in 1 sec I can see that I am lagging this week and need to catch up.

Its another thing that I need to remember to carry this dongle with me, I always carry my cell with me on walk and I saw that Iphone6 has a pedometer built into it so that would eliminate the need for this when I upgrade to Iphone6.

Large scale migration to Elasticsearch: We recently migrated billions of files to Elasticsearch and the migration took months but data kept us on toes and telling us if the compass is pointing towards north or not. We built various dashboard to monitor migration rates and as migration was running day and night I would start my day with checking how many more files we migrated and whether we need to add more servers, jvms, memory or CPU to meet the goal. Here is a graph in one data center.
Data kept us on track and we were able to spot many issues before they were able to create a disaster.

Exception Analysis:  We do exceptions and 5xx status analysis on incoming requests daily and in 2 week sprint we strive to fix  as many as we can, but after the release either new issues pops up or customers use the flow in a different way causing some components to buckle under pressure but one thing that has kept us on toes is data. By looking at the one screen report we can tell how did this data center do yesterday and which areas require immediate attention vs areas that require attention in 1-2 days. This leads to lesser no of customer escalations as we are able to spot many issues before them. Little little things adds up and  having this report daily strives us to optimize more and we can spend quality time on doing things we love which is writing code for scalable systems.

In short data points you that there is a problem and once a bug has been implanted in your brain that a problem exists you would try to fix it so you can get back to normal routine.

Wednesday, May 6, 2015

Have I started hating mysql and falling in love with distributed databases

It seems Mysql is rock solid if you want:
  1. Transactions
  2. ACID support
So I would still recommend mysql for any thing that is mission critical data and it should be the primary datastore for your transactions. But what about derived data, immutable data or analytical data?

In past I have built large scale cluster of mysql server storing metadata about billions of files and folders used by tens of thousands of customers daily and its scaling fine and working good, its still growing at a healthy rate and holding up.  But this requires a lot of baby sitting if you have 100s of nodes and you need to do
  1. replication
  2. add more nodes
  3. rebalancing data
  4. monitoring entire cluster
  5. Sharding
  6. Backup/restore
You have to write a lot of tooling and lot of monitoring/babysitting to scale the cluster. Plain stock Mysql will scale up to a limit but vertically scaling has its own issues. So +1 for Mysql but not everything should be stuffed there.

Recently me and my team built full text search/indexing on same dataset using elasticsearch and so far it hasn’t disappointed me. With just 1 engineer and 1 devops guy we are able to build a cluster per datacenter to store same data. The thing I liked most about Elasticsearch was half way through migration we started facing performance issues and we just added more nodes and the cluster rebalanced itself.  Also Elasticsearch has tools like kopf/HQ where I can monitor all nodes in one place.  For e.g. this is one of the smallest cluster that we just started migrating and as it grows if we see high load averages then we can add more data or client nodes.

I don’t need an lot of DBAs to manage the cluster as elasticsearch has built in support for
  1. replication
  2. adding more nodes
  3. rebalancing data
  4. monitoring entire cluster
  5. Sharding
I had earlier built an event store to store events on top of mysql but I was storing only few months of events into it. Now I have to build a store that can store events for 7 years and I dont want to use Mysql for it as I dont want to baby sit it. Our events data is way huge than the metadata. Because every change to a file generates an event and over 7 years this could be a huge no of records. I dont want to manage an army of mysql servers so researching for some database that has good querying support and can store long lived data with eventual consistency and rock solid durability. You could store them using kafka or scribe but they lack good querying support.  I am still not sure if Elasticsearch is a good store for this because I don’t need search, I need querying based on some paths and time based querying. Also most querying will be on recent data and past data would be queried rarely. Logstash uses Elasticsearch in same fashion so I need to find something else and then compare with Elasticsearch.

Also I see a trend emerging that in today's world DBAs need to get out of cocoon and know more than just relational databases. There are a lot of new tools like OpenTSDB, HBase, Mongo, Cassandra, ElasticSearch, BigQuery that are now getting used to store BigData so they need to catch up and catch up fast. Google recently released BigTable need to check it out. I had checked BigQuery but writes are cheap, its the reads that are costly when using things like BigQuery.

Rate limiting APIs and Java services when operating at Scale to solve Thundering herd problem

When you are operating at scale and handling peak traffic of 1K+ request per sec on a jvm then no matter what you do you would get hit by a Thundering herd problem. There would be operations that happens once in a while but take more than 10 sec and if there are too many of them happening then you could choke backend services or worse cause a downtime. So you need to Rate limit these long running operations that only X can run at a time, this way you are leaving room for running lots of short lived transactions.

When you have millions of users then not all users are doing these long running operations and not all traffic is coming from online users. We are a cloud storage company and we give sync client to users so 80%+ traffic at a given time is coming from these clients that are trying to sync changes between cloud and local system behind the scenes.  Our application is written using REST apis and these clients  are using the same REST apis that our web ui is using.  Also some customers may have 10K users and some may have 10. So it may happen that a big customer may always starve the small customers.  So you need two things:
  1. Fair distribution of use of REST apis
  2. Separate priority for online vs bot traffic.
  3. Rate limit or protect a tier if a Thundering herd occurs
Most of these sync clients can exponential backoff, they handle 503 response and retry after some time with exponential delay with an upper threshold.

For fair distribution of use among customers I was using a pool of thread pools with hashing based on customerId and a salt, the technique is described here

for Rate limiting I was using ThreadPools. When requests would come I would create a callable for the action to be executed and submit it to threadpool and wait for the result. The Thread pool had a fixed processing capacity and a fixed queue length. After the queue is full we would send 503s.

But thread pools have many disadvantages:
  1. You already have a http thread and now that is idle as you delegated to a threadpool to do the job.
  2. New relic somehow goes nuts when you delegate to a threadpool and doesnt record any trace info, AppDynamics is smart and it recognizes it but I am not a big fan of AppDynamics.
  3. Logging context gets messed up and you now have to propagate it to the new thread.
  4. Exception handling is messed up as it would get wrapped in ExecutorService exceptions
  5. Under high load we ran into an issue where in tomcat if you wrote to response from 2 threads and sync clients abort a connection then it hangs the thread causing all threads to be gobbled up in a course of 6-12 hours.
I was looking for a RateLimiter that would give me a gate with a finite opening and a finite queue length, kinda like restaurant where you have a finite no of tables and a finite queue before they start accepting more guests. I didnt found anything so I cooked up a one in kitchen. 

The code is going live this weekend, we did perf test and the results looks promising. To be conservative I had to add a switch so that in case of issues you can revert to old way with the flip of a flag in config at runtime.

public class RateLimiter {
    private DiagnosticSemaphore allQueue;
    private DiagnosticSemaphore processingQueue;
    private volatile boolean shutdown;

    public RateLimiter(int numProcessing, int numWaiting) {
        this.allQueue = new DiagnosticSemaphore(numWaiting + numProcessing, true);
        this.processingQueue = new DiagnosticSemaphore(numProcessing, true);

    public T executeWithRateLimit(Callable callable) throws Exception {
        boolean allQueueAdded = false;
        boolean processingQueueAdded = false;
        try {
            allQueueAdded = allQueue.tryAcquire();
            if (!allQueueAdded) {
                throw new RejectedExecutionException();
            try {
                processingQueueAdded = true;
            } catch (InterruptedException e) {
                throw new ApplicationRuntimeException(e);
            } finally {
                if (processingQueueAdded) {
        } finally {
            if (allQueueAdded) {

    private void handleShutdown() {
        if (shutdown) {
            throw new RejectedExecutionException("Not accepting requests as shutting down");

    public void shutdownNow() {
        shutdown = true;
        for (Thread t : allQueue.getQueuedThreads()) {

    static final class DiagnosticSemaphore extends Semaphore {
        private static final long serialVersionUID = 1L;

        public DiagnosticSemaphore(int permits, boolean fair) {
            super(permits, fair);

        public Collection getQueuedThreads() {
            return super.getQueuedThreads();