Skip to main content

Posts

Showing posts from May, 2013

Benefits of indirection

We shard our databases and keep the metadata information as to what shard is located in what database host in a table.  We recently had a database spike where due to thundering herd problem, lots of our clients would come and execute a very costly query concurrently.  The only solution is to avoid making this query and change the application architecture but the problem would take 1-2 months to solve. We didn't had the luxury to wait that long.  So we were looking for solutions to buy time.

Luckily to buy time we can throw more hardware, we had recently ordered a pair of mysql servers for some other purpose so we repurposed it.  Because we had desgined our application to have a layer of indirection on how to look what customer is located on what shard and what shard is located on what database host.  We were able to quickly spin off 2 slaves and connect them to the db host having load isues. As soon as replication was done we cut off the db access and wait for replication to be 10…