Tuesday, June 21, 2016

Little things and frustrations

Small shitty things frustrates me, like someone came up with an idea to put epic on each Jira task.  Well its suited for big Tasks that has lot of subtasks but 50% + of my cases don’t require an Epic.


I am lazy and dont like to waste my time. I am creating an issue and filled all things and now this process requires me an Epic, total BS.  Yes this is total BS so either the process requires fixing where epic is first required field so I wont invest filling so many fields or this needs to be removed/made optional.

Wednesday, July 15, 2015

Data motivates you to do more

I always believed in "Trusting data" over human gut but lately I have been observing a simple fact that being data driven has a side effect, "it motivates you and  keeps you on track". Some recent examples are:

Fitbit: Last year I started the afternoon walk because by 2:00PM after doing calls and replying to a lot of emails the brain would be fried and I wont have energy left to code or think.  Doing these 30 min walk daily recharges the brain. I  had an Omron pedometer sitting around for almost an year and I seldom took it with me on walks. The problem with it was that it used to store last 30 days data of my steps and other things but it didn’t had a good way to graphically see the data. When it comes to data "less is more" but also one more important aspect is that you need to present your data in graphs so it doesn’t take a huge amount of cognitive effort to make sense of it.  Recently my employer gave a fitbit to everyone who participated in summer challenge and it isn’t a better  pedometer than my old Omron but immediately I saw that it can sync data stored on pedometer via bluetooth to my fitbit app and after a week I see this. Immediately in 1 sec I can see that I am lagging this week and need to catch up.

Its another thing that I need to remember to carry this dongle with me, I always carry my cell with me on walk and I saw that Iphone6 has a pedometer built into it so that would eliminate the need for this when I upgrade to Iphone6.


Large scale migration to Elasticsearch: We recently migrated billions of files to Elasticsearch and the migration took months but data kept us on toes and telling us if the compass is pointing towards north or not. We built various dashboard to monitor migration rates and as migration was running day and night I would start my day with checking how many more files we migrated and whether we need to add more servers, jvms, memory or CPU to meet the goal. Here is a graph in one data center.
Data kept us on track and we were able to spot many issues before they were able to create a disaster.

Exception Analysis:  We do exceptions and 5xx status analysis on incoming requests daily and in 2 week sprint we strive to fix  as many as we can, but after the release either new issues pops up or customers use the flow in a different way causing some components to buckle under pressure but one thing that has kept us on toes is data. By looking at the one screen report we can tell how did this data center do yesterday and which areas require immediate attention vs areas that require attention in 1-2 days. This leads to lesser no of customer escalations as we are able to spot many issues before them. Little little things adds up and  having this report daily strives us to optimize more and we can spend quality time on doing things we love which is writing code for scalable systems.

In short data points you that there is a problem and once a bug has been implanted in your brain that a problem exists you would try to fix it so you can get back to normal routine.





Wednesday, May 6, 2015

Have I started hating mysql and falling in love with distributed databases

It seems Mysql is rock solid if you want:
  1. Transactions
  2. ACID support
So I would still recommend mysql for any thing that is mission critical data and it should be the primary datastore for your transactions. But what about derived data, immutable data or analytical data?

In past I have built large scale cluster of mysql server storing metadata about billions of files and folders used by tens of thousands of customers daily and its scaling fine and working good, its still growing at a healthy rate and holding up.  But this requires a lot of baby sitting if you have 100s of nodes and you need to do
  1. replication
  2. add more nodes
  3. rebalancing data
  4. monitoring entire cluster
  5. Sharding
  6. Backup/restore
You have to write a lot of tooling and lot of monitoring/babysitting to scale the cluster. Plain stock Mysql will scale up to a limit but vertically scaling has its own issues. So +1 for Mysql but not everything should be stuffed there.

Recently me and my team built full text search/indexing on same dataset using elasticsearch and so far it hasn’t disappointed me. With just 1 engineer and 1 devops guy we are able to build a cluster per datacenter to store same data. The thing I liked most about Elasticsearch was half way through migration we started facing performance issues and we just added more nodes and the cluster rebalanced itself.  Also Elasticsearch has tools like kopf/HQ where I can monitor all nodes in one place.  For e.g. this is one of the smallest cluster that we just started migrating and as it grows if we see high load averages then we can add more data or client nodes.
















I don’t need an lot of DBAs to manage the cluster as elasticsearch has built in support for
  1. replication
  2. adding more nodes
  3. rebalancing data
  4. monitoring entire cluster
  5. Sharding
I had earlier built an event store to store events on top of mysql but I was storing only few months of events into it. Now I have to build a store that can store events for 7 years and I dont want to use Mysql for it as I dont want to baby sit it. Our events data is way huge than the metadata. Because every change to a file generates an event and over 7 years this could be a huge no of records. I dont want to manage an army of mysql servers so researching for some database that has good querying support and can store long lived data with eventual consistency and rock solid durability. You could store them using kafka or scribe but they lack good querying support.  I am still not sure if Elasticsearch is a good store for this because I don’t need search, I need querying based on some paths and time based querying. Also most querying will be on recent data and past data would be queried rarely. Logstash uses Elasticsearch in same fashion so I need to find something else and then compare with Elasticsearch.

Also I see a trend emerging that in today's world DBAs need to get out of cocoon and know more than just relational databases. There are a lot of new tools like OpenTSDB, HBase, Mongo, Cassandra, ElasticSearch, BigQuery that are now getting used to store BigData so they need to catch up and catch up fast. Google recently released BigTable http://venturebeat.com/2015/05/06/google-introduces-cloud-bigtable-managed-nosql-database-to-process-data-at-scale/ need to check it out. I had checked BigQuery but writes are cheap, its the reads that are costly when using things like BigQuery.