Programming fun at startup

"The only thing constant in a Startup is Change" If you aren't changing fast enough then order and complacency sets in which leads to mediocrity and you can soon become obsolete. We do biweekly releases and want to move to weekly and then daily releases. You can increase the release cadence, if you are able to automate most of the testing. But automation can't detect everything and things may break and the intent is to establish a culture of how can you prevent it but if you cant prevent it then how fast can diagnose and recover from it. For a long time I had been pushing hard for a working centralized logging and after almost an year of trying to scale the ELK framework finally our team has been able to put a fast centralized logging system that's ingesting Terabytes of log per minute from thousands of nodes in production. This weekend we deployed a major architecture change by creating swimlanes in our cloud by directing different types of traffic to di

We should not repeat mistakes and if its the end user that is committing same mistake again and again then the UX should be changed to make it difficult for user to perform such actions. Make the user apply some cognitive thinking before he can do a detrimental action and make the action obscure. For e.g. we are a filesystem company and "Trash delete" is a detrimental action, over past few years I had several tickets where user accidentally clicked Empty trash and deleted all his files or he clicked Delete Permanently instead of Restore. He was able to do all this  because in the UX the actions are close to each other There are users who did this and immediately created tickets under panic and some even blaming that system did delete instead of restore.  I know in UI it may look bad but I have seen several companies solving this by making user think twice before invoking detrimental action like asking a captcha or make the delete and restore flow differ

This week I saw the movie Dangal trailer and the actor says "If you win silver medal, today or tomorrow people will forget you but if you win Gold medal you will become a source of inspiration".  Also today morning I saw K L Rahul got out on 199 runs and the commentators are complaining about his loss of focus on one ball and he got out http://timesofindia.indiatimes.com/sports/cricket/news/india-v-england-5th-test-day-3-rahuls-199-drives-india-to-commanding-position/articleshow/56045818.cms This reminds me of Startups and Finishers, give up too close to the finish line and you wont be remembered but if you finish it you may become an inspiration.

Burnout is very common in startup and it hits you more often when you are at the same startup for 7+ years. Much of burnout is felt when you are working hard but progress is slow and you have high expectations of yourself and the team. I read this excellent article http://andrewdumont.me/avoiding-burnout/ and it felt almost Deja vu to me.  I work from home so it hits even hard to me as I have less people to share it with. Here are the ways I have been trying to curtail burnout and hope it help others too: Workplace :Keep your workplace organized and stick to a routine.  I have an office room at home and I am in work mode when I am in that office. Morning :Keep 1 hour in mornings on something outside work. I read a lot in morning and don't come online or check emails until that 1 hour is done. Workout : rain, heat or winter, I try to walk for 30 min in afternoon as much as possible. This gives you some time for meta thinking and de stresses you from the morning chaos. Many sol

A quality of a good Startup engineer is "Finishing what you started". I value finishers more than people who talk good theory and start many things but don’t finish most of them. Most of the time people get excited and start many things concurrently but leave them in limbo or they finish 95% of the thing.  Over 7 years at my current Startup I have observed that in many projects doing first 95% may take you X units of time but its the remaining 5% that demands 10x cognitive effort and time from you. Its this last 5% where people either burnout or give up. Its the finishers who keep the compass pointing to North star, never give up and do what ever it takes to Finish( in indian mythology saam, daam, dand, bhed). Recently One of my respected colleague "Sachin" left for a sabbatical. He was one of the Finishers whom I would hire any day on my team or would choose as a buddy to watch your back.  "Sachin" has a zeal like a Rhino or Triceratops to Finish what h

Woke up by earthquake at 7:00 AM in morning and then couldn't get to sleep. I took a bath, made my tea and started checking emails and saw that after last night deployment three storage node out of 100s of nodes were running into Full GC. What was special about the 3 nodes was that each one was in a different Data centre but it was named same app02.  This got me curious I asked the node to be taken out of rotation and take a heap dump.  Yesterday night a new release has happened and I had upgraded spymemcached library version as new relic now natively supports instrumentation on it so it was a suspect. And the hunch was a bullseye, the heap dump clearly showed it taking 1.3G and full GCs were taking 6 sec but not claiming anything.   I have a quartz job in each jvm that takes a thread dump every 5 minutes and saves last 300 of them, checking few of them quickly showed a common thread among all 3 data centres. It seems there was a long running job that was trying to replica