I would say I didn't prepared a hell lot but I did 2 hours in night every day and every weekend around 8 hours for 2-3 months. I did 20-30 leetcode medium problems from this list https://leetcode.com/explore/interview/card/top-interview-questions-medium/. I watched the first 12 videos of Lecture Videos | Introduction to Algorithms | Electrical Engineering and Computer Science | MIT OpenCourseWare I did this course https://www.educative.io/courses/grokking-the-system-design-interview I researched on topics from https://www.educative.io/courses/java-multithreading-for-senior-engineering-interviews and leetcode had around 10 multithreading questions so I did those I watched some 10-20 videos from this channel https://www.youtube.com/channel/UCn1XnDWhsLS5URXTi5wtFTA
Posts
"The only thing constant in a Startup is Change" If you aren't changing fast enough then order and complacency sets in which leads to mediocrity and you can soon become obsolete. We do biweekly releases and want to move to weekly and then daily releases. You can increase the release cadence, if you are able to automate most of the testing. But automation can't detect everything and things may break and the intent is to establish a culture of how can you prevent it but if you cant prevent it then how fast can diagnose and recover from it. For a long time I had been pushing hard for a working centralized logging and after almost an year of trying to scale the ELK framework finally our team has been able to put a fast centralized logging system that's ingesting Terabytes of log per minute from thousands of nodes in production. This weekend we deployed a major architecture change by creating swimlanes in our cloud by directing different types of traffic to di...
We should not repeat mistakes and if its the end user that is committing same mistake again and again then the UX should be changed to make it difficult for user to perform such actions. Make the user apply some cognitive thinking before he can do a detrimental action and make the action obscure. For e.g. we are a filesystem company and "Trash delete" is a detrimental action, over past few years I had several tickets where user accidentally clicked Empty trash and deleted all his files or he clicked Delete Permanently instead of Restore. He was able to do all this because in the UX the actions are close to each other There are users who did this and immediately created tickets under panic and some even blaming that system did delete instead of restore. I know in UI it may look bad but I have seen several companies solving this by making user think twice before invoking detrimental action like asking a captcha or make the delete and restore flow differ...
This week I saw the movie Dangal trailer and the actor says "If you win silver medal, today or tomorrow people will forget you but if you win Gold medal you will become a source of inspiration". Also today morning I saw K L Rahul got out on 199 runs and the commentators are complaining about his loss of focus on one ball and he got out http://timesofindia.indiatimes.com/sports/cricket/news/india-v-england-5th-test-day-3-rahuls-199-drives-india-to-commanding-position/articleshow/56045818.cms This reminds me of Startups and Finishers, give up too close to the finish line and you wont be remembered but if you finish it you may become an inspiration.
Burnout is very common in startup and it hits you more often when you are at the same startup for 7+ years. Much of burnout is felt when you are working hard but progress is slow and you have high expectations of yourself and the team. I read this excellent article http://andrewdumont.me/avoiding-burnout/ and it felt almost Deja vu to me. I work from home so it hits even hard to me as I have less people to share it with. Here are the ways I have been trying to curtail burnout and hope it help others too: Workplace :Keep your workplace organized and stick to a routine. I have an office room at home and I am in work mode when I am in that office. Morning :Keep 1 hour in mornings on something outside work. I read a lot in morning and don't come online or check emails until that 1 hour is done. Workout : rain, heat or winter, I try to walk for 30 min in afternoon as much as possible. This gives you some time for meta thinking and de stresses you from the morning chaos. Man...
A quality of a good Startup engineer is "Finishing what you started". I value finishers more than people who talk good theory and start many things but don’t finish most of them. Most of the time people get excited and start many things concurrently but leave them in limbo or they finish 95% of the thing. Over 7 years at my current Startup I have observed that in many projects doing first 95% may take you X units of time but its the remaining 5% that demands 10x cognitive effort and time from you. Its this last 5% where people either burnout or give up. Its the finishers who keep the compass pointing to North star, never give up and do what ever it takes to Finish( in indian mythology saam, daam, dand, bhed). Recently One of my respected colleague "Sachin" left for a sabbatical. He was one of the Finishers whom I would hire any day on my team or would choose as a buddy to watch your back. "Sachin" has a zeal like a Rhino or Triceratops to Finish what h...
Woke up by earthquake at 7:00 AM in morning and then couldn't get to sleep. I took a bath, made my tea and started checking emails and saw that after last night deployment three storage node out of 100s of nodes were running into Full GC. What was special about the 3 nodes was that each one was in a different Data centre but it was named same app02. This got me curious I asked the node to be taken out of rotation and take a heap dump. Yesterday night a new release has happened and I had upgraded spymemcached library version as new relic now natively supports instrumentation on it so it was a suspect. And the hunch was a bullseye, the heap dump clearly showed it taking 1.3G and full GCs were taking 6 sec but not claiming anything. I have a quartz job in each jvm that takes a thread dump every 5 minutes and saves last 300 of them, checking few of them quickly showed a common thread among all 3 data centres. It seems there was a long running job that was trying ...
I have seen my Kid grow from 0 to 6 year and one startup growing from 20 to 150 employees and another growing from 5 to 300+ employees. A common trend I seen is hanging on to old things. Not sure if its due to sentimental value attached to things or inability to make hard choices. My Kid still has his toys that he used to play when he was 1 year old. He wouldn’t let us throw them or give it to someone, I am recently working with him to let go of old things. Problem with old things is that they are a drag, they just sit around the house and are a clutter. Its the same with old books, I have many of them and one day I need to go and clean them once I read a book, I rarely read them again so why not donate them to library. A common observation with Startups I have worked is feature bloat due to holding on to old features. Features that are not maintained but are in use by a handful of customers causes drag. Recently I wanted to refactor folder listing code t...
Small shitty things frustrates me, like someone came up with an idea to put epic on each Jira task. Well its suited for big Tasks that has lot of subtasks but 50% + of my cases don’t require an Epic. I am lazy and dont like to waste my time. I am creating an issue and filled all things and now this process requires me an Epic, total BS. Yes this is total BS so either the process requires fixing where epic is first required field so I wont invest filling so many fields or this needs to be removed/made optional.
I always believed in "Trusting data" over human gut but lately I have been observing a simple fact that being data driven has a side effect, "it motivates you and keeps you on track". Some recent examples are: Fitbit: Last year I started the afternoon walk because by 2:00PM after doing calls and replying to a lot of emails the brain would be fried and I wont have energy left to code or think. Doing these 30 min walk daily recharges the brain. I had an Omron pedometer sitting around for almost an year and I seldom took it with me on walks. The problem with it was that it used to store last 30 days data of my steps and other things but it didn’t had a good way to graphically see the data. When it comes to data "less is more" but also one more important aspect is that you need to present your data in graphs so it doesn’t take a huge amount of cognitive effort to make sense of it. Recently my employer gave a fitbit to everyone who participated in summe...
It seems Mysql is rock solid if you want: Transactions ACID support So I would still recommend mysql for any thing that is mission critical data and it should be the primary datastore for your transactions. But what about derived data, immutable data or analytical data? In past I have built large scale cluster of mysql server storing metadata about billions of files and folders used by tens of thousands of customers daily and its scaling fine and working good, its still growing at a healthy rate and holding up. But this requires a lot of baby sitting if you have 100s of nodes and you need to do replication add more nodes rebalancing data monitoring entire cluster Sharding Backup/restore You have to write a lot of tooling and lot of monitoring/babysitting to scale the cluster. Plain stock Mysql will scale up to a limit but vertically scaling has its own issues. So +1 for Mysql but not everything should be stuffed there. Recently me and my team built ...
When you are operating at scale and handling peak traffic of 1K+ request per sec on a jvm then no matter what you do you would get hit by a Thundering herd problem. There would be operations that happens once in a while but take more than 10 sec and if there are too many of them happening then you could choke backend services or worse cause a downtime. So you need to Rate limit these long running operations that only X can run at a time, this way you are leaving room for running lots of short lived transactions. When you have millions of users then not all users are doing these long running operations and not all traffic is coming from online users. We are a cloud storage company and we give sync client to users so 80%+ traffic at a given time is coming from these clients that are trying to sync changes between cloud and local system behind the scenes. Our application is written using REST apis and these clients are using the same REST apis that our web ui is using. ...
Looks like these days bitching about email is my favourite topic. What can I do no matter what I do I just cant keep up with email flow. The new kind of emails just keep increasing, I was doing a good job 2 years ago and close to inbox zero daily and was writing a lot of code but two things have increased the email flow to me recently: Internal reports about production montioring from Newrelic, haproxy, exception analysis by team. Review requests. This is a recent one. Most review requests are sent to Java review group and somehow I atleast need to spend 1-2 min on every email to first decide whether this is for me or not. Even if there are 30 review request per day it consumes 30 min of my time just to make sense of it. Daily when I see 100 emails to be answered I feel disheartened that I cant write any code today. Both above items are necessary evil, #1 has increased production stability so my weekends are less busy. #2 I am hoping...
Monitoring tools exposes a lot of data and we use Nagios, cacti, graphite,newrelic, mixpanel, flurry, boundary and many more tools. But one of the ask for Support and marketing teams is how can they internally know if something is wrong. We cant expect them to wade through so many systems and so many applications to make sense of what is operational and what is not. For e.g. we use a lot of services to serve the cloud filer server solution and this is the first page of status of our services in new relic and it spans 2 pages. Support team and management relies on Operations team to notify them if an issue is on going. To make this easy I did a Proof of concept integration application responsible for serving main website with Statuspage.io. The idea is simple Create public metrics in statuspage.io that are human readable. Query new relic and various systems for application status. Map new relic green/red/yellow status and other system status to statuspage.io s...
We run a complex system with multiple services and every 2 or 3 week we update the Java applications. I want to do it every week as most applications are stateless and can be patched anytime but the application serving the main website is using sticky session. We are working to make it failover sessions, once we do that, we can do mid week deployment and that will allow us to go faster than 3 weeks. This week I pushed a huge infrastructure change related to user Id generation. I had asked ops team to check the status of new relic after the midnight deployment and it looked like this so everyone was happy. I woke up and checked new relic mobile app and things looked ok to me. After finishing my morning routines I ran my daily exception report and one thing that caught the eye was 90K exceptions in last 12 hours in one of the files I had changed. To gauge the impact I went in new relic and it showed me an error rate of 0.07 in one of the app I then check...
It seems I have become an EmailSlave. The first half of the day is spent in just answering to emails. There are so many emails where I am copied but I need not be. There are many emails where its a 1-2 page email and somewhere down someone says @KP please answer this. So it seems daily my work schedule is: Signin to newrelic and check anomalies for 15 min. Check emails related production exception report and yes there are a ton of these report daily. Need a better tool here as this model is not scalable. I need to reduce the incoming data at me to only see relevant data like what newrelic does. May be I need to create a webapp out of these emails. Check emails for next few minutes before team calls Do team calls Then again back to checking emails until a I have taken a best shot at answering everyone waiting for my reply. Attend team meetings on Tue/Thu Being an architect and coder at heart I don't feel satisfied at end of the day if there is nothing tangible...
My wired mouse in home setup is as old as I started working from home for my employer. Suddenly this week the scroll wheel would scroll up fine but it would not scroll more than one page, you had to scroll up half page before it would resume scroll down. At first I thought its my Firefox upgrade but soon I noticed same issue in eclipse and terminal. In evening I use the laptop without wired mouse and it would work fine. That led me to conclusion that issue is with mouse. In USA people have a replace mentality, if something is not working and cheap, you can replace it. I was this close to ordering a new mouse on amazon when I thought let me shake it to see if something is stuck, but nothing came out. I then saw a small screw in bottom, I unscrew it and I hit the jackpot, I was like wth "Lint". My friend had similar issues with Lint in Iphone5 when his iphone wont charge and we used a breadtie to get out the lint and it was charging fine. I was like yay problem solve...
http://astartupguy.blogspot.com/2014/12/can-you-remain-fullstack-developer.html
I am intrigued by this webhook concept and it seems a very nice way for B2B communication. Webhooks are powerful and it eliminates Poll for integrating with third parties. All you need is to have some REST api registered that will be called when an event occurs. A good e.g. of webhook for a cloud storage provider can be "automatically print this document on registered printers when a file is dropped in this folder". Lets assume all the customer needs to do is register a webhook "http://xyz.foo.com/printer/xxd344/print?token=authTokenXXX and the cloud storage provider can then call this url and POST the body of document in input. Recently I had a chance to play with webhooks when I was trying to move jenkins to EC2 for my f riend and as part of it I moved his svn to assembla.com. I saw webhooks and I thought I can integrate commits into svn hosted by assembla.com to pivotaltracker.com tickets. It took just 1 hour to do it but it was fun, apparently there are po...
NewRelic really shines at discovering these data driven performance issues. Earlier we would find them late or these would be buried but now they seem so obvious if the engineer is paying attention. I was casually trolling new relic and sorted all apps by avg time per api and one of our core application in one DC was taking twice the avg time for each call than all other DCs. I immediately compared that DC with other DCs and I saw was a graph like below in DC1 and I saw this in DC2 Clearly DC1 is spending abnormal amount of time in database. So I went to database view and saw this in DC1 and I saw this in DC2 Clearly something is weird in DC1 even though its same codebase. 309K queries per minute seems abnormal. Within 5 min I found out its a n query problem. Aparently some customer has 4000 users and he has created 3000 groups and the group_member table has 40K rows for this customer. Normally all of our customers would create 10-50 gro...