Programming fun at startup

This weekend was painful. Hardware came but out of 12 mysql servers, 3 had hardware issues, so only 8 mysql available (4 pair of Master-slave). I didn't wanted to take a risk of having a cluster up with only 1 master so left that one intentionally. Started migration on 1st DC and after 4 hours the DC guys were doing some inbound network maintenance and took the in bound network to DC down and it took them 3 hours to restore it, so 3 hours were lost. We were up till 6:00 AM on Saturday morning and only able to finish 8-10 nodes. We worked on Saturday night and as usual Berkely db gave us last minute jitters, one node was badly fragmented and took 10 hours to migration and one other node stuck in middle of migration so had to stop migration and run db_recovery and then resume again. In all 12-15 nodes migrated on that day. We are now almost done 60% done in all DCs over last 3 weeks. But all this pain is worth having a calm Monday where no system alerts or no Sales/Operations...

I hate when developers writes useless comments in code like this try{ }finally {             // Closing the connection             helper.closeConnection(cnn);   } Anyone reading the function name close connection can figure out this is closing the connection. I try to give function a meaningful name so that it conveys the intent and comments are not required. I write method comments only when function is doing some complex algorithm or some trick. I also hate comments like this     /**      *      * @param customer      * @param ids      * @return      */ This is also useless comment generated by eclipse automatically but the developer didn't added a comment. Screen size is limited and I would rather see code in it than useless comments.

Interesting to know how different people can come to same architecture to solve scalability issues.  I just read this article published today called as cell architecture http://highscalability.com/blog/2012/5/9/cell-architectures.html   and I came up with same architecture in our company as highlighted in the  diagram below and I am calling it as "Cluster" instead of "cell" or "pod" but the concept is same. You can read a bit more on the link I published above but the main reason why we chose this architecture were: Failure of a cell doesn't cause the entire DC to go down. We can update one cell and watch out for a week before pushing the release to all cells. We can have difference capacity for different cells (Enterprise customers vs trial customers). We can add more cells or mores mysql host to once cell if it has a capacity problem in one component. Ideally you want to make it as homogeneous as possible but let say for some reason in o...

As discussed in Part2 of the series we do Horizontal sharding for any schemas that will store 100M+ rows. As we are a cloud file server no data is shared between two customers a perfectly isolation can be achieved easily. One year back when I was thinking on designing the schema there were many alternatives One shard per customer mapped to one database schema : Rejected this idea because mysql stores 1 or more files per table in physical file system and linux file system chokes after some no of files in a folder. We had faced this problem when storing the real files on filers (topic of another blog post). One shard per customer maped to one set of tables in database schema : This would solve the issue of multiple files in a folder but again it would lead to too many files on the disk and operating system can choke on it. Also we have customers to do a trial for 15 day and never signup, so too much for ops team to manage for these trials. Many customers in one shard ...

No hardware this weekend due to delays by dell so only  2 nodes were migrated. Ran into another crappy Berkely DB issue where for no reason it would get stuck in native code. java.lang.Thread.State: RUNNABLE at com.sleepycat.db.internal.db_javaJNI.Dbc_get(Native Method) at com.sleepycat.db.internal.Dbc.get(Dbc.java:61) at com.sleepycat.db.Cursor.getNext(Cursor.java:350) at com.xxx.migration.Bdb2MysqlMigrator.migrateFolder(Bdb2MysqlMigrator.java:291) It was already 4:00 AM in the morning and I was pissed that Berkely db is giving last minute pains. In India they say " Bujhta Diya last main Tej Jalta hai " or "dying lantern runs more bright at end" .  We tried doing a defrag of database but it didnt helped. Ultimately we moved the customer db to its own folder and then ran catastrophic recovery and restarted the migration. From past Berkely db migrations I had the experience that you would get these kind of issues and you might have to restart migrations. ...

in Mysql sharding at my company part1 I covered that when a customer registers we assign him next free shard. Now this problem seems simple at first but it has its own twists. You could have different strategies to determine next free shard id Based on no of rows in shard Based on no of customers in shard After lots of thinking I use the first approach. When a customer registers I pick up the next free shard by looking up information schema and query least used 8 shards and then randomly pick one of those.  Picking shards based on no of customers was rejected because we use a nagios monitor to test registration and that causes lots of dummy registrations and also QA team does registrations and some times people will register a trial use for 15 days and wont convert as they are either spammers or just want to use us for sharing large files for <15 days. The reason to not always pick the first least used shard is that 6 months down the line if we add 2 more shards to clust...