Wednesday, July 23, 2014

Keep slogging help is on the way

When working in startup, don’t lose hope, keep slogging as once you reach scale help is on the way.

So far I have been scaling Mysql by throwing more hardware and focusing only on performance issues detected by new relic or my custom report but there were some data driven anomalies where 99% calls to this query would take 1sec but one call to same query would take 10sec.  I wasn’t focusing much on it because it was a blip in the graph and there were too many other issues to focus on.

Now we got a full time Mysql engineer who is looking at these queries and hunting down suspects. Today he found this query

select sum(points) from (
            select g.all_versions_size as points from folders_trash f
            inner join groups g on f.folder_id = g.folder_id
            union all
            select e1.size as points from groups_trash g1
            inner join entries e1 on g1.group_id=e1.group_id
            union all
            select e2.size as points from entries_trash e2
            ) s

Now this query works 99% of the time faster because most customers have very few data in trash but some customers have 400K+ rows in trash and for them this  was creating a temp table with 400K rows causing the blip.

Changing this query to something like below would create only 3 temp table row, the query became fast and uses less resources.

select sum(points) from (
            select sum(g.all_versions_size) as points from folders_trash f
            inner join groups g on f.folder_id = g.folder_id
            union all
            select sum(e1.size) as points from groups_trash g1
            inner join entries e1 on g1.group_id=e1.group_id
            union all
            select sum(e2.size) as points from entries_trash e2
            ) s

So don't lose hope, find creative ways initially like throwing more hardware to the problem if you can. When you reach scale expert help will come on the way :).

Wednesday, July 16, 2014

yellowstone national park lost reciept

Last month I went to yellowstone and we were supposed to stay for 6 days.  Upon entry we were issued a reciept and I lost it. On 4th day we had to go out of the park and come back in. I talked to a ranger about the situation and he said before exiting stop by the gate and ask the ranger at gate.  I did the same and he was kind enough to issue a gate pass to me valid for 2 more days.

Blogging this in case someone else runs into same situation and is googling :).

Spring proxyFactoryBean not applying interceptors and circular dependencies

We use AOP a lot but due to legacy reasons we dont use aspectJ yet and we are using Spring ProxyFactoryBean and use interceptors to apply AOP behavior like light weight instrumentation or transactions.

Lately we ran into two or three issues where on some proxyFactoryBeans the interceptors wont be applied.  My hunch was it was some circular dependency between beans so I added DEBUG on org.springframework.beans in log4j and it spitted tons of logs, I did a grep

 grep "that is not fully initialized yet - a consequence of a circular reference" ~/apache-tomcat-7.0.30/logs/tomcat.log |awk -F "singleton" '{print $2}'|sort|uniq  

it would print some message like 



bean 'gdriveClient' that is not fully initialized yet - a consequence of a circular reference
 bean 'gdriveRestServiceHelper' that is not fully initialized yet - a consequence of a circular reference
 bean 'filerServiceRequestHandlerFactory' that is not fully initialized yet - a consequence of a circular reference
 bean 'objectReplicationManager' that is not fully initialized yet - a consequence of a circular reference
But this was a painful way to  analyse it and I wanted to future proof the code so that developers in future cant introduce a circular dependency.

Today I found that instead of loading context like 
ClassPathXmlApplicationContext context = new ClassPathXmlApplicationContext("/applicationContext.xml");
You  can do something like

ClassPathXmlApplicationContext
context = new ClassPathXmlApplicationContext();
context.setAllowCircularReferences(false); 
context.setConfigLocation("/applicationContext.xml");
context.refresh();

and this would fail on first circular reference encountered.  This prints in log entire hierarchy of bean it was trying to load so its very easy to chain the beans to find circular reference. I haven’t committed the code yet but the idea is to fix all existing circular dependencies and then enable this code.