Skip to main content

Posts

Showing posts from May, 2010

Java com.sun.net.httpserver. HttpServer OutOfMemory Issue

We had to extract text out of documents that people were uploading to the site so we can index them. As this was going to be just a text extraction services I didnt wanted to go for some full blown tomcat server so wrote a quick http server using java

        this.httpServer = HttpServer.create(addr, 0);
        HttpContext context = this.httpServer.createContext("/", new DocumentProcessHandler());
        this.httpThreadPool = Executors.newFixedThreadPool(this.noOfThreads);
        this.httpServer.setExecutor(this.httpThreadPool);
        context.getFilters().add(new HttpParameterFilter());
        this.httpServer.start();Not adding all code for brevity. 

After being live for 2-3 days the process started crashing and it was a regular pattern. I added -verbose:gc -XX:+HeapDumpOnOutOfMemoryError to the startup script and had a crash dump after 3 days. Using eclipse Memory Analyzer and using Histogram to look at the heap dump revealed that we had 90K HttpConnection objects. wo…

RabbitMQ for Thumbnail generation in cloud

We are an cloud based fileshare and backup company. A frequent requirement for customers is to share their data with other people, for e.g. some customers will have people who will upload stock photos and the customer want to first look at thumbnail before really downloading the large picture. We were earlier using disk based queuing and there was a process that was running at night that used to come around and generate thumbnails. This was working earlier as most of the customers didn't had time sensitive thumbnail generation requirements. Now as we had grown the customer base has grown and people wanted their thumbnails to be generated instantaneously. So I came up with the architecture as shown in the image.



We use the new RabbitMQ with persister so that if it goes down the message are preserved. App nodes are tomcat servers that uses Java api to push messages to Rabbit as soon as a file is uploaded by the user and we also queue a message to scribe in case pushing a message to …

BlackBerry curve wired handsfree with normal 3.5 mm jack

Blackberry is just great, my wired handsfree with blackberry broke and was trying to pair my motorola wireless handsfree with blackberry, but couldn't get it working (may be because its 3 years old tech). But Blackberry is just great you can plugin any normal 3.5 mm jack headphones and it will work without a charm as it would use the phone microphone.