Skip to main content

Programatically extracting quoted reply from an email

When files are uploaded to our cloud file server, we wanted to send notification email per file with its own unique email address. I will discuss how to have so many unique email address without creating a user on mail server for each file and scale out the solution in some later blog. People can just hit reply button on the generated notification email and comment on the uploaded file. When reply email reaches back the server we want to extract the comment that user added after stripping out the quoted reply form the mail client and add the clean comment to file. Seems like an easy problem isnt it, but unfortunately there is no easy way to detect the quoted reply from an incoming email because different mail clients use different way to quote a reply. On top of it quoted reply of html emails are different than plain text quoted replies.
  1. Angle Brackets "> xxx zzz"
  2. "---Original Message---"
  3. "On such-and-such day, so-and-so wrote:"
  4. html email reply in thunderbird uses blockquote tags.
  5. yahoo/hotmail uses some div tags
Got an brilliant idea from someone to add a hash marker in the outbound notification email  so that when it comes back we can strip the text after that marker. Then I found other sites are already doing this like redmine or issueburner already does that. These guys add a marker text in outbound email like below

##### Please do not write below this line #####
Hi kalpesh,

The issue has been updated.

Updated by:     Kris Katta
Comment added:     this is a test comment
Kris Katta's Reply..

This is a test reply

To track the status of your request and set up a profile for yourself, follow the link below:





Now all that is left to extract the mail header so using some regex you can strip that. I  have handled thunderbird and outlook and will soon add yahoo/hotmail. Below is some sample code.


/**
 * @author kpatel
 */
public class QuotedReplyExtractor {
    public static final String 
REPLY_MARKER = "--- Please reply ABOVE THIS LINE to comment on this file ---";

    private static final List patterns = new ArrayList();
    static {
        patterns
                .add(Pattern.compile(".*on.*?wrote:", Pattern.CASE_INSENSITIVE));
        patterns.add(Pattern.compile("-+original\\s+message-+\\s*",
                Pattern.CASE_INSENSITIVE));
    }

    public String stripQuotedReply(String comment) {
        int startIndex = comment.indexOf(REPLY_MARKER);
        if (startIndex > 0) {
            comment = comment.substring(0, startIndex);
        }
        for (Pattern pattern : patterns) {
            Matcher matcher = pattern.matcher(comment);
            if (matcher.find()) {
                startIndex = matcher.start();
                comment = comment.substring(0, startIndex);
            }
        }
        return comment;
    }

} 

Comments

Post a Comment

Popular posts from this blog

Killing a particular Tomcat thread

Update: This JSP does not work on a thread that is inside some native code.  On many occasions I had a thread stuck in JNI code and it wont work. Also in some cases thread.stop can cause jvm to hang. According to javadocs " This method is inherently unsafe. Stopping a thread with Thread.stop causes it to unlock all of the monitors that it has locked". I have used it only in some rare occasions where I wanted to avoid a system shutdown and in some cases we ended up doing system shutdown as jvm was hung so I had a 70-80% success with it.   -------------------------------------------------------------------------------------------------------------------------- We had an interesting requirement. A tomcat thread that was spawned from an ExecutorService ThreadPool had gone Rogue and was causing lots of disk churning issues. We cant bring down the production server as that would involve downtime. Killing this thread was harmless but how to kill it, t

Adding Jitter to cache layer

Thundering herd is an issue common to webapp that rely on heavy caching where if lots of items expire at the same time due to a server restart or temporal event, then suddenly lots of calls will go to database at same time. This can even bring down the database in extreme cases. I wont go into much detail but the app need to do two things solve this issue. 1) Add consistent hashing to cache layer : This way when a memcache server is added/removed from the pool, entire cache is not invalidated.  We use memcahe from both python and Java layer and I still have to find a consistent caching solution that is portable across both languages. hash_ring and spymemcached both use different points for server so need to read/test more. 2) Add a jitter to cache or randomise the expiry time: We expire long term cache  records every 8 hours after that key was added and short term cache expiry is 2 hours. As our customers usually comes to work in morning and access the cloud file server it can happe

Preparing for an interview after being employed 11 years at a startup

I would say I didn't prepared a hell lot but  I did 2 hours in night every day and every weekend around 8 hours for 2-3 months. I did 20-30 leetcode medium problems from this list https://leetcode.com/explore/interview/card/top-interview-questions-medium/.  I watched the first 12 videos of Lecture Videos | Introduction to Algorithms | Electrical Engineering and Computer Science | MIT OpenCourseWare I did this course https://www.educative.io/courses/grokking-the-system-design-interview I researched on topics from https://www.educative.io/courses/java-multithreading-for-senior-engineering-interviews and leetcode had around 10 multithreading questions so I did those I watched some 10-20 videos from this channel https://www.youtube.com/channel/UCn1XnDWhsLS5URXTi5wtFTA