Skip to main content

Finally uploading to S3 using java sdk

One of the reasons I joined my employer was that I would get some experience with AWS and S3. But so far the ride is a roller coaster ride and I have been working on something totally different and first we created our own S3 like system and then uploading to S3 and AWS was handled by another rockstar friend in the team :).  Now it almost seems like I wont get a chance to work on it so I was like let me anyway write a simple program, so I signed up for AWS and tried this. Its a simple thing and took me only 2-3 hours but its worth spending it because for past 3 weeks I am working on cleaning technical debt accumulated over years and that doesnt give that much joy. Doing this give me some joy.

On that note one of the team mate shared this today to me

public class S3StorageService {
    private static final AppLogger logger = AppLogger.getLogger(S3StorageService.class);
    private String bucketName;
    private String accessId;
    private String secretKey;
    private String tempFolder;

    public String uploadObject(String guid, InputStream in) {
        File file = getTempStoragePath(guid);
        try {
            saveToTempFile(in, file);
            String key = getObjectKey(guid);
            AmazonS3 s3Client = getS3Client();
            s3Client.putObject(new PutObjectRequest(bucketName, key, file));
            return key;
        } finally {
    private File getTempStoragePath(String guid) {
        File baseDir = new File(tempFolder);
        File hashedPath = new File(new File(new File(baseDir, guid.substring(0, 2)), guid.substring(2, 4)),
                guid.substring(4, 6));
        return new File(hashedPath, guid);
    public void deleteObject(String guid) {

        String key = getObjectKey(guid);
        AmazonS3 s3Client = getS3Client();
        s3Client.deleteObject(new DeleteObjectRequest(bucketName, key));

    public void downloadObject(String guid, OutputStream out) throws StorageServiceException {
        String key = getObjectKey(guid);
        AmazonS3 s3Client = getS3Client();
        S3Object object = s3Client.getObject(new GetObjectRequest(bucketName, key));
        InputStream objectData = null;
        try {
            objectData = object.getObjectContent();
            IOUtils.copy(objectData, out);
        } catch (IOException e) {
            throw new StorageServiceException(e);
        } finally {
    private String getObjectKey(String guid) {
        return  "Files/" + guid;

    private AmazonS3 getS3Client() {
        AWSCredentials awsCredentials = new BasicAWSCredentials(accessId, secretKey);
        AmazonS3 s3Client = new AmazonS3Client(awsCredentials);
        return s3Client;


Popular posts from this blog

RabbitMQ java clients for beginners

Here is a sample of a consumer and producer example for RabbitMQ. The steps are
Download ErlangDownload Rabbit MQ ServerDownload Rabbit MQ Java client jarsCompile and run the below two class and you are done.
This sample create a Durable Exchange, Queue and a Message. You will have to start the consumer first before you start the for the first time.

For more information on AMQP, Exchanges, Queues, read this excellent tutorial
import com.rabbitmq.client.Connection; import com.rabbitmq.client.Channel; import com.rabbitmq.client.*; public class RabbitMQProducer { public static void main(String []args) throws Exception { ConnectionFactory factory = new ConnectionFactory(); factory.setUsername("guest"); factory.setPassword("guest"); factory.setVirtualHost("/"); factory.setHost(""); factory.setPort(5672); Conne…

What a rocky start to labor day weekend

Woke up by earthquake at 7:00 AM in morning and then couldn't get to sleep. I took a bath, made my tea and started checking emails and saw that after last night deployment three storage node out of 100s of nodes were running into Full GC. What was special about the 3 nodes was that each one was in a different Data centre but it was named same app02.  This got me curious I asked the node to be taken out of rotation and take a heap dump.  Yesterday night a new release has happened and I had upgraded spymemcached library version as new relic now natively supports instrumentation on it so it was a suspect. And the hunch was a bullseye, the heap dump clearly showed it taking 1.3G and full GCs were taking 6 sec but not claiming anything.

I have a quartz job in each jvm that takes a thread dump every 5 minutes and saves last 300 of them, checking few of them quickly showed a common thread among all 3 data centres. It seems there was a long running job that was trying to replicate pending…

Spring query timeout or transaction timeout

If you are using spring to manage transactions then you can specify default transaction timeout using

    <bean id="transactionManager"
        <property name="dataSource" ref="dataSource" />
        <property name="defaultTimeout" value="30" /> <!--30 sec--->             

or you can override the timeout in the annotation

    @Transactional(readOnly = false, timeout=30)

or if you are doing it programatic transactions then you can do

DataSourceTransactionManager transactionManager = new DataSourceTransactionManager(dataSource);

 or you can override the timeout for one particular transaction

TransactionTemplate transactionTemplate = new TransactionTemplate();