Skip to main content

Finally uploading to S3 using java sdk

One of the reasons I joined my employer was that I would get some experience with AWS and S3. But so far the ride is a roller coaster ride and I have been working on something totally different and first we created our own S3 like system and then uploading to S3 and AWS was handled by another rockstar friend in the team :).  Now it almost seems like I wont get a chance to work on it so I was like let me anyway write a simple program, so I signed up for AWS and tried this. Its a simple thing and took me only 2-3 hours but its worth spending it because for past 3 weeks I am working on cleaning technical debt accumulated over years and that doesnt give that much joy. Doing this give me some joy.

On that note one of the team mate shared this today to me

http://www.businessinsider.com/syndromes-drive-coders-crazy-2014-3



public class S3StorageService {
    private static final AppLogger logger = AppLogger.getLogger(S3StorageService.class);
    private String bucketName;
    private String accessId;
    private String secretKey;
    private String tempFolder;

    @Override
    public String uploadObject(String guid, InputStream in) {
        File file = getTempStoragePath(guid);
        try {
            saveToTempFile(in, file);
            String key = getObjectKey(guid);
            AmazonS3 s3Client = getS3Client();
            s3Client.putObject(new PutObjectRequest(bucketName, key, file));
            return key;
        } finally {
            FileUtils.deleteQuietly(file);
        }
    }
    private File getTempStoragePath(String guid) {
        File baseDir = new File(tempFolder);
        File hashedPath = new File(new File(new File(baseDir, guid.substring(0, 2)), guid.substring(2, 4)),
                guid.substring(4, 6));
        hashedPath.mkdirs();
        return new File(hashedPath, guid);
    }
    @Override
    public void deleteObject(String guid) {

        String key = getObjectKey(guid);
        AmazonS3 s3Client = getS3Client();
        s3Client.deleteObject(new DeleteObjectRequest(bucketName, key));
    }

    @Override
    public void downloadObject(String guid, OutputStream out) throws StorageServiceException {
        String key = getObjectKey(guid);
        AmazonS3 s3Client = getS3Client();
        S3Object object = s3Client.getObject(new GetObjectRequest(bucketName, key));
        InputStream objectData = null;
        try {
            objectData = object.getObjectContent();
            IOUtils.copy(objectData, out);
        } catch (IOException e) {
            throw new StorageServiceException(e);
        } finally {
            StreamUtils.closeQuietly(objectData);
        }
    }
    private String getObjectKey(String guid) {
        return  "Files/" + guid;
    }

    private AmazonS3 getS3Client() {
        AWSCredentials awsCredentials = new BasicAWSCredentials(accessId, secretKey);
        AmazonS3 s3Client = new AmazonS3Client(awsCredentials);
        return s3Client;
    }
}

Comments

Popular posts from this blog

RabbitMQ java clients for beginners

Here is a sample of a consumer and producer example for RabbitMQ. The steps are
Download ErlangDownload Rabbit MQ ServerDownload Rabbit MQ Java client jarsCompile and run the below two class and you are done.
This sample create a Durable Exchange, Queue and a Message. You will have to start the consumer first before you start the for the first time.

For more information on AMQP, Exchanges, Queues, read this excellent tutorial
http://blogs.digitar.com/jjww/2009/01/rabbits-and-warrens/

+++++++++++++++++RabbitMQProducer.java+++++++++++++++++++++++++++
import com.rabbitmq.client.Connection; import com.rabbitmq.client.Channel; import com.rabbitmq.client.*; public class RabbitMQProducer { public static void main(String []args) throws Exception { ConnectionFactory factory = new ConnectionFactory(); factory.setUsername("guest"); factory.setPassword("guest"); factory.setVirtualHost("/"); factory.setHost("127.0.0.1"); factory.setPort(5672); Conne…

Spring query timeout or transaction timeout

If you are using spring to manage transactions then you can specify default transaction timeout using

    <bean id="transactionManager"
        class="org.springframework.jdbc.datasource.DataSourceTransactionManager">
        <property name="dataSource" ref="dataSource" />
        <property name="defaultTimeout" value="30" /> <!--30 sec--->             
    </bean>

or you can override the timeout in the annotation

    @Transactional(readOnly = false, timeout=30)


or if you are doing it programatic transactions then you can do


DataSourceTransactionManager transactionManager = new DataSourceTransactionManager(dataSource);
transactionManager.setDefaultTimeout(30);

 or you can override the timeout for one particular transaction

TransactionTemplate transactionTemplate = new TransactionTemplate();
transactionTemplate.setTimeout(30);

Python adding pid file

I have a thumbnail generator that launches multiple processes and the correct way to shut it down is to send kill -HUP to the parent process. To automate I had to write a pid file from python, it was a piece of cake
def writePidFile(): pid = str(os.getpid()) f = open('thumbnail_rabbit_consumer.pid', 'w') f.write(pid) f.close()