chendo

Dr Nic needed to upload his screencast to S3 so he could distribute his propagandapresentation about Javascript testing to the rest of the world. We tried to upload it from our work connection, but it kept failing with random errors, timing out, and killing the response time of the net since it was choking the upstream.

And so I suggested uploading the 400mb or so file to a remote server via a protocol that can easily be throttled, then uploading it to S3 from there. Since it was my idea, I was ‘honoured’ with handling the logistics of this.

The first step was picking a method to get the file on the remote server without killing your net. My first thought was rsync, and a quick Google search said that there’s an option called –bwlimit which allowed you set the I/O speed (file I/O, not network I/O, but it worked anyway). So about three hours later, the screencast was on the server and then it was down to getting this on to S3.

I found a tool named s3cmd, which is a Python-based command-line tool for S3 management. After you install it (apt-get install s3cmd is what I used on our Ubuntu server), then, you need to run s3cmd --configure, where you enter in your access key, secret key and an optional encryption password (I didn’t do this). Then, I ran s3cmd put < filename> s3://<bucket name> and it was done in less than a minute. The wonders of a 100mbit connection

Tags: amazon, mocra, s3, Technology