Need to transfer files from an FTP server to an S3 bucket quickly? I found an interesting solution.
- LFTP - A command line FTP client. https://lftp.yar.ru/
- AWS CLI - The standard AWS command line interface. https://aws.amazon.com/cli/
- LFTP will access a specified directory on an FTP server.
- The files within the directory will be downloaded into the current working directory.
- When finished downloading, the AWS CLI will upload all of the files to a specified bucket.
- The file structure on S3 will match the exact structure from the FTP server.
Step by step:
Here's the the install command for several operating systems:
$ sudo apt-get install lftp
$ sudo yum install lftp
Mac w/ Homebrew:
$ brew install lftp
Install the Amazon AWS CLI
Instructions are clearly described in the docs: https://docs.aws.amazon.com/cli/latest/userguide/install-bundle.html#install-bundle-user
My only comment here is that I've found success on multiple systems with the "Install the AWS CLI without Sudo" method.
Run the AWS CLI configuration
$ aws configureand follow the prompts to input your credentials. Here's an example:
$ aws configure AWS Access Key ID [None]: AKIAIOSFODNN7EXAMPLE AWS Secret Access Key [None]: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY Default region name [None]: us-west-2 Default output format [None]: json
LET IT RIP
Now this is where the magic happens...
Run the entire command below and replace everything between brackets to meet your needs.
$ lftp -c 'open <ip_address>; user <username> <password>; mirror -e <remote_folder> <local_folder>; quit' && aws s3 sync <local_folder> s3://<s3_bucket_location>/
Here's an example without the placeholders:
$ lftp -c 'open mywebsite.com; user myUserName myPassWord; mirror -e ./images/ .; quit' && aws s3 sync . s3://images/
Notice that I'm using
.to pull everything into my current directory, but you could change that to anything.
The amount of time this entire process takes will depend on your internet connection, the connection of the remote FTP server, the amount of files, and the size of the files.
Drink a beer
Grab an IPA while you wait for your files to transfer and then tell everyone you worked a ton of overtime to get this done... I'm kidding ;)
This worked for my needs because it's the perfect balance of quick setup and almost complete automation. Obviously it would be even better if there was a direct transfer between FTP and S3, but I haven't found any reliable methods for that yet. Let me know if you have!