Need to transfer files from an FTP server to an S3 bucket quickly? I found an interesting solution.

Tools Needed:

Brief Overview:

  • LFTP will access a specified directory on an FTP server.
  • The files within the directory will be downloaded into the current working directory.
  • When finished downloading, the AWS CLI will upload all of the files to a specified bucket.
  • The file structure on S3 will match the exact structure from the FTP server.

Step by step:

  1. Install LFTP

    Here's the the install command for several operating systems:

    CentOS: $ sudo apt-get install lftp
    Ubuntu: $ sudo yum install lftp
    Mac w/ Homebrew: $ brew install lftp

  2. Install the Amazon AWS CLI

    Instructions are clearly described in the docs: https://docs.aws.amazon.com/cli/latest/userguide/install-bundle.html#install-bundle-user

    My only comment here is that I've found success on multiple systems with the "Install the AWS CLI without Sudo" method.

  3. Run the AWS CLI configuration

    Run $ aws configure and follow the prompts to input your credentials. Here's an example:

    $ aws configure
    AWS Access Key ID [None]: AKIAIOSFODNN7EXAMPLE
    AWS Secret Access Key [None]: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
    Default region name [None]: us-west-2
    Default output format [None]: json 


  4. LET IT RIP

    Now this is where the magic happens...

    Run the entire command below and replace everything between brackets to meet your needs.

    $ lftp -c 'open <ip_address>; user <username> <password>; mirror -e <remote_folder> <local_folder>; 
    quit' && aws s3 sync <local_folder> s3://<s3_bucket_location>/

    Here's an example without the placeholders:

    $ lftp -c 'open mywebsite.com; user myUserName myPassWord; mirror -e ./images/ .; 
    quit' && aws s3 sync . s3://images/

    Notice that I'm using . to pull everything into my current directory, but you could change that to anything.

    The amount of time this entire process takes will depend on your internet connection, the connection of the remote FTP server, the amount of files, and the size of the files.

  5. Drink a beer

    Grab an IPA while you wait for your files to transfer and then tell everyone you worked a ton of overtime to get this done... I'm kidding ;)

Conclusion

This worked for my needs because it's the perfect balance of quick setup and almost complete automation. Obviously it would be even better if there was a direct transfer between FTP and S3, but I haven't found any reliable methods for that yet. Let me know if you have!