How to send data to Faraday

Check the format and required information 

First, please see http://knowledgebase.faraday.io/help-and-support/data-sources/what-fields-are-required-when-sending-data-to-faraday

Decide what you're going to upload

For example, every day, you upload 2 files:

• orders_2018-09-29.zip.gpg
  ...which contains a single CSV file, orders.csv
  ...every line of which has name, address 
• customers_2018-09-29.zip.gpg
  ...which contains a single CSV file, customers.csv
  ...every line of which has name, address, email 

Extra fee applies

• Multiple CSVs per file
• CSVs without headers
• Tab separated or fixed width
• Joining in identifying information from other files 

Common mistakes

• Putting 2 CSVs in a single ZIP file
  ...instead, upload 2 ZIP files with 1 CSV each! 
• Uploading an orders file with no identifying information, only an order ID
  ...instead, put name, address, etc. on every order line before you upload!
• Encrypting before zipping
  ...instead, zip before encrypt! 

1. Send via Amazon S3 (our bucket)

First, we will create a bucket on our end.

To provision access to a bucket, Faraday will need an AWS username like:

• arn:aws:iam::$YOUR_AWS_ACCOUNT_ID:user/acme_to_faraday

Note: when you upload, you will need to set your ACL to:

• bucket-owner-full-control

We'll also need for you to put the following in your IAM user's inline policy (example showing if the bucket name was "acme_to_faraday"):

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetBucketLocation"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::acme_to_faraday"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject"
            ],
            "Resource": [
                "arn:aws:s3:::acme_to_faraday/to_faraday/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:GetObjectAcl",
                "s3:GetObjectVersion"
            ],
            "Resource": [
                "arn:aws:s3:::acme_to_faraday/*"
            ]
        }
    ]
} 

2a. Send via Amazon S3 (your bucket)

Upload your files to a bucket you control.
Then, provision us access to a folder by defining a bucket policy:

(in the example below, it is for a folder to_faraday/ in the bucket acme-faraday-interchange )

{
  "Version": "2012-10-17",
  "Id": "FaradayAccessToBucket",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::113233973114:user/stagecraft-download_s3"
      },
      "Action": [
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::acme-faraday-interchange"
      ]
    },
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::113233973114:user/stagecraft-download_s3"
      },
      "Action": [
        "s3:GetObject",
        "s3:GetObjectAcl",
        "s3:GetObjectVersion"
      ],
      "Resource": "arn:aws:s3:::acme-faraday-interchange/*"
    }
  ]
}

Then, we have to add the bucket name to the IAM user here https://console.aws.amazon.com/iam/home?region=us-east-1#/users/stagecraft-download_s3

2b. Upload with SFTP

We will require your SSH public key. They look like this:

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdtiZZsY6tR0JUWD1OtnkSFOK9wE3kiq5zNBwRKobpCsuSE3/vHZr20MeqhP7HhHvN/mlO81xvY9Hz4AfmslJWJAxw0kDbUj9Gz639FlnalUlNcMkSoG2iSYn/krB/6YG76Eb64CF14X+9aLKgyKrwgOc90AF0aD22Ig3ddsj0HW1YyxB0xH3NQVrdALQME+FILA+Fv20Ihmu8I/en8pdAcFa3vQDpE5aWc36w2/0xKU7luTwXjvJV0bOuDqMIe84KU4oHB0JrhnLTVxUDl+lEsmjZJqj7FBVV8Wg8Wqk0t3Iz1+4k2U68VPpCIY2m9fXkoh/rFJUYT4rRP+lwhJmt 

Needless to say, they are NOT secret - that's why they're called "public" keys.
(Do not EVER send anybody, including Faraday, your SSH private key.)
If you don't have an SSH keypair, you can generate one like this https://www.digitalocean.com/docs/droplets/how-to/add-ssh-keys/create-with-openssh/.

Inputting details in our app:

You can navigate to: https://app.faraday.io/sources and hit "Create source"

Fill out the fields here:

examples:

File glob: to_faraday/$file_name_YYYYMMDD.csv
AWS bucket: acme-faraday-interchange
AWS region: us-east-1

Connecting and sending data:

Using your favorite SFTP client:

• Host:  s-9733f4cc1a6049ed9.server.transfer.us-east-1.amazonaws.com 
• Port: 22 
• Username: varies by client (for example, acme1 ) 
• Password: not enabled, you must utilize a SSH public key as explained above 
• Folder: to_faraday/  

Let's pretend you work for Acme, Inc. Your SSH private key file is /path/to/acme_rsa  (remember, this is a secret, don't sent it to us) and the username given to you by Faraday is acme1 .

You will upload files like this:

$ sftp -i /path/to/acme_rsa acme1@s-9733f4cc1a6049ed9.server.transfer.us-east-1.amazonaws.com

Connected to acme1@s-9733f4cc1a6049ed9.server.transfer.us-east-1.amazonaws.com

$ sftp> put helloworld.csv.gz to_faraday/helloworld-2019-01-15.csv.gz

Uploading helloworld.csv.gz to /faraday-acme/to_faraday/helloworld-2019-01-15.csv.gz

You should be set!

A few notes:

• You won't be able to download or even see files once you send them to us - that's a security measure called a "blind dropbox."
• You need to upload it to the to_faraday/ folder as shown above. 
• See above for requirements on file format, etc. 
• The most secure option is to encrypt your file with GPG (aka PGP) before you send it to us. See https://knowledgebase.faraday.io/help-and-support/optimize/how-to-encrypt-files-with-gpg.