File format

Required

  • Best and most complete possible identifying information on every line (first name + last name or full name, address, city, state, postcode, email, phone)

  • Relevant dates in ISO 8601 YYYY-MM-DD

  • Statuses, product names, etc. (in plain text not numeric codes)

  • CSV format

  • Comma delimited

  • String quoted

  • Headers included

  • UTF-8 encoded

  • One CSV per compressed file

Optional but recommended

Decide what you're going to upload

For example, every day, you upload 2 files:

  • orders_2018-09-29.zip.gpg
    ...which contains a single CSV file, orders.csv
    ...every line of which has name, address 

  • customers_2018-09-29.zip.gpg
    ...which contains a single CSV file, customers.csv
    ...every line of which has name, address, email 

Extra fee applies

  • Multiple CSVs per file

  • CSVs without headers

  • Tab separated or fixed width

  • Joining in identifying information from other files 

Common mistakes

  • Putting 2 CSVs in a single ZIP file
    ...instead, upload 2 ZIP files with 1 CSV each

  • Uploading an orders file with no identifying information, only an order ID
    ...instead, put name, address, etc. on every order line before you upload!

  • Encrypting before zipping
    ...instead, zip before encrypt


1. Send via Amazon S3 (our bucket)

First, we will create a bucket on our end.

To provision access to a bucket, Faraday will need an AWS username like:

  • arn:aws:iam::$YOUR_AWS_ACCOUNT_ID:user/acme_to_faraday

Note: when you upload, you will need to set your ACL to:

  • bucket-owner-full-control

We'll also need for you to put the following in your IAM user's inline policy (example showing if the bucket name was "acme_to_faraday"):

{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:GetBucketLocation"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::acme_to_faraday"
]
},
{
"Effect": "Allow",
"Action": [
"s3:PutObject"
],
"Resource": [
"arn:aws:s3:::acme_to_faraday/to_faraday/*"
]
},
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:GetObjectAcl",
"s3:GetObjectVersion"
],
"Resource": [
"arn:aws:s3:::acme_to_faraday/*"
]
}
]
}

2a. Send via Amazon S3 (your bucket)

Upload your files to a bucket you control.
Then, provision us access to a folder by defining a bucket policy:

(in the example below, it is for a folder to_faraday/ in the bucket acme-faraday-interchange )

{
  "Version": "2012-10-17",
  "Id": "FaradayAccessToBucket",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::113233973114:user/stagecraft-download_s3"
      },
      "Action": [
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::acme-faraday-interchange"
      ]
    },
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::113233973114:user/stagecraft-download_s3"
      },
      "Action": [
        "s3:GetObject",
        "s3:GetObjectAcl",
        "s3:GetObjectVersion"
      ],
      "Resource": "arn:aws:s3:::acme-faraday-interchange/*"
    }
  ]
}

Then, we have to add the bucket name to the IAM user here https://console.aws.amazon.com/iam/home?region=us-east-1#/users/stagecraft-download_s3

2b. S3 upload with SFTP user

We will require your SSH public key. They look like this:

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdtiZZsY6tR0JUWD1OtnkSFOK9wE3kiq5zNBwRKobpCsuSE3/vHZr20MeqhP7HhHvN/mlO81xvY9Hz4AfmslJWJAxw0kDbUj9Gz639FlnalUlNcMkSoG2iSYn/krB/6YG76Eb64CF14X+9aLKgyKrwgOc90AF0aD22Ig3ddsj0HW1YyxB0xH3NQVrdALQME+FILA+Fv20Ihmu8I/en8pdAcFa3vQDpE5aWc36w2/0xKU7luTwXjvJV0bOuDqMIe84KU4oHB0JrhnLTVxUDl+lEsmjZJqj7FBVV8Wg8Wqk0t3Iz1+4k2U68VPpCIY2m9fXkoh/rFJUYT4rRP+lwhJmt 

Needless to say, they are NOT secret - that's why they're called "public" keys.
(Do not EVER send anybody, including Faraday, your SSH private key.)
If you don't have an SSH keypair, you can generate one like this https://www.digitalocean.com/docs/droplets/how-to/add-ssh-keys/create-with-openssh/.

Inputting details in our app:

You can navigate to: https://app.faraday.io/sources and hit "Create source"

Fill out the fields here:

examples:

File glob: to_faraday/$file_name_YYYYMMDD.csv
AWS bucket: acme-faraday-interchange
AWS region: us-east-1

Connecting and sending data:

Using your favorite SFTP client:

  • Host:  s-9733f4cc1a6049ed9.server.transfer.us-east-1.amazonaws.com 

  • Port: 22 

  • Username: varies by client (for example, acme1

  • Password: not enabled, you must utilize a SSH public key as explained above 

  • Folder: to_faraday/  

Let's pretend you work for Acme, Inc. Your SSH private key file is /path/to/acme_rsa  (remember, this is a secret, don't sent it to us) and the username given to you by Faraday is acme1 .

You will upload files like this:

$ sftp -i /path/to/acme_rsa acme1@s-9733f4cc1a6049ed9.server.transfer.us-east-1.amazonaws.com

Connected to acme1@s-9733f4cc1a6049ed9.server.transfer.us-east-1.amazonaws.com

$ sftp> put helloworld.csv.gz to_faraday/helloworld-2019-01-15.csv.gz

Uploading helloworld.csv.gz to /faraday-acme/to_faraday/helloworld-2019-01-15.csv.gz

You should be set!

A few notes:

Did this answer your question?