Check the format and required information
First, please see http://knowledgebase.faraday.io/help-and-support/data-sources/what-fields-are-required-when-sending-data-to-faraday
Decide what you're going to upload
For example, every day, you upload 2 files:
- orders_2018-09-29.zip.gpg
...which contains a single CSV file, orders.csv
...every line of which has name, address - customers_2018-09-29.zip.gpg
...which contains a single CSV file, customers.csv
...every line of which has name, address, email
Extra fee applies
- Multiple CSVs per file
- CSVs without headers
- Tab separated or fixed width
- Joining in identifying information from other files
Common mistakes
- Putting 2 CSVs in a single ZIP file
...instead, upload 2 ZIP files with 1 CSV each! - Uploading an orders file with no identifying information, only an order ID
...instead, put name, address, etc. on every order line before you upload! - Encrypting before zipping
...instead, zip before encrypt!
1. Send via Amazon S3 (our bucket)
First, we will create a bucket on our end.
To provision access to a bucket, Faraday will need an AWS username like:
arn:aws:iam::$YOUR_AWS_ACCOUNT_ID:user/acme_to_faraday
Note: when you upload, you will need to set your ACL to:
bucket-owner-full-control
We'll also need for you to put the following in your IAM user's inline policy
(example showing if the bucket name was "acme_to_faraday"):
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:GetBucketLocation"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::acme_to_faraday"
]
},
{
"Effect": "Allow",
"Action": [
"s3:PutObject"
],
"Resource": [
"arn:aws:s3:::acme_to_faraday/to_faraday/*"
]
},
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:GetObjectAcl",
"s3:GetObjectVersion"
],
"Resource": [
"arn:aws:s3:::acme_to_faraday/*"
]
}
]
}
2a. Send via Amazon S3 (your bucket)
Upload your files to a bucket you control.
Then, provision us access to a folder by defining a bucket policy:
(in the example below, it is for a folder to_faraday/
in the bucket acme-faraday-interchange
)
{
"Version": "2012-10-17",
"Id": "FaradayAccessToBucket",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::113233973114:user/stagecraft-download_s3"
},
"Action": [
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::acme-faraday-interchange"
]
},
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::113233973114:user/stagecraft-download_s3"
},
"Action": [
"s3:GetObject",
"s3:GetObjectAcl",
"s3:GetObjectVersion"
],
"Resource": "arn:aws:s3:::acme-faraday-interchange/*"
}
]
}
Then, we have to add the bucket name to the IAM user here https://console.aws.amazon.com/iam/home?region=us-east-1#/users/stagecraft-download_s3
2b. Upload with SFTP
We will require your SSH public key. They look like this:
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdtiZZsY6tR0JUWD1OtnkSFOK9wE3kiq5zNBwRKobpCsuSE3/vHZr20MeqhP7HhHvN/mlO81xvY9Hz4AfmslJWJAxw0kDbUj9Gz639FlnalUlNcMkSoG2iSYn/krB/6YG76Eb64CF14X+9aLKgyKrwgOc90AF0aD22Ig3ddsj0HW1YyxB0xH3NQVrdALQME+FILA+Fv20Ihmu8I/en8pdAcFa3vQDpE5aWc36w2/0xKU7luTwXjvJV0bOuDqMIe84KU4oHB0JrhnLTVxUDl+lEsmjZJqj7FBVV8Wg8Wqk0t3Iz1+4k2U68VPpCIY2m9fXkoh/rFJUYT4rRP+lwhJmt
Needless to say, they are NOT secret - that's why they're called "public" keys.
(Do not EVER send anybody, including Faraday, your SSH private key.)
If you don't have an SSH keypair, you can generate one like this https://www.digitalocean.com/docs/droplets/how-to/add-ssh-keys/create-with-openssh/.
Inputting details in our app:
You can navigate to: https://app.faraday.io/sources and hit "Create source"
Fill out the fields here:
examples:
File glob: to_faraday/$file_name_YYYYMMDD.csv
AWS bucket: acme-faraday-interchange
AWS region: us-east-1
Connecting and sending data:
Using your favorite SFTP client:
- Host:
s-9733f4cc1a6049ed9.server.transfer.us-east-1.amazonaws.com
- Port: 22
- Username: varies by client (for example,
acme1
) - Password: not enabled, you must utilize a SSH public key as explained above
- Folder:
to_faraday/
Let's pretend you work for Acme, Inc. Your SSH private key file is /path/to/acme_rsa
(remember, this is a secret, don't sent it to us) and the username given to you by Faraday is acme1
.
You will upload files like this:
$ sftp -i /path/to/acme_rsa acme1@s-9733f4cc1a6049ed9.server.transfer.us-east-1.amazonaws.com
Connected to acme1@s-9733f4cc1a6049ed9.server.transfer.us-east-1.amazonaws.com
$ sftp> put helloworld.csv.gz to_faraday/helloworld-2019-01-15.csv.gz
Uploading helloworld.csv.gz to /faraday-acme/to_faraday/helloworld-2019-01-15.csv.gz
You should be set!
A few notes:
- You won't be able to download or even see files once you send them to us - that's a security measure called a "blind dropbox."
- You need to upload it to the
to_faraday/
folder as shown above. - See above for requirements on file format, etc.
- The most secure option is to encrypt your file with GPG (aka PGP) before you send it to us. See https://knowledgebase.faraday.io/help-and-support/optimize/how-to-encrypt-files-with-gpg.