Integrating Faraday with Amazon S3

Faraday can integrate with Amazon S3 as both a Source and Destination for CSVs, meaning that it can be used to pull customer data from, and also be where your predictions are pushed to. This article covers both Source and Destination creation, as well as using Monitor to push predictions.

Table of Contents:

Preparing your CSVs for Faraday integration
Creating CSV on Amazon S3 as a Source in Faraday
Creating CSV on Amazon S3 as a Destination in Faraday
Sending data to your Amazon S3 bucket with Monitor

Preparing your CSV for Faraday integration

First and foremost, in order to ensure smooth, hands-off CSV syncing between Faraday and Amazon S3, your CSVs must have the same columns every time. Take a look at What data Faraday expects for an idea of what should be included in your CSVs.

How to upload different datasets via CSV:

If you're interested in providing several datasets to build predictions upon, you can do so by having different folders inside your S3 bucket:

s3://mybucket/customers/
s3://mybucket/transactions/
s3://mybucket/coupon_codes/
s3://mybucket/products/

As noted above, inside of every folder, every file must have the same columns.

Note: if you plan to provide multiple datasets, you will need to create a Source for each bucket by following the instructions in Creating Amazon S3 as a Source in Faraday.

How to format CSVs:

The first row should be headers
Dates in YYYY-MM-DD (aka ISO 8601)
Comma-delimited (not tab)
Cells with commas should be surrounded by double quotes
Double quotes inside of cells should be doubled up ("")
UTF-8 encoded
One email/address/phone per cell

Creating CSV on Amazon S3 as a Source in Faraday

Go to the Faraday app’s Sources console, click Create source at the top.

Search for CSV on Amazon S3 as a Source location, enter a descriptive name, and click Next.
Enter your Amazon S3 details, and click Next.
Under Finalize, click Create source. Faraday can now begin the integration process. Once you finalize this source integration, our Customer Success team will be able to finish the configuration and we can begin utilizing your CSV on Amazon S3 data!

Creating CSV on Amazon S3 as a Destination in Faraday

Go to the Faraday app’s Destinations console, click Create destination at the top.

Search for CSV on Amazon S3 as a Destination location, enter a descriptive name, and click Next.
Enter your Amazon S3 details here, and click Next.
examples (yours may differ):
- Whether to gzip: Yes/No
- Client-provided AWS bucket: acme-faraday-interchange
- Client-provided AWS region: us-east-1
- Filename prefix with optional strfdate string (optional): Faraday_insights_%Y%m%d
- Name of file to create (optional): (we will work with your suggestion)
- Client-provided directory: from_faraday/ - (alternatively this can be included in the Filename prefix)
- Whether to quote all fields with double quotes: Yes/No
- Omit delivery ID (and fdysec) from filename: `filename` param MUST be specified in this case: Yes/No (but if "Yes" then you'll need to fill out the "Filename prefix" field)
  Note: for the last option, we normally send out deliveries in this format, which can be overridden:
```
fdysec_delivery_e538f7s4.csv
```
  The thought is that perhaps you wanted something more descriptive - and you can!
Under Finalize, click Create destination. Faraday can now begin the integration process. Once your Account Manager confirms everything is all set up, CSV on Amazon S3 will be available as a destination across the app!

Sending data to your Amazon S3 bucket with Monitor

Faraday has a product called Monitor that will automate delivery on a schedule and can push to your Amazon S3 bucket. If you're interested in having this process automated, talk to your Account Manager.

Amazon S3 - our bucket

If you want Faraday to deliver to an S3 bucket that we control, talk to your Account Manager and we'll get it set up for you. We'll let you know what the bucket name is and you can input an inline policy for your IAM user in the following manner (example showing if our bucket name was "acme_from_faraday"):

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetBucketLocation"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::acme_from_faraday"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:GetObjectAcl",
                "s3:GetObjectVersion"
            ],
            "Resource": [
                "arn:aws:s3:::acme_from_faraday/*"
            ]
        }
    ]
}

Amazon S3 - your bucket

Let's assume you are Acme, Inc. and you own an Amazon S3 bucket called:

s3://acme-faraday-interchange

Use the bucket policies below (there are two, you have to choose one) to grant access to Faraday's account id 113233973114 and service account deliver_s3 .
Once that is complete, we just need to know what region your bucket is in.

Note: you will not need to provide an AWS access key - you already gave us access via the service account.

Bucket policy for Faraday-only buckets

If this bucket is only going to be used by Faraday, use a bucket policy like this:

{
  "Version": "2012-10-17",
  "Id": "FaradayAccessToBucket",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::113233973114:user/deliver_s3"
      },
      "Action": [
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::acme-faraday-interchange"
      ]
    },
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::113233973114:user/deliver_s3"
      },
      "Action": [
        "s3:PutObject",
        "s3:PutObjectAcl"
      ],
      "Resource": "arn:aws:s3:::acme-faraday-interchange/*",
      "Condition": {
        "StringEquals": {
          "s3:x-amz-acl": "bucket-owner-full-control"
        }
      }
    },
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::113233973114:user/deliver_s3"
      },
      "Action": [
        "s3:GetObject",
        "s3:GetObjectAcl",
        "s3:GetObjectVersion"
      ],
      "Resource": "arn:aws:s3:::acme-faraday-interchange/*"
    }
  ]
}

Bucket policy for shared buckets

If you are giving us access to a bucket that is shared by other users/vendors/etc (for example, if you are using Snowflake), use a policy like this. As you can see, we restrict s3:ListBucket and other actions to our folder only:

{
  "Version": "2012-10-17",
  "Id": "FaradayAccessToBucket",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::113233973114:user/deliver_s3"
      },
      "Action": [
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::acme-faraday-interchange"
      ],
      "Condition": {
        "StringLike": {
          "s3:prefix": [
            "faraday/*"
          ]
        }
      }
    },
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::113233973114:user/deliver_s3"
      },
      "Action": [
        "s3:PutObject",
        "s3:PutObjectAcl"
      ],
      "Resource": "arn:aws:s3:::acme-faraday-interchange/faraday/*",
      "Condition": {
        "StringEquals": {
          "s3:x-amz-acl": "bucket-owner-full-control"
        }
      }
    },
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::113233973114:user/deliver_s3"
      },
      "Action": [
        "s3:GetObject",
        "s3:GetObjectAcl",
        "s3:GetObjectVersion"
      ],
      "Resource": "arn:aws:s3:::acme-faraday-interchange/faraday/*"
    }
  ]
}