docs / destinations / filesystem_s3

Amazon S3

deployable todayappend and replace only

Parquet files in your S3 bucket, written under a role you own and can revoke.

Ingest lands typed, deduplicated tables in Amazon S3 from any API it connects, including Abilene TX 311, Albany NY 311, Ann Arbor MI 311 and Anne Arundel County MD 311. You own the Amazon S3 account and the credentials; Ingest keeps no copy of the data.

Every destination receives the same layout; see the tables Ingest creates.

Setup

Before you start

Ingest saves your data as files in one folder of your own S3 bucket. To allow that, you create a role in your AWS account that can use that folder and nothing else. Deleting the role later cuts off Ingest's access. No AWS access keys are created or shared. Only your workspace's Ingest role can use this role: the trust policy in step 2 names it and checks your workspace's ID.

You need to sign in to the AWS console as someone who can create IAM roles, such as an administrator. If that is not you, send this page to whoever manages your AWS account.

Both policies in this guide are filled in for you when you open it from your Ingest dashboard. On the public docs page they show placeholders instead. In your dashboard, open Destinations, choose Add a destination, then Amazon S3.

Using Cloudflare R2 or MinIO? Contact Ingest. Those connect with access keys instead of a role.

1. Choose a bucket and a folder

  • In the S3 console, choose General purpose buckets and note the name of the bucket to use. No bucket yet? Choose Create bucket, enter a Bucket name, keep every other setting as it is, and choose Create bucket.
  • In Bucket and folder at the bottom, enter s3://, then the bucket name, then a folder name for Ingest's files, for example s3://acme-data/ingest. Any folder name works; ingest is only a suggestion. The folder appears when Ingest first writes to it. The role can reach only this folder, so the rest of the bucket stays out of Ingest's reach.
  • Leave Format empty. Your files will be Parquet, which most analytics tools can read. Other values: jsonl.gz and csv.gz. Plain files do not support upserts.

2. Create the role

  • In the IAM console, choose Roles, then Create role.
  • Choose Custom trust policy. In the Custom trust policy box, replace everything with this, then choose Next. A trust policy says who may use the role: here, only your workspace's Ingest role, and only with your workspace's ID as the external ID.
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": { "AWS": "<ingest-tenant-role-arn>" },
    "Action": "sts:AssumeRole",
    "Condition": { "StringEquals": { "sts:ExternalId": "<your-workspace-slug>" } }
  }]
}
  • On the Add permissions page, choose Next without selecting anything. Step 3 gives the role its access.
  • For Role name, enter IngestAccess-lake, then choose Create role. The name must start with IngestAccess-, because Ingest may only use roles named that way.

3. Let the role use your folder

  • In the list of roles, choose IngestAccess-lake. On its Permissions tab, choose Add permissions, then Create inline policy.
  • Choose JSON, replace everything in the policy editor with this, then choose Next. It lets the role read, write, list and delete files inside your folder only. Finding the bucket's region needs one more statement, which cannot be limited to a folder. On the public docs page, replace your-bucket and your-folder yourself.
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:AbortMultipartUpload"],
      "Resource": "arn:aws:s3:::your-bucket/your-folder/*"
    },
    {
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::your-bucket",
      "Condition": { "StringLike": { "s3:prefix": ["your-folder", "your-folder/*"] } }
    },
    {
      "Effect": "Allow",
      "Action": "s3:GetBucketLocation",
      "Resource": "arn:aws:s3:::your-bucket"
    }
  ]
}
  • For Policy name, enter ingest-folder-access, then choose Create policy.
  • Does your bucket encrypt with a KMS key you created? Give this role access to that key too. Allow kms:GenerateDataKey and kms:Decrypt on the customer managed key; its key policy must also allow the role. The bucket's Properties tab shows its Default encryption. SSE-S3, the default for every bucket, needs nothing extra.

4. Connect it to Ingest

  • On the role's page, copy the ARN from its Summary. It starts with arn:aws:iam::. Paste it into IAM role Ingest assumes (IngestAccess-…) at the bottom.
  • Click Test & configure at the bottom. Ingest writes a small test table into your folder, reads it back, then deletes it. It also saves and reads back pipeline state. If the test data cannot be deleted, the result says what was left. If the test fails, fix the reported problem and choose Test again.

Staging

The worker writes files to your chosen bucket and folder. No separate staging bucket is required.

Writes

Plain files cannot merge. For upserts into your S3 lake, use Amazon Athena with Glue, or confirm a supported Iceberg REST catalog setup.

Conformance

Every adapter passes the same suite before it ships: create a table with every canonical type, append 100k rows, atomic replace, merge (latest dedup_sort wins; child tables delete-then-insert by root id), schema evolution, NULL-key rejection, package replay without duplicates, state and schema round-trip, row-count read-back, identifier edge cases, double-underscored table names (__quarantine, child tables), and rollback on a mid-load failure.

Questions

Can Ingest load data into Amazon S3?
Yes. Amazon S3 is live on Ingest's hosted service: the adapter passes all 15 conformance cases against the real service, and the setup steps are on this page.
Does Ingest merge into Amazon S3, or only append?
Amazon S3 appends and replaces; it does not merge. Plain files have no update in place, so a resource that needs merge is filtered out with a reason before a pipeline is created, rather than failing on its first run.
What does Ingest need to connect to Amazon S3?
Bucket and folder. Secrets are entered in a form, never in a chat, and held in a secret store under your own path: only the key names ever reach Ingest's database.
Which sources can Ingest load into Amazon S3?
Every connector Ingest has certified (Abilene TX 311, Albany NY 311, Ann Arbor MI 311, Anne Arundel County MD 311, Anthropic, Asana, Auburn WA 311, Aurora IL 311, Austin TX 311, Baltimore MD 311, Baton Rouge LA 311, Beaumont TX 311, Bellevue WA 311, Bellingham WA 311, Berkeley CA 311, Bethlehem PA 311, Bloomington IN 311, Boca Raton FL 311, Boston MA 311, Bridgeport CT 311, Brockton MA 311, Brookline MA 311, Buffalo NY 311, Burlington VT 311, Cambridge MA 311, Cambridge MA Open Data, Charleston County SC 311, Charlotte NC 311, Chula Vista CA 311, Cincinnati OH 311, Chicago IL 311, New York City NY 311, Clayton County GA 311, Cleveland OH 311, Cobb County GA 311, Columbus OH 311, Corona CA 311, Dallas TX 311, Davenport IA 311, Denver CO 311, Detroit MI 311, Elgin IL 311, Encinitas CA 311, Everett WA 311, Federal Way WA 311, Fort Lauderdale FL 311, Gilbert AZ 311, Google Ad Manager, Google Analytics 4, Greenwich CT 311, Hartford CT 311, Honolulu HI 311, Houston TX 311, Howard County MD 311, Huntington WV 311, Huntsville AL 311, Indianapolis IN 311, Jersey City NJ 311, Jira, Kansas City MO 311, Kent WA 311, Knox County TN 311, Los Angeles CA 311, Lexington KY 311, Lincoln NE 311, LinkedIn Ads, Livonia MI 311, Louisville KY 311, Lubbock TX 311, Macon-Bibb County GA 311, Manchester NH 311, Maui County HI 311, Medford MA 311, Memphis TN 311, Miami FL 311, Miami-Dade County FL 311, Minneapolis MN 311, Montgomery County MD 311, Multnomah County OR 311, Naperville IL 311, Nashua NH 311, Nashville TN 311, Newark NJ 311, NOAA, New Orleans LA 311, Norfolk VA 311, Oakland CA 311, Omaha NE 311, Open Brewery DB, Paterson NJ 311, Peoria IL 311, Philadelphia PA 311, Pinellas County FL 311, Pittsburgh PA 311, PokéAPI, Portland ME 311, PostgreSQL, Prince George's County MD 311, Provo UT 311, Raleigh NC 311, Ramp, Renton WA 311, The Rick and Morty API, Sacramento CA 311, Salesforce, San Antonio TX 311, San Francisco CA 311, San Gabriel CA 311, San Jose CA 311, San Mateo County CA 311, San Rafael CA 311, Santa Fe NM 311, Scottsdale AZ 311, Seattle WA 311, Shopify, Somerville MA 311, Springfield MO 311, St. Petersburg FL 311, Tacoma WA 311, Toledo OH 311, Topeka KS 311, Vallejo CA 311, Washington DC 311, Wayne County MI 311, West Des Moines IA 311, Wichita KS 311, Worcester MA 311 and Youngstown OH 311), plus any API it authors on request. The data lands in a destination you own, under credentials you control. Ingest keeps no copy of it.

Sources certified against a real API today: Abilene TX 311, Albany NY 311, Ann Arbor MI 311, Anne Arundel County MD 311, Anthropic, Asana, Auburn WA 311, Aurora IL 311, Austin TX 311, Baltimore MD 311, Baton Rouge LA 311, Beaumont TX 311, Bellevue WA 311, Bellingham WA 311, Berkeley CA 311, Bethlehem PA 311, Bloomington IN 311, Boca Raton FL 311, Boston MA 311, Bridgeport CT 311, Brockton MA 311, Brookline MA 311, Buffalo NY 311, Burlington VT 311, Cambridge MA 311, Cambridge MA Open Data, Charleston County SC 311, Charlotte NC 311, Chula Vista CA 311, Cincinnati OH 311, Chicago IL 311, New York City NY 311, Clayton County GA 311, Cleveland OH 311, Cobb County GA 311, Columbus OH 311, Corona CA 311, Dallas TX 311, Davenport IA 311, Denver CO 311, Detroit MI 311, Elgin IL 311, Encinitas CA 311, Everett WA 311, Federal Way WA 311, Fort Lauderdale FL 311, Gilbert AZ 311, Google Ad Manager, Google Analytics 4, Greenwich CT 311, Hartford CT 311, Honolulu HI 311, Houston TX 311, Howard County MD 311, Huntington WV 311, Huntsville AL 311, Indianapolis IN 311, Jersey City NJ 311, Jira, Kansas City MO 311, Kent WA 311, Knox County TN 311, Los Angeles CA 311, Lexington KY 311, Lincoln NE 311, LinkedIn Ads, Livonia MI 311, Louisville KY 311, Lubbock TX 311, Macon-Bibb County GA 311, Manchester NH 311, Maui County HI 311, Medford MA 311, Memphis TN 311, Miami FL 311, Miami-Dade County FL 311, Minneapolis MN 311, Montgomery County MD 311, Multnomah County OR 311, Naperville IL 311, Nashua NH 311, Nashville TN 311, Newark NJ 311, NOAA, New Orleans LA 311, Norfolk VA 311, Oakland CA 311, Omaha NE 311, Open Brewery DB, Paterson NJ 311, Peoria IL 311, Philadelphia PA 311, Pinellas County FL 311, Pittsburgh PA 311, PokéAPI, Portland ME 311, PostgreSQL, Prince George's County MD 311, Provo UT 311, Raleigh NC 311, Ramp, Renton WA 311, The Rick and Morty API, Sacramento CA 311, Salesforce, San Antonio TX 311, San Francisco CA 311, San Gabriel CA 311, San Jose CA 311, San Mateo County CA 311, San Rafael CA 311, Santa Fe NM 311, Scottsdale AZ 311, Seattle WA 311, Shopify, Somerville MA 311, Springfield MO 311, St. Petersburg FL 311, Tacoma WA 311, Toledo OH 311, Topeka KS 311, Vallejo CA 311, Washington DC 311, Wayne County MI 311, West Des Moines IA 311, Wichita KS 311, Worcester MA 311, Youngstown OH 311.

Other destinations Ingest writes to: Google Cloud Storage, Azure Blob Storage, Apache Iceberg, Ingest Managed Lakehouse, MotherDuck, Postgres, Amazon Athena, Databricks, Redshift, BigQuery, ClickHouse, MySQL / generic SQL, Snowflake.

Get your API data into Amazon S3