File
Buckets and local paths via fsspec: `csv`, `jsonl`, `json`, `parquet`, `xlsx`.
How it works
Files are listed, then stream-converted into Arrow batches. State is a ledger of (name, etag|mtime),
so a file that has not changed is not re-read.
Credential proof
List the prefix.
Questions
- Which File sources has Ingest certified?
- None yet. Ingest authors a File connector from the vendor's own API documentation when a customer asks for it.
- Can Ingest sync any File API?
- Ingest authors the connector from the vendor's published API documentation, runs it against the real API, and lists it as deployable only once it passes. A connector is a declarative spec, never per-source code, which is why a new one is authored rather than programmed.
- Where can File data land?
- Amazon S3, Google Cloud Storage, Azure Blob Storage, Apache Iceberg, Ingest Managed Lakehouse, MotherDuck, Postgres, Amazon Athena, Databricks, Redshift, BigQuery, ClickHouse, MySQL / generic SQL and Snowflake. The data lands in a destination you own, under credentials you control. Ingest keeps no copy of it.
File sources, and where they land
The other source kinds Ingest reads: GraphQL, gRPC, REST, SOAP, SQL, WebSocket.
Where the tables land: Amazon S3, Google Cloud Storage, Azure Blob Storage, Apache Iceberg, Ingest Managed Lakehouse, MotherDuck, Postgres, Amazon Athena, Databricks, Redshift, BigQuery, ClickHouse, MySQL / generic SQL, Snowflake.