gRPC
A channel per resource, TLS by default, with auth carried in metadata.
Descriptors: check them in
Descriptors come either from server reflection at run time or from a descriptor set checked into the connector directory. Prefer the checked-in descriptor set. Reflection drifts under you between runs; a file in the repo does not. When reflection is used, it is recorded.
Messages
Unary returns one response; server-streaming yields one record per message. page_token pagination
follows AIP-158, re-issuing with next_page_token until it comes back empty.
Protobuf becomes a dict with preserving_proto_field_name. int64 maps to bigint, Timestamp and
Duration to timestamp and double, enums to their names.
Status codes
Retryable: UNAVAILABLE, RESOURCE_EXHAUSTED, DEADLINE_EXCEEDED, ABORTED.
Fatal: UNAUTHENTICATED, PERMISSION_DENIED, INVALID_ARGUMENT, NOT_FOUND, UNIMPLEMENTED.
Credential proof
A unary health check, or the first method on the service.
Questions
- Which gRPC sources has Ingest certified?
- None yet. Ingest authors a gRPC connector from the vendor's own API documentation when a customer asks for it.
- Can Ingest sync any gRPC API?
- Ingest authors the connector from the vendor's published API documentation, runs it against the real API, and lists it as deployable only once it passes. A connector is a declarative spec, never per-source code, which is why a new one is authored rather than programmed.
- Where can gRPC data land?
- Amazon S3, Google Cloud Storage, Azure Blob Storage, Apache Iceberg, Ingest Managed Lakehouse, MotherDuck, Postgres, Amazon Athena, Databricks, Redshift, BigQuery, ClickHouse, MySQL / generic SQL and Snowflake. The data lands in a destination you own, under credentials you control. Ingest keeps no copy of it.
gRPC sources, and where they land
The other source kinds Ingest reads: File, GraphQL, REST, SOAP, SQL, WebSocket.
Where the tables land: Amazon S3, Google Cloud Storage, Azure Blob Storage, Apache Iceberg, Ingest Managed Lakehouse, MotherDuck, Postgres, Amazon Athena, Databricks, Redshift, BigQuery, ClickHouse, MySQL / generic SQL, Snowflake.