Ingest
Use standard Kafka producers, Arrow FlightRPC, HTTP/2 Arrow payloads, or PostgreSQL streaming INSERT. Confluent Schema Registry-compatible Avro is supported on the Kafka path.
Streaming ingestion lakehouse
Stream data into DuckLake without a batch staging layer
BoilStream accepts Kafka-protocol, Arrow FlightRPC, HTTP/2 Arrow, and PostgreSQL streaming INSERT traffic, makes rows queryable in a hot DuckDB tier, and writes durable Parquet snapshots into DuckLake-managed object storage.
What is a streaming DuckLake?
It is a DuckLake catalog whose tables receive continuous events. BoilStream keeps recent rows in a hot tier for low-latency SQL while concurrently finalizing older data as Parquet in object storage. Query with PostgreSQL-compatible tools and connect native DuckDB clients through the BoilStream extension.
One data path from producer to lakehouse
Traditional streaming-to-lakehouse stacks often assemble a broker, a stream processor, object-storage writers, compaction jobs, and a separate query engine. BoilStream combines the ingestion, streaming SQL, hot visibility, and DuckLake write path in one deployable system.
The storage stays open: bulk data is Apache Parquet and the catalog follows the DuckLake specification. Remote DuckDB clients can connect through the BoilStream community extension and attach the selected catalog by UUID and execute SQL remotely.
Query recent rows
Query recent events through PostgreSQL-compatible clients while BoilStream prepares analytical history in Parquet.
Retain durable history
Completed data is written as Parquet to S3, Azure Blob, GCS, MinIO, or filesystem storage and registered with the DuckLake catalog.
How the streaming and lakehouse layers fit together
| Stage | BoilStream component | Result |
|---|---|---|
| Producer connection | Kafka wire protocol, FlightRPC, HTTP/2 Arrow, or PGWire | Existing clients can send typed event batches. |
| Recent-data visibility | Per-topic hot DuckDB tier | PGWire can combine locally committed rows with cold files. |
| Continuous processing | Streaming views and windowed materialized views | Filters, transforms, and aggregates create derived topics. |
| Durability | Parquet files plus DuckLake catalog metadata | Open-format history lives in customer-controlled object storage. |
| Remote access | PostgreSQL-compatible clients and DuckDB with the BoilStream extension | Bring familiar SQL tools to analytics and applications. |
Create and query a streaming DuckLake
Streaming catalogs use the __stream suffix. Create the catalog, list its UUID, and attach it under a local alias. Then create and query tables with SQL.
INSTALL boilstream FROM community;
LOAD boilstream;
PRAGMA boilstream_login(
'https://your-server.example/user@example.com',
'password',
'123456'
);
PRAGMA boilstream_create_ducklake(
'events__stream',
'Real-time event catalog'
);
FROM boilstream_ducklakes();
-- Replace the placeholder with the catalog_id returned above.
PRAGMA boilstream_attach_ducklake('<catalog-uuid>', 'events');
USE events;
CREATE TABLE web_events (event_id BIGINT, event_time TIMESTAMP);
SELECT * FROM web_events
ORDER BY event_time DESC
LIMIT 100;The table is initially empty. For inserts, expected results, and connection instructions, use the BoilStream Quick Start.
When this architecture is a good fit
- You want DuckDB-compatible analytics over continuously arriving events.
- You need recent data quickly but want durable history in open Parquet files.
- You already have Kafka producers and want to simplify the downstream processing path.
- You want SQL transformations, windowed aggregates, and full-text search in the same system.
When to keep a general-purpose event broker
BoilStream is optimized for the ingestion-to-analytics path. Keep a general-purpose broker when your architecture depends on a broad connector ecosystem, long broker-side retention as the system of record, or application messaging semantics outside BoilStream's supported protocols and processing model.
Streaming DuckLake FAQ
Does DuckLake support streaming data?
DuckLake supports data inlining for small changes. BoilStream adds the continuous ingestion, hot query tier, Parquet finalization, and catalog coordination needed to operate a streaming data path around DuckLake.
Where does BoilStream store the data?
Recent data is held in the hot DuckDB tier. Durable cold data is stored as Parquet in a configured S3-compatible, Azure, GCS, MinIO, or filesystem backend and registered in DuckLake.
Can existing Kafka clients produce to BoilStream?
Yes. BoilStream implements the Kafka wire protocol for producers and supports Confluent Schema Registry-compatible Avro on that ingestion path.
How do clients query the data?
Clients can use PostgreSQL-compatible SQL tools or native DuckDB with the BoilStream extension. See the Quick Start for connection instructions.