Hot search tier
Index incoming data with a refresh interval configured for your workload.
SQL-native search
Full-text search for streaming DuckDB data
Make incoming logs, documents, and events searchable with SQL. BoilStream combines Tantivy full-text search with DuckDB analytics and DuckLake storage in one system.
Can DuckDB do full-text search?
DuckDB has an
fts extension for indexes inside a DuckDB database. BoilStream serves a different streaming use case: enable indexing on the fields you need, then search new data from SQL alongside your analytical queries.
Search without a separate indexing pipeline
A conventional analytics-and-search architecture often copies the same events into a columnar store and a separate search cluster. BoilStream can route one incoming batch to both Parquet persistence and a Tantivy index, keeping search configuration with the streaming table.
Enable indexing for the fields that need tokenized search. Other supported scalar columns allow exact matches or filters. See the search guide for supported column types.
Cold search tier
Keep searchable history in object storage, managed alongside your DuckLake tables.
SQL query surface
The multilake_search() table function returns supported user columns plus a relevance _score for matching rows. Combine search scores with filters and SQL analysis.
Enable search on a streaming table
Full-text search is available for tables in __stream DuckLake catalogs. The text-field setting selects the string columns that Tantivy tokenizes for full-text queries.
ALTER TABLE knowledge__stream.main.documents SET (
tantivy_enabled = true,
tantivy_text_fields = 'title,body'
);BoilStream creates a search table named documents__tantivy_idx and indexes subsequent inserts. Choose the fields your users need to search, such as titles, messages, or document text.
Query full-text results from SQL
-- Rank matches and return up to ten results
SELECT id, title, body, _score
FROM multilake_search(
'knowledge__stream',
'documents__tantivy_idx',
'distributed systems'
)
ORDER BY _score DESC
LIMIT 10;
-- Restrict the query to one field
SELECT id, title, _score
FROM multilake_search(
'knowledge__stream',
'documents__tantivy_idx',
'title:Rust'
);| Input column | Tantivy mapping | Search behavior |
|---|---|---|
| Configured UTF-8 text field | TEXT | Tokenized full-text search |
| Other UTF-8 field | STRING | Exact-match indexing |
| Integer or float | Numeric field | Indexed numeric queries and filters |
| Timestamp | Date field | Indexed date-range queries |
| Boolean | Boolean field | Indexed boolean filter |
Where integrated streaming search helps
Logs and events
Search error messages, traces, and event payload text while keeping analytical columns in the same ingestion flow.
Knowledge and support
Index documents, articles, or support tickets as they arrive and retrieve relevance-ranked context for people or AI agents.
Product catalogs
Search titles and descriptions while retaining numeric, timestamp, and boolean fields for structured filtering.
When should I still use a dedicated search service?
Keep a dedicated service when you need its mature ecosystem, specialized analyzers, complex relevance tooling, or independent search scaling. BoilStream is strongest when the same continuously ingested data needs SQL analytics, DuckLake retention, and integrated full-text retrieval.
DuckDB and Tantivy search FAQ
Does BoilStream require Elasticsearch?
No. Full-text indexing, search storage, and SQL query access are integrated into BoilStream.
How quickly do new rows become searchable?
Search refresh is configurable. Use the search guide to choose settings for your workload.
Can I use Tantivy without writing Parquet?
Yes. Tantivy-only mode disables Parquet persistence while keeping search indexes in local and object storage. It is intended for search-first workloads that do not need columnar analytics.
Where are cold indexes stored?
Search indexes are stored in your configured object storage and managed through DuckLake.