DuckDB has earned widespread acclaim across the data engineering ecosystem for its vectorized execution engine and zero-dependency embedded design. For developers seeking rapid SQL analytics on Parquet files without spinning up heavy clusters, it has frequently been hailed as the SQLite of analytical processing. However, when analytics platform PostHog expanded its data warehouse to two petabytes—encompassing approximately one trillion rows and 15.3 million table snapshots—the architectural assumptions of embedded, single-node query execution clashed with the demands of high-concurrency cloud workloads.
In an extensive post-mortem authored by Benjamin Knofe-Vider, PostHog detailed the structural limitations that forced them to sunset their in-house DuckLake architecture. While DuckDB performed admirably on isolated local tests, operating it at scale inside Kubernetes cloud containers exposed four severe operational bottlenecks that degraded customer query reliability and inflated infrastructure operational overhead.
The first major challenge was memory spilling in containerized environments. While DuckDB is capable of spilling intermediate join buffers to disk when queries exceed physical RAM, executing out-of-core operations over virtualized cloud storage proved prohibitively slow. Unpredictable query patterns frequently triggered Linux kernel Out-Of-Memory (OOM) killer terminations, crashing active worker pods and disrupting concurrent user sessions.
Second, DuckLake's centralized metadata catalog became a severe write serialization bottleneck. Under continuous telemetry ingestion, the catalog implementation scanned partition-level statistics across entire tables on every transaction commit. What took milliseconds on small test datasets began requiring upwards of three minutes per commit as snapshot counts climbed past 15 million, creating massive write backpressure.
Compounding the catalog delays was unchecked metadata bloat. Because PostHog frequently generates temporary tables and intermediate materializations for user queries, the catalog accumulated hundreds of thousands of orphaned metadata rows. An internal audit revealed that over 99% of all entries in the catalog statistics table belonged to dead or dropped tables that had never been purged, degrading all subsequent catalog lookups.
To replace DuckLake, PostHog is transitioning its core warehousing infrastructure to Trino (formerly PrestoSQL) coupled with HogLake, an open Delta and Iceberg-compatible storage format designed for high-throughput append workloads on Amazon S3. The new architecture distributes heavy partition scans across horizontal worker pools, leverages HTTP Application Load Balancers for connection pooling, and decouples write commits from global statistics recalculations.