Personalization Data Lake Architecture: Bronze, Silver, Gold for Recommendations

TL;DR.

• A personalization data lake architecture is medallion (bronze/silver/gold) plus a real-time tier — the lake alone cannot serve p50 < 50 ms ranking.
• Bronze stores raw events partitioned by event_date/event_hour; silver joins identity and dedupes; gold materializes per-user feature tables partitioned by user_id_hash.
• Apache Hudi wins for personalization workloads needing upserts and CDC; Iceberg wins when most reads are analytical; Delta wins inside Databricks.
• The real cost driver is bytes-scanned, not storage — converting CSV to Parquet on S3 can cut Athena bills by 5-10x on the same data.
• Real-time ranking reads from an online store (Redis, DynamoDB, ScyllaDB); the lake is the system of record that backfills it.

A generic data warehouse will not power a recommendation system. The query patterns are wrong, the latencies are wrong, and the cost model collapses the moment you try to scan a billion-row event table for every ranking call. This post is about what a personalization data lake architecture actually looks like when the workload is ranking — not dashboards.

We will walk the bronze/silver/gold layers with personalization-specific decisions (partitioning, table format, online/offline split), the table-format pick between Iceberg, Delta, and Hudi, and the cost reality at scale on S3 with Athena and Trino versus Snowflake. By the end you should know how to lay out the lake so it feeds both batch training jobs and real-time ranking without buckling.

What a personalization data lake architecture actually is

The medallion architecture — bronze (raw), silver (cleaned), gold (feature-ready) — comes from the lakehouse world and is documented in detail by Databricks and Microsoft Fabric. The basic idea is three logical layers in object storage, each strictly more refined than the one before it.

That basic shape applies to personalization, but with three specific twists.

First, the gold layer is feature tables, not BI cubes. A gold table for personalization is shaped like user_id, feature_vector, computed_at, partitioned by a hash of user_id. It is read row-by-row at low latency, not scanned in aggregate. That is a different physical layout than the star schemas the medallion pattern was originally designed for.

Second, there is always a real-time path that bypasses the lake. No matter how clever your lake layout is, you are not serving p50 < 50 ms ranking off Parquet files in S3. The lake feeds an online feature store (Redis, DynamoDB, ScyllaDB, ×marble's own KV layer), and the ranker reads from there. The lake is the system of record. The online store is the cache that gets ranked against.

Third, the gold layer's contract is "feature available within an SLA," not "transformed correctly." A ranking model that wakes up to a stale feature table will silently regress. Freshness becomes a first-class quality dimension alongside correctness, and you instrument both.

If you have not read our reference architecture for real-time personalization, start there for the end-to-end picture; this post zooms in on the storage layer.

Bronze layer — raw events, no transformation

The bronze layer is where every event lands exactly as it arrived. Page views, clicks, plays, dwell-time pings, purchase events, server-side conversion webhooks — all of it, no transformation. The Databricks medallion guide puts it bluntly: source system table structures as-is.

A few non-negotiables for personalization bronze:

• Append-only. No updates, no deletes, no upserts. If correction is needed, write a new event with a corrected_at flag in silver. Bronze is the audit log.
• Partition by event time, not ingestion time. event_date=2026-05-12/event_hour=14/ keeps queries that ask "what happened between 14:00 and 15:00" from scanning the whole day. Most engineers default to ingestion time and regret it within six months.
• Store as Parquet with Snappy compression. Converting CSV or JSON to Parquet on the read side is 5-10x more expensive on Athena, because Athena charges by bytes scanned. (Athena docs on partitioning and columnar formats.)
• Include event_id, received_at, producer_version. Without these you cannot deduplicate or replay.

A bronze table layout that has held up for us in production looks like this:

s3://lake/bronze/events/
  ├── event_date=2026-05-11/
  │     ├── event_hour=00/
  │     │     └── part-0000.snappy.parquet
  │     └── event_hour=01/...
  └── event_date=2026-05-12/...

You will sometimes see bronze partitioned by source system too (source=web/, source=ios/). That is fine as long as it sits inside the date partition, not outside it — date is the partition key your queries will care about 95% of the time.

Silver layer — cleaned, deduped, joined to identity

Silver is where the data becomes trustworthy. The same medallion pattern from Databricks and the Microsoft Fabric guide calls this "validated, cleaned, and enriched." For personalization specifically, silver has four jobs:

1. Deduplicate. Production event pipelines double-send. Hash event_id and drop duplicates, or use Hudi's record-level indexing if you are on that table format.
2. Resolve identity. A click from anonymous_id=abc followed by a login from user_id=42 needs to be stitched. The silver layer holds the result of identity resolution so downstream code can assume one row, one user.
3. Join in entity metadata. Item, content, product, or video metadata — whatever the recommender ranks — joined onto each event so downstream gold-layer feature jobs do not re-join from scratch.
4. Cast and enforce schema. Strings become enums, JSON blobs become typed columns, malformed rows go to a quarantine table. Schema-on-read is great for bronze and a disaster for everything downstream.

Silver is typically partitioned the same way as bronze (by event date), and is the layer most ad-hoc analytics queries hit. For an in-depth view of what identity resolution looks like as a pipeline stage, see our cold-start problem and day-zero personalization post.

Upserts: where Hudi earns its keep

Identity stitching is upsert-heavy. When a user logs in two weeks after their first anonymous session, you need to backfill user_id onto all those old rows. With Delta or Iceberg in pure Copy-on-Write mode, that means rewriting entire Parquet files. With Hudi's Merge-on-Read, the update appends a small log file that compacts later — far cheaper at scale. Onehouse's comparison documents the Peloton case study where Hudi let them move ingestion from daily to every 10 minutes.

Gold layer — feature tables shaped for ranking

Gold is where personalization diverges sharpest from analytics. A BI gold table is wide, denormalized, and read by humans through dashboards. A personalization gold table is a feature snapshot for a specific use case, read by ranking jobs and feature-store sync jobs.

In practice we keep a handful of distinct gold tables per use case:

• gold.user_features_realtime — latest snapshot per user, partitioned by user_id_hash (mod 256 or 1024 buckets). This is what the online feature store syncs from.
• gold.user_features_historical — point-in-time feature snapshots, partitioned by snapshot_date. This is what training jobs read for backfilled labels.
• gold.item_features — current state per item, much smaller cardinality, often under 100M rows.
• gold.user_item_interactions_recent — last 90 days of user×item events, partitioned by event_date. The ranker reads this for "recent activity" features.

The hash-bucket partition on user features is the key trick. Without it, queries like "give me features for these 10K users" scan the whole table. With 1024 buckets, the same query scans roughly 1/1024 of the data and finishes in seconds on Trino or Athena.

If your team is still thinking in terms of "a recommendation engine," our take on recommendation engine vs personalization layer is worth reading — in this view of personalization data lake architecture, the lake's gold tables are the personalization layer's substrate, not the engine itself.

Picking a table format: Iceberg, Delta, or Hudi

The three open lakehouse table formats — Apache Iceberg, Delta Lake, and Apache Hudi — all give you ACID transactions, schema evolution, and time travel on Parquet files in object storage. The differences only start to matter at personalization scale. The Onehouse comparison post goes deep; the short version:

• Hudi is built for streaming, upserts, and CDC. Merge-on-Read appends small log files and compacts in the background, so frequent updates do not rewrite full Parquet files. Native record-level indexing makes dedup and identity stitching cheap. Best for: personalization workloads with high-frequency upserts and minute-level freshness goals.
• Iceberg is built for analytics breadth. Excellent schema evolution, broad query-engine support (Trino, Spark, Snowflake, BigQuery, Athena), and the de facto standard for catalog interop. Best for: lakes where the gold tier is read by many engines and update rates are moderate.
• Delta Lake is the Databricks-native pick. Deep Spark integration, mature tooling, but the writer ecosystem outside Databricks is narrower than Iceberg's. Best for: shops already on Databricks where the team is not going to introduce a second platform.

Our default for greenfield personalization lakes is Hudi for bronze and silver (where upserts dominate) and Iceberg for gold (where analytical reads dominate). It is two table formats, but the operational pain is small compared to forcing one format to do both jobs. Engineering teams at Uber, Peloton, and Robinhood report similar splits.

The partitioning that actually matters

Partitioning is where data-lake projects quietly succeed or fail. Three rules we apply on every personalization lake:

Partition by what you filter on. If 90% of your gold-table reads look like WHERE user_id IN (...), partition by user_id_hash. If they look like WHERE event_date BETWEEN ..., partition by event_date. Resist the urge to partition by anything else, however logical it feels.

Cardinality matters. A country_code partition column with 200 values is fine. A user_id partition column with 200 million values will destroy your metadata layer and make queries slower, not faster. Hash to buckets (1024 is a good default for user_id) when cardinality is too high.

Avoid small-file hell. If your stream writes a partition every minute, each partition has tens of files within an hour, and within a week you have millions. Iceberg's rewrite_data_files and Hudi's clustering both fix this; budget for it in the platform team's roadmap from day one. Otherwise queries that should take seconds will take minutes purely because the engine is opening files instead of scanning rows.

The same principles apply to vector indexes if you are storing embeddings in the lake — which is increasingly common. Our post on knowledge graphs vs vector embeddings covers when and why you would store embeddings in gold-tier tables versus a dedicated vector DB.

Cost reality: S3 + Athena/Trino versus Snowflake

The cost conversation around a personalization data lake architecture gets unrealistic in vendor pitches. The honest version:

On a pure TPC-H benchmark Snowflake came out 49% cheaper than Athena on Iceberg across 18 of 22 queries, while Athena was faster on 8 of 22 queries (benchmark write-up from Yuval Yogev). That benchmark is not personalization, but it sets a useful baseline: Snowflake's cost-per-query at scale is competitive with — sometimes better than — open formats once you account for the engineering time to tune partitions and file sizes.

A few patterns we have seen hold up across real personalization workloads:

• S3 storage is cheap; Athena scans are not. S3 Standard is roughly $0.023/GB-month. Athena charges $5 per terabyte scanned. A poorly partitioned table that scans 10 TB to answer a question costs $50 per call — and dashboards call it 100 times a day.
• Athena's concurrency is capped. The default concurrent query limit is 25, and Athena is a multi-tenant pool. Heavy production workloads need workgroups and reserved capacity, which closes the cost gap with Snowflake.
• Trino on EKS is more flexible but you operate it. Open-source Trino self-hosted gives you concurrency control and pluggable connectors, but you now run a query engine. For teams under 50 engineers we usually recommend Athena until the bill makes Trino worth the headcount.
• Snowflake is predictable; that is the entire pitch. You pay for warehouse-hours, not bytes scanned. For workloads where scan volume is uncertain (ad-hoc analytics, ML feature exploration), the warehouse model removes a class of cost surprises.

A defensible split we have shipped: bronze and silver on Hudi + S3, queried mostly by batch Spark jobs; gold on Iceberg + S3, queried by Trino for ad-hoc and Athena for cheap one-off scans; an aggressive online feature store (Redis or ScyllaDB) for the actual p50 < 50 ms ranking path. Snowflake enters the picture if and only if non-engineers (analysts, growth team) need stable SQL access to the same gold tables — at which point you mount the Iceberg tables as external in Snowflake instead of duplicating storage.

Real-time and batch in the same lake

A personalization data lake architecture has to serve two clocks. Batch training jobs read months of history. Real-time ranking reads "what did this user do in the last 10 seconds." The lake supports both, but only if you design for both from day one.

The pattern we apply:

1. Bronze ingests in two streams: a batch stream (S3 → Hudi via Spark structured streaming, micro-batched every 5 minutes) and a hot stream (Kafka → online feature store directly, never touches the lake on the write path). Both end up in bronze eventually, but the hot path does not wait for the lake.
2. Silver runs as an incremental Hudi/Iceberg upsert job every 5-10 minutes, triggered by the bronze commit log. Identity stitching, dedup, schema enforcement.
3. Gold runs hourly for training tables, every 10-15 minutes for online-feature snapshots. The gold-to-online-store sync is its own job — typically a Kafka topic that the online store consumes.
4. The ranker reads only from the online store. It does not touch the lake at request time. Ever. If the online store is cold, you cold-start the user; you do not block the request waiting for S3.

This split — batch lake plus real-time path — is the same pattern that shows up in every credible reference architecture for ranking systems. Our five patterns for adding personalization post walks the higher-level shapes; this post is what the storage layer looks like underneath.

How ×marble fits in

If you would rather not build this stack yourself, this is roughly what we run inside ×marble. The medallion lake, the Hudi-on-bronze/Iceberg-on-gold split, the online feature store, the partitioning conventions — packaged as a managed personalization knowledge graph. You point us at your event stream and item catalog; we handle the lake-to-online-store sync, the feature freshness SLAs, and the ranking surface.

The same engine powers our product surfaces: Vivo for personalized AI video briefings, Video for personalized YouTube discovery, and Marble × Music for music personalization. The data lake architecture under each of them is what this post described — we built the platform once and ran it three times.

FAQ

What is a personalization data lake architecture?

A personalization data lake architecture is a medallion-layered lakehouse (bronze for raw events, silver for cleaned and identity-resolved data, gold for feature tables) paired with a real-time online feature store. The lake is the system of record for training and analytical workloads; the online store serves the actual ranking calls at sub-100 ms latency. The two together replace the older "data warehouse plus offline batch model" pattern that cannot serve real-time recommendations.

What is bronze silver gold personalization?

Bronze stores raw event data exactly as captured (clicks, plays, page views) partitioned by event date. Silver applies cleaning, deduplication, identity resolution, and schema enforcement — turning raw events into trustworthy entity-resolved rows. Gold materializes per-user and per-item feature tables shaped for ranking, partitioned by hashed user ID so that lookups for specific users are fast. Each layer serves a different consumer: bronze for audit and replay, silver for analytics, gold for ML feature jobs.

Should I use Iceberg, Delta, or Hudi for a personalization lake?

For personalization workloads specifically, Apache Hudi has the strongest fit on bronze and silver because of its Merge-on-Read upsert path and native record-level indexing, both critical for identity stitching and CDC. Apache Iceberg is the better pick for gold-layer feature tables read by many query engines, thanks to its broad ecosystem and schema-evolution story. Delta Lake is the default if you are already inside the Databricks ecosystem; outside it, the writer support is narrower.

Can a data lake serve real-time recommendations directly?

No. Object storage like S3 has read latencies in the tens to hundreds of milliseconds even for small files, and ranking calls need to return well under 100 ms end-to-end across multiple features. The lake feeds an online feature store (Redis, DynamoDB, ScyllaDB, or a purpose-built KV layer) which serves the actual request. The lake is your system of record and training source; the online store is the cache that gets ranked against.

How much does a personalization data lake cost at scale?

The dominant cost driver is bytes scanned, not storage. S3 Standard storage runs about $0.023/GB-month, while Athena queries cost about $5 per terabyte scanned — a poorly partitioned table that scans 10 TB per query costs $50 per call. Converting raw CSV or JSON to Parquet typically reduces scan costs by 5-10x. At higher scale, self-hosted Trino on EKS or a Snowflake external Iceberg setup becomes more cost-stable than per-query Athena.

Further reading

• Reference architecture for real-time personalization — the end-to-end stack this post drills into one layer of.
• Recommendation engine vs personalization layer — why the lake's gold tier is the substrate, not the engine.
• Knowledge graphs vs vector embeddings — when to store embeddings in gold tables vs a dedicated vector DB.
• Cold-start problem and day-zero personalization — what identity stitching in silver is solving.
• Databricks medallion architecture overview — canonical reference for the bronze/silver/gold pattern.
• Apache Hudi vs Delta vs Iceberg feature comparison — the most detailed technical comparison currently public.
• Athena vs Snowflake on Iceberg, TPC-H benchmark — the cost numbers cited in the cost section.