In this lecture, I want to introduce you to the medallion architecture.

The medallion architecture is essentially a data design pattern used to logically organize data in a lake house.

There are typically three layers in the medallion architecture.

Bronze Layer

• First you have the bronze layer, also known as the ingestion layer.
• This layer contains raw data ingested from various sources in various formats.
• The layer stores data in a distributed file store such as the Azure Data Lake Storage.
• The objective of this layer is to ingest data as is.
• It should provide a historical archive of source data.

Silver Layer

• This contains refined and enriched tables from the bronze layer.
• You can provide transformations, merge your data, cleanse your data and so on.
• The silver layer brings data from different sources into an enterprise view, and it should enable self-service analytics.
• So, for example, data scientists and analysts should be able to use the data from this layer to answer business problems.
• The objective here isn’t to produce a huge number of transformations, but just enough so that the data can still capture the relevant business entities, concepts and transactions.

If you produce too many transformations, then this can be lost.

Gold Layer

• This layer contains curated business level tables.
• The data here should be consumption ready so it can contain data that directly addresses business questions.
• The layer is used for reporting, and it can contain more denormalized and optimized data models. So Renormalized means that the data can contain more columns with duplicated data. This kind of data is not typically suitable for relational databases.

Generally speaking, there is no strict policy on what kind of tables should be contained in each layer.

For example, it’s advised to keep the data in the silver layer in the third normal form.

But depending on your situation and real-world requirement, there is of course flexibility to adapt this with good reason.

There are some resources online around the medallion architecture and I’ll share them as links in the article posted in the next lecture and with the course resource notebook so you can review in more detail if you wish.

So now that you know the medallion architecture, I’ll provide you with a challenge section to implement this in practice along with everything else you’ve learned so far.

https://www.databricks.com/glossary/medallion-architecture