Okay, so for the first assignment of this section will be adding data to the silver layer.

Right now we have the following data files in the bronze layer, as you can see.

We want to add tables to the silver layer that have been assigned appropriate data types.

Merged where possible and remove any unnecessary columns.

So in general, the data in the silver layer should contain columns that are not knowledgeable and 100%

complete.

The files should be in the parquet file format.

This is because the parquet file format can retain data types and properties.

Okay, so in this overlay, I’d like the following tables.

I’d like an order’s table that contains the order ID as type integer.

The order underscore timestamp column as type timestamp.

This column should be renamed from the order underscore date time column.

I’d like the customer underscore ID as type integer the store underscore name as type string.

Now just as a hint.

Note that the order underscore timestamp column at present isn’t in the standard format to convert directly

to a timestamp, so you’ll need to first read it in as a string and then use the to underscore timestamp

function along with the date time patterns.

You can access the two underscore timestamp function via the Spark API.

It’s under the function section and if you filter by date time functions.

So here it is.

You’ll also need the spot date time patterns, so you can reference this link here.

I’ll leave both of these links in the Course Resources notebook containing my detailed solutions as

another hint.

Note that the orders table in the silver layer also contains the store name.

You can get this column by merging the stores table with the orders table.

Okay, So we also have an order items table, and I’ve detailed the columns and the data types to include

here.

We have a products table and a customer’s table too.

Again, I’ve lifted the columns and the data types that are required.

Okay, So that’s the requirement.

Before you get started, please create a folder called Section six.

Oh six

assignments and then create a notebook.

In that folder and call it oh, one bronze to silver.

You can write your solution code in this notebook.

Note this assignment is meant to mirror a real world scenario.

And it’s supposed to be difficult.

So give it your best attempt.

And in the next lecture, I’ll provide a detailed walkthrough.

In addition to this, there’s also a solutions notebook available for you to review in detail as part

of the course resources.

So if you get stuck, please give that a review before the next lecture where I go over the solutions.