Part 2: Add\/Modify Columns | Rename Tables | Data Lineage<\/strong><\/p>\n\n\n\n This tutorial walks step by step through advanced Delta Live Tables (DLT) features in Databricks. It is based on the transcript you provided but rewritten into a structured learning guide<\/strong> with examples you can try.<\/p>\n\n\n\n In the previous tutorial, you built your first DLT pipeline<\/strong> with:<\/p>\n\n\n\n In this tutorial, you will:<\/p>\n\n\n\n \ud83d\udc49 You can inspect pipeline details in the Databricks Workflows \u2192 Delta Live Tables<\/strong> UI.<\/p>\n\n\n\n Streaming tables in DLT process only new data<\/strong> since the last checkpoint.<\/p>\n\n\n\n When you rerun the pipeline:<\/p>\n\n\n\n Suppose the business asks for sum of total price<\/strong> in addition to the count.<\/p>\n\n\n\n \ud83d\udc49 DLT automatically alters the schema<\/strong> of the table. No manual DDL needed.<\/p>\n\n\n\n If you forget to import DLT manages lifecycle automatically. To rename:<\/p>\n\n\n\n Behind the scenes:<\/p>\n\n\n\n \ud83d\udc49 You can inspect the checkpoint location in your storage container ( Unity Catalog tracks table and column lineage<\/strong> automatically.<\/p>\n\n\n\n Column-level lineage: click on \ud83d\udc49 Works for DLT and non-DLT tables.<\/p>\n\n\n\n <\/p>\n\n\n\n <\/p>\n","protected":false},"excerpt":{"rendered":" Delta Live Tables (DLT) Internals & Incremental Load Part 2: Add\/Modify Columns | Rename Tables | Data Lineage This tutorial walks step by step through advanced Delta… <\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-826","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/826","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/comments?post=826"}],"version-history":[{"count":1,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/826\/revisions"}],"predecessor-version":[{"id":827,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/826\/revisions\/827"}],"wp:attachment":[{"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/media?parent=826"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/categories?post=826"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/tags?post=826"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
\n\n\n\nIntroduction<\/h2>\n\n\n\n
\n
\n
\n\n\n\nDLT Pipeline Internals<\/h2>\n\n\n\n
\n
\n\n\n\nIncremental Load Using DLT<\/h2>\n\n\n\n
Example \u2014 Append incremental data<\/h3>\n\n\n\n
-- Insert 10,000 new rows into raw orders table\nINSERT INTO dev.bronze.orders_raw\nSELECT * FROM dev.samples_tpch.orders TABLESAMPLE (1 PERCENT) LIMIT 10000;\n<\/code><\/pre>\n\n\n\n\n
\n\n\n\nAdd\/Modify Columns in Materialized Views<\/h2>\n\n\n\n
Python DLT<\/h3>\n\n\n\n
import dlt\nfrom pyspark.sql.functions import count as f_count, sum as f_sum\n\n@dlt.table(\n name=\"orders_aggregated_gold\",\n table_properties={\"quality\": \"gold\"},\n comment=\"Aggregation by market segment with sum of total price\"\n)\ndef orders_aggregated_gold():\n df = dlt.read(\"orders_silver\")\n return (df.groupBy(\"c_mktsegment\")\n .agg(\n f_count(\"o_orderkey\").alias(\"count_orders\"),\n f_sum(\"o_totalprice\").alias(\"sum_totalprice\")\n ))\n<\/code><\/pre>\n\n\n\n
\n\n\n\nDebugging Mode in DLT<\/h2>\n\n\n\n
\n
Connect to Pipeline<\/code>).<\/li>\n\n\n\nExample<\/h3>\n\n\n\n
sum<\/code>, DLT will fail validation and show the error in the event logs<\/strong>. Fix by:<\/p>\n\n\n\nfrom pyspark.sql.functions import sum as f_sum\n<\/code><\/pre>\n\n\n\n
\n\n\n\nRename DLT Tables<\/h2>\n\n\n\n
@dlt.table(\n name=\"orders_silver\", # changed from \"joined_silver\"\n table_properties={\"quality\":\"silver\"},\n comment=\"Renamed from joined_silver\"\n)\ndef orders_silver():\n return dlt.read(\"join_v\")\n<\/code><\/pre>\n\n\n\n\n
orders_silver<\/code>, removes joined_silver<\/code>.<\/li>\n<\/ul>\n\n\n\n
\n\n\n\nInternals of DLT Datasets<\/h2>\n\n\n\n
\n
\n\n\n\nHow Streaming Tables Handle Incremental Data<\/h2>\n\n\n\n
\n
_delta_log<\/code> + checkpoints<\/code>).<\/p>\n\n\n\n
\n\n\n\nLineage in Unity Catalog<\/h2>\n\n\n\n
Example<\/h3>\n\n\n\n
\n
dev.etl.orders_aggregated_gold<\/code> in Unity Catalog.<\/li>\n\n\n\n\n
orders_silver<\/code> \u2190 orders_bronze<\/code> + customer_bronze<\/code><\/li>\n\n\n\norders_raw<\/code>, customer_raw<\/code><\/li>\n<\/ul>\n<\/li>\n<\/ol>\n\n\n\ncount_orders<\/code> to trace back to o_orderkey<\/code>.<\/p>\n\n\n\n
\n\n\n\n\u2705 Key Takeaways<\/h2>\n\n\n\n
\n
name<\/code> in code \u2192 lifecycle handled automatically.<\/li>\n\n\n\n
\n\n\n\n