Fabric Espresso: Loading existing Delta Tables to Silver! Welcome to our Fabric Espresso series! In this video, Stijn shows how to tap into existing delta tables residing on ADLS Gen2 with Microsoft Fabric! We will show you how to link those tables using shortcuts and how to start querying it with the SQL endpoint!
"Exploring Microsoft Fabric and Delta Tables in Depth" likely refers to learning about Microsoft Fabric's Lakehouse and its use of Delta Lake tables. Here's what I found:
Microsoft Fabric Lakehouse is a data architecture platform for storing, managing, and analyzing both structured and unstructured data in a single location. For seamless data access across all compute engines in Microsoft Fabric, Delta Lake is chosen as the unified table format.
Microsoft Fabric Runtime for Apache Spark is built on the same foundation as Azure Synapse Analytics Runtime for Apache Spark, but it has key differences that provide more streamlined behavior across all engines in the Microsoft Fabric service. Key performance features are turned on by default, but advanced Apache Spark users can revert configurations to previous values to better align with specific scenarios.
Microsoft Fabric Lakehouse and the Apache Spark engine support all table types, including both managed and unmanaged tables. This includes views and regular non-Delta Hive table formats. Tables defined using PARQUET, CSV, AVRO, JSON, and any Apache Hive compatible file format work as expected.
The Lakehouse Explorer user interface experience varies depending on the table type, and currently, it only renders table objects.
The Lakehouse Explorer provides a tree-like view of the objects in the Microsoft Fabric Lakehouse item. It has a key capability of discovering and displaying tables that are described in the metadata repository and in OneLake storage. Auto-discovery also applies to tables defined over OneLake shortcuts.
Microsoft Fabric Lakehouse supports tables defined over OneLake shortcuts to provide utmost compatibility and no data movement. If multiple tables are present in the destination, one shortcut should be created per table. Folders with files should use Apache Spark to use the destination directly using relative paths and load the data into Lakehouse native Delta tables for maximum performance.
There is a feature called "Load to Tables" that streamlines loading data into Delta tables. This feature provides a visual experience for loading common file formats to Delta to boost analytical productivity for all personas.
Microsoft Fabric Lakehouse proactively enables important parameters to minimize common problems associated with big data tables, such as compaction and small file sizes, and to maximize query performance. There are scenarios where these parameters need changes, and Delta Lake table optimization guides provide in-depth instructions on how to efficiently maintain Delta tables for maximum performance.
It's worth noting that as of the time the information was gathered, Microsoft Fabric was still in the preview stage.
Exploring Microsoft Fabric and Delta Tables in Depth
Microsoft Fabric offers a powerful solution for accessing and analyzing data using Delta tables. Delta tables allow for the creation and management of high-quality, multidimensional tables that can significantly improve the data processing capabilities of an organization. With Microsoft Fabric, users can easily link to existing Delta tables on ADLS Gen2 and query them via SQL endpoints. The OneLake shortcuts and Data engineering tools provided by Microsoft Fabric make the integration process simple and efficient, allowing users to harness the full potential of Delta tables for their data management programs.
In this Fabric Espresso series video, Stijn shows how to tap into existing delta tables residing on ADLS Gen2 with Microsoft Fabric. He explains how to link these tables using shortcuts and how to start querying them with the SQL endpoint. Learn more about the Lakehouse SQL Endpoint, OneLake shortcuts, and data engineering in Fabric on the Microsoft Fabric website. Stijn Wynants, the Fasttrack Engineer featured in the video, is also available to connect with on LinkedIn, on his website sql-stijn.com, and on Twitter. #MICROSOFTFABRIC #FABRIC
ONELAKE, MICROSOFTFABRIC, FABRIC, ADLS Gen2, SQL Endpoint, Data Engineering