Microsoft Fabric - Your data is in the Lakehouse, but now what?
You've got your data into OneLake and a Lakehouse, but now what? What can you do with that data after you've landed it in Microsoft Fabric?
Learn about Your data is in the Lakehouse, but now what? | Microsoft Fabric (Public Preview)
Microsoft Fabric is an all-in-one analytics solution that covers everything from data movement to data science, real-time analytics, and business intelligence. It offers a comprehensive suite of services, including data lake, data engineering, and data integration, all in one place. A key aspect of Microsoft Fabric is the concept of a "Lakehouse", which combines the benefits of traditional data lakes and data warehouses.Here are some key points about data in a Lakehouse in Microsoft Fabric:
Data Ingestion: There are multiple ways to get data into a Lakehouse in Microsoft Fabric, such as connecting to existing SQL Server and copying data into delta table on the Lakehouse, uploading files from your computer, copying and merging multiple tables from other Lakehouses into a new delta table, connecting to a streaming source to land data in a Lakehouse, and referencing data without copying it from other internal Lakehouses or external sources. You can also use file upload from a local computer, run a copy tool in pipelines, set up a dataflow, or use Apache Spark libraries in notebook code. The approach you choose to load data will depend on the use case, for example, for small file upload from a local machine, you can use Local file upload; for large data source, you can use Copy tool in pipelines; and for complex data transformations, you can use Notebook code.
Lakehouse Architecture: A Lakehouse in Fabric can be implemented either as a lakehouse or data warehouse architecture, or you can combine these two together. The architecture uses a medallion model where the bronze layer has the raw data, the silver layer has the validated and deduplicated data, and the gold layer has highly refined data. A lakehouse implementation involves creating a Fabric workspace, creating a Lakehouse, ingesting data, transforming data, and loading it into the Lakehouse. You can load data from the bronze, silver, and gold zones as delta lake tables. Additionally, you can connect to your Lakehouse using TDS/SQL endpoint and create a Power BI report to analyze data. You can also orchestrate and schedule data ingestion and transformation flow with a pipeline.
Data Flow and Transformation: In a Lakehouse in Microsoft Fabric, data is ingested from various data sources using connectors integrated into the Fabric pipeline. The data is then transformed and stored in a standardized Delta Lake format, which allows all the Fabric engines to access and manipulate the same dataset stored in OneLake without duplicating data. Data transformation can be done using either pipelines/dataflows or notebook/Spark for a code-first experience. After transformation, the data can be consumed using Power BI for reporting and visualization. Each Lakehouse has a built-in TDS/SQL endpoint for easy connectivity and querying of data in the Lakehouse tables from other reporting tools.
Please note that I could not find the specific step-by-step guide to load data into a Lakehouse due to time constraints, but I can continue the search if you would like more information.
More links on about Your data is in the Lakehouse, but now what? | Microsoft Fabric (Public Preview)
- Ingest data into the lakehouse - Microsoft Fabric
- May 23, 2023 — In this tutorial, you will ingest additional dimensions and fact tables from the Wide World Importers (WWI) into the lakehouse.
- Copy sample data into Lakehouse and transform with ...
- May 23, 2023 — Microsoft Fabric is currently in PREVIEW. This information relates to a prerelease product that may be substantially modified before it's ...
- Explore the data in your lakehouse with a notebook
- Jun 8, 2023 — Learn how to use a notebook to explore lakehouse data, connect a lakehouse and a notebook, and generate code cells in a notebook.
- Introducing Synapse Data Engineering in Microsoft Fabric
- May 23, 2023 — Build a lakehouse for all your organizational data ... Synapse Data Engineering public preview is shipping with 'Runtime 1.1' which includes ...
- Microsoft Fabric Launch Event | May 24-25
- May 23, 2023 — Microsoft Fabric is now in public preview! Unify your data to unlock intelligence faster with a lake-first, open, and governed analytics ...
- Introducing Microsoft Fabric and Copilot in Microsoft Power BI
- May 23, 2023 — Copilot in Power BI, now in private preview, combines advanced generative AI with your data to help everyone uncover and share insights ...
- Lakehouse tutorial - build a report - Microsoft Fabric
- Jun 12, 2023 — Microsoft Fabric is in preview. Prerequisites. Prepare and transform the data using notebooks and Spark runtime. Build a report. Power BI is ...
- How to Convert Tableau Hyper for Use in a Microsoft ...
- May 27, 2023 — With the recently announced public preview of Microsoft Fabric, you can also easily work with data sourced from Tableau Hyper files in a Fabric ...
Keywords