Essential Dataverse Feature: Synapse Link for Azure Analytics

The Revolutionizing Feature in Azure Synapse Analytics: Synapse Link

According to the latest YouTube video by "Serverless SQL", every Azure Synapse Analytics engineer called the Synapse Link is a necessity. This feature enables seamless data synchronization from various data sources, including Cosmos DB, SQL Server, and now, the Dataverse into a Synapse Analytics workspace, enabling easier query execution. This synchronization capability offers users the convenience of collating multiple data sources into a single Synapse workspace without the need for custom data loading processes.

The brand new Synapse Link for Dataverse allows data synchronization from Power Apps, i.e., Dynamics 365, into a Synapse Analytics workspace. This synchronization makes the data accessible for querying through Serverless SQL Pools and Spark. This means that data typically stored and manipulated within Power Apps, like Dynamics 365 sales data, can now be made live, and accessible using near-real-time synchronization offered by the Synapse Link for Dataverse feature.

Implementing this feature involves setting up a Synapse Link for Dataverse, then synchronizing the Dynamics 365 Sales data with an existing Synapse Analytics workspace. This set up initiates real-time synchronization of valuable business data, enabling access to live datasets for dynamic reporting, predictive analysis, and making strategic business decisions.

For comprehensive instructions on setting up Synapse Link for Dataverse, follow this link for an in-depth guide.

Data Synchronization with Synapse Link

There are two ways to configure Synapse Link, syncing with the analytics suite or with Azure Data Lake Gen2 account. Upon configuring to sync with the Analytics suite, the Dataverse data gets synchronized to an Azure Data Lake Gen2 storage account and deploys a Lake Database in the suite.

This Lake Database holds tables originating from Dynamics, which have been prepped for export. Primarily, two tables are present for each entity, a "near-real-time" table, and a "per hour" snapshot table. This arrangement minimizes the possibility of any locking during reading process which may occur on the "near-real-time" tables as data is written to CSV files.

The other option is syncing with Azure Data Lake, which synchronizes the Dataverse data to an Azure Data Lake Gen2 storage account. This synchronized data can be queried or loaded using any service that can connect to an Azure Data Lake Gen2 account and read/process CSV files.

The Prospects of Synapse Link

At present, CSV is the only supported file format for data synchronization and export, with the Parquet format expected to be supported by Q1 2023. The Synapse Link feature is continuously being updated with more functionality, and will soon allow setup with private endpoints. Future blogs promise using Append Only for suppressing deletes in Synapse/Data Lake and a deeper exploration into the snapshotting of data in the Data Lake.

The concept of "near" real-time synchronization with this feature is based on how fast the changed data is added to a queue based on the frequency of changes made to the data. In most cases, data updating is completed within a minute, with bulkier updates taking a bit more time. Resynchronization options are also available in case of replication issues or issues with the destination synced data.

Read the full article Every Azure Synapse Analytics Engineer Needs to Know this Essential Dataverse Feature - Synapse Link

Learn about Every Azure Synapse Analytics Engineer Needs to Know this Essential Dataverse Feature - Synapse Link

Synapse Link, a part of the Azure Synapse Analytics suite of services, offers seamless data synchronization from sources such as Cosmos DB, SQL Server, Azure SQL Database, and the Dataverse into a Synapse Analytics workspace. Effectively merging numerous data sources into a Synapse workspace negates the need for setting up custom data loading processes.

A practical application of this feature is Synapse Link for SQL, which can synchronize data from Azure SQL Database and SQL Server into Dedicated SQL Pools. Additionally, with the launch of Synapse Link for Dataverse, the synchronization of data from Power Apps (including Dynamics 365) into a Synapse Analytics workspace is now possible. The synchronized data becomes readily available for querying with Serverless SQL Pools and Spark.

Synapse Link for Dataverse supports near real-time synchronization of data from Power Apps like Dynamics 365 Sales. This post walks through the setup of a Synapse Link for Dataverse and synchronizing Dynamics 365 Sales data with a Synapse Analytics workspace. It will explore what transpires when the initial setup is run, provide an understanding of what "near real-time" signifies, and address potential issues that might arise.

• Synapse Link holds two configuration options. The first option synchronizes Dataverse data to an Azure Data Lake Gen2 storage account and deploys a Lake Database in Synapse Analytics. The Lake Database functions by holding the Dynamics tables configured for export. Two tables exist for every entity – a "near-real-time" table, and a "per hour" snapshot table – to minimize any locking during reading.

• The second configuration option synchronizes Dataverse data to an Azure Data Lake Gen2 storage account. This data is then queryable or loadable using any service capable of connecting to an Azure Data Lake Gen2 account and reading/processing CSV files.

Please note that these are relevant at the time of this article, and there may be additional configuration options in the future. Currently, Synapse Link only supports the CSV format for exporting; however, the future expansion of the Parquet format support is expected (Q4 2022/Q1 2023).

To walk through a Synapse Link setup, this tutorial will employ Dynamics 365 Sales, which is a model-driven app that utilizes Dataverse. A Dynamics 365 Sales environment is required for the walkthrough, a 30-day Sales trial is suggested. The Dynamics environment should be set in the same region as the Synapse workspace and the Azure Data Lake Gen2 account.

Additionally, a user needs to be licensed via Office 365 admin portal to access Dynamics 365. The Power Apps Developer (free version) can be assigned to the user from the Office 365 admin area. A license is usually granted automatically when a trial is created. Permissions required for setting up Synapse Link with Synapse Analytics are distributed across Dynamics 365 and Azure.

Once the necessary permissions are in place, Synapse Link for Dataverse will need to be set up with the Synapse Analytics workspace. This includes a sequence of steps such as logging into Power Apps, selecting the relevant environment and setting up a new link via Dataverse -> Azure Synapse Link.

The new link needs to have the 'Connect to your Azure Synapse Analytics workspace' enabled, and it also needs relevant information such as Subscription, Resource Group, Synapse Workspace, and Storage Account details. Remember, the sequence and item names might vary with updates so it's always better to check Microsoft's latest documentation.

The configuration of Synapse Link with a Synapse Workspace apart, it’s important to understand that during the setup, each table synchronized from Dynamics is created in the Lake Database along with all column names and data types specified.

Subsequent Refreshes occur after the initial synchronization has completed. This stage presents the opportunity to automatically start synchronizing any data changes within Dynamics 365. The actual time taken to synchronize data will depend on the volume of data that has changed.

Furthermore, you need to know how to navigate the Synapse Analytics workspace and exploit its many features including clicking on the Data tab on the left menu, expanding Lake Database to see the lake database created during Synapse Link Setup, and expanding Tables to see the tables selected for export, plus metadata tables (E.G. StateMetadata).

In conclusion, mastering the Synapse Analytics, and specifically the Synapse Link feature, can go a long way towards streamlining your data synchronization and consolidation processes. More advanced topics and updates on this technology will be covered in subsequent blogs.

More links on about Every Azure Synapse Analytics Engineer Needs to Know this Essential Dataverse Feature - Synapse Link

Every Azure Synapse Analytics Engineer Needs to Know ...

Dec 31, 2022 — Every Azure Synapse Analytics Engineer Needs to Know this Essential Dataverse Feature - Synapse Link. by HubSite 365 about Serverless SQL.

Keywords

Azure Synapse Analytics, Synapse Link features, Essential Dataverse Feature, Azure Engineer, Azure Synapse, Dataverse Synapse Link, SEO Analytics Azure, Synapse Analytics Engineer, Azure Synapse Link, Dataverse for Azure Synapse Engineer.