CHALLENGES
- Extract data from 4 on-prems SQL databases, an Azure SQL database, and from an online CRM via a REST API
- Make them available to a few selected people in the business units (internal customers) in SQL format or via Power BI
- That data be updated at the frequencies required by internal customers: daily or hourly according to business topics + every 2 minutes during office hours for a very specific operational need
SOLUTION
- Architecture involving Microsoft Fabric for data management and analysis, including lakehouses and SQL endpoints for analysis.
- Temporary adaptation of the architecture pending updates to Microsoft Fabric to simplify direct data extraction.
- Use of Azure Data Factory for on-premises data extraction and Azure Data Lake for temporary storage.
BENEFITS
- Easily accessible production reports (Power BI in the browser), which update automatically and contain quality data validated by subject matter expert
- Centralization of data required by analysts.
- Improved data security by design: Instead of the analysts extracting data from production servers and store them as Excel files on their workstation, the data remains at all times in Fabric.
- Improved data governance. Documentation of calculation rules and requirements via a matrix bus, Azure DevOps, and data mapping documents.