In this article we will describe in detail the steps to follow to complete the automatic deployment of Azure Data Factory pipelines in the Development (dev), Staging (stg) and Production (prd) environments. In software development, the use of integration (CI) and continuous deployment (CD) is done to release better code in a fast way. This possibility also exists for data engineers working with Azure Data Factory. That is, we will have the possibility of moving pipelines between the different environments. Furthermore, working in this way, several people in the team may be working at the same time on the same data pipeline. In this case, we are going to work on an example of automating the deployment from dev to prd. All thanks to ADF integration with Azure DevOps Automation. Let’s see how.
The infrastructure necessary to complete this process is
Azure Resource Group
Azure SQL Database
Azure Key Vault
Azure Data Factory
About Alberto Alonso Marcos
My name is Alberto Alonso. Actually I work with Sogeti Spain in Business Intelligence Department with Microsoft Technologies. My profile is very orientated to customer, and how the DATA can improve the organization. My first steps in the data management were in the Pharmaceutical Sector. (I´m pharmaceutical too). I worked hard to extract and built procedures for gathering all the information across the organization. Measurement all kind of events. Aggregating different sources like ERP, LIMS, HVAC, OEE tools, and productivity machine reports.
More on Alberto Alonso Marcos.