Azure Data Factory

Course Contents
• Introduction of Azure
• Introduction of Azure Data Factory
• Data Factory components
• Differences between v1 and v2
• Triggers
• Control Flow
• SSIS in ADFv2
• Demo

Introduction of Azure
• Azure is Microsoft's cloud computing platform, provides cloud services that gives you the freedom to build, manage,
and deploy applications on a massive global network using your favorite tools and frameworks. A quick explanation on
how Azure works

• Cloud computing is the delivery of computing services over the Internet using a pay-as-you-go pricing
model. In other words it's a way to rent compute power and storage from someone’s data center.
• Microsoft categorizes Azure cloud services into below product types:
• Compute
• Storage
• Networking
• Web
• Databases
• Analytics and IOT
• Artificial Intelligence
• DevOps
Introduction of Azure
n of Azure Data Factory
• Azure Data Factory is a cloud-based data integration service to compose data storage, movement, and
processing services into automated data pipelines.
• It compose of data processing, storage, and movement services to create and manage analytics pipelines,
also provides orchestration, data movement and monitoring services.
• In the world of big data, raw, unorganized data is often stored in relational, non-relational, and other storage
systems, big data requires service that can orchestrate and operationalize processes to refine these enormous
stores of raw data into actionable business insights.
• Azure Data Factory is a managed cloud service that's built for these complex hybrid extract-transform-load
(ETL), extract-load-transform (ELT), and data integration projects.
• Azure Data Factory is a data ingestion and transformation service that allows you to load raw data from over
70 different on-premises or cloud sources. The ingested data can be cleaned, transformed, restructured, and
loaded back into a data warehouse.

• Currently, there are two versions of the service: version 1 (V1) and version 2 (V2).

Introduction of Azure Data Factory

• The pipelines (data-driven workflows) in Azure Data Factory typically perform the following four steps:

• Connect and collect: The first step in building an information production system is to connect to all the
required sources of data and processing, such as software-as-a-service (SaaS) services, databases, file shares,
 and FTP web services. The next step is to move the data as needed to a centralized location for subsequent
processing.
• Transform and enrich: After data is present in a centralized data store in the cloud, process or transform the
collected data by using compute services such as HDInsight Hadoop, Spark, Data Lake Analytics, and Machine
Learning.
Data Factory Components
• Publish: After the raw data has been refined into a business-ready consumable form, load the data into
Azure Data Warehouse, Azure SQL Database, Azure Cosmos DB, or whichever analytics engine your business
users can point to from their business intelligence tools.
• Monitor: After you have successfully built and deployed your data integration pipeline, providing business
value from refined data, monitor the scheduled activities and pipelines for success and failure rates.
• Azure Data Factory is composed of four key components. These components work together to provide the
platform on which you can compose data-driven workflows with steps to move and transform data.
• Pipeline: A data factory might have one or more pipelines. A pipeline is a logical grouping of activities that
performs a unit of work. For example, a pipeline can contain a group of activities that ingests data from an
Azure blob, and then runs a Hive query on an HDInsight cluster to partition the data.
• Activity: Activities represent a processing step in a pipeline. For example, you might use a copy activity to
copy data from one data store to another data store. Data Factory supports three types of activities: data
movement activities, data transformation activities, and control activities.
• Datasets: Datasets represent data structures within the data stores, which simply point to or reference the
data you want to use in your activities as inputs or outputs.

• Linked services: Linked services are much like connection strings, which define the connection information
that's needed for Data Factory to connect to external resources. For example, an Azure Storage-linked
service specifies a connection string to connect to the Azure Storage account.
• Linked services are used for two purposes in Data Factory :
• To represent a data store that includes, but isn't limited to, an on-premises SQL Server database, Oracle database, file
share, or Azure blob storage account.
• To represent a compute resource that can host the execution of an activity. For example, the HDInsight Hive activity runs
on an HDInsight Hadoop cluster.
Data Factory Components
• Overview of Data Factory flow

• Overview of Data Factory flow

Data Factory Components

• Other components of Data Factory.

Data Factory Components
• Triggers: Triggers represent the unit of processing that determines when a pipeline execution needs to be
kicked off. There are different types of triggers for different types of events.
• Pipeline runs: A pipeline run is an instance of the pipeline execution. Pipeline runs are typically instantiated
by passing the arguments to the parameters that are defined in pipelines. The arguments can be passed
manually or within the trigger definition.
• Parameters: Parameters are key-value pairs of read-only configuration. Parameters are defined in the
pipeline. Activities within the pipeline consume the parameter values.
• Control flow: Control flow is an orchestration of pipeline activities that includes chaining activities in a
sequence, branching, defining parameters at the pipeline level, and passing arguments while invoking the
pipeline on-demand or from a trigger. It also includes custom-state passing and looping containers, that is,
For-each iterators.
Differences between v1 and v2

Feature Version 1 Version 2

Data Factory Components
Datasets A named view of data that references Datasets are the same in the current
the data, can be utilized in activities as version. However, you do not need to
inputs and outputs. define availability schedules for datasets.
Datasets identify data within different
data stores, such as tables, files, folders,
and documents

Availability defines the processing

window slicing model for the dataset
(for example, hourly, daily, and so on).
Differences between v1 and v2
Linked services are much like connection Linked services are the same as in Data
Linked services strings, which define the connection Factory V1, but with a new connectVia
information that's necessary for Data property to utilize the Integration
Factory to connect to external resources. Runtime compute environment of the
current version of Data Factory.

Feature Version 1 Version 2

Differences between v1 and v2
Pipelines A data factory can have one or more Pipelines are groups of activities that are
pipelines. A pipeline is a logical grouping performed on data. However, the
of activities that together perform a task. scheduling of activities in the pipeline has
been separated into new trigger
resources.
The Data Factory V1 concepts of
startTime, endTime, and isPaused are no
longer present in the current version of
Data Factory.
Differences between v1 and v2
Activities Activities define actions to perform on In this version of Data Factory, activities
your data within a pipeline. Data still are defined actions within a pipeline
movement (copy activity) and data The current version of Data Factory
transformation activities (such as Hive, introduces new control flow activities.
Pig, and MapReduce) are supported.

Feature Version 1 Version 2

Differences between v1 and v2
Hybrid data movement Now called Integration Runtime, Data Data Management Gateway is now called
and activity dispatch Management Gateway supported moving Self-Hosted Integration Runtime. It
data between on-premises and cloud. provides the same capability as it did in
V1.
The Azure-SSIS Integration Runtime in the
current version of Data Factory also
supports deploying and running SQL
Server Integration Services (SSIS)
packages in the cloud.

Parameters NA Parameters are key-value pairs of

readonly configuration settings that are
defined in pipelines.
Differences between v1 and v2
Feature Version 1 Version 2

Expressions Data Factory V1 allows to use functions In this version of Data Factory, one can
and system variables in data selection use expressions anywhere in a JSON
queries and activity/dataset properties. string value.

Pipeline runs NA A single instance of a pipeline execution.

Each pipeline run has a unique pipeline
run ID. The pipeline run ID is a GUID that
uniquely defines that particular pipeline
run.
Differences between v1 and v2
Activity runs NA
An instance of an activity execution
within a pipeline.

Trigger runs NA
An instance of a trigger execution. For
more information.

Scheduling
Scheduling is based on pipeline Scheduler trigger or execution via
start/end times and dataset availability. external scheduler.

Feature Version 1 Version 2

Differences between v1 and v2
Chaining activities In V1, must configure the output of an In this version of Data Factory, in the
activity as an input of another activity to current version, one can chain activities in
chain them. a sequence within a pipeline, by using the
dependsOn property in an activity
definition to chain it with an upstream
activity.

Branching activities NA Can branch activities within a pipeline.

The If-condition activity provides the
same functionality that an if statement
provides in programming languages.
Differences between v1 and v2
Custom state passing NA Activity outputs including state can be
consumed by a subsequent activity in the
pipeline. By using this feature, we can
build workflows where values can pass
through activities.

Feature Version 1 Version 2

Looping containers NA The ForEach activity defines a repeating

control flow in your pipeline. This activity
iterates over a collection and runs
specified activities in a loop.
Differences between v1 and v2
Trigger-based flows NA Pipelines can be triggered by on-demand
(event-based, i.e. blob post) or wall-clock
time.

Invoking a pipeline from NA The Execute Pipeline activity allows a

another pipeline Data Factory pipeline to invoke another
pipeline.

Delta flows NA A key use case in ETL patterns is “delta

loads”. New capabilities in this current
version, such as lookup activity, flexible
scheduling, and control flow, enable this
use case.
Differences between v1 and v2

Feature Version 1 Version 2

Other control flow NA ForEach activity, Web activity, Lookup

activities activity, Get metadata activity, Wait
activity.

Deploy SSIS packages to NA We can Azure-SSIS if you want to move

Azure our SSIS workloads to the cloud, create a
data factory by using the current version,
and provision an Azure-SSIS Integration
Runtime.
Differences between v1 and v2
Custom activities In V1, we implement (custom) DotNet In a custom activity in this version, you
activity code by creating a .NET class don't have to implement a .NET interface.
library project with a class that You can directly run
implements the Execute method of the commands, scripts, and your own custom
IDotNetActivity interface. Therefore, you code compiled as an executable.
need to write your custom code in .NET
Framework 4.5.2 and run it on
Windows-based Azure Batch Pool nodes.
Triggers
How do pipelines getstarted

1. on-demand
2. Wall-clockSchedule
3. Tumbling Window (aka time-slices inv1)
4. Event
How do pipelines getstarted
1.PowerShell:
Triggers
Invoke-AzureRmDataFactoryV2Pipeline +Parameters

2.Rest API:
https://management.azure.com/subscriptions/mySubId/resourceGroups/myResou
rceGroup/providers/Microsoft.DataFactory/factories/{yourDataFactory}/pipelines
/{yourPipeline}/createRun?api-version=2017-03-01-preview

3..NET:
client.Pipelines.CreateRunWithHttpMessagesAsync(+parameter
s)

4. AzurePortal
(Data factory -> <Author & Monitor> -> Pipeline runs)
Triggers
 Triggers
Run pipeline by schedule
Triggers
Tumbling Window
Tumbling window triggers are a type of trigger that fires at a periodic time interval from a
specified start time, while retaining state. Tumbling windows are a series of fixedsized,
non-overlapping, and contiguous time intervals.
Triggers
Triggers

Event Based Trigger

Triggers
Data integration scenarios often require Data Factory customers to trigger pipelines based
on events. Data Factory is now integrated with Azure Event Grid, which lets you trigger
pipelines on an event.
Control Flow
Activities Known from v1 -Data TransformationActivities

Data transformationactivity Computeenvironment

Hive HDInsight[Hadoop]

Pig HDInsight[Hadoop]

MapReduce HDInsight[Hadoop]
HadoopStreaming HDInsight[Hadoop]

Spark HDInsight[Hadoop]

AzureVM
Machine Learning activities: Batch
Executionand
UpdateResource

StoredProcedure
Azure SQL, Azure SQL Data Warehouse,
orSQL Server

U-SQL Azure Data LakeAnalytics

Control Flow
New! Control FlowActivitiesin v2
Controlactivity Description

allows a Data Factory pipeline to invoke anotherpipeline.

ExecutePipeline
Activity

ForEachActivity used to iterate over a collection and executes specified activities ina loop.

WebActivity call a custom REST endpoint and pass datasets and linkedservices
LookupActivity
look up a record/ table name/ value from any external source to be referencedby
succeeding activities. Could be used for incrementalloads!

retrieve metadata of any data in Azure Data Factory e.g. did another pipelinefinish
GetMetadata
Activity

Do UntilActivity similar to Do-Until looping structure in programminglanguages.

do something based on condition that evaluates to true orfalse.

IfCondition
Activity

Control Flow
New! Control FlowActivitiesin v2
Controlactivity Description

Append to add a value to an existing array variable defined in a Data Factory pipeline.
Variable
Activity
Filter activity to apply a filter expression to an input array.

Set Variable to set the value of an existing variable of type String, Bool, or Array defined in a Data
Activity Factory pipeline.

Validation to ensure the pipeline only continues execution once it has validated the attached
activity dataset reference exists
Wait activity
the pipeline waits for the specified period of time before continuing with execution of
subsequent activities.

Webhook
activity to control the execution of pipelines through your custom code.

Data flow to run your ADF data flow in pipeline debug (sandbox) runs and in pipeline triggered
activity runs. (This Activity is in public preview)

SSIS in ADFv2
Managed Cloud

EnvironmentPick# nodes & node size

Resizable
 SQL Standard Edition, Enterprise coming soon

Azure
SSIS Integration
Project Runtime
Compatible
Same SSIS runtime across Windows, Linux, Azure
Cloud

SSIS + SQL Server

SQL Managed instance + SSIS
(in ADFv2) Access on premises
data via VNet

GetStarted
Hourly pricing (no SQL Server license
SSIS in ADFv2
Integration runtime -Differentcapabilities
1. DataMovement
Move data between data stores, built-in connectors, format
conversion, column mapping, and performantand scalable
data transfer
2. ActivityDispatch
Dispatchandmonitortransformationactivities(e.g.StoredProcon
SQLServer,HiveonHDInsight..)

3. SSIS package execution

Execute SSIS packages
51

SSIS in ADFv2
Combinations of IR types, networks and
capabilities

IR type Public network Private network

SSIS in ADFv2
Azure Data movement
Activity dispatch

Self-hosted Data movement Data movement

Activity dispatch Activity dispatch
 Azure-SSIS SSIS SSIS
packageexecution packageexecution

https://docs.microsoft.com/en-us/azure/data-factory/concepts-integration-runtime 52

Integrationruntimes
1. Azure IntegrationRuntime

• move data between cloud data stores • fully managed • serverless

compute service (PaaS) on Azure • cost will occur only for time of
duration • user could define data movement units 29 • compute size
auto scaled for copy jobs
SSIS in ADFv2
Integrationruntimes
2. Self-hosted IntegrationRuntime
SSIS in ADFv2
• perform data integration securely in a private network
environment w/o direct line-of-sight from the public cloud
environment • Installed on-premises in your environment •
Supports copy activity between a cloud data stores and a data
store in private network • Supports dispatching the transform
activities • Works in your corporate network or virtual private
network30 • Only Outbound http based connections to open
internet • Scale out supported

Integrationruntimes
3. Azure-SSIS IntegrationRuntime
SSIS in ADFv2
• fully managed cluster of Azure VMs for native execution of SSIS
packages.
SSIS in ADFv2
• Access to on-premises data access using Vnet (classic in
preview) • SSIS Catalog on Azure SQL DB or SQL Managed
Instance • scale up: set node size • scale out: number of nodes
• reduce costs by start/stop of service
SSIS in ADFv2
 SSIS in ADFv2
Determining which IR touse
• IRisreferencedaslinkedserviceinthedatafactory •
Transformation Activity: target compute needslinked service •
Copy Activity: source and sink need linked service, the
computation is determined automatically (see detailson msdn) •
IntegrationruntimelocationscandifferfromitsData Factory location
which usesit

https://docs.microsoft.com/en-us/azure/data-factory/concepts-integration-runtime
SSIS in ADFv2
Samples ADF and IRlocations
SSIS in ADFv2
SSIS in ADFv2
Scaleable IntegrationServices
How to scale up/out using 3 Settings on Azure SSIS IR

1. Configurablenumberofnodes on which SSIS is executed

$AzureSSISNodeSize = "Standard_A4_v2" # minimum size,
others avail.*
2. Configurable size ofnodes
$AzureSSISNodeNumber = 2 #between 1 and 10 nodes*
3. Configurable maximum parallel executions pernode
InPowershell CommandLet:
SSIS in ADFv2
SetAzureRmDataFactoryV2IntegrationRuntime
$AzureSSISMaxParallelExecutionsPerNode = 2 # between 1-8*
SSIS in ADFv2
SSIS in ADFv2
SSIS in ADFv2
SSIS in ADFv2
Notes from thefield
1. Connect in SSMS directly to the DB SSISDB to see SSIS
Catalog 2. Deploy from Visual Studio only in Project
Deployment Mode, workaround SSMS Import 3. In
Preview no SSIS 3rd Party components supported (e.g.
Theobald for SAP, cozyroc..) 4. V2 IR supports only V2
pipelines and V1 IR supports only V1. Both cannot be
used interchangeably. Even though it is the same installer
5. Use same location for Azure-SSIS IR and the used
SSIS in ADFv2
(SQL37 Azure) DB for SSIS Catalog
ExecutionMethods
1. SSIS packages can be executed via SSMS

2. SSIS packages can be executed via CLI

›Run dtexec.exe from the command prompt

3. SSIS packages can be executed via custom code/PSH using SSIS MOM
.NET SDK/API
SSIS in ADFv2
› Microsoft.SqlServer.Management.IntegrationServices.dll is installed in
.NET GAC with SQL Server/SSMS installation
4. SSIS packages can be executed via T-SQL scripts executing SSISDB
sprocs
›Execute SSISDB sprocs [catalog].[create_execution] +
[catalog].[set_execution_parameter_value] +
[catalog].[start_execution]

SchedulingMethods
1. SSIS package executions can be directly/explicitly scheduled via ADFv2 App
(Work in
SSIS in ADFv2
Progress)
› For now, SSIS package executions can be indirectly/implicitly scheduled via
ADFv1/v2 Sproc
Activity
2. If you use Azure SQL MI server to host SSISDB
› SSIS package executions can also be scheduled via Azure SQL MI Agent
(Extended Private Preview)

3. If you use Azure SQL DB server to host SSISDB

› SSIS package executions can also be scheduled via Elastic Jobs (Private
Preview)
SSIS in ADFv2
4. If you keep on-prem SQL Server
› SSIS package executions can also be scheduled via on-prem SQL Server
Agent
Demo

comments

Azure Data Factory Azure SQL Database Power BI Dashboard

ingest copy and transform visualize

(just for fun)
storage blob
copy
Data Management
Gateway
SQL database
(on-premises)
What is Azure Data Factory?
• Data Factory is a fully managed, cloud-based, data-integration service that
automates the movement and transformation of data. Like a factory that runs
equipment to transform raw materials into finished goods, Azure Data Factory
orchestrates existing services that collect raw data and transform it into
readytouse information.
• By using Azure Data Factory, you can create data-driven workflows to move data
between on-premises and cloud data stores. And you can process and transform
data by using compute services such as Azure HDInsight, Azure Data Lake
Analytics, and the SQL Server Integration Services (SSIS) integration runtime.
• With Data Factory, you can execute your data processing either on an Azurebased
cloud service or in your own self-hosted compute environment, such as SSIS, SQL
Server, or Oracle. After you create a pipeline that performs the action you need,
 you can schedule it to run periodically (hourly, daily, or weekly, for example), time
window scheduling, or trigger the pipeline from an event occurrence.

Introduction to Azure Data Factory

• The data landscape is more varied than ever with unstructured and
structured data originating from many cloud and on-premises sources.
• Data Factory enables you to process on-premises data like SQL Server,
together with cloud data like Azure SQL Database, Blobs, and Tables.
• These data sources can be composed, processed, and monitored through
simple, highly available, fault-tolerant data pipelines.
• Combining and shaping complex data can take more than one try to get it
right, and changing data models can be costly and time consuming.
•
Using Data Factory you can focus on transformative analytics while the
service 'takes care of the plumbing'.

What are the top-level concepts of Azure Data

Factory?
• An Azure subscription can have one or more Azure Data Factory
instances (or data factories).
•
• Azure Data Factory contains four key components that work together
as a platform on which you can compose data-driven workflows with
steps to move and transform data.
Pipelines

A data factory can have one or more pipelines. A pipeline is a logical grouping of activities to perform a unit of work. Together, the
activities in a pipeline perform a task. For example, a pipeline can contain a group of activities that ingest data from an Azure blob
and then run a Hive query on an HDInsight cluster to partition the data. The benefit is that you can use a pipeline to manage the
activities as a set instead of having to manage each activity individually. You can chain together the activities in a pipeline to
operate them sequentially, or you can operate them independently, in parallel.

•
Activities Activities represent a processing step in a pipeline. For example, you can use a Copy activity to copy data from one data
store to another data store. Similarly, you can use a Hive activity, which runs a Hive query on an Azure HDInsight cluster to transform
or analyze your data. Data Factory supports three types of activities: data movement activities, data transformation activities, and
control activities.
Datasets
•
• Datasets represent data structures within the data stores, which simply point to or reference the data you want to use in your
activities as inputs or outputs.

•
Linked services Linked services are much like connection strings, which define the connection information needed for Data Factory
to connect to external resources. Think of it this way: A linked service defines the connection to the data source, and a dataset
represents the structure of the data. For example, an Azure Storage linked service specifies the connection string to connect to the
Azure Storage account. And an Azure blob dataset specifies the blob container and the folder that contains the data.
Linked services have two purposes in Data Factory:
To represent a data store that includes, but is not limited to, an on-premises SQL Server instance, an Oracle database instance, a file share, or an
Azure Blob storage account..

• To represent a compute resource that can host the execution of an activity. For example, the HDInsight Hive activity runs on an HDInsight Hadoop
cluster.
Triggers

• Triggers represent units of processing that determine when a pipeline execution is kicked off. There are different types of triggers for different types of
events.
Pipeline runs
•
• A pipeline run is an instance of a pipeline execution. You usually instantiate a pipeline run by passing arguments to the parameters that are defined in
the pipeline. You can pass the arguments manually or within the trigger definition.
Parameters

• Parameters are key-value pairs in a read-only configuration. You define parameters in a pipeline, and you pass the arguments for the defined
parameters during execution from a run context. The run context is created by a trigger or from a pipeline that you execute manually. Activities within
the pipeline consume the parameter values.

• A dataset is a strongly typed parameter and an entity that you can reuse or reference. An activity can reference datasets, and it can consume the
properties that are defined in the dataset definition.

• A linked service is also a strongly typed parameter that contains connection information to either a data store or a compute environment. It's also an
entity that you can reuse or reference.
• Control flows
• Control flows orchestrate pipeline activities that include chaining
activities in a sequence, branching, parameters that you define at the
pipeline level, and arguments that you pass as you invoke the
pipeline on demand or from a trigger. Control flows also include
custom state passing and looping containers (that is, foreach
iterators).
Data Factory Overview

Congratulations on continuing the path to learn Azure Courses!

Welcome to Azure Data Factory.

Hope you have a good foundational understanding of Azure cloud services. If not, you should go
through the Azure Databases and Azure Storage courses before going any further.

In this course, you will be introduced to fundamental concepts of data factory, creating data factory,
building pipelines, copying data, transforming data.

Happy Learning!

Prelude

In the world of Big data, the raw and unorganized data is stored in relational, non-relational, and
other storage systems. The raw data doesn't have the proper context or meaning to provide
meaningful insights to data scientists, analysts, or business decision makers. It requires some service
to orchestrate and refine these raw data into actionable business insights.

Azure Data Factory is a managed cloud integration service that is built for these complex hybrid
extract-load-transform (ELT), extract-transform-load (ETL), and data integration projects.
Integration Runtime

The image shows how the different integration runtimes can be used in combination to
offer rich data integration capabilities and network support.
Azure Data Factory

• It is a cloud-based data integration service that supports to create data-driven

workflows (pipelines) in the cloud for data transformation, data orchestration, and
automating data movement.

• Azure Data Factory (ADF) allows to create and schedule pipelines that ingest data
from different data stores.
• It can process the transformation of data by using compute services such as Azure
HDInsight (Hadoop), Azure Data Lake Analytics, and Azure Machine Learning.
Key Components in ADF
There are a set of top-level concepts to be familiar with them before diving into the Azure
Data Factory.

• Pipeline:
It is the logical grouping of activities that perform a task, where each can operate
individually or independently in parallel. A data factory can contain one or more
pipelines. The major benefit of these pipelines is, it allows us to manage a set of
operations instead of managing each operation individually.

• Activity:
An activity is represented as a processing step or task in a pipeline such as you can
have a copy activity for copying data between data stores. It performs three kinds of
activities such as data transformation, data movement, and control activities.

Key Components in ADF

• Datasets:
Datasets represent data structures with a data store that points to data that needs to
use in your activities as inputs and outputs.

• Linked Services:
Linked services are used to represent connection objects for sources, destinations
and compute resources that contains the connection strings (connection information
needed for data factory to connect with external resources).
These four components comprise the ADF that works together to compose pipelines with
steps to move and transform data.
Note: An Azure subscription can have more than one data factory instances.

Triggers and Pipeline Runs

• Triggers:
Triggers represent the processing unit that determines when a pipeline execution
needs to be kicked off. These are of different types for different events.

• Pipeline Runs:
It is an instance of the pipeline execution that is instantiated by passing arguments to
the parameters, which are defined in pipelines. The arguments can be passed
manually or within the trigger definition.

ADF Pricing
The pricing is broken down into four ways that you are paying for this service.

1. Azure activity runs vs Self-hosted activity runs:

There are different pricing models for these. For the Azure activity runs, it is about copying
activity. So you are moving data from an Azure Blob to an Azure SQL database or Hive
activity running the high script on an Azure HDInsight cluster. With self-hosted activity runs,
you can copy activity moving from an on-premises SQL Server to an Azure Blob Storage, a
stored procedure to an Azure Blob Storage, or a stored procedure activity running a stored
procedure on an on-premises SQL Server.
2. Volume of data moved:

It is measured in data movement units (DMUs). You should be aware of it, as this will change
from default to auto, by using all the DMUs it can handle. This is paid on an hourly basis.
Let’s say you specify and use two DMUs. It takes an hour to move that data. The other
option is that you could use eight DMUs and it takes 15 minutes. This price is going to end
up the same. You’re using 4X the DMUs, but it’s happening in a quarter of the time.

ADF Pricing
3. SSIS integration run times:

Here, you’re using A-series and D-series compute levels. When you go through these, you
will understand that it depends on the compute requirements to invoke the process (how
much CPU, how much RAM, how much attempt storage you need).

4. The inactive pipeline:

You’re paying a small account for pipelines (about 40 cents currently). A pipeline is
considered inactive if it’s not associated with a trigger and hasn’t been run for over a week.
Yes, it’s a minimal charge, but they do add up. When you start to wonder where some of
those charges come from, it’s good to keep this in mind.
Supported Regions
The regions currently supporting for provisioning the data factory are West Europe, East US, and East
US 2.

However, the data factory can access data stores and compute resources from other regions to move
data between data stores or process data using compute services, the service that powers data
movement in data factory is available globally in many areas.

Provisioning Data Factory

Azure subscription is the prerequisite for the data factory instance provision. The video
shows how to provision the ADF from the Azure portal.

If you have trouble playing this video, please click here for help.

No transcript is available for this video.

Copy Data Using Copy Data Tool

Copy Data option in ADF allows copying data between data sources. The video shows a
demo of copying data from Azure blob to SQL database using Azure portal and Visual
Studio Cloud explorer.

Play

11:50

-16:55
 Mute

Settings

Enter fullscreen

Play
If you have trouble playing this video, please click here for help.

No transcript is available for this video.

On-Premises Data Sources, Data Gateway

As you know, Azure lets you connect with the on-premises data sources such as SQL server,
and SQL server analysis services. for queries or processing by provisioning the data
gateway.
The data gateway provides secure data transfer between on-premises data sources and your
Azure services in the cloud. It requires the following steps:

1. Download and run setup in an on-premise computer.

2. Registering your gateway by specifying a name and recovery key.
3. Creating a gateway resource in Azure in a subscription.
4. Connecting data sources to the gateway resource.

Note: You can register your gateway resource in any region, but it recommended to be in
the same region of your data factory.
On-Premises Data Gateway
The picture shows how the data gateway works between on-premises data sources and
Azure services.
ADF Management Tools

Azure Data Factory can be managed such as creating ADF, creating pipelines, monitoring
pipelines through various ways such as:
 1. Azure Portal
2. Azure PowerShell
3. Azure Resource Manager Templates
4. Using REST API
5. Using .NET SDK

Datasets

Datasets identify data such as files, folders, tables, documents within different data stores. For
example, an Azure SQL dataset specifies the schema and table in the SQL database from which the
activity should read the data.

Before creating a dataset, you have to create a linked service to link your data store to the data
factory.

Both linked service and datasets are defined in JSON format in ADF.

Linked Service Structure

Linked Service structure is defined in a JSON format as below for an AzureStorage
linked service.
{ "name":
"AzureStorageLinkedService",
"properties": {
"type": "AzureStorage",
"typeProperties": {
"connectionString": {
"type": "SecureString",
"value":
"DefaultEndpointsProtocol=https;AccountName=<accountname>;AccountKey=<accountkey>"
}
},
"connectVia": { "referenceName": "<name
of Integration Runtime>", "type":
"IntegrationRuntimeReference"
}
}
}

Dataset Structure
Dataset structure is defined in JSON format for an AzureBlob dataset as shown below.

{
"name": "AzureBlobInput",

"properties": {
 "type": "AzureBlob",
"linkedServiceName": {
"referenceName": "MyAzureStorageLinkedService",
"type": "LinkedServiceReference", },

"typeProperties": {
"fileName": "input.log",
"folderPath": "adfgetstarted/inputdata",

"format": {
"type": "TextFormat",
"columnDelimiter": ","
}
}
}
}
Pipeline Overview
A typical pipeline in an Azure data factory performs the above four activities represented in
the picture.

Workflow of Pipelines
Connect and Collect:
The first step in building a pipeline is connecting to all the required sources of data and
processing the movement of data as needed to a centralized location for subsequent
processing. Without data factory, it requires to build custom data movement components or
write services to move to integrate these data sources.
Transform and Enrich
The collected data that is presented in the centralized data store is transformed or processed
by using compute services such as HDInsight Hadoop, Spark, Data Lake Analytics, and
Machine Learning that is produced as feed to production environments.

Workflow of Pipelines
Publish:
After the raw data was refined it is loaded into Azure Data Warehouse, Azure SQL Database,
Azure CosmosDB, or whichever analytics engine your business users can point to from their
business intelligence tools.
Monitor:
After the successful build and deployment of your data integration pipeline, you can monitor
the scheduled activities and pipelines for success and failure rates. ADF has built-in support
for monitoring pipeline Azure Monitor, API, PowerShell, Log Analytics, and health panels on
the Azure portal.

Creating a Pipeline
The video shows creating a pipeline from scratch in the Azure portal.

Play

05:19

-06:35

Mute

Settings

Enter fullscreen

Play
If you have trouble playing this video, please click here for help.

No transcript is available for this video.

Monitoring Pipelines
The video shows monitoring an up and running pipeline from ADF resource explorer in
Azure Portal.

Play

05:07
 -07:03 Mute

Settings

Enter fullscreen

Play
If you have trouble playing this video, please click here for help.

No transcript is available for this video.

Relationship Between Four Components

The picture shows the relationship between the four key components in ADF.
Scheduling Pipelines
A pipeline is active only between its start time and end time that does not execute before,
or after the start and end times, respectively. If it is paused, it does not get executed
irrespective of its start and end time. For a pipeline to run, it should not be paused.
You can define the pipeline start and end times in the pipeline definition in JSON format as
below.

"start": "2017-04-01T08:00:00Z",
"end": "2017-04-01T11:00:00Z"
"isPaused": false
Specifying Schedule for an Activity

You must specify schedulers for the activities that are executing in the pipeline to run the
pipeline.
Example of an activity that is scheduled to run hourly in the JSON format.

"scheduler": {
"frequency": "Hour",
"interval": 1 },

Specify Schedule to Dataset

Similar to scheduling activity in a pipeline, you can specify a schedule to the dataset, such
as, you can specify a schedule that takes input datasets and produce output data in a
frequency (hourly, daily, weekly, monthly).
For example, the below JSON format file defines the input and output data available in an
hourly frequency:
{
"name": "AzureSqlInput",
"properties": {
"published": false,
"type": "AzureSqlTable",
"linkedServiceName": "AzureSqlLinkedService", "typeProperties":
{ "tableName":
"MyTable" },
"availability": {
"frequency":
"Hour", "interval":
1 },
"external": true,
"policy": {}
}
}

Note: You must specify schedule frequency and intervals in the availability section.

Scheduling Pipelines: Demo

The video shows scheduling the pipelines at regular intervals in visual studio, you can try
this in the portal as well.
If you have trouble playing this video, please click here for help.

No transcript is available for this video.

Troubleshooting a Scheduled Pipeline

The video shows how to troubleshoot a scheduled pipeline by monitoring it and fixing the
failed execution activity.

If you have trouble playing this video, please click here for help.

No transcript is available for this video.

Visual Authoring
The Azure Data Factory user interface experience (UX) lets you visually author and deploy
resources for your data factory without having to write any code.
You can drag activities to a pipeline canvas, perform test runs, debug iteratively, and deploy
and monitor your pipeline runs. There are two approaches for using the UX to perform visual
authoring:

• Author directly with the Data Factory service.

• Author with Visual Studio Team Services (VSTS) Git integration for collaboration,
source control, or versioning.

Visual authoring with the Data Factory service differs from visual authoring with VSTS in two
ways:
 • The Data Factory service doesn't include a repository for storing the JSON entities for
your changes.
• The Data Factory service isn't optimized for collaboration or version control.

Integration JSON in ADF

The Integration Runtime (IR) is the compute infrastructure used by Azure Data Factory to provide
data integration capabilities across different network environments such as data movement, activity
dispatch, and SSIS package execution.

While moving data from data stores in public and private networks, it provides support for built-in
connectors, format conversion, column mapping, and scalable data transfer.

The IR (Integration runtime) provides the bridge between the activity and linked Services.

Types of Integration Runtime

There are three different types of IR which you have to choose the type that best serve the
data integration capabilities you are looking for:

• Azure
• Self-hosted
• Azure-SSIS

Azure IR type is recommended to choose for data movement and activity dispatch
activities over a public network whereas the self-hosted IR type is recommended over both
public and private networks for the same activities.
The Azure SSIS IR type is recommended to choose for SSIS package execution over both
public and private networks.
Azure Integration Runtime

An Azure integration runtime is capable of:

Running copy activity and data movement activities between cloud data stores.

Activity dispatching the following data transform activities in public network: HDInsight Hadoop,
Machine Learning Batch Execution and update resource activities, Data Lake Analytics U-SQL activity
and other custom activities.

Azure IR supports connecting data stores over a public network with public accessible endpoints.

Self-Hosted Integration Runtime

Self-hosted IR is to perform data integration securely in private network. You can install a self-hosted
IR on-premises environment behind your corporate firewall, or inside a virtual private network.

It is capable of:

Running copy activity and data movement activity between data stores.

Activity dispatching the following transform activities against compute resources in On-Premise or
Azure Virtual Network: HDInsight Hadoop, Machine Learning Batch Execution and update resource
activities, Data Lake Analytics U-SQL activity, and other custom activities.

Lift and Shift SSIS

It is easy to move your SQL Server Integration Services (SSIS) workloads, projects, and packages to the
Azure cloud, as it deploys, runs and manages SSIS projects and packages in the SSIS Catalog on Azure
SQL Database or with familiar tools such as SQL Server Management Studio (SSMS).
Moving your on-premises SSIS workload to Azure will reduce your operational costs and provides
maximum scalability.

Azure SSIS Integration Runtime

To lift and shift existing SSIS workload, you can create an Azure-SSIS IR. which is dedicated to run or
execute SSIS packages. It can be provisioned in either public network or private network.

Integration Runtime Location:

The IR Location defines the location of its back-end compute, and essentially the location where the
data movement, activity dispatching, and SSIS package execution are performed. The IR location can
be different from the location of the data factory it belongs to.
Azure Data Factory Working

The above GIF represents the working of ADF, it requires to install three IRs in the whole
scenario of data movement and data transformation from on-premise to Azure.
 1. Installs self-hosted IR type in the on-premise that provides integration capabilities
overs private network.
2. Installs Azure IR type at the transformation part.
3. Installs Azure IR type at the Azure data store that provides integration capabilities
over the public network.

Scenarios for Copy Data Using ADF

There are three types of copy data scenarios performed across different environments between data
stores such as:

On-premise to Azure

Azure cloud data store instance to another Azure cloud data store instance

SaaS Application to Azure

There are two ways to create a pipeline for copy activity:

From ADF editor and monitor tile, choosing editor option to manually create key components
(dataset, linked services) and perform copy activity.

Using copy data tool in ADF.

Scenario 1: On-Premises to Azure

Let's assume a scenario copying data from on-premise SQL server to Azure Data Lake Store using
Copy data tool in ADF. Follow the below steps to perform this kind of activity:

Choose Copy data option from the ADF

It has six steps to be evaluated in a sequence such as Properties, Source, Destination, Settings,
Summary, and Deployment.

Properties:

Provide the task name, task description, and task schedule.

Source Settings
Source:
1. Choose the SQL server data store and provide details of it such as connection name,
Server name, Database name, Authentication type, username, and password.
2. It requires to run Integration runtime to provide capabilities for data movement. For
this, you have to choose create integration runtime and launch express setup on
this computer. This will install selfhosted IR on your computer.
3. Choose the existing tables from your SQL server that needs to be copied. It also
allows copying a table by filtering a column. Destination and Settings
Destination:

Choose the data store data lake store and provide the details of it such as connection name, network
environment, azure subscription, data lake store account name, authentication type, tenant.

Choose the output file or folder (where to copy data).

Choose file format settings.

Settings:
Choose actions that need to be performed such as:

Abort copy on the first incompatible row.

Skip all incompatible rows.

Skip and log all incompatible rows.

Summary and Deployment

Summary:

Shows you the properties and source settings, click next for deployment.

Deployment:

Shows the deployment status and other options that allow to edit pipeline and monitor.

Watch this video in open network (not using tcs LAN) from 9th minute to have a better understanding
about how to perform copy activity from on-premise to Azure.

Scenario 2: Azure to Azure

Copying data from one Azure data store to another data store using copy tool is explained in the first
topic (Introduction to Azure data factory), please refer to card no: 11.
Note: Check this video on open network, it explains performing copy activity manually by creating
linked service, datasets for data stores using Azure portal. This is a similar process for any type of
scenario to create pipeline.

Scenario 3: SaaS Application to Azure

The video shows copying data from a dynamic CRM tool to Azure data lake store using
copy data tool.

If you have trouble playing this video, please click here for help.

No transcript is available for this video.

Transform Data in ADF

Data transformation is executed as a transformation activity that executes in a computing
environment such as an Azure HDInsight cluster or an Azure Batch.
Data factory supports various transformation activities that can be added to pipelines either
individually or chained with another activity. Which are:

1. HDInsight (Hadoop) compute environment is used for Hive, Pig, MapReduce,

Hadoop Streaming, Spark data transformation activities.
2. Azure VM compute environment is used for Machine learning batch execution and
update resource data transformation activities.
3. Azure SQL, Azure SQL data warehouse compute environment is used for Stored
procedure data transformation activity.
 4. Azure Data Lake Analytics compute environment is used for U-SQL data
transformation activity.

U-SQL Transformations
Data Lake Analytics U-SQL Activity that runs a U-SQL script on an Azure Data Lake Analytics
compute linked service.

1. Create an Azure Data Lake Analytics account before creating a pipeline with a Data
Lake Analytics U-SQL Activity.
2. The Azure Data Lake Analytics linked service requires a service principal
authentication to connect to the Azure Data Lake Analytics service.
3. To use service principal authentication, register an application entity in Azure Active
Directory (Azure AD) and grant it the access to both the Data Lake Analytics and the
Data Lake Store it uses.

U-SQL Transformations Activity: Demo

The video shows creating a pipeline for data transformation activity that uses the Azure
data lake compute environment.

If you have trouble playing this video, please click here for help.

No transcript is available for this video.

Hive Transformations: Demo

Similar to the Data lake activity, HDHadoop Hive activity executes Hive queries.
 Play

15:43

-18:20

Mute

Settings

Enter fullscreen

Play
If you have trouble playing this video, please click here for help.

No transcript is available for this video.

Custom Query and Parameters

You can parameterize a linked service and pass dynamic values at run time. Linked service
parameterization is supported in the Data Factory UI in the Azure portal for the following
data stores. For all other data stores, you can parameterize the linked service by selecting
the Code icon on the Connections tab and using the JSON editor.

• Azure SQL Database

• Azure SQL Data Warehouse
• SQL Server
• Oracle
• Cosmos DB
• Amazon Redshift
• MySQL Azure Database for MySQL
Custom Activity
If you need to transform data in a way that is not supported by Data Factory, you can
create a custom activity with your own data processing logic and use the activity in the
pipeline. You can configure the custom .NET activity to run using either an Azure Batch
service or an Azure HDInsight cluster.
You can create a Custom activity with your data movement or transformation logic and use
the activity in a pipeline.
The parameters required to define for a custom activity in JSON format are: name, activities,
typeproperties.
Refer to this link for azure batch basics as the custom activity runs your customized code
logic on an Azure Batch pool of virtual machines.

Custom Activity Example

The following JSON snippet defines a pipeline with a simple Custom Activity. The activity
definition has a reference to the Azure Batch linked service.
{ "name": "MyCustomActivityPipeline",
"properties": { "description": "Custom activity
sample", "activities": [{
"type": "Custom",
"name": "MyCustomActivity",
"linkedServiceName": { "referenceName":
"AzureBatchLinkedService", "type":

"LinkedServiceReference" },
"typeProperties": {
"command": "helloworld.exe",
"folderPath": "customactv2/helloworld",
"resourceLinkedService": {
"referenceName": "StorageLinkedService",

"type": "LinkedServiceReference"
}
}
}]
}
}
Hands-on scenario
Your company has a requirement to maintain employee details in the SQL database. You are
given a text file containing employee details that has to be migrated. As a cloud computing
professional, you plan to simplify the task by using Azure Data Factory. i) Create a storage
account: Location: (US) East US 2, Performance: Standard, Account Kind: Storage V2
(general-purpose v2), Replicapagetion: Locally-redundant storage (LRS). ii) Create a Blob
storage and upload the file containing employee data (refer to the sample employee data
provided in the following table). iii) Create SQL database: Server: Create new, Location: (US)
East US 2, Compute + storage: Basic, Network Connectivity: Public endpoint, Allow Azure
services and resources to access the server: Yes, Add current client IP address: Yes. iv) Set a
firewall rule for the SQL database to allow IP addresses from 0.0.0.0 to 255.255.255.255 and
write an SQL query to create a table database (refer to the SQL query provided in the
following table). v) Create Data Factory: Region: East US 2, Configure Git later: mark as later.
vi) Create a Copy data tool in the data factory to move data from blob storage to the SQL
database.. Employeedata.txt
FirstName|LastName

John|Brito

Alwin|Chacko

SQL Query
CREATE TABLE dbo.emp
(
ID int IDENTITY(1,1) NOT NULL,
FirstName
varchar(50), LastName varchar(50)
)
GO
CREATE CLUSTERED INDEX IX_emp_ID ON dbo.emp (ID);

Note:
Use the credentials given in the hands-on to log in to the Azure Portal, create a new resource
group and use the same resource group for all resources. The
Username/Password/Services Name can be as per your choice, after completing the
handson, delete all the resources created.
Course Summary

So far in this course, you have learned the following topics:

Provisioning an Azure Data Factory

Creating Pipelines for various activities

Transforming data through different compute environments

Monitoring ADF

Hope you had a better understanding of the data factory concept. Hope you had hands-on practice in
creating pipelines and activities.
Azure Updates
Like many other cloud infrastructure platforms today,
Azure is continuously developing updates to its services and components.
Every effort has been taken to update course content where there are significant changes to
product capability. However, there will be occasions when the course content does not
exactly match the latest version of the product.
Hence, we encourage you to check Azure updates as a starting point for the latest
information about updates in Azure

Raj Kumar Manda

Thank You