Scribd
Upload
4 views2 pages

ADF Class Notes

The document outlines a real-time use case for an Azure Data Factory (ADF) pipeline that processes daily sales data from CSV files stored in Azure Blob Storage and loads it into an Azure SQL Database. The pipeline includes activities such as fetching metadata, iterating over files, copying data, and optionally calling a stored procedure for validation. It also provides design tips like using parameterized datasets, logging, and error handling strategies.

Uploaded by

lavanyayadav301998
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
4 views2 pages

ADF Class Notes

The document outlines a real-time use case for an Azure Data Factory (ADF) pipeline that processes daily sales data from CSV files stored in Azure Blob Storage and loads it into an Azure SQL Database. The pipeline includes activities such as fetching metadata, iterating over files, copying data, and optionally calling a stored procedure for validation. It also provides design tips like using parameterized datasets, logging, and error handling strategies.

Uploaded by

lavanyayadav301998
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 2

Azure Data Factory (ADF) - Class Notes

Real-Time Use Case: Sales Data Load Pipeline

Scenario:

A retail company receives daily sales data from various branches as CSV files in an Azure Blob Storage

container.

These files need to be processed and loaded into an Azure SQL Database for reporting purposes.

ADF Pipeline Flow

1. Trigger:

- Schedule trigger runs the pipeline every night at 2 AM.

2. Activities:

- Get Metadata Activity:

Fetches file names from Blob Storage.

- ForEach Activity:

Iterates over each file returned by the metadata activity.

- Copy Activity (inside ForEach):

Source: CSV file from Blob

Sink: Azure SQL Sales table

- Stored Procedure Activity (optional):


Azure Data Factory (ADF) - Class Notes

Calls a procedure to validate and summarize data post-load.

3. Monitoring:

- Check the pipeline run status and performance in ADF Monitor.

Pipeline Design Tips

- Use parameterized datasets for dynamic file names and paths.

- Use logging tables in SQL DB to record successful/failed loads.

- Implement error handling using the Web Activity or Fail Activity.

You might also like

pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy