www.learnomate.

org

A Steps towards the bright future

Learnomate Technologies is the Information technology company which provide training

on different IT Technologies.
Out of that GCP Data Engineer is the one of the technology.
Course structure design in such a way that student will learn from Basic concepts to
advance.

Welcome to the world of Google Cloud Platform (GCP) Data Engineering! In this training
course, you'll dive into the exciting realm of managing and analyzing data on one of the
world's most powerful cloud platforms.
This course is designed to equip you with the knowledge and skills necessary to excel in
the role of a GCP Data Engineer. Whether you're new to the field or looking to enhance
your existing expertise, this training program will provide you with a comprehensive
understanding of GCP's data services, tools, and best practices.

+91 7757062955 +91 8983069523 info@learnomate.org

 www.learnomate.org

A Steps towards the bright future

Course Overview

Module 1: GCP Introduction

Why we need Cloud.
Overview of Google Cloud Platform (GCP)
Key GCP Services and Products
How to create Free Tier Account in GCP

Module 2: GCP Interfaces

Cloud Console
▶Navigating the GCP Console
▶Configuring the GCP Console for Efficiency
▶ Using the GCP Console for Service Management

Cloud Shell
▶ Introduction to GCP Shell
▶ Command-line Interface (CLI) Basics
▶ GCP Shell Commands for Service Deployment and Management

Cloud SDK
▶ Overview of GCP Software Development Kits (SDKs)
▶ Installing and Configuring SDKs
▶ Writing and Executing GCP SDK Commands

Module 3: GCP Locations

Regions
▶ Understanding GCP Regions
▶ Selecting Regions for Service Deployment
▶ Impact of Region on Service Performance

+91 7757062955, +91 8983069523 info@learnomate.org

 www.learnomate.org

A Steps towards the bright future

Zones
▶ Exploring GCP Zones
▶ Distributing Resources Across Zones
▶ High Availability and Disaster Recovery Considerations

Importance
▶ Significance of Choosing the Right Location
▶ Global vs. Regional Resources
▶ Factors Influencing Location Decisions

Module 4: GCP IAM & Admin

Identities
▶ Introduction to Identity and Access Management (IAM)
▶ Users, Groups, and Service Accounts
▶ Best Practices for Identity Management

Roles
▶ GCP IAM Roles Overview
▶ Defining Custom Roles
▶ Role-Based Access Control (RBAC) Implementation

Policy
▶ Resource-based Policies
▶ Understanding and Implementing Organization Policies
▶ Auditing and Monitoring Policies

Resource Hierarchy
▶ GCP Resource Hierarchy Structure
▶ Managing Resources in a Hierarchy
▶ Organizational Structure Best Practices

+91 7757062955,+91 8983069523 info@learnomate.org

 www.learnomate.org

A Steps towards the bright future

Module 5 :Linux Basics
Overview of Linux
Basic Command Line Interface (CLI)
- Navigation: ls, cd, pwd
- File operations: cp, mv, rm, mkdir, rmdir
- Viewing file contents: cat, less, more, head, tail
GCP service-related commands

Module 6: Python for Data Engineer

CHAPTER 1 - Python Basics
Strings
Operators
Numbers (Int, Float)
Booleans

CHAPTER 2 - Data Types & Data Structures

Lists
Tuple
Dictionary
Sets

CHAPTER 3 - Python Programming Constructs

if, elif, else statements
for loops, while loops
Exception Handling
File I/O operations

CHAPTER 4- Modular Programming in Python

Functions
Lambda Functions and Classes

+91 7757062955, +91 8983069523 info@learnomate.org

 www.learnomate.org

A Steps towards the bright future

Module 7 : Google Cloud Storage
Introduction to Cloud Storage
▶Overview of Cloud Storage as a scalable and durable object storage service.
▶Understanding buckets and objects in Cloud Storage.
▶Use cases for Cloud Storage, such as data backup, multimedia storage, and
website content delivery.

Cloud Storage Operations

▶Creating and managing Cloud Storage buckets.
▶Uploading and downloading objects to and from Cloud Storage.
▶Setting access controls and permissions for buckets and objects.

Data Transfer and Lifecycle Management

▶Strategies for efficient data transfer to and from Cloud Storage.
▶Implementing data lifecycle policies for automatic object deletion or archival.
▶Utilizing Transfer Service for large-scale data transfers.

Versioning and Object Versioning

▶Enabling and managing versioning for Cloud Storage buckets.
▶Understanding how object versioning works.
▶Use cases for object versioning in data resilience and recovery.

Integration with Other GCP Services

▶Integrating Cloud Storage with BigQuery for data analytics.
▶Using Cloud Storage as a data source for Dataflow and Dataproc.
▶Exploring options for serving static content on websites.

+91 7757062955, +91 8983069523 info@learnomate.org

 www.learnomate.org

A Steps towards the bright future

Module 8 : Cloud SQL
Introduction to Cloud SQL
Overview of Cloud SQL as a fully managed relational database service.
Supported database engines and use cases for Cloud SQL.

Creating and Managing Cloud SQL Instances

Creating MySQL or PostgreSQL instances.
Configuring database settings, users, and access controls.
Importing and exporting data in Cloud SQL.

Backups and High Availability

Configuring automated backups and performing manual backups.
Implementing high availability with failover replicas.
Strategies for restoring data from backups.

Scaling and Performance Optimization

Vertical and horizontal scaling options in Cloud SQL.
Performance optimization tips for database queries.
Monitoring and troubleshooting database performance.

Integration with Other GCP Services

Connecting Cloud SQL with App Engine, Compute Engine, and Kubernetes Engine.
Using Cloud SQL as a backend database for applications.
Data synchronization with Cloud Storage and BigQuery.

End to End Database migration Project

Offline: Export and Import method
Online: DMS method

+91 7757062955, +91 8983069523 info@learnomate.org

 www.learnomate.org

A Steps towards the bright future

Module 9 : BigQuery (SQL development)
Introduction to BigQuery
Overview of BigQuery as a fully managed, serverless data warehouse.
Use cases for BigQuery in business intelligence and analytics.
Various method of creating table in BigQuery
BigQuery Data Sources and File Formats
Native table and External Tables

SQL Queries and Performance Optimization

Writing and optimizing SQL queries in BigQuery.
Understanding query execution plans and best practices.
Partitioning and clustering tables for performance.

Data Integration and Export

Loading data into BigQuery from Cloud Storage, Cloud SQL, and other sources.
Exporting data from BigQuery to various formats.
Real-time data streaming into BigQuery.

Configuring access controls and permissions in BigQuery.

Implementing encryption for data in BigQuery.
Auditing and monitoring for security compliance.

BigQuery Views
Views, Materialized Views, Authorized Views

Integration with Other GCP Services

Integrating BigQuery with Dataflow for ETL processes.
Using BigQuery in conjunction with Data Studio for visualization.
Building data pipelines with BigQuery and Composer.

Case Study-1: Spotify

Case Study-2: Social Media

+91 7757062955, +91 8983069523 info@learnomate.org

 www.learnomate.org

A Steps towards the bright future

Module 10 : DataProc (Pyspark Development)
Introduction to DataProc
Introduction to Hadoop and Apache Spark
Understanding the difference between Spark and MapReduce
What is Spark and Pyspark.
Understanding Spark framework and its functionalities
Overview of DataProc as a fully managed Apache Spark and Hadoop service.
Use cases for DataProc in data processing and analytics.

Cluster Creation and Configuration

Creating and managing DataProc clusters.
Configuring cluster properties for performance and scalability.
Preemptible instances and cost optimization.

Running Jobs on DataProc

Submitting and monitoring Spark and Hadoop jobs on DataProc.
Use of initialization actions and custom scripts.
Job debugging and troubleshooting.

Integration with Storage and BigQuery

Reading and writing data from/to Cloud Storage and BigQuery.
Integrating DataProc with other storage solutions.
Performance optimization for data access.

Scaling and Automation

Autoscaling DataProc clusters based on workload.
Using Dataprep or other tools for data preparation before processing.
Automation and scheduling of recurring jobs.

Case Study-1: Data Cleaning of Employee Travel Records

End to End Batch Pyspark pipeline using Dataproc, BigQuery, GCS

+91 7757062955, +91 8983069523 info@learnomate.org

 www.learnomate.org

A Steps towards the bright future

Module 11 : DataFlow (Apache Beam development)
Introduction to DataFlow
Use cases for DataFlow in real-time analytics and ETL.
Understanding the difference between Apache Spark and Apache Beam
How Dataflow is different from Dataproc

Building Data Pipelines with Apache Beam

Writing Apache Beam pipelines for batch and stream processing.
Custom Pipelines and Pre-defined pipelines
Transformations and windowing concepts.

Integration with Other GCP Services

Integrating DataFlow with BigQuery, Pub/Sub, and other GCP services.
Real-time analytics and visualization using DataFlow and BigQuery.
Workflow orchestration with Composer.

End to End Streaming Pipeline using Apache beam with Dataflow, Python app, PubSub,
BigQuery, GCS
Template method of creating pipelines

Module 12 : Cloud Pub/Sub

Introduction to Pub/Sub
Understanding the role of Pub/Sub in event-driven architectures.
Key Pub/Sub concepts: topics, subscriptions, messages, and acknowledgments.

Creating and Managing Topics and Subscriptions

Using the GCP Console to create Pub/Sub topics and subscriptions.
Configuring message retention policies and acknowledgment settings.

Publishing and Consuming Messages

Writing and deploying code to publish messages to a topic.
Implementing subscribers to consume and process messages from subscriptions.

+91 7757062955, +91 8983069523 info@learnomate.org

 www.learnomate.org

A Steps towards the bright future

Integration with Other GCP Services

Connecting Pub/Sub with Cloud Functions for serverless event-driven computing.
Integrating Pub/Sub with Dataflow for real-time stream processing.

Streaming use-case using Dataflow

Module 13 : Cloud Composer (DAG Creations)

Introduction to Composer
Overview of Airflow Architecture
Use cases for Composer in managing and scheduling workflows.

Creating and Managing Workflows

Creating and configuring Composer environments.
Defining and scheduling workflows using Apache Airflow.
Monitoring and managing workflow executions.

Integration with Data Engineering Services

Orchestrating workflows involving BigQuery, DataFlow, and other services.
Coordinating ETL processes with Composer.
Integrating with external systems and APIs.

Error Handling and Troubleshooting

Handling errors and retries in Composer workflows.
Debugging and troubleshooting failed workflow executions.
Logging and monitoring for Composer workflows

Level-1-DAG: Orchestrating the BigQuery pipelines

Level-2-DAG: Orchestrating the DataProc pipelines
Level-3-DAG: Orchestrating the Dataflow pipelines
Implementing CI/CD in Composer Using Cloud Build and GitHub

+91 7757062955, +91 8983069523 info@learnomate.org

 www.learnomate.org

A Steps towards the bright future

Module 14: Cloud Functions
Cloud Functions Introduction

Setting up Cloud Functions in GCP

Event-driven architecture and use cases

Writing and deploying Cloud Functions

Triggering Cloud Functions:

HTTP triggers
Pub/Sub triggers
Cloud Storage triggers

Monitoring and logging Cloud Functions

Usecase-1: Loading the files from GCS to BigQuery as soon as it is uploaded.

+91 7757062955, +91 8983069523 info@learnomate.org

 www.learnomate.org

A Steps towards the bright future

Azur Data Engineering Tools

Module 15 : Azur Introduction

Introduction Azure Platform and Overview

Multiple Interaction Methods

Console
Shell
SDK

Azure Account Creation

Service Overview, subscription and Resource Container

Azure DataLake Storage (ADLS)

Introduction to Azure Data Lake Storage (ADLS)
Core Components of ADLS Gen2
Blobs, Containers, and Storage Accounts
Hierarchical Namespace o Why Use Azure Data Lake Storage?
Scalability, Cost-Effectiveness, Security
Integration with Analytics Services

Access and Management of ADLS

Data Management via Console, SDK, CLI, REST API
Role-Based Access Control (RBAC) and Access Control Lists (ACLs)

Pricing Overview
Pricing Tiers (Hot, Cool, Archive)
Cost Factors

Best Practices for ADLS

Data Partitioning
Tiering Strategy for Cost Optimization
Monitoring and Usage Tracking

+91 7757062955, +91 8983069523 info@learnomate.org

 www.learnomate.org

A Steps towards the bright future

Interactive Access Methods
Managing ADLS via Console
CLI/Shell Interaction
Programmatic Access via SDKs

Azure SQL DB
Introduction to Azure SQL Database
Key Features
Creating a SQL db for practice

Module 16 : AzurData Factory

What is ADF and Use cases

Core components of ADF and understanding & configuring

Integration runtime
Linked Services
Datasets
Pipelines
Activities
Triggers

Key Features of Azure Data Factory

Pipelines in ADF
Creating and Configuring Pipelines
Activities and Triggers
Monitoring and Managing Pipeline Runs
Orchestration

Use case: End to End Incremental pipeline with ADF, ADLS, SQL DB.

Best Practices for ADF

Designing Efficient Pipelines
Data Partitioning and Parallelism

Monitoring and Optimization for Performance

+91 7757062955, +91 8983069523 info@learnomate.org

 www.learnomate.org

A Steps towards the bright future

Module 17 : Databricks on Azure/GCP

Introduction to Databricks Lakehouse Platform

Databricks Architecture and Core Components

Explore the architecture, including clusters, jobs, and the workspace environment.
Key components like Databricks Runtime, Spark, and Delta Lake.

Setting Up and Administering a Databricks Workspace

Step-by-step guide on workspace creation and management.
Admin tasks including user management, permissions, and billing.

Databricks Unity Catalog

Unified governance solution for managing data and metadata across the lakehouse.
Organizing data assets and enforcing security policies.

Managing Data with Delta Lake

Understanding Delta Lake’s ACID transactions and versioning capabilities.
Best practices for efficient data management.
Creating Delta Lake Tables
Querying and managing data across multiple file formats.

Working with Notebooks and Clusters

Collaborative data science using Databricks notebooks.
Cluster setup, management, and autoscaling.

Building ELT Pipelines with Spark SQL and Python

Performance Optimization in Databricks
Incremental Data Processing with Delta Lake
Real-time data processing with Spark Structured streaming
Databricks Autoloader for file-based ingestion utility

Case Study: Creating an End-to-End Incremental Workflow

+91 7757062955, +91 8983069523 info@learnomate.org

 www.learnomate.org

A Steps towards the bright future

By the End of the course What Students can Expect

Proficient in SQL Development:

Mastering SQL for querying and manipulating data within Google BigQuery and Cloud
SQL.
Writing complex queries and optimizing performance for large-scale datasets.
Understanding schema design and best practices for efficient data storage.

Pyspark Development Skills:

Proficiency in using PySpark for large-scale data processing on Google Cloud.
Developing and optimizing Spark jobs for distributed data processing.
Understanding Spark's RDDs, Data Frames, and transformations for data manipulation.

Apache Beam Development Mastery:

Creating data processing pipelines using Apache Beam.
Understanding the concepts of parallel processing and data parallelism.
Implementing transformations and integrating with other GCP services.

DAG Creations with Cloud Composer:

Designing and implementing Directed Acyclic Graphs (DAGs) for orchestrating
workflows.
Using Cloud Composer for workflow automation and managing dependencies.
Developing DAGs that integrate various GCP services for end-to-end data processing.

Architecture Planning:
Proficient in architecting end-to-end data solutions on GCP and Azure.
Understanding the principles of designing scalable, reliable, and cost-effective data
architectures.

Certification Readiness
Prepare for the Google Cloud Professional Data Engineer (PDE) and
Associate Cloud Engineer (ACE) certifications through a combination of theoretical
knowledge and hands-on experience.

+91 7757062955, +91 8983069523 info@learnomate.org

 www.learnomate.org

A Steps towards the bright future

Contact Details

If you required any further information, please fill free to contact us.

Learnomate Technologies Pvt. Ltd

Main Branch:
(Sai Luxuria, Office No 15, 3rd Floor,Bhumkar Chowk,
Wakad, Pune, Maharashtra, 411057 India)

Contact Details:

Call/WhatsApp: +91 7757062955

+91 7822917585
Email: info@learnomate.org
---------------------------------------------------------------------------------------
Kalewadi Branch.
Office no.216, Solitaire business hub, 2nd floor, Kaspate Wasti, Wakad, Pune,
Maharashtra 411057

Contact Details:

Call/WhatsApp: +91 8983069523

Email: info@learnomate.org
------------------------------------------------------------------------------------------------

+91 7757062955, +91 8983069523 info@learnomate.org

 www.learnomate.org

A Steps towards the bright future

THANK YOU

https://www.youtube.com/@learnomate

https://www.linkedin.com/company/learnom
atetechnologies/

https://www.facebook.com/learnomate

https://www.instagram.com/learnomate/

+91 7757062955, +91 8983069523 info@learnomate.org