Cloud Computing

Virtualization

It refers to the process of creating virtual representation of various

computing resources.

It assigns a logical name for a physical resource and then provides a pointer
to that physical resource when a request is made.
When an application makes a request to the virtual resource using its logical
name, the virtualization layer translates this request by looking up the
mapping to the corresponding physical resource and provides a pointer to that
physical resource for processing the request.

This mapping is dynamic and effortless as per rapidly changing conditions.

Types of Virtualization
Access: A client can request access to a cloud service from any location
using any device with Internet connectivity.
Application: Cloud environment hosts multiple instances of same application
and requests are directed to a specific instance based on predefined
conditions/routing algorithms.
These predefined conditions could be workload, availability, or proximity
to users.
This type of virtualization enhances scalability, performance, and fault
tolerance.

CPU: It involves partitioning physical computers into multiple VMs, each

running its own OS and applications.
This allows for better resource utilization by combining multiple
workloads onto single physical computer, leading to cost savings and
improved efficiency.
 CPU Virtualization could be achieved via load balancing techniques, where
computational tasks are distributed across multiple physical CPUs to
optimize performance and reliability

Storage: It abstracts physical storage devices and presents them as a

unified virtual resource pool.
Data is stored across multiple storage devices, and virtualization
techniques manage data placement, replication, and access to ensure high
availability, scalability, and performance.
It includes features such as de-duplication, compression, and replication
for redundancy and disaster recovery process.
 Load Balancing

It is an optimization technique that forms a crucial aspect of managing the

distribution of service requests to the available resources.

It ensures that the workload is evenly distributed across multiple servers or

resources to optimize performance, maximize resource utilization, and
prevent any single server from becoming overloaded.

It is used to increase utilization and throughput, lower latency, reduce

response time, and avoid system overload.
It converts Unreliable system into reliable one through managed redirection
and redundancy.

It also provides fault tolerance when coupled with a failover mechanism.

It uses popular round robin, weighted round robin, fastest response time,
least connections and weighted least connections Load balancing algorithms
when a service request arrives.

A session ticket is created to direct all related traffic from client of a

particular session, so that proper routing could be achieved.
Without session ticket a load balancer would not be able to correctly failover
a request from source to destination

The session ticket could be created using session data stored in database, or
use client’s browser to store client side cookie, or use rewrite engine that
modifies URL.

Out of all these methods session cookie method has least amount of overhead
as it allows load balancer an independent selection of resources.
The objectives of load balancing in cloud computing are as follows:
Resource Utilization Optimization:
Optimize the utilization of computing resources such as virtual machines,
servers, and network bandwidth.

By distributing workloads evenly across available resources, thus

maximizing resource efficiency.

Scalability and Elasticity:

Dynamically allocating resources in response to changing workload demands.
 As the demand for computing resources fluctuates, load balancing
algorithms automatically scale resources up or down to accommodate
varying workloads, ensuring consistent performance and responsiveness.

High Availability and Fault Tolerance:

By distributing workloads across multiple redundant resources, load
balancing minimizes the impact of failures or hardware malfunctions on the
overall system performance.

In case of a failure or degradation of a resource, load balancing algorithms

redirect traffic to healthy resources, thereby maintaining uninterrupted
service availability.
Improved Performance and Response Time:
Load balancing aims to minimize response time and latency by directing
incoming requests to the least busy or closest resources.

By distributing workloads efficiently, load balancing algorithms reduce

congestion and bottlenecks, leading to faster processing times and
improved performance for end-users.

Cost Optimization:
Optimize costs by enabling efficient resource utilization and avoiding
unnecessary resource provisioning.
 By dynamically allocating resources based on workload demands, load
balancing reduces the need for over-provisioning and minimizes idle
resources, leading to cost savings for cloud service providers and users.

Traffic Management and Quality of Service (QoS):

Load balancing facilitates effective traffic management by directing
incoming requests to the most appropriate resources based on factors such
as server load, network conditions, and geographic location.

By ensuring equitable distribution of resources and prioritizing critical

workloads, load balancing helps maintain consistent Quality of Service
(QoS) levels for different types of applications and users.
 Types of load balancing
Static Load Balancing:
Distributing incoming requests or workloads based on predefined rules or
algorithms.

Resources are allocated statically, without considering real-time resource

utilization or workload characteristics.

While simple to implement, static load balancing may lead to uneven

resource utilization and inefficient allocation under dynamic workload
conditions.
Dynamic Load Balancing:
Dynamic load balancing adjusts resource allocation in real-time based on
changing workload conditions, resource availability, and performance
metrics.

Dynamic load balancers continuously monitor resource utilization, network

traffic, response times, and other metrics to make informed decisions
about workload distribution.

Dynamic load balancing ensures better resource utilization, scalability, and

responsiveness compared to static approaches, especially in highly dynamic
and heterogeneous cloud environments.
Global Load Balancing:
Global load balancing distributes incoming traffic across multiple
geographically distributed data centers or cloud regions to improve
availability, latency, and disaster recovery capabilities.

Global load balancers use DNS-based or Anycast-based routing techniques

to direct users to the nearest or least loaded data center based on their
geographic location or network conditions.

By distributing traffic across multiple regions, global load balancing

enhances fault tolerance, minimizes latency, and optimizes performance for
users accessing cloud services from different locations.
Layer 4 Load Balancing:
Layer 4 load balancing operates at the transport layer (TCP/UDP) of the
OSI model and forwards incoming traffic based on network-level
information such as IP addresses and port numbers.

Layer 4 load balancers distribute traffic among backend servers.

Layer 4 load balancers offer high performance and scalability but lack
application-awareness and content-based routing capabilities.
Layer 7 Load Balancing:
Layer 7 load balancing operates at the application layer (HTTP/HTTPS) of
the OSI model and can make routing decisions based on application-specific
parameters such as URL, HTTP headers, cookies, and payload content.

Layer 7 load balancers perform advanced content-based routing, SSL

termination, session persistence, and application-level health checks.

Layer 7 load balancers provide more granular control over traffic routing
and can optimize application performance, security, and user experience.
Each type of load balancing has its advantages and limitations, and the choice
depends on factors such as the nature of workloads, scalability requirements,
performance goals, budget constraints, and deployment environment.

Many cloud providers offer load balancing services that integrate different
types of load balancing to meet various customer needs.
 Load balancing algorithms
Least Connection:
The Least Connection algorithm directs incoming requests to the server
with the fewest active connections or sessions.

By dynamically routing traffic to the least loaded server, Least Connection

helps distribute the workload evenly and prevents overloading of individual
servers.

This algorithm is particularly effective in scenarios where the workload per

connection varies significantly or where connections have long durations.
Least Response Time:
The response time is the total time that the server takes to process the
incoming requests and send a response.

This algorithm aims to minimize user perceived latency and improve overall
application performance.

It is suitable for latency-sensitive applications or scenarios where response

time variations are significant across servers.
Weighted Round Robin:
Weighted Round Robin extends the basic Round Robin algorithm by
assigning different weights or priorities to each server based on its
capacity or processing power.

Servers with higher capacities are assigned higher weights, allowing them
to handle a greater proportion of incoming requests.

Weighted Round Robin enables administrators to balance workload

distribution according to server capabilities and optimize resource
utilization accordingly.
Weighted Least connection:
Weighted least connection extends the least connection algorithm.

It takes into account differing application server characteristics (power

and connections) by assigning weights to each server.

These weights are based on the relative processing power and available
resources of each server.

For example, a server with more processing power might be assigned a

higher weight compared to a server with lesser resources.
 Load balancing decisions are based on both active connections and the
assigned server weights.

If there are multiple servers with the lowest number of connections, the
server with the highest weight is preferred for load balancing.

Weighted least connections load balancing is suitable for server pools

where the servers are not identical.

Cloud providers often offer load balancing services with built-in support for
these algorithms, allowing users to configure and customize load balancing
behavior according to their needs.
 Load balancing architecture
This architecture typically involves the following key components:
Load Balancer:
The load balancer is a central component responsible for receiving incoming
requests from clients and distributing them.

Load balancers can operate at different layers of the OSI model,

depending on the level of traffic.

Backend Servers or Virtual Machines:

Backend servers or virtual machines are the computing resources that
actually process incoming requests forwarded by the load balancer.
Health Check Mechanism:
A health check mechanism is employed to monitor the health and
availability of backend servers in real-time.

Health checks periodically assess the responsiveness and availability of

servers by sending probe requests and analyzing responses.

If a server is detected as unhealthy or unresponsive, the load balancer

automatically removes it from the pool of available servers to prevent it
from receiving new requests until it becomes healthy again.

Load Balancing Algorithms

 Types of Load Balancing Architecture
Centralized Architecture:
In a centralized load balancing architecture, a single load balancer or a
cluster of load balancers sits at a central point in the network, receiving all
incoming traffic and distributing it to backend servers.

All traffic passes through the central load balancer(s)

Backend servers are typically homogeneous and centrally managed.

Configuration changes and updates to load balancing policies are made

centrally.
Monitoring and health checks are often performed centrally, with the load
balancer(s) responsible for detecting server failures and adjusting traffic
accordingly.

Simplified management and configuration.

Easier to implement and maintain in smaller environments.

Single point of failure

Scalability limitations: Struggle to handle large volumes of traffic or

accommodate rapid changes in demand.
Distributed Load Balancing Architecture:
In a distributed load balancing architecture, load balancing functionality is
distributed across multiple nodes or components in the network.

Load balancing functionality may be integrated into individual servers, edge

devices, or software-defined networking (SDN) controllers.

Traffic is distributed among multiple load balancers or routing nodes, often

based on proximity or network topology.

Backend servers may be heterogeneous and distributed across multiple

data centers or cloud regions.
Decentralized monitoring and health checking, with each load balancer
responsible for a subset of servers.

Improved fault tolerance and resilience: Distributed architectures are less

susceptible to single points of failure.

Scalability: Can scale horizontally by adding more load balancing nodes.

Geographic distribution: Can route traffic to the nearest server or data

center, reducing latency.

Complexity: Managing and coordinating distributed load balancers may

introduce complexity in configuration and troubleshooting.
 Synchronization: Ensuring consistency and synchronization among
distributed components can be challenging.

Dynamic Load Balancing Architecture:

Dynamic load balancing architectures focus on adapting to changing
conditions and optimizing resource usage in real-time.

Load balancing decisions are based on real-time metrics such as server

load, response times, and network conditions.

Adaptive algorithms dynamically adjust routing decisions based on changing

conditions, traffic patterns, and application requirements.
Autonomic computing principles may be employed to enable
self-configuration, self-optimization, and self-healing capabilities.

Optimized resource utilization: Dynamic architectures can efficiently

distribute traffic based on current conditions, maximizing performance and
minimizing response times.

Agility and responsiveness: Systems can adapt to fluctuations in demand,

infrastructure failures, and other dynamic factors.

Improved user experience: Dynamic load balancing can prioritize critical

applications or users based on changing priorities or policies.
Complexity: Implementing dynamic load balancing algorithms and
mechanisms may require sophisticated monitoring, analytics, and
automation.

Overhead: Continuous monitoring and adaptation may introduce additional

computational overhead and network traffic.
 Performance metric and benchmarks
These metrics provide insights into various aspects of load balancer
performance.
Response Time:
Response time is the duration between sending a request to a server and
receiving the corresponding response.

Metric: Average response time, measured in milliseconds (ms) or seconds

(s).

Benchmark: Lower response time indicates better performance.

Benchmarks vary depending on application requirements, but typically aim
for sub-second response times.
Throughput:
Throughput is the rate at which tasks or requests are processed by the
system over a period of time.

Metric: Requests per second (RPS) or transactions per second (TPS).

Benchmark: Higher throughput indicates better performance. Benchmarks

depend on the system's capacity and workload characteristics but aim for
maximizing the number of requests processed per unit time.
Latency:
Latency is the delay incurred when transferring data between a client and a
server.

Metric: Round-trip latency, measured in milliseconds (ms).

Benchmark: Lower latency indicates better performance. Benchmarks aim

for minimizing the delay between request and response, especially for
real-time or latency-sensitive applications.
Server Utilization:
Server utilization measures the percentage of a server's capacity that is
being utilized to process requests.

Metric: CPU utilization, memory utilization, network bandwidth utilization.

Benchmark: Optimal server utilization balances resource usage without

overloading servers. Benchmarks aim for maintaining utilization levels below
capacity to prevent performance degradation and ensure scalability.
Scalability:
Scalability refers to the ability of the load balancing system to handle
increasing workload and resource demands by adding more resources or
nodes.

Metric: Scalability factor or scalability ratio (e.g., the ratio of

performance improvement to the increase in resources).

Benchmark: Scalability benchmarks measure how effectively the load

balancing system scales with increasing workload or resources. Benchmarks
aim for linear or near-linear scalability to ensure efficient resource
allocation and performance as the system grows.
Fault Tolerance:
Fault tolerance measures the system's ability to continue operating in the
presence of failures or errors, including server failures, network failures,
or load balancer failures.

Metric: Mean Time Between Failures (MTBF), Mean Time to Recover

(MTTR), availability percentage.

Benchmark: High fault tolerance ensures minimal service disruption and

downtime in the event of failures. Benchmarks aim for maximizing
availability and minimizing recovery time, often measured in "nines" (e.g.,
99.99% uptime).
Session Persistence:
Session persistence ensures that subsequent requests from the same client
are consistently routed to the same backend server to maintain session
state.

Metric: Session stickiness or session affinity.

Benchmark: Session persistence benchmarks measure the effectiveness of

maintaining session continuity across requests. Benchmarks aim for
ensuring seamless user experience and preventing session-related issues
such as data loss or session timeout.
Adaptability:
Adaptability measures the load balancing system's ability to dynamically
adjust to changing conditions, such as fluctuating workloads, resource
availability, or network conditions.

Metric: Adaptive response time, dynamic load distribution.

Benchmark: Effective adaptability ensures optimal performance and

resource utilization under varying conditions. Benchmarks aim for rapid and
accurate adaptation to changes in workload or environment.
Algorithm Overhead:
Algorithm overhead refers to the additional computational or processing
resources consumed by the load balancing algorithm itself.

Metric: CPU overhead, memory overhead, network overhead.

Benchmark: Lower algorithm overhead indicates better efficiency.

Benchmarks aim for minimizing overhead to maximize resource availability
for processing user requests.
Energy Efficiency:
Energy efficiency measures the amount of energy consumed by the load
balancing system to perform its operations.

Metric: Energy consumption, power usage effectiveness (PUE).

Benchmark: Higher energy efficiency reduces operational costs and

environmental impact. Benchmarks aim for optimizing energy usage while
maintaining performance and reliability.
 Workload Managers
These are sophisticated load balancers; they actively manage the distribution
of tasks based on various factors such as resource utilization, response time,
work queue length, connection latency and capacity.

Key features of workload managers:

Health Monitoring and Dynamic Scaling: Checks if the resource in the pool
is available and responsive. If resource turns unhealthy or overloaded, it
dynamically removes such resources and bring standby resources online to
maintain optimal performance and availability.
Priority Activation and Asymmetric Loading: Priority activation allows load
balancers to prioritize certain servers or resources over others, ensuring
critical tasks are handled properly. Asymmetric loading involves assigning
different workloads to resources based on their capacity, optimizing
resource utilization.

Traffic Optimization: HTTP traffic compression along with buffering are

carried out to reduce bandwidth usage, improve website performance,
improves efficiency and reduces latency.
Security and Authentication: Decryption of SSL traffic, access control
policies, authentication of users, filtering malicious traffic etc. are
performed by workload managers to enhance security and reduce server
load.

Packet Shaping and Content Filtering: Network traffic is shaped by

prioritizing or throttling packets based on predefined rules. Content
filtering capabilities enable them to inspect and filter traffic based on
content, allowing for better resource allocation and protection against
malicious attacks.
 What does a Virtual Machine Monitor (VMM) do?
A VMM essentially multiplexes multiple virtual machines (VM) on the same
physical hardware.

It's akin to how an operating system (OS) handles multiple processes on a

CPU.

The VMM switches between running these VM, saving their contexts, and
then switching to others as needed, much like an OS switch between
processes.

However, there are challenges in achieving this.

Unlike regular processes, which are designed to only perform unprivileged
operations and rely on the OS for privileged tasks (Tasks like installing
device drivers, modifying system files), a guest OS behaves differently.

When running inside a VM, a guest OS expects full access to hardware and
the ability to execute privileged instructions independently.

Additionally, since VMs should be isolated from each other, the VMM needs
to ensure they share resources safely, including hardware.

Hence, it becomes a bit more challenging to configure the OS to operate in

this manner.
One common method used to design VMMs is called "trap and emulate."

This technique takes advantage of the multiple privilege levels present in

CPUs.

For instance, CPUs like x86 have different privilege levels, or "rings,"
typically four of them.
In this setup, user processes operate in the least privileged ring (ring
three), while the operating system resides in the most privileged ring (ring
zero), where it executes privileged instructions.

The guest OS applications run in ring three, just like regular user
processes.

The VMM and the host OS, however, operate in ring zero, providing them
with the highest level of privilege.

To ensure security, the guest OS operates in an intermediate ring, such as

ring one.
This arrangement allows the guest OS to be more privileged than regular
user processes but less privileged than the VMM.

In essence, the fundamental concept of a "trap and emulate" VM is that

the guest OS runs at a lower privilege level than the VMM.

Whenever the guest OS attempts a privileged operation, it "traps" to the

VMM, similar to how a user process traps to the OS for privileged tasks.
 How does "trap and emulate" actually function?
Imagine a scenario where the guest OS needs to make a system call, handle
an interrupt, or perform any other privileged action.

Typically, such actions would be directed to the guest OS.

However, in the case of "trap and emulate," where the guest OS operates
at a lower privilege level, these actions are redirected to the VMM instead.

When an interrupt or privileged action occurs, instead of going directly to

the guest OS, it is intercepted by the VMM.
The VMM, being in a higher privilege level (usually ring zero), then
redirects the action to the guest OS's trap handling code.

The guest OS, equipped with functions to handle such traps, processes the
action as if it were a regular system call.

After handling the trap, if the guest OS needs to return to the user
application, it executes a privileged instruction like Iret.

This instruction also traps to the VMM, which then knows to direct the
execution flow back to the guest user code.
The underlying principle is straightforward: whenever the guest OS needs
to perform a privileged action, it traps to the VMM.

The VMM then handles the action on behalf of the guest OS, whether it
involves returning to the user process or handling input/output operations.

This way, sensitive operations such as managing data structures or

accessing CPU registers are managed by the VMM in collaboration with the
guest OS, ensuring security and proper functioning within the virtualized
environment.
In essence, the VMM acts as an intermediary between the user application
and the guest OS, ensuring smooth and secure operation within the
virtualized environment.

What are the problems with trap and emulate?

The trap and emulate technique, while useful for virtualization,
encounters several challenges.

One major issue is that the guest OS may detect that it's operating at a
lower privilege level than expected.
Guest OS are typically designed to operate at the highest privilege level
available.

This expectation aligns with the general assumption that an OS should

function with the highest level of privileges.

Consequently, if a guest OS detects that it's not running at the highest

privilege level, it may choose to terminate its operations to prevent
potential issues or security vulnerabilities.

This can lead to unexpected behavior or even crashes.

For instance, on x86 architecture, the guest OS can check its privilege
level through certain CPU registers.

Typically, OSes are designed to run at the highest privilege level, so

running at a lower level violates this assumption.

A major issue arises with certain x86 instructions when they are
executed at a lower privilege level.

These instructions, known as sensitive instructions, alter hardware

settings but can function in both privileged and unprivileged modes.
Typically, sensitive instructions that alter hardware are expected to be
privileged, allowing them to trigger a trap to the Virtual Machine Monitor
(VMM) for emulation.

However, some x86 instructions can function equally well in both

privileged and unprivileged modes.

Consequently, when a guest OS operates in ring zero, these instructions

behave as expected.

However, when the guest OS operates in ring one, they execute without
trapping to the VMM.
During the development of the x86 instruction set architecture,
virtualization was not a primary consideration.

It was widely assumed that operating systems would always run at the
highest privilege level, rendering these corner cases inconsequential.

As a result, the potential risks associated with allowing sensitive

instructions to operate in unprivileged modes were not thoroughly
considered.

For example, consider the "popf" instruction in x86, which writes values
from the stack into CPU registers like "eflags".
When executed in ring zero (privileged mode), all flags are correctly set.
However, in ring one (unprivileged mode), only accessible flags are set,
omitting crucial ones like the interrupt flag.

Consequently, running the OS in ring one can lead to incorrect behavior

due to these sensitive instructions not trapping as expected.

A key concept in addressing these challenges is the Popek Goldberg

theorem, which states that for efficient trap and emulate-based
virtualization, sensitive instructions should always be privileged.
Ideally, sensitive instructions should always trigger a trap to the VMM,
allowing the VMM to handle the privileged operation by emulating it.

This ensures proper control and security in a virtualized environment.

In an optimal scenario, sensitive instructions would be a subset of

privileged instructions.

However, the x86 architecture deviates from this ideal.

The set of sensitive instructions in x86 is not strictly a subset of

privileged instructions.
This misalignment poses challenges in implementing a
trap-and-emulate-based VM for x86.

Consequently, various techniques have been proposed to address this

discrepancy and enable effective virtualization of x86 systems.
 What are these techniques?
Paravirtualization: This involves modifying the guest OS code to remove
privileged operations and make the OS aware of virtualization.

Instead of invoking privileged operations directly, the guest OS makes

hyper calls to the VMM for such actions.

While easy to implement, it requires changes to the OS source code,

making it incompatible with generic unmodified OS kernels.

Full Virtualization: Unlike paravirtualization, full virtualization doesn't require

changes to the OS source code.
Instead, it involves translating CPU instructions dynamically to handle
sensitive but unprivileged instructions by trapping them to the VMM.

This allows existing OS binaries to be run in a virtualized environment

without modification.

Although it incurs higher overhead due to dynamic translation, it offers

greater compatibility with existing OS images.

So, this technique was first pioneered by VMware in their VMware

workstation, and it is the most common technique used today.
Hardware-Assisted Virtualization: This technique leverages hardware support
from modern CPUs to facilitate virtualization.

CPUs equipped with hardware virtualization support have a special

execution mode, allowing the guest OS to run directly in this mode without
needing to operate at a lower privilege level like in paravirtualization.

So, what exactly is this VMX mode of execution? Well, in x86 architecture,
there are typically four privilege levels, known as rings, in the regular
non-VMX mode, also called root mode.

Additionally, there exists another set of four rings in a special VMX mode
for virtualization.
In this setup, the guest OS operates at ring zero within this special VMX
mode, while the guest applications run at ring three.

This arrangement eliminates the need to run the guest OS at ring one, thus
avoiding potential issues encountered previously.

But how does the VMM maintain control? The VMM operates at ring zero in
the non-VMX mode, also known as the root mode of the CPU.

When the VMM needs to execute a guest OS, it switches to the VMX mode
and runs the guest OS at ring zero within this special mode.
However, it's essential to note that this ring zero in VMX mode isn't as
powerful as the regular ring zero.

The VMM can configure specific points at which the guest OS must trap
back into the VMM, allowing it to maintain some control.

For instance, certain privileged actions by the guest OS can be configured

to trigger traps back to the regular ring zero, enabling the VMM to take
appropriate actions.
Thus, while the guest OS operates happily at ring zero, the VMM retains
control over its execution, ensuring that the guest OS doesn't have full
access or the ability to perform all privileged operations on the hardware.

This approach offers efficient virtualization but requires hardware

support, which is commonly available in modern CPUs.

This is what is used by the KVM QEMU hypervisor in Linux.