Outline

• Cloud Computing definition

• Characteristics of cloud computing
• Cloud deployment models
• Cloud service models
• Cloud Services
• Cloud Applications
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Chapter 1
Introduction & Concepts
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
 Characteristics of Cloud Computing
• On-demand self service:
• Cloud computing resources can be provisioned on-demand by the users,
without requiring interactions with the cloud service provider. The process
of provisioning resources is automated.
• Broad network access:
• Cloud computing resources can be accessed over the network using standard access
mechanisms that provide platform-independent access through the use of
heterogeneous client platforms such as workstations, laptops, tablets and
smartphones.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Definition of Cloud Computing
The U.S. National Institute of Standards and Technology (NIST) defines
cloud computing as:
• Cloud computing is a model for enabling ubiquitous, convenient, on-demand
network access to a shared pool of configurable computing resources (e.g.,
networks, servers, storage, applications, and services) that can be rapidly
provisioned and released with minimal management effort or service
provider interaction.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
 Characteristics of Cloud Computing
• Measured service:
• Cloud computing resources are provided to users on a pay-per-use model.
The usage of the cloud resources is measured and the user is charged based
on some specific metric.
• Performance:
• Cloud computing provides improved performance for applications since the
resources available to the applications can be scaled up or down based on the
dynamic application workloads.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Characteristics of Cloud Computing
• Resource pooling:
• The computing and storage resources provided by cloud service providers are
pooled to serve multiple users using multi-tenancy. Multi-tenant aspects of
the cloud allow multiple users to be served by the same physical hardware.
• Rapid elasticity:
• Cloud computing resources can be provisioned rapidly and elastically. Cloud
resources can be rapidly scaled up or down based on demand.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
 Characteristics of Cloud Computing
• Reliability:
• Applications deployed in cloud computing environments generally have a
higher reliability since the underlying IT infrastructure is professionally
managed by the cloud service.
• Multi-tenancy:
• The multi-tenanted approach of the cloud allows multiple users to make use of the
same shared resources.
• In virtual multi-tenancy, computing and storage resources are shared among
multiple users.
• In organic multi-tenancy every component in the system architecture is shared
among multiple tenants
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Characteristics of Cloud Computing
• Reduced costs:
• Cloud computing provides cost benefits for applications as only as much
computing and storage resources as required can be provisioned dynamically,
and upfront investment in purchase of computing assets to cover worst case
requirements is avoid.
• Outsourced Management:
• Cloud computing allows the users (individuals, large organizations, small and
medium enterprises and governments) to outsource the IT infrastructure
requirements to external cloud providers.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Software-as-a-Service (SaaS)
• Software/Interface
• SaaS provides the users a complete software application or the user interface to the
application itself.
• Outsourced Management
• The cloud service provider manages the underlying cloud infrastructure including servers,
network, operating systems, storage and application software, and the user is unaware of
the underlying architecture of the cloud.
• Thin client interfaces
• Applications are provided to the user through a thin client interface (e.g., a browser). SaaS
applications are platform independent and can be accessed from various client devices such
as workstations, laptop, tablets and smartphones, running different operating systems.
• Ubiquitous Access
• Since the cloud service provider manages both the application and data, the users are able to
access the applications from anywhere.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Cloud Service Models
• Software as a Service (SaaS)
• Applications, management and user interfaces provided over a network
• Platform as a Service (PaaS)
• Application development frameworks, operating systems and deployment
frameworks
• Infrastructure as a Service (IaaS)
• Virtual computing, storage and network resource that can be provisioned on
demand
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Platform-as-a-Service (PaaS)
• Development & Deployment:
• PaaS provides the users the capability to develop and deploy application in
the cloud using the development tools, application programming interfaces
(APIs), software libraries and services provided by the cloud service provider.
• Provider Manages Infrastructure:
• The cloud service provider manages the underlying cloud infrastructure
including servers, network, operating systems and storage.
• User Manages Application:
• The users, themselves, are responsible for developing, deploying, configuring
and managing applications on the cloud infrastructure.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Software-as-a-Service (SaaS)
SaaS
Benefits Characteristics Examples
- Lower costs - Multi-tenancy - Google Apps
- No infrastructure required - On-demand software - Salesforce.com
- Seamless upgrades - Open integration protocols - Facebook
- Guaranteed performance - Social network integration - Zoho
- Automated backups Adoption - Dropbox
- Easy data recovery - Individual users: High - Taleo
- Small & medium enterprises: High
- Secure - Large organizations: High - Microsoft Office 365
- High adoption - Government: Medium - Linkedin
- On-the move access - Slideshare
Book website: www.cloudcomputingbook.info - CareCloud Bahga & Madisetti, © 2014
Infrastructure-as-a-Service (IaaS)
• Resource Provisioning
• Provides the users the capability to provision computing and storage resources.
• Virtual Machines
• These resources are provided to the users as virtual machine instances and virtual
storage. Users can start, stop, configure and manage the virtual machine instances
and virtual storage.
• Provider Managers Infrastructure:
• The cloud service provider manages the underlying infrastructure.
• Pay-per-use/Pay-as-you-go:
• Virtual resources provisioned by the users are billed based on a pay-per-use/pay-as-
you-go paradigm.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Platform-as-a-Service (PaaS)
PaaS
Benefits Characteristics Examples
- Lower upfront & operations costs - Multi-tenancy - Google App Engine
- No IT infrastructure management - Open integration protocols - Windows Azure Platform
costs
- App development tools & SDKs - Force.com
- Improved scalability
- Analytics - RightScale
- Higher performance Adoption - Heroku
- Secured access - Individual users: Low - Github
- Quick & easy development - Small & medium enterprises:
Medium - Gigaspaces
- Seamless integration - Large organizations: High - AppScale
- Government: Medium - OpenStack
Book website: www.cloudcomputingbook.info - LongJump Bahga & Madisetti, © 2014
Cloud Deployment Models
• Public Cloud
• Available for public use or a large industry group
• Private Cloud
• Operated for exclusive use of a single organization
• Community Cloud
• Available for shared use of several organizations supporting a specific
community
• Hybrid Cloud
• Combines multiple clouds (public and private) that remain unique but bound
together to offer application and data portability
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Infrastructure-as-a-Service (IaaS)
IaaS
Benefits Characteristics Examples
- Shift focus from IT management to - Multi-tenancy - Amazon Elastic Compute Cloud (EC2)
core activities
- Virtualized hardware - RackSpace
- No IT infrastructure management
- Management & monitoring tools - GoGrid
costs
- Disaster recovery - Eucalyptus
- Pay-per-use/pay-per-go pricing
Adoption - Joyent
- Guaranteed performance
- Individual users: Low - Terremark
- Dynamic scaling
- Small & medium enterprises: - OpSource
- Secure access Medium
- Large organizations: High - Savvis
- Enterprise grade infrastructure
- Government: High - Nimbula
- Green IT adoption
Book website: www.cloudcomputingbook.info - Enamoly Bahga & Madisetti, © 2014
Cloud Computing Applications
• Banking & Financial Apps
• E-Commerce Apps
• Social Networking
• Healthcare Systems
• Energy Systems
• Intelligent Transportation Systems
• E-Governance
• Education
• Mobile Communications
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Cloud Service Examples
• IaaS:
• Amazon EC2
• Google Compute Engine
• Windows Azure VMs
• PaaS:
• Google App Engine
• SaaS:
• Salesforce
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Further Reading
• Peter Mell, Timothy Grance, The NIST Definition of Cloud Computing, NIST Special Publication 800-145, Sep 2011.
• VMware, Understanding Full Virtualization, Paravirtualization, and Hardware Assist, 2007.
• A. Bahga, V. Madisetti, Analyzing Massive Machine Maintenance Data in a Computing Cloud, IEEE Transactions on Parallel &
Distributed Systems, Vol. 23, Iss. 10, Oct 2012.
• A. Bahga, V. Madisetti, On a Cloud-Based Information Technology Framework for Data Driven Intelligent Transportation Systems,
Journal of Transportation Technologies, Vol. 3, No. 2, April 2013.
• A. Bahga, V. Madisetti, A Cloud-Based Approach to Interoperable Electronic Health Records (EHRs), IEEE Journal of Biomedical and
Health Informatics, Vol. 17, Iss. 5, Sep 2013.
• Network Functions Virtualization, http://www.etsi.org/technologies-clusters/technologies/nfv, Retrieved 2013.
• Amazon Elastic Compute Cloud, http://aws.amazom.com/ec2, 2012.
• Google Compute Engine, https://developers.google.com/compute/, Retrieved 2013.
• Windows Azure, http://www.windowsazure.com/, Retrieved 2013.
• Google App Engine, http://appengine.google.com, 2012.
• Salesforce, http://salesforce.com, 2012.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
 Outline
• Concepts and enabling technologies of cloud computing
• Virtualization
• Load balancing
• Scalability & Elasticity
• Deployment
• Replication
• Monitoring
• MapReduce
• Identity and Access Management
• Service Level Agreements
• Billing
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Chapter 2
Concepts & Technologies
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Hypervisor
• The virtualization layer consists of a hypervisor or a
virtual machine monitor (VMM).
• Hypervisor presents a virtual operating platform to a
guest operating system (OS).
Type-1 Hypervisor
• Type-1 Hypervisor
• Type-I or the native hypervisors run directly on the host
hardware and control the hardware and monitor the guest
operating systems.
• Type-2 Hypervisor
• Type 2 hypervisors or hosted hypervisors run on top of a
conventional (main/host) operating system and monitor Type-2 Hypervisor
the guest operating systems.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Virtualization
• Virtualization refers to the partitioning the
resources of a physical system (such as
computing, storage, network and memory)
into multiple virtual resources.
• Key enabling technology of cloud
computing that allow pooling of resources.
• In cloud computing, resources are pooled
to serve multiple users using multi-tenancy.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
 Load Balancing
• Cloud computing resources can be scaled up on
demand to meet the performance requirements of
applications.
• Load balancing distributes workloads across multiple
servers to meet the application workloads.
• The goals of load balancing techniques include:
• Achieve maximum utilization of resources
• Minimizing the response times
• Maximizing throughput
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Types of Virtualization
• Full Virtualization
• In full virtualization, the virtualization layer completely decouples the guest OS from the
underlying hardware. The guest OS requires no modification and is not aware that it is being
virtualized. Full virtualization is enabled by direct execution of user requests and binary
translation of OS requests.
• Para-Virtualization
• In para-virtualization, the guest OS is modified to enable communication with the hypervisor to
improve performance and efficiency. The guest OS kernel is modified to replace non-virtualizable
instructions with hyper-calls that communicate directly with the virtualization layer hypervisor.
• Hardware Virtualization
• Hardware assisted virtualization is enabled by hardware features such as Intel’s Virtualization
Technology (VT-x) and AMD’s AMD-V. In hardware assisted virtualization, privileged and sensitive
calls are set to automatically trap to the hypervisor. Thus, there is no need for either binary
translation or para-virtualization.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
 Load Balancing - Persistence Approaches
• Since load balancing can route successive requests from a user
session to different servers, maintaining the state or the information
of the session is important.
• Persistence Approaches
• Sticky sessions
• Session Database
• Browser cookies
• URL re-writing
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Load Balancing Algorithms
• Round Robin load balancing
• Weighted Round Robin load balancing
• Low Latency load balancing
• Least Connections load balancing
• Priority load balancing
• Overflow load balancing
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
 Scaling Approaches
• Vertical Scaling/Scaling up:
• Involves upgrading the hardware resources (adding additional computing, memory,
storage or network resources).
• Horizontal Scaling/Scaling out
• Involves addition of more resources of the same type.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Scalability & Elasticity
• Multi-tier applications such as e-Commerce,
social networking, business-to-business, etc.
can experience rapid changes in their traffic.
• Capacity planning involves determining the
right sizing of each tier of the deployment of
an application in terms of the number of
resources and the capacity of each resource.
• Capacity planning may be for computing,
storage, memory or network resources.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Replication
• Replication is used to create and maintain multiple copies of the data in the cloud.
• Cloud enables rapid implementation of replication solutions for disaster recovery for
organizations.
• With cloud-based data replication organizations can plan for disaster recovery without
making any capital expenditures on purchasing, configuring and managing secondary site
locations.
• Types:
• Array-based Replication
• Network-based Replication
• Host-based Replication
Array-based Replication
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Deployment
• Cloud application deployment design is an iterative process that involves:
• Deployment Design
• The variables in this step include the number of servers in each tier, computing, memory and storage capacities
of severs, server interconnection, load balancing and replication strategies.
• Performance Evaluation
• To verify whether the application meets the performance requirements with the deployment.
• Involves monitoring the workload on the application and measuring various workload parameters such as
response time and throughput.
• Utilization of servers (CPU, memory, disk, I/O, etc.) in each tier is also monitored.
• Deployment Refinement
• Various alternatives can exist in this step such as vertical scaling (or scaling up), horizontal scaling (or scaling
out), alternative server interconnections, alternative load balancing and replication strategies, for instance.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Software Defined Networking
• Software-Defined Networking (SDN) is a networking architecture that separates the
control plane from the data plane and centralizes the network controller.
• Conventional network architecture
• The control plane and data plane are coupled. Control plane is the part of the network that carries the
signaling and routing message traffic while the data plane is the part of the network that carries the
payload data traffic.
• SDN Architecture
• The control and data planes are decoupled and the network controller is centralized.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Monitoring
• Monitoring services allow cloud users to
collect and analyze the data on various Examples of Monitoring Metrics
monitoring metrics.
Type Metrics
CPU CPU-Usage, CPU-Idle
• A monitoring service collects data on
various system and application metrics Disk Disk-Usage, Bytes/sec (read/write),
from the cloud computing instances. Operations/sec
Memory Memory-Used, Memory-Free, Page-Cache
Interface Packets/sec (incoming/outgoing),
• Monitoring of cloud resources is
Octets/sec(incoming/outgoing)
important because it allows the users to
keep track of the health of applications
and services deployed in the cloud.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
OpenFlow
• OpenFlow is the broadly accepted SDN protocol for the Southbound
interface.
• With OpenFlow, the forwarding plane of the network devices can be
directly accessed and manipulated.
• OpenFlow uses the concept of flows to identify network traffic based
on pre-defined match rules.
• Flows can be programmed statically or dynamically by the SDN
control software.
OpenFlow switch
• OpenFlow protocol is implemented on both sides of the interface comprising of one or more
between the controller and the network devices. flow tables and a group
table, which perform packet
lookups and forwarding,
and OpenFlow channel to
an external controller.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
SDN - Key Elements
• Centralized Network Controller
• With decoupled the control and data planes and centralized network controller, the network
administrators can rapidly configure the network.
• Programmable Open APIs
• SDN architecture supports programmable open APIs for interface between the SDN application and
control layers (Northbound interface). These open APIs that allow implementing various network services
such as routing, quality of service (QoS), access control, etc.
• Standard Communication Interface (OpenFlow)
• SDN architecture uses a standard communication interface between the control and infrastructure layers
(Southbound interface). OpenFlow, which is defined by the Open Networking Foundation (ONF) is the
broadly accepted SDN protocol for the Southbound interface.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
NFV Architecture
• Key elements of the NFV architecture are
• Virtualized Network Function (VNF): VNF is a
software implementation of a network function
which is capable of running over the NFV
Infrastructure (NFVI).
• NFV Infrastructure (NFVI): NFVI includes compute,
network and storage resources that are virtualized.
• NFV Management and Orchestration: NFV
Management and Orchestration focuses on all
virtualization-specific management tasks and covers
the orchestration and lifecycle management of
physical and/or software resources that support the
infrastructure virtualization, and the lifecycle
management of VNFs.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Network Function Virtualization
• Network Function Virtualization (NFV) is a technology that leverages virtualization to
consolidate the heterogeneous network devices onto industry standard high volume
servers, switches and storage.
• Relationship to SDN
• NFV is complementary to SDN as NFV can provide the infrastructure on which SDN can run.
• NFV and SDN are mutually beneficial to each other but not dependent.
• Network functions can be virtualized without SDN, similarly, SDN can run without NFV.
• NFV comprises of network functions implemented in software that run on virtualized resources in the cloud.
• NFV enables a separation the network functions which are implemented in software from the underlying
hardware.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Identity and Access Management
• Identity and Access Management (IDAM) for cloud describes the authentication and authorization
of users to provide secure access to cloud resources.
• Organizations with multiple users can use IDAM services provided by the cloud service provider
for management of user identifiers and user permissions.
• IDAM services allow organizations to centrally manage users, access permissions, security
credentials and access keys.
• Organizations can enable role-based access control to cloud resources and applications using the
IDAM services.
• IDAM services allow creation of user groups where all the users in a group have the same access
permissions.
• Identity and Access Management is enabled by a number of technologies such as OpenAuth,
Role-based Access Control (RBAC), Digital Identities, Security Tokens, Identity Providers, etc.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
MapReduce
• MapReduce is a parallel data processing model for
processing and analysis of massive scale data.
• MapReduce phases:
• Map Phase: In the Map phase, data is read from a
distributed file system, partitioned among a set of
computing nodes in the cluster, and sent to the nodes as a
set of key-value pairs.
• The Map tasks process the input records independently of
each other and produce intermediate results as key-value
pairs.
• The intermediate results are stored on the local disk of the
node running the Map task.
• Reduce Phase: When all the Map tasks are completed, the
Reduce phase begins in which the intermediate data with
the same key is aggregated.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Further Reading
• Network Functions Virtualization, http://www.etsi.org/technologies-clusters/technologies/nfv,
Retrieved 2013.
• OpenFlow Switch Specification, https://www.opennetworking.org, Retrieved 2013.
• J. Dean, S. Ghemawat, MapReduce: Simplified Data Processing on Large Clusters, OSDI 2004.
• VMware, Understanding Full Virtualization, Paravirtualization, and Hardware Assist, 2007.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Billing
Cloud service providers offer a number of billing models described as follows:
• Elastic Pricing
• In elastic pricing or pay-as-you-use pricing model, the customers are charged based on the usage of cloud resources.
• Fixed Pricing
• In fixed pricing models, customers are charged a fixed amount per month for the cloud resources.
• Spot Pricing
• Spot pricing models offer variable pricing for cloud resources which is driven by market demand.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
 Outline
• Compute Services
• Storage Services
• Database Services
• Application Services
• Content Delivery Services
• Analytics Services
• Deployment & Management Services
• Identity & Access Management Services
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Chapter 3
Cloud Services & Platforms
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Compute Services
• Compute services provide dynamically scalable compute capacity in the cloud.
• Compute resources can be provisioned on-demand in the form of virtual machines.
Virtual machines can be created from standard images provided by the cloud
service provider or custom images created by the users.
• Compute services can be accessed from the web consoles of these services that
provide graphical user interfaces for provisioning, managing and monitoring these
services.
• Cloud service providers also provide APIs for various programming languages that
allow developers to access and manage these services programmatically.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Cloud Reference Model
• Infrastructure & Facilities Layer
• Includes the physical infrastructure such as datacenter facilities, electrical
and mechanical equipment, etc.
• Hardware Layer
• Includes physical compute, network and storage hardware.
• Virtualization Layer
• Partitions the physical hardware resources into multiple virtual resources
that enabling pooling of resources.
• Platform & Middleware Layer
• Builds upon the IaaS layers below and provides standardized stacks of
services such as database service, queuing service, application frameworks
and run-time environments, messaging services, monitoring services,
analytics services, etc.
• Service Management Layer
• Provides APIs for requesting, managing and monitoring cloud resources.
• Applications Layer
• Includes SaaS applications such as Email, cloud storage application,
productivity applications, management portals, customer self-service
portals, etc.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Compute Services – Google Compute Engine
• Google Compute Engine is a compute service provided by
Google.
• Launching Instances
• To create a new instance, the user selects an instance machine type, a zone in which the
instance will be launched, a machine image for the instance and provides an instance name,
instance tags and meta-data.
• Disk Resources
• Every instance is launched with a disk resource. Depending on the instance type, the disk
resource can be a scratch disk space or persistent disk space. The scratch disk space is deleted
when the instance terminates. Whereas, persistent disks live beyond the life of an instance.
• Network Options
• Network option allows you to control the traffic to and from the instances. By default, traffic
between instances in the same network, over any port and any protocol and incoming SSH
connections from anywhere are enabled.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Compute Services – Amazon EC2
• Amazon Elastic Compute Cloud (EC2) is a compute service provided by Amazon.
• Launching EC2 Instances
• To launch a new instance click on the launch instance button. This will open a wizard
where you can select the Amazon machine image (AMI) with which you want to launch
the instance. You can also create their own AMIs with custom applications, libraries
and data. Instances can be launched with a variety of operating systems.
• Instance Sizes
• When you launch an instance you specify the instance type (micro, small, medium,
large, extra-large, etc.), the number of instances to launch based on the selected AMI
and availability zones for the instances.
• Key-pairs
• When launching a new instance, the user selects a key-pair from existing keypairs or
creates a new keypair for the instance. Keypairs are used to securely connect to an
instance after it launches.
• Security Groups
• The security groups to be associated with the instance can be selected from the
instance launch wizard. Security groups are used to open or block a specific network
port for the launched instances.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Storage Services
• Cloud storage services allow storage and retrieval of any amount of data, at any time from anywhere on the
web.
• Most cloud storage services organize data into buckets or containers.
• Scalability
• Cloud storage services provide high capacity and scalability. Objects upto several tera-bytes in size can be uploaded and
multiple buckets/containers can be created on cloud storages.
• Replication
• When an object is uploaded it is replicated at multiple facilities and/or on multiple devices within each facility.
• Access Policies
• Cloud storage services provide several security features such as Access Control Lists (ACLs), bucket/container level policies, etc.
ACLs can be used to selectively grant access permissions on individual objects. Bucket/container level policies can also be
defined to allow or deny permissions across some or all of the objects within a single bucket/container.
• Encryption
• Cloud storage services provide Server Side Encryption (SSE) options to encrypt all data stored in the cloud storage.
• Consistency
• Strong data consistency is provided for all upload and delete operations. Therefore, any object that is uploaded can be
immediately downloaded after the upload is complete.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Compute Services – Windows Azure VMs
• Windows Azure Virtual Machines is the compute service from
Microsoft.
• Launching Instances:
• To create a new instance, you select the instance type and the machine image.
• You can either provide a user name and password or upload a certificate file for securely
connecting to the instance.
• Any changes made to the VM are persistently stored and new VMs can be created from the
previously stored machine images.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Storage Services – Google Cloud Storage
• GCS is the Cloud storage service from Google
• Buckets
• Objects in GCS are organized into buckets.
• Access Control Lists
• ACLs are used to control access to objects and buckets. ACLs can be
configured to share objects and buckets with the entire world, a Google
group, a Google-hosted domain, or specific Google account holders.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Storage Services – Amazon S3
• Amazon Simple Storage Service(S3) is an online cloud-based
data storage infrastructure for storing and retrieving any
amount of data.
• S3 provides highly reliable, scalable, fast, fully redundant and
affordable storage infrastructure.
• Buckets
• Data stored on S3 is organized in the form of buckets. You must create a
bucket before you can store data on S3.
• Uploading Files to Buckets
• S3 console provides simple wizards for creating a new bucket and
uploading files.
• You can upload any kind of file to S3.
• While uploading a file, you can specify the redundancy and encryption
options and access permissions.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Database Services
• Cloud database services allow you to set-up and operate relational or non-relational databases in the cloud.
• Relational Databases
• Popular relational databases provided by various cloud service providers include MySQL, Oracle, SQL Server, etc.
• Non-relational Databases
• The non-relational (No-SQL) databases provided by cloud service providers are mostly proprietary solutions.
• Scalability
• Cloud database services allow provisioning as much compute and storage resources as required to meet the application
workload levels. Provisioned capacity can be scaled-up or down. For read-heavy workloads, read-replicas can be created.
• Reliability
• Cloud database services are reliable and provide automated backup and snapshot options.
• Performance
• Cloud database services provide guaranteed performance with options such as guaranteed input/output operations per
second (IOPS) which can be provisioned upfront.
• Security
• Cloud database services provide several security features to restrict the access to the database instances and stored data,
such as network firewalls and authentication mechanisms.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Storage Services – Windows Azure Storage
• Windows Azure Storage is the cloud storage service from
Microsoft.
• Windows Azure Storage provides various storage services such
as blob storage service, table service and queue service.
• Blob storage service
• The blob storage service allows storing unstructured binary data or binary
large objects (blobs).
• Blobs are organized into containers.
• Block blobs - can be subdivided into some number of blocks. If a failure
occurs while transferring a block blob, retransmission can resume with the
most recent block rather than sending the entire blob again.
• Page blobs - are divided into number of pages and are designed for
random access. Applications can read and write individual pages at
random in a page blob.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Database Services – Amazon DynamoDB
• Amazon DynamoDB is the non-relational (No-SQL) database
service from Amazon.
• Data Model
• The DynamoDB data model includes include tables, items and attributes.
• A table is a collection of items and each item is a collection of attributes.
• To store data in DynamoDB you have to create a one or more tables and
specify how much throughput capacity you want to provision and reserve
for reads and writes.
• Fully Managed Service
• DynamoDB is a fully managed service that automatically spreads the data
and traffic for the stored tables over a number of servers to meet the
throughput requirements specified by the users.
• Replication
• All stored data is automatically replicated across multiple availability zones
to provide data durability.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Database Services – Amazon RDS
• Amazon Relational Database Service (RDS) is a web service that
makes it easy to setup, operate and scale a relational database
in the cloud.
• Launching DB Instances
• The console provides an instance launch wizard that allows you to select
the type of database to create (MySQL, Oracle or SQL Server) database
instance size, allocated storage, DB instance identifier, DB username and
password. The status of the launched DB instances can be viewed from the
console.
• Connecting to a DB Instance
• Once the instance is available, you can note the instance end point from
the instance properties tab. This end point can then be used for securely
connecting to the instance.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Storage Services – Google Cloud Datastore
• Google Cloud Datastore is a fully managed non-relational
database from Google.
• Cloud Datastore offers ACID transactions and high availability
of reads and writes.
• Data Model
• The Cloud Datastore data model consists of entities. Each entity has one or
more properties (key-value pairs) which can be of one of several
supported data types, such as strings and integers. Each entity has a kind
and a key. The entity kind is used for categorizing the entity for the
purpose of queries and the entity key uniquely identifies the entity.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Storage Services – Google Cloud SQL
• Google SQL is the relational database service from Google.
• Google Cloud SQL service allows you to host MySQL databases
in the Google’s cloud.
• Launching DB Instances
• You can create new database instances from the console and manage
existing instances. To create a new instance you select a region, database
tier, billing plan and replication mode.
• Backups
• You can schedule daily backups for your Google Cloud SQL instances, and
also restore backed-up databases.
• Replication
• Cloud SQL provides both synchronous or asynchronous geographic
replication and the ability to import/ export databases.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Storage Services – Windows Azure Table Service
• Windows Azure Table Service is a non-relational (No-SQL) database service from Microsoft.
• Data Model
• The Azure Table Service data model consists of tables having multiple entities.
• Tables are divided into some number of partitions, each of which can be stored on a separate machine.
• Each partition in a table holds a specified number of entities, each containing as many as 255 properties.
• Each property can be one of the several supported data types such as integers and strings.
• No Fixed Schema
• Tables do not have a fixed schema and different entities in a table can have different properties.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Storage Services – Windows Azure SQL DB
• Windows Azure SQL Database is the relational database
service from Microsoft.
• Azure SQL Database is based on the SQL server, but it does not
give each customer a separate instance of SQL server.
• Multi-tenant Service
• SQL Database is a multi-tenant service, with a logical SQL Database server
for each customer.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Google App Engine
• Google App Engine is the platform-as-a-service
(PaaS) from Google, which includes both an
application runtime and web frameworks.
• Runtimes
• App Engine provides runtime environments for Java,
Python, PHP and Go programming language.
• Sandbox
• Applications run in a secure sandbox environment
isolated from other applications.
• The sandbox environment provides a limited access to
the underlying operating system.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Application Runtimes & Frameworks
• Cloud-based application runtimes and frameworks allow developers to develop and host
applications in the cloud.
• Support for various programming languages
• Application runtimes provide support for programming languages (e.g., Java, Python, or Ruby).
• Resource Allocation
• Application runtimes automatically allocate resources for applications and handle the application
scaling, without the need to run and maintain servers.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Windows Azure Web Sites
• Windows Azure Web Sites is a Platform-as-a-Service (PaaS) from Microsoft.
• Azure Web Sites allows you to host web applications in the Azure cloud.
• Shared & Standard Options.
• In the shared option, Azure Web Sites run on a set of virtual machines that may contain multiple web
sites created by multiple users.
• In the standard option, Azure Web Sites run on virtual machines (VMs) that belong to an individual user.
• Azure Web Sites supports applications created in ASP.NET, PHP, Node.js and Python
programming languages.
• Multiple copies of an application can be run in different VMs, with Web Sites
automatically load balancing requests across them.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Google App Engine
• Web Frameworks
• App Engine provides a simple Python web application framework called webapp2. App Engine also supports any framework written in
pure Python that speaks WSGI, including Django, CherryPy, Pylons, web.py, and web2py.
• Datastore
• App Engine provides a no-SQL data storage service.
• Authentication
• App Engine applications can be integrated with Google Accounts for user authentication.
• URL Fetch service
• URL Fetch service allows applications to access resources on the Internet, such as web services or other data.
• Other services
• Email service
• Image Manipulation service
• Memcache
• Task Queues
• Scheduled Tasks service
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
 Queuing Services - Google Task Queue Service
• Google Task Queues service is a queuing service from Google and is a part of the Google App Engine platform.
• Task queues allow applications to execute tasks in background.
• Tasks
• Task is a unit of work to be performed by an application. The task objects consist of application-specific URL with a request handler for
the task, and an optional data payload that parameterizes the task.
• Push Queue
• Push Queue is the default queue that processes tasks based on the processing rate configured in the queue definition.
• Pull Queue
• Pull Queues allow task consumers to lease a specific number of tasks for a specific duration. The tasks are processed and deleted before
the lease ends.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Queuing Services - Amazon Simple Queue Service
• Amazon Simple Queue Service (SQS) is a queuing service from
Amazon.
• Short Messages
• SQS is a distributed queue that supports messages of up to 256 KB in size.
• Multiple Writers/Readers
• SQS supports multiple writers and readers and locks messages while they are
being processed.
• High Availability
• To ensure high availability for delivering messages, SQS service trade-offs on
the first in, first out capability and does not guarantee that messages will be
delivered in FIFO order.
• Applications that require FIFO ordering of messages can place additional
sequencing information in each message so that they can be re-ordered after
retrieving from a queue.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Email Services
• Cloud-based email services allow applications hosted in the cloud to send emails.
• Amazon Simple Email Service
• Amazon Simple Email Service is bulk and transactional email-sending service from Amazon
• SES is an outbound-only email-sending service that allows applications hosted in the Amazon cloud to send emails such
as marketing emails, transactional emails and other types of correspondence
• To ensure high email deliverability, SES uses content filtering technologies to scan the outgoing email messages
• SES service can be accessed and used from the SES console, the Simple Mail Transfer Protocol (SMTP) interface, or the
SES API
• Google Email Service
• Google Email service is part of the Google App Engine platform that allows App Engine applications to send email
messages on behalf of the app’s administrators, and on behalf of users with Google Accounts.
• App Engine apps can also receive emails. Apps send messages using the Mail service and receive messages in the form
of HTTP requests initiated by App Engine and posted to the app.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Queuing Services - Windows Azure Queue Service
• Windows Azure Queue service is a queuing service from Microsoft.
• Azure Queue service allows storing large numbers of messages that can be accessed from anywhere in the
world via authenticated calls using HTTP or HTTPS.
• Short Messages
• The size of a single message can be up to 64KB.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Notification Services - Amazon Simple Notification Service
• Amazon Simple Notification Service is a push
messaging service from Amazon.
• SNS has two types of clients:
• Publishers
• Publishers communicate asynchronously with subscribers by
producing and sending messages to topics. A topic is a logical
access point and a communication channel.
• Subscribers.
• Subscribers are the consumers who subscribe to topics to receive
notifications.
• SNS can deliver notifications as SMS, email,
or to SQS queues, or any HTTP endpoint.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Notification Services
• Cloud-based notification services or push messaging services allow applications to push messages
to internet connected smart devices such as smartphones, tablets, etc.
• Push messaging services are based on publish-subscribe model in which consumers subscribe to
various topics/channels provided by a publisher/producer.
• Whenever new content is available on one of those topics/channels, the notification service pushes
that information out to the consumer.
• Push notifications are used for such smart devices as they help in displaying the latest information
while remaining energy efficient.
• Consumer applications on such devices can increase their consumer engagement with the help of
push notifications.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Windows Azure Notification Hubs
• Windows Azure Notification Hubs is a push notification service from Microsoft.
• Common Interface
• Provides a common interface to send notifications to all major mobile platforms including Windows Store/Windows
Phone 8, iOS, and Android.
• Platform Notification Systems
• Platform specific infrastructures called Platform Notification Systems (PNS) are used to deliver notification messages.
• Devices register their PNS handles with the Notification Hub.
• Each notification hub contains credentials for each supported PNS.
• These credentials are used to connect to the PNSs and send push notifications to the applications.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Google Cloud Messaging
• Google Cloud Messaging for Android provides push messaging for Android devices.
• GCM allows allows applications to send data from the application servers to their users’ Android
devices, and also to receive messages from devices on the same connection.
• Notifying Android Apps
• GCM is useful for notifying applications on Android devices that there is new data to be fetched from the application
servers.
• Short Messages
• GCM supports messages with payload data upto 4 KB.
• Send-to-Sync
• GCM provides a ‘send-to-sync’ message capability that can be used to inform an application to sync data from the
server.
• GCM for Chrome
• Google Cloud Messaging for Chrome is another notification service from Google that allows messages to be delivered
from the cloud to apps and extensions running in Chrome.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Content Delivery Services
• Cloud-based content delivery service include Content Delivery Networks (CDNs).
• CDN is a distributed system of servers located across multiple geographic locations to serve content to end-
users with high availability and high performance.
• CDNs are useful for serving static content such as text, images, scripts, etc., and streaming media.
• CDNs have a number of edge locations deployed in multiple locations, often over multiple backbones.
• Requests for static for streaming media content that is served by a CDN are directed to the nearest edge
location.
• Amazon CloudFront
• Amazon CloudFront is a content delivery service from Amazon. CloudFront can be used to deliver dynamic, static and streaming content
using a global network of edge locations.
• Windows Azure Content Delivery Network
• Windows Azure Content Delivery Network (CDN) is the content delivery service from Microsoft.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Media Services
• Cloud service providers provide various types of media services that can be used by applications for
manipulating, transforming or transcoding media such as images, videos, etc.
• Amazon Elastic Transcoder
• Amazon Elastic Transcoder is a cloud-based video transcoding service from Amazon.
• Elastic Transcoder can be used to convert video files from their source format into various other formats that can be
played on devices such as desktops, mobiles, tablets, etc.
• Google Images Manipulation Service
• Google Images Manipulation service is a part of the Google App Engine platform. Image Manipulation service provides
the capability to resize, crop, rotate, flip and enhance images.
• Windows Azure Media Services
• Windows Azure Media Services provides the various media services such as encoding & format conversion, content
protection and on-demand and live streaming capabilities.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Deployment & Management Services
• Cloud-based deployment & management services allow you to easily deploy and manage applications in the
cloud. These services automatically handle deployment tasks such as capacity provisioning, load balancing,
auto-scaling, and application health monitoring.
• Amazon Elastic Beanstalk
• Amazon provides a deployment service called Elastic Beanstalk that allows you to quickly deploy and manage applications in the AWS
cloud.
• Elastic Beanstalk supports Java, PHP, .NET, Node.js, Python, and Ruby applications.
• With Elastic Beanstalk you just need to upload the application and specify configuration settings in a simple wizard and the service
automatically handles instance provisioning, server configuration, load balancing and monitoring.
• Amazon CloudFormation
• Amazon CloudFormation is a deployment management service from Amazon.
• With CloudFront you can create deployments from a collection of AWS resources such as Amazon Elastic Compute Cloud, Amazon
Elastic Block Store, Amazon Simple Notification Service, Elastic Load Balancing and Auto Scaling.
• A collection of AWS resources that you want to manage together are organized into a stack.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Analytics Services
• Cloud-based analytics services allow analyzing massive data sets stored in the cloud either in cloud storages or
in cloud databases using programming models such as MapReduce.
• Amazon Elastic MapReduce
• Amazon Elastic MapReduce is the MapReduce service from Amazon based the Hadoop framework running on Amazon EC2 and S3
• EMR supports various job types such as Custom JAR, Hive program, Streaming job, Pig programs and Hbase
• Google MapReduce Service
• Google MapReduce Service is a part of the App Engine platform and can be accessed using the Google MapReduce API.
• Google BigQuery
• Google BigQuery is a service for querying massive datasets. BigQuery allows querying datasets using SQL-like queries.
• Windows Azure HDInsight
• Windows Azure HDInsight is an analytics service from Microsoft. HDInsight deploys and provisions Hadoop clusters in the Azure cloud
and makes Hadoop available as a service.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
 Open Source Private Cloud Software - CloudStack
• Apache CloudStack is an open source cloud software that can be used for creating private
cloud offerings.
• CloudStack manages the network, storage, and compute nodes that make up a cloud
infrastructure.
• A CloudStack installation consists of a Management Server and the cloud infrastructure that
it manages.
• Zones
• The Management Server manages one or more zones where each zone is typically a single datacenter.
• Pods
• Each zone has one or more pods. A pod is a rack of hardware comprising of a switch and one or more clusters.
• Cluster
• A cluster consists of one or more hosts and a primary storage. A host is a compute node that runs guest
virtual machines.
• Primary Storage
• The primary storage of a cluster stores the disk volumes for all the virtual machines running on the hosts in
that cluster.
• Secondary Storage
• Each zone has a secondary storage that stores templates, ISO images, and disk volume snapshots.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Identity & Access Management Services
• Identity & Access Management (IDAM) services allow managing the authentication and authorization of users
to provide secure access to cloud resources.
• Using IDAM services you can manage user identifiers, user permissions, security credentials and access keys.
• Amazon Identity & Access Management
• AWS Identity and Access Management (IAM) allows you to manage users and user permissions for an AWS account.
• With IAM you can manage users, security credentials such as access keys, and permissions that control which AWS resources users can
access.
• Using IAM you can control what data users can access and what resources users can create.
• IAM also allows you to control creation, rotation, and revocation security credentials of users.
• Windows Azure Active Directory
• Windows Azure Active Directory is an Identity & Access Management Service from Microsoft.
• Azure Active Directory provides a cloud-based identity provider that easily integrates with your on-premises active directory
deployments and also provides support for third party identity providers.
• With Azure Active Directory you can control access to your applications in Windows Azure.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
 Open Source Private Cloud Software - OpenStack
• Eucalyptus is an open source private cloud software for building private
and hybrid clouds that are compatible with Amazon Web Services (AWS)
APIs.
• Node Controller
• NC hosts the virtual machine instances and manages the virtual network endpoints.
• The cluster-level (availability-zone) consists of three components
• Cluster Controller - which manages the virtual machines and is the front-end for a cluster.
• Storage Controller – which manages the Eucalyptus block volumes and snapshots to the instances
within its specific cluster. SC is equivalent to AWS Elastic Block Store (EBS).
• VMWare Broker - which is an optional component that provides an AWS-compatible interface for
VMware environments.
• At the cloud-level there are two components:
• Cloud Controller - which provides an administrative interface for cloud management and performs
high-level resource scheduling, system accounting, authentication and quota management.
• Walrus - which is equivalent to Amazon S3 and serves as a persistent storage to all of the virtual
machines in the Eucalyptus cloud. Walrus can be used as a simple Storage-as-a-Service
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Open Source Private Cloud Software - Eucalyptus
• Apache CloudStack is an open source cloud software that can be used for creating private
cloud offerings.
• CloudStack manages the network, storage, and compute nodes that make up a cloud
infrastructure.
• A CloudStack installation consists of a Management Server and the cloud infrastructure that
it manages.
• Zones
• The Management Server manages one or more zones where each zone is typically a single datacenter.
• Pods
• Each zone has one or more pods. A pod is a rack of hardware comprising of a switch and one or more clusters.
• Cluster
• A cluster consists of one or more hosts and a primary storage. A host is a compute node that runs guest
virtual machines.
• Primary Storage
• The primary storage of a cluster stores the disk volumes for all the virtual machines running on the hosts in
that cluster.
• Secondary Storage
• Each zone has a secondary storage that stores templates, ISO images, and disk volume snapshots.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Further Reading
• Amazon Elastic Compute Cloud, http://aws.amazom.com/ec2
• Google Compute Engine, https://developers.google.com/compute/
• Windows Azure, http://www.windowsazure.com/
• Google App Engine, http://appengine.google.com
• Google App Engine, https://developers.google.com/appengine/
• Google Cloud Storage, https://developers.google.com/storage/
• Google BigQuery, https://developers.google.com/bigquery/
• Google Cloud Datastore, http://developers.google.com/datastore/
• Google Cloud SQL, https://developers.google.com/cloud-sql/
• CloudStack, http://cloudstack.apache.org
• Eucalyptus, http://www.eucalyptus.com
• OpenStack, http://www.openstack.org
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
 Outline
• Overview of Hadoop ecosystem
• MapReduce architecture
• MapReduce job execution flow
• MapReduce schedulers
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Chapter 4
Hadoop & MapReduce
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Apache Hadoop
• A Hadoop cluster comprises of a Master node, backup node and a number
of slave nodes.
• The master node runs the NameNode and JobTracker processes and the
slave nodes run the DataNode and TaskTracker components of Hadoop.
• The backup node runs the Secondary NameNode process.
• NameNode
• NameNode keeps the directory tree of all files in the file system, and tracks
where across the cluster the file data is kept. It does not store the data of these
files itself. Client applications talk to the NameNode whenever they wish to
locate a file, or when they want to add/copy/move/delete a file.
• Secondary NameNode
• NameNode is a Single Point of Failure for the HDFS Cluster. An optional
Secondary NameNode which is hosted on a separate machine creates
checkpoints of the namespace.
• JobTracker
• The JobTracker is the service within Hadoop that distributes MapReduce tasks to
specific nodes in the cluster, ideally the nodes that have the data, or at least are
in the same rack.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Hadoop Ecosystem
• Apache Hadoop is an open source framework for distributed batch processing of big data.
• Hadoop Ecosystem includes:
• Hadoop MapReduce
• HDFS
• YARN
• HBase
• Zookeeper
• Pig
• Hive
• Mahout
• Chukwa
• Cassandra
• Avro
• Oozie
• Flume
• Sqoop
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
MapReduce
• MapReduce job consists of two phases:
• Map: In the Map phase, data is read from a distributed
file system and partitioned among a set of computing
nodes in the cluster. The data is sent to the nodes as a set
of key-value pairs. The Map tasks process the input
records independently of each other and produce
intermediate results as key-value pairs. The intermediate
results are stored on the local disk of the node running
the Map task.
• Reduce: When all the Map tasks are completed, the
Reduce phase begins in which the intermediate data with
the same key is aggregated.
• Optional Combine Task
• An optional Combine task can be used to perform data
aggregation on the intermediate data of the same key for
the output of the mapper before transferring the output
to the Reduce task.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Apache Hadoop
• TaskTracker
• TaskTracker is a node in a Hadoop cluster that accepts Map, Reduce
and Shuffie tasks from the JobTracker.
• Each TaskTracker has a defined number of slots which indicate the
number of tasks that it can accept.
• DataNode
• A DataNode stores data in an HDFS file system.
• A functional HDFS filesystem has more than one DataNode, with data
replicated across them.
• DataNodes respond to requests from the NameNode for filesystem
operations.
• Client applications can talk directly to a DataNode, once the
NameNode has provided the location of the data.
• Similarly, MapReduce operations assigned to TaskTracker instances
near a DataNode, talk directly to the DataNode to access the files.
• TaskTracker instances can be deployed on the same servers that host
DataNode instances, so that MapReduce operations are performed
close to the data.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
 MapReduce Job Execution Workflow
• The JobTracker submits the work to the TaskTracker nodes when they poll for tasks. To choose a task for a
TaskTracker, the JobTracker uses various scheduling algorithms (default is FIFO).
• The TaskTracker nodes are monitored using the heartbeat signals that are sent by the TaskTrackers to
JobTracker.
• The TaskTracker spawns a separate JVM process for each task so that any task failure does not bring down
the TaskTracker.
• The TaskTracker monitors these spawned processes while capturing the output and exit codes. When the
process finishes, successfully or not, the TaskTracker notifies the JobTracker. When the job is completed, the
JobTracker updates its status.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
MapReduce Job Execution Workflow
• MapReduce job execution starts when the client applications submit jobs to the Job tracker.
• The JobTracker returns a JobID to the client application. The JobTracker talks to the NameNode to determine
the location of the data.
• The JobTracker locates TaskTracker nodes with available slots at/or near the data.
• The TaskTrackers send out heartbeat messages to the JobTracker, usually every few minutes, to reassure the
JobTracker that they are still alive. These messages also inform the JobTracker of the number of available
slots, so the JobTracker can stay up to date with where in the cluster, new work can be delegated.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
YARN Components
• Resource Manager (RM): RM manages the global assignment of
compute resources to applications. RM consists of two main
services:
• Scheduler: Scheduler is a pluggable service that manages and enforces the
resource scheduling policy in the cluster.
• Applications Manager (AsM): AsM manages the running Application Masters in
the cluster. AsM is responsible for starting application masters and for monitoring
and restarting them on different nodes in case of failures.
• Application Master (AM): A per-application AM manages the
application’s life cycle. AM is responsible for negotiating resources
from the RM and working with the NMs to execute and monitor
the tasks.
• Node Manager (NM): A per-machine NM manages the user
processes on that machine.
• Containers: Container is a bundle of resources allocated by RM
(memory, CPU, network, etc.). A container is a conceptual entity
that grants an application the privilege to use a certain amount of
resources on a given machine to run a component task.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
MapReduce 2.0 - YARN
• In Hadoop 2.0 the original processing engine of Hadoop
(MapReduce) has been separated from the resource
management (which is now part of YARN).
• This makes YARN effectively an operating system for
Hadoop that supports different processing engines on a
Hadoop cluster such as MapReduce for batch processing,
Apache Tez for interactive queries, Apache Storm for
stream processing, etc.
• YARN architecture divides architecture divides the two
major functions of the JobTracker - resource management
and job life-cycle management - into separate components:
• ResourceManager
• ApplicationMaster.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
FIFO Scheduler
• FIFO is the default scheduler in Hadoop that maintains a work queue
in which the jobs are queued.
• The scheduler pulls jobs in first in first out manner (oldest job first)
for scheduling.
• There is no concept of priority or size of job in FIFO scheduler.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Hadoop Schedulers
• Hadoop scheduler is a pluggable component that makes it open to support
different scheduling algorithms.
• The default scheduler in Hadoop is FIFO.
• Two advanced schedulers are also available - the Fair Scheduler, developed
at Facebook, and the Capacity Scheduler, developed at Yahoo.
• The pluggable scheduler framework provides the flexibility to support a
variety of workloads with varying priority and performance constraints.
• Efficient job scheduling makes Hadoop a multi-tasking system that can
process multiple data sets for multiple jobs for multiple users
simultaneously.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Capacity Scheduler
• The Capacity Scheduler has similar functionally as the Fair Scheduler but
adopts a different scheduling philosophy.
• Queues
• In Capacity Scheduler, you define a number of named queues each with a
configurable number of map and reduce slots.
• Each queue is also assigned a guaranteed capacity.
• The Capacity Scheduler gives each queue its capacity when it contains jobs, and
shares any unused capacity between the queues. Within each queue FIFO scheduling
with priority is used.
• Fairness
• For fairness, it is possible to place a limit on the percentage of running tasks per user,
so that users share a cluster equally.
• A wait time for each queue can be configured. When a queue is not scheduled for
more than the wait time, it can preempt tasks of other queues to get its fair share.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Fair Scheduler
• The Fair Scheduler allocates resources evenly between multiple jobs and also provides capacity
guarantees.
• Fair Scheduler assigns resources to jobs such that each job gets an equal share of the available
resources on average over time.
• Tasks slots that are free are assigned to the new jobs, so that each job gets roughly the same
amount of CPU time.
• Job Pools
• The Fair Scheduler maintains a set of pools into which jobs are placed. Each pool has a guaranteed capacity.
• When there is a single job running, all the resources are assigned to that job. When there are multiple jobs in
the pools, each pool gets at least as many task slots as guaranteed.
• Each pool receives at least the minimum share.
• When a pool does not require the guaranteed share the excess capacity is split between other jobs.
• Fairness
• The scheduler computes periodically the difference between the computing time received by each job and
the time it should have received in ideal scheduling.
• The job which has the highest deficit of the compute time received is scheduled next.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Further Reading
• Apache Hadoop, http://hadoop.apache.org
• Apache Hive, http://hive.apache.org
• Apache HBase, http://hbase.apache.org
• Apache Chukwa, http://chukwa.apache.org
• Apache Flume, http://flume.apache.org
• Apache Zookeeper, http://zookeeper.apache.org
• Apache Avro, http://avro.apache.org
• Apache Oozie, http://oozie.apache.org
• Apache Storm, http://storm-project.net
• Apache Tez, http://tez.incubator.apache.org
• Apache Cassandra, http://cassandra.apache.org
• Apache Mahout, http://mahout.apache.org
• Apache Pig, http://pig.apache.org
• Apache Sqoop, http://sqoop.apache.org
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Hadoop Cluster Setup
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
 Outline
• Cloud Application Design Considerations
• Cloud Application Reference Architectures
• Design Methodologies
• Data Storage
• Data Analytics
• Deployment & Management
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Chapter 5
Cloud Application Design
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Reference Architectures –
e-Commerce, Business-to-Business, Banking and Financial apps
• Load Balancing Tier
• Load balancing tier consists of one or more load balancers.
• Application Tier
• For this tier, it is recommended to configure auto scaling.
• Auto scaling can be triggered when the recorded values for
any of the specified metrics such as CPU usage, memory
usage, etc. goes above defined thresholds.
• Database Tier
• The database tier includes a master database instance and
multiple slave instances.
• The master node serves all the write requests and the read
requests are served from the slave nodes.
• This improves the throughput for the database tier since
most applications have a higher number of read requests
than write requests.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Design Considerations for Cloud Applications
• Scalability
• Scalability is an important factor that drives the application designers to move to cloud computing environments. Building applications
that can serve millions of users without taking a hit on their performance has always been challenging. With the growth of cloud
computing application designers can provision adequate resources to meet their workload levels.
• Reliability & Availability
• Reliability of a system is defined as the probability that a system will perform the intended functions under stated conditions for a
specified amount of time. Availability is the probability that a system will perform a specified function under given conditions at a
prescribed time.
• Security
• Security is an important design consideration for cloud applications given the outsourced nature of cloud computing environments.
• Maintenance & Upgradation
• To achieve a rapid time-to-market, businesses typically launch their applications with a core set of features ready and then
incrementally add new features as and when they are complete. In such scenarios, it is important to design applications with low
maintenance and upgradation costs.
• Performance
• Applications should be designed while keeping the performance requirements in mind.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Reference Architectures –
Analytics apps
• Figure shows a typical deployment architecture for compute
intensive applications such as Data Analytics, Media
Transcoding, etc.
• Comprises of web, application, storage, computing/analytics
and database tiers.
• The analytics tier consists of cloud-based distributed batch
processing frameworks such as Hadoop which are suitable
for analyzing big data.
• Data analysis jobs (such as MapReduce) jobs are submitted
to the analytics tier from the application servers.
• The jobs are queued for execution and upon completion the
analyzed data is presented from the application servers.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Reference Architectures –
Content delivery apps
• Figure shows a typical deployment architecture for content
delivery applications such as online photo albums, video
webcasting, etc.
• Both relational and non-relational data stores are shown in
this deployment.
• A content delivery network (CDN) which consists of a global
network of edge locations is used for media delivery.
• CDN is used to speed up the delivery of static content such
as images and videos.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
SOA Layers
• Business Systems
• This layer consists of custom built applications and legacy systems such as
Enterprise Resource Planning (ERP), Customer Relationship Management
(CRM), Supply Chain Management (SCM), etc.
• Service Components
• The service components allow the layers above to interact with the business
systems. The service components are responsible for realizing the
functionality of the services exposed.
• Composite Services
• These are coarse-grained services which are composed of two or more
service components. Composite services can be used to create enterprise
scale components or business-unit specific components.
• Orchestrated Business Processes
• Composite services can be orchestrated to create higher level business
processes. In this layers the compositions and orchestrations of the
composite services are defined to create business processes.
• Presentation Services
• This is the topmost layer that includes user interfaces that exposes the
services and the orchestrated business processes to the users.
• Enterprise Service Bus
• This layer integrates the services through adapters, routing, transformation
and messaging mechanisms.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Service Oriented Architecture
• Service Oriented Architecture (SOA) is a well established architectural
approach for designing and developing applications in the form
services that can be shared and reused.
• SOA is a collection of discrete software modules or services that form
a part of an application and collectively provide the functionality of an
application.
• SOA services are developed as loosely coupled modules with no hard-
wired calls embedded in the services.
• The services communicate with each other by passing messages.
• Services are described using the Web Services Description Language
(WSDL).
• WSDL is an XML-based web services description language that is used
to create service descriptions containing information on the functions
performed by a service and the inputs and outputs of the service.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
CCM Application Design Methodology
• CCM approach for application
design involves:
• Component Design
• Architecture Design
• Deployment Design
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Cloud Component Model
• Cloud Component Model is an application design methodology that provides a
flexible way of creating cloud applications in a rapid, convenient and platform
independent manner.
• CCM is an architectural approach for cloud applications that is not tied to any
specific programming language or cloud platform.
• Cloud applications designed with CCM approach can have innovative hybrid
deployments in which different components of an application can be deployed on
cloud infrastructure and platforms of different cloud vendors.
• Applications designed using CCM have better portability and interoperability.
• CCM based applications have better scalability by decoupling application
components and providing asynchronous communication mechanisms.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
CCM Architecture Design
• In Architecture Design step, interactions
between the application components are
defined.
• CCM components have the following
characteristics:
• Loose Coupling
• Components in the Cloud Component Model are loosely
coupled.
• Asynchronous Communication
• By allowing asynchronous communication between
components, it is possible to add capacity by adding
additional servers when the application load increases.
Asynchronous communication is made possible by using
messaging queues.
• Stateless Design Architecture design of an e-Commerce application.
• Components in the Cloud Component Model are
stateless. By storing session state outside of the
component (e.g. in a database), stateless component
design enables distribution and horizontal scaling.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
CCM Component Design
• Cloud Component Model is created for the application
based on comprehensive analysis of the application’s
functions and building blocks.
• Cloud component model allows identifying the
building blocks of a cloud application which are
classified based on the functions performed and type
of cloud resources required.
• Each building block performs a set of actions to
produce the desired outputs for other components.
• Each component takes specific inputs, performs a pre-
defined set of actions and produces the desired CCM map for an e-Commerce application
outputs.
• Components offer their functions as services through
a functional interface which can be used by other
components.
• Components report their performance to a
performance database through a performance
interface.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
SOA vs CCM
Similarities
SOA CCM
Standardization & SOA advocates principles of CCM is based on reusable
Re-use reuse and well defined components which can be used
relationship between service by multiple cloud applications.
provider and service consumer.
Loose coupling SOA is based on loosely CCM is based on loosely
coupled services that minimize coupled components that
dependencies. communicate asynchronously
Statelessness SOA services minimize resource CCM components are stateless.
consumption by deferring the State is stored outside of the
management of state components.
information.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
CCM Deployment Design
• In Deployment Design step, application components
are mapped to specific cloud resources such as web
servers, application servers, database servers, etc.
• Since the application components are designed to be
loosely coupled and stateless with asynchronous
communication, components can be deployed
independently of each other.
• This approach makes it easy to migrate application
components from one cloud to the other.
• With this fiexibility in application design and
deployment, the application developers can ensure
that the applications meet the performance and cost
requirements with changing contexts.
Deployment design of an e-Commerce application
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Model View Controller
• Model View Controller (MVC) is a popular software design pattern for web applications.
• Model
• Model manages the data and the behavior of the applications. Model processes events sent by the controller. Model has no information about the views and controllers.
Model responds to the requests for information about its state (from the view) and responds to the instructions to change state (from controller).
• View
• View prepares the interface which is shown to the user. Users interact with the application through views. Views present the information that the model or controller tell
the view to present to the user and also handle user requests and sends them to the controller.
• Controller
• Controller glues the model to the view. Controller processes user requests and updates the model when the user manipulates the view. Controller also updates the view
when the model changes.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
SOA vs CCM
Differences
SOA CCM
End points SOA services have small and well-defined set CCM components have very large number of
of endpoints through which many types of endpoints. There is an endpoint for each resource
data can pass. in a component, identified by a URI.
Messaging SOA uses a messaging layer above HTTP by CCM components use HTTP and REST for
using SOAP which provide prohibitive messaging.
constraints to developers.
Security Uses WS-Security , SAML and other CCM components use HTTPS for security.
standards for security
Interfacing SOA uses XML for interfacing. CCM allows resources in components represent
different formats for interfacing (HTML, XML,
JSON, etc.).
Consumption Consuming traditional SOA services in a CCM components and the underlying component
browser is cumbersome. resources are exposed as XML, JSON (and other
formats) over HTTP or REST, thus easy to consume
in the browser.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Relational Databases
• A relational database is database that conforms to the relational model that was popularized by
Edgar Codd in 1970.
• The 12 rules that Codd introduced for relational databases include:
• Information rule
• Guaranteed access rule
• Systematic treatment of null values
• Dynamic online catalog based on relational model
• Comprehensive sub-language rule
• View updating rule
• High level insert, update, delete
• Physical data independence
• Logical data independence
• Integrity independence
• Distribution independence
• Non-subversion rule
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
RESTful Web Services
• Representational State Transfer (REST) is a set of architectural principles by which you can
design web services and web APIs that focus on a system’s resources and how resource
states are addressed and transferred.
• The REST architectural constraints apply to the components, connectors, and data
elements, within a distributed hypermedia system.
• A RESTful web service is a web API implemented using HTTP and REST principles.
• The REST architectural constraints are as follows:
• Client-Server
• Stateless
• Cacheable
• Layered System
• Uniform Interface
• Code on demand
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
ACID Guarantees
• Relational databases provide ACID guarantees.
• Atomicity
• Atomicity property ensures that each transaction is either “all or nothing”. An atomic transaction ensures that all parts
of the transaction complete or the database state is left unchanged.
• Consistency
• Consistency property ensures that each transaction brings the database from one valid state to another. In other
words, the data in a database always conforms to the defined schema and constraints.
• Isolation
• Isolation property ensures that the database state obtained after a set of concurrent transactions is the same as would
have been if the transactions were executed serially. This provides concurrency control, i.e. the results of incomplete
transactions are not visible to other transactions. The transactions are isolated from each other until they finish.
• Durability
• Durability property ensures that once a transaction is committed, the data remains as it is, i.e. it is not affected by
system outages such as power loss. Durability guarantees that the database can keep track of changes and can recover
from abnormal terminations.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Relational Databases
• Relations
• A relational database has a collection of relations (or tables). A relation is a set of tuples (or rows).
• Schema
• Each relation has a fixed schema that defines the set of attributes (or columns in a table) and the constraints on the attributes.
• Tuples
• Each tuple in a relation has the same attributes (columns). The tuples in a relation can have any order and the relation is not sensitive to
the ordering of the tuples.
• Attributes
• Each attribute has a domain, which is the set of possible values for the attribute.
• Insert/Update/Delete
• Relations can be modified using insert, update and delete operations. Every relation has a primary key that uniquely identifies each
tuple in the relation.
• Primary Key
• An attribute can be made a primary key if it does not have repeated values in different tuples.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Non-Relational Databases - Types
• Key-value store
• Key-value store databases are suited for applications that require storing unstructured data without a fixed schema.
Most key-value stores have support for native programming language data types.
• Document store
• Document store databases store semi-structured data in the form of documents which are encoded in different
standards such as JSON, XML, BSON, YAML, etc.
• Graph store
• Graph stores are designed for storing data that has graph structure (nodes and edges). These solutions are suitable for
applications that involve graph data such as social networks, transportation systems, etc.
• Object store
• Object store solutions are designed for storing data in the form of objects de?ned in an object-oriented programming
language.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Non-Relational Databases
• Non-relational databases (or popularly called No-SQL databases) are becoming popular with the growth of
cloud computing.
• Non-relational databases have better horizontal scaling capability and improved performance for big data at
the cost of less rigorous consistency models.
• Unlike relational databases, non-relational databases do not provide ACID guarantees.
• Most non-relational databases offer “eventual” consistency, which means that given a sufficiently long period
of time over which no updates are made, all updates can be expected to propagate eventually through the
system and the replicas will be consistent.
• The driving force behind the non-relational databases is the need for databases that can achieve high
scalability, fault tolerance and availability.
• These databases can be distributed on a large cluster of machines. Fault tolerance is provided by storing
multiple replicas of data on different machines.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Further Reading
• Roy T. Fielding, Richard N. Taylor, Principled Design of the Modern Web Architecture, ACM
Transactions on Internet Technology (TOIT), 2002.
• A. Bahga, V. Madisetti, Rapid Prototyping of Advanced Cloud-Based Systems, IEEE Computer, vol.
46, iss. 11, Nov 2013.
• E.F. Codd, A Relational Model of Data for Large Shared Data Banks, Communications of the ACM
13 (6): 377–387, 1970.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
 Outline
• Introduction to Python
• Installing Python
• Python Data Types & Data Structures
• Control Flow
• Functions
• Modules
• Packages
• File Input/Output
• Date/Time Operations
• Classes
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Chapter 6
Python Basics
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Python - Benefits
• Easy-to-learn, read and maintain
• Python is a minimalistic language with relatively few keywords, uses English keywords and has fewer syntactical constructions
as compared to other languages. Reading Python programs feels like English with pseudo-code like constructs. Python is easy
to learn yet an extremely powerful language for a wide range of applications.
• Object and Procedure Oriented
• Python supports both procedure-oriented programming and object-oriented programming. Procedure oriented paradigm
allows programs to be written around procedures or functions that allow reuse of code. Procedure oriented paradigm allows
programs to be written around objects that include both data and functionality.
• Extendable
• Python is an extendable language and allows integration of low-level modules written in languages such as C/C++. This is
useful when you want to speed up a critical portion of a program.
• Scalable
• Due to the minimalistic nature of Python, it provides a manageable structure for large programs.
• Portable
• Since Python is an interpreted language, programmers do not have to worry about compilation, linking and loading of
programs. Python programs can be directly executed from source
• Broad Library Support
• Python has a broad library support and works on various platforms such as Windows, Linux, Mac, etc.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Python
• Python is a general-purpose high level programming language and suitable for providing a solid
foundation to the reader in the area of cloud computing.
• The main characteristics of Python are:
• Multi-paradigm programming language
• Python supports more than one programming paradigms including object-oriented programming and structured
programming
• Interpreted Language
• Python is an interpreted language and does not require an explicit compilation step. The Python interpreter
executes the program source code directly, statement by statement, as a processor or scripting engine does.
• Interactive Language
• Python provides an interactive mode in which the user can submit commands at the Python prompt and interact
with the interpreter directly.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Numbers
• Numbers
• Number data type is used to store numeric values. Numbers are immutable data types, therefore changing the value of a number data
type results in a newly allocated object.
#Integer #Addition #Division
>>>a=5 >>>c=a+b >>>f=b/a
>>>type(a) >>>c >>>f
<type ’int’> 7.5 0.5
>>>type(c) >>>type(f)
#Floating Point <type ’float’> <type float’>
>>>b=2.5
>>>type(b) #Subtraction #Power
<type ’float’> >>>d=a-b >>>g=a**2
>>>d >>>g
#Long 2.5 25
>>>x=9898878787676L >>>type(d)
>>>type(x) <type ’float’>
<type ’long’>
#Multiplication
#Complex >>>e=a*b
>>>y=2+5j >>>e
>>>y 12.5
(2+5j) >>>type(e)
>>>type(y) <type ’float’>
<type ’complex’>
>>>y.real
2
>>>y.imag
5
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Python - Setup
• Windows
• Python binaries for Windows can be downloaded from http://www.python.org/getit .
• For the examples and exercise in this book, you would require Python 2.7 which can be directly downloaded from:
http://www.python.org/ftp/python/2.7.5/python-2.7.5.msi
• Once the python binary is installed you can run the python shell at the command prompt using
> python
• Linux
#Install Dependencies
sudo apt-get install build-essential
sudo apt-get install libreadline-gplv2-dev libncursesw5-dev libssl-dev libsqlite3-dev tk-dev libgdbm-dev libc6-dev libbz2-dev
#Download Python
wget http://python.org/ftp/python/2.7.5/Python-2.7.5.tgz
tar -xvf Python-2.7.5.tgz
cd Python-2.7.5
#Install Python
./configure
make
sudo make install
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Lists
• Lists
• List a compound data type used to group together other values. List items need not all have the same type. A list contains items
separated by commas and enclosed within square brackets.
#Mixed data types in a list
#Create List #Removing an item from a list >>>mixed=[’data’,5,100.1,8287398L]
>>>fruits=[’apple’,’orange’,’banana’,’mango’] >>>fruits.remove(’mango’) >>>type(mixed)
>>>type(fruits) >>>fruits <type ’list’>
<type ’list’> [’apple’, ’orange’, ’banana’, ’pear’] >>>type(mixed[0])
<type ’str’>
#Get Length of List #Inserting an item to a list >>>type(mixed[1])
>>>len(fruits) >>>fruits.insert(1,’mango’) <type ’int’>
4 >>>fruits >>>type(mixed[2])
[’apple’, ’mango’, ’orange’, ’banana’, ’pear’] <type ’float’>
#Access List Elements >>>type(mixed[3])
>>>fruits[1] #Combining lists <type ’long’>
’orange’ >>>vegetables=[’potato’,’carrot’,’onion’,’beans’,’r
>>>fruits[1:3] adish’] #Change individual elements of a list
[’orange’, ’banana’] >>>vegetables >>>mixed[0]=mixed[0]+" items"
>>>fruits[1:] [’potato’, ’carrot’, ’onion’, ’beans’, ’radish’] >>>mixed[1]=mixed[1]+1
[’orange’, ’banana’, ’mango’] >>>mixed[2]=mixed[2]+0.05
>>>eatables=fruits+vegetables >>>mixed
#Appending an item to a list >>>eatables [’data items’, 6, 100.14999999999999, 8287398L]
>>>fruits.append(’pear’) [’appl
>>>fruits e’, #Lists can be nested
[’apple’, ’orange’, ’banana’, ’mango’, ’pear’] ’mang >>>nested=[fruits,vegetables]
o’, >>>nested
’orang [[’apple’, ’mango’, ’orange’, ’banana’, ’pear’],
e’, [’potato’, ’carrot’, ’onion’, ’beans’, ’radish’]]
’banan
a’,
’pear’, ’potato’, ’carrot’, ’onion’, ’beans’, ’radish’]
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Strings
• Strings
• A string is simply a list of characters in order. There are no limits to the number of characters you can have in a string.
#Create string #Print string #strip: Returns a copy of the string with the
>>>s="Hello World!" >>>print s #leading and trailing characters removed.
>>>type(s) Hello World!
<type ’str’> >>>s.strip("!")
#Formatting output ’Hello World’
#String concatenation >>>print "The string (The string (Hello World!)
>>>t="This is sample program." has 12 characters
>>>r = s+t
>>>r #Convert to upper/lower case
’Hello World!This is sample program.’ >>>s.upper()
’HELLO WORLD!’
#Get length of string >>>s.lower()
>>>len(s) ’hello world!’
12
#Accessing sub-strings
#Convert string to integer >>>s[0]
>>>x="100" ’H’
>>>type(s) >>>s[6:]
<type ’str’> ’World!’
>>>y=int(x) >>>s[6:-1]
>>>y ’World’
100
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Dictionaries
• Dictionaries
• Dictionary is a mapping data type or a kind of hash table that maps keys to values. Keys in a dictionary can be of any data type, though
numbers and strings are commonly used for keys. Values in a dictionary can be any data type or object.
#Create a dictionary #Get all keys in a dictionary #Check if dictionary has a key
>>>student={’name’:’Mary’,’id’:’8776’,’major’:’CS’} >>>student.keys() >>>student.has_key(’name’)
>>>student [’gender’, ’major’, ’name’, ’id’] True
{’major’: ’CS’, ’name’: ’Mary’, ’id’: ’8776’} >>>student.has_key(’grade’)
>>>type(student) #Get all values in a dictionary False
<type ’dict’> >>>student.values()
[’female’, ’CS’, ’Mary’, ’8776’]
#Get length of a dictionary
>>>len(student) #Add new key-value pair
3 >>>student[’gender’]=’female’
>>>student
#Get the value of a key in dictionary {’gende
>>>student[’name’] r’: ’female’, ’major’: ’CS’, ’name’: ’Mary’, ’id’: ’8776’}
’Mary’
#A value in a dictionary can be another dictionary
#Get all items in a dictionary >>>student1={’name’:’David’,’id’:’9876’,’major’:’ECE’}
>>>student.items() >>>students={’1’: student,’2’:student1}
[(’gender’, ’female’), (’major’, ’CS’), (’name’, ’Mary’), >>>students
(’id’, ’8776’)] {’1’:
{’gende
r’: ’female’, ’major’: ’CS’, ’name’: ’Mary’, ’id’: ’8776’}, ’2’:
{’
major’: ’ECE’, ’name’: ’David’, ’id’: ’9876’}}
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Tuples
• Tuples
• A tuple is a sequence data type that is similar to the list. A tuple consists of a number of values separated by commas and enclosed
within parentheses. Unlike lists, the elements of tuples cannot be changed, so tuples can be thought of as read-only lists.
#Create a Tuple #Get an element from a tuple
>>>fruits=("apple","mango","banana","pineapple") >>>fruits[0]
>>>fruits ’apple’
(’apple’, ’mango’, ’banana’, ’pineapple’) >>>fruits[:2]
(’apple’, ’mango’)
>>>type(fruits)
<type ’tuple’> #Combining tuples
>>>vegetables=(’potato’,’carrot’,’onion’,’radish’)
#Get length of tuple >>>eatables=fruits+vegetables
>>>len(fruits) >>>eatables
4 (’apple’, ’mango’, ’banana’, ’pineapple’, ’potato’, ’carrot’, ’onion’, ’radish’)
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Control Flow – if statement
• The if statement in Python is similar to the if statement in other languages.
>>>a = 25**5 >>>if a>10000: >>>s="Hello World" >>>student={’name’:’Mary’,’id’:’8776’}
>>>if a>10000: if a<1000000: >>>if "World" in s: >>>if not student.has_key(’major’):
print "More" print "Between 10k and 100k" s=s+"!" student[’major’]=’CS’
else: else: print s
print "Less" print "More than 100k" >>>student
elif a==10000: Hello World! {’major’: ’CS’, ’name’: ’Mary’, ’id’: ’8776’}
More print "Equal to 10k"
else:
print "Less than 10k"
More than 100k
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Type Conversions
• Type conversion examples
#Convert to string #Convert to long
>>>a=10000 >>>long(b)
>>>str(a) 2013L
’10000’
#Convert to list
#Convert to int >>>s="aeiou"
>>>b="2013" >>>list(s)
>>>int(b) [’a’, ’e’, ’i’, ’o’, ’u’]
2013
#Convert to set
#Convert to float >>>x=[’mango’,’apple’,’banana’,’mango’,’banana’]
>>>float(b) >>>set(x)
2013.0 set([’mango’, ’apple’, ’banana’])
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Control Flow – while statement
• The while statement in Python executes the statements within the while loop as long as the while condition is
true.
#Prints even numbers upto 100
>>> i = 0
>>> while i<=100:
if i%2 == 0:
print i
i = i+1
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Control Flow – for statement
• The for statement in Python iterates over items of any sequence (list, string, etc.) in the order in which they
appear in the sequence.
• This behavior is different from the for statement in other languages such as C in which an initialization,
incrementing and stopping criteria are provided.
#Looping over characters in a string #Looping over items in a list #Looping over keys in a dictionary
helloString = "Hello World" fruits=[’apple’,’orange’,’banana’,’mango’] student
=
for c in helloString: i=0 ’nam
print c for item in fruits: e’:
print "Fruit-%d: %s" % (i,item) ’Mar
i=i+1 y’, ’id’: ’8776’,’gender’: ’female’, ’major’: ’CS’
for key in student:
print "%s: %s" % (key,student[key]
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Control Flow – break/continue statements
• The break and continue statements in Python are similar to the statements in C.
• Break #Break statement example
• Break statement breaks out of the for/while loop >>>y=1
>>>for x in range(4,256,4):
y=y*x
if y > 512:
break
print y
4
32
384
• Continue #Continue statement example
• Continue statement continues with the next iteration. >>>fruits=[’apple’,’orange’,’banana’,’mango’]
>>>for item in fruits:
if item == "banana":
continue
else:
print item
apple
orange
mango
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Control Flow – range statement
• The range statement in Python generates a list of numbers in arithmetic progression.
#Generate a list of numbers from 0 – 9 #Generate a list of numbers from 10 - 100 with increments
of 10
>>>range (10)
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9] >>>range(10,110,10)
[10, 20, 30, 40, 50, 60, 70, 80, 90,100]
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Functions
• A function is a block of code that takes information in (in the form of students = { '1': {'name': 'Bob', 'grade': 2.5},
parameters), does some computation, and returns a new piece of '2': {'name': 'Mary', 'grade': 3.5},
'3': {'name': 'David', 'grade': 4.2},
information based on the parameter information. '4': {'name': 'John', 'grade': 4.1},
'5': {'name': 'Alex', 'grade': 3.8}}
• A function in Python is a block of code that begins with the keyword
def averageGrade(students):
def followed by the function name and parentheses. The function “This function computes the average grade”
parameters are enclosed within the parenthesis. sum = 0.0
for key in students:
sum = sum + students[key]['grade']
average = sum/len(students)
• The code block within a function begins after a colon that comes after return average
the parenthesis enclosing the parameters.
avg = averageGrade(students)
print "The average garde is: %0.2f" % (avg)
• The first statement of the function body can optionally be a
documentation string or docstring.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Control Flow – pass statement
• The pass statement in Python is a null operation.
• The pass statement is used when a statement is required syntactically but you do not want any command or
code to execute.
>fruits=[’apple’,’orange’,’banana’,’mango’]
>for item in fruits:
if item == "banana":
pass
else:
print item
apple
orange
mango
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Functions - Passing by Reference
• All parameters in the Python functions are passed by reference.
• If a parameter is changed within a function the change also reflected back in the calling function.
>>>def displayFruits(fruits):
print "There are %d fruits in the list" % (len(fruits))
for item in fruits:
print item
print "Adding one more fruit"
fruits.append('mango')
>>>fruits = ['banana', 'pear', 'apple']
>>>displayFruits(fruits)
There are 3 fruits in the list
banana
pear
apple
#Adding one more fruit
>>>print "There are %d fruits in the list" % (len(fruits))
There are 4 fruits in the list
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Functions - Default Arguments
• Functions can have default values of the parameters.
• If a function with default values is called with fewer parameters or without any parameter, the default values of the
parameters are used
>>>def displayFruits(fruits=[’apple’,’orange’]):
print "There are %d fruits in the list" % (len(fruits))
for item in fruits:
print item
#Using default arguments
>>>displayFruits()
apple
orange
>>>fruits = [’banana’, ’pear’, ’mango’]
>>>displayFruits(fruits)
banana
pear
mango
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Functions - Variable Length Arguments
• Python functions can have variable length arguments. The variable length arguments are passed to as a tuple to the
function with an argument prefixed with asterix (*)
>>>def student(name, *varargs):
print "Student Name: " + name
for item in varargs:
print item
>>>student(’Nav’)
Student Name: Nav
>>>student(’Amy’, ’Age: 24’)
Student Name: Amy
Age: 24
>>>student(’Bob’, ’Age: 20’, ’Major: CS’)
Student Name: Bob
Age: 20
Major: CS
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Functions - Keyword Arguments
• Functions can also be called using keyword arguments that identifies the arguments by the parameter name when the
function is called.
>>>def #Correct use #name is a formal argument.
printStudentRecords(name,age=20,major=’CS’): >>>printStudentRecords(name=’Alex’) #**kwargs is a keyword argument that receives all
print "Name: " + name Name: Alex arguments except the formal argument as a
print "Age: " + str(age) Age: 20 dictionary.
print "Major: " + major Major: CS
>>>def student(name, **kwargs):
#This will give error as name is required argument >>>printStudentRecords(name=’Bob’,age=22,major=’EC print "Student Name: " + name
>>>printStudentRecords() E’) for key in kwargs:
Traceback (most recent call last): Name: Bob print key + ’: ’ + kwargs[key]
File "<stdin>", line 1, in <module> Age: 22
TypeError: printStudentRecords() takes at least 1 Major: ECE >>>student(name=’Bob’, age=’20’, major = ’CS’)
argument (0 given) Student Name: Bob
>>>printStudentRecords(name=’Alan’,major=’ECE’) age: 20
Name: Alan major: CS
Age: 20
Major: ECE
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Packages
• Python package is hierarchical file structure that consists of # skimage package listing
modules and subpackages. skimage/ Top level package
__init__.py Treat directory as a package
• Packages allow better organization of modules related to a single
application environment. color/ color color subpackage
__init__.py
colorconv.py
colorlabel.py
rgb_colors.py
draw/ draw draw subpackage
__init__.py
draw.py
setup.py
exposure/ exposure subpackage
__init__.py
_adapthist.py
exposure.py
feature/ feature subpackage
__init__.py
_brief.py
_daisy.py
...
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Modules
#student module - saved as student.py #Using student module
• Python allows organizing the program def averageGrade(students): >>>import student
code into different modules which sum = 0.0 >>>students = '1': 'name': 'Bob', 'grade': 2.5,
improves the code readability and for key in students: '2': 'name': 'Mary', 'grade': 3.5,
management. sum = sum + students[key]['grade'] '3': 'name': 'David', 'grade': 4.2,
average = sum/len(students) '4': 'name': 'John', 'grade': 4.1,
'5': 'name': 'Alex', 'grade': 3.8
• A module is a Python file that defines return average
some functionality in the form of functions def printRecords(students): >>>student.printRecords(students)
or classes. print "There are %d students" %(len(students)) There are 5 students
i=1 Student-1:
• Modules can be imported using the import for key in students: Name: Bob
Grade: 2.5
keyword. print "Student-%d: " % (i)
Student-2:
print "Name: " + students[key]['name']
print "Grade: " + str(students[key]['grade']) Name: David
• Modules to be imported must be present i = i+1 Grade: 4.2
in the search path. Student-3:
Name: Mary
# Importing a specific function from a module Grade: 3.5
>>>from student import averageGrade Student-4:
Name: Alex
# Listing all names defines in a module Grade: 3.8
>>>dir(student) Student-5:
Name: John
Grade: 4.1
>>>avg = student. averageGrade(students)
>>>print "The average garde is: %0.2f" % (avg)
3.62
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
File Handling
# Example of reading a certain number of bytes # Example of seeking to a certain position
>>>fp = open('file.txt','r') >>>fp = open('file.txt','r')
>>>fp.read(10) >>>fp.seek(10,0)
'Python sup' >>>content = fp.read(10)
>>>fp.close() >>>print content
ports more
>>>fp.close()
# Example of getting the current position of read # Example of writing to a file
>>>fp = open('file.txt','r') >>>fo = open('file1.txt','w')
>>>fp.read(10) >>>content='This is an example of writing to a file in
'Python sup' Python.'
>>>currentpos = fp.tell >>>fo.write(content)
>>>print currentpos >>>fo.close()
<built-in method tell of file object at 0x0000000002391390>
>>>fp.close()
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
File Handling
• Python allows reading and writing to files using the file # Example of reading an entire file
object. >>>fp = open('file.txt','r')
>>>content = fp.read()
>>>print content
This is a test file.
• The open(filename, mode) function is used to get a file >>>fp.close()
object.
# Example of reading line by line
>>>fp = open('file1.txt','r')
• The mode can be read (r), write (w), append (a), read and >>>print "Line-1: " + fp.readline()
write (r+ or w+), read-binary (rb), write-binary (wb), etc. Line-1: Python supports more than one programming paradigms.
>>>print "Line-2: " + fp.readline()
Line-2: Python is an interpreted language.
>>>fp.close()
• After the file contents have been read the close function is
called which closes the file object. # Example of reading lines in a loop
>>>fp = open(’file1.txt’,’r’)
>>>lines = fp.readlines()
>>>for line in lines:
print line
Python supports more than one programming paradigms.
Python is an interpreted language.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Classes
• Python is an Object-Oriented Programming (OOP) language. Python provides all the standard features of Object
Oriented Programming such as classes, class variables, class methods, inheritance, function overloading, and
operator overloading.
• Class
• A class is simply a representation of a type of object and user-defined prototype for an object that is composed of three things: a name,
attributes, and operations/methods.
• Instance/Object
• Object is an instance of the data structure defined by a class.
• Inheritance
• Inheritance is the process of forming a new class from an existing class or base class.
• Function overloading
• Function overloading is a form of polymorphism that allows a function to have different meanings, depending on its context.
• Operator overloading
• Operator overloading is a form of polymorphism that allows assignment of more than one function to a particular operator.
• Function overriding
• Function overriding allows a child class to provide a specific implementation of a function that is already provided by the base class. Child class
implementation of the overridden function has the same name, parameters and return type as the function in the base class.
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Date/Time Operations
• Python provides several functions for date and time access and conversions.
• The datetime module allows manipulating date and time in several ways.
• The time module in Python provides various time-related functions.
# Examples of manipulating with date # Examples of manipulating with time
>>>from datetime import date >>>import time
>>>nowtime = time.time()
>>>now = date.today() >>>time.localtime(nowtime)
>>>print "Date: " + now.strftime("%m-%d-%y") time.struct_time(tm_year=2013, tm_mon=7, tm_mday=24, tm_ec=51, tm_wday=2, tm_yday=205,
Date: 07-24-13 tm_isdst=0)
>>>print "Day of Week: " + now.strftime("%A") >>>time.asctime(time.localtime(nowtime))
Day of Week: Wednesday 'Wed Jul 24 16:14:51 2013'
>>>print "Month: " + now.strftime("%B") >>>time.strftime("The date is %d-%m-%y. Today is a %A. It is %H hours, %M minutes and %S seconds now.")
Month: July 'The date is 24-07-13. Today is a Wednesday. It is 16 hours, 15 minutes and 14 seconds now.'
>>>then = date(2013, 6, 7)
>>>timediff = now - then
>>>timediff.days
47
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
 Class Inheritance
• In this example Shape is the base class and Circle is the derived class. The class Circle inherits the attributes of the Shape class.
• The child class Circle overrides the methods and attributes of the base class (eg. draw() function defined in the base class Shape is overridden in child
class Circle).
# Examples of class inheritance class Circle(Shape): class Point: >>>p = Point(2,4)
class Shape: def __init__(self, c,r): def __init__(self, x, y): >>>circ = Circle(p,7)
def __init__(self): print "Child class constructor" self.xCoordinate = x Child class constructor
print "Base class constructor" self.center = c self.yCoordinate = y >>>circ.getColor()
self.color = ’Green’ self.radius = r ’Green’
self.lineWeight = 10.0 self.color = ’Green’ def setXCoordinate(self,x): >>>circ.setColor(’Red’)
self.lineWeight = 10.0 self.xCoordinate = x >>>circ.getColor()
def draw(self): self.__label = ’Hidden circle label’ ’Red’
print "Draw - to be implemented" def getXCoordinate(self): >>>circ.getLineWeight()
def setColor(self, c): def setCenter(self,c): return self.xCoordinate 10.0
self.color = c self.center = c >>>circ.getCenter().getXCoordinate()
def getColor(self): def getCenter(self): def setYCoordinate(self,y): 2
return self.color return self.center self.yCoordinate = y >>>circ.getCenter().getYCoordinate()
4
def setLineWeight(self,lwt): def setRadius(self,r): def getYCoordinate(self): >>>circ.draw()
self.lineWeight = lwt self.radius = r return self.yCoordinate Draw Circle (overridden function)
>>>circ.radius
def getLineWeight(self): def getRadius(self): 7
return self.lineWeight return self.radius
def draw(self):
print "Draw Circle (overridden function)"
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Class Example
• The variable studentCount is a # Examples of a class >>>s = Student(’Steve’,’98928’)
class Student: Constructor called
class variable that is shared by studentCount = 0
all instances of the class >>>s.addGrade(’Math’,’90’)
Student and is accessed by def __init__(self, name, id): >>>s.addGrade(’Physics’,’85’)
print "Constructor called" >>>s.printGrades()
Student.studentCount. self.name = name Physics: 85
self.id = id Math: 90
• The variables name, id and Student.studentCount = Student.studentCount + 1
grades are instance variables self.grades={} >>>mathgrade = s.getGrade(’Math’)
which are specific to each >>>print mathgrade
def __del__(self): 90
instance of the class. print "Destructor called"
>>>count = s.getStudentCount()
• There is a special method by def getStudentCount(self): >>>print count
the name __init__() which is return Student.studentCount 1
the class constructor.
def addGrade(self,key,value): >>>del s
• The class constructor self.grades[key]=value Destructor called
def getGrade(self,key):
initializes a new instance return self.grades[key]
when it is created. The
function __del__() is the class def printGrades(self):
for key in self.grades:
destructor print key + ": " + self.grades[key]
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014
Further Reading
• Code Academy Python Tutorial, http://www.codecademy.com/tracks/python
• Google's Python Class, https://developers.google.com/edu/python/
• Python Quick Reference Cheat Sheet, http://www.addedbytes.com/cheat-sheets/python-cheat-sheet/
• PyCharm Python IDE, http://www.jetbrains.com/pycharm/
Book website: www.cloudcomputingbook.info Bahga & Madisetti, © 2014