Distributed

Systems
CPSC 5520
Kevin Lundeen
Naming
 What is naming?

• Names are used to denote entities in a distributed

system
• To operate on an entity, we need to access it at an
access point
• Access points are entities that are named by means
of an address
• IP endpoint
• We also have location-independent names
• Extra level of abstraction between entity and
address(es)
Naming
Name Characteristics

• Identifiers have 1-to-1 relationship with entity

• Invariant over time, too

Name Architectures

• Flat versus hierarchical

• Content in Names
• Human-Readable
• Attribute-Based Naming
 Flat Namespaces
Naming Hierarchical
Namespaces
Schemes Attribute-Based
Lookups
Types of Naming Schemes:
Hierarchical Names
• Hierarchically named entities are leaves in a tree
• The name of an entity is a path to a leaf from the root
• Doesn’t have to exactly be a tree: can be a directed graph with distinguished
”root” and “leaves”
• Typically human-readable, but not necessarily
• Examples:
• Homogeneous:
• DNS host names: fubar.garble.edu
• Homogeneity hiding heterogeneous resources (mount points):
• File names in a distributed directory structure:
/mnt6/eduedu/students/warp.php
• Heterogeneous:
• URL: scheme:[//authority]path[?query][#fragment]
http://fubar.garble.edu/top/students/warp.php?q=6#footnotes

• Implemented as multi-tier systems

• Each component in name can correspond to a service
Types of Naming Schemes:
Flat Names
• Flat named entities are all at the same level:
• the name in its entirety is used as a key to the resource
• Typically not human-readable
7CCA8A82-D1E0-4455-A416-E0EAEEDBE9E4
• Or partially human-readable
O GBJP CE 127.3400 F37AC207
• Examples:
• UUID
• Product ID
• Phone book (thinking of the name in last, first format)˜
• Implemented with:
• Distributed Hash Tables (DHT)
• Disruption-insensitive bucket schemes like consistent hashing
• Hierarchical overlays and other schemes
Types of Naming Schemes:
Attribute-Based Naming
• Kind of shoehorns the concept of naming for an indexed lookup
based on attributes
• Think SQL where we are looking for a particular row in a large table
• We can specify predicates on various properties of the row to narrow down
the selection of qualifying rows
• Similarly in distributed systems, if we want to find an entity or set of entities
with certain attributes (or more generally, with certain predicates being true)
• Example
• LDAP
search("(C=NL)(O=VU University)(OU=*)(CN=Main
server)")
• Implemented as a distributed index or set of indices
• Can use well-studied query optimization techniques from DBMS research
 DNS
Domain
Name
System

Hierarchical
Namespace
Example
DNS
Domain Name System
One of the most successful and longest-lived distributed system

• Distributed dictionary • Features

• Key: Hierarchical host name • Convenience, like a phone book, to
• Value: IP address(es) and other have human-friendly host names
related info • Hierarchical namespace using
domains
• Part of the implementation
• But not required by IP • But also pleasing to humans
• People have been dialing phone • Location and administrative
numbers for 100 years; could transparency
equally do it for IP addresses
without DNS

desktop.seattleu.edu.
Before DNS
• Talk to Jake Feinler at Stanford
Research Institute! (Actually talk…)
• In the 1970’s, she’d add your host
name and its IP address to her
HOSTS.TXT file
• Don’t have an IP address? Talk to Jon
Postel at USC
• Then folks would occasionally take
the updated version via file sharing
(ftp)
• Or they might put their own name
for your server in their HOSTS.TXT
• There were no domains
By Elizabeth Feinler - Elizabeth Feinler (subject), CC BY-SA 3.0,

• She pioneered the idea of domains, https://commons.wikimedia.org/w/index.php?curid=14983224

including .EDU
A colleague of Postel’s, Paul Mockapetris, invented DNS in early 1980’s
DNS Conceptual Architecture

• Name
• Domain labels delimited by dots
• Labels are composed of Roman letters, digits, and hyphen (up to 63 characters)
• Meant to be human-meaningful
• Can use “punycode” encoding to allow general unicode for “internationalized domain
names”
• Read from right to left, the dot signifies a “within” relation with its parent
domain
• Final (often-omitted) dot connects to the root domain
• Last (right-most) domain is a top-level domain (TLD) and intended to be one
of a sparse set: edu, org, com, mil, gov, etc.
• Subdomain of arbitrary depth is to the left of TLD
• Limited to max 253 characters including the dots
 DNS
Conceptual
Architecture
(continued)

• Each domain corresponds

(conceptually) to a single host
• So all domains together
represent a giant tree
structure
• Each host has a list of
and knowledge of it child
subdomains
• Resolver
• The querying node/program
• Iteratively asks each
server starting at root for
the location of the
desired child
 • Root, which knows about all TLDs, is replicated in thirteen different
domains.
DNS Actual • a.root-servers.net, b.root-servers.net, …, m.root-servers.net
• (see root-servers.org)
Architecture • And each of these is replicated (with the same IP address) hundreds
of times
• TLDs are typically registries
• Multiple layers below TLD can be combined on a single node as desired
DNS Actual Architecture
(cont.)
• Administrative architecture is also distributed.
• Root is administered (indirectly) by ICANN, including establishing new TLDs
• Each TLD is administered (typically) by a registrar
• For example, the .com TLD is administered by Verisign
• (Verisign also handles .edu, .gov, .net, and many more TLDs)
• Each subdomain is administered by the registrant
• Their authoritative server for their domain

• Resolvers do sometimes work iteratively, but also:

• Use local cache
• Time to live part of response from a name server
• Use recursive resolving service
• provided by an ISP, for example
DNS: Design Evaluation

• How does DNS stack up with our 4-prong distributed system design
criteria from Chapter 1?
1. Support sharing of resources
 Gives a distributed database of most hosts on the internet
2. Distribution transparency
 Resolver seems to ”just work” for typical user
 Access, Location, Relocation, Replication, Concurrency Transparency
 (What about migration and failure transparency?)
3. Openness
 Very open framework. Note incorporation into many other protocols (e.g. HTTP)
4. Scalability
 Size Scalable: Over a billion named hosts and counting
 Geographically Scalable: Worldwide
 Administratively Scalable: ICANN, registries, local administration
 DNS
Walkthrough
Application wants to know IP
address of fu.bar.goober.com

hsd1.wa.comcast.net
75.75.75.75
ISP’s Recursive Name
Local resolver Server
(sponsored
by O/S)

Local
Local /etc/ cache
cache resolve.conf

names.goober.com a.gtld-servers.net g.root-server.net

107.14.5.0 192.5.6.30 67.172.255.165*
Authoritative server Proximal replica of Proximal replica of
for goober.com one of the TLD name one of the 13 root
domain servers for .com servers
*known a priori
Other DNS Topics

• Transport Protocol
• Usually UDP
• But TCP is used to address larger packets, security, etc.
• Failures
• Security
• Reverse lookup
• Have IP, but want to know corresponding host name
• Often done through tricky special domains
35.105.184.in-addr.arpa
• Blacklisting
• Similar idea
 Distributed
Hash
Tables

Typical Flat
Namespace
The DHT Implementation
Hashing

• In the typical data structure curriculum

• we moved from lists to trees
• then to hash tables
• Similarly, in distributed naming schemes
• we move from rings to hierarchies
• then to DHT’s
• DHT’s are used on flat namespaces
• Overlay topology is P2P
• Some schemes have a bucket table in a distinguished peer
• likely replicated for load balancing, failure, and recovery
• Other schemes treat all the peers identically
• On these, you need to know at least one peer to make a
query
• Each hash bucket is hosted on a peer
• A peer usually hosts many buckets
Let’s Review Hash Tables
• Idea is to get query time (and also fast insert/delete)
• Have a hash function, , which maps a key to an hash code, which is just a bit
string of say of m bits
• Hash function must be
• Constant time
• Uniformly distributed
• Idempotent (no side effects; always returns the same thing for a given key)
• Have a bucket mapper, , that maps the hash code to a bucket, usually a small
non-negative integer
• Buckets are usually required to be “small” if you want guaranteed
• Have constant time retrieval of a bucket from our scheme
• For in-memory hash tables, this is usually just an array of buckets or bucket pointers
• So a query, , algorithm is:
1. Calculate bucket number,
2. Retrieve this bucket
3. Look through the objects in this bucket for one with an equivalent key
Naïve Hash Tables

• Hash function:
• Just something convenient
• Most OO languages require a hash() method and provide some decent ones
• Many hash functions are poorly distributed
• “Use the memory address” – aligned on 8-byte boundaries and within a very small range
• “Use a checksum” – empty and small objects clump up
• Bucket mapper:
• Usually just modulo bucket count
• This is a great workaround for poorly distributed hash functions!
• When buckets get too full?
• Require a priori knowledge of size
• Or, every once in a while, dramatically increase the bucket count
• With the modulo bucket mapper, this means nearly every object moves to a new bucket
Naïve Hash Tables
h(key) % 7 == 0
Bucket
Address hashes: 0, 7, 14, 21, 28, …
Table
[0] h(key) % 7 == 1
[1]
hashes: 1, 8, 15, 22, 29, …
[2]
[3]
[4] h(key) % 7 == 2
[5] hashes: 2, 9, 16, 23, 30, …
[6]

h(key) % 7 == 6

hashes: 6, 13, 20, 27, 35, …

Naïve Hash Tables – Resize!
h(key) % 11 == 0
Bucket
Address hashes: 0, 11, 22, 33, 44, …, 77, …
Table
[0] h(key) % 11 == 1
[1]
hashes: 1, 12, 23, 34, 45, …, 78, …
[2]
[3]
[4] h(key) % 11 == 2
[5] hashes: 2, 13, 24, 35, 46, …, 79, …
[6]
[7] • Almost every object moves to a different
[8] …
bucket!
[9] • But that’s the price we pay to
[10]
compensate for our poor hash
function
h(key) % 11 == 10

hashes: 10, 21, 32, 43, 54, …, 87, …

 Good So why not a
idea!
better hash
function?
• Message digest algorithms (MD5,
SHA-1, etc.)
• Are very uniform, even for small
messages, even for large hash code
domains (like 512-bits)
• Are constant time and relatively easy
to compute
• Are idempotent
• Also have some additional
properties, suitable for cryptography
• infeasible to generate an object with
same hash
Then “rinse and repeat”
• infeasible to find two distinct objects
with the same hash
• avalanche effect – small object changes
make big hash code changes
CC BY-SA 2.5, https://commons.wikimedia.org/w/index.php?curid=1446602
Better Hash Tables
0 ≤ sha1(key) < 220
Bucket
Address hashes: 0, 1, 2, …, 220-1
Table
[0] 220 ≤ sha1(key) < 240
[1]
hashes: 220, 220 + 1, 220 + 2, …, 240 - 1
[2]
[3]
[4] 240 ≤ sha1(key) < 260
[5] hashes: 240, 240 + 1, 240 + 2, …, 260 - 1
[6]
[7]
…

2140 ≤ sha1(key) < 2160

hashes: 2140, 2140 + 1, …, 2160 - 1

Better Hash Tables – Resize!
0 ≤ sha1(key) < 210
Bucket
Address hashes: 0, 1, 2, …, 210-1
Table
[0] 210 ≤ sha1(key) < 220
[1]
hashes: 210, 210 + 1, 210 + 2, …, 220 - 1
[2]
[3]
[4] 220 ≤ sha1(key) < 230 (use the bucket[1] from before)
[5] hashes: 220, 220 + 1, 220 + 2, …, 230 - 1
[6]
[7]
[8] …
[9] • Even doubling number of buckets leaves
[10] half the objects in the same buckets!
[11]
[12]
[13] 2150 ≤ sha1(key) < 2160
[14]
[15] hashes: 2150, 2150 + 1, …, 2160 - 1
Consistent Hashing
Karger, et al 1997

• Each bucket has a key, too, and is similarly hashed to find where it
belongs
• Each bucket is responsible for the range of hash codes from its own hash up
to the next greater one
• Next one is called the successor, for the bucket which hashes to h,
• This allows us to add any new bucket, anytime
• just move any objects from predecessor bucket with hash codes ≥ new
bucket’s hash code
• Usually depicted and modeled as a circle of buckets
• The starting range (0 to the first bucket) is given to the last bucket
• Removing a node is similarly easy
• But if it just fails without warning, well then we need some scheme for the
objects to migrate even though their bucket is dead
• Usually done with replicas in adjacent nodes
Consistent
Hashing
 2160 0
bu
ck
et
2 147
+9
92

bucket 2
5
3

bu

bu
ck

ck
e

et
bucket 281 + 17
t21

53

2
03
DHT Overlays

• Better bucket management is not enough if we want true P2P

architecture.
• No one node knows all the others
• Everyone has to know at least one other
• And a querier also needs to know at least one peer in the network
• Architectures like Chord† attempt to make it efficient to find the
node which has the desired bucket
• Finger tables: exponential spatial indices
• Every node knows succ(h+1), succ(h+2), succ(h+4), … + succ(h+2m)

†Stoica, et al, 2001

 DHT
Dances
Follow along in the dance handout
Hash function for playing cards

10:49
:48
:47
:46
The End of Naming!