BASICS OF DISTRIBUTED SYSTEMS

NETWORK FILE SYSTEMS

ACCESSING FILES

 File sharing with socket-based programs

 HTTP, FTP, telnet:
 Explicit access
 User-directed connection to access remote resources

 We want more transparency

 Allow user to access remote resources just as local ones

 NAS: Network Attached Storage

FILE SERVICE MODELS
 Upload/Download model  Remote access model
 Read file: copy file from server to client  File service provides functional interface:
 Write file: copy file from client to server  create, delete, read bytes, write bytes, etc…
 Advantage  Advantages
 Simple  Client gets only what’s needed
 Problems  Server can manage coherent view of file system
 Wasteful: what if client needs small piece?  Problem
 Problematic: what if client doesn’t have enough  Possible server and network congestion
space?
 Servers are accessed for duration of file access
 Consistency: what if others need to modify the
same file?  Same data may be requested repeatedly
SEMANTICS OF FILE SHARING
 Sequential Semantics  Session Semantics
 Read returns result of last write  Relax the rules
 Easily achieved if  Changes to an open file are initially visible only
to the process (or machine) that modified it.
 Only one server
 Need to hide or lock file under modification
 Clients do not cache data from other clients
 BUT  Last process to modify the file wins.
 Performance problems if no cache
 Clients get obsolete data
 We can write-through
 Must notify all clients holding copies
 Requires extra state, generates extra traffic
REMOTE FILE SERVICE

 File Directory Service

 Maps textual names for file to internal locations that can be used by file service

 File service
 Provides file access interface to clients

 Client module (driver)

 Client-side interface for file and directory service
 If done right, helps provide access transparency
 E.g., implement the file system under the VFS layer
SYSTEM DESIGN ISSUES

 Transparency
 Integrated into OS or access via APIs?
 Consistency
 What happens if more than one user accesses the same file?
 What if files are replicated across servers?
 Security
 The local OS is no longer in charge
 Reliability
 What happens when the server or client dies?
 State
 Should the server keep track of clients between requests?
ACCESSING REMOTE FILES

 For maximum transparency, implement the client module as a file system type under VFS
STATEFUL OR STATELESS DESIGN?
 Stateless
 Stateful
 Server maintains no information on client
 Server maintains client-specific state
accesses
 Shorter requests
 Each request must identify file and offsets
 Better performance in processing requests
 Server can crash and recover
 Cache coherence is possible
 No state to lose
 Server can know who’s accessing what
 Client can crash and recover
 File locking is possible
 No open/close needed
 They only establish state
 No server space used for state
 Don’t worry about supporting many clients
 Problems if file is deleted on server
 File locking not possible
CACHING

 Hide latency to improve performance for repeated accesses

 File data can reside in several places
 Server’s disk
 Server’s buffer cache
 Client’s buffer cache
risk of cache consistency
 Client’s disk problems across multiple systems
APPROACHES TO CACHING

 Write-through
 What if another client reads its own (out-of-date) cached copy?
 All accesses will require checking with server
 Or … server maintains state and sends invalidations
 Delayed writes (write-behind)
 Data can be buffered locally (watch out for consistency – others won’t see updates!)
 Remote files updated periodically
 One bulk wire is more efficient than lots of little writes
 Problem: semantics become ambiguous
APPROACHES TO CACHING

 Read-ahead (prefetch)
 Request chunks of data before it is needed
 Minimize wait times if that data is later needed

 Write on close
 Admit that we have session semantics

 Centralized control
 Keep track of who has what open and cached on each node
 Stateful file system with signaling traffic
NFS
NETWORK FILE SYSTEM
SUN MICROSYSTEMS
NFS DESIGN GOALS

 Any machine can be a client or server

 Must support diskless workstations
 Device files refer back to local drivers
 Heterogeneous systems
 Access transparency: normal file system calls
 Recovery from failure:
 Stateless, UDP, client retries
 Stateless → no locking!
 High Performance
 use caching and read-ahead
VFS ON CLIENT; SERVER ACCESSES LOCAL FILE SYSTEM
MOUNTING
 Server: edit /etc/exports
 Client: mount fluffy:/users/paul /home/paul

 Send pathname to server

 Request permission to access contents
 Server returns file handle

 File device #, inode #, instance #

DIRECTORY AND FILE ACCESS PROTOCOL
 NFS has 16 functions
 (version 2; six more added in version 3)
PROBLEMS WITH NFS
 File consistency
 Assumes clocks are synchronized
 Open with append cannot be guaranteed to work
 getattr & write(offset) are separate operations
 Locking cannot work
 Separate lock manager added (but this adds stateful behavior)
 No reference counting of open files at the server
 You can delete a file that you (or others) have open!
 File permissions may change
 Invalidating access to file
 Global UID space assumed
 No encryption or authentication
 Requests via unencrypted RPC
 Authentication were later methods available
 Diffie-Hellman, Kerberos, Unix-style
 Rely on user-level software to encrypt
AUTOMOUNTER
AFS
ANDREW FILE SYSTEM
CARNEGIE MELLON UNIVERSITY (1986(V2), 1989(V3))
AFS

 Design Goal
 Support information sharing on a large scale e.g., 10,000+ clients

 History
 Became a commercial spin-off: Transarc
 IBM acquired Transarc
 Open source under IBM Public License
 OpenAFS (openafs.org)
AFS DESIGN ASSUMPTIONS

 Most files are small

 Reads are more common than writes
 Most files are accessed by one user at a time
 Files are referenced in bursts (locality)
 Once referenced, a file is likely to be referenced again
AFS DESIGN DECISIONS

 Whole file serving

 Send the entire file on open

 Whole file caching

 Client caches entire file on local disk
 Client writes the file back to server on close
 if modified
 keeps cached copy for future accesses
AFS DESIGN

 Each client has an AFS disk cache

 Part of disk devoted to AFS (e.g. 100 MB)
 Client manages cache in LRU manner

 Clients communicate with set of trusted servers

 Each server presents one identical name space to clients
 All clients access it in the same way
 Location transparent
AFS SERVER: CELLS

 Servers are grouped into administrative entities called cells

 Cell: collection of
 Servers
 Administrators
 Users
 Clients

 Each cell is autonomous but cells may cooperate and present users with
one uniform name space
AFS SERVER: VOLUMES

 Disk partition contains

 file and directories grouped into volumes

 Volume
 Administrative unit of organization
 E.g., user’s home directory, local source, etc.
 Each volume is a directory tree (one root)
 Assigned a name and ID number
 A server will often have 100s of volumes
NAMESPACE MANAGEMENT

 Clients get information via cell directory server (Volume Location

Server) that hosts the Volume Location Database (VLDB)
 Goal:
 everyone sees the same namespace

 /afs/cellname/path
 /afs/mit.edu/home/paul/src/try.c
FILES, DIRECTORIES, VOLUMES, CELLS
COMMUNICATION WITH THE SERVER

 Communication is via RPC over UDP

 Access control lists used for protection
 Directory granularity
 UNIX permissions ignored (except execute)
AFS SUMMARY

 AFS benefits
 AFS scales well
 Uniform name space
 Read-only replication
 Security model supports mutual authentication, data encryption
 AFS drawbacks
 Session semantics
 Directory based permissions
 Uniform name space
CODA
CONSTANT DATA AVAILABILITY
CARNEGIE-MELLON UNIVERSITY (1990-1992)
CODA GOALS

 Descendant of AFS
 Goals
 Provide better support for replication than AFS
 support shared read/write files

 Support mobility of PCs

MOBILITY

 Goal: Improve fault tolerance

 Provide constant data availability in disconnected environments
 Via hoarding (user-directed caching)
 Log updates on client
 Reintegrate on connection to network (server)
MODIFICATIONS TO AFS

 Support replicated file volumes

 A volume can be replicated on a group of servers
 Volume Storage Group (VSG)
 Volume ID used to identify files is a Replicated Volume ID
 One-time lookup
 Replicated volume ID list of servers and local volume IDs
 Cache results for efficiency
 Read files from any server
 Write to all available servers
 Extend mechanism to support disconnected operation
CODA SUMMARY

 Session semantics as with AFS

 Replication of read/write volumes
 Clients do the work of writing replicas (extra bandwidth)
 Client-detected reintegration
 Disconnected operation
 Client modification log
 Hoard database for needed files
 User-directed pre-fetch
 Log replay on reintegration
DFS (AFS V3)
DISTRIBUTED FILE SYSTEM
DFS

 Goal
 AFS: scalable performance but session semantics were hard to live with
 Create a file system similar to AFS but with a strong consistency model
 History
 Part of Open Group’s Distributed Computing Environment
 Descendant of AFS - AFS version 3.x
 Assume (like AFS):
 Most file accesses are sequential
 Most file lifetimes are short
 Majority of accesses are whole file transfers
 Most accesses are to small files
CACHING AND SERVER COMMUNICATION

 Increase effective performance with

 Caching data that you read
 Safe if multiple clients reading, nobody writing
 read-ahead
 Safe if multiple clients reading, nobody writing
 write-behind (delaying writes to the server)
 Safe if only one client is accessing file
 Goal:
 Minimize times client informs server of changes, use fewer messages with more data vs. lots of
messages with little data
DFS TOKENS

 Cache consistency
maintained by tokens
 Token
 Guarantee from server that a
client can perform certain
operations on a cached file
 Server grants & revokes tokens
LIVING WITH TOKENS

Server grants and revokes tokens

 Multiple read tokens OK
 Multiple read and a write token or multiple write tokens not OK if byte
ranges overlap
 Revoke all other read and write tokens
 Block new request and send revocation to other token holders
DFS KEY POINTS

 Caching
 Token granting mechanism
 Allows for long term caching and strong consistency
 Caching sizes: 8K – 256K bytes
 Read-ahead (like NFS)
 Don’t have to wait for entire file before using it as with AFS

 File protection via access control lists (ACLs)

 Communication via authenticated RPCs
 Essentially AFS v2 with server-based token granting
 Server keeps track of who is reading and who is writing files
 Server must be contacted on each open and close operation to request token
SMB
SERVER MESSAGE BLOCK PROTOCOL
MICROSOFT (1987)
SMB GOALS

 File sharing protocol for Windows 9x - Windows 10, Window NT-20xx

 Protocol for sharing Files, devices, communication abstractions
(named pipes), mailboxes
 Servers: make file system and other resources available to clients
 Clients: access shared file systems, printers, etc. from servers
 Design Priority: locking and consistency over client caching
SMB DESIGN

 Request-response protocol
 Send and receive message blocks
 name from old DOS system call structure
 Send request to server (machine with resource)
 Server sends response

 Connection-oriented protocol
 Persistent connection – “session”

 Each message contains:

 Fixed-size header
 Command string (based on message) or reply string
MESSAGE BLOCK

 Header: [fixed size]

 Protocol ID
 Command code (0..FF)
 Error class, error code
 Tree ID – unique ID for resource in use by client (handle)
 Caller process ID
 User ID
 Multiplex ID (to route requests in a process)
 • Command: [variable size]
 Param count, params, #bytes data, data
SMB COMMANDS
 Files  Print-related
 get disk attributes  Open/close spool file
 create/delete directories  Write to spool
 search for file(s)  Query print queue
 create/delete/rename file  User-related
 lock/unlock file area  Discover home system for user
 open/commit/close file  Send message to user
 get/set file attributes  Broadcast to all users
 Receive messages
PROTOCOL STEPS

 Establish connection
 Negotiate protocol - negprot
 Authenticate - sesssetupX
 Make a connection to a resource – tcon
 Send open/read/write/close/… SMBs
SMB EVOLVES
COMMON INTERNET FILE SYSTEM
SMB 2
SMB 3
1996, 2006, 2012
SMB EVOLVES

History
 SMB was reverse-engineered for non-Microsoft platforms
 samba.org
 E.g., Linux uses Samba to access file shares from Windows

 Microsoft released SMB protocol to X/Open in 1992

 Common Internet File System (CIFS)
 SMB as implemented in 1996 for Windows NT 4.0
SMB2 AND SMB3

 Original SMB was…

 Chatty: common tasks often required multiple round trip messages
 Not designed for WANs
 SMB2 (2007)
 Protocol dramatically cleaned up
 New capabilities added
 SMB2 became the default network file system in macOS Mavericks (10.9)
 SMB3 (2012)
 Added RDMA and multichannel support; end-to-end encryption
 RDMA = Remote DMA (Direct Memory Access)
 Windows 8 / Windows Server 2012: SMB 3.0
 SMB3 became the default network file system in macOS Yosemite (10.10)
SMB2 ADDITIONS & CREDIT-BASED FLOW CONTROL

 Reduced complexity
 From >100 commands to 19
 Pipelining support
 Send additional commands before the response to a previous one is received
 Credit-based flow control
 Goal: keep more data in flight and use available network bandwidth
 Server starts with a small # of “credits” and scales up as needed
 Server sends credits to client
 Client needs credits to send a message and decrements credit balance
 Allows server to control buffer overflow
SMB3 KEY FEATURES

 Multichannel support for network scaling

 Transparent network failover
 “SMBDirect” – support for Remote DMA in clustered environments
 Enables direct, low-latency copying of data blocks from remote memory without CPU
intervention
 Direct support for virtual machine files
 Volume Shadow Copy
 Enables volume backups to be performed while apps continue to write to files.

 End-to-end encryption
NFS VERSION 4
NETWORK FILE SYSTEM
SUN MICROSYSTEMS (NOW ORACLE)
NFS VERSION 4 ENHANCEMENTS

 Stateful server
 Compound RPC
 Group operations together
 Receive set of responses
 Reduce round-trip latency
 Stateful open/close operations
 Ensures atomicity of share reservations for windows file sharing
 Supports exclusive creates
 Client can cache aggressively
NFS VERSION 4 ENHANCEMENTS

 create, link, open, remove, rename

 Inform client if the directory changed during the operation
 Strong security
 Extensible authentication architecture
 File system replication and migration
 Mirror servers can be configured
 Administrator can distribute data across multiple servers
 Clients don’t need to know where the data is: server will send referrals

 No concurrent write sharing or distributed cache coherence

NFS VERSION 4 ENHANCEMENTS

 Stateful locking
 Clients inform servers of lock requests
 Locking is lease-based; clients must renew leases
 Improved caching
 Server can delegate specific actions on a file to enable more aggressive client caching
 Close-to-open consistency
 File changes propagated to server when file is closed
 Client checks timestamp on open to avoid accessing stale cached copy
 Clients must disable caching to share files
 Callbacks
 Notify client when file/directory contents change
  NFS
 RPC-based access
 AFS
SUMMARY  Long-term caching

Main types of network file

 DFS
systems  AFS + tokens for consistency and efficient caching
 CODA
 Read/write replication & disconnected operation
 SMB/CIFS
 RPC-like access with strong consistency
 Oplocks (tokens) to support caching
 DFS Namespaces: add-on to provide a consistent view of
volumes (AFSstyle)
QUESTIONS?
NOW, BY E-MAIL, …