Web Application Architecture Principles Protocols and Practices 2nd Edition Leon Shklar
Web Application Architecture Principles Protocols and Practices 2nd Edition Leon Shklar
com
https://ebookgate.com/product/web-application-architecture-
principles-protocols-and-practices-2nd-edition-leon-shklar/
OR CLICK HERE
DOWLOAD NOW
https://ebookgate.com/product/wordpress-web-application-
development-2nd-edition-rakhitha-nimesh-ratnayake/
ebookgate.com
https://ebookgate.com/product/practical-poetics-in-architecture-1st-
edition-leon-van-schaik/
ebookgate.com
https://ebookgate.com/product/programming-principles-and-practices-
using-c-2nd-edition-bjarne-stroustrup/
ebookgate.com
https://ebookgate.com/product/brain-injury-treatment-theories-and-
practices-1st-edition-jose-leon-carrion/
ebookgate.com
Network defense and countermeasures principles and
practices 2nd Edition Chuck Easttom
https://ebookgate.com/product/network-defense-and-countermeasures-
principles-and-practices-2nd-edition-chuck-easttom/
ebookgate.com
https://ebookgate.com/product/clinical-application-of-neuromuscular-
techniques-volume-1-the-upper-body-2nd-edition-leon-chaitow-nd-do/
ebookgate.com
https://ebookgate.com/product/water-chlorination-and-chloramination-
practices-and-principles-2nd-edition-american-water-works-association/
ebookgate.com
https://ebookgate.com/product/game-audio-programming-principles-and-
practices-somberg/
ebookgate.com
https://ebookgate.com/product/web-engineering-principles-and-
techniques-woojong-suh/
ebookgate.com
Table of Contents
Title Page
Copyright Page
Dedication
About the Authors
Preface to the Second Edition
Acknowledgements
CHAPTER 1 - Introduction
2
4.1 Standard Generalized Markup Language
4.2 HTML
4.3 HTML Rendering
4.4 Summary
4.5 Bibliography
3
7.11 Web Links
7.12 Endnotes
8.1 Pre-History
8.2 JavaScript
8.3 Cascading Style Sheets
8.4 DHTML
8.5 AJAX
8.6 Case Study: 5-Star Rating
8.7 Summary
8.8 Bibliography
8.9 Web Links
8.10 Endnotes
4
11.4 Benefits and Drawbacks of Using Rails
11.5 Whither Enterprise Java?
11.6 Summary
11.7 Bibliography
11.8 Web Links
11.9 Endnotes
CHAPTER 14 - Conclusions
Index
5
This book delivers a thorough and rigorous introduction to fundamental architectural elements of the web;
this knowledge is a critical foundation for any development engineer working on our advanced web
applications. Just as important, this book provides a challenging, non-trivial example application
demonstrating current best practices which the reader can work through to connect theory to practice. For our
engineers coming from other backgrounds to advanced web development, this book has been a very efficient,
effective learning tool and I would recommend it highly to others who desire a deep understanding of web
architecture and applications.
6
7
Copyright © 2009 John Wiley & Sons Ltd, The Atrium, Southern Gate, Chichester,
West Sussex PO19 8SQ, England
Telephone (+44) 1243 779777
Email (for orders and customer service enquiries): cs-books@wiley.com
Visit our Home Page on www.wiley.com
All Rights Reserved. No part of this publication may be reproduced, stored in a retrieval system or transmitted
in any form or by any means, electronic, mechanical, photocopying, recording, scanning or otherwise, except
under the terms of the Copyright, Designs and Patents Act 1988 or under the terms of a licence issued by the
Copyright Licensing Agency Ltd, Saffron House, 6-10 Kirby Street, London EC1N 8TS, UK, without the
permission in writing of the Publisher. Requests to the Publisher should be addressed to the Permissions
Department, John Wiley & Sons Ltd, The Atrium, Southern Gate, Chichester, West Sussex PO19 8SQ,
England, or emailed to permreq@wiley.com, or faxed to (+44) 1243 770620.
Designations used by companies to distinguish their products are often claimed as trademarks. All brand
names and product names used in this book are trade names, service marks, trademarks or registered
trademarks of their respective owners. The Publisher is not associated with any product or vendor mentioned
in this book.
This publication is designed to provide accurate and authoritative information in regard to the subject matter
covered. It is sold on the understanding that the Publisher is not engaged in rendering professional services. If
professional advice or other expert assistance is required, the services of a competent professional should be
sought.
John Wiley & Sons Inc., 111 River Street, Hoboken, NJ 07030, USA
Jossey-Bass, 989 Market Street, San Francisco, CA 94103-1741, USA
Wiley-VCH Verlag GmbH, Boschstr. 12, D-69469 Weinheim, Germany
John Wiley & Sons Australia Ltd, 42 McDougall Street, Milton, Queensland 4064, Australia
John Wiley & Sons (Asia) Pte Ltd, 2 Clementi Loop #02-01, Jin Xing Distripark, Singapore 129809
John Wiley & Sons Canada Ltd, 6045 Freemont Blvd, Mississauga, Ontario, L5R 4J3, Canada
Wiley also publishes its books in a variety of electronic formats. Some content that appears in print may not
be available in electronic books.
Shklar, Leon
Web application architecture : principles, protocols, and practices / Leon Shklar and Rich Rosen. - 2nd ed.
p. cm.
Includes bibliographical references and index.
ISBN 978-0-470-51860-1 (pbk.)
8
1. Web site development. 2. Application software-Development. 3. Web sites-Design. 4. Software
architecture.
I. Rosen, Rich. II. Title.
TK5105.888.S492 2009
005.1’2-dc22
2008052051
A catalogue record for this book is available from the British Library
ISBN 978-0-470-51860-1
Typeset in 10/12.5 Times by Laserwords Private Limited, Chennai, India
Printed and bound in Great Britain by Bell and Bain, Glasgow
9
To my beautiful girls: my wife Rita and daughter Victoria.
To the memory of my parents, Hasya and Arcady Shklar, and my grandparents, Tasya and Boris
Korkin.
Leon Shklar
To my parents, Arthur and Toby, and to Celia. Also, to the memory of my high school maths teacher, Jack
Garfunkel, who instilled in his students the value of thinking things through logically, and the value of writing
not for those who already know what you’re talking about, but for those who don’t.
Rich Rosen
10
About the Authors
Leon Shklar currently works for Thomson Reuters where he is the head of technology for Reuters Media.
Previously, Leon headed up the development team for the online edition of the Wall Street Journal at Dow
Jones. Prior to joining Dow Jones, he spent six years at Bell Communications Research and almost as long in
the world of dot-coms and Internet software. Leon holds a Ph.D. in Computer Science from Rutgers
University.
Rich Rosen is a senior developer in the Fixed Income Systems Group at Interactive Data Corporation.
Previously, he was an Application Architect at Dow Jones. Rich began his career at Bell Labs, where his work
with relational databases and the Internet prepared him for the world of Web application development. He is
a co-author of Mac OS X for Unix Geeks, 4th Edition (O’Reilly). Rich holds an M.S. in Computer Science from
Stevens Institute of Technology.
11
Preface to the Second Edition
The expression “web time” connotes a world in which rapid change is the norm, where time is exponentially
condensed. Technological advances that once upon a time might have taken years to transpire now occur in a
matter of months or even days. What’s more, these advances often result in radical paradigm shifts that
change the way we interact with our technology, and with the world at large.
The first edition of this book was published in 2003. Since then, there have been many technological
advances and paradigm shifts, causing some of what we wrote to become dated. New frameworks such as
Ruby on Rails have arisen as a reaction to increasing complexity in the application development process.
AJAX has taken client-side interactivity to a new level, blazing new frontiers in web application functionality.
Search has become a fundamental part of our everyday web experience. Even the core protocols and markup
languages representing the foundation of web technology have evolved since we first wrote about them over
five years ago. Back then, who could have imagined the ascendance of today’s most popular web applications,
such as YouTube, Facebook, eBay, and Wikipedia, or the advances in real-time interactivity that are now
commonplace?
For the second edition of this book, we provide new material covering these changes in the web technology
landscape, while striving to bring existing content up-to-date. We have included new chapters on search
technology and client-side interactivity (JavaScript, DHTML, and AJAX). We have added a second sample
application implemented using Ruby on Rails as a complement to the original Struts application, which has
been updated for the new edition. The chapters on Internet protocols, markup languages, server and browser
architecture, and application development approaches have all been revised and enhanced. It is our hope that
this updated edition will provide readers with deeper insights into the principles, protocols, and practices
associated with the design and development of web applications.
Leon Shklar
Rich Rosen
October 31, 2008
12
13
Acknowledgments
I am grateful to my wife Rita for inspiration and music, and for still being around after all the work that went
into the two editions of this book. My special thanks to our daughter Victoria for her insightful ideas
throughout the project.
Leon Shklar
Ongoing and everlasting thanks to my wife, Celia, for the joy her singing brings into my life, and for enduring
the continued insanity associated with the writing process. Thanks also to my parents and to Celia’s parents
for their love and support.
Rich Rosen
We would both like to thank the following people for their help:
• our editor, Jonathan Shipley, and his assistants, Georgia King and Claire Jardine, for their
professionalism and flexibility throughout the project;
• our technical reviewers, Sue Fitzgerald, Ciara n O’Leary, Ilmi Yoon, Roger Beresford, Yuanzhu
Peter Chen, Ray Cheung and Wei Ding, for taking the time to examine the text and provide us
with valuable insights and advice;
• our colleagues, Otis Gospodnetich and Keith Kim, for their comments and suggestions for the
search chapter, and Heow Eide-Goodman for his comments and suggestions for the chapters on
development approaches and Rails application development.
14
CHAPTER 1
Introduction
IN THIS CHAPTER
OBJECTIVES
15
1.1 History and Pre-History of the Web
Back in 1989, at CERN (the scientific research laboratory near Geneva, Switzerland), Tim Berners-Lee
presented a proposal for an information management system that would enable the sharing of knowledge and
resources over a computer network. We now know this system as the worldwide web (the web). Building on
the foundation of existing Internet protocols and services, it lives up to its name as a ubiquitous network
providing information and communication services to hundreds of millions of people around the world.
From the very beginnings of Internet technology, many people shared the dream of using the Internet as a
universal medium for exchanging information over computer networks. Internet file-sharing services (such as
FTP and Gopher) and message forum services (such as Netnews) provided increasingly powerful mechanisms
for information exchange and brought us closer to fulfilling those goals.
Ted Nelson’s Xanadu project aspired to make that dream a reality, but the goals were lofty and never fully
realized. Nelson coined the term “hypertext” as a way of describing “non-sequential writing - text that
branches and allows choice to the reader.” Unlike the static text of print media, hypertext was intended for use
with an interactive computer screen. It is open, fluid and mutable, and can be connected to other pieces of
hypertext by “links”.
It took Tim Berners-Lee to “marry together” (in his words) the notion of hypertext with the power of the
Internet, bringing those initial dreams to fruition in a way that the earliest developers of both hypertext and
Internet technology might never have imagined. His vision was to connect literally everything, in a uniform
and universal way. Berners-Lee originally promoted the web as a virtual library, a document control system for
sharing information resources among researchers. On-line documents could be accessed via a unique
document address, a universal resource locator (URL). These documents could be cross-referenced via hypertext
links.
From its humble beginnings, the web has expanded exponentially to serve a wide variety of purposes for a
wide variety of people:
• Given its origins, it seems natural that educational institutions and research laboratories were
among the very first users of the web, employing it to share documents and other resources across
the Internet.
• Popular adoption of the web by individuals followed gradually, originally through on-line services,
such as America On-line (now AOL) that slowly but surely integrated with the web and the
Internet. Initially, personal usage of the web was limited to e-mail and web surfing. Eventually,
people began building their own web sites where they posted everything from on-line photo albums
to personal journals that would become known as “blogs.”
• Over time, businesses saw the potential of the web and began to employ it for e-commerce,
providing an on-line medium for buying and selling goods interactively. As more and more people
were connected to the web, the draw of a new source of revenue and the fear that competitors would
get there first and undercut traditional revenue streams made e-commerce increasingly important to
business. E-commerce applications are now being used for everything - from displaying catalogs of
16
available merchandise to providing the means for customers to purchase goods and services securely
on-line.
• As the impact of e-commerce grew, the back-end applications supporting e-commerce became
increasingly important to the companies using it. The front-end, customer-facing web sites needed
to be up to date, synchronized with inventory systems so that customers would know what items
were or weren’t immediately available. Automated fulfillment systems connected to the on-line
ordering mechanisms became commonplace. With that, secure login and registration, including
collection of credit card information and payment processing, became an absolute requirement for
viable e-commerce sites.
• With the maturing of the web, a tug of war arose between sites offering free and paid content.
Google was successful in tilting the balance in favor of free content supported by ads. Even the Wall
Street Journal, which pioneered the notion of paid subscriptions for on-line content, has been
opening up more and more of its content to non-subscribers. Critical technological challenges for
free sites include tracking visitor behavior, providing adaptive personalization, and offering advanced
community-building tools.
The web did not come into existence in a vacuum. It was built on top of core Internet protocols that had
been in existence for many years prior to the inception of the web. Understanding the relationship between
web technology and the underlying Internet protocols is fundamental to the design and implementation of
true web applications. In fact, it is the exploitation of that relationship that distinguishes a web page or web
site from a web application.
17
1.2 From Web Pages to Web Sites
The explosive growth of the web at least partially can be attributed to its grass-roots proliferation as a tool for
personal publishing. The fundamental technology behind the web is relatively simple. A computer connected to
the Internet running a web server was all that was necessary to serve documents. Both CERN and the National
Center for Supercomputer Applications (NCSA) at the University of Illinois had developed freely available web
server software. A small amount of HTML knowledge (and the proper computing resources) got you
something that could be called a web site. Early web sites were just loosely connected sets of pages, branching
hierarchically from a home page; now, a web site is much more than just a conglomeration of web pages.
When the web was in its infancy, academic institutions and technology companies owned the only
computers that were connected to the Internet and could run server software. In those days, personal
computers sitting on people’s desks were still a rarity. If you wanted access to any sort of computing power,
you used a terminal that let you “log in” to a large server or mainframe over a direct connection or dial-up
phone line.
Still, creating and posting web pages quickly became popular among people that had access to scarce
computing power. The original HTML language was simple enough that, even without the more
sophisticated tools we have at our disposal today, it was an easy task for someone to create a web page. (Some
would say too easy.) In the end, all that was needed to create first generation web pages was a simple text
editor.
There is a big difference between a bunch of web pages and a web site. A web site is more than just a group
of web pages that happen to be connected to each other through hypertext links:
• First, there are content-related concerns. Maintaining thematic consistency of content is important
in giving a site some degree of identity.
• There are also aesthetic concerns. In addition to having thematically related content, a web site
should also have a common look and feel across all of its pages, so that site visitors know what they
are looking at. This means utilizing a common style: page layout, graphic design, and typographical
elements.
• Then there are architectural concerns. As a site grows in size and becomes more complex, it is
critically important to organize its content. This includes not just the layout of content on individual
pages, but also the interconnections between the pages (site navigation). If the site becomes so
complex that visitors cannot navigate their way through, even with the help of site maps and
navigation bars, then it needs to be reorganized and restructured.
18
1.3 From Web Sites to Web Applications
Initially, what people shared over the Internet consisted mostly of static information found in files. They
might edit those files, but there were few truly dynamic information services.
Granted, there were a few exceptions: search applications for finding files in on-line archives and services
that provided information about the current weather, or the availability of cans from a soda-dispensing
machine. (One of the first web applications that Tim Berners-Lee demonstrated at CERN was designed to
look up numbers in an on-line phone book using a web browser.) However, for the most part, the information
resources on the web were static documents.
The advent of the dynamic web, which resulted from the proliferation of dynamic information services,
changed all that. The new services ranged from CGI scripts to search engines to packages that connected web
applications to relational databases. No longer was it sufficient to build a web site (as opposed to a motley
collection of web pages). It became necessary to design a web application.
What is a “web application?” It is a client-server application that uses a web browser as its client program. It
delivers interactive services through web servers distributed over the Internet (or an intranet). A web site
simply delivers content from static files. A web application can present dynamically tailored content based on
request parameters, tracked user behaviors, and security considerations.
Web applications power information portals, retail sites, and corporate intranets. They not only provide
information and interact with site visitors, but also collect and update information, maintain access controls,
and support on-line transactions.
A prime example of a dynamic web application is an on-line shopping site. Site visitors browse on-line
catalogs, search for items they want to purchase, add those items to an on-line shopping cart, and place their
orders through a secure interface that encrypts personal information. To enable this functionality, the web
application communicates with warehouses that maintain catalog and inventory information, storing this
information in a database. Users can search the on-line database to obtain information about products. The
application provides a way for users to select items to be added to their shopping carts, maintains shopping
cart contents for the duration of the browser session, and records order information in the database. It
communicates with outside financial institutions to verify credit card and address information, fulfills the
order by directing warehouses to ship the purchased items, and sends confirmation e-mails to purchasers
allowing them to track shipment of their orders.
19
1.4 Web 2.0: On-line Communities and Collaboration
Since 2000, another major trend has arisen in the on-line world, incorporating applications that support user-
generated content, on-line communities, and collaborative mechanisms for updating on-line content. This
trend is often referred to as Web 2.0, because it is closely tied to advances in web technology that herald a new
generation of web applications.
In many respects, Web 2.0 is a harking back to the web as a network for presenting personal hyperlinked
information. Information flow on the web is no longer one way, from the web site to the surfer. Site visitors
contribute information of their own, ranging from reviews and ratings for movies, music, and books to
personal journals, how-to information, and news for popular consumption. These journals go by the name
blogs (short for “web logs”) and the whole blogging movement has resurrected the idea of the personal web
page and elevated its status.
Personal blogs and community web sites encouraging user input may bear a resemblance to the more
personal web of old, but the technology behind them does not. Whereas individual owners could easily
maintain the simple static web sites of old, this has become impractical given the enormous volume and
increasingly malleable nature of content out there today. Blogs, user forums, and collaborative community
sites may look like the simpler web sites of the past, but the underlying functionality supporting them is based
on sophisticated web application technology incorporating user authentication, access control, and content
management services.
20
1.5 The Brave New World of AJAX
Another radical change in the web application landscape was the advent of AJAX (an acronym that stands for
Asynchronous JavaScript and XmlHttpRequest). While the technology advances behind it accumulated
gradually, AJAX represents a paradigm shift that has changed the way web applications are conceived and
developed.
Tim Berners-Lee’s original concept of HTTP was that of a simple stateless request - response protocol.
Such a protocol allows for independent processing of requests and responses, without requiring a persistent
connection. This simplicity fostered wide acceptance of the HTTP protocol but imposed significant
limitations. The only method available for updating page content was to replace the entire page. This
limitation was unsatisfying to those who were used to client-server applications that communicated with their
associated servers directly and continuously. For instance, in such an application, a server-side change to a
value in a spreadsheet cell could immediately be reflected on the screen without refreshing the whole
spreadsheet.
HTTP, and the web experience in general, have evolved in a number of ways since their inception,
introducing elements that allow web transactions to transcend the limitations of a stateless request-response
protocol (e.g., cookies and JavaScript). AJAX is the latest such element, essentially allowing what amounts to
an “end-run” to communicate with a web server indirectly: instead of submitting a direct request to see a new
page from the server, a subordinate background request is submitted (usually through a JavaScript object
called XmlHttpRequest) and the response to that request is used to dynamically modify the content of the
current page in lieu of replacing its content entirely.
The main impact of AJAX is that the web experience is brought closer to that of client-server applications,
allowing for a more immediate dynamic interaction between the user and the web application.
21
1.6 Focus of This Book
The purpose of this book is to provide a guide for learning the underlying principles, protocols, and practices
associated with designing flexible and efficient web applications. Our target audience is senior undergraduate
or graduate students who have already learned the basics of web development and professional developers who
have had some exposure to web application development and want to expand their knowledge.
We expect our readers to have some familiarity with HTML and JavaScript - not at the level of experienced
web designers but enough to create web pages and embed simple JavaScript code to submit forms to a web
server. We recommend at least some exposure to Internet protocols, such as Telnet, FTP, SMTP, HTTP and
SSL. We appreciate that many of our readers may not be familiar with Linux or Unix operating systems.
However, it is helpful to have some understanding of the command-line interfaces provided by interactive
shells, as some of our examples employ such an interface.
As we have said, there is a major difference between web pages, web sites, and web applications. There are
excellent resources dedicated to web page and web site design, and the References section at the end of this
chapter lists some of the best that we know. When we examined the current literature available on the subject
of web application development, we found there were three main categories of book currently available.
• technical overviews
• reference books
• focused tutorials.
Technical overviews are usually very high level, describing technology and terminology in broad terms. They
do not go into enough detail to enable the reader to design and build serious web applications. They are most
often intended for managers who want a surface understanding of concepts and terminology without going
too deeply into specific application development issues. Frequently, such overviews attempt to cover
technology in broad brushstrokes; their subject may be Java, XML, e-commerce, or Web 2.0. Such books
approach the spectrum of technology so broadly that the coverage of any specific area is too shallow for serious
application developers.
Reference books are useful, naturally, as references, but not for the purpose of learning about the technology.
They are great to keep on your desk to look things up once you are already deeply familiar with the
technology, but they are not oriented toward elucidation and explanation.
The focused tutorials concentrate on the usage of specific platforms and products to develop web
applications. Books in this category provide in-depth coverage of very narrow areas, concentrating on how to
use a particular language or platform without necessarily explaining the underlying principles. Such books may
be useful in teaching programmers to develop applications for a specific platform, but they do not provide
enough information about the enabling technologies, focusing instead on the platform-specific
implementation. Should a developer be called upon to rewrite an application for another platform, the
knowledge acquired from these books is rarely transferable to the new platform.
Given the rate of change of the web technologies, today’s platform of choice is tomorrow’s outdated legacy
system. When new development platforms emerge, developers without a fundamental understanding of the
22
inner workings of web applications have to learn the new platforms from the ground up. The challenge is their
lack of understanding of what the systems they implemented using specialized application programming
interfaces (APIs) did behind the API calls. What is missing is the ability to use fundamental technological
knowledge across platforms.
What was needed was a book that covered the basic principles of good application design, the underlying
protocols associated with web technology, and the best practices for creating scalable, extensible, maintainable
applications. With this in mind, we endeavored to write such a book.
The need for such a book is particularly apparent when interviewing job candidates for web application
development positions. Too many programmers have detailed knowledge of particular languages and
interfaces but they are lost when asked questions about the underlying technologies and how they relate to real
problems (e.g., why is it necessary for a server to add a trailing slash to a URL and redirect the request back to
itself?). Such knowledge is not purely academic - it is critical when designing and debugging complex systems.
Too often, developers with proficiency only within a specific application development platform (such as
Active Server Pages, Cold Fusion, PHP, or Perl CGI scripting) are not capable of transferring that proficiency
directly to another platform. The fundamental understanding of core technologies is critical to enable
developers to grow with the rapid technological advances in web application development.
What do we have in mind when we refer to the general principles that need to be understood in order to
properly design and develop web applications? We mean the core protocols and languages associated with web
applications. This includes HyperText Transfer Protocol (HTTP) and HyperText Markup Language (HTML),
which are fundamental to the creation and transmission of web pages, but it also includes older Internet
protocols such as Telnet and FTP, and the protocols used for message transfer such as SMTP and IMAP. A
web application architect must also be familiar with JavaScript, XML, relational databases, graphic design and
multimedia. Web application architects must be well-versed in application server technology and have a
strong background in information architecture.
If you find people with all these qualifications, please let us know: we would love to hire them! Rare is the
person who can not only design a web site but can also perform all the other tasks associated with web
application development: working with graphic designers, creating database schemas, producing interactive
multimedia programs, and configuring e-commerce transactions. More realistically, we can seek someone who
is an expert in one particular area (e.g., e-commerce transactions or browser programming) but who also
understands the wider issues associated with designing web applications.
We hope that, by reading this book, you can acquire the skills needed to design and build complex
applications for the web. There is no “one easy lesson” for learning the ins and outs of application design.
Hopefully, this book will enhance your ability to design and build sophisticated web applications that are
scaleable, maintainable, and extensible.
We examine various approaches to the process of web application development, covering both client-side
presentation technology and server-side application technology. On the client side, we look at both markup
languages and programming languages, from HTML and XML to CSS and JavaScript. On the server side, we
look at the full range of approaches, starting with Server Side Includes (SSI) and CGI, covering template
languages such as Cold Fusion and ASP, examining the intricacies of Java Platform, Enterprise Edition (Java
EE), and finally looking at newer “rapid development” approaches such as Ruby on Rails. At each level, we
23
concentrate not only on the particular development platform, but also on the underlying principles that span
multiple platforms.
24
1.7 What Is Covered in This Book
• Chapter 2: Core Internet Protocols - This chapter offers an examination of the underlying
Internet protocols that form the foundation of the web. It offers some perspectives on the history of
TCP/IP, as well as some details about using several of these protocols in web applications.
• Chapter 3: Birth of the Web: HTTP - The HTTP protocol is covered in detail in this chapter,
with explanations of how requests and responses are transmitted and processed.
• Chapter 4: HTML and Its Roots - In the first of two chapters about markup languages, we go
back to SGML to learn more about the roots of HTML (and of XML). Rather than providing a
tutorial on web design with HTML, the focus is on HTML as a markup language and its place in
web application development.
• Chapter 5: XML Languages and Applications - This chapter covers XML and related
specifications, including XML Schema, XPath, XSLT, and XSL FO, as well as XML applications
such as XHTML and WML.
• Chapter 6: Web Servers - The operational intricacies of web servers is the topic of this chapter,
with in-depth discussion of what web servers must do to support interactions with clients such as
web browsers and HTTP proxies.
• Chapter 7: Web Browsers - As the previous chapter dug deep into the inner workings of web
servers, this chapter provides similar coverage of the inner workings of web browsers.
• Chapter 8: Active Browser Pages: From JavaScript to AJAX - Here we cover the mechanisms for
providing dynamic interactivity in web pages, including JavaScript, DHTML, and AJAX.
• Chapter 9: Approaches to Web Application Development - This chapter contains a survey of
available web application approaches, including CGI, Servlets, PHP, Cold Fusion, ASP, JSP, and
frameworks such as Struts. It classifies and compares these approaches to help readers make
informed decisions when choosing an approach for their project, emphasizing the benefits of using
the Model-View-Controller (MVC) design pattern in implementing applications.
• Chapter 10: Web Application Primer: Virtual Realty Listing Services - Having examined the
landscape of available application development approaches, we decide on Struts along with the Java
Standard Tag Library (JSTL). We give the reasons for our decisions and build a sample employing
the principles we have been discussing in previous chapters. We then suggest enhancements to the
application as exercises to be performed by the reader, including the introduction of an
administrative interface component, using Hibernate for object-relational mapping, and using Java
Server Faces for presentation.
• Chapter 11: Web Application Primer: Ruby on Rails - In this chapter, we revisit the Virtual
Realty Listing Services application and implement its administrative interface using the Ruby on
Rails framework. We describe the general structure of a Rails application, walk through the process
of building an application in Rails, and compare the Java EE - Struts approach with Rails in terms
25
of both ease of development and flexibility of deployment.
• Chapter 12: Search Technologies - Here we describe not only the process of indexing site content
for internal search functionality, but also the mechanisms for structuring a site’s content for optimal
indexing by external search engines. Jakarta’s Lucene and other tools are covered.
• Chapter 13: Trends and Directions - Finally, we look to the future, providing coverage of the
most promising developments in web technology, as well as speculating about the evolution of web
application frameworks.
Chapter 2 starts us off with a study of the core protocols supporting Internet technology, in general, and
the web in particular. Although some might see this as review, it is a subject worth going over to gain both a
historical and a technological perspective.
26
1.8 Bibliography
Berners-Lee, Tim, 2000. Weaving the Web: The Original Design and Ultimate Destiny of the World Wide Web.
New York: HarperBusiness.
Gehtland, Justin, Galbraith, Ben and Almaer, Dion, 2006. Pragmatic AJAX: A Web 2.0 Primer. Raleigh (NC):
Pragmatic Bookshelf.
Hadlock, Kris, 2007. AJAX for Web Application Developers. Indianapolis (IN): SAMS Publishing (Developer’s
Library).
Nelson, Theodor Holm, 1982. Literary Machines 931 . Sausalito (CA): Mindful Press.
Rosenfeld, Louis and Morville, Peter, 2006. Information Architecture for the World Wide Web, 3rd Edition.
Sebastopol (CA): O’Reilly & Associates.
Williams, Robin and Tollett, John, 2005. The Non-Designer’s Web Book, 3rd Edition. Berkeley (CA): Peachpit
Press.
27
CHAPTER 2
IN THIS CHAPTER
• TCP/IP
• Telnet
• Electronic mail
• Messaging
• Security and encryption
• FTP and file server protocols
• And then came the web ...
OBJECTIVES
As we mentioned in the previous chapter, Tim Berners-Lee did not come up with the worldwide web in a
vacuum. The web as we know it was built on top of core Internet protocols that had been in existence for many
years. Understanding the ideas behind those underlying protocols is important for the discipline of building
robust web applications.
In this chapter, we examine the core Internet protocols that make up the TCP/IP protocol suite, which is the
foundation for the web protocols that are discussed in Chapter 3. We begin with a brief historical overview of
the forces that led to the creation of TCP/IP. We then go over the layers of the TCP/IP stack, and show
where various protocols fit into it. Our description of the client-server paradigm used by TCP/IP applications
is followed by discussion of the various TCP/IP application services, including Telnet, electronic mail (e-
28
mail), message forums, live messaging, and file servers. While some of these protocols and services may be
deprecated, if not obsolete, knowledge of how they work and what they do provides critical insights into the
present and future of web protocols and web applications.
29
2.1 Historical Perspective
The roots of web technology can be found in the original Internet protocols (known collectively as TCP/IP)
developed in the 1980s. These protocols were an outgrowth of work to design a network called the
ARPANET.
The ARPANET was named for the Advanced Research Projects Agency (ARPA) of the US Department of
Defense (DoD). It came into being as a result of efforts in the 1970s to develop an open, common,
distributed, and decentralized computer networking architecture. The DoD’s goal was to resolve numerous
problems with existing network architectures.
First and foremost among these problems was that the typical network topology was centralized. A
computer network had a single point of control directing communication between all the systems belonging to
that network. From a military perspective, such a topology had a critical flaw: destroy that central point of
control and all possibility of communication was lost.
Another issue was the proprietary nature of existing network architectures. Most of them were developed
and controlled by private corporations, who had a vested interest both in pushing their own products and in
keeping their technology to themselves. Furthermore, the proprietary nature of the technology limited the
interoperability between different systems. It was important, even then, to ensure that the mechanisms for
communicating across computer networks were not proprietary or controlled in any way by private interests,
lest the entire network become dependent on the whims of a single corporation.
Thus, the DoD funded an endeavor to design the protocols for the next generation of computer
communications networking architectures. Establishing a decentralized, distributed network topology was
foremost among the design goals for the new networking architecture. Such a topology would allow
communications to continue without disruption, even if any one system was damaged or destroyed. In such a
topology, the network “intelligence” would not reside in a single point of control. Instead, it would be
distributed among many systems throughout the network.
To facilitate this (and to accommodate other network reliability considerations), ARPANET employed a
packet-switching technology, whereby a network “message” could be split up into packets, each of which might
take a different route over the network, arrive in completely mixed-up order, and still be reassembled and
understood by the intended recipient.
To promote interoperability, the protocols needed to be open: readily available to anyone who wanted to
connect their system to the network. An infrastructure was needed to design the set of agreed-upon protocols,
and to formulate new protocols for new technologies that might be added to the network in the future. An
Internet Working Group (INWG) was formed to examine the issues associated with connecting heterogeneous
networks in an open, uniform manner. This group provided an open platform for proposing, debating, and
approving protocols.
The Internet Working Group evolved over time into other bodies, such as the Internet Activities Board
(IAB), later renamed the Internet Architecture Board, the Internet Assigned Numbers Authority (IANA), the
Internet Engineering Task Force (IETF) and the Internet Engineering Steering Group (IESG). These bodies
30
defined the standards that “govern” the Internet. They established the formal processes for proposing new
protocols, discussing and debating the merits of these proposals, and ultimately approving them as accepted
Internet standards.
Proposals for new protocols (or updated versions of existing protocols) are provided in the form of Requests
for Comments, also known as RFCs. Once approved, the RFCs are treated as the standard documentation for
the new or updated protocol.
31
2.2 TCP/IP Architecture
The original ARPANET was the first fruit borne of the DoD endeavor. The protocols behind the
ARPANET evolved over time into the TCP/IP Suite, a layered taxonomy of data communications protocols.
The name TCP/IP refers to two of the most important protocols within the suite, Transmission Control
Protocol (TCP) and Internet Protocol (IP), but the suite is comprised of many other significant protocols and
services.
32
2.2.1 Protocol layers
The protocol layers (above the “layer” of physical interconnection) associated with TCP/IP are:
Because this taxonomy contains layers, implementations of these protocols are often known as a protocol
stack.
The Network Interface layer is responsible for the lowest level of data transmission within TCP/IP,
facilitating communication with the underlying physical network.
The Internet layer provides the mechanisms for intersystem communications, controlling message routing,
validity checking, and composition and decomposition of message headers. The protocol known as IP (which
stands, oddly enough, for Internet Protocol) operates on this layer, as does the Internet Control Message
Protocol (ICMP), which handles the transmission of control and error messages between systems. Ping is an
Internet service that operates through ICMP.
The Transport layer provides message transport services between applications running on remote systems.
This is the layer in which the Transmission Control Protocol (TCP) operates. TCP provides reliable, connection-
oriented message transport. Most of the well-known Internet services make use of TCP as their foundation.
However, some services that do not require the reliability (and overhead) associated with TCP make use of
User Datagram Protocol (UDP). For instance, streaming audio and video services would gladly sacrifice a few
lost packets to get faster performance out of their data streams, so these services often operate over UDP,
which trades reliability for performance.
The Application layer is the highest level within the TCP/IP protocol stack. It is within this layer that most
of the services we associate with the Internet operate.
33
2.2.2 Comparison with OSI model
During the period that TCP/IP was being developed, the International Standards Organization (ISO) was
working on its own layered protocol scheme, called Open Systems Interconnection (OSI). While the TCP/IP
taxonomy consists of five layers (if you include the physical connectivity medium as a layer), OSI had seven
layers: Physical, Data Link, Network, Transport, Session, Presentation, and Application.
There is some parallelism between the two models. TCP/IP’s Network Interface layer is sometimes called
the Data Link layer to mimic the OSI reference model, while the Internet layer corresponds to OSI’s
Network layer. Both models share the notion of a Transport layer, which serves roughly the same functions in
each model. The Application layer in TCP/IP combines the functions of the Session, Presentation, and
Application layers of OSI.
OSI never caught on and, while some people waited patiently for its adoption and propagation, it was
TCP/IP that became the ubiquitous foundation of the Internet as we know it today.
34
2.2.3 The client - server paradigm
TCP/IP applications tend to operate according to the client-server paradigm. This simply means that, in these
applications, servers (also called services and daemons, depending on the lingo of the underlying operating
system) execute by waiting for requests from client programs to arrive and processing those requests.
Client programs can be applications used by human beings, or they could be servers that need to make their
own requests that can only be fulfilled by other servers. More often than not, the client and the server run on
separate machines, and communicate via a connection across a network.
Over the years, user interfaces to client programs have evolved from command-line interfaces (CLI) to graphical
user interfaces (GUI).
Command-line programs have their origins in the limitations of the oldest human interface to computer
systems: the teletype keyboard. In the earliest days of computing, even simple text-based CRT terminals were
not available - let alone today’s monitors with advanced graphics capabilities. The only way to enter data
interactively was through a teletypewriter interface, one character at a time. A command-line interface (CLI)
prompts the user for the entry of a “command” (the name of a program) and its “arguments” (the parameters
passed to the program). The original PC-DOS operating system and Unix shells use command-line interfaces.
Screen-mode programs allow users to manipulate the data on an entire CRT screen, rather than on one line.
This means that arrow keys can be used to move a cursor around the screen or to scroll through pages of a text
document. Screen-mode programs are still restricted to character-based interfaces.
GUI programs make use of a visual paradigm that offers users a plethora of choices. For most, this is a
welcome alternative to manually typing in the names of files, programs, and command options. The graphics
are not limited to just textual characters, as in screen-mode programs. The GUI paradigm relies on windows,
icons, mouse, pointers, and scrollbars (WIMPS) to display graphically the set of available files and
applications.
Whether command-line or GUI-based, client programs provide the interface by which end users
communicate with servers to make use of TCP/IP services. Although debates rage as to whether GUIs or
CLIs are “better”, each has advantages and disadvantages for different types of users.
Under the hood, communications between client and server programs take the form of request- response
interactions. The imposition of this constraint on Internet communication protocols means that even the
most primitive command-line interface can make use of TCP/IP services. More sophisticated GUI-based
client programs often hide their command-line details from their users, employing point-and-click and drag-
and-drop functionality to support underlying command-line directives.
After the server acknowledges the success of the connection, the client sends commands on a line-by-line
35
basis. There are single-line and block commands. A single-line command, as the name implies, includes an
atomic command directive on a single line. A block command begins with a line indicating the start of the
command and terminates with a line indicating its end. For example, in the Simple Mail Transfer Protocol
(SMTP), a line beginning with the word HELO initiates an SMTP session, while a line containing just the
word DATA begins a block that ends with a line containing only a period. The server then responds to each
command, usually beginning with a line containing a response code.
A stateful protocol allows a client to support a sequence of commands. The server is required to maintain
the “state” of the connection throughout the transmission of successive commands, until the connection is
terminated. The sequence of transmitted and executed commands is often called a session. Most Internet
services (including SMTP) are session-based and make use of stateful protocols.
HTTP
HTTP is a stateless protocol (see Chapter 3). An HTTP request usually consists of a single
command and a single response. There is no built-in ability to maintain state between
transmitted commands.
Early implementations of client - server architectures did not make use of open protocols. This meant that
client programs had to be as “heavy” as the server programs. A “lightweight” client (also called a thin client)
could only exist in a framework where common protocols and application controls were associated with the
client machine’s operating system. Without such a framework, many of the connectivity features had to be
included directly in the client program, adding to its “weight”. One advantage of using TCP/IP for client-
server applications was that the protocol stack was installed on the client machine as part of the operating
system and the client program itself could be more of a thin client.
Web applications are a prime example of thin clients. Rather than building a custom program to perform
desired application tasks, web applications use the web browser, a program that is already installed on most
end-user systems. You cannot create a client much thinner than a program that users already have on their
desktops!
TCP/IP client programs open a socket, which is simply a TCP connection between the client
machine and the server machine. Servers listen for connection requests that come in through
specific ports. A port is not a physical interface between the computer and the network, but is
simply a numeric reference within a request that indicates which server program is its intended
recipient.
There are established conventions for matching port numbers with specific TCP/IP services.
For example, Telnet services listen for connection requests on port 23, SMTP servers listen to
port 25, and web servers (by default) listen to port 80.
36
37
2.3 TCP/IP Application Services
In this section, we discuss some of the common TCP/IP application services, including Telnet, e-mail,
message forums, live messaging, and file servers. Where appropriate, we compare the original TCP/IP services
with modern counterparts.
38
2.3.1 Telnet
The Telnet protocol operates within the Application layer. It was developed to support Network Virtual
Terminal functionality, which means the ability to “log in” to a remote machine over the Internet. The latest
specification for the Telnet protocol is defined in Internet RFC 854.
Before the advent of personal computers, access to computing power was limited to those who could
connect to a larger server or mainframe computer, either through a dial-up phone line or through a direct
local connection. Whether you phoned in remotely or sat down at a terminal connected directly to the server,
you used a command-line interface to log in. You connected to a single system and your interactions were
limited to that system.
With the arrival of Internet services, you could use the Telnet protocol to log in remotely to systems that
were accessible over the Internet. As we mentioned earlier, Telnet clients are configured by default to connect
to port 23 on the server machine, but the target port number can be overridden in most client programs. This
means you can use a Telnet client program to connect and “talk” to another TCP-based service if you know its
address and its port number, have proper authentication credentials, and understand the format of its
transactions with the server.
Secure Shell
The Telnet protocol is now considered a poor mechanism for connecting to remote systems,
because Telnet servers and clients transmit all their data (including passwords) unencrypted.
Secure Shell (SSH) establishes connections and transfers data over secure channels, encrypting
transmitted data and promoting security (see Section 2.3.5).
39
2.3.2 E-mail
Electronic mail, or e-mail, was probably the first “killer app” in what we now call cyberspace. Since the
Internet had its roots in military interests, naturally the tone of e-mail started out being formal, rigid, and
business-like. Once the body of people using e-mail expanded, and once these people realized what it could be
used for, things lightened up.
Electronic mailing lists provided communities where people with like interests could exchange messages.
These lists were closed systems, in the sense that only subscribers could post messages to the list or view
messages posted by other subscribers. Obviously, lists grew and list managers had to maintain them. Over
time, automated mechanisms were developed to allow people to subscribe (and, even more importantly, to
unsubscribe) without human intervention. These mailing lists evolved into message forums, where people could
publicly post messages on an electronic bulletin board for everyone to read.
These services certainly existed before the birth of the Internet. But in those days, users read and sent their
e-mail by logging in to a system directly (usually via telephone dial-up or direct local connection) and running
programs on that system (usually with a command-line interface). The methods for using these services varied
greatly from system to system, and e-mail connectivity between disparate systems was hard to come by. With
the advent of TCP/IP, the mechanisms for providing these services became more consistent, and e-mail
became uniform and ubiquitous.
The transmission of e-mail is performed through the SMTP protocol. The reading of e-mail is usually
performed through either POP or IMAP.
SMTP
As an application layer protocol, Simple Mail Transfer Protocol (SMTP) normally runs on top of TCP, though
it can theoretically use any underlying transport protocol. The latest specification for the SMTP protocol is
defined in Internet RFC 821 and the structure of SMTP messages is defined in Internet RFC 822. The
application called “sendmail” is an implementation of the SMTP protocol for Unix systems.
SMTP, like other TCP/IP services, runs as a server, service, or daemon. In a TCP/IP environment, SMTP
servers usually run on port 25. They wait for requests to send e-mail messages, which can come from local
system users or from across the network. They are also responsible for evaluating the recipient addresses found
in e-mail messages, determining whether they are valid, and whether their final destination is another
recipient (e.g., a forwarding address or the set of individual recipients subscribed to a mailing list).
If the message embedded in the request is intended for a user with an account on the local system, then the
SMTP server delivers the message to that user by appending it to their mailbox. Depending on the
implementation, the mailbox can be anything from a simple text file to a complex database of e-mail
messages. If the message is intended for a user on another system, then the server must figure out how to
transmit the message to the appropriate system. This may involve direct connection to the remote system or it
may involve connection to a gateway system. A gateway is responsible for passing a message on to other
gateways or sending it directly to its ultimate destination.
40
Before the advent of SMTP, the underlying mechanisms for sending mail varied from system to system.
Once SMTP became ubiquitous as the mechanism for e-mail transmission, these mechanisms became more
uniform. Applications responsible for transmitting e-mail messages, such as SMTP servers, are known as Mail
Transfer Agents (MTAs). Likewise, the applications responsible for retrieving messages from a mailbox,
including POP servers and IMAP servers, are known as Mail Retrieval Agents (MRAs).
E-mail client programs are engineered to allow users to read and send mail. Such programs are known as
Mail User Agents (MUAs). MUAs talk to MRAs to read mail and to MTAs to send mail. In a typical e-mail
client, once the user has composed and submitted a message, the client program directs it to the SMTP
server. First, the client connects to the server by opening a TCP socket to port 25 (the SMTP port) of the
server (even if the server is running on the user’s machine). Figure 2.1 shows an example of an interaction
between a client and an SMTP server.
The client program identifies itself (and the system on which it is running) to the server via the HELO
command. The server decides (based on this identification information) whether to accept or reject the
request. If the server accepts the request, it waits for the client to send further information. One line at a time,
the client transmits commands to the server, sending information about the originator of the message (using
the MAIL command) and each of the recipients (using a series of RCPT commands). Once all this is done,
the client tells the server it is about to send the actual data: the message itself. It does this by sending a
command line consisting of the word DATA. Every line that follows, until the server encounters a line
containing only a period, is considered part of the message body. Once it has sent the body of the message,
the client signals the server that it is done, and the server transmits the message to its destination (either
directly or through gateways). Having received confirmation that the server has transmitted the message, the
client closes the socket connection using the QUIT command.
41
Originally, SMTP servers executed in a very open fashion: anyone knowing the address of an SMTP server
could connect to it and send messages. In an effort to discourage spamming (the sending of indiscriminate
mass semi-anonymous e-mails), many SMTP server implementations allow the system administrator to
configure the server so that it only accepts connections from a discrete set of systems, perhaps only those
within their local domain. Additionally, SMTP servers can use an extension to the SMTP protocol called
SMTP-AUTH to restrict the ability to send e-mail only to authenticated users.
When building web applications that include e-mail functionality (specifically the sending of e-mail), make
sure your configuration includes the specification of a working SMTP server system, which will accept your
requests to transmit messages. To maximize application flexibility, the address of the SMTP server should be
a parameter that can be modified at run time by an application administrator.
MIME
Originally, e-mail systems transmitted messages in the form of standard ASCII text. If a user
wanted to send a file in a non-text or “binary” format (e.g., an image or sound file), it had to be
encoded before it could be placed into the body of the message. The sender had to
communicate the nature of the binary data directly to the receiver, e.g., “The block of encoded
binary text below is a GIF image.”
Multimedia Internet Mail Extensions (MIME) provided a uniform mechanism for
including encoded attachments within a multipart e-mail message. MIME supports the
definition of boundaries separating the text portion of a message (the “body”) from its
attachments, as well as the designation of attachment encoding methods, including “Base64”
and “quoted-printable.” MIME was originally defined in Internet RFC 1341, but the most
recent specifications can be found in Internet RFCs 2045 through 2049.
MIME also supports the notion of content typing for attachments (and for the body of a
message). MIME-types are standard naming conventions for defining the type of data
contained in an attachment. A MIME-type is constructed as a combination of a top-level data
type and a subtype. There is a fixed set of top-level data types, including text, image, audio,
video, and application. The subtypes describe the data in enough detail to select an appropriate
processing program. For example, there are specific programs for processing text/html,
text/plain, image/jpeg, and audio/mp3 files.
POP
The Post Office Protocol (POP) gives users direct access to their e-mail messages stored on remote systems.
POP3 is the most recent version of the POP protocol, first defined in Internet RFC 1725 and revised in
Internet RFC 1939. Most of the popular e-mail clients (including Microsoft Outlook and Mozilla
Thunderbird) use POP3 to access user e-mail. (Even proprietary systems, such as Lotus Notes, offer
administrators the option to configure remote e-mail access through POP.)
Before the Internet, people read and sent e-mail by logging in to a system and running command-line
programs. User messages were usually stored locally in a mailbox file on that system. Even with the advent of
42
Internet technology, many people continued to access e-mail by establishing a Telnet connection to the
system containing their mailbox and running command-line programs (e.g., from a Unix shell) to read and
send mail. (Some people who prefer command-line programs still do!)
Let us look at the process by which POP clients communicate with POP servers to provide user access to e-
mail. First, the POP client must connect to the POP server (which usually runs on port 110), so it can identify
and authenticate the user to the server. This is usually done by sending the user name (user id) and password
one line at a time, using the USER and PASS commands. (Sophisticated POP servers may make use of the
APOP command, which allows the secure transmission of the user name and password as a single encrypted
entity across the network.)
Once connected and authenticated, the POP protocol offers the client a variety of commands it can
execute. Among them is the UIDL command, which responds with an ordered list of message numbers,
where each entry is followed by a unique message identifier. POP clients can use this list (and the unique
identifiers it contains) to determine which messages in the list qualify as “new” (i.e., not yet seen by the user
through this particular client). Having obtained this list, the client can execute the command to retrieve a
message (RETR n). It can also execute commands to delete a message from the server (DELE n) or retrieve
just the header of a message (TOP n 0).
Message headers contain metadata about a message, such as the addresses of its originator and recipients, its
subject, etc. Each message contains a header block (see Figure 2.2) containing a series of lines, followed by a
blank line indicating the end of that block.
The information that e-mail clients include in message lists (e.g., the From, To, and Subject of each
message) comes from the message headers. As e-mail technology advanced, headers began representing more
sophisticated information, including MIME-related data (e.g., content types) and attachment encoding
schemes.
Figure 2.3 provides an example of a simple command-line interaction between a client and a POP server.
GUI-based clients often hide the mundane command-line details from their users. The normal sequence of
operation for most GUI-based POP clients today is as follows:
1. Get the user id and password (the client may already have this information or may need to
prompt the user).
2. Connect the user and verify identity.
3. Obtain the UIDL list of messages.
4. Compare the identifiers in this list to a list that the client keeps locally, to determine which
messages are “new”.
5. Retrieve all the new messages and present them to the user in a selection list.
6. Delete the newly retrieved messages from the POP server (optional).
Although this approach is simple, it is inefficient. All the new messages are always downloaded to the
client. This is inefficient because some of these messages may be quite long or have extremely large
attachments. Users must wait for all of the messages (including large or unwanted messages) to download
before viewing any of the messages they want to read. The current proliferation of spam makes this even more
important today than it was in the past. It would be more efficient for the client to retrieve only message
43
headers and display the information in a message list. It could then allow users the option to selectively
download desired messages for viewing or to delete unwanted messages without downloading them. A web-
based e-mail client helps remove some of this inefficiency.
IMAP
Some of the inefficiencies of the POP protocol can be alleviated by the Internet Message Access Protocol
(IMAP). IMAP was intended as a successor to POP, offering sophisticated services for managing messages in
remote mailboxes. IMAP servers provide support for multiple remote mailboxes or folders, so that users can
move messages from an incoming folder (the “inbox”) into other folders kept on the server. In addition, they
also provide support for saving sent messages in one of these remote folders and for multiple simultaneous
operations on mailboxes. IMAP servers commonly listen on TCP port 143.
IMAP4, the most recent version of the IMAP protocol, was originally defined in Internet RFC 1730, but
the most recent specification can be found in Internet RFC 2060. The approach used by IMAP differs in
many ways from that of POP. POP clients download e-mail messages from the server. The default behavior
for many POP clients is to delete messages from the server following the download; in practice, many users
elect to leave viewed messages on the server, rather than deleting them after viewing. This is because many
44
people who travel extensively want to check e-mail while on the road, but want to see all of their messages
(even the ones they’ve seen) when they return to their “home machine.” While the POP approach “tolerates”
but does not encourage this sort of user behavior, the IMAP approach eagerly embraces it. IMAP was
conceived with “nomadic” users in mind: users who might check e-mail from literally anywhere, who want
access to all of their saved and sent messages wherever they happen to be. IMAP not only allows the user to
leave messages on the server, it provides mechanisms for storing messages in user-defined folders for easier
accessibility and better organization. Moreover, users can save sent messages in a designated remote folder on
the IMAP server. While POP clients support saving of sent messages, they usually save those messages
locally, on the client machine.
IMAP e-mail client programs work very similarly to POP e-mail clients. In fact, many e-mail client
programs allow users to operate in either POP or IMAP mode. However, the automatic downloading of the
content and attachments for incoming messages does not occur by default in IMAP clients. Instead, an IMAP
client downloads only the header information associated with new messages, requesting the body of an
individual message only when user expresses interest in seeing it.
POP vs IMAP
Although it is possible to write a POP client that downloads only the header information until
the user expressly looks at a message, most do not. POP clients tend to operate in “burst”
mode, getting all the messages on the server in one “shot.” While this may be in some respects
inefficient, it is useful for those whose on-line access is not persistent. By getting all the
messages in one burst, users can work off-line with the complete set of downloaded messages,
connecting to the Internet again only when they want to send responses and check for new
mail.
IMAP clients assume the existence of a persistent Internet connection, allowing discrete
actions to be performed on individual messages, while maintaining a connection to the IMAP
server. Thus, for applications where Internet connectivity may not be persistent (e.g., a
handheld device where Internet connectivity is paid for by the minute), POP might be a better
choice than IMAP.
Because IMAP offers many more options than POP, the possibilities for what can go on in a user session
are much richer. After connection and authentication, users can check new messages, recently seen messages,
unanswered messages, flagged messages, and drafts of messages yet to go out. They can view messages either
in their entirety or in part (e.g., header, body, attachment). They can delete or move messages to other folders.
They can also respond to messages or forward them to others.
IMAP need not be used strictly for e-mail messages. Because security features allow mailbox folders to be
designated as “read only”, IMAP can be used for “message board” functionality as well. However, such
functionality is usually reserved for message forum services.
Today, web-based e-mail systems, such as Hotmail, Yahoo! Mail, and Gmail, provide users with the means to
45
access e-mail through a web browser, without requiring a specialized locally configured e-mail client program
to access designated POP/IMAP and SMTP servers. This means users can read their e-mail from practically
anywhere, without having to use their own computer. Users behind network firewalls, which block the ports
required for POP3, IMAP, and SMTP access to outside e-mail servers, can read and send e-mail through
web-based services.
Almost all web-based e-mail services employ standard Internet e-mail protocols to store and send e-mail.
Many ISPs provide web-based e-mail services that use IMAP as the underlying storage and retrieval
mechanism, but access to their SMTP and IMAP servers from the Internet is usually blocked. Blocked IMAP
and SMTP servers can be accessed only through the web-based e-mail interfaces. Some services, including
Gmail, expose their POP3 servers to allow e-mail clients retrieve messages (Yahoo! Mail does this only for
paying customers). Open IMAP access is slowly becoming more commonplace, with both AOL and Gmail
now providing this service. The vast majority of services tightly control access to SMTP servers for sending e-
mail to prevent abuse by spammers.
46
2.3.3 Message forums
Message forums are on-line services that allow users to write messages to be posted on the equivalent of an
electronic bulletin board and to read messages that others have posted. These messages are usually organized
into categories so that people can find the kinds of messages they are looking for.
For years, on-line message forums existed in various forms. Perhaps the earliest form was the electronic
mailing list. As we mentioned earlier, mailing lists are closed systems: only subscribers can view or post
messages. In some situations, a closed private community may be exactly what the doctor ordered, but if the
goal is to have open public participation, message forums are more appropriate.
Although message forums were originally localized, meaning that messages appeared only on the system
where they were posted, the notion of distributed message forums took hold. Cooperative networks (e.g.,
FIDONET) allowed systems to share messages, by forwarding them to their network “neighbors.” This
enabled users to see all the messages posted by anyone on any member system.
The original Internet-based version of message forums was Netnews. Netnews organizes messages into
newsgroups, which form a large hierarchy of topics and categories. Among the main divisions are comp (for
computing-related newsgroups), sci (for scientific newsgroups), soc (for socially oriented newsgroups), talk
(for newsgroups devoted to talk) and alt (an unregulated hierarchy for “alternative” newsgroups). The naming
convention for newsgroups is reminiscent of domain names in reverse, e.g., comp.infosystems.www.
Netnews existed before the proliferation of the Internet. It grew out of Usenet, an
interconnected network of Unix systems. Before the Internet took hold, Unix systems
communicated with each other over UUCP, a protocol used to transmit mail and news over
phone lines. It has been suggested, only half in jest, that the proliferation of Unix by Bell
Laboratories in the 1980s was an effort by AT&T to increase long-distance phone traffic, since
e-mail and Netnews were transmitted by long-distance calls between these Unix systems.
Today, Netnews is transmitted using an Internet protocol called Network News Transfer Protocol (NNTP).
The NNTP specification is defined in the Internet RFC 977. NNTP clients allow users to read (and post)
messages in newsgroups by connecting to NNTP servers. These servers propagate the newsgroup messages
throughout the world by regularly forwarding them to “neighboring” servers.
Netnews functionality is directly incorporated into browsers such as Mozilla, where it is included in the
Thunderbird component. It is possible to create web applications that provide Netnews access through normal
web browser interactions. Google Groups (originally called Deja News) provides an infrastructure for
accessing current and archived newsgroup messages, using the Google search engine to find desired messages.
On-line forums continue to exist in a wide variety of formats. They are commonly implemented as
database-driven services local to their related web sites. Today, the Netnews protocol is relegated to a niche
role in the on-line forums world, but much can be learned from it when it comes to building collaborative
technology for web applications. For the most part, Netnews functionality has been supplanted by web-based
47
forums and blogging services.
48
2.3.4 Chat and messaging protocols
The Instant Messaging (IM) service of America On-line (now AOL) may be responsible for making the
notion of IM-ing someone part of our collective vocabulary but, long before AOL, there was the talk protocol
which enabled users who were logged in to network-connected Unix systems to talk to each other.
A talk server would run on a Unix machine, waiting for requests from other talk servers. (Since talk was a
bi-directional service, servers had to run on the machines at both ends of a conversation.) A user would invoke
a talk client program to communicate with a person on another machine somewhere else on the network, e.g.,
elvis@graceland.org. The talk client program would communicate with the local talk server, which would ask
the talk server on the remote machine whether the other person is on-line. If so, and if that other person was
accepting talk requests, the remote talk server would establish a connection and the two people would use a
screen-mode interface to have an on-line conversation.
Today, the vast majority of Internet users eschew command-line interfaces and the notion of being logged
in to a particular system to communicate with others (the way people used to connect to AOL or
Compuserve) is alien to most people. A protocol such as talk would not work in its original form in today’s
diverse Internet world.
Efforts to create an open, interoperable Instant Messaging protocol have been unsuccessful thus far.
Proprietary instant-messaging systems (such as those from AOL, Yahoo!, and Microsoft) are exclusive, and
intense competition yields a lack of cooperation between instant messaging providers that further limits the
degree of interoperability we can get from them. Many shareware or open-source client programs for instant
messaging implement multiple protocols to provide access to multiple instant-messaging services. For now,
the best solution would seem to be provided by tools that attempt to connect to the widest variety of services.
The providers of these services make this difficult by keeping their specifications proprietary and updating
them frequently, so it is a struggle to keep ahead of these changes. XMPP (formerly known as Jabber) is an
open source instant-messaging protocol that is gaining traction but has not achieved the popularity of better-
known commercial services.
49
2.3.5 Security and encryption
Virtually all the data we’ve discussed up to this point is transmitted “in the clear,” meaning it is readable as
plain text. A “packet sniffer” (a program that scans a TCP/IP network and collects transmitted data) can easily
retrieve all this data. (Although the packet nature of TCP/IP would require observing programs to reorder
and properly sequence all the gathered packets to make any sense of them, such a task is not difficult.) This
means that all of your network traffic (e-mail, chat messages, Telnet connections, etc.) is available to anyone
with the tools to scan that traffic. Securing your data transmissions means encrypting the data so that a hacker
cannot easily decrypt it without knowing the specific “key” associated with the encryption process.
Secure Shell (SSH) serves as a secure replacement for Telnet, encrypting the traffic that would normally be
exposed in a standard Telnet connection. SSH is more than just an alternative to Telnet. Using public key
authentication and the capability called tunneling, SSH enables sophisticated mechanisms for secure data
transmission.
Public key authentication requires a key pair consisting of a private key and a public key. The basic principle
is that messages encrypted with the public key can only be decrypted with the corresponding private key and
vice versa. The key pair is generated by its owner using a key generation utility that is usually included with
SSH client programs. For an individual user, the private key is kept on the user’s computer at a location
known to the SSH client program. The public key is placed on the server machine at a location known to the
SSH server program. Keys are complex (usually 1024 to 2048 bits in size) and breaking the encryption is
sufficiently difficult to deter most attempts.
In practice, this exchange of keys is bidirectional. A server publishes its public key, often called a “host key.”
Clients connecting with that server download its host key and persist it locally. During subsequent
connections, the server uses its private key to encrypt a message for the client. If the client can successfully
decrypt this message using the downloaded host key, this confirms that the server is the machine it claims to
be.
The SSH server tries to authenticate a user attempting to connect using public key authentication. The
authentication process works as follows:
50
Other documents randomly have
different content
la Patria, unida en perpetuo y amistoso vínculo con la Religión, cual
lo exige la prosperidad de esta nuestra desgraciada España.
Madrid y Febrero de 1904.
Fr. Bernardino,
Arzobispo dimisionario de Manila y electo
de Valencia.
APÉNDICE
DOCUMENTO NÚM. 1
*
* *
*
* *
ebookgate.com