Figure 2.4-1: A bird's eye view of the Internet e-mail system.
Figure 2.4-1 presents a high-level view of the Internet mail system.
We see from this diagram that it has three major components: user agents,
mail
servers, and the Simple Mail Transfer Protocol (SMTP). We now
describe each of these components in the context of a sender, Alice 
, sending an email message to a recipient, Bob 
.
User agents allow users to read, reply to, forward, save, and compose messages.
(User agents for electronic mail are sometimes called
mail
readers, although we will generally avoid this term in this book.)
When Alice is finished composing her message, her user agent sends
the message to her mail server, where the message is placed in the mail
server's outgoing message queue. When Bob wants to read a message, his
user agent obtains the message from his mailbox in his mail server.
In
the late 1990s, GUI (graphical user interface) user agents became popular,
allowing users to view and compose multimedia messages. Currently, Eudora,
Microsoft's Outlook Express, and Netscape's Messenger are among the popular
GUI user agents for email. There are also many text-based
email user interfaces in the public domain, including mail, pine and elm.
Mail servers form the core of the e-mail infrastructure. Each recipient, such as Bob, has a mailbox located in one of the mail servers. Bob's mailbox manages and maintains the messages that have been sent to him. A typical message starts its journey in the sender's user agent, travels to the sender's mail server, and then travels to the recipient's mail server, where it is deposited in the recipient's mailbox. When Bob wants to access the messages in his mailbox, the mail server containing the mailbox authenticates Bob (with user names and passwords). Alice's mail server must also deal with failures in Bob's mail server. If Alice's server cannot deliver mail to Bob's server, Alice's server holds the message in a message queue and attempts to transfer the message later. Reattempts are often done every 30 minutes or so; if there is no success after several days, the server removes the message and notifies the sender (Alice) with an email message.
The Simple Mail Transfer Protocol (SMTP) is the principle application-layer protocol for Internet electronic mail. It uses the reliable data transfer service of TCP to transfer mail from the sender's mail server to the recipient's mail server. As with most application-layer protocols, SMTP has two sides: a client side which executes on the sender's mail server, and server side which executes on the recipient's mail server. Both the client and server sides of SMTP run on every mail server. When a mail server sends mail (to other mail servers), it acts as an SMTP client. When a mail server receives mail (from other mail servers) it acts as an SMTP server.
To illustrate the basic operation of SMTP, let's walk through a common scenario. Suppose Alice wants to send Bob a simple ASCII message:
It is important to observe that SMTP does not use intermediate mail servers for sending mail, even when the two mail servers are located at opposite ends of the world. If Alice's server is in Hong Kong and Bob's server is in Mobile, Alabama, the TCP "connection" is a direct connection between the Hong Kong and Mobile servers. In particular, if Bob's mail server is down, the message remains in Alice's mail server and waits for a new attempt -- the message does not get placed in some intermediate mail server.
Let's now take a closer look at how SMTP transfers a message from a sending mail server to a receiving mail server. We will see that the SMTP protocol has many similarities with protocols that are used for face-to-face human interaction. First, the client SMTP (running on the sending mail server host) has TCP establish a connection on port 25 to the server SMTP (running on the receiving mail server host). If the server is down, the client tries again later. Once this connection is established, the server and client perform some application-layer handshaking. Just as humans often introduce themselves before transferring information from one to another, SMTP clients and servers introduce themselves before transferring information. During this SMTP handshaking phase, the SMTP client indicates the email address of the sender (the person who generated the message) and the email address of the recipient. Once the SMTP client and server have introduced themselves to each other, the client sends the message. SMTP can count on the reliable data transfer service of TCP to get the message to the server without errors. The client then repeats this process over the same TCP connection if it has other messages to send to the server; otherwise, it instructs TCP to close the connection.
Let us take a look at an example transcript between client (C) and server (S). The host name of the client is crepes.fr and the host name of the server is hamburger.edu. The ASCII text prefaced with C: are exactly the lines the client sends into its TCP socket; and the ASCII text prefaced with S: are exactly the lines the server sends into its TCP socket. The following transcript begins as soon as the TCP connection is established:
It is highly recommended that you use Telnet to carry out a direct dialogue with an SMTP server. To do this, issue telnet serverName 25 . When you do this, you are simply establishing a TCP connection between your local host and the mail server. After typing this line, you should immediately receive the 220 reply from the server. Then issue the SMTP commands HELO, MAIL FROM, RCPT TO, DATA, and QUIT at the appropriate times. If you Telnet into your friend's SMTP server, you should be able to send mail to your friend in this manner (i.e., without using your mail user agent).
Comparison with HTTP
Let us now briefly compare SMTP to HTTP. Both protocols are used to transfer files from one host to another; HTTP transfers files (or objects) from Web server to Web user agent (i.e., the browser); SMTP transfers files (i.e., email messages) from one mail server to another mail server. When transferring the files, both persistent HTTP and SMTP use persistent connections, that is, they can send multiple files over the same TCP connection. Thus the two protocols have common characteristics. However, there are important differences. First, HTTP is principally a pull protocol -- someone loads information on a Web server and users use HTTP to pull the information off the server at their convenience. In particular, the TCP connection is initiated by the machine that wants to receive the file. On the other hand, SMTP is primarily a push protocol -- the sending mail server pushes the file to the receiving mail server. In particular, the TCP connection is initiated by the machine that wants to send the file.
A second important difference, which we alluded to earlier, is that SMTP requires each message, including the body of each message, to be in seven-bit ASCII format. Furthermore, the SMTP RFC requires the body of every message to end with a line consisting of only a period -- i.e., in ASCII jargon, the body of each message ends with "CRLF.CRLF", where CR and LF stand for carriage return and line feed, respectively. In this manner, while the SMTP server is receiving a series of messages from an SMTP client over a persistent TCP connection, the server can delineate the messages by searching for "CRLF.CRLF" in the byte stream. (This operation of searching through a character stream is referred to as "parsing".) Now suppose that the body of one of the messages is not ASCII text but instead binary data (for example, a JPEG image). It is possible that this binary data might accidentally have the bit pattern associated with ASCII representation of "CR LF . CR LF" in the middle of the bit stream. This would cause the SMTP server to incorrectly conclude that the message has terminated. To get around this and related problems, binary data is first encoded to ASCII in such a way that certain ASCII characters (including ".") are not used. Returning to our comparison with HTTP, we note that neither non-persistent nor persistent HTTP has to bother with the ASCII conversion. For non-persistent HTTP, each TCP connection transfers exactly one object; when the server closes the connection, the client knows it has received one entire response message. For persistent HTTP, each response message includes a Content-length: header line, enabling the client to delineate the end of each message.
A third important difference concerns how a document consisting of text and images (along with possibly other media types) is handled. As we learned in Section 2.3, HTTP encapsulates each object in its own HTTP response message. Internet mail, as we shall discuss in greater detail below, places all of the message's objects into one message.
A typical message header looks like this:
The MIME Extension for Non-ASCII Data
While the message headers described in RFC 822 are satisfactory for sending ordinary ASCII text, they are not sufficiently rich enough for multimedia messages (e.g., messages with images, audio and video) or for carrying non-ASCII text formats (e.g., characters used by languages other than English). To send content different from ASCII text, the sending user agent must include additional headers in the message. These extra headers are defined in [RFC 2045] and [RFC 2046], the MIME extension to [RFC 822]. Two key MIME headers for supporting multimedia are the Content-Type: header and the Content-Transfer-Encoding: header. The Content-Type: header allows the receiving user agent to take an appropriate action on the message. For example, by indicating that the message body contains a JPEG image, the receiving user agent can direct the message body to a JPEG decompression routine. To understand the need of the Content-Transfer-Encoding: header, recall that non-ASCII text messages must be encoded to an ASCII format that isn't going to confuse SMTP. The Content-Transfer-Encoding: header alerts the receiving user agent that the message body has been ASCII encoded and the type of encoding used. Thus, when a user agent receives a message with these two headers, it first uses the value of the Content-Transfer-Encoding: header to convert the message body to its original non-ASCII form, and then uses the Content-Type: header to determine what actions it should take on the message body.
Let's take a look at a concrete example. Suppose Alice wants to send Bob a JPEG image. To do this, Alice invokes her user agent for email, specifies Bob's email address, specifies the subject of the message, and inserts the JPEG image into the message body of the message. (Depending on the user agent Alice uses, she might insert the image into the message as an "attachment".) When Alice finishes composing her message, she clicks on "Send". Alice's user agent then generates a MIME message, which might look something like this:
base64
encoded data .....
.........................
......base64
encoded data
When Bob reads his mail with his user agent, his user agent operates on this same MIME message. When Bob's user agent observes the Content-Transfer-Encoding: base64 header line, it proceeds to decode the base64-encoded message body. The message also includes a Content-Type: image/jpeg header line; this indicates to Bob's user agent that the message body (after base64 decoding) should be JPEG decompressed. Finally, the message includes the MIME-Version: header, which, of course, indicates the MIME version that is being used. Note that the message otherwise follows the standard RFC 822/SMTP format. In particular, after the message header there is a blank line and then the message body; and after the message body, there is a line with a single period.
Let's now take a closer look at the Content-Type: header. According to the MIME specification, [RFC 2046], this header has the following format:
where the "parameters" (along with the semi-colon) is optional. Paraphrasing [RFC 2046], the Content-Type field is used to specify the nature of the data in the body of a MIME entity, by giving media type and subtype names. After the type and subtype names, the remainder of the header field is a set of parameters. In general, the top-level type is used to declare the general type of data, while the subtype specifies a specific format for that type of data. The parameters are modifiers of the subtype, and as such do not fundamentally affect the nature of the content. The set of meaningful parameters depends on the type and subtype. Most parameters are associated with a single specific subtype. MIME has been carefully designed to be extensible, and it is expected that the set of media type/subtype pairs and their associated parameters will grow significantly over time. In order to ensure that the set of such types/subtypes is developed in an orderly, well-specified, and public manner, MIME sets up a registration process which uses the Internet Assigned Numbers Authority (IANA) as a central registry for MIME's various areas of extensibility. The registration process for these areas is described in [RFC 2048].
Currently there are seven top-level types defined. For each type, there is a list of associated subtypes, and the lists of subtypes are growing every year. We describe five of these types below:
To obtain a better understanding of multipart/mixed, let's look at an example. Suppose that Alice wants to send a message to Bob consisting of some ASCII text, followed by a JPEG image, followed by more ASCII text. Using her user agent, Alice types some text, attaches a JPEG image, and then types some more text. Her user agent then generates a message something like this:
From: alice@crepes.fr
To: bob@hamburger.edu
Subject: Picture of yummy crepe with commentary
MIME-Version: 1.0
Content-Type:
multipart/mixed; Boundary=StartOfNextPart
--StartOfNextPart
Dear Bob,
Please find a picture of an absolutely
scrumptious crepe.
--StartOfNextPart
Content-Transfer-Encoding:
base64
Content-Type: image/jpeg
base64
encoded data .....
.........................
......base64
encoded data
As mentioned earlier, the list of registered MIME types grows every year. The RFC [2048] describes the registration procedures which use the Internet Assigned Numbers Authority (IANA) as a central registry for such values. A list of the current MIME subtypes is maintained at numerous sites. The reader is also encouraged to glance at Yahoo's MIME Category Page.
The Received Message
As we have discussed, an email message consists of many components.
The core of the message is the message body, which is the actually data
being sent from sender to receiver. For a multipart message, the message
body itself consists of many parts, with each part preceded with one or
more lines of peripheral information. Preceding the message body is a blank
line and then a number of header lines. These header lines include RFC
822 header lines such as From:, To:
and Subject: header lines. The
header lines also include MIME header lines such as Content-type:
and
Content-transfer-encoding:
header lines. But we would be remiss if we didn't mention another class
of header lines that are inserted by the SMTP receiving server.
Indeed, the receiving server, upon receiving a message with RFC 822 and
MIME header lines, appends a Received:
header line to the top of the message; this header line specifies the name
of the SMTP server that sent the message ("from"), the name of the SMTP
server that received the message ("by") and the time at which the receiving
server received the message. Thus the message seen by the destination user
takes the following form:
base64 encoded data .......
........................................
.......base64 encoded data
Given that Bob (the recipient) executes his user agent on the his local PC, it is natural to consider placing a mail server on the his local PC as well. There is a problem with this approach, however. Recall that a mail server manages mailboxes and runs the client and server sides of SMTP. If Bob's mail server were to reside on his local PC, then Bob's PC would have to remain constantly on, and connected to the Internet, in order to receive new mail, which can arrive at any time. This is impractical for the great majority of Internet users. Instead, a typical user runs a user agent on the local PC but accesses a mailbox from a shared mail server - a mail server that is always running, that is always connected to the Internet, and that is shared with other users. The mail server is typically maintained by the user's ISP, which could be a residential or an institutional (university, company, etc.) ISP.
With user agents running on users' local PCs and mail servers hosted by ISPs, a protocol is needed to allow the user agent and the mail server to communicate. Let us first consider how a message that originates at Alice's local PC makes its way to Bob's SMTP mail server. This task could simply be done by having Alice's user agent communicate directly with Bob's mail server in the language of SMTP: Alice's user agent would initiate a TCP connection to Bob's mail server, issue the SMTP handshaking commands, upload the message with the DATA command, and then close the connection. This approach, although perfectly feasible, is not commonly employed, primarily because it doesn't offer the Alice any recourse to a crashed destination mail server. Instead, Alice's user agent initiates a SMTP dialogue with her own mail server (rather than with the recipient's mail server) and uploads the message. Alice's mail server then establishes a new SMTP session with Bob's mail server and relays the message to Bob's mail server. If Bob's mail server is down, then Alice's mail server holds the message and tries again later. The SMTP RFC defines how the SMTP commands can be used to relay a message across multiple SMTP servers.
But there is still one missing piece to the puzzle! How does a recipient like Bob, running a user agent on his local PC, obtain his messages, which are sitting on a mail server within Bob's ISP? The puzzle is completed by introducing a special access protocol that transfers the messages from Bob's mail server to the local PC. There are currently two popular mail access protocols: POP3 (Post Office Protocol - Version 3) and IMAP (Internet Mail Access Protocol). We shall discuss both of these protocols below. Note that Bob's user agent can't use SMTP to obtain the messages: obtaining the messages is a pull operation whereas SMTP is a push protocol. Figure 2.4-3 provides a summary of the protocols that are used for Internet mail: SMTP is used to transfer mail from the sender's mail server to the recipient's mail server; SMTP is also used to transfer mail from the sender's user agent to the sender's mail server. POP3 or IMAP are used to transfer mail from the recipient's mail server to the recipient's user agent.
POP3
POP3, defined in [RFC 1939], is an extremely simple mail access protocol. Because the protocol is so simple, its functionality is rather limited. POP3 begins when the user agent (the client) opens a TCP connection to the the mail server (the server) on port 110. With the TCP connection established, POP3 progresses through three phases: authorization, transaction and update. During the first phase, authorization, the user agent sends a user name and a password to authenticate the user downloading the mail. During the second phase, transaction, the user agent retrieves messages. During the transaction phase, the user agent can also mark messages for deletion, remove deletion marks, and obtain mail statistics. The third phase, update, occurs after the client has issued the quit command ending the POP3 session; at this time, the mail server deletes the messages that were marked for deletion.
In a POP3 transaction, the user agent issues commands, and the server responds to each command with a reply. There are two possible responses: +OK (sometimes followed by server-to-client data), whereby the server is saying that the previous command was fine; and -ERR, whereby the server is saying that something was wrong with the previous command.
The authorization phase has two principle commands: user<user name> and pass<password>. To illustrate these two commands, we suggest that you Telnet directly into a POP3 server, using port 110, and issue these commands. Suppose that mailServer is the name of your mail server. You will see something like:
Now let's take a look at the transaction phase. A user agent using POP3 can often be configured (by the user) to "download and delete" or to "download and keep". The sequence of commands issued by a POP3 user agent depend on which of these two modes the user agent is operating in. In the download-and-delete mode, the user agent will issue the list, retr and dele commands. As an example, suppose the user has two messages in his or her mailbox. In the dialogue below C: (standing for client) is the user agent and S: (standing for server) is the mail server. The transaction will look something like:
A problem with this download-and-delete mode is that the recipient, Bob, may be nomadic and want to access his mail from multiple machines, including the office PC, the home PC and a portable computer. The download-and-delete mode scatters Bob's mail over all the local machines; in particular, if Bob first reads a message on a home PC, he will not be able to reread the message on his portable later in the evening. In the download-and-keep mode, the user agent leaves the messages on the mail server after downloading them. In this case, Bob can reread messages from different machines; he can access a message from work, and then access it again later in the week from home.
During a POP3 session between a user agent the mail server, the POP3 server maintains some state information; in particular, it keeps track of which messages have been marked deleted. However, the POP3 server is not required to carry state information across POP3 sessions. For example, no message is marked for deletion at the beginning of each session. This lack of state information across sessions greatly simplifies the implementation of a POP3 server.
IMAP
Once Bob has downloaded his messages to the local machine using POP3, he can create mail folders and move the downloaded messages into the folders. Bob can then delete messages, move messages across folders, and search for messages (say by sender name or subject). But this paradigm -- folders and messages in the local machine -- poses a problem for the nomadic user, who would prefer to maintain a folder hierarchy on a remote server that can be accessed by from any computer. This is not possible with POP3.
To solve this and other problems, the Internet Mail Access Protocol (IMAP), defined in [RFC 1730], was invented. Like POP3, IMAP is a mail access protocol. It has many more features than POP3, but it is also significantly more complex. (And thus the client and server side implementations are significantly more complex.) IMAP is designed to allow users to manipulate remote mailboxes as if they were local. In particular, IMAP enables Bob to create and maintain multiple message folders at the mail server. Bob can put messages in folders and move messages from one folder to another. IMAP also provides commands that allow Bob to search remote folders for messages matching specific criteria. One reason why an IMAP implementation is much more complicated than a POP3 implementation is that the IMAP server must maintain a folder hierarchy for each of its users. This state information persists across a particular user's successive accesses to the IMAP server. Recall that a POP3 server, by contrast, does not maintain anything about a particular user once the user quits the POP3 session.
Another important feature of IMAP is that it has commands that permit a user agent to obtain components of messages. For example, a user agent can obtain just the message header of a message or just one part of a multipart MIME message. This feature is useful when there is a low-bandwidth connection between the user agent and its mail server, for example, a wireless or slow-speed modem connection. With a low-bandwidth connection, the user may not want to download all the messages in its mailbox, particularly avoiding long messages that might contain, for example, an audio or video clip.
An IMAP session consists of the establishment of a connection between the client (i.e., the user agent) and the IMAP server, an initial greeting from the server, and client-server interactions. The client/server interactions are similar to, but richer than, those of POP3. They consist of a client command, server data, and a server completion result response. The IMAP server is always in one of four states. In the non-authenticated state, which starts when the connection starts, the user must supply a user name and password before most commands will be permitted. In the authenticated state, the user must select a folder before sending commands that affect messages. In the selected state, the user can issue commands that affect messages (retrieve, move, delete, retrieve a part in a multipart message, etc.). Finally, the logout state is when the session is being terminated. The IMAP commands are organized by the state in which the command is permitted. You can read all about IMAP at the official IMAP site.
CM e-mail differs from traditional text mail in many ways. These differences
include much larger messages, more stringent end-to-end delay requirements,
and greater sensitivity to recipients with highly heterogeneous Internet
access rates and local storage capabilities. Unfortunately, the current
e-mail infrastructure has several inadequacies that obstruct the widespread
adoption of CM e-mail. First, many existing mail servers do not have the
capacity to store large CM objects; recipient mail servers typically reject
such messages, which makes sending CM messages to such recipients impossible.
Second, the existing mail paradigm of transporting
entire messages
to the recipient's mail server before recipient rendering can lead to excessive
waste of bandwidth and storage. Indeed, stored CM is often not rendered
in its entirety [Padhye 1999], so that bandwidth
and recipient storage is wasted by receiving data that is never rendered.
(For example, one can imagine listening to the first fifteen seconds of
a long audio email from a rather long-winded colleague, and then deciding
to delete the remaining 20 minutes of the message without listening to
it.) Third, current mail access protocols (POP3, IMAP and HTTP) are inappropriate
for streaming CM to recipients. (Streaming CM is discussed in detail in
Chapter 6.) In particular, the current mail access protocols do not provide
functionality that allows a user to pause/resume a message or to reposition
within a message; furthermore, streaming over TCP is often leads to poor
reception (see Chapter 6). These inadequacies will hopefully be addressed
in the upcoming years. Possible solutions are discussed in [Gay
1997] [Hess 1998] [Shurman
1996] and [Turner 1999].
[Gay 1997] V. Gay and B. Dervella, "MHEGAM
- A Multimedia Messaging System," IEEE Multimedia Magazine, Oct-Dec.
1997, pp. 22-29.
[Hess 1998] C. Hess, D. Lin and K.
Nahrstedt, "VistaMail: An Integrated Multimedia Mailing System," IEEE
Multimedia Magazine, Oct.-Dec, 1988, pp. 13-23.
[Hughes 1998] L. Hughes, Internet
E-mail: Protocols, Standards and Implementation, Artech House, Norwood,
MA, 1998.
[Padhye 1999] J. Padhye and J. Kurose,
"An Empirical Study of Client Interactions with a Continuous-Media Courseware
Server," IEEE Internet Computing, April 1999.
[RFC 821] J.B. Postel, "Simple Mail
Transfer Protocol," [RFC
821], August 1982.
[RFC 822] D.H. Crocker, "Standard for
the Format of ARPA Internet Text Messages," [RFC
822], August 1982.
[RFC 977] B. Kantor and P. Lapsley,
"Network News Transfer Protocol," [RFC
977], February 1986.
[RFC 1730] M. Crispin, "Internet Message
Access Protocol - Version 4," [RFC
1730], December 1994.
[RFC 1911] G. Vaudreuil, "Voice Profil
for Internet Mail," [RFC
1911], February 1996.
[RFC 1939] J. Myers and M. Rose, "Post
Office Protocol - Version 3," [RFC
1939], May 1996.
[RFC 2045] N. Borenstein and N. Freed,
"Multipurpose Internet Mail Extensions (MIME) Part One: Format of Internet
Message Bodies,"
[RFC
2045], November 1996.
[RFC 2046] N. Borenstein and N. Freed,
"Multipurpose Internet Mail Extensions (MIME) Part Two: Media Types," [RFC
2046], November 1996.
[RFC 2048] N. Freed, J. Klensin and
J. Postel "Multipurpose Internet Mail Extensions (MIME) Part Four: Registration
Procedures,"
[RFC
2048], November 1996.
[Schurmann 1996] G. Schurmann,
"Multimedia Mail," Multimedia Systems, ACM Press, Oct. 1996, pp.
281-295.
[Turner 1999] D.A. Turner and K.W.
Ross, "Continuous-Media Internet E-Mail: Infrastructure Inadequacies and
Solutions, http://www.eurecom.fr/~ross/MMNetLab.htm
If you are interested in an Internet Draft relating to a certain subject or protocol enter the keyword(s) here.