DEV Community: jrpereirajr

Implementing an IMAP Client in InterSystems IRIS - part II

jrpereirajr — Sat, 12 Feb 2022 15:29:51 +0000

In the first part we got a quick introduction on the IMAP protocol commands, now it's time to use IRIS and implement them and create our own IMAP client!

IRIS Email Framework

The IRIS platform has default interfaces and classes for working with email. Developers originally designed those artifacts for POP3 implementation. However, this doesn’t mean that we can’t use and extend these interfaces and classes to implement an IMAP client. So let’s talk about them:

%Net.Fetch

MailProtocol: This is the base class for email retrieval. The IMAP client extends it.
%Net.MailMessage: This is the MIME message. It extends %Net.MailMessagePart.
%Net.MailMessagePart: This encapsulates a MIME message part for multipart messages. This class has an array for itself, enabling a tree representation of message subparts.
%Net.MIMEReader: This utility class has methods to parse a message’s MIME content, generating a %Net.MIMEPart instance.
%Net.MIMEPart: This encapsulates the message’s MIME parts and provides methods to get information about them.

Implementing an IMAP Client

In this section, we present implementation details about an IMAP client, an inbound interoperability adapter, and a simple production example. Note that, in favor of saving space, we won’t show most implementation methods. Instead, we link to each one’s full implementation details. You can find the complete source code on GitHub.

Creating a Basic IMAP Client

As we discussed before, IMAP is a plain text-based protocol over TCP. This means the base code to implement a client for such a protocol is a TCP client.

The IRIS platform provides standard ObjectScript commands to perform I/O operations: OPEN, USE, READ, WRITE, and CLOSE.

Here is a simple example of how to connect to the MS Outlook server, log in, then log out:

ClassMethod SimpleTest()
{
    // connection configuration
    SET dev = "|TCP|993"
    SET host = "outlook.office365.com"
    SET port = "993"
    SET mode = "C"
    SET sslConfig = "ISC.FeatureTracker.SSL.Config"
    SET timeout = 30
    // connection to MS Outlook IMAP server
    OPEN dev:(host:port:mode:/TLS=sslConfig):timeout
    THROW:('$TEST) ##class(%Exception.General).%New("Sorry, can't connect...")
    USE dev
    READ resp($INCREMENT(resp)):timeout
    WRITE "TAG1 LOGIN user@outlook.com password", !
    READ resp($INCREMENT(resp)):timeout
    WRITE "TAG2 LOGOUT", !
    READ resp($INCREMENT(resp)):timeout
    CLOSE dev
    // come back to default device (terminal) and prints responses
    USE 0
    ZWRITE resp
}

This is its output:

USER>d ##class(dc.Demo.Test).SimpleTest()
resp=3
resp(1)="* OK The Microsoft Exchange IMAP4 service is ready. [QwBQ..AA==]"_$c(13,10)
resp(2)="TAG1 OK LOGIN completed."_$c(13,10)
resp(3)="* BYE Microsoft Exchange Server IMAP4 server signing off."_$c(13,10)_"TAG2 OK LOGOUT completed."_$c(13,10)

There are some highlights in this code:

We set the mode variable to C, which is carriage return mode. This setting is mandatory for IMAP.
The flag /TLS establishes a secure layer of communication (SSL). We must set this flag value to a valid SSL IRIS connection.
The OPEN command initiates the connection.
The special boolean variable $TEST returns 1 when a command with a timeout is successful or 0 if the timeout expires. In this example, if the OPEN command exceeds 30 seconds, the code throws an exception.
After a connection is established successfully, the command USE owns the TCP device, redirecting all READ and WRITE commands to this device.
The WRITE command issues commands to the IMAP server, and the READ command gets their output.
To finish the connection, we must use the CLOSE command.
After owning the device, all calls to READ and WRITE commands execute on the device specified in the dev variable, after using the USE dev command. To come back to the terminal and write to it again, you need to issue a USE 0 command first.

Each READ command has a limited buffer to store the server response. When the response size exceeds this limit, you need to issue another READ command to read the complete response. Of course, it’s possible to increase the buffer size, but a better approach is to be ready to deal with such a situation.

As we discussed before, IMAP requires a tag for each command. This tag is helpful to check if the code retrieved the complete response or if it needs to issue another READ command. In this case, we implement the ReadResponse method to ensure the code reads the whole message.

Implementing the %Net.FetchMailProtocol Interface for IMAP

The %Net.FetchMailProtocol abstract class abstracts email retrieval on the IRIS platform. We implement the following methods:

Connect: This establishes a connection to the IMAP server and logs in a user.
GetMailBoxStatus: This gets the size of the mailbox and how many messages are in it.
GetSizeOfMessages: This gets the size of one or all messages identified by a message number.
GetMessageUIDArray: This gets an array with one or all message UIDs in the inbox.
GetMessageUID: This gets the UID corresponding to a message number.
Fetch: This retrieves a message’s content, possibly multipart content, identified by a message number. It retrieves the message content encapsulated in a %Net.MailMessage object.
FetchFromStream: This is the same as Fetch, but gets content from an encapsulated EML message content in a %BinaryStream object, instead of calling the IMAP server.
FetchMessage: This is the same as Fetch, but returns specific message headers in ByRef variables.
FetchMessageInfo: This retrieves only message headers and the text of the message.
DeleteMessage: This adds a message to the deletion array.
RollbackDeletes: This cleans up the deletion array.
QuitAndCommit: This deletes all messages in the deletion array and disconnects from the IMAP server.
QuitAndRollback: This cleans up the deletion array and disconnects from the IMAP server.
Ping: This pings the IMAP server to keep the session alive.

First, we create a new class to implement the interface: dc.Demo.IMAP. This class inherits several properties, which we must set to establish a connection to the IMAP server.

We create a helper class as well: dc.Demo.IMAPHelper. This class parses methods for IMAP responses, gets all parts of a multipart message, and stores peripheral features, including a method to send commands and ensure the entire response is read.

The first method we implement is the Connect method. This method establishes a connection to the IMAP server using the configuration encapsulated in the class properties. It issues a login as well. This method uses the IRIS platform’s OPEN command to establish the connection to the IMAP server and the IMAP command LOGIN to authenticate to the server.

The next method we implement is GetMailBoxStatus. This method uses the SELECT command to select a mailbox and it brings some additional information as well, like how many messages are in the mailbox.

IMAP doesn’t have a ready-to-use command to get the size of all messages. Of course, it’s possible to iterate through all messages and sum their sizes. However, this strategy will probably cause slowness issues. So in this implementation, we don’t retrieve the size for all messages.

The next method is GetSizeOfMessages. This method gets the size of one or more messages in the inbox. When no message number is defined, this method throws an exception due to the same IMAP limitation we explained for the GetMailBoxStatus method. We use the IMAP command FETCH (RFC822.SIZE) to retrieve a message size by its number.

The GetMessageUIDArray method comes next, which uses the IMAP commands SELECT and UID SEARCH [ALL | ] and parses its response to get the UID array.

The next method is GetMessageUID. This method gets a UID for a defined message number and uses the same logic as the GetMessageUIDArray method.

Following this is the Fetch method. It uses the IMAP commands SELECT and FETCH BODY to retrieve message content, which is coded in MIME format. Fortunately, the IRIS platform has a reader for MIME content, the %Net.MIMEReader class. This class gets the message in a stream and returns the parsed message in a %Net.MIMEPart object.

After getting the MIME content, the method creates a %Net.MailMessage object, fills it with data from the %Net.MIMEPart object, and returns it.

The MIME content is encapsulated in a %Net.MIMEPart object that maps into a %Net.MailMessagePart object through the GetMailMessageParts method in the dc.Demo.IMAPHelper class.

The next method is FetchFromStream. This method receives a stream object with an EML message and converts it to a %Net.MailMessage object. This method does not retrieve content from the server.

Following are the FetchMessage and FetchMessageInfo methods, which are special cases of the Fetch method.

The DeleteMessage method marks a message for deletion, whereas the [RollbackDeletes}(https://github.com/jrpereirajr/iris-imap-inbound-adapter-demo/blob/c477d3b37b2ff63824ab66bebb61d18017d449db/src/dc/Demo/imap/IMAP.cls#L526) method just cleans up the array of messages marked for deletion.

Next is the QuitAndCommit method. It disconnects from the IMAP server and calls the method CommitMarkedAsDeleted for message deletion.

The method QuitAndRollback just disconnects from the IMAP server and cleans up the array of messages marked for deletion.

The last method, Ping, issues a NOOP command to keep the IMAP session alive.

Implementing an Inbound Interoperability Adapter for IMAP

The base class for email interoperability inbound in the IRIS platform is EnsLib.EMail.InboundAdapter. This inbound adaptor requires these configurations:

The email server host address
The email server port
A credential ID which stores the username and password for accessing the server
An SSL configuration

This class was extended to create a new IMAP inbound adapter class: dc.Demo.IMAPInboundAdapter.

To use this new adapter, we set which mailbox to use in the Mailbox production parameter. Its default value is INBOX.

The implementation is simple, it just overrides the MailServer property and sets its type to dc.Demo.POP3ToIMAPAdapter IMAP client. This adapter maps the POP3 flow to the IMAP one, as the base adapter class was designed for POP3 commands.

Thus, this POP3 to IMAP adapter enables us to perform all the original inbound adapter logic using IMAP commands instead of POP3 commands.

In the dc.Demo.POP3ToIMAPAdapter class, we use the IMAP client IMAPClient of type dc.Demo.IMAP as a proxy for server communication. However, as dc.Demo.POP3ToIMAPAdapter extends %Net.POP3, it must override all abstract methods in %Net.FetchMailProtocol.

Also, we had to implement new methods that the %Net.POP3 client had implemented directly: ConnectPort and FetchMessageHeaders. In the same way, we created ConnectedGet and SSLConfigurationSet methods to set and get properties that %New.POP3 also implemented directly.

Setting up a Simple Production

To make all these classes work together, we set up a simple production. Check out Creating a Production to get more information about IRIS Interoperability productions.

This production includes a business service and a business operation, which uses the IMAP inbound adapter to check for new messages. This code was inspired by the Demo.Loan.FindRateProduction interoperability sample.

In short, this production:

Uses the GetMessageUIDArray method to get all available messages in the configured mailbox
Loops over them, tracing their output, fetched by the Fetch method
Checks if each message subject matches a criterion — starting with "[IMAP test]"
Responds to the sender if the message subject matches the criteria, otherwise ignores the message
Deletes all of the messages so that it won’t analyze them again

In this example, we configure an IMAP server from Yahoo Mail imap.mail.yahoo.com, on port 993. We also use the default IRIS SSL configuration “ISC FeatureTacker.SSL.Config”.

Next, we configure a credential called imap-test containing a username and password, as follows:

As the image below shows, the production starts and keeps querying the IMAP server for new messages. When there are new messages, the inbound adapter grabs their information, like the header and subject, and lets production take further action based on this information.

In this example, the production checks if the message subject starts with "[IMAP test]" and sends back a message to the sender.

When a message doesn’t match the criteria, production just ignores it.

Conclusion

In this article, we discussed an IMAP client implementation. First, we explored some essential background on IMAP and its main commands. Then, we detailed the implementation, covering the client itself and how to connect it to the IRIS platform. We also presented an extension to the default interoperability adapter to use IMAP, and a simple production example.

Now that you know more about IMAP and its settings and you know how to connect it to IRIS, you can set up email capabilities in your applications. To learn more about the IMAP topics we discussed here, explore the resources below.

Resources

Implementing an IMAP Client in InterSystems IRIS - part I

jrpereirajr — Sat, 12 Feb 2022 15:29:38 +0000

This article explains how to use the InterSystems IRIS platform to write a basic IMAP client. First, we present an overview of IMAP, then we discuss the main IMAP commands and client implementation. Finally, we offer a simple use of this IMAP client on the IRIS interoperability application.

Note that this article isn’t a detailed IMAP explanation. For more detailed information, please check the references of this article.

IMAP Overview

The Internet Message Access Protocol (IMAP) lets users retrieve emails. Mark Crispin proposed IMAP in the 1980s, and since then, the protocol has been published and revised in RFC 3501. At the time of writing this article, its current version is IMAP4rev1.

It’s important to note that this protocol is designed for retrieving only. You must use another protocol, the Simple Mail Transfer Protocol (SMTP), if you need to send emails. There’s also an older protocol for email retrieval that’s as popular as IMAP, called Post Office Protocol version 3 (POP3).

Trying Basic IMAP Commands

Like POP3, IMAP is a plaintext protocol. You can easily try some of its commands by yourself, using a TCP client like Telnet or OpenSSL.

First, you need your IMAP host and email server port. For instance, at the time of writing this article, this is the host and port to connect to the Microsoft Outlook service:

outlook.office365.com:993

Let’s connect to this server using our TCP client. Enter the command below and hit ENTER. Note the use of the -crlf flag. This flag is mandatory for IMAP as carriage return and line feed (CRLF) is its line terminator.

$ openssl s_client -connect outlook.office365.com:993 -crlf -quiet

After you hit enter, the IMAP server shows you some information and stays waiting for input.

depth=2 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert ...
* OK The Microsoft Exchange IMAP4 service is ready. [...]

Now we can perform our first IMAP command: CAPACITY. As its name suggests, this command presents features that the server can provide. Note that the first line is the IMAP command itself, and the others lines are its output - this is the same for all other IMAP commands presented here.

TAG1 CAPABILITY
* CAPABILITY IMAP4 IMAP4rev1 AUTH=PLAIN AUTH=XOAUTH2 SASL-IR UIDPLUS ID UNSELECT CHILDREN IDLE NAMESPACE LITERAL+
TAG1 OK CAPABILITY completed.

Note the token before the CAPABILITY command: TAG1. A tag must precede every IMAP command and is any string you want. They identify what command instance the server is responding to. Generally, a simple counter with a prefix fits all your needs.

The IMAP server finishes the response with the tag you used to issue the command and a flag indicating the command’s status, followed by miscellaneous information related to the command and status. The statuses allowed are: OK (success), NO (failure), and BAD (wrong command).

The command LOGIN expects user and password as arguments. First, let’s check how the server responds if someone gives an invalid username and password:

TAG2 LOGIN user@outlook.com wrong-password
TAG2 NO LOGIN failed.

Note the NO status after the command tag.

Now, let’s proceed to a valid login:

TAG3 LOGIN user@outlook.com password
TAG3 OK LOGIN completed.

Now we’re logged in. To finish our session, we use the LOGOUT command:

TAG4 LOGOUT
* BYE Microsoft Exchange Server IMAP4 server signing off.
TAG4 OK LOGOUT completed.

We need other commands to implement our IMAP client, so we need to reconnect and log into our IMAP server again.

Let’s get some information about the inbox next.

First, we must tell the server which mailbox we want to access, INBOX in this case.

TAG5 SELECT "INBOX"
* 14 EXISTS
* 0 RECENT
...
TAG5 OK [READ-WRITE] SELECT completed.

Now we are ready to access messages in our inbox, but we need a way to refer to them first. There are two ways to do that: by message number or by a unique identifier (UID).

Message numbers are just counters, starting from 1 and going up to the number of messages within the inbox. When you delete a message, all following messages have their numbers decremented by 1. You can think of this as an array that has one of its elements removed.

UIDs, on the other hand, keep their value whatever happens to the other messages.

You can get UIDs using the UID SEARCH command. This command accepts a message number if you'd like to get a specific UID or the ALL parameter to get all UIDs in the directory.

In the example below, we search for UID for message number 1, which is 483.

TAG6 UID SEARCH 1
* SEARCH 483
TAG6 OK SEARCH completed.

Now let’s retrieve message information, like headers, body, and attachments. We use the FETCH command for this. This command has plenty of parameters, which you can explore in detail on RFC 3501. In this article, we only address parameters related to the needs of the IMAP client implementation.

The first information we need for this demonstration is the message size. We can get this information using the RFC822.SIZE parameter:

TAG7 FETCH 1 RFC822.SIZE
* 1 FETCH (RFC822.SIZE 70988)
TAG7 OK FETCH completed.

This indicates that the size of message number 1 is 70,988 bytes.

Note that it’s also possible to use UIDs rather than message numbers to fetch message information:

TAG8 UID FETCH 483 RFC822.SIZE
* 1 FETCH (RFC822.SIZE 70988 UID 483)
TAG8 OK FETCH completed.

You can also get the basic From, To, Date, and Subject message headers:

TAG9 FETCH 1 (FLAGS BODY[HEADER.FIELDS (FROM TO DATE SUBJECT)])
* 1 FETCH (FLAGS (\Seen) BODY[HEADER.FIELDS (FROM TO DATE SUBJECT)] {157}
Date: Thu, 22 Apr 2021 15:49:05 +0000
From: Another User <anotheruser@outlook.com>
To: user@outlook.com
Subject: Greetings from Another User!
 FLAGS (\Seen))
TAG9 OK FETCH completed.

Now let’s retrieve the message body. We can retrieve all the body content using this command:

TAG10 FETCH 1 BODY[]
* 1 FETCH (BODY[] {9599}
...
MIME-Version: 1.0
--00000000000041bd3405c3403048
Content-Type: multipart/alternative; boundary="00000000000041bd3205c3403046"
--00000000000041bd3205c3403046
Content-Type: text/plain; charset="UTF-8"
...
--00000000000041bd3405c3403048
Content-Type: image/png; name="download.png"
Content-Disposition: attachment; filename="download.png"
...
TAG10 OK FETCH completed.

We can see in this code that some blocks are delimited by hex numbers between -- --, called parts. A message with parts is called a multipart message. It’s possible to get such parts directly by passing the part index to the command.

In order to delete messages, the protocol has the commands STORE and EXPUNGE, which mark a message as deleted and commit such an operation.

TAG11 STORE 1 +FLAGS (\Deleted)
* 0 EXISTS
* 0 RECENT
TAG11 OK [READ-WRITE] SELECT completed; now in selected state
TAG12 EXPUNGE
EXPUNGE: TAG12 OK EXPUNGE completed

The last command is simple: NOOP. This command does nothing but is used to implement a keep-alive strategy. By default, the IMAP session closes after 30 minutes without commands. So, issuing the NOOP command keeps the connection active.

TAG17 NOOP
TAG17 OK NOOP completed.

And this finishes our IMAP overview. If you’d like more information, there are a lot of good articles on the Web (I've selected some in resources section), and of course always remember the RFC 3501.

Resources

Next part, you'll use those commands within IRIS and see them in action!

See you! :)