Exchange 2013 Shadow Redundancy

Shadow redundancy is a feature introduced from exchange 2010 to provide redundancy for messages for the entire time they're in transit. The solution involves a technique similar to the transport dumpster. With shadow redundancy, the deletion of a message from the transport databases is delayed until the transport server verifies that all of the next hops for that message have completed delivery. If any of the next hops fail before reporting back successful delivery, the message is resubmitted for delivery to that next hop. 

Improvement in Exchange 2013

The major improvement to shadow redundancy in Microsoft Exchange Server 2013 is that the transport server now makes a redundant copy of any messages it receives before it acknowledges successfully receiving the message back to the sending server. The sending server's support or lack of support for shadow redundancy doesn't matter. This helps to ensure that all messages in the Exchange 2013 transport pipeline are made redundant while they're in transit.

Shadow redundancy components

Transport high availability boundary	DAG for DAG environment and AD site for non DAG environment
Primary message	The original message submitted to transport for delivery
Shadow message	The redundant copy of the message that the shadow server retains until it confirms the primary message was successfully processed by the primary server.
Primary server	The transport server that's currently processing the primary message.
Shadow server	The transport server that holds the shadow message for the primary server. A transport server may be the primary server for some messages and the shadow server for other messages simultaneously.
Shadow queue	The delivery queue where the shadow server stores shadow messages. For messages with multiple recipients, each next hop for the primary message requires separate shadow queues.
Discard status	The information a transport server maintains for shadow messages that indicate the primary message has been successfully processed.
Discard notification	The response a shadow server receives from a primary server indicating a shadow message is ready to be discarded.
Safety Net	The Exchange 2013 improved version of the transport dumpster. Messages that are successfully processed or delivered to a mailbox recipient by the Transport service on a Mailbox server are moved into Safety Net. For more information, see Safety Net.
Shadow Redundancy Manager	The transport component that manages shadow redundancy.
Heartbeat	The process that allows primary servers and shadow servers to verify the availability of each other.

  

How shadow message created

When receiving emails from outside the boundary

1. An SMTP server transmits a message to the Transport service on a Mailbox server. The Mailbox server is the primary server, and the message is the primary message.
2. While the original SMTP session with the SMTP server is still active, the Transport service on primary server opens a new, simultaneous SMTP session with the Transport service on a different Mailbox server in the organization to create a redundant copy of the message.
• If the primary server is a member of a DAG, the primary server connects to a different Mailbox server in the same DAG. If the DAG spans multiple Active Directory sites, a Mailbox server in a different Active Directory site is preferred by default. This setting is controlled by the ShadowMessagePreference parameter on the Set-TransportService cmdlet. The default value is PreferRemote, but you can change it to RemoteOnly or LocalOnly.
• If the primary server isn't a member of a DAG, the primary server connects to a different Mailbox server in the same Active Directory Site, regardless of the value of the ShadowMessagePreference parameter.
3. The primary server transmits a copy of the message to the Transport service on other Mailbox server, and Transport service on the other Mailbox server acknowledges that the copy of the message was created successfully. The copy of the message is the shadow message, and the Mailbox server that holds it is the shadow server for the primary server. The message exists in a shadow queue on the shadow server.
4. After the primary server receives acknowledgement from the shadow server, the primary server acknowledges the receipt of the primary message to the original SMTP server in the original SMTP session, and the SMTP session is closed.

When sending emails to outside the boundary

When an Exchange 2013 transport server transmits a message outside the transport high availability boundary, and the SMTP server on the other side acknowledges successful receipt of the message, the transport server moves the message into Safety Net.

How shadow redundancy works

1. After successful creation of a shadow message, the primary and shadow server need to stay in contact to track the progress of the message.

2. When the primary server successfully transmits the message to the next hop, and the next hop acknowledge the receipt, the primary server updates the status of “Discard Status” as delivery complete.

3. Shadow server opens SMTP session (which is nothing but heartbeat) with primary server to determine the discard status of the shadow message. 

4. Primary server responds with the discard status and then shadow messages will be removed or moved to safety Net.

5. If the shadow server can’t open SMTP session with primary server, the shadow server promotes itself as a primary server, promotes shadow message to primary message, and submits to next hop.

Shadow redundancy manager

Shadow Redundancy Manager is the core component of an Exchange 2013 transport server that's responsible for managing shadow redundancy.

• The shadow server for each primary message being processed.
• The discard status to be sent to shadow servers.

• Maintaining the list of primary servers for each shadow message.
• Comparing the original database ID and the current database ID of the queue database where the primary copy of the message is stored.
• Checking the availability of each primary server for which a shadow message is queued.
• Processing discard notifications from primary servers.
• Removing the shadow messages from the shadow queues after all expected discard notifications are received.
• Deciding when the shadow server should take ownership of shadow messages, becoming a primary server.

Message processing after an outage

1. Transport server comes back with new database - When the transport server becomes unavailable, each server that has shadow messages queued for that server will assume ownership of those messages and resubmit them. The messages then get delivered to their destinations.

2. Transport server comes back with old database - After the server comes back online, it will deliver the messages in its queues, which have already been delivered by the servers that hold shadow copies of messages. This will result in duplicate delivery of these messages. Exchange mailbox users won't see duplicate messages due to duplicate message detection. However, recipients on non-Exchange messaging systems may receive duplicate copies of messages.

Important

Shadow redundancy can't protect messages in transit, during and simultaneous failure of two or more transport servers, in a single server topology.