Exchange server 2016 mail flow and transport pipeline

Exchange server 2016 mail flow and transport pipeline

Transport pipeline

Transport pipeline is a collection of services, connections, components, and queues that work together to route all messages to the categorizer in the Transport service on an Exchange 2016 Mailbox server inside the organization.

Front End Transport service on Mailbox servers   This service acts as a stateless proxy for all inbound and (optionally) outbound external SMTP traffic for the Exchange 2016 organization. The Front End Transport service doesn't inspect message content, doesn't communicate with the Mailbox Transport service, and doesn't queue any messages locally.

Transport service on Mailbox servers   This service is virtually identical to the Hub Transport server role in Exchange Server 2010. The Transport service handles all SMTP mail flow for the organization, performs message categorization, and performs message content inspection. Unlike Exchange 2010, the Transport service never communicates directly with mailbox databases. That task is now handled by the Mailbox Transport service. The Transport service routes messages among the Mailbox Transport service, the Transport service, the Front End Transport service, and (depending on your configuration) the Transport service on Edge Transport servers.

Mailbox Transport service on Mailbox servers   This service consists of two separate services:

 

·         Mailbox Transport Submission service   This service connects to the local mailbox database using an Exchange remote procedure call (RPC) to retrieve messages. The service submits the messages over SMTP to the Transport service on the local Mailbox server or on other Mailbox servers. The Mailbox Transport Submission service has access to the same routing topology information as the Transport service.

·         Mailbox Transport Delivery service   This service receives SMTP messages from the Transport service on the local Mailbox server or on other Mailbox servers and connects to the local mailbox database using RPC to deliver the messages.

 

The Mailbox Transport service doesn't communicate with the Front End Transport service, the Mailbox Transport service, or mailbox databases on other Mailbox servers. It also doesn't queue any messages locally.

Transport service on Edge Transport servers   This service is very similar to the Transport service on Mailbox servers. If you have an Edge Transport server installed in the perimeter network, all mail coming from the Internet or going to the Internet flows through the Transport service Edge Transport server. This service is described in more detail later in this topic.

Mail flow from external senders

1.     A message from outside the organization enters the transport pipeline through the default Receive connector named "Default Frontend <Mailbox server name>" in the Front End Transport service.

2.     The message is sent to the Transport service on the local Mailbox server or on a different Mailbox server. The Transport service listens for messages on the default Receive connector named "Default <Mailbox server name>".

3.     The message is sent from the Transport service to the Mailbox Transport Delivery service on the local Mailbox server or on a different Mailbox server.

4.     The Mailbox Transport Delivery service uses RPC to deliver the message to the local mailbox database.

 

Mail flow from internal senders

SMTP messages from inside the organization enter the transport pipeline through the Transport service on a Mailbox server in one of the following ways:

 

·         Through a Receive connector.

·         From the Pickup directory or the Replay directory.

·         From the Mailbox Transport Submission service.

·         Through agent submission.

 

Outbound mail flow

 

1.     The Mailbox Transport Submission service uses RPC to retrieve the outbound message from the local mailbox database.

2.     The Mailbox Transport Submission service uses SMTP to send the message to the Transport service on the local Mailbox server or on a different Mailbox server.

3.     In the Transport service, the default Receive connector named "Default <Mailbox server name>" accepts the message.

4.     What happens next depends on the configuration of the Send connector:

 

o    Default   The Transport service uses the Send connector you created to send the message to the Internet.

o    Outbound proxy   The Transport service uses the Send connector you created to send the message to the Front End Transport service on the local Mailbox server or on a remote Mailbox server. In the Front End Transport service, the default Receive connector named "Outbound Proxy Frontend <Mailbox server name>" accepts the message. The Front End Transport services sends the message to the Internet.

 

Internal mail flow

 

1.     The Mailbox Transport Submission service uses RPC to retrieve the outbound message from the local mailbox database.

2.     The Mailbox Transport Submission service uses SMTP to send the message to the Transport service on the local Mailbox server.

3.     Transport service determines the mail destination and route it to the transport service on the mailbox server where the recipient mailbox is residing.

4.     Mailbox transport delivery service receives the message and connect to user’s mailbox database to deliver.

 

Office 365 Directory Synchronization

Office 365 Directory Synchronization

What is Dir Sync:

If you are already using Active Directory in your on premise environment, you probably have invested a lot of time creating user accounts, populating their attributes and adding them to the appropriate groups. Directory Synchronization takes all that information, users, groups, contacts, email addressesess, phone numbers, names, etc and synchronizes it from your Active Directory to Office 365. The synchronization is ongoing which allows you to continue to manage users, groups and contacts from your local Active Directory. The synchronization is 1 way (from Active Directory to Office 365) and at this time, once enabled, cannot be disabled.

Installation and Configuration

The installation of Directory Sync is very straight forward. It’s pretty much a next, next, next install, with you only having to choose the install path. The install process installs several components behind the scenes:

·         Microsoft Online Directory Synchronization, which is really just Identity Lifecycle Manager 2007

·         Microsoft SQL Server 2008 R2

·         Microsoft Online Services Sign-In Assistant

When the installation is complete, you’ll need to launch the configuration wizard to configure the synchronization. Again not a lot of options, you must specify your Office 365 Credentials, your Active Directory Credentials and choose whether or not you’ll be using Rich Coexistence. The Rich Coexistence option is only available if you have Exchange 2010. Enabling Rich Coexistence also allows the Microsoft Online Directory Sync tool write Access to your local Active Directory.

During the installation, an account named MSOL_AD_Sync is created in the Users OU in the Active Directory. This is the account that Dir Sync uses to interact with the Active Directory. The Enterprise Admin Credentials specified during the configuration are not used on an ongoing basis. If you selected the Rich Coexistence Option, there is a group created, MS_AD_Sync_RichCoexistence.

The MSOL_AD_Sync User account has the following permissions granted:

·         Replicate Directory Changes

·         Replication Synchronization

·         Replicating Directory Changes All

MS_AD_Sync_RichCoexistence has the following permissions granted:

·         Write msExchangeSafeRecipientsHash on User objects

·         Write msExchangeBlockedSendersHash on Users objects

·         Write msExchUCVoiceMailSettings on Users objects

·         Write msExchArchiveStatus on Users objects

·         Write msExchSafeSendersHash on Users objects

·         Write ProxyAddresses on User,Contact, and Group objects

How we can view and modify the Dir Sync configuration

The key thing to realize is that Directory Synchronization is really just Microsoft Identity Lifecycle Manager 2007. You can launch the configuration interface by running:

C:\Program Files\Microsoft Online Directory Sync\SYNCBUS\UIShell\miisclient.exe

Operations Console

Upon launching the MIIS Client you will see the Operations Area. The Microsoft Online Directory Synchronization uses 2 Management agents. The SourceAD management agent reads the information from the source Active Directory. The TargetWebService management agent writes the information to the Microsoft Online Directory. The operations area will allow you see a log of what the Management Agents have done. In this example we’ll take a look at the initial synchronization which is located at the bottom of the history and has a Profile Name of Full Import Full Sync. Once selected, in the lower left pane of the window you’ll see Synchronization Statistics which allow you to see what objects have been synchronized as well as which objects have been filtered out and therefore won’t be synchronized to the Metaverse and Microsoft Online.

Source AD Management Agent

Adds

In the image above there are 193 Adds. This appears to be all the objects read from the Active Directory. Not all of these objects will be replicated to Microsoft Online as filters (explained later) will prevent them from synchronizing. Clicking on the Adds will allow you to see what objects have been read as shown below.

Projections

The Projections show which objects have been written to the Metaverse. In the example above you’ll see that of the 193 objects read, only 14 are written to the Metaverse. Just because an object is written to the Metaverse does not mean it will be written to Microsoft Online, as the TargetWebService also has filter rules when synchronizing from the Metaverse to Microsoft Online. Again you can click on Projections to see what objects have been written to the Metaverse.

Filtered Disconnectors

Filtered Disconnectors show the objects what are not synchronized due to the objects matching a defined filter rule.

TargetWebService

The TargetWebService is the Agent that reads information from the Metaverse and writes information to Microsoft Online. You’ll notice in the Agent Operations log that the TargetWebService performs 2 operations on each run, an import and an export.

Clicking on the Profile name Export and then clicking Adds, appears to show what has actually been synchronized to Microsoft Online. In this case 7 objects were synchronized to Microsoft Online. This screenshot is from the same session as earlier, in which 14 objects were written to the Metaverse, but as you see above, only 7 were synchronized to Microsoft Online. The reason is that the objects did not have a display Name specified. When you click Adds, you’ll notice that the objects GUIDs are listed instead of names, but you can click on the GUID to display the objects properties

Metaverse Search

The Metaverse Search area will allow you see the objects, and their attributes, that have been synchronized to the Metaverse. To see the objects, simply click the Search button.

The Search Results will show all objects in the Metaverse. In the image above you’ll notice that 14 objects are in the Metaverese and displayName is only column displayed. You can click on the Column settings link to display additional attributes. You can also limit the search by changing the Scope by Object Type drop down list or clicking the Add Clause action to filter by attribute.

The Metaverse search area could be useful if you are trying to determine why certain objects aren’t synchronizing to Microsoft Online.

Joiner

The Joiner will allow you to see the objects which have not been synchronized. Selecting the Management Agent SourceAD and Disconnector Type Disconnectors, and then clicking search will display all the disconnectors.

Controlling the Frequency of Directory Synchronization

By default the Directory Synchronization runs every 3 hours. If you want to increase or decrease the frequency you will need to modify the following file:

C:\program files\Microsoft Online Directory Sync\Microsoft.Online.DirSync.Scheduler.exe.Config

Just open with notepad and you’ll see a line that looks like:

<add key=”SyncTimeInterval” value=”3:0:0″ />

By default the SyncTimeInterval is set to 3 hours. If you wanted it to run every 30 minutes you’d change the value to “0:30:0”. We wouldn’t recommend going any lower than 15-30 minutes.

Port

We need to open 443 towards internet for Dir Sync to work.

What is outlook auto complete or nick name file (.NK2)

What is outlook auto complete or nick name file (.NK2)

Every time that you type an email address or name in the message window of MS-Outlook, it automatically offer you a list of users and email address that you can choose. This feature is known as 'AutoComplete' and Outlook automatically build this emails list according to user activity and save it into a file with .NK2 extension.

The location of the NK2 file of Outlook

The location of the NK2 AutoCompelete file created by Outlook might be different from one computer to another, depending on the operating system and the version of Outlook. 

Here's the rules for finding the location of your NK2 file:

For Outlook 2003/2007 with Windows 2000, Windows XP, or Windows Server 2003: 

The location of the nk2 file is C:\Documents and Settings\[User Profile]\Application Data\Microsoft\Outlook 
The name of the NK2 file is identical to the Outlook profile name, with .nk2 extension.

For Outlook 2003/2007 with Windows Vista, Windows 7, or Windows server 2008: 

The location of the nk2 file is C:\Users\[User Profile]\AppData\Roaming\Microsoft\Outlook 
The name of the NK2 file is identical to the Outlook profile name, with .nk2 extension.

For Outlook 2010/2013 with Windows Vista, Windows 7, or Windows server 2008: 

The nk2 file is located in C:\Users\[User Profile]\AppData\Local\Microsoft\Outlook\RoamCache 
The name of the NK2 file is in the following format: Stream_Autocomplete_X_AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA.dat The X is the file index (usually 0) and AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA is a random 16-byte key in hexadecimal format.

For Outlook 2010/2013 with Windows XP: 

The nk2 file is located in C:\Documents and Settings\[User Profile]\Local Settings\Application Data\Microsoft\Outlook\RoamCache 
The name of the NK2 file is in the following format: Stream_Autocomplete_X_AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA.dat The X is the file index (usually 0) and AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA is a random 16-byte key in hexadecimal format.

A common question from users

One of my email contacts has a new email address as their company changed their domain name. I have deleted and re-entered her correct email address about half a dozen times in the address book and yet my PC still thinks I am sending her emails to her old domain.com instead of her new domain.com. When I entered her name it defaults to the "olddomain.com" address even though it is nowhere to be found on my PC. I have to select her "newdomain.com" address from the address book for it to work. Has anyone experienced where Outlook is keeping this old email address info? How can I can purge it?

This is an all too common experience for many users.

Use the arrow key to select the incorrect address and press the Delete key.

You can clear the entire autocomplete file (*.NK2): in older versions you need to delete the NK2 file from your hard drive.

Outlook 2010 And Newer

Outlook 2010 and 2013 does not use the NK2 file; it stores the autocomplete cache in the mailbox or data file and caches the addresses in an autocomplete stream at C:\Users\username\AppData\Local\Microsoft\Outlook\RoamCache. You can type or paste %localappdata%\Microsoft\Outlook\RoamCache in the address bar of Windows Explorer then press Enter to open the folder location where the cache is stored.

Outlook 2010 also creates a contact for people you send mail to, as insurance if the autocomplete cache becomes corrupt. Look for these contacts in the Suggested Contacts folder. This unpopular feature was removed from Outlook 2013.

Clear the autocomplete cache using the File, Options, Mail, Empty Auto-complete list button.

You can also use the /cleanautocompletecache switch to clear the cache.

Nickname Cache Is Empty :

If the users are losing addresses of people they recently replied to (but not older addresses), it's because Outlook crashes or does not close properly and the changes are not written to the NK2 file. The existing *.NK2 shouldn't be replaced by a clean *.NK2 if this happens, but new addresses you've sent new mail to or replied to won't be saved to the file. Less used addresses are replaced by new addresses when the NK2 contains 1000 addresses in Outlook 2003 or 2000 address if using Outlook 2007.

If all addresses disappear, it means something is corrupt and Outlook is replacing the NK2 with a new, clean one.

Import Addresses From NK2:

When you upgrade from Outlook 2007 and earlier in-place and use the same profile, Outlook 2010 or 2013 should import the NK2 you used with your previous Outlook version automatically on the first run. If Outlook didn't import the NK2 automatically, or you are using a new computer and didn't think to move the NK2 file, or you made a new profile, you can import the NK2 after the first run by using the ImportNK2 switch.


To import .nk2 files into Outlook 2010 or Outlook 2013, follow these steps:

1. Make sure that the .nk2 file is in the following folder:
   %appdata%\Microsoft\Outlook
   The .nk2 file must have the same name as your current Outlook profile. (By default, the profile name is "Outlook.")
2. Press Windows Key + R to open the Run dialog.
3. In the Open box, type (or paste) outlook.exe /importnk2, and click OK to import the NK2 into your profile.

The .nk2 file is renamed with a .old file name extension on the first start of Outlook 2010. If you need to re-import the .nk2 file, remove the .old file name extension.

Released: March 2016 Quarterly Exchange Updates

Microsoft released the quarterly updates for Exchange Server are now available on the Microsoft Download Center. Exchange Server 2016 receives its first Cumulative Update, and Exchange Server 2013 Cumulative Update 12 is also released. Exchange Server 2007 and Exchange Server 2010 Update Rollups provide an updated OWA S/MIME control signed with a SHA-2 certificate. More information and highlights of all these releases can be found below. 

Updated OWA S/MIME control

All of the packages released today include an update to the OWA S/MIME control. The control itself has not changed, but has now been signed with a SHA-2 compliant certificate. All of the updates released will install the updated control onto the Exchange Server. Users who have installed the control into their browser will need to re-install this onto devices where the previous version was installed. Installing the control is straight forward and can be done quickly using OWA Options, Exchange Control Panel or Exchange Admin Center depending upon the release of Exchange you are using. 

New distribution package for Exchange Server 2016 updates

With the introduction of Cumulative Updates for Exchange Server 2016, we are making a change to the update package type for this product version. Previous versions of Exchange used self-extracting packages to deliver service packs and cumulative updates. We have heard requests to release these updates as .ISO’s. With the capability to mount .ISO’s directly in Windows Server 2012 and later, we think it makes sense to ship Cumulative Updates as .ISO’s. At this time, we are not planning to do this for Exchange Server 2013 Cumulative Updates but could be persuaded to do so if enough people ask for it. One down side to this approach is that the package is much larger. However, copying a single .ISO vs. the ever growing number of files and folders over the network is much more efficient and faster. We hope you like this change. 

Change to Mailbox Anchoring for Remote PowerShell

We heard your feedback on the changes to load balancing Remote PowerShell introduced into Exchange Server 2013 and 2016. As announced by Ross here, we have reverted this behavior in the Cumulative Updates being released today. 

Additional languages for Outlook on the Web

Exchange Server 2016 Cumulative Update 1 adds support for 17 additional languages in Outlook on the Web. These languages will appear automatically in the language selection drop down after a server is updated to Cumulative Update 1. 

.Net 4.6.1 Support

We know that many of you have been asking about .Net 4.6.1 and Exchange. Rest assured we are working closely with the .Net Framework team to resolve issues preventing us from supporting .Net 4.6.1 with Exchange Server. While we are not there yet, we hope to be very soon. Support for .Net 4.6.1 is planned for future Cumulative Updates for Exchange Server 2013 and 2016. 

Slow installations on Windows Server 2012 R2

For customers who are running Exchange on Windows Server 2012 R2, we want to make certain you are aware of a condition which can substantially increase the amount of time it takes to install Exchange Updates on this OS. Working with the .Net team, we have discovered that systems which have applied Windows Update KB3097966 can take 50% more time to install Exchange. The .Net team is working on a resolution to this and will include a fix in a future product update. In the meantime, customers who have deployed this Windows update can take a one-time action on their server before installing Exchange or a Cumulative Update to bring installation time back to normal. This procedure needs to be done once on every Exchange server running Windows Server 2012 R2. The command to execute is: 

“%windir%\Microsoft.NET\Framework64\v4.0.30319\ngen.exe update”

Errors and warnings encountered running this command can be safely ignored provided the final exit status code of 0 is reported in the output. 

Support for Standalone Hybrid Configuration Wizard in Exchange Server 2010

Customers using Exchange Server 2010 in Hybrid mode with Office 365 will notice a new link in the EMC to use the Updated Standalone Hybrid Configuration Wizard. We encourage all customers to use this updated version of the Hybrid Configuration Wizard. 

Release Details

KB articles which contain greater depth on what each release includes are available as follows: 

• Exchange Server 2016 Cumulative Update 1 (KB3134844), Download, UM Lang Packs
• Exchange Server 2013 Cumulative Update 12 (KB3108023), Download, UM Lang Packs
• Exchange Server 2010 Service Pack 3 Update Rollup 13 (KB3141339), Download
• Exchange Server 2007 Service Pack 3 Update Rollup 19 (KB3141352), Download

How to migrate exchange 2010 to office 365 - Lab simulation

To simulate exchange 2010 to O365, i have subscribed with Azure and configured virtual machines. We will get below features from Azure virtual machines

1. Pre assigned public IP
2. We can open required ports
3. We can access high speed internet from inside the virtual machines
4. We can configure virtual machine very quickly

To know further information about Azure virtual machine please click here

My domain name - LYNCITMAIL.IN

I have simulated the migration and created a document. Please download the document from here.

Note:- 

I did not included screenshot for each steps. Instead i have given you some important screenshots which needs some attention.

I have enabled password synchronization during Dir Sync configuration instead of ADFS.  

What happens to my data after my organization’s Office 365 subscription ends?

What happens to my data after my organization’s Office 365 subscription ends? The most common answer circulated in the community refers to a grace period of 30 days, during which you can still retrieve your data.

The answer’s not wrong, but here’s some more detail about the tenant lifecycle after an Office 365 subscription is cancelled, as it relates to the organization’s data.

During the first 30 days after an Office 365 subscription ends, the Office 365 tenant account is in this grace period, known as expired state. During this period, users can still access data. If the subscription ended unintentionally, a rare event I’d argue given the many alerts you get to prevent termination of subscription due to issues such as non-payment, this is a good time to set things right.

After 30 days, the tenant account enters disabled state for 90 days. During this period, users no longer have access to data. The admin can still log in, backup data if required, or reactivate the subscription. At the end of the disabled state, which is 120 days after your subscription has expired, the account enters the de-provisioning state. This is when the data – from user accounts to email data and documents, is deleted permanently.

There are a few compliance-related questions arising out of end of subscription.

1. 1. How quickly will you delete data after my organization’s Office 365 service ends?
   Some time after 120 days. The jobs that delete data do so based on service load. You can expect data to be permanently deleted in a reasonable timeframe after the 120 days have elapsed.
2. 2. How can I ensure my organization’s Office 365 data is deleted quickly after service ends?
   Many security and compliance-minded organizations want to ensure there’s no residual data in a cloud service after they end service. Office 365 customers can request expedited deprovisioning by calling Support. Expedited deprovisioning ensures your data is deleted within 3 days.
3. 3. Is data immutability maintained after service ends? (In other words, are mailboxes placed on In-Place Hold or Litigation Hold retained after service ends?)
   By far one of the most frequently asked questions. Data immutability refers to the ability to preserve data – in essence, protecting it from destruction and tampering. See links to additional resources on Immutability, In-Place Hold and Litigation Hold below.
   
   No. Microsoft’s responsibility as a service provider ends after your service ends, which is when you stop being a customer/subscriber of the service. As noted above, data is permanently deleted when your tenant account enters the deprovisioning state, within a reasonable time after 120 days of end of subscription, or within 3 days if you request expedited deprovisioning. Mailboxes placed on In-Place Hold or Litigation Hold, including inactive mailboxes, are also deleted as part of deprovisioning.

How Lync/SfB clients authenticates to Lync/SfB Server - Simple explanation

How Lync/SfB clients authenticates to Lync/SfB Server

Internal users

When the user is inside the corporate network and signs in to Lync Server from a domain joined computer using their corporate Active Directory user account, Lync/SfB clients uses the following three protocols to authenticate the user when signing in to Lync/SfB Server:

o   Kerberos v5

o   TLS-DSK

o   NTLM v2

1. If the user has no certificate, Lync/SfB clients attempts to sign-in the user to Front-End Server using Kerberos (SIP traffic).

2. The Front-End Server rejects the authentication request, and redirects the Lync/SfB client to the Web Services (https://lync.contoso.com/CertProv/CertProvisioningService.svc) to request a certificate (SIP traffic)

3. Lync/SfB clients authenticates the user to Web Services using NTLM (HTTPS traffic).

4. Once authenticated, Lync/SfB clients requests a certificate for the user (HTTPS traffic). This client certificate is then stored in the user’s Personal certificate store.

5. Lync/SfB client re-authenticates the user to the Front End Server using TLS-DSK (SIP traffic).

6. For all subsequent sign-in requests, Lync/SfB client authenticates to Server using the TLS-DSK protocol with the user’s certificate instead of using Kerberos or NTLM

Note - This user certificate is valid for a period of 180 days, and is automatically renewed one month prior to expiration regardless of whether the user is connected internally or externally.

External users

For remote users connecting over the SIP channel via the Edge Server, Lync/SfB client can only use the following two authentication protocols:

o   TLS-DSK

o   NTLM v.2

1. If the user does not have a certificate, Lync/SfB client attempts to sign-in the user to Server using NTLM through the Edge Server (SIP traffic).

2. The Edge Server rejects the authentication request, and redirects the Lync/SfB client to the Web Services (https://lyncexternal.contoso.com/CertProv/CertProvisioningService.svc) through the reverse proxy to request a certificate (SIP traffic).

3. Lync/SfB client authenticates the user to Web Services using NTLM v2 (HTTPS traffic).

4. After authentication, Lync/SfB client requests a certificate for the user (HTTPS traffic). This client certificate is then stored in the user’s Personal certificate store.

5. Lync/SfB client re-authenticates the user through the Edge Server using TLS-DSK (SIP traffic).