Exchange 2013 Cumulative Update 11

Ex2013 LogoThe Exchange Team released Cumulative Update 11 for Exchange Server 2013 (KB3099522). This update raises Exchange 2013 version number to 15.0.1156.6.

  • KB 3120594 Appointment on the Outlook calendar isn’t updated to a meeting when attendees are added
  • KB 3108345 “The app couldn’t be downloaded” error occurs when you try to install an application from the Intranet in Exchange Server 2013
  • KB 3108011 Error message occurs in Outlook after you change a single instance of a recurring meeting by using an iOS device
  • KB 3107781 Exchange ActiveSync device doesn’t keep messages for 30 days as configured
  • KB 3107379 Noderunner.exe consumes excessive CPU resources by parsing an attached document in Exchange Server 2013
  • KB 3107337 Mailbox migration from Exchange Server 2007 to Exchange Server 2013 is very slow
  • KB 3107291 Exception occurs when you run the Invoke-MonitoringProbe cmdlets to set probes for IMAP and POP3 in Exchange Server 2013
  • KB 3107205 “Custom error module does not recognize this error” error when OWA web parts fail to load
  • KB 3107174 Pages that use the People pop-up URL don’t load in Chrome when you access OWA or the Exchange Server Administration Center
  • KB 3106613 Outlook Web App shows partial contacts in an Exchange Server 2013 environment
  • KB 3106475 POP3 and IMAP4 are not supported to use TLS protocol 1.1 or 1.2 in Exchange Server 2013
  • KB 3106421 Very long URLs in an email message do not open in OWA in Internet Explorer
  • KB 3105760 Exchange Server 2016 mailbox server can be added to an Exchange Server 2013 DAG
  • KB 3105690 Outlook clients that use MAPI over HTTP to connect to Microsoft Exchange Server 2013 mailboxes are intermittently disconnected
  • KB 3105685 The lsass.exe process leaks an amount of handles in Exchange Server 2013
  • KB 3105654 Cannot edit Inbox rules in Outlook Web App by using Chrome
  • KB 3105625 ActiveSync device downloads emails while it’s in quarantine in an Exchange Server 2013 environment
  • KB 3105389 WSMan-InvalidShellID error when you create remote PowerShell sessions in an Exchange Server 2013 environment
  • KB 3100519 No responses are sent from a room mailbox when a booked meeting extends beyond the date you set in Exchange Server 2013
  • KB 3093866 The number of search results can’t be more than 250 when you search email messages in Exchange Server 2013
  • KB 3088911 Inline attachments are sent as traditional when you smart forward an HTML email in an iOS device in Exchange Server 2013
  • KB 3088487 IOPS Write increase causes email delivery delays in an Exchange Server 2013 environment
  • KB 3076376 IMAP clients that use Kerberos authentication protocol are continually prompted for credentials in Exchange Server 2013
  • KB 3068470 “Something went wrong” error in Outlook Web App and ECP in Exchange Server 2013
  • KB 3048372 Exchange Calendar items are shifted incorrectly when some Windows DST updates are applied
  • KB 2968265 OWA cannot be accessed after you upgrade Exchange Server 2013

 

Notes:

  • This CU introduces an important change in the mechanism how Exchange Management Shell sessions will be initiated as of Exchange 2013 CU11 (and to be introduced in Exchange 2016, as well), called Mailbox Anchoring. More on this later in this article.
  • When using Exchange hybrid deployments or Exchange Online Archiving (EOA), you are required to stay current (version N) or be one version behind (N-1).
  • Cumulative Update may include schema or Active Directory changes (e.g. Role-Based Access Control). Make sure you run PrepareSchema /PrepareAD.  If you want to speed up the Cumulative Update installation process, you can temporarily disable certificate revocation checking as described here.

Note that Cumulative Updates can be installed directly, i.e. no need to install RTM prior to installing Cumulative Updates. Note that once installed, you can’t uninstall a Cumulative Update nor any of the installed Exchange server roles. The order of upgrading servers is irrelevant, unlike with previous generations of Exchange.

Finally, and I can’t emphasize this enough: For any Hotfix, Rollup, Service Pack or Cumulative Update, I’d recommend to thoroughly test this in a test and acceptance environment first, prior to implementing it in production. When you lack such facilities, hold out a week or two and monitor the comments on the release article or TechNet forum for any issues.

You can download Exchange 2013 Cumulative Update 11 here; UM Language Packs can be found here.

MAILBOX ANCHORING
This CU introduces an important change in the administrative model. In short, you need to home your administrative mailbox on the Exchange platform level you want to administer Exchange from (mailbox anchoring), as you will connect (or be proxied) to an Exchange Management Shell (EMS) session on that host. In other words, use an administrative account with a mailbox on Exchange 2013 to administer Exchange 2013, use an admin mailbox on Exchange 2016 for Exchange 2016. The logic behind this is to work around mixed-version environment issues, as newer Exchange versions may introduce changes, like new or enhanced cmdlets but also deprecated functionality. New general recommendation is to keep arbitration mailboxes as well as administrative mailboxes on the most current version.

If the admin has no mailbox, or if it’s unavailable, arbitration mailboxes – primarily SystemMailbox{bb558c35-97f1-4cb9-8ff7-d53741dc928c} – are considered for hosting your EMS session. Also, that ‘Connected to <Server>’ message when you open up an EMS session will no longer always mean your EMS session is hosted on that server; it could mean your EMS session is being proxied through there, which can create challenges when you’re running multiple sites with low bandwidth links – you may need to move your admin mailbox around or create one for local administration to enjoy better response times. You can only discover which host your session runs on by inspecting the local environment, using elements like the env:COMPUTERNAME variable or [System.Net.Dns]::GetHostName().

Also, it might be wise to spread administrative mailboxes over different servers or databases, in case your arbitration mailboxes become unavailable together with that one administrative mailbox, as you need to recover one of those just so you can set up an EMS session. The last resort for running an EMS cmdlets – against all best practices and recommendations, as it bypasses Role-Based Access Control for example – is  to load the Exchange module using Add-PSSnapIn. But be advised, you may not have all required permissions, for example your admin account may not have direct Active Directory permissions (and which is one of the reasons you shouldn’t just load the snap-in under normal circumstances).

The Exchange Team put up a separate blog to explain this change in behavior here.

Exchange 2010 SP3 RU12 & Exchange 2007 SP3 RU18

Exchange 2010 LogoThe Exchange Team released Rollup 12 for Exchange Server 2010 Service Pack 3 (KB3096066) as well as Rollup 18 for Exchange Server 2007 Service Pack 3 (KB3078672). These update raise version numbers to 14.3.279.2 and 8.3.445.0 respectively.

Apart from a Daylight Savings Time update, documented here, these Rollups contain the following fixes:

Exchange 2010 SP3 Rollup 12:

  • KB 3048372 Exchange Calendar items are shifted incorrectly when some Windows DST updates are applied
  • KB 3096125 CryptographicException error when Edge Transport service crashes in an Exchange Server 2010 environment
  • KB 3097219 Organizer’s name isn’t displayed in the subject of the recurring meeting requests in Exchange Server 2010
  • KB 3106421 Very long URLs in an email message don’t open in OWA in Internet Explorer
  • KB 3115809 Mailboxes can be accessed when the DefaultNetworkCredentials option is selected when you use Exchange Web Services Managed API to connect to Exchange Server

Exchange Server 2007 SP3 Rollup 18:

  • KB 3106421 Very long URLs in an email message don’t open in OWA in Internet Explorer

Notes:

    • If you want to speed up the update process for systems without internet access, follow the procedure described here to disable publisher’s certificate revocation checking.
    • If you got an Exchange 2010 DAG, and want to properly update the DAG members, check the instructions here.
    • As for any Hotfix, Rollup, Service Pack or Cumulative Update, apply this update to a acceptance environment first, prior to implementing it in production. When you lack such facilities, hold out a certain period and monitor the comments on the release article or TechNet forum for any issues.

Rollups are cumulative per service pack level, i.e. they contain fixes released in earlier update Rollups for the same product level (RTM, SP). This means you can apply the latest Rollup after installing a fresh installation of RTM or SPx version, for that product level.

You can download Exchange 2010 SP3 Rollup 12 here and Exchange 2007 SP3 Rollup 18 here.

OWA vulnerable to backdoor hack?

fudLast Update: October 10th, 2015

Yesterday, news rose of a security vulnerability in Outlook Web Access (OWA). A company called Cybereason claimed to have discovered an OWA backdoor hack of which they published in a report, “Webmail Server APT: new persistent attack methodology targeting Microsoft Outlook Web Application (OWA)” (APT stands for Advanced Persistent Threat). Supposedly, an OWA backdoor in ‘OWA Server’, the term used for Exchange Server in the report, allows a hacker to collect clear text usernames and passwords.

News sites quickly picked up the story, with catchy headlines such as:

  • New Outlook mailserver attack steals massive number of passwords (Arstechnica)
  • Microsoft OWA falls victim to password-pinching APT attack (Inquirer)
  • Potent OWA backdoor scores 11000 corporate creds from single biz (The Register)
  • Hackers Breach Microsoft OWA Server, Steal 11,000 User Passwords (SoftPedia)
  • Researchers find credential-stealing webmail server APT attack (ComputerWeekly)

The news was copied a lot without fact checking, and Microsoft felt the need to publicly make a statement: “No new security vulnerability in Outlook Web Access (OWA)”. Unfortunately that doesn’t stop media from reporting, as they are driven by a model based on page views and clicks. And such headlines most certainly will attract viewers.

Looking closer at the report, I’m inclined to think the company wanted to push for business and free publicity by spreading FUD (Fear, Uncertainty and Doubt), not uncommon in the security world. The report states that it is required to have installed (report does not disclose how) a malicious ISAPI filter on the ‘OWA Server’, without details on how this was achieved. Most likely they have used (or are referring to) the OWAAuth ISAPI filter also mentioned in a threat report (TG-3390) from Dell, dated August, 2015. The OWAAuth.dll filter authenticates users through Forms-Based Authentication against Active Directory.  Capturing and decoding client traffic is what these ISAPI filters can do, so that’s not worrying. Unfortunately, Cybereason report does not state the version of the ‘OWA Server’ or operating system. Was it current, and fully patched?

Key question is how did this filter get on the Exchange server in the first place? A properly managed environment does not allow for this type of access. So, the problem is likely not with the ‘OWA Server’ or the operating system. In a response on a blog reporting on this issue, Cybereason clarified that, “The hackers managed to obtain access to this server using stolen credentials.” Well, there is the confirmation of the real issue at hand: This is not an ‘OWA Server’ issue. The person could in theory have done anything with those stolen credentials.

In their response, the Cybereason spokesperson also stated that:

“The problem is that this server was in a very unique position. On one hand it’s completely internet facing and on the other hand, it is a focal point for the full credentials of all employees in the organization. Companies should be wary of using this server without requiring VPN (although this is usually its biggest advantage) and at the very least, require 2FA (2 factor authentication).”

I agree on the multi-factor authentication statement, especially for administrative or high profile accounts. However, claiming that VPN would prevent the issue is strange, as with most typical organizations that same set of stolen credentials would allow for setting up a VPN connection, maybe requiring some guesswork on the endpoint, but in the end enabling access to the same environment and practicing the same malicious behavior. Also, it is best practice to use a  more regular account for e-mail and connectivity, requiring another set of credentials for administrative privileges.

So, while the report may be based on a real world scenario, always have a healthy dose of common sense when reading these ‘research reports’ from companies selling security products and services. Manage your Active Directory and Exchange environment properly, use MFA for privileged accounts and remote access, and life should be good.

Other Exchange fellows also debunked the report:

Update (Sep9): If you are nevertheless still concerned, and want to do a quick scan of the currently loaded ISAPI modules on your Exchange servers, you can run the cmdlet below (be advised it’s a one-liner!). You should be able to spot ISAPI modules loaded from unusual locations or reporting an unexpected version number:

Get-ExchangeServer | ForEach-Object { Invoke-Command -ComputerName $_.Name -ScriptBlock { Get-WmiObject -Namespace 'Ro
ot\MicrosoftIISv2' -Class IISFilterSetting -Authentication 6 | ForEach-Object { (Get-Item $_.FilterPath | Select -ExpandPropert
y VersionInfo) } } } | Sort-Object PSComputerName,FileName | Format-Table -AutoSize PSComputerName, ProductVersion, FileName

isapifilt1

Update (Sep10): Cybereason provided some more details through Twitter and will publish a FAQ next week. However, more details were already given in an interview with ThreatPost (by Kaspersky Lab), in which Cybereason states that:

  • The harvesting took place over a period of months.
  • Stolen credentials were used to load a malicious, unsigned ISAPI filter, OWAAuth.dll.
  • The malicious OWAAuth.dll was residing in a non-standard location.
  • The malicious OWAAuth.dll was persistently loaded by modifying the registry.
  • Other modules were loaded, amongst them PlugX which has been around for a while, and which is the actual backdoor providing remote control mechanisms.

There are lots of similarities with the Cybereason case and Dell CTU’s TG-3390 analysis (use of PlugX, OWAAuth.dll). Since the harvesting took place over a longer period, were administrators not aware of the theft or not paying attention. Could it be that there’s a sudden increase of organizations and administrators not properly dealing with stolen passwords and password policies in general?

Meanwhile, Cybereason also claims the report, “was a malware analysis report and never about an OWA exploit”. While they have no control over the media, wording like “Cybereason Labs Reports on OWA Backdoor Attack” implies something differently. They also state one of the main concerns is, “Corporate Microsoft OWA servers are high prevalence in financial institutions”, which seems odd statement. Possibly, it’s a clue on where they hope to push business from, but from my personal experience these organizations are the most likely to have implemented multi-factor authentication and provide limited – if any at all – remote access functionality.

2015 Microsoft MVP Award

I am proud and happy to announce I got re-awarded the Microsoft MVP Award for Exchange Server for the third year in a row:

mvp2015

MVP awards are given to individuals by Microsoft in recognition of their contributions to the technical community, such as this writing on blogs or books, presenting, forum contributions or The UC Architects podcast.

I’d like to take this opportunity to thank my readers, followers, fellow MVPs and of course the Microsoft employees that have encouraged, helped and supported me over years.

My MVP profile can be found here.

IT/Dev Connections 2015 App

IMG_0608A quick note that if you are attending IT/Dev Connections this year, you can now build your schedule using a mobile app. The app allows you to browse and pick from 190 sessions, view speaker bios, etc.

The app is available for:

For other devices, you can use the generic mobile website here.

Note: You can still register for the event. New registrations can use SPKRSOC15 when registering for a $400 off!

KEMP LoadMaster & HA Virtual ID

imageA small heads-up on something which you need to configure when deploying a Highly Available setup of physical or virtual KEMP LoadMaster devices in environments with redundant network routing components, but this may apply to other components with similar functionality as well. While in typical environments the LoadMaster’s default setting will never be an issue, it can easily be overlooked or not immediately considered suspect when you do have issues, for example in hosted environments.

Note: If you are looking for more information on load balancing Exchange 2013 using KEMP LoadMaster devices, Exchange-fellow Jeff Guillet did an excellent multi-part write-up on this topic here.

When configuring multiple LoadMaster’s in a High Availability setup, one of the settings is the HA Virtual ID parameter, which is located System Configuration > Miscellaneous Options > HA Parameters. This setting configures the routing identifier used by the LoadMaster as part of the VRRP or Virtual Router Redundancy Protocol (see RFC5798).

The HA Virtual ID is used to construct a unique MAC address, so that all devices in the same VRRP group can communicate. The MAC address uses a format as defined by VRRP, and is 00:00:5E:00:01:<ID> for IPv4 and 00:00:5E:00:02:<ID> for IPv6.  One device, the Master being the Active LoadMaster, owns the VRRP group and manages its MAC address and shared IP address.

As you can imagine, using the same identifier for multiple non-related devices on the same segment may cause unexpected behavior, like LoadMasters being unable to communicate with eachother, both HA LoadMasters thinking they are the Active HA node, or other disruptive behavior. This is likely caused by a device other than LoadMasters managing the VRRP group.

Therefor, it is recommended to always change the default value of ‘1’, but always consult with the network or hosting people which value to use, as different vendors use their own default ID. For example, Cisco may use a different default value than FortiNet or CheckPoint for their redundant networking components. Of course, you also need to use different values when using multiple HA LoadMaster deployments on the same segment.

Ignite 2015 Session Download Script

ignite ButtonYesterday was the first day of Ignite, and Exchange fellow Tony Redmond put up a nice summary of the first day, keynoted included, here.

For those not attending Microsoft Ignite, attending different sessions or not able to enter a session because the room was full, Microsoft publishes Ignite sessions on Channel 9. Because you may want to watch sessions offline, some people created scripts to retrieve all session videos and slidedecks.

Here is a slightly modified script, originally from Claus Nielsen, to download all Microsoft Ignite 2015 videos and slidedecks as the become available on Channel9. You can select sessions based on category or speaker, which helps narrowing down the contents offered at Ignite to sessions you are interested in. The script also allows you to download other session videos and decks, for example from Build 2015 or last year’s TechEd NA.

You can download the script from the TechNet Gallery here.

Microsoft Ignite 2015 Countdown

ignite ButtonIn only 2 weeks, the Microsoft Ignite event will be held at Chicago, USA. With the demise of Microsoft Exchange Conference, this is the major Microsoft conference this year. Its the place where people involved with Exchange will get updated on next version of Exchange. It is also the place to be informed in related areas, such as Office 365, Office 2016 or Azure, or catch up with your peers.

Microsoft recently revealed a small glimpse of what’s coming in Exchange 2016, such as modern attachments in OWA, which will allow you to send links to attachments stored on OneDrive for Business instead of embedding them in the message.  The article not only provides teasers as Exchange on-premises will – hopefully – be brought more up to par with the Exchange Online offering. It will also give many people peace of mind as there will be another version of Exchange on-premises.

In just 3 days, a whopping number of 82 sessions related to Exchange or Exchange Online will be held, so creating a schedule could be challenging. I expect these sessions t to reveal a lot more details on Exchange 2016 and its new features or enhancements. Million dollar question: will the IOPS requirement again change significantly? Be advised that the schedule is still not 100% fixed, so check back often for updates or plan for alternative sessions.

I am sure Microsoft will make this new consolidated conference a success. For those attending or presenting, I wish you a great time in Chicago at Ignite or one of the side events or at one of the many parties such as ENow’s Scheduled Maintenance. Unfortunately, I will not be attending Microsoft Ignite. For myself or others looking for session contents, Microsoft stated Ignite sessions will be recorded and be made available within 48 hours.

On another note, I will be at IT/DEV Connections later this year in Las Vegas. With Jaap Wesselius, I will be hosting a workshop on ‘Managing Exchange Online and Exchange On-Premises using Powershell’. If you plan to visit another conference this year, be sure to consider Connections, which will be held from September 14th-17th in Las Vegas, USA. Connections is independent, will have lots of sessions on Exchange on-premises as well as Office 365 topics. Sessions will be hosted by well-known speakers from the industry.

The UC Architects Podcast Ep47

iTunes-Podcast-logo[1]Episode 47 of The UC Architects podcast is now available,which was recorded at the Norwegian Lync Day. This episode is hosted by Steve Goodman, who is joined by John A Cook and Ståle Hansen. Editing was done by Andrew Price.

Some of the topics discussed in this episode are:

  • Outlook for iOS and Android
  • Microsoft Ignite Session Catalog
  • Blocking Outlook App for iOS & Android
  • Planning and Migrating a Small Organization from Exchange 2003 to Exchange 2013
  • AWS Quick Start Reference Deployment – Exchange Server 2013
  • Considering an Exchange 2013 DAG without AAP?
  • Using a Microsoft Azure VM as a DAG witness server: Exchange 2013 Help
  • Securing Exchange and Lync 2013 with Multi-Factor Authentication
  • Exchange Server 2010 Reaches the End of Mainstream Support
  • Office for Android
  • Azure AD Improvements
  • Amazon Workmail
  • Setting up a multi-forest Azure AD Sync deployment
  • Office 365 Exchange Online Message Size Onboarding Limit Increased to 150mb
  • Drive Shipping and Network Based Data Import for Office 365
  • Skype for Business Video Interoperability Server (VIS)
  • Cumulative Update 10 for Lync Server 2013 released December 31, 2014
  • Fix for Google Chrome, stability
  • Lync 2013 Standard Automatic APP CU 10 December 2014
  • Lync SDN For Dummies – Part 2.1
  • Lync Admin Tools (free): make configuration, administrative and troubleshooting of Lync easier
  • Video calling between Skype and Lync is temporarily disabled
  • What’s New in Skype4B: SILK is default codec for P2P sessions Synchronize Lync Presence with Skype – Lync Exchange – UC Blog
  • Lync Dude: Simple Understanding of Lync Windows Fabric  Failover
  • Lync and Skype video calling is coming to your Android and iOS Devices
  • Passive Auth for Lync 2013 Android mobile client
  • Events

More information on the podcast including references and a link to download the podcast here or you can subscribe to the podcasts using iTunes, Zune or use the RSS feed.

About
The UC Architects is a bi-weekly community podcast by people with a passion for Unified Communications; our main focus is on Exchange, Lync or related subjects.

EighTwOne 2014 Stats

stats chartOi! A tad late as I was enjoying a trip and celebrating New Year in the beautiful country of Brazil, but happy new year to all dear readers and followers.

The start of a new year gives opportunity to reflect on the past year which has been quite busy, apart from writing blog posts and creating and supporting scripts:

Looking at 2015, it will see the daylight of Ignite, the MOAME (Mother Of All Microsoft Events), and merger of events like MEC, LyncConf, TENA, SPC and MMS which will be held from May 4 – 8 in Chicago.

It promises to be an interesting year for Exchange On-Premises, with Microsoft’s cloud-first, mobile-first strategy, and Exchange Online / Office 365 as well. The next version of Exchange (’16’) it expected to be announced at Ignite, and it will become clear which features, new or currently available via Office 365, will make it to the next version of Exchange ‘on-premises’, and which ones will not. With the next version of Windows Server (’10’) expected end of 2015, it is very likely that Exchange ’16’ will require Windows Server 2012 R2 (or 2012) or Windows Server ’10’, adding support for Windows Management Framework 5 (PowerShell). Note that WMF5 might become available as an individual component, before the release of Windows Server ’10’. If things follow earlier OS/WMF dependencies for Exchange, the combination of Windows Server 2012 R2 with WMF5 is highly unlikely to become a supported combination for hosting Exchange.

There is also lots of development towards Azure, looking at the recent support for hosting your File-Share Witness on an Azure VM. This option, originally announced as a ‘possible feature’ in the works for CU2 back at TechEd North America 2013, introduces an interesting alternative for site fail-over scenarios.

Some stats of 2014:

Apart from the Archives, Versions, Builds and Dates, Schema Versions and Toolkit pages, these were the Top Posts (yes, Exchange 2010 is still a strong lead through search engines):

statscountries2014

Top 5 Visitor Countries:

  • United States (95,660)
  • United Kingdom (28,733)
  • Germany (22,730)
  • Australia (14,986)
  • Netherlands (14,242)

Top 5 Referrers:

  • social.technet.microsoft.com (TechNet forum)
  • blogs.technet.com (Technet blogs)
  • exchangeserverpro.com (Paul Cunningham)
  • experts-exchange (community)
  • community.spiceworks.com (community)

Top 5 Search Terms:

  • exchange target address hybrid query
  • kb2506143
  • powershell ise for exchange
  • msexcheseparamcachesizemax
  • exchange versions