MEC 2014: Austin, 31 March-2 April 2014

This is pretty darn exciting: Microsoft has announced the official date and time of the Microsoft Exchange Conference (MEC) in 2014. It will be held in Austin, home of at least one of the original MECs (the first one, maybe? I wasn’t there so I’m not sure) from 31 March to 2 April 2014. 

I am sure that nothing bad will come of Microsoft’s decision to include April Fool’s Day as part of the conference. Nope, not at all.

On a personal note, I am excited that the conference will be in Austin. It’s one of my favorite cities, and I’ll be making side trips to see family (Hi, Lee Anne!) and friends while there. I also believe that we should have an Exchange-themed visit to the Salt Lick BBQ. Stay tuned for details!

Loading PowerShell snap-ins from a script

So I wanted to launch an Exchange Management Shell (EMS) script to do some stuff for a project at work. Normally this would be straightforward, but because of the way our virtualized lab environment works, it took me some fiddling to get it working.

What I needed to do was something like this:

c:\windows\system32\powershell\v1.0\powershell.exe -command "someStuff"

That worked fine as long as all I wanted to do was run basic PowerShell cmdlets. Once I started trying to run EMS cmdlets, things got considerably more complex because I needed a full EMS environment. First I had to deal with the fact that EMS, when it starts, tries to perform a CRL check. On a non-Internet-connected system, it will take 5 minutes or so to time out. I had completely forgotten this, so I spent some time fooling around with various combinations of RAM and virtual CPUs trying to figure out what the holdup was. Luckily Jeff Guillet set me straight when he pointed me to this article, helpfully titled “Configuring Exchange Servers Without Internet Access.” That cut the startup time waaaaay down.

However, I was still having a problem: my scripts wouldn’t run. They were complaining that “No snap-ins have been registered for Windows PowerShell version 2″. What the heck? Off to Bing I went, whereupon I found that most of the people reporting similar problems were trying to launch PowerShell.exe and load snap-ins from web-based applications. That puzzled me, so I did some more digging. Running my script from the PowerShell session that appears when you click the icon in the quick launch bar seemed to work OK. Directly running the executable by its path (i.e. %windir%\system32\powershell\v1.0\powershell.exe) worked OK too… but it didn’t work when I did the same thing from my script launcher.

Back to Bing I went. On about the fifth page of results, I found this gem at StackExchange. The first answer got me pointed in the right direction. I had completely forgotten about file system virtualization, the Windows security feature that, as a side effect, helps erase the distinction between x64 and x86 binaries by automatically loading the proper executable even when you supply the “wrong” path. In my case, I wanted the x64 version of PowerShell, but that’s not always what I was getting because my script launcher is a 32-bit x86 process. When it launched PowerShell.exe from any path, I was getting the x86 version, which can’t load x64 snap-ins and thus couldn’t run EMS.

The solution? All I had to do was read a bit further down in the StackExchange article to see this MSDN article on developing applications for SharePoint Foundation, which points out that you must use %windir%\sysnative as the path when running PowerShell scripts after a Visual Studio build. Why? Because Visual Studio is a 32-bit application, but the SharePoint snap-in is x64 and must be run from an x64 PowerShell session… just like Exchange.

Armed with that knowledge, I modified my scripts to run PowerShell using sysnative vice the “real” path and poof! Problem solved. (Thanks also to Michael B. Smith for some bonus assistance.)

Exchange 2010 SP3 released

It’s here! 

 Microsoft today released Service Pack 3 for Exchange 2010 (available from http://www.microsoft.com/en-us/download/details.aspx?id=36768).

SP3 enables full coexistence between Exchange 2010 and Exchange 2013, plus it adds a couple of new features, such as Windows Server 2012 support. See the full feature list here.

After you’ve upgraded allof your 2010 servers to SP3, you will be able to install Exchange 2013 in your forest. Opinions differ on whether you should– for example, fellow MVPs Tony Redmond, Michael B. Smith, and Jason Sherry all agree that you should wait. For my part, I’m OK with deploying 2013 into production; it does have some rough edges, but I think CU1 is right around the corner, so adding 2013 in your environment now is certainly defensible.

Excessive transaction log growth with iOS 6.1 devices

Well, it appears that Apple has done it again: reports are starting to surface of runaway transaction log growth when mobile devices running iOS 6.1 synchronize with Exchange Server. Tony has a good synopsis here.

Those of you who have been administering Exchange for a while may think this sounds familiar– that’s because there was a very similar problem with Microsoft Entourage back in the day, as detailed by Jeremy Kelly here. Remarkably, a couple of years later, we got the same bug in a slightly different guise, as described in KB 935848. In both cases, the problem was that the client was too stupid to detect certain types of failures, so it would keep retrying the failed operation, which would keep failing. This endless loop quickly resulted in large volumes of transaction log files on the Exchange server.

Luckily, Exchange 2010 and 2013 include throttling to prevent misbehaving clients from using up an excessive share of resources. However, the throttling controls available regulate EAS based on the amount of time user requests take, the number of concurrent connections, or the number of device partnerships. None of these parameters are useful in preventing the iOS 6.1-related problem; it’s not that the individual requests take up an excessive amount of time, it’s that there are so many requests that they generate an excessive log volume. (This video may provide a useful explanation for the phenomenon.)

Exchange 2013 includes the ability to specifically block misbehaving Exchange ActiveSync devices based on “suspicious” behavior. I will have a lot more to say about that in the near future, although that spiffy feature doesn’t help anyone now suffering the problem. For now, all we can do is the following:

• Block iOS 6.1 devices using an Exchange ActiveSync device access rule
• Discourage your users from upgrading, although I expect this to be an ineffective strategy
• If you have a support relationship with Apple, report this problem to them. If you’re a developer, file a RADAR issue. If you have enterprise technical support with Apple, use it. I’ve seen reports that the ordinary consumer-level technical support (i.e. the $49 pay-per-incident support, as well as AppleCare) doesn’t have any way to report this particular problem in an actionable way.

Thoughts for another time: the rapid adoption rate of iOS devices has many benefits for users, including largely avoiding the fragmentation problems that plague Android with issues (like this “smishing” fix that virtually no one has). However, when Apple ships a buggy update, which is common, that rapid adoption multiplies the pain of the bug.

Update 1535 CST 8 Feb: Ina Fried at AllThingsD is reporting that Vodafone is telling iPhone 4S users not to upgrade to iOS 6.1.

Michael B. Smith’s list of Exchange 2013 gotchas

Fellow MVP and all-around ace Michael B. Smith just published an article accurately titled “Exchange Server 2013 Gotchas.” Rather than editorialize, I’ll just ask that you read the article and consider the list of issues he’s found.

I don’t agree that all the things he lists as faults are actually faults; for example, I have no problem with Outlook 2003 no longer being supported, and it doesn’t matter to me that installation is slow. Many of the other items he lists have fairly low impact, such as the absence of the Edge Transport role (which very few of my customers use) or the fact that there is no longer a separate UM role– very few organizations have enough concurrent calls to warrant investing in a dedicated UM server. No one likes to see functionality disappear, such as the loss of S/MIME support in OWA, but at the same time we’ve gotten a ton of new functionality (including offline mode and touch mode in OWA, to name two possible reasons why S/MIME was deprioritized.)

Having said that, there are a number of his points that I agree with wholeheartedly. The documentation is clearly not finished; there are many missing pieces, and this is compounded by Microsoft’s still-unresolved decision to change the way TechNet URLs work. The Exchange 2013 message hygiene functionality is useful but limited, and the fact that there is still no way to deploy Exchange 2013 into an existing Exchange 2007/2010 organization is difficult to swallow– Microsoft must know that this is a huge bottleneck for deployments, so there must be some very good reason why the needed coexistence patches have not yet been released.

I don’t think I’d agree with his opinion that “Exchange 2013 is not ready for prime time,” though. Shipping is a feature in itself, and while that’s no excuse to rush a poorly designed or unstable product out the door, I think that Microsoft generally does a good job of balancing market demand for releases against the engineering and support effort required to prepare those releases. With Kevin Allison’s many public statements about getting to a more stable, predictable release cadence for Exchange (both as an on-premises product and a service), I think the new normal is going to be seeing more frequent, more incremental releases. Exchange 2010 RTM, of course, lacked a number of features (such as the ability to manage public folders) that were added in SP1, so the idea of shipping what’s ready to hit a schedule and fleshing it out in an incremental release is nothing new.

Michael’s larger point is well taken: before you deploy Exchange 2013, you should be very sure that you understand how it differs from previous versions and how those differences may affect your messaging operations. As Tony and I work on Exchange 2013 Inside Out, we are taking careful note of the kinds of issues Michael points out, as well as how Microsoft responds to them; we’ll keep doing so throughout the release of Exchange 2010 SP3 and, eventually, Exchange 2013 SP1, to make sure that the book reflects the best available knowledge when it’s released later this year.

In possibly related news, if you’re interested in software engineering and release management issues, I suspect Steven Sinofsky’s new blog, Learning by Shipping, might be of interest to you.

Now, back to the unified messaging chapter… and go Seahawks!

Microsoft wins UK case vs Motorola Mobility/Google

Earlier this year I had the unique (to me) opportunity to serve as a technical expert witness in a court case in the UK. Tony’s already written about the case but I wanted to add my perspective.

I was contacted by Bird and Bird to see if I might be willing and able to act as a technical expert in a court case; that’s all they said at first. The nature of the questions they were asking soon clued me in that the case involved Exchange ActiveSync and multiple-point-of-presence (MPOP) support for presence publishing– two completely separate technologies which Motorola/Google had lumped together in this case.  .

My role was to perform a wire-level analysis of the protocols in question: EAS, SIP/SIMPLE as implemented in Lync, and the Windows Live Messenger protocol. For each of these protocols, my job was to produce an accurate, annotated packet capture showing exactly what happened when multiple devices synchronized with the same account, and when the status on one device changed.

This isn’t what most people think of when they think of expert testimony; in courtroom dramas and books, it always seems like the expert is being asked to provide an opinion, or being cross-examined on the validity of their opinion. No one wanted my opinion in this case (which is perfectly normal), just for me to to accurately and impartially report what was happening on the wire.

This proved to be incredibly interesting from a technical standpoint. Like most administrators, it had never really occurred to me to look into the depths of the EAS protocol itself to see exactly what bits were being passed on the wire. After a great deal of study of the ActiveSync protocol documentation and many a late night slaving away over Wireshark and Network Monitor captures, I’d produced a report that showed the actual network traffic that passed between client(s) and server for a variety of test scenarios, along with an explanation of the contents of the packets and how they related to user action on the device.

Along the way, I gained a new appreciation for the economy of design of these protocols– it’s surprising how efficient they are when you look at them at such a low level. (And a shout out to Eric Lawrence for his incredibly useful Fiddler tool, which made it much easier for me to get the required data into a usable format.) I found a few bugs in Wireshark, learned more than I wanted to about SSL provisioning on Windows Phone 7.5 devices, and generally had a grand time. I particularly enjoyed working with the attorneys at Bird and Bird, who were quite sharp and had lovely accents to boot. (I’m not sure they enjoyed my accent quite as much, but oh well.)

When I finished my report, I submitted it to Bird and Bird and that was the last I heard of the case until today, when Mr. Justice Arnold issued his ruling. It was submitted as part of Microsoft’s justification explaining why their implementations did not infringe on Motorola’s patent; the purpose of having an annotated set of packet captures was to clearly illustrate the differences between the claimed innovations in the patent and Microsoft’s implementation to show why Microsoft wasn’t infringing.

Florian Mueller has a good summary of the case that highlights something I didn’t know: the patent at issue is the only one on which an Android manufacturer is currently enforcing an injunction against Apple. I am no patent attorney, but it would seem that Apple might have grounds to have this injunction lifted. It will be interesting to see what happens in the related German court cases that Muller cites, but it’s hard for this layman to see any other likely result besides a Microsoft win… but we will see.

Announcing Exchange 2013 Inside Out

Big news, at least to me!

Tony Redmond and I are delighted to announce a new joint project: Exchange 2013 Inside Out, a two-volume set that we will write for Microsoft Press, with an anticipated publication date in fall 2013. Tony is writing part 1, which covers the mailbox server role, the store, DAG, compliance, modern public folders and site mailboxes. I’m writing part 2, which covers client access, connectivity, transport, unified messaging, and Office 365 integration. This division looks as if I got more work to do, but Tony assures everyone that he can easily fill a book on just one topic.

Why two books where Exchange 2010 Inside Out merited just one? Well, just look at that book and reflect that it contains some 400,000 words in a 2-pound tome. Apart from the weight, it takes a long time to write such a book and there are tons of changes and new material in Exchange 2013 that we want to cover. The option of writing a single 500,000 word volume was just not attractive. Thankfully Microsoft Press agreed with us.

We’ve deliberately decided to take our time writing. There’s no point in rushing out a book based on a product immediately after it is released because no real-world experience exists. Microsoft runs an excellent Technology Adoption Program (TAP) that helps the development group understand how new versions of Exchange behave in production environments through early deployments, but we prefer to see how the software evolves and behaves as it is deployed more widely. This can’t really happen until after Microsoft releases Exchange 2010 SP3 and whatever update is necessary for Exchange 2007 SP3 to allow coexistence with Exchange 2013. Writing based on a firm foundation of real-world deployment experience has always seemed to make a lot of sense to us and we see no reason to change now.

Although the two volumes of Exchange 2013 Inside Out will stand alone, we will absolutely make sure that each volume complements the other. We will be technical editors for each other’s volumes, giving us equal opportunity to insert bad jokes and Exchange war stories across the breadth of both volumes.

Mostly because we have no firm dates in mind, we’re not releasing any details of our schedule, we hope that we will be able to offer an early-access program to readers through the Microsoft Press prePress program, so stay tuned!

Postel’s Law, Exchange ActiveSync, and iOS 6

Over the last week the Exchange community has learned a bit more about problems with Apple’s Exchange ActiveSync implementation in iOS 6; Microsoft has released a KB article outlining the problem and suggesting some workarounds, and Tony Redmond this morning pointed to a TUAW article that I hadn’t previously seen which asserts that the hijacking problem is a known issue with previous versions of iOS.

Tony’s article is titled “The emerging need for more supervision over ActiveSync implementations.” It’s certainly hard to disagree with that basic premise. Just over 18 months ago, Microsoft launched an Exchange ActiveSync logo certification program with the goal of inducing third parties who sell EAS-compatible devices or software to verify that they properly implement the client side of EAS. A quick check of this page maintained by Microsoft’s legal department shows just over four dozen EAS licensees, but only 3 vendors are listed on the EAS logo page itself. There are several possible explanations, ranging from lack of vendor interest in getting certified to a Microsoft failure to update the contents of the page.

Whatever the reason, I think Tony is right that Microsoft needs to be more proactive in requiring their EAS licensees to perform more robust testing on their clients.

Furthermore, I’m starting to think that the certification and testing program isn’t sufficient in and of itself, which brings me to Postel’s Law.

Some years ago, I worked for a company that made e-mail encryption software. This was during a time when even basic SMTP interoperability between different vendors’ systems was not assured– the Electronic Mail Association had biannual “plugfests” where vendors such as Lotus, Microsoft, and Netscape got together to ensure that their products could exchange SMTP mail properly. Whenever we ran into an interop problem, my boss would cite Postel’s Law:

Be conservative in what you do, be liberal in what you accept from others.

It seems pretty clear that Apple’s iOS EAS implementation has bugs– long-standing ones, if the TUAW article is right– and that Apple has a responsibility to fix them. I am coming to the opinion, though, that the server-side EAS implementation is too lenient about what it accepts. For years Exchange administrators had to suffer through a dance that went something like this:

1. A new version of Outlook would ship with a bug that would occasionally corrupt calendar or message items in some interesting way.
2. The corrupted items would cause the client to crash; in more severe cases, they could also crash the store process or cause backups, mailbox exports, etc. to fail.
3. The Exchange team would release a hotfix to keep that particular flavor of badly formed item from causing such a problem.
4. The Outlook team would release a hotfix that would keep that version of Outlook from emitting that particular flavor of corruption.
5. In the next release of Exchange, the team would add business logic to reject items with the specific flaws generated by Outlook.
6. GOTO 1

Over time, the Exchange business logic got better, as did Outlook’s track record of not creating bad items in the first place. This approach has served Exchange administrators and users pretty well, and it’s clear that the lessons learned from this painful process have been applied in Exchange Web Services. For example, EWS will reject attempts to create calendar items with end dates that come before start dates– something that Outlook used to occasionally do out of sheer perversity.

However, it looks as though, at least in this specific case, EAS doesn’t have business logic in place to catch the modifications that lead to the hijacking behavior. That’s something that Microsoft can fix in three steps.

• First, they can update the EAS spec so that it clearly defines which fields of each data item are to be considered read-only. This gives implementers a fair shot to understand where they may need to change their clients.
• Second, they can update Exchange itself so that it rejects client requests that violate the updated spec.
• Third, they can vigorously “encourage” their major ISV partners to test more aggressively against the updated spec. I have to wonder, for example, how much, if any, testing Apple has done against Exchange 2013…

I’ve been putting the blame on Apple for this problem, and while I don’t think that doing so is unfair or inaccurate, I also believe that the Exchange team can do a better job on the server side of blocking bad client requests, and I hope they’re madly planning how to do so in Exchange 2013 SP1.

More on bugs in iOS6 Exchange ActiveSync

As I hoped they would (but couldn’t mention in my previous post), Microsoft has weighed in with a KB article on the iOS 6 “hijacked meeting” bug. The article ID is 2768774, and it makes a couple of cogent points: first, that Microsoft is aware of, and is actively investigating, the bug. Second, that “Microsoft cannot mitigate this issue”; they recommend contacting Apple. The interesting part to me was the section on workarounds. Microsoft’s suggestions include:

• telling users not to make calendar changes using the iOS app
• recommending that users not update to iOS6 (as if! iOS device users are legendarily quick to update to new versions)
• blocking iOS 6 devices using Exchange ActiveSync allow/block/quarantine
• blocking users who have delegates, or who are granted delegate access, from using EAS
• using POP/IMAP instead of EAS (this made me laugh out loud– really!)

I have been testing Savvy Software’s Agenda app recently and like it quite a bit. However, it, like almost all the other calendar apps of which I’m aware,  it uses the underlying iOS calendar data store and sync mechanism, so I’d expect it to fall victim to the same bug. NitroDesk has ported its Touchdown app to iOS (though its App Store reviews are pretty poor), and Emtrace has the MoxierMail EAS client (which I still need to review.) So pick your poison: none of Microsoft’s suggested workarounds are great, but their hands are tied. It’s pretty clear this is an Apple problem and it will be incumbent on Apple to fix it… which I am sure they will, eventually. 

 

 

iOS 6 and Exchange ActiveSync misbehavior

Brace yourself for a surprise: there’s at least one major bug in Apple’s Exchange ActiveSync client in iOS 6.

cue shocked silence…

This is not surprising, of course; EAS is a fairly complex protocol and Apple has displayed a somewhat cavalier attitude towards verifying that their EAS clients behave properly. If you don’t believe me, all you have to do is look at this list of known EAS issues with third-party devices and see how many of them involve iOS.

Anyway, the latest widely reported  issue is that meetings sometimes appear to be hijacked– Alice will send out a meeting invitation to Bob and Carole, but somehow Carole will end up as the meeting organizer, thus gaining the ability to cancel or change meetings.

The hijacking bug isn’t the only one; users have reported a few other iOS 6 EAS issues, including its apparent failure to handle the case where the user’s primary SMTP address is different from the user name (e.g. an SMTP address of paul@robichaux.net coupled with a UPN of paul.robichaux@robichaux.net, for example), but these other issues all have easy workarounds; meeting hijacking is the most pestiferous.

The usual pattern for these bugs is fairly predictable:

1. Apple (or another EAS licensee) releases an update.
2. People start complaining in various fora that some EAS-related functionality is broken. However, these reports are only rarely made directly to either Microsoft or the vendor.
3. A critical mass of reports accumulates and begins to draw attention. This is often accelerated by the opening of support cases with either Microsoft or the ISV.
4. The source of the problem is identified, a resolution is developed, and everything is fixed.
5. GOTO 1

Now, stop laughing. That really is what usually happens. Note that Microsoft’s hands are somewhat tied during this process. Until they get feedback from customers that something is broken, they can’t very well investigate it. I imagine that it is very frustrating for the EAS team to see people blaming Exchange for what end up being bugs in the EAS client implementation. But I digress.

Tony points out a few nuances of how the process works, including suggestions for figuring out which devices are actually in use. (Note that one improvement in iOS 6 is that it reports a value for the DeviceOS property returned by Get-ActiveSyncDeviceStatistics; older versions just left that field blank.) Leaving that aside, though, it’s interesting to consider what’s known about the calendar hijacking bug. The best explanation I’ve seen, ironically, comes from the Z-Push development team. (Z-Push, you may recall, is an open-source EAS implementation that has nothing to do whatsoever with Exchange.) However, they are apparently first to market with a public explanation of the problem with iOS 6 that causes meeting hijacking. I won’t repeat it here; it’s worth reading the original. The root of the problem appears to be that iOS 6 emits meetings with zero attendees, and that Exchange accepts these as valid. I’m not sure whether Exchange’s acceptance is a desired behavior or not but I’m pretty sure that the device should never be emitting a zero-attendee meeting. It’s possible that there are cases where this is not true, which is why the Z-Push folks are holding their patch in QA for now instead of pushing it into the main tree of their product.

The tricky issue here, of course, is how to get the problem fixed for those of us who aren’t using Z-Push. Microsoft could conceivably make a change to Exchange’s business logic for calendar items, preventing ill-formed meeting items from being propagated. Apple could likewise fix their client so that it doesn’t send out ill-formed items in the first place. Both sides have an interest in providing a smooth EAS experience for iOS users, but each side has a different set of engineering and delivery constraints that make the process of actually getting the fix out to customers a challenge.

Microsoft hasn’t publicly said much about this bug, other than that it is being investigated. (And Apple, AFAIK, has said absolutely nothing about it, which is regrettably typical.) Your best bet is to keep an eye on KB 2563324 for updates so you’ll know when Microsoft believes they have a solid understanding of the problem and the best way to fix it.

Meanwhile, the Z-Push team claims that turning off the Exchange calendar attendant feature would eliminate the problem, at the cost of some useful functionality. The iOS 6 bug is rare enough in most environments that I’d advise living with it rather than giving up the attendant functionality, but that’s a choice you’ll have to make based on your users and their needs.

I wonder whether Emtrace’s MoxierMail client has this problem? I’ve got an evaluation copy but haven’t been able to evaluate it yet; might be time to move that up a couple of notches on the priority scale….

Some MEC schedule and content updates

Today the Exchange team updated the MECIsBack.com website to share more details of what awaits us in a mere 48 days! The complete schedule is a pretty broad outline, but the session list is quite tantalizing.

Day 1 starts with an opening keynote by Rajesh Jha, but the real goodies start with a technical keynote covering the architecture of what Microsoft is calling “the new Exchange.” (It’s interesting, btw, that SharePoint, Lync, Office 2013, and Windows 8/2012 aren’t calling their products “the new X”. I like the Exchange branding.)  There are a total of 8 additional breakout sessions, all on Exchange 2013, scheduled for the rest of day 1. This is definitely a good news/bad news situation, as these 8 sessions are stuffed into three time slots so you cannot attend them all. That means that we’ll all have to choose which sessions seem most interesting. The arrangement reminds me a bit of past MVP summits when we had to make choices such as “would I rather go to the ‘what’s new in PowerShell’ or ‘storage architecture changes’ session?” This is rather jarring given how lame the last few years’ worth of TechEd content has been for Exchange, but it’s a good problem to have. Fortunately the MEC folks will have the Exchange 2013 day-1 sessions recorded for later viewing. (Personally, I think I will probably hit the high availability, security, and “Apps for Outlook and OWA” sessions.)

Days 2 and 3 are all chalk talks. Microsoft is calling them “classroom sessions” but I picture something more informal than the typical lecture sessions, with lots of back-and-forth Q&A. The preview session content list includes a bunch of sessions both on Exchange 2013 and Exchange 2010. There are some interesting tidbits hidden in the session list: “What’s New In Support Programs with Exchange,” for instance, sounds intriguing given that Microsoft has not yet publicly said anything about upcoming support changes. The sessions on site mailboxes, modern public folders, and what’s new in anti-malware (you did know Exchange 2013 includes malware filtering now, right?) look worthwhile as well.

Microsoft hasn’t yet announced exactly which speakers will be presenting the new Exchange 2013 content. However, if you look at the speaker list you can make some informed guesses. I’d expect all of the Exchange 2013 sessions to be covered by Microsoft speakers (I love it that the Microsoft product group folks are listed under the heading of “Exchange Team Personalities”– I can attest that many of the Exchange folks are, in fact, lively personalities), and if you know who does what on the product team you can probably match session titles to personalities pretty easily.

I’m presenting two sessions: E14.302, “Developing Mobile Applications with Exchange Web Services,” and E14.303, “10 Things You Didn’t Know About Exchange Unified Messaging.” Other presenters include unindicted co-conspirator Tony Redmond, fellow MCM instructor Brian Reid, the formidable Glen Scales, ex-3Sharpie Devin Ganger, and a host of others whose names you’ll probably recognize.

Interestingly, Microsoft is still looking for suggestions for sessions– drop mecideas@microsoft.com a line if there are specific things you want to talk about that aren’t covered. The exhibitors list is now up to date as well, with most of the usual suspects represented– Quest, Binary Tree, Sherpa, and so on.

One open question: there are two evening events, plus an option post-event activity… I wonder what the MEC planners have up their sleeves for us? I can’t wait to find out. See you there!

Man-in-the-middle attacks against Exchange ActiveSync

I love the BlackHat security conference, although it’s been a long-distance relationship, as I’ve never been. The constant flow of innovative attacks (and defenses!) is fascinating, but relatively few of the attacks focus on things that I know enough about to have a really informed opinion. At this year’s BlackHat, though, security researcher Peter Hannay presented a paper on a potential vulnerability in Exchange ActiveSync that can result in malicious remote wipe operations. (Hannay’s paper is here, and the accompanying presentation is here.)

In a nutshell, Hannay’s attack depends on the ability of an attacker to impersonate a legitimate Exchange server, then send the device a remote wipe command, which the device will then obey. The attack depends on the behavior of the EAS protocol provisioning mechanism, as described in MS-ASPROV.

Before discussing this in more detail, it’s important to point out three things. First, this attack doesn’t provide a way to retrieve or modify data on the device (apart from erasing it, which of course counts as “modifying” it in the strictest sense.) Second, the attack depends on use of a self-signed certificate. Self-signed certificates are installed and used by Exchange 2007, 2010, and 2013 by default, but Microsoft doesn’t recommend their use for mobile device sync (see the 2nd paragraph here); contrary to Hannay’s claim in the paper, my experience has been that relatively few Exchange sites depend on self-signed certs.

The third thing I want to highlight: this is an interesting result and I’m sure that the EAS team is studying it closely to ensure that the future attacks Hannay contemplates, like stealing data off the device, are rendered impossible. There’s no current cause for worry.

The basis of this attack is that EAS provides a policy update mechanism that allows the server to push an updated security policy to the device when the policy changes. There are 3 cases when the EAS Provision command can be issued by the server:

• when the client contacts the server for the first time. In this case, the client should pull the policy and apply it. (I vaguely remember that iOS devices prompt the user to accept the policy, but Windows Phone devices don’t.)
• when the policy changes on the server, in which case the server returns a response indicating that the client needs to issue another Provision command to get the update.
• when the server tells the device to perform a remote wipe.

The client sends a policy key with each command it sends to the server, so the server always knows what version of the policy the device has; that’s how it knows when to send back the response indicating that the device should reprovision.

If the client doesn’t have a policy, or if the policy has changed on the server, the client policy key won’t match the current server policy key, so the server sends back a response indicating that the client must reprovision before the server will talk to it.

There seems to be a flaw in Hannay’s paper, though.

The mechanism he describes in the paper is that used by EAS 12.0 and 12.1, as shipped in Exchange 2007. In that version of EAS, the server returns a custom HTTP error, 449, to tell the device to get a new policy. A man-in-the-middle attack in this configuration is simple: set up a rogue server that pretends to be the victim’s Exchange server, using a self-signed certificate, then when any EAS device attempts to connect, send back HTTP 449. The client will then request reprovisioning, at which time the MITM device sends back a remote wipe command.

Newer versions of Exchange return an error code in the EAS message itself; the device, upon seeing this code, will attempt to reprovision. (The list of possible error codes is in the section “When should a client provision?” in this excellent MSDN article). I think this behavior would be harder to spoof, since the error code is returned as part of an existing EAS conversation.

In addition, there’s the whole question of version negotiation. I haven’t tested it, but I assume that most EAS devices are happy to use EAS 12.1. I don’t know of any clients that allow you to specify that you only want to use a particular version of EAS. It’s also not clear to me what would happen if you send a device using EAS 14.x (and thus expecting to see the policy status element) the HTTP 449 error.

Having said all that, this is still a pretty interesting result. It points to the need for better certificate-management behavior on the devices, since Hannay points out that Android and iOS devices behaved poorly in his tests. Windows Phone seems to do a better job of handling unexpected certificate changes, although it’s also the hardest of the 3 platforms to deal with from a perspective of installing and managing legitimate certificates.

More broadly, Hannay’s result points out a fundamental flaw in the way all of these devices interact with EAS, one that I’ve mentioned before: the granularity of data storage on these devices is poor. A remote-wipe request from a single Exchange account on the device arguably shouldn’t wipe out data that didn’t come from that server. The current state of client implementations is that they erase the entire device– apps, data, and all– upon receiving a remote wipe command. This is probably what you want if your device is lost or stolen (i.e. you don’t want the thief to be able to access your personal or company data), but when you leave a company you probably don’t want them wiping your entire device. This is an area where I hope for, and expect, improvement on the part of EAS client implementers.

Licensing change for Exchange multi-mailbox search

Microsoft today announced that you no longer need an enterprise client access license (ECAL) to use the multi-mailbox search feature. This is a welcome change, of course, since it means that it is now OK to run multi-mailbox searches against mailboxes that are licensed with a standard CAL. In essence, Microsoft is giving standard CAL holders something for free that formerly cost money. ECAL holders, of course, aren’t getting anything extra out of the deal, but I’d argue that the other ECAL features (including legal hold and the Personal Archive feature) probably make up for that.

The interesting question to me is: why this change at all, and why now? It’s common for Microsoft to adjust licensing terms with new releases of Exchange, so my guess is that we’ll see some differences in how the data loss prevention (DLP) and information management features of Exchange 2013 are licensed and which specific mix of CALs you need to use them.

Stay tuned…

Stalking the wily ADAccess event 2112

Timing is everything.

A week ago, I got a late-night phone call about a problem with an Exchange server that seemed to be related to an expired certificate; the admin had replaced the expired cert on one member of a two-node DAG, but not the other. He noticed the errors in the event log when troubleshooting a seemingly unrelated problem, installed the new cert, and then boom! Bad stuff started happening.  Problem was, the reported problem was that inbound SMTP from a hosted filtering service that doesn’t use TLS wasn’t flowing, so it didn’t seem likely that certificate expiration would be involved. By the time he called me, he had installed the new certificate and rebooted the affected server, and all seemed to be well.

Fast forward to Sunday night. I’d planned to patch these same servers to get them on to Exchange 2010 SP2 UR3, in part because I’d noticed a worrisome number of events generated by the MSExchange ADAccess service, chiefly event ID 2112:

Process MSEXCHANGEADTOPOLOGYSERVICE.EXE (PID=8356). The Exchange computer HQ-EX02.blahblah does not have Audit Security Privilege on the domain controller HQ-DC01.blahblah. This domain controller will not be used by Exchange Active Directory Provider.

This was immediately followed by MSExchange ADAccess event ID 2102 with the rather ominous message that

Process MSEXCHANGEADTOPOLOGYSERVICE.EXE (PID=8356). All Domain Controller Servers in use are not responding:

However, the event ID 2080 logged by ADAccess indicated that all but 1 of the GCs were up and providing necessary services, including indicating that their SACL allowed Exchange the necessary access. I couldn’t puzzle it out in the time I had allotted, so I decided to take a backup (see rule #3) and  wait to tackle the patching until I could be physically present. That turned out to be a very, very good idea.

Last night, I sat down to patch the affected systems. I began with the passive DAG node, updating it to SP2 and then installing UR3. I half-thought that this process might resolve the cause of the errors (see rule #2), but after a reboot I noticed they were still being logged. I suspected that the reported 2102 errors might be bogus, since I knew all of the involved GCs were running and available. As I started to dig around, I learned that this error often appears when there’s a problem with permissions; to be more specific, this SCOM article asserts that the problem is that the Exchange server(s) don’t have the SeSecurityPrivilege user right on the domain controllers. However, I was still a little skeptical. I checked the default DC GPO and, sure enough, the permissions were present, so I moved on to do some further investigation.

Another possible cause is that the Exchange servers’ computer accounts aren’t in the Exchange Servers group, or that permissions on that group were jacked up somehow, but they appeared to be fine so I discounted that as a likely cause.

Along the way I noticed that the FBA service wouldn’t start, but its error message was meaningless– all I got from the service control manager was a service-specifc error code that resisted my best attempts to Bing it. Without that service, of course, you can’t use OWA with FBA mode, which would be a problem so I made a mental note to dig into that later.

A little more searching turned up this article, which is dangerously wrong: it suggests adding the Exchange computer accounts to the Domain Admins security group. Please, please, don’t do this; not only does it not fix the problem, it can cause all sorts of other tomfoolery that you don’t want to have to deal with.

Still more digging revealed two common problems that were present on this server: the active NIC wasn’t first in the binding order and IPv6 was disabled on the two enabled NICs. Now, you and I both know that IPv6 isn’t required to run Exchange.. but Microsoft does not support disabling or removing IPv6 on Windows servers. And you know what they say about what “unsupported” means! So, I enabled IPv6 on the two adapters and got the binding order sorted out, then bounced the AD topology service and…

… voila! Everything seemed to be working normally, so I ran some tests to verify that *over was working as it should, then started patching the DAG primary– only to have setup fail partway through. Upon reboot, w3svc was caught in an endless loop of trying to load some of the in-proc OWA DLLs; it kept trying endlessly until I power-cycled the server. The problem with this is that the Active Manager service was starting, so the current active node would try to sync with it before mounting its copy of the DAG databases, but it never got an answer! Net result, no mounted databases on either server, and an unhappy expression on my face as the clock ticked past 11pm.

I put the primary member server in safe mode, then set the Exchange and w3svc services to manual star and rebooted it. Rather than spend a lot of time trying to pin down exactly what happened, I ran setup in recovery mode; it installed the binaries, after which the services restarted normally. I did a switchover back to the original primary node, verified mail flow, and went home. Life was good.

Until, that is, this morning, when I got an e-mail: “OWA is down.” I checked the servers and, sure enough, the errors were back and the FBA service was again refusing to start. After some creative swearing, I once again started digging around in the guts of the server. I couldn’t shake the feeling that this was a legitimate permissions problem of some kind.

At that point, I found this article, which pointed out something critical about GPOs: you have to check the effective policy, not just the one defined in the default policy. Sure enough, when I used RSoP to check the effective policy on the DCs, the Exchange servers did not have SeSecurityPrivilege on the DCs because there was a separate GPO to control audit logging permissions, and it had recently been changed to remove the Exchange Servers group. That was easy to fix: I added the Exchange Servers group to the GPO, ran gpudate, rebooted the passive node, and found that the services all started normally and ran without error. A quick switchover let me restart the topology service on the primary DAG member, after which it too ran without errors. End result: problem solved.

It’s still not entirely clear to me why that particular service needs to have the SeSecurityPrivilege right assigned. I’m trying to find that out and will update this post once I do. In the meantime, if you have similar symptoms, check to verify that the effective policy is correct.

What "supported" really means

If I had a nickel for every time I had had a discussion like the below…

<Customer> wants to <do something>. I don’t think it’s a good idea and tried to explain that to them. They want to do it anyway. Is it supported?

The particular discussion that triggered this post was a conversation among MCMs concerning a customer who wanted to know if they could configure an Exchange 2010 server so that it was dual-homed, with one NIC on the LAN and another in their DMZ. There are a number of good reasons not to do this, most related to one of two things: the inability to force Windows and/or Exchange to use only one of the installed NICs for certain operations, or the lack of knowledge about how to configure everything properly in such a configuration. For example, you’d have to be careful to get static routes right so that you only passed the traffic you wanted on each interface. You’d also have to be careful about which AD sites your server appeared to be a member of.

The big issue for me: that configuration would add complexity. Any time you add complexity, you should be able to clearly articulate what you’re gaining in exchange. Performance, scalability, flexibility, security, cost savings.. there has to be some reason to make it worth complicating things. This is a pretty fundamental principle of designing anything technical, from airplanes to washing machines to computer networks, and you violate it at your peril.

In this case, the gain is that the customer wouldn’t need to use TMG or a similar solution. That seems like an awfully small gain for the added complexity burden and the supportability issues it raises.

You might be wondering why I’d bring up supportability in this context. The cherry on the sundae was this comment from the fellow who started the thread: “It’s not written that you can’t do it, so they assume that means you can.” This is a dangerous attitude in many contexts, but especially so here.

I’ve said it before (and so has practically everyone who has ever written about Exchange), but it bears repeating:

Just because something is not explicitly unsupported, that doesn’t mean it is supported.

Microsoft doesn’t– indeed, can’t– test every possible configuration of Exchange. Or Windows. Or any of their other products (well, maybe except for closed consumer systems like Windows Phone and Xbox 360). So there’s a simple process to follow when considering whether something meets your requirements for supportability:

1. Does Microsoft explicitly say that what you want to do is, or is not, supported?
2. If they don’t say one way or the other, are you comfortable that you can adequately test the proposed change in your environment to make sure that it only has the desired effects?

Point 1 is pretty straightforward. If Microsoft says something’s explicitly supported, you’re good to go. If they explicitly say something is unsupported, you’re still good, provided you don’t do it.

Brief digression: when Microsoft says something’s unsupported, it can mean one of three specific things:

• We tested it. It doesn’t work. Don’t do it. (Example: a long list of things involving Lync device provisioning.)
• We tested it. It works. It’s a bad idea for some other unrelated reason. Don’t do it. (Example: going backupless with a 2-copy DAG.)
• We didn’t test it. We don’t know if it works. You could probably figure out some way to make it work.  If it doesn’t work, on your own head be it. (Example: the prior stance on virtualization of Exchange roles.)

OK, where was I? Oh yeah: if Microsoft doesn’t make an explicit statement one way or another, that is not an unconditional green light for you to do whatever you want. Instead, it’s an invitation for you to think carefully about what you’ll gain from the proposed configuration. If what you want to do is common, then there will probably be a support statement for it already; the fact that there isn’t should give you pause right there. If you believe the gain is worth the potential risk that comes from an increase in complexity, and you can demonstrate through testing (not just a SWAG) that things will work, only then should you consider proceeding.

(n.b. permission is hereby granted for all you Exchange folks out there to copy this and send it to your customers next time they ask you for something dangerous, ignorant, unsupportable, or otherwise undesirable.)