shawnbass.com - Network Administration blog

Beware of McAfee AntiVirus Enterprise 8.7 upgrade

shawn@shawnbass.com — Wed, 26 Nov 2008 19:17:14 GMT

A client of mine recently rolled out their upgrade from McAfee 8.0 Enterprise to 8.7 Enterprise. On the Citrix server environment we had two major application failures that were the result of the McAfee upgrade uninstalling the MS XML 4.0 Parser on the servers. There were two applications on the production Citrix environment that required the MS XML 4.0 Parser and they both stopped working following the upgrade. Unfortunately, the issue was not caught in Lab/UAT testing and was only found after McAfee went out to production. To make matters worse, something went wrong with the deployment mechanism and the McAfee upgrade went to all servers in one blast whereas the server team usually pushes in 2-3 separate sets of server groups. All of that combined = FAIL

So McAfee is responsible for the outage, right? WRONG!

What the Mcafee installer was doing was proper behavior. It had incremented the shared component registry entry for MSXML4.dll when it was installed on the servers. The shared DLL component dependency counters are stored in HKLM\Software\Microsoft\Windows\Current Version\SharedDLLS. Here's an example listing 5 installed apps that depend on MSXML4.dll on my workstation PC (see the DWORD 5 next to MSXML4.DLL).

Every time a program installs that depends on MSXML4.dll (or any other shared windows DLL), it's supposed to read the existing counter value, increment it, and then write it back out. Upon uninstalling that program, the installer engine is supposed to read the counter, decrement it, and write it back out. However, if the value of the counter is 1 and a program is uninstalling that has a shared dependency on that DLL, it's supposed to uninstall that component and remove the shared component counter from the registry. See, everyone is supposed to tidy up after themselves.

Windows Shared DLLs + Dumb Developers = FAIL

So in my client's situation, they had two applications that apparently required MSXML 4.0 Parser, but those apps never bothered to increment the shared DLL counter in the registry when they were installed. This is usually due to a dumb developer that doesn't realize that the component has to be registered with the installer engine in order to properly install and register the shared DLL component. So when McAfee Enterprise 8.0i went to uninstall, the installer engine noticed that the value of the shared component registry entry was 1and subsequently ran a removal process for the MSXML 4.0 Parser. This equals failure for the two apps that need MSXML.

So how do you know you've been affected?

1) If you're lucky, the application will complain about not being able to find the MSXML4.dll, or not being able to instantiate a object for the MSXML4 DOM. If you're not that lucky, then...

2) FileMon / ProcessMonitor from SysInternals will pinpoint the issue. Simply run it before launching your application and when you reach the place in code where the app is crashing or throwing an unexpected error, you'll see messages in the log where it was attempting to locate MSXML4.DLL in the search path and failing.

How do you fix it?

Download MSXML 4.0 SP2 Parser from Microsoft and install it. You can do the install manually or push it with whatever ESD system you're using. The MSI installs quite nicely with Citrix Installation Manager if you only have that.

Stay tuned for another entry from me regarding this issue with respect to application virtualization since it has some unique properties that make this particularly challenging...

Failure to run Windows Update with error 0x80070020 on Server 2003

shawn@shawnbass.com — Wed, 26 Nov 2008 18:12:43 GMT

I was recently running some Windows Update patches on one of my client's Server 2003 boxes (they aren't using a patch management solution) when I ran into a strange error that stated the Windows Update site could not be accessed and the error message listed was 0x80070020. This message did not appear while trying to install the updates, it appeared right after the selection for Express vs Custom. I chose Custom and it's supposed to query the available patches for the server, but instead I was greeted with error 0x80070020. In doing some reason on this error I found that it's related to a file locking issue. Some people on various forums reported issues with BitDefender AntiVirus on Windows XP and Vista as the source of the issue. While they're not running BitDefender on their server (they run eTrust), it was worth looking into. I also found KB883825 which also lists Anti-Virus as a potential source of issues for Windows Update. So I temporarily stopped the Realtime A/V scanner and voila! Windows Update website began working again.

Al Solorzano on Group Policy Preferences

shawn@shawnbass.com — Mon, 02 Jun 2008 14:52:24 GMT

Al Solorzano recently posted a nice article on Group Policy Preferences over here. If you're not familiar with Group Policy Preferences, it's the technology that Microsoft acquired from Desktop Standard. GPP is a nice way of extending management reach to registry settings that were previously not able to be managed by Group Policy (at least not without those PITA Administrative Templates anyway). There's also some powerful capabilities for filtering that are largely lacking from Group Policy. While out at Citrix Synergy a little over a week ago, I had the pleasure of some great conversation with Al at a group dinner outing. Al is extremely knowledgable about a variety of technologies and I highly recommend following his blog (if you don't already). Al's blog can be found here.

XP SP3 officially re-released

shawn@shawnbass.com — Tue, 06 May 2008 20:47:44 GMT

Microsoft has officially re-released Windows XP Service Pack 3 today. Get the full download here.

Citrix announces Workflow Studio tool for automating repetitive tasks. Is this FullArmor in disguise?

shawn@shawnbass.com — Thu, 14 Feb 2008 15:06:51 GMT

Citrix has recently announced a new product called Workflow Studio that is a tool for putting together a visual workflow for completing repetitive tasks. This workflow would then leverage PowerShell scripts to complete the individual tasks. What's interesting about this tool is that Citrix has mentioned that it came from an unmentioned technology acquistion/partnership. From my perspective, it sounds an awful lot like FullArmor's Workflow Studio product that was announced in July 2007. It's also got the PowerShell community wondering the same thing. Folks like Karl Prosser of the amazing PowerShell Analyzer and PowerShell Plus products questions the same thing in a blog entry on his site. For those looking for more information on Workflow Studio, Rich Crusco from Frameworkx.com put together a multi-part series on Workflow Studio. Is this the Full Armor product or what?

Exchange 2007 SIS and Entourage 2008 new features

shawn@shawnbass.com — Tue, 12 Feb 2008 00:11:11 GMT

A ton of things have been happening in the Exchange world that I've not been keeping up with very well, but I did come across two items that were of particular interest to me that I thought I'd share.

First, the MS Exchange Team has put up a blog item discussion a feature of Exchange that's been around a long time (Single Instance Storage). SIS is a technology that was introduced in Exchange 4.0 that allows for an email to only exist once in the Information Store for multiple different users (if those users are on the same system). So if an email (especially an email that contains attachments) is sent out to 10 different users, the Exchange server will only keep one copy of the email and all users will reference that copy. There are some changes with SIS in Exchange 2007 namely that it only performs Single Instance Storage of message attachments, not of message bodies. This makes perfect sense when you think about it considering that most of the storage that you'll consume on your Exchange server is related to attachments more so that simple text. Simple text is small and generally irrelevant whereas attachments kill you. This is particularly true since 90% of the organizations I've consulted for seem to think that Email = File Server. You can read further about the reasons why the Exchange Team made this decision in the blog post here. Ultimately it came down to a trade off between storage savings and IO Operations. Ultimately storage is cheap, IO is not.

The second thing I want to cover in this blog item is a discussion about the newly released Office 2008 for the Mac and Entourage (the Outlook equivalent for you the non-Mac people out there. Office 2008 for Mac is a completely new beast versus Office 2004 for Mac. Luckily Amir Haque from Microsoft has written about many of the new features in Entourage 2008. The two part blog item can be found here: Part 1 and Part 2. An excellent read.

Woohoo! Jeffrey Snover (creator of PowerShell) confirms suspicion that PowerShell 2.0 will include remoting!

shawn@shawnbass.com — Mon, 03 Sep 2007 15:52:57 GMT

PowerShell 1.0 was released in November 2006 and has already received over a million downloads. I recently came across an article published a few days ago where SearchWinIT.com interviewed Jeffrey Snover (creator/architect of PowerShell) where Jeffrey has confirmed that PowerShell 2.0 will have support for remoting. One of the primary limitations with PowerShell currently is that many of the existing Cmdlets and Providers are designed for local operation only. Those admins that wish to use it for remote operations are forced to either run the code on the actual remote systems, roll your own via WMI/.NET/COM or rely on third party remoting solutions such as the PowerShell Remoting project on CodePlex. I can't wait to see what PowerShell 2.0 will bring. Exciting times ahead!

Network Monitor 3.1 is released!

shawn@shawnbass.com — Tue, 03 Jul 2007 01:37:23 GMT

NetMon 3.1 is released and available on the Microsoft Connect site (the final release on the MS Download site will be posted in a few weeks). Here's a rundown of the new features:

Wireless (802.11) capturing and monitor mode on Vista – With supported hardware, (Native WIFI), you can now trace wireless management packets. You can scan all channels or a subset of the ones your wireless NIC supports. You can also focus in on one specific channel. We now show the wireless metadata for normal wireless frames. This is really cool for t-shooting wireless problems. See signal strength and transfer speed as you walk around your house!
RAS tracing support on Vista – Now you can trace your RAS connections so you can see the traffic inside your VPN tunnel. Previously this was only available with XP.
Right click add to filter – Now there's an easier way to discover how to create filters. Right click in the frame details data element or a column field in the frame summary and select add to filter. What could be easier!
Microsoft Update enabled – Now you will be prompted when new updates exist. NM3.1 will occasionally check for a new version and notify you when one is available.
New look filter toolbar – We've changed the UI related to apply and remove filters. You can now apply a filter without having to UN-apply it first.
New reassembly engine – Our reassembly engine has been improved to handle a larger variety of protocol reassembly schemes.
New public parsers – These include ip1394, ipcp, ipv6cp, madcap, pppoE, soap, ssdp, winsrpl, as well as improvements in the previously shipped parsers.
Numerous Bug Fixes – We've taken your reported problems on the connect site and fixed many of the confirmed bugs.
Faster Parser Loading – We've significantly improved the time it takes to load the parsers. Now rebuilding takes a fraction of the time it used to.

For those of you who are huge Ethereal/Wireshark fans, you really should checkout NetMon as it's really shaping up to be a nice product. Plus you don't need to buy SMS to use it anymore

Shawn

Windows 2003 Server SP2 is a critical update and will begin to be forced on June 12, 2007 via Automatic Updates (there is an opt-out though)

shawn@shawnbass.com — Wed, 04 Apr 2007 21:04:00 GMT

If you haven't had a chance to test 2003 SP2 AND you're configured to automatic updates, you may be interested in knowing that Microsoft has a fix that will allow your servers to bypass the forced install of 2003 SP2 that will begin on June 12th, 2007. If you're interested in using this opt-out, visit Microsoft's download site here for the download and instructions.

Shawn

The exact reason why you should not allow PST files on the network

shawn@shawnbass.com — Wed, 04 Apr 2007 20:29:56 GMT

I've always known that you're not supposed to use PST files across the network (LAN or WAN), but up until recently I did not have the specific proof as to the magnitude of problems it can cause (outside of corruption in the PST). I was recently looking for this information to pass to a client of mine, and I came across a great blog article from the Server Performance Team at Microsoft. Turns out there's all sorts of issues from I/O deadlocking to paged pool depletion, etc. Head over their blog and view the article.

Shawn