OK, important note - this article posits a Slammer-type worm that can use the vulnerabilities patched by MS03-043 to replicate. The article uses the term "in minutes" - sensationalism? Perhaps, but perhaps not.

In any case, make sure you're devices are patched with MS03-043. You can get the XPE version here.

Flaw could unleash another Slammer | CNET News.com

Back in October, I mentioned a new QFe we had, KB824706, which would let you apply desktop QFE's directly to XPE runtimes. It's a cool thing - we had to get some DCR's into the desktop QFE installer, and add some components to support the installer, but it works very well with only a 10Mb footprint hit at a maximum (the dependent components max out at 10Mb - if you've got some of them in your image, your footprint hit is smaller). There were a few caveats we documented (stuff like EWF images, remote boot, kernel updates, etc.), but the solution was sound. With everything in place, I released the new QFE.

The next day, I had to pull it from the website. It killed me, but it wasn't my decision. The story is short and sordid, and delves into subject areas that rightly make techies cringe - corporate legal departments and license agreements.

You see, XPE customers fall into two basic camps - OEM's and end-users (I thought they were all OEM's, but MS Legal corrected me). OEM customers use our tools to develop solutions using XPE runtimes that they then sell for profit (think someone making ATM machines). End-user customers use our tools to develop solutions using XPE runtimes that they then use internally (think a big retailer building their own in-house POS system). There's also a third class of customer, the true end-user, who gets MS OS's from licensed OEM's or via retail channels and uses it as a general purpose OS on their machine (think buying a PC from Gateway or Dell with the OS preloaded).

The problem arises because desktop QFE's are licensed for true end-users, not OEM's of any kind. OEM's require a different EULA, which allows them to redistribute the QFE to their customers. If you've ever bought a PC with the OS preloaded on it, you've bought from an OEM, and they support the system completely. For an OEM to hand out QFE's on their PC's or devices, they need a special EULA that covers redistribution of MS bits. In the retail OEM channel for the desktop, this is taken care of at a different level - unfortunately, nobody ever thought about the Embedded developer when crafting these EULA's (mainly because MS wasn't involved in the embedded market at the time).

In short, the desktop QFE's come packaged with a EULA that doesn't cover any of our embedded customers, which means you can't legally send out an image containing a raw desktop QFE. The QFE's we author that update the database do come wrapped in our own EULA tailored for XPE developers, so there's no problem there - this argument only applies to using desktop QFE's to update devices you sell, or update devices already in the field for your customers.

The first solution is rewrapping the desktop QFE with a EULA for each class customer, but that's a non-starter for me - it means having to repackage the desktop QFE's once for each unique EULA, and then making them available to only those customers covered by that EULA. This solution doesn't scale past one EULA and one customer, and it requires re-engineering the OEM secure web site to tailor downloads for different customer classes, turning what is a licensing issue into a technical one. The real solution (in my opinion) is a single EULA covering all our customers, which is something we're fighting to get. We're working on some interim solutions to get customers moving, but it's a slog - imagine trying to explain your job to your grandmother, and you approach the difficulty I'm having explaining embedded systems development to lawyers and paralegals. It's much the same when the lawyers try to explain the licensing situations to me - we speak two different languages.

For the time being, this update is limbo - I wish it weren't so, but there you have it.

I think it's important to address the story that's been circulating recently about certain ATM's being infected with Welchia.

There are patches available for XPE that address both MS03-039 and MS03-026. Applying the latest patch to your XPE devices will protect you from the vulnerabilities described by each patch. It is important to note that the ATM's described in the story above were not patched.

In short, this story isn't about XPE being vulnerable - it's no more vulnerable than Windows XP Pro, and in some cases, can even be less vulnerable. For example, some recent patches to XP Pro address vulnerabilities in Windows Help and Support Center and Messenger service. If your device does not include these features (by not including the components for these features), you aren't vulnerable to these problems, even if you don't have the patch in place.

Hope this helps - note that everything in this blog represents my own opinions and views, and nothing can or shold be construed to be official from Microsoft.

You know, the hardest part about getting QFE's out the door depends on the type of QFE we're doing...

When we're authoring an existing XP desktop QFE for XPE, the toughest part is creating the test matrix so we can test it. There are a lot of files and registry keys I need to put into the components, and all of them need to be listed - accurately - in the matrix so my testers can verify I did the work properly. That's a little time consuming, and when the format of the component changes, the matrix has to change as well.

Another part of building the matrix involves finding someone to test the fix itself. The XPE Test team can test the SLD file and the packaging, but we're not experts on individual features. Tracking down the right people to verify that the QFE works on embedded runtimes the same way it works on desktop systems is sometimes tricky - actually convincing them to take the time to do the testing can sometimes be trickier. It's getting better, as the rest of the Windows organization now knows who we are and what we do, but it's still a big part of releasing a QFE.

Now, when we're releasing a QFE for our stuff (EWF, SDI, etc.), things get a little easier - the matrix still needs to be filled out, but by the time I've got the new bits for authoring, someone on the test team has already made sure they work properly. Identifying the tester is trivial, and getting them on board to test is a minor thing. However, the hardest and most time-consuming part of any QFE we put it out still needs to be done - writing and publishing the Knowledge Base article.

Every QFE that is issued has a corresponding KB article that customers can search on. For desktop QFE's and security bulletins, the KB article has already been written, so our XPE QFE's just reference that. For fixes to XPE binaries, though, we need to write the KB's ourselves.

Writing a KB article isn't as simple as it seems. The content needs to be formatted properly, the right boilerplate text added, and the right properties added so it shows up as a QFE. Once it's written, like everything else that goes out of MS, it needs to be tested - in the case of KB articles and other written docs, it's a technical review of the material. Once that's done, it goes to an editor to be cleaned up for publishing, then published by our content team. All in all, the process can take a few weeks to do - as you can imagine, there are far fewer editors for KB articles than there are people writing them.

So, when you see an XPE QFE in the Download Center, but can't find the KB article we reference, check to see if the QFE is XPE specific. If so, it's probably because we were able to release the QFE quicker than we could get the KB article published. Most of the time, if the KB article isn't published when we release the QFE to the web, the salient content from the article is included on the Download page. And if you need more, ping up in the newsgroup.

Wow, it's been a month since my last post - guess I better start filling in some of the recent events around here.

As most of you know, the XP Embedded team has a small group of people called the Strike Team. The job of the Strike Team is to service the OS (via QFE's primarily), handle customer escalations, and work support for our Beta and other programs. Since we don't current have a Beta program, we're primarily a customer contact point and QFE generator. That was full time work for three people.

Well, in the last month, the team is now one - yours truly. The other members of the team left for greener pastures, leaving me with everything - customer escalations, marketing asks, QFE's, service packs, etc. In short, lots of work and lots of stress, not lots of time for updating blogs.

That's gonna change Real Soon Now®... We've hired on a new guy who you all will be meeting soon. He's still coming up to speed, and I'm having him concentrate on QFE's for now, but he'll be in the newsgroups and customer events soon as well. A third member is in the works, bringing the team back up to it's full complement. Which means I'll have time to work on cool new things and get back into the newsgroups.

Speaking of cool, we've implemented some new processes internally that should help decrease the delta between the desktop QFE release and our QFE release. It helps that the desktop has moved to a monthly release schedule - we're moving to the same schedule and working closer with the desktop guys so we know what they're releasing before it's released, and can do our own authoring and testing. The first results of that should be visible this week - we've got MS03-049 working through our release process now. We're also working through our backlog and should be caught up soon.

Well, that's that for now - hopefully I'll have more early next week....

OK, I promised a story about why MS03-039 came out so quickly for XPE, but other QFE's take weeks or months...

Simply put, after the Blaster worm (exploiting the vulnerability patched with MS03-026), we were pretty skittish when it came to RPC fixes. Since MS03-039 was another RCP fix, we were on it like flies on manure. Also, we had learned some lessons with 026, like where there was a test tool for the fix. With the test tool in hand and everyone sufficiently alarmed at the prospect of another Blaster, 039 was completed and released in record time.

To put it another way, everyone involved made it their number one priority. Obviously, this kind of reactive fire-drill mode doesn't scale well.

The best turn-around time I can look forward to achieving with QFE's is about one week. That includes time to author the new components, get them tested, code sign the packages, and release them to the web. There's a longer lead time needed to get bits out into the OEM channel for non-evaluation customers (check the download site - the QFE's posted there are for eval customers only, not for redistribution on shipping devices).

To cut down on that week, we're releasing a new QFE today, KB 824706, which enables you to run later Windows XP Pro QFE packages directly on XPE runtimes. XP Pro QFE's shipped after 7/6/03 will run on XPE runtimes, provided you've got the right dependencies in place. Our Desktop QFE Installer Support component adds those dependencies. Footprint hit is about 10 Mb, less if you've got a more feature rich device. Check out the web page and KB article (which may not be published yet) for more details.

There's a new version of Q824704 at the URL below:

Download details: XPE QFE Rollup, Summer 2003

Let me tell you, this one was a bit of a surprise, but looking at it in reverse, it's understandable. Here's the story...

Normally, I author all the QFE's. To do this, I grab versions of the SLD's we use to build our main database, make the modifications, and send them off for testing. This one was a little different - someone else actually made all the changes to the components. Instead of grabbing SLD files used to build the DB, she grabbed the SLD files that were provided by the other MS teams.

For better understanding, you need to know about cooked and uncooked components. When a feature team gives us a SLD file, it has the files, registry keys, and FBA resources already in it, and maybe some file dependencies. We put those SLD files through a process we call cooking, which (among other things) resolves all the file dependencies into component dependencies. Those cooked components are what we bring together to build the XPE database, and what I use to build QFE's.

Since uncooked components were used to make this particular QFE, the fact that some of those components had been QFE'd previously was missed. When we test, we test the QFE alone to make sure it doesn't screw anything up - if we tried to test every combination of QFE's together, we'd never ship anything. So, when the QFE was ready to ship, it was already conflicting with existing QFE's (Q328310 and Q814078). By conflict, I mean we had two different components with the same revision number, but different GUID's and resources. Whatever got added last won, which meant you'd be missing part of a QFE somewhere - definitely bad ju-ju.

Anyway, to fix it, I took the previously QFE'd components from Q328310 and Q814078, made the changes that were needed from Q824704, bumped the revision, and sent it off for retest and publishing. We're in the middle of implementing a new QFE process that should eliminate this problem, using components in a master XPE database as source for QFE authoring and testing. This one just fell through the cracks. On the bright side, it's the first and only screw-up of it's kind we've had, and the post-mortem on the issue was done before I could get the component re-authored and off to test.

And that's what we like to call a small part of QFE Hell (TM).

OK, there's now an XPE version of MS03-039 ready for download at the link below.

Download details: Windows XP Embedded with SP1 (824146)

And for those of you who wonder why it takes so long to get QFE's otherwise, and why this one showed up the next dat from the Pro release, that's my next story....

Heads up - there's a new RPC QFE on it's way. The XPE version is coming within a few days, but you might wanna get your desktop systems prepped now.

Download details: Security Update for Windows NT 4.0 Workstation (KB824146)

Quick update this morning - Sean Liming's new book on Windows XP Embedded is available via Barnes and Noble at the link below. It's a good book - I've read it (helped review it as it was being written), and it's a much more complete treatment of XPE than the Cseri book.
Barnes & Noble.com - Windows XP Embedded Advanced

Perhaps the best place to really start with XPE QFE's is to go over, as best I can, the QFE process from my point of view.

Most QFE's don't start with the Embedded team - they start with the Windows team. There are exceptions, but for the purpose of argument, we'll assume every QFE published for XPE starts with bits patched for the Windows XP product. I routinely author security builletins, available through the MSDN Security Bulletins page and Windows Update.

QFE packages for the desktop contain two very important things - a set of patched files, and an INF file that direcs the live installation of those files. The INF also contains registry keys that are needed by the patch. This INF tells me which files need to be updated and which registry keys need to be added to an XPE runtime to implement the QFE.

From there, I go a'hunting - I find the components that own the updated files and registry keys. In some case, no component owns the registry keys - most QFE's come with a set of tracking registry keys which need to be added to the runtime. Once I've identified the components, I find the original SLD files we used to build the database and update them.

The updating process is fairly straight-forward, but a little confusing. First, the components have certain metadata changes made to help us ID the new component as a QFE, like the name, version, revision, and additional extended properties. The repository is also changed to point to the SP1 QFE repository found in Q811279. A new package for the QFE components is created in Component Designer to give us a quick way to remove the QFE later.

Once that's done, I rename all the patched files, prefixing them with the QFE number. This is to avoid any file name collisions with the older, unpatched files already in the SP1 repository. Then I change the Source Name property of the file resources representing the patched files to reference the new name.

The new registry keys are somewhat tougher - if there is more than one component being affected by updated files, I need to pick one component to hold the new keys. Sometimes it's easy, sometimes not, but it always gets done. Updated registry keys are much easier - just change the keys in the owning component.

With all that done, it's time to package everything up into an installation package. We developed an installer for XPE QFE's that automagically imports all the components and copies the files into the SP! QFE repository. It's driven by it's own INF file, which needs to be written. The the whole thing gets tested twice - once by me, and once by our stellar Test team. We also loop in the testers for the feature being updated to help us make sure the functionality works as expected on our runtimes.

Once everything is fully tested and has everyone's OK, I digitall sign the executable package and publish it to the Microsoft Download Center. Once it's published, I send that info to our web masters for inclusion on the Microsoft Embedded web site.

That's about it - pretty dry, but necessary for a complete understanding of why it takes a while for QFE's to get out to you. I need to start with a complete QFE package for the desktop, go through some reauthoring and a lot of testing, then the publishing process which requires more verification along the way. Throw in some cross-group cooperation with other test teams, many of whom I've never met before and may not know anything about XPE, and mix in some hot QFE's that need to be done immediately that throw the whole schedule off, and you can see the difficulties.

Next time, I'll tell you what kind of stress of worm writer can induce....

Here is the first of what I hope to a set of weekly posts on the subject of Windows XP Embedded QFE's.

It's the rivetting tale of a young boy, set loose on the world with nothing but a dirty MS Natural Keyboard Pro and a talent for flipping bits, seeking his fortune in the workaday, hob-nob, up and down, in and out, business world. Using nothing but his keen eyes, freckled nose, and rather hair fingers, he captivates audiences world-wide with his nail-biting exploits, trying to feverishly patch operating systems and keep customers happy. I'm your host, Jon Fincher, Program Manager with the Windows Embedded team at Microsoft and XPE QFE Lead (a lofty title that conveys much glory, but very little booty). Try to hang on to the sides of the boat as we navigate these treacherous and exciting waters - I'm not coming back to pick up stragglers.

OK, so it's not William Burroughs, but it's a start. But why a blog? Well, I've written some columns for MSDN, which have been moderately well received. This blog is a personal venture (sanctioned by my boss, of course), and being so, I can be a little more relaxed and free flowing. Plus, in keeping with the spirit of the Internet, I figure if something is interesting to me, it must be interesting to everyone, enough to use up hard drive space on someone (else's) server. So I'm gonna write and talk and be painfully clever and exercise all the dry wit I learned watching BritComs and be wildly successful at it.... OK, maybe not, but if someone other than my immediate family members reads this blog, I'll score it as a victory.

We'll start with some meat in the next post - there are some stories to tell. Names will be changed to protect the innocent. Once again, I'm your host Noj Rehcnif - see you next time.

The jokes get better, I promise, just stay with me....