My friend Mike Hall (famous for his Windows CE insights) has a blog setup on MSDN. If you're not interested in reading my scant ravings on XPE, you might want to check out his slant on things new and exciting in the Windows CE space.

Mikehall's WebLog

OK, this isn't specific to QFE's in XPE, but it is a very cool device.

Cornice unveils tiny 2 gigabyte magnetic media "storage element"

OK, important note - this article posits a Slammer-type worm that can use the vulnerabilities patched by MS03-043 to replicate. The article uses the term "in minutes" - sensationalism? Perhaps, but perhaps not.

In any case, make sure you're devices are patched with MS03-043. You can get the XPE version here.

Flaw could unleash another Slammer | CNET News.com

Back in October, I mentioned a new QFe we had, KB824706, which would let you apply desktop QFE's directly to XPE runtimes. It's a cool thing - we had to get some DCR's into the desktop QFE installer, and add some components to support the installer, but it works very well with only a 10Mb footprint hit at a maximum (the dependent components max out at 10Mb - if you've got some of them in your image, your footprint hit is smaller). There were a few caveats we documented (stuff like EWF images, remote boot, kernel updates, etc.), but the solution was sound. With everything in place, I released the new QFE.

The next day, I had to pull it from the website. It killed me, but it wasn't my decision. The story is short and sordid, and delves into subject areas that rightly make techies cringe - corporate legal departments and license agreements.

You see, XPE customers fall into two basic camps - OEM's and end-users (I thought they were all OEM's, but MS Legal corrected me). OEM customers use our tools to develop solutions using XPE runtimes that they then sell for profit (think someone making ATM machines). End-user customers use our tools to develop solutions using XPE runtimes that they then use internally (think a big retailer building their own in-house POS system). There's also a third class of customer, the true end-user, who gets MS OS's from licensed OEM's or via retail channels and uses it as a general purpose OS on their machine (think buying a PC from Gateway or Dell with the OS preloaded).

The problem arises because desktop QFE's are licensed for true end-users, not OEM's of any kind. OEM's require a different EULA, which allows them to redistribute the QFE to their customers. If you've ever bought a PC with the OS preloaded on it, you've bought from an OEM, and they support the system completely. For an OEM to hand out QFE's on their PC's or devices, they need a special EULA that covers redistribution of MS bits. In the retail OEM channel for the desktop, this is taken care of at a different level - unfortunately, nobody ever thought about the Embedded developer when crafting these EULA's (mainly because MS wasn't involved in the embedded market at the time).

In short, the desktop QFE's come packaged with a EULA that doesn't cover any of our embedded customers, which means you can't legally send out an image containing a raw desktop QFE. The QFE's we author that update the database do come wrapped in our own EULA tailored for XPE developers, so there's no problem there - this argument only applies to using desktop QFE's to update devices you sell, or update devices already in the field for your customers.

The first solution is rewrapping the desktop QFE with a EULA for each class customer, but that's a non-starter for me - it means having to repackage the desktop QFE's once for each unique EULA, and then making them available to only those customers covered by that EULA. This solution doesn't scale past one EULA and one customer, and it requires re-engineering the OEM secure web site to tailor downloads for different customer classes, turning what is a licensing issue into a technical one. The real solution (in my opinion) is a single EULA covering all our customers, which is something we're fighting to get. We're working on some interim solutions to get customers moving, but it's a slog - imagine trying to explain your job to your grandmother, and you approach the difficulty I'm having explaining embedded systems development to lawyers and paralegals. It's much the same when the lawyers try to explain the licensing situations to me - we speak two different languages.

For the time being, this update is limbo - I wish it weren't so, but there you have it.

I think it's important to address the story that's been circulating recently about certain ATM's being infected with Welchia.

There are patches available for XPE that address both MS03-039 and MS03-026. Applying the latest patch to your XPE devices will protect you from the vulnerabilities described by each patch. It is important to note that the ATM's described in the story above were not patched.

In short, this story isn't about XPE being vulnerable - it's no more vulnerable than Windows XP Pro, and in some cases, can even be less vulnerable. For example, some recent patches to XP Pro address vulnerabilities in Windows Help and Support Center and Messenger service. If your device does not include these features (by not including the components for these features), you aren't vulnerable to these problems, even if you don't have the patch in place.

Hope this helps - note that everything in this blog represents my own opinions and views, and nothing can or shold be construed to be official from Microsoft.

You know, the hardest part about getting QFE's out the door depends on the type of QFE we're doing...

When we're authoring an existing XP desktop QFE for XPE, the toughest part is creating the test matrix so we can test it. There are a lot of files and registry keys I need to put into the components, and all of them need to be listed - accurately - in the matrix so my testers can verify I did the work properly. That's a little time consuming, and when the format of the component changes, the matrix has to change as well.

Another part of building the matrix involves finding someone to test the fix itself. The XPE Test team can test the SLD file and the packaging, but we're not experts on individual features. Tracking down the right people to verify that the QFE works on embedded runtimes the same way it works on desktop systems is sometimes tricky - actually convincing them to take the time to do the testing can sometimes be trickier. It's getting better, as the rest of the Windows organization now knows who we are and what we do, but it's still a big part of releasing a QFE.

Now, when we're releasing a QFE for our stuff (EWF, SDI, etc.), things get a little easier - the matrix still needs to be filled out, but by the time I've got the new bits for authoring, someone on the test team has already made sure they work properly. Identifying the tester is trivial, and getting them on board to test is a minor thing. However, the hardest and most time-consuming part of any QFE we put it out still needs to be done - writing and publishing the Knowledge Base article.

Every QFE that is issued has a corresponding KB article that customers can search on. For desktop QFE's and security bulletins, the KB article has already been written, so our XPE QFE's just reference that. For fixes to XPE binaries, though, we need to write the KB's ourselves.

Writing a KB article isn't as simple as it seems. The content needs to be formatted properly, the right boilerplate text added, and the right properties added so it shows up as a QFE. Once it's written, like everything else that goes out of MS, it needs to be tested - in the case of KB articles and other written docs, it's a technical review of the material. Once that's done, it goes to an editor to be cleaned up for publishing, then published by our content team. All in all, the process can take a few weeks to do - as you can imagine, there are far fewer editors for KB articles than there are people writing them.

So, when you see an XPE QFE in the Download Center, but can't find the KB article we reference, check to see if the QFE is XPE specific. If so, it's probably because we were able to release the QFE quicker than we could get the KB article published. Most of the time, if the KB article isn't published when we release the QFE to the web, the salient content from the article is included on the Download page. And if you need more, ping up in the newsgroup.

Wow, it's been a month since my last post - guess I better start filling in some of the recent events around here.

As most of you know, the XP Embedded team has a small group of people called the Strike Team. The job of the Strike Team is to service the OS (via QFE's primarily), handle customer escalations, and work support for our Beta and other programs. Since we don't current have a Beta program, we're primarily a customer contact point and QFE generator. That was full time work for three people.

Well, in the last month, the team is now one - yours truly. The other members of the team left for greener pastures, leaving me with everything - customer escalations, marketing asks, QFE's, service packs, etc. In short, lots of work and lots of stress, not lots of time for updating blogs.

That's gonna change Real Soon Now®... We've hired on a new guy who you all will be meeting soon. He's still coming up to speed, and I'm having him concentrate on QFE's for now, but he'll be in the newsgroups and customer events soon as well. A third member is in the works, bringing the team back up to it's full complement. Which means I'll have time to work on cool new things and get back into the newsgroups.

Speaking of cool, we've implemented some new processes internally that should help decrease the delta between the desktop QFE release and our QFE release. It helps that the desktop has moved to a monthly release schedule - we're moving to the same schedule and working closer with the desktop guys so we know what they're releasing before it's released, and can do our own authoring and testing. The first results of that should be visible this week - we've got MS03-049 working through our release process now. We're also working through our backlog and should be caught up soon.

Well, that's that for now - hopefully I'll have more early next week....

OK, I promised a story about why MS03-039 came out so quickly for XPE, but other QFE's take weeks or months...

Simply put, after the Blaster worm (exploiting the vulnerability patched with MS03-026), we were pretty skittish when it came to RPC fixes. Since MS03-039 was another RCP fix, we were on it like flies on manure. Also, we had learned some lessons with 026, like where there was a test tool for the fix. With the test tool in hand and everyone sufficiently alarmed at the prospect of another Blaster, 039 was completed and released in record time.

To put it another way, everyone involved made it their number one priority. Obviously, this kind of reactive fire-drill mode doesn't scale well.

The best turn-around time I can look forward to achieving with QFE's is about one week. That includes time to author the new components, get them tested, code sign the packages, and release them to the web. There's a longer lead time needed to get bits out into the OEM channel for non-evaluation customers (check the download site - the QFE's posted there are for eval customers only, not for redistribution on shipping devices).

To cut down on that week, we're releasing a new QFE today, KB 824706, which enables you to run later Windows XP Pro QFE packages directly on XPE runtimes. XP Pro QFE's shipped after 7/6/03 will run on XPE runtimes, provided you've got the right dependencies in place. Our Desktop QFE Installer Support component adds those dependencies. Footprint hit is about 10 Mb, less if you've got a more feature rich device. Check out the web page and KB article (which may not be published yet) for more details.

There's a new version of Q824704 at the URL below:

Download details: XPE QFE Rollup, Summer 2003

Let me tell you, this one was a bit of a surprise, but looking at it in reverse, it's understandable. Here's the story...

Normally, I author all the QFE's. To do this, I grab versions of the SLD's we use to build our main database, make the modifications, and send them off for testing. This one was a little different - someone else actually made all the changes to the components. Instead of grabbing SLD files used to build the DB, she grabbed the SLD files that were provided by the other MS teams.

For better understanding, you need to know about cooked and uncooked components. When a feature team gives us a SLD file, it has the files, registry keys, and FBA resources already in it, and maybe some file dependencies. We put those SLD files through a process we call cooking, which (among other things) resolves all the file dependencies into component dependencies. Those cooked components are what we bring together to build the XPE database, and what I use to build QFE's.

Since uncooked components were used to make this particular QFE, the fact that some of those components had been QFE'd previously was missed. When we test, we test the QFE alone to make sure it doesn't screw anything up - if we tried to test every combination of QFE's together, we'd never ship anything. So, when the QFE was ready to ship, it was already conflicting with existing QFE's (Q328310 and Q814078). By conflict, I mean we had two different components with the same revision number, but different GUID's and resources. Whatever got added last won, which meant you'd be missing part of a QFE somewhere - definitely bad ju-ju.

Anyway, to fix it, I took the previously QFE'd components from Q328310 and Q814078, made the changes that were needed from Q824704, bumped the revision, and sent it off for retest and publishing. We're in the middle of implementing a new QFE process that should eliminate this problem, using components in a master XPE database as source for QFE authoring and testing. This one just fell through the cracks. On the bright side, it's the first and only screw-up of it's kind we've had, and the post-mortem on the issue was done before I could get the component re-authored and off to test.

And that's what we like to call a small part of QFE Hell (TM).

OK, there's now an XPE version of MS03-039 ready for download at the link below.

Download details: Windows XP Embedded with SP1 (824146)

And for those of you who wonder why it takes so long to get QFE's otherwise, and why this one showed up the next dat from the Pro release, that's my next story....