- All articles

All articles

- WPF: Flexible Content Display With Flow Documents

- OPC: A New Standard For Packaging Your Data

- Excel Services: Develop A Calculation Engine For Your Apps

- Mobility: Make Your WPF Apps Power-Aware

- Share Code: Write Code Once For Both Mobile And Desktop Apps

- NTFS: Enhance Your Apps With File System Transactions

- C# 3.0: The Evolution Of LINQ And Its Impact On The Design Of C#

- OFFICE UI: New VSTO Features Help You Customize Word And Outlook

- Got Game?: Unleash Your Imagination With XNA Game Studio Express

- WPF: Customizing Controls For Windows Presentation Foundation

- Migration: Convert A Java Web Application To ASP.NET Using JLCA

- Collaborate: Help Teams Work Together With Web Services And Groove 2007

- Identity: Secure Your ASP.NET Apps And WCF Services With Windows CardSpace

- Aero Glass: Create Special Effects With The Desktop Window Manager

- Event Tracing: Improve Debugging And Performance Tuning With ETW

- IIS 7.0: Explore The Web Server For Windows Vista And Beyond

- .NET Security: Support Certificates In Your Applications With The .NET Framework 2.0

- Digital Media: Add Video To Controls And 3D Surfaces With WPF

- WiX Tricks: Automate Releases With MSBuild And Windows Installer XML

- ASP.NET 2.0: Manage Web Users With Custom Profile Providers

- SQL Server 2005: Regular Expressions Make Pattern Matching And Data Extraction Easier

- Vista and Office: View Data Your Way With Our Managed Preview Handler Framework

- SideShow Gadgets: Get Started Writing Gadgets For Windows SideShow Devices

- Debug Leaky Apps: Identify And Prevent Memory Leaks In Managed Code

WPF

Flexible Content Display With Flow Documents

Markus Egger

This article discusses:

- Windows Presentation Foundation

- Flow Documents

- Flow Document rendering

- Flow Document editing

This article uses the following technologies:

.NET Framework 3.0

Contents

Flow Document Basics

Viewing Flow Documents

Displaying Flow Documents

Creating Flow Documents

Exploring Layout Possibilities

Above and Beyond

Windows® Presentation Foundation (WPF) provides a great set of features. So many features, in fact, that even some very significant ones don’t get nearly the attention they deserve. A perfect example is the Flow Documents feature, which allows developers to create documents natively in WPF. In "XPS Documents: A First Look at APIs For Creating XML Paper Specification Documents" in the January 2006 issue of MSDN® Magazine, Bob Watson took a close look at XPS Documents in WPF, but Flow Documents are different. XPS (the XML Paper Specification) is geared towards printing and page-oriented content, while Flow Documents are aimed at screen reading and provide a more dynamic and arguably more sophisticated model. Flow Documents work for almost everything related to text content, from product descriptions to entire books.

The representation of text is undoubtedly one of the more important UI features. In WPF interfaces, you often use controls such as Label to display text. In many scenarios, however, you need more than a simple display of a few words. Flow Documents provide a more sophisticated approach, though they are fundamentally very simple. They define text flows in a fashion reminiscent of HTML documents, but they are more capable and provide significantly more advanced layout options.

Flow Documents are typically defined using Extensible Application Markup Language (XAML), the XML-based standard markup language. XAML is particularly intuitive for Flow Documents, mainly because of the similarity to HTML. The following Flow Document sample creates a single paragraph of text with simple bold formatting applied to a few words:

<FlowDocument

xmlns=’http://schemas.microsoft.com/winfx/2006/xaml/presentation’

xmlns:x=’http://schemas.microsoft.com/winfx/2006/xaml’>

<Paragraph>The quick <Bold>brown fox</Bold> jumps over the lazy dog.

</Paragraph>

</FlowDocument>

As you can see, the similarities to HTML are even more apparent than in other XAML UIs. The actual element names are different, but at least for simple documents, the paradigm is very similar. Flow Documents generally start with a FlowDocument root element that contains a number of blocks. Blocks are elements within the flow, typically paragraphs of text as in the example above (although there are other block types as well). Paragraphs can in turn contain other elements, such as the two bolded words in this example. Note that as with any other XAML document, the root element must have the XAML-specific namespace definitions in order to be recognized. This is an implementation detail specific to XAML and has nothing to do with Flow Documents. Note that the namespace definition is only necessary in standalone Flow Documents. (Flow Documents can be part of a larger XAML UI, in which case the UI’s root element gets the namespace definitions.)

Of course, users never see the XAML for Flow Documents (unlike HTML source that can be viewed in the browser), just as they don’t see the XAML for any other UI element. Instead, users see the final rendering of the document. For this particular example, you can see the result in a number of ways. Perhaps the simplest is to type it into XamlPad, a utility that ships with the Windows SDK (see Figure 1).

Figure 1 Very Simple Flow Document Displayed in XamlPad (Click the image for a larger view)

Of course, this is a very simple example; the definition of a document and the embedded layout can be much more sophisticated. Flow Documents support all the formatting you’d expect, such as italic, underline, font colors and typefaces, and much more. Figure 2 shows a slightly more advanced example, the result of which can be seen in Figure 3.

Figure 2 More Formatting and a Bulleted List

<FlowDocument

xmlns=’http://schemas.microsoft.com/winfx/2006/xaml/presentation’

xmlns:x=’http://schemas.microsoft.com/winfx/2006/xaml’>

<Paragraph FontFamily=”Calibri” FontWeight=”Bold” FontSize=”24”>

WPF Flow Documents</Paragraph>

<Paragraph>WPF Flow Documents are <Italic>quite sophisticated</Italic>.

They support all common layout options, as well as many you probably

do <Span Foreground=”Red”>not expect</Span>.</Paragraph>

<List>

<ListItem><Paragraph>First List Item</Paragraph></ListItem>

<ListItem><Paragraph>Second List Item</Paragraph></ListItem>

<ListItem><Paragraph>Third List Item</Paragraph></ListItem>

</List>

<Paragraph>Of course, Flow Documents also support the definition

of <Bold><Span FontFamily=”Comic Sans MS” FontSize=”24”

Foreground=”Blue”>in line</Span></Bold> font sizes and font faces.

</Paragraph>

</FlowDocument>

Figure 3 Flow Document with Slightly More Advanced Formatting (Click the image for a larger view)

This example shows several paragraphs with inline formatting. It also provides a first example of another type of block, namely a List, which, not surprisingly, contains a number of list items. Note that each list item in turn is also just a container for more blocks. So rather than simply putting text inside of a list item, I added a paragraph element to each. I could have added multiple paragraphs to each item or any other block type for that matter. This lets you create advanced layouts within a single bullet point of a list, something that is generally a problem in formats such as HTML, which only flow simple text strings into each list element.

Flow Document Basics

Now that you’ve seen some basic Flow Documents, let’s take a step back and look at some of the fundamentals again. As you’ve already seen, Flow Documents are collections of blocks. Internally, all blocks are WPF classes that derive from the System.Windows.Documents.Block class. Block in turn derives (a few steps further up the chain) from ContentElement, which is a fairly low-level class in WPF specifically optimized for document definition. This approach is somewhat similar to the controls you use to define WPF interfaces, which all derive from UIElement. Both inheritance trees are similar in concept, but not exactly the same. This means that WPF controls and blocks cannot be combined directly. For example, a button’s caption can’t be set to a paragraph of text, nor can a paragraph directly contain a button. There are some subtle differences in these controls and blocks, which result from the fact that layout within a content control and layout within a block work quite a bit differently. Luckily, the gap that needs to be bridged between the two types of WPF elements is rather narrow. Buttons, for instance, can contain TextBlock objects, which are made up of formatted text, and blocks can contain any WPF control by means of the special BlockUIContainer block class. This means that Flow Documents can contain all kinds of WPF elements (including interactive user interfaces, media, and 3D elements), and, looking at it from the other angle, Flow Documents can become part of any WPF user interface either as an advanced layout element for control contents or as a true document, such as the description of an item in a point of sale application.

The list of available blocks is theoretically open-ended, since developers can derive their own block classes and thus create their own enhancements to the document-rendering engine. This provides freedom that is not available in any other document-rendering engines that I am aware of. However, the number of blocks the average document creator is exposed to is typically limited. A list of the most important types of blocks is shown in Figure 4.

Figure 4 Important Block Types

Block

Description

Paragraph

Contains (potentially richly formatted) text.

List

Contains lists of various kinds (numbered, bulleted, and so on).

Table

Contains tables similar to those in Microsoft Word or HTML.

BlockUIContainer

Contains various UI elements that are made part of the overall flow.

Section

Contains a group of other blocks. Sections are handy for applying common attributes to a group of blocks, such as the same font attributes to multiple paragraphs.

When creating WPF Flow Documents in XAML, you really just instantiate certain types. Consider the following XAML snippet (from here on, I’ll omit the namespace definition to keep the examples simple):

<FlowDocument>

<Paragraph>Hello World!</Paragraph>

</FlowDocument>

This instantiates a FlowDocument class and a Paragraph class (which has its text set to "Hello World!"). The paragraph is added to the FlowDocument’s blocks collection. Note that as with all XAML, the element names are case-sensitive and map exactly to the classes that are available as part of the WPF. You could also create the same document programmatically, like so:

FlowDocument doc = new FlowDocument();

Paragraph para = new Paragraph();

para.Inlines.Add(“Hello World!”);

doc.Blocks.Add(para);

Of course, this is much less intuitive than the declarative approach XAML provides, so the programmatic approach is only taken in special scenarios. (I sometimes use this approach when I need to create a richly formatted report that is more of a real document, rather than the tabular output created by many reporting engines.)

In many scenarios, paragraphs have richly formatted content themselves, which is also done by instantiating classes, like this:

<Paragraph>Hello <Bold>World!</Bold></Paragraph>

Here, the paragraph contains two segments of text—"Hello" (which uses the default format) and "World!" (which is bold). This is interesting because it means that this XAML doesn’t just instantiate a paragraph and set its text as a simple string; instead it creates a paragraph with two child segments, each of which contains text with different formatting. In WPF, these segments are called inlines. Just like a Flow Document can contain multiple blocks of various types, paragraphs can contain inlines of various types. Inlines come in a number of variations. Some inlines are so-called Spans, which represent a segment of text with certain formatting options applied. The Bold element used in this example is a special case of a Span, with the default font weight set to bold. Another type of an inline is a Run, which is a segment of text with default formatting. Therefore, the XAML above is really just shorthand for this:

<Paragraph>

<Run>Hello </Run>

<Bold>World!</Bold>

</Paragraph>

Of course, it is much more convenient that you do not have to define each inline in XAML, but if you were to create the same example programmatically, it is important to understand the concept of inlines, since they cannot be omitted in code. Here is the programmatic equivalent of the previous two XAML examples:

Paragraph para = new Paragraph();

para.Inlines.Add(new Run(“Hello “));

Bold b = new Bold();

b.Inlines.Add(“World!”);

para.Inlines.Add(b);

Bold is a special version of a Span, with its default font weight set to bold; the Bold type is subclassed from Span and overrides the FontWeight property. There are similar special Spans, such as Italic and Underline. However, these special Spans are not strictly needed, because you could also use a default Span and set the appropriate properties instead:

<Paragraph>Hello <Span FontWeight=”Bold”>World!</Span></Paragraph>

Of course, the ability to directly specify attributes such as bold and italic by wrapping a certain section of text into Bold and Italic tags is very convenient and intuitive, so it is much more common to use <Bold> than <Span FontWeight="Bold">. Nevertheless, the <Span> element is very useful since there are numerous properties that can be set beyond simple font bolding, and most of those formatting options do not have individual Span types. In fact, for many very common formatting options, there is no special Span. A typical example is setting font faces. Unlike HTML, Flow Documents do not have <Font> elements. Instead, fonts are set like so:

<Paragraph>Hello <Span FontFamily=”Comic Sans MS” FontSize=”24”>

World!</Span></Paragraph>

Many properties, such as FontFamily, can be found consistently on all Flow Document classes. For instance, if you wanted to set the font for a complete paragraph rather than just an inline, you can do it without a Span:

<Paragraph FontFamily=”Comic Sans MS” FontSize=”24”>Hello World!</Paragraph>

There are also inlines other than Spans and Runs. Here some of the other more interesting inlines:

Figure Figures are somewhat unusual inlines, because they contain blocks. Thus, in a way, Figures are almost like mini Flow Documents within Flow Documents. Figures are often used for advanced layout features, such as images within paragraphs where the normal text flows around the image.

Floater Floaters are lightweight figures. They do not support all the figure positioning options, but if all you need is the ability to do simple alignment outside the standard paragraph alignment, floaters may be useful.

LineBreak LineBreak elements do exactly what their name suggests: they introduce line breaks within paragraphs.

InlineUIContainer The InlineUIContainer is the inline equivalent of the BlockUIContainer. If you need to combine any sort of WPF control with your other inlines (such as having a button flow inside a paragraph’s text), the InlineUIContainer is what you need.

Figures are used all the time in Flow Documents (as are LineBreaks, but they hardly require a detailed discussion). The following example uses a figure to display an image as part of a larger flow:

<Paragraph>

<Figure Width=”200”>

<BlockUIContainer>

<Image Source=”Pictures\Humpback Whale.jpg” />

</BlockUIContainer>

<Paragraph Foreground=”Blue” FontFamily=”Consolas”>

The Whale</Paragraph>

</Figure>

The quick brown fox jumps over the lazy dog. The quick brown...

</Paragraph>

Note that there is no Image block in WPF Flow Documents. Instead, images are embedded as BlockUIContainers with standard WPF Image controls. (The same approach is used for content such as videos or interactive 3D models inside of Flow Content). Figure 5 shows the rendering of a document similar to this one.

Figure 5 Text Flows around Image and Caption (Click the image for a larger view)

Viewing Flow Documents

You have now seen how to create some simple Flow Documents and view them in XamlPad. What I’ve ignored so far is how you would view Flow Documents in the wild. After all, you wouldn’t expect your users to bring out XamlPad and paste the document’s XAML. One way of viewing a XAML Flow Document is to save it as a file with a .xaml extension and double-click it in Windows Explorer. This launches the default application associated with XAML files (typically Internet Explorer®), which shows the document. Figure 6 shows the result.

Figure 6 XAML Flow Document in Internet Explorer (Click the image for a larger view)

The fact that Internet Explorer (and other browsers) can display XAML content is of particular interest, since this is your ticket to displaying Flow Documents as part of your Web applications. In other words, if you upload XAML Flow Documents to your Web server and someone browses to the file, he will see something similar to Figure 6 (assuming the user has the Microsoft® .NET Framework 3.0 installed). Of course, this also works dynamically. If your ASP.NET Web application (or any other server-side technology) generates a XAML Flow Document on the fly and returns it as its output (and assuming the content type is set appropriately to "application/xaml+xml"), the user will see a Flow Document as part of your Web application, which of course is extremely useful in many scenarios. Figure 7 shows a simple ASP.NET page that generates a Flow Document.

Displaying Flow Documents

You may have noticed that whenever a Flow Document is displayed (either in the browser or in XamlPad), there seems to be a tad more in the display than just the document itself. In particular, there are controls rendered along the bottom of the document. As you can see in Figure 8, by default Flow Documents are rendered by means of a FlowDocumentReader control, which provides a set of standard features such as zoom, paging, various view modes, and even a find feature. As it turns out, Flow Documents need to be hosted by some sort of control that is capable of displaying them. The default viewer for Flow Documents is a FlowDocumentReader control, which is instantiated automatically unless you explicitly use a different control. WPF currently provides three different controls for viewing Flow Documents:

Figure 8 Control Buttons in the FlowDocumentReader Control (Click the image for a larger view)

FlowDocumentScrollViewer This control displays documents in a continuous flow with a scrollbar, similar to Web pages or the Web Layout in Microsoft Word. Figure 9 shows a document in a scroll viewer.

Figure 9 Using the FlowDocumentScrollViewer Control (Click the image for a larger view)

FlowDocumentPageViewer This control displays Flow Documents in individual pages, with page flipping instead of scrolling. This is similar to the Full Screen Reading mode in Word. Figure 10 shows a page viewer. There the document from Figure 9 is rendered in a FlowDocumentPageViewer control and the scrollbar is replaced by a paging mechanism. The simple flow layout approach has been replaced with an advanced, multi-column, paged layout.

Figure 10 Using the FlowDocumentPageViewer Control (Click the image for a larger view)

FlowDocumentReader This control combines the scroll viewer and the page viewer and allows the user to switch between the two approaches. This is the default control used for Flow Documents, and it is often a great choice for applications that feature sophisticated text display. In Figure 11, the same document shown in Figures 9 and 10 was rendered in a FlowDocumentReader, which combines the scroll viewer and page viewer approaches. In addition, it enables the search feature which is hidden by default in the other controls (the other viewers do support find, and it can be shown either by executing the ApplicationCommands.Find command or by using Ctrl+F from the keyboard). The reader control also supports a multi-page view, which slightly alters the page-based rendering and the way columns and figures are rendered.

Figure 11 Using the FlowDocumentReader Control (Click the image for a larger view)

The control you choose will depend on your situation, though the FlowDocumentReader is compelling for all but the most basic uses. It is versatile and powerful and it supports paged layouts, which is superior to scrolling in many scenarios. A detailed discussion of the topic is beyond the scope of this article, but it turns out that scrolling and associated effects, such as doubling, is one of the main reasons people prefer print over digital text. The paged approach is more natural in many situations and should help to make digital reading more widely accepted.

So how do you define which control to use? A simple—though rather brute-force—approach is to add the desired control to the document’s XAML:

<FlowDocumentScrollViewer

xmlns=’http://schemas.microsoft.com/winfx/2006/xaml/presentation’

xmlns:x=’http://schemas.microsoft.com/winfx/2006/xaml’>

<FlowDocument>

<Paragraph>The quick <Bold>brown fox</Bold> jumps over the lazy

dog.</Paragraph>

</FlowDocument>

</FlowDocumentScrollViewer>

In this example, the document root has been made a FlowDocumentScrollViewer tag. This means you are not just defining a pure document anymore. Instead, you are defining a complete XAML interface, which happens to be using a scroll viewer as its root. The content of the scroll viewer is the Flow Document from the very first example. (Note that the namespace definitions are now with the scroll viewer tag, rather than the Flow Document tag). Figures 9 through 11 were created using this approach, with different viewer controls used as the root element

Why do I call this a brute force approach? Because, from an architectural point of view, it has some problems caused by mixing the user interface definition with its actual data. It is much more desirable to keep the document separate from its interface. Mixing the reader with the document is a little like creating a SQL Server™ table and somehow defining that it can only be displayed in a Windows Forms DataGrid. There are several ways of keeping the document separate from the UI definition. If you want to display Flow Documents as part of a Web application using the demonstrated ASP.NET approach, you can define the ASP.NET page with the desired viewer control and simply merge in the actual content (which is stored separately, perhaps in a database) using standard ASP.NET code.

In a typical WPF application, on the other hand, you can simply define your user interface using standard WPF, Windows, and XAML Browser Application (XBAP) approaches, and then load your document dynamically. Figure 12 shows a simple example that uses a fictitious library of my articles, which are displayed in a listbox in the top-left corner. When the user selects an article from the list, the document is loaded dynamically into the Flow Document Reader control that takes up most of the form. Note that standard WPF techniques, such as alpha blending, work in this setup. You’ll notice that the actual Flow Document is semi-transparent and the photo of me in the background shines through Also note that the app uses a listbox, imagery, a label, and a FlowDocumentReader control to create a library of fictitious articles.

Figure 12 Using Listbox, Image, Label, and FlowDocumentReader Controls (Click the image for a larger view)

The trickiest part of this example is loading the actual document into the viewer control. This is done by means of the System.Windows.Markup.XamlReader class, which allows for the dynamic loading of any XAML content, including, but not limited to, Flow Documents. Here is the line of code I wired up to the selection change event of the listbox:

documentReader.Document =

(FlowDocument)XamlReader.Load(

File.OpenRead(fileName));

The Load method returns an object, since the root element in the XAML file could represent many different types. In my example, I know that the return value is a FlowDocument, so I simply perform a cast and assign that document to the FlowDocumentReader control’s Document property (I named the control instance documentReader in this example). Keep in mind that this is just an example. Production-quality code definitely needs some error handling here as well.

Note that all the things you know about WPF apply to this example. For instance, the reader controls are just standard WPF controls that support styling. This means you can completely change the appearance of all UI elements such as the zoom bar, the view mode switches, or the paging controls. (The only element you have limited control over is the search box, though you do not have to use it at all if you don’t like it.)

Also, while my example shows a Windows-based application, the same application could be deployed as an XBAP and run inside a Web browser (again, of course, assuming the user has the .NET Framework 3.0 installed). Note that Microsoft Silverlight™ (formerly code-named "WPF/E") is not enough as Silverlight supports only a subset of WPF and does not support Flow Documents.

Creating Flow Documents

How are Flow Documents authored? Of course developers can always author Flow Documents using low-level tools such as XamlPad. However, in real-life settings, this is unlikely. Typically, Flow Documents are either created using WYSIWYG editors or by means of a content transformation from an existing document format. Since Flow Documents can be defined in XAML, it is particularly easy to convert existing XML content. However, it is also possible to convert HTML and Word documents with a reasonable amount of effort (although coding is required, as no out-of-the-box tools have emerged for this as of yet).

For WYSIWYG editing, WPF provides a ready-made control. The WPF RichTextBox control can edit XAML Flow Documents natively. The control’s name incorrectly suggests that it is made for Rich Text Format (RTF). Though it supports RTF, this control really operates mostly on FlowDocuments. In fact, the control really mirrors the Flow Document view controls, except that it supports editing. Some would even argue that the RichTextBox control should be considered another way of displaying Flow Documents.

Type the following example into XamlPad to see the RichTextBox control in action:

<RichTextBox

xmlns=’http://schemas.microsoft.com/

winfx/2006/xaml/presentation’

xmlns:x=’http://schemas.microsoft.com/winfx/2006/xaml’>

<FlowDocument>

<Paragraph>

The quick brown fox jumps over the lazy dog.

</Paragraph>

</FlowDocument>

</RichTextBox>

Just like the reader controls, the RichTextBox has a Document property, which you can automatically populate with a Flow Document in this session. This actually creates a UI that looks very similar to the FlowDocumentScrollViewer control, except that the text is editable. Note that this textbox control always handles Flow Documents in a scroll fashion. There is no way to edit Flow Documents in a RichTextBox in paged or multi-column mode. However, the result of the editing operation is a standard Flow Document that can be displayed in any of the viewer mechanisms you’ve seen already, including multi-column and paged modes.

One of the features of RichTextBox worth mentioning is the integrated spell checking. You can enable it like so:

<RichTextBox SpellCheck.IsEnabled=”true”>

<FlowDocument>...</FlowDocument>

</RichTextBox>

Figure 13 shows the spell checker in action.

Figure 13 RichTextBox Control with Spell-Checking (Click the image for a larger view)

The only tricky part about using the control is loading and saving. In most scenarios, you probably won’t code the content of the RichTextBox into the UI XAML as in the earlier example. Instead, you will load and save the document dynamically. The load operation for text in a RichTextBox is identical to loading a Flow Document for a viewer control (see above). Saving a document is essentially the opposite: you take the document object and serialize it back to XAML, which can be done like so:

System.Windows.Markup.XamlWriter.

Save(richTextBox.Document)

This returns the XAML as a string, which you can then store to a file or a database or use any other way you can imagine.

The RichTextBox is very handy, but a word of caution is in order. While Flow Documents represent the most sophisticated technology available for rendering on-screen documents, the RichTextBox control is nowhere near as sophisticated. It is great for editing small documents and snippets of text, but you won’t be writing a book, magazine, or marketing brochure with it. For such long formats, it renders too simplistically, since it doesn’t support anything but a scroll layout (which means that there is no good visual way to create the advanced layouts I will discuss shortly). Also, the approach to saving documents is often unsatisfactory. The XmlWriter class simply uses a live, in-memory document and turns it into XAML, but unfortunately, it isn’t aware of many concepts that are important for large-scale Flow Document efforts, such as styles. As a result, the XAML faithfully preserves the look of the document, but it is often nasty and very large. The RichTextBox control is certainly useful, but don’t expect it to be a desktop publishing solution for on-screen content (although such an app is sorely needed).

Exploring Layout Possibilities

Now that you understand how to author and view Flow Documents, let’s return to the documents themselves and look at a few more features. Flow Documents are very sophisticated, and exploring all available features is beyond the scope of this article, but there are a few more things I would like to discuss.

One feature that always amazes me is called optimal paragraph. When enabled, this feature aims to distribute white space as evenly as possible within a given paragraph, resulting in a much improved reading experience. Optimal paragraph works particularly well in combination with hyphenation, another built-in feature that (surprise, surprise), performs on-the-fly hyphenation of either entire Flow Documents or individual paragraphs.

Turning optimal paragraph and hyphenation on is a straightforward operation:

<FlowDocument IsOptimalParagraphEnabled=”true”

IsHyphenationEnabled=”true”>

Figure 14 shows the same document rendered with these features enabled and disabled. The difference between the two versions is subtle, but significant. Note that the left version appears much calmer, mainly due to more evenly distributed but also overall reduced white space between words. Especially when reading large amounts of text on screen, this seemingly small difference is of crucial importance.

Figure 14 Optimal Paragraph and Hyphenation (Click the image for a larger view)

As you’ve seen already, the FlowDocumentReader control takes a multi-column approach to text rendering. This is another very important readability feature because people do not enjoy reading a single line across the entire width of a wide-screen display. The width of the actual columns varies depending on a number of factors, such as total available width for content display, zoom factor, and defined column width. By default, a Flow Document’s column width is 20 times the font size, which with the default font size is approximately 300 device-independent pixels (3 1/8 inch on a size-accurate display). You can override this default setting easily:

<FlowDocument ColumnWidth=”400”>

This results in columns with a width of roughly 400 pixels. However, there are other factors that impact the actual width. If the zoom is at 50 percent for instance, the actual column width is only 200 pixels. Also, the column width as it stands is to be seen more as a minimum column width. This means that if the total width available is 900 pixels, the rendered result contains two columns that are wide enough to fill up the entire 900 pixels, making each column wider than the defined 400 pixels. This is usually desirable as it results in a very appealing rendering. However, if you don’t like that behavior and want columns that are truly 400 pixels wide, you can make sure that the column widths are not flexible:

<FlowDocument ColumnWidth=”400” IsColumnWidthFlexible=”false”>

Now, all the columns are exactly 400 pixels wide (at 100 percent zoom), and remaining space is just left as white space.

Another column-related setting you may want to play with is the gap between columns. This can be adjusted through the ColumnGap property (the setting is also based on device-independent pixels):

<FlowDocument ColumnGap=”25”>

A related setting is the column rule, which allows for the definition of a visual element between the columns. Consider this example (the result of which can be seen in Figure 15):

Figure 15 Flow Document with a Simple Rule between Columns (Click the image for a larger view)

<FlowDocument ColumnRuleWidth=”5” ColumnRuleBrush=”Red”>

Of course in many publications, documents are not just laid out in simple columns. Often, other elements exist that are taken out of the regular flow. You have already seen such examples with the images placed inside documents. Figure 12 shows an arrangement that is popular with graphic artists. The image sits between two columns, with the text flowing around it, putting the image squarely in the middle of the content without interrupting the flow much in either column. This is a common layout choice that simply hasn’t been available in dynamic on-screen reading environments I’m aware of prior to Flow Documents.

The key to the creation of such layouts is the figure block, which allows for the definition of content that does not flow with the rest of the document. Putting an image inside of a figure tag is one example, but there are many other uses for figures. For instance, you could use a figure to define a heading that stretches across the entire width of the document:

<Paragraph>

<Figure HorizontalAnchor=”ContentLeft” VerticalAnchor=”ContentTop”

Width=”1Content”>

<Paragraph FontSize=”36” FontWeight=”Bold”>Go With

The Flow</Paragraph>

</Figure>

Windows Presentation Foundation in Windows Vista provides a great set

of features.

</Paragraph>

In this code the figure contains another paragraph, which is the text used as the headline. Note that there are some handy properties you can use to create advanced, flexible documents. Consider the width of the figure, for instance. Rather than setting the width to a certain number of pixels, I set it to be exactly the width of the content, which will automatically adjust the figure’s width to however wide the entire content is.

Take a look at Figure 16. There you’ll notice that the heading (positioned through a figure) is set to span across the width of the entire content, which pushes all four columns down. The image is horizontally and vertically anchored to the page center.

Figure 16 Heading Spans Four Columns (Click the image for a larger view)

Note that figures with a width relative to the content do not always have to be as wide as the content. The following example for instance sets the width of the figure to be 75 percent of the width of the content:

<Figure Width=”0.75Content”>

The width can also be relative to other things, for instance, to the width of columns. The following example figure is always two columns wide (unless only one column is displayed, in which case the width is reduced to one column):

<Figure Width=”2Column”>

Of course, figure heights can be defined in a similar fashion (although often figures just grow vertically with the content).

Another important aspect is the position of the figure. In the code snippet, it is set to be horizontally anchored to the left and vertically to the top. This means that the figure appears at the top-left corner of the current page of content, regardless of where it is actually defined. In this example, the figure was defined as the first element of the document anyway, but even if there would have been paragraphs before that heading, it would have been moved up and to the left due to these settings. The photos in Figure 12 and Figure 16 have been moved between the columns in a similar fashion, by setting its horizontal anchor to "PageCenter". (The available property values for all these settings can be found in the WPF documentation).

You may have noticed that there was a lot of manual coding. For instance, whenever the font face needs to be changed, you add that information to a block or an inline. So far, this has not been a huge problem since most of the examples were small. However, if you have a 50-page chapter of a book and would like to change the font in every paragraph, then doing so manually every time becomes tedious. Luckily, there is a better approach: like everything else in WPF, Flow Documents support styling. Styles can be defined as named resources in the actual Flow Document. Here is a style that defines font information:

<FlowDocument>

<FlowDocument.Resources>

<Style x:Key=”MyStyle”>

<Setter Property=”TextElement.FontSize” Value=”12” />

<Setter Property=”TextElement.FontFamily” Value=”Bodoni MT” />

</Style>

<FlowDocument.Resources>

...

</FlowDocument>

The style can then be applied to paragraphs (and other elements) in the following fashion:

<Paragraph Style=”{StaticResource MyStyle}”>The quick... </Paragraph>

Due to the nature of Flow Documents, styles are very commonly used. I recommend that for anything but the simplest scenarios, you define most of your formatting options in styles, rather than with attributes on individual inlines. Styles keep your documents compact and also make them easier to maintain.

Above and Beyond

I hope this article provided you with a bit more than just a fundamental understanding of Flow Documents and their capabilities, and piqued your interest. There are many more advanced features that are definitely worth looking into, including sophisticated styling of viewer controls, subclassing and extending documents, blocks, and inlines, digital rights management, the text and ink annotation feature, and advanced font formatting.

Markus Egger is an international speaker and the publisher of CoDe Magazine. Markus is also the President and Chief Software Architect of EPS Software Corp., consulting for object-oriented development, Internet development, B2B, and Web services. See his blog at www.markusegger.com/blog.

OPC

A New Standard For Packaging Your Data

Jack Davis and Andrey Shur

This article discusses:

- How content is organized in a package

- Parts and relationships

- Addressable URIs

- Creating your own OPC file format

This article uses the following technologies:

Open Packaging Conventions, .NET Framework 3.0

Code download available at: Packaging2007_08.exe (984 KB)

Browse the Code Online

Contents

File Format Organization

Packages

Package Parts

Package Relationships

Package Digital Signatures

Package Properties

Creating and Accessing Packages and Parts

Creating and Accessing Relationships

Pack URI

Authentication and Validation

Rights Management and Encryption

Packaging a Web Page

Exploring a Package

Reading a Package

Signing and Validating a Package

Designing a File Format

M any applications integrate content with various additional resources. A Web browser, for example, displays a page that integrates HTML, image files, style sheets, and other types of content. Similarly, a word processor builds a document that combines text, style definitions, image files, and other elements. For the most part, applications use one of two approaches to organize the content: a flat-file organization where content is stored as separate files organized on disk, or binary container files where all the content is packaged in a single custom file. More and more, applications are tending towards the latter.

In the move toward open standards, a new file packaging technology has evolved as part of the 2007 Microsoft® Office System Open XML specification that was recently approved by the ECMA International standards organization. An underlying component of this standard is Open Packaging Conventions (OPC), which defines a structured means to store application data together with related resources using a standard ZIP file. This new packaging technology is already being used in several Microsoft products, including the 2007 Office System applications. The XML Paper Specification (XPS), which defines the new print-spool and document presentation format for Window Vista™, also implements the storage and transport of high-fidelity documents based on OPC. See the "Online Resources" sidebar for more information about the technologies discussed in this article.

So what makes the portable container technology offered by Open Packaging Conventions different? Since it is an open standard, OPC provides a container technology you can use without having to code your own custom binary container files. And it supports a number of enhanced features, including content addressable URIs, MIME types, relational structuring, and authentication and validation. With the Microsoft .NET Framework 3.0, the packaging APIs also offer options for encryption with rights management. Moreover, because they adhere to an open standard, package-based files can be accessed through high-level services such as workflow applications and virus scanners.

API support for the ECMA Open Packaging Conventions is built into Windows Vista and included as part of the .NET Framework 3.0 for use with Windows® XP and Windows Server® 2003. In this article, we’ll examine the new standard, showing how you can use the .NET Framework 3.0 APIs to organize your application’s storage of multiple data streams in a single portable package.

File Format Organization

Microsoft Word 2007 and Excel® 2007 both use Open Packaging for document storage, however, they use different schemas and file organizations. The specific organization of a package’s contents defines its format, which is typically reflected in the extension (like .docx or .xlsx).

Using packaging, you can define your own file organization, filename extension, and file type association for your application. As an example, we’ll create our own custom package type, .htmx, that will store a Web page along with the local files it depends on (style sheets, scripts, image files, and so on).

Packages

A Package is the basic storage unit of the Open Packaging Conventions standard. In the implementation for the .NET Framework 3.0, System.IO.Packaging.Package is defined as an abstract class from which specific physical implementations are derived. The default and primary physical implementation of a Package in the .NET Framework 3.0 is the ZipPackage derived class that uses a standard ZIP archive (see Figure 1). The following code shows the basic steps used to create and open a new package:

Figure 1 File Formats Using Open Packaging Conventions

using System.IO.Packaging;

...

// Path and name for the package file.

string packageFile = @”C:\webpages\packaging.htmx”;

// Create and open the specified package file for writing.

// (The ‘using’ statement ensures that package is

//  closed and disposed when it goes out of scope.)

using (Package package = Package.Open(packageFile, FileMode.Create))

{

... // Store content files as parts in the package.

}

Open Packaging uses ZIP files because they are a known industry format that is easy to work with, inspect, and access. In fact, you can take a package-based file (a .docx file, for example), rename it with a .zip extension, and access its contents using a standard ZIP utility, such as the Compressed Folders feature built into Windows Explorer.

A package stores two basic elements: PackageParts (which are referred to simply as parts) and PackageRelationships (which are referred to as relationships). Parts represent the actual content being packaged (see Figure 2). Higher-level components are provided by building upon the basic part and relationship elements. These include PackageDigitalSignatures, PackageProperties, and components to support encryption and rights management.

Figure 2 Basic Elements of a Package

Package Parts

A part is a data stream, analogous to a file in a file system or ZIP archive. Parts can hold any type of data—binary, text, images, and so on. Figure 3 outlines a sample package that contains several parts with different types of content.

Figure 3 Packaging Various Forms of Content

When a part is stored in a package, it is defined with a unique URI-formatted PartName along with a MIME ContentType. Part names start with a forward slash, defining an absolute path based from the package root. You can easily add or remove a part to or from the package, write to a part, update it, or read the part from the package, simply by referencing its unique URI part name.

While the PartName URIs can appear to represent a folder hierarchy, there are actually no separate folders in a package. A package cannot contain empty folders and there is no concept of a default current directory. Because there can be no empty folders, there is no need to create, delete, or otherwise manage folders separately. If the package is opened in a graphical ZIP utility, the utility will often display the paths of the part names in a folder representation. Similarly if the package is unzipped, the utility will recreate the apparent folder hierarchy. The folder representation, however, is an artifact of the ZIP utility and is not inherent in the package itself.

The code sample shown in Figure 4 illustrates the process of creating a part and then writing data from a disk file to the part stored in the package.

Package Relationships

A relationship defines an association between two items: a specified source and a specified target. The source of a relationship can be either the package itself (a package-level relationship) or a specified part in the package (a part-level relationship). The target of a relationship can be either a specified part in the package or a specified external resource. An external resource can be another package, a part within another package, or any other type URI-addressable data entity. Thus, there are four combinations of relationships that can be defined:

- Between a package (source) and a specified part (target) contained in the package.

- Between a package (source) and a specified resource (target) external to the package.

- Between a specified part (source) within the package and a second specified part (target) also contained within the package.

- Between a specified part (source) within the package and a specified resource (target) external to the package.

The source part is considered the owner of the relationship. If the source part is deleted, all the relationships owned by that part are also deleted. Relationships are defined and stored separately in special relationship parts in the package. Note that creating or deleting a relationship does not physically alter the source or the target elements in any way.

Relationships offer a number of advantages:

Relational Structuring Relationships allow associations between content and resources to be determined without needing to read or parse content streams. Preprocessing tasks can aggregate, cache, and access needed resources without having to understand the details of the content in use.

Content Discoverability Relationship types combined with part MIME content types enable quick discoverability of the structure, associations, and content of the parts contained in the package.

Schema Independence Content and resource associations defined by relationships are schema independent. This is especially helpful for content defined in XML markup where parsing would typically require knowledge of the specific schema in use.

Reference Integrity For XML-based content, you can specify resources in markup by a reference to a relationship ID, rather than embedding resource URI references directly in markup. This simplifies the XML markup and eliminates the possibility that content references and relationship references might become out of sync.

Figure 5 illustrates a sample HTML page packaged with its style sheets and image file resources, as well as the relationships that define the associations. A package-level relationship is used to identify the URI of the PackagePart that contains the base HTML content for the page. By querying for the package’s root-html relationship, the URI of HTML Web page part can be directly determined. With the HTML part as the source, part-level relationships for the style sheet, scripts, and image resources can be easily located and resolved. (For each resource referenced in the HTML page, there is a corresponding part-level relationship.)

Figure 5 Sample HTML Page Package (Click the image for a larger view)

The code shown in Figure 6 demonstrates how to create a package, create and add a part, and then create a package relationship.

Figure 6 Create a Package, Part, and Package Relationship

using System.IO.Packaging;

// Path and name of the package file.

string packageFile = @”C:\webpages\packaging.htmx”;

// Path and name of the file to store in the package.

string partFile = @”C:\inetpub\wwwroot\packaging.htm”;

// Part name URI, MIME content type, and compression option.

Uri partNameUri = PackUriHelper.CreatePartUri(

new Uri(“/packaging.htm”, UriKind.Relative));

string partType = “text/html”; // MIME type

CompressionOption compression = CompressionOption.Normal;

for the package relationship type.

string packageRelationshipType = “http://schemas.openxmlformats.org/” +

“package/2007/relationships/htmx/root-html”;

// Create and open the package file.

using (Package package = Package.Open(packageFile, FileMode.Create))

{

// Create the part.

PackagePart part = package.CreatePart(

partNameUri, partType, compression);

// Write the data from the file to the part.

using (FileStream fileStream = File.OpenRead(partFile))

{

CopyStream(fileStream, part.GetStream()); // Copy data to part.

}

// Create a package-level relationship to the Web page part.

package.CreateRelationship(

part.Uri, TargetMode.Internal, packageRelationshipType);

}

Using relationships makes it easy to open and discover the associations between the content and resource items stored in the package. Figure 7 is an example of opening a package and then using the root-html package relationship to locate the root content part contained in the package. The root part can then be queried to return relationships that identify additional resources that the root element needs.

Figure 7 Locating the Root Content Part in a Package

using System.IO.Packaging;

// Path and name of the package file.

string packageFile = @”C:\webpages\packaging.htmx”;

// Name for the root package relationship type.

string rootHtmlRelationshipType = “http://schemas.openxmlformats.org/” +

“package/2007/relationships/htmx/root-html”;

string resourceRelationshipType = “http://schemas.openxmlformats.org/” +

“package/2007/relationships/htmx/required-resource”;

// Open the package for reading. - - - - - - - - - - - - - - -

using (Package package =

Package.Open(packageFile, FileMode.Open, FileAccess.Read))

{

// A package can contain multiple root items, iterate

// through each.  Get the “root-html” package relationship.

foreach (PackageRelationship relationship in

package.GetRelationshipsByType(rootHtmlRelationshipType))

{

// Get the part referenced by the relationship TargetUri.

PackagePart rootPart = package.GetPart(relationship.TargetUri);

// Open and access the part data stream.

using (Stream dataStream = rootPart.GetStream())

{

... //  Access the root part’s data stream.

}

// A part can have associations with other parts.

// Locate and iterate through each associated part.

// Iterate through each “required-resource” part.

foreach (PackageRelationship resourceRelationship in

rootPart.GetRelationshipsByType(

resourceRelationshipType))

{

// Open the Resource Part and write the contents to a file.

PackagePart resourcePart = package.GetPart(

resourceRelationship.TargetUri);

... // Party with the resource part.

}

}

}

Package Digital Signatures

Built upon a composition of parts and relationships, a package can also contain PackageDigitalSignature items (digital signatures for short). A digital signature uses an X.509 certificate to securely sign parts and relationships in the package. The signature provides two features. It identifies and authenticates the individual or entity that has signed a given set of parts and relationships, and it validates that the signed parts and relationships have not been modified.

The digital signature does not prevent a part or relationship from being changed, but a validation check against the signature will fail if the signed item has been altered. The application can then take appropriate action—for example, prevent the part from being opened or notify the user that the data is not secure.

The diagram in Figure 8 illustrates a package that contains parts and relationships along with a digital signature that has been used to sign specific items. If any of the signed items are modified in any way, validation against the digital signature will fail.

Figure 8 Signing Parts and Relationships (Click the image for a larger view)

For more information about package digital signatures, see the MSDN® article "The Digital Signing Framework of the Open Packaging Conventions".

Package Properties

A package can also store a set of public metadata as PackageProperties. This set includes 16 common properties that describe the package and its contents. Use of these properties is optional. Figure 9 summarizes the properties available with a package.

Figure 9 Properties Available with a Package

Property

Description

Category

A categorization of the package content (example: Letter, Proposal, Resume)

ContentStatus

The status of the package content (example: Draft, Reviewed, Final).

ContentType

The type of content that is contained in the package (example: Whitepaper, Bulletin).

Created

The date and time the package was created (see also Modified).

Creator

The name of the individual or entity that created the package (see also LastModifiedBy).

Description

A description of the package content.

Identifier

A unique identification assigned to the package and its content.

Keywords

A delimited set of keywords to support search and indexing.

Language

An RFC 3066 tag that identifies the language for which the package content is written.

LastModifiedBy

The name of the individual or entity that last modified the package (see also Creator).

LastPrinted

The date and time the package content was last printed.

Modified

The date and time the package content was last modified (see also Created).

Revision

The revision number of the package content (see also Version).

Subject

The topic of the package content.

Title

A name given to the package content.

Version

The version number of the package content (see also Revision).

Creating and Accessing Packages and Parts

You can use the Package.Open method to create a new package or open an existing package. Package.Open provides overloads that instantiate packages, either through a specified filename or by a given file stream. In addition, Package.Open includes settings that let you specify options for file mode (System.IO.FileMode) and file access (System.IO.FileAccess).

After a package is instantiated and opened, you can use the part management methods to create and access the parts contained in the package (see Figure 10). After obtaining a part instance—either through the CreatePart method or the GetPart method—you can call the part GetStream method to return the stream for reading or writing the content data. We discuss how to access the data content of a part in more detail later.

Figure 10 Package Part Methods

Package Method

Description

CreatePart

Creates a new part in the package.

DeletePart

Deletes a specified part from the package.

GetPart

Returns the part with a specified name.

GetParts

Returns a collection of all the parts contained in the package.

PartExists

Determines whether a specified part exists in the package.

Creating and Accessing Relationships

When using the System.IO.Packaging APIs, you can create package-level and part-level relationships through the CreateRelationship methods of the Package and PackagePart classes. The CreateRelationship methods use five items to define a relationship:

- The source package or part

- A URI for the target part or external resource

- A targetMode that identifies if the target is internal or external to the package

- A URI-formatted relationshipType

- A unique relationshipID

Once created, relationships can be accessed through the GetRelationshipsByType, GetRelationship, or GetRelationships methods of the Package and PackagePart classes (see Figure 11). We provide more detail about how relationships are stored in a package in the section "Exploring a Package."

Figure 11 Relationship Methods

Package/Part Method

Description

CreateRelationship

Creates a package-level or part-level relationship to a specified target.

DeleteRelationship

Deletes a specified relationship owned by this package or part.

GetRelationshipsByType

Returns all relationships owned by this package or part that match a specified relationshipType.

GetRelationship

Returns the relationship owned by this package or part that matches a specified relationshipID.

GetRelationships

Returns all relationships owned by this package or part.

RelationshipExists

Determines whether a specified relationship owned by this package or part exists.

Pack URI

In order to access a part contained in a package, an address for both the package and the selected part needs to be specified. A package can be identified through conventional URI schemes, such as http, ftp, or file. However, conventional URI schemes do not provide a means to specify an individual part within the package. Based on the extensible architecture for URI schemes (RFC 3986), the Open Packaging Conventions standard defines a pack URI scheme that can address individual parts contained in a package.

Creating a pack URI is fairly simple. Given a conventional URI for a package file and the name of a part contained in that package, a fully-qualified pack URI address that identifies the part can be created in two steps.

Say, for instance, the URI is http://www.proseware.com/mypackage.docx and the part name is /images/chocolate.jpg. The first step is to encode the URI of the package to form a pack authority. This is done by replacing "?", "@", ":", "%", and "," characters with their percent-encoded equivalents ("?"="%3f", "@"="%40", ":"="%3a", "%"="%25", ","="%2c") and replacing all forward slash characters with commas. Thus, the URI would look like this:

http%3a,,www.proseware.com,mypackage.docx

The second step is to combine the "pack://" scheme prefix, pack authority, and part name to form the pack URI for the specified part. The result for our sample would be:

pack://http%3a,,www.proseware.com,mypackage.docx/images/chocolate.jpg

There are additional steps to handle other forms of URIs, but understanding the basic process outlined here is sufficient for most uses.

There are some important classes you ought to be familiar with. These include:

PackUriHelper The PackUriHelper class provides methods that make it easy to create pack URIs, as well as to extract and return the original package URI or part name from a given pack URI.

PackWebRequest To access data through URIs, .NET-based applications can use the abstract WebRequest and WebResponse classes from which scheme-specific subclasses are implemented and registered. Based on the scheme of the URI specified to the WebRequest.Create method, the appropriate subclass is automatically selected and used. With the .NET Framework 3.0, WebRequest and WebResponse directly support pack URIs. A feature specific to pack URIs is that streams returned by the GetResponseStream method are fully seekable (if seek is not supported by the network protocol, seek operations are handled client-side transparently by the pack implementation). Through the use of pack URIs your application can directly access any part contained in a package on any URI accessible data source.

Authentication and Validation

Packaging services include support for digital signatures that can be used to securely sign and validate content within a package. Key PackageDigitalSignatureManager class methods include VerifyCertificate, Sign, VerifySignature, and RemoveSignature.

Rights Management and Encryption

Beyond the functionality defined by the Open Packaging Conventions, the System.IO.Packaging APIs in the .NET Framework 3.0 offer support for protecting packaged content both for privacy and security. Using encryption with rights management in combination with a Windows Rights Management server, packages can be encrypted to limit access to specific individuals or groups. For more information on rights management and encryption, see the "Rights Managed Package Publish" and "Rights Managed Package Viewer" .NET Framework 3.0 samples provided with the Windows SDK.

Packaging a Web Page

We’ll use a simple Web page with its local resources to illustrate some of the basic operations with packages. Note, however, that this example will limit us to some extent since Web pages are based on a fixed HTML schema and we want to avoid altering the existing page content. An extensible XML-based schema provides more flexibility for additional packaging features. We’ll discuss these and other options in the "Package File Format Design" section.

Now that we’ve covered the basics of packaging, we’ll write a sample program, PackageHtmxWrite, for creating our own custom .htmx package (the full source for PackageHtmxWrite is available on the MSDN Magazine Web site). We will store in the package a Web page together with its local style sheets, scripts, and image file resources. To identify the starting part that contains the HTML page, we’ll create a package-level relationship that points to the HTML part that’s stored in the package. We’ll also create part-level relationships that define associations between the Web page and each local resource stored in the package. The relationships will make the associations between each of the components directly discoverable and eliminate the need to search the package to locate the Web page or to parse the HTML content to locate the related resources.

Since the Web page and its local resources are typically defined through relative paths, we can use the relative path and filename of each item as the part name for storing the data stream in the package. For example, given the HTML tag <img src="images/packaging-sign.png">, we can use the path and filename of the src attribute to define the URI name /images/packaging-sign.png for storing the image as a part in the package. The local resources of the Web page are specified within the HTML, therefore we need to parse the HTML once to identify all of the resource files used in the page.

PackageHtmxWrite’s File | Open command lets the user choose a Web page to load and display in the Web browser control. After the page is loaded, the File | Save As command allows the users to specify the location and filename for storing the Web page and its local resources in an .htmx file. Specifically, File | Save As invokes methods that parse the page’s HTML and create a list of the local resource files to be included in the package.

The WritePackage method then performs a series of steps to create the package. First it creates the package with the user-specified path and filename. Then, for the root HTML Web page, it creates a part for storing the root page; this involves storing the HTML data into the Web page part and creating a package-level relationship that identifies the root HTML Web page part. For each local resource referenced in the Web page, the method creates a part for storing the resource, stores the resource’s data into the part, and creates a part-level relationship from the Web page part that targets the resource part. The WritePackage code shown in Figure 12 illustrates the process.

Figure 12 Creating a Package

private void WritePackage(

string packageFilepath, string parsedHtml, Hashtable resourceHash)

{

// Create and open the specified package file for writing.

// (The ‘using’ statement ensures that the package is

// closed and disposed when it goes out of scope.)

using (Package package = Package.Open(

packageFilepath, FileMode.Create))

{

string docPath =

webBrowser.Url.AbsoluteUri.Substring(rootPrefix.Length);

string docPartName = Path.GetFileName(packageFilepath);

docPartName = Path.ChangeExtension(docPartName, “htm”);

if (!docPath.EndsWith(“/”))

{

// docPath is in the format of “/folder1/folder2/1.html”

// Make it “/folder1/folder2/”

docPath = docPath.Remove(docPath.LastIndexOf(‘/’)+1);

}

docPartName = docPath + docPartName;

// The Web page and its resource files will likely be an absolute

// path with a common folder prefix.  Determine the common folder

// prefix to remove when the parts are stored.

string commonFolderString =

GetCommonFolderString(resourceHash.Values, docPartName);

// Remove prefix of common folders from the Web page part name.

docPartName = docPartName.Substring(commonFolderString.Length);

// Create a Uri name for the Web page part.

Uri webpagePartUri = PackUriHelper.CreatePartUri(

new Uri(docPartName, UriKind.Relative));

// Create a package part to store the Web page HTML.

PackagePart webpagePart = package.CreatePart(webpagePartUri,

System.Net.Mime.MediaTypeNames.Text.Html,

CompressionOption.Normal);

// Write the Web page HTML to the Web page part.

using (StreamWriter sw = new StreamWriter(

webpagePart.GetStream(), System.Text.Encoding.UTF8))

{

sw.Write(parsedHtml); // Write the Web page part.

}

// Create a package-level relationship to the Web page part.

package.CreateRelationship(webpagePart.Uri,

TargetMode.Internal, _packageRelationshipType);

// Create and write each of the Web page’s local resource parts.

foreach (Uri resourceUri in resourceHash.Keys)

{

string contenttype;

// Open the stream for each resource file.

using (Stream s = GetResourceStream(

resourceUri, out contenttype))

{

// Nothing to do if the resource file has no stream.

if (s == null) continue;

// Get the package part name for the resource.

ResourceInfo ri = (ResourceInfo)resourceHash[resourceUri];

string partName = ri.partName;

// Remove the leading common folders.

partName = partName.Substring(commonFolderString.Length);

// Create the part name URI for the resource.

Uri partUri = PackUriHelper.CreatePartUri(

new Uri(partName, UriKind.Relative));

// Use normal compression unless the data has already

// been compressed or is an octet-stream.

CompressionOption compression = CompressionOption.Normal;

if (   contenttype.StartsWith(“image”)

|| contenttype.StartsWith(“video”)

|| contenttype.StartsWith(“audio”)

|| contenttype.StartsWith(“application/octet-stream”))

compression = CompressionOption.NotCompressed;

// Create the part for storing the resource file

// in the package.

PackagePart resourcePart =

package.CreatePart(partUri, contenttype, compression);

// Copy the data from the file to the package part.

CopyStream(s, resourcePart.GetStream());

// Create a part-level relationship from the Web page part

// (owner) to the resource part (relationship target).

webpagePart.CreateRelationship(partUri,

TargetMode.Internal, resourceRelationshipType);

}

}

}

}

Exploring a Package

When executed, the PackageHtmxWrite sample creates an .htmx package that contains the original Web page along with its local resources. Since the package is a ZIP file, we can use the standard Windows ZIP functionality to examine the components stored in the package. Figure 13 illustrates the organization of the parts contained in the packaging.htmx sample included in the download accompanying this article. Here’s a quick tour of the contents.

Figure 13 Organization of Parts in Packaging.htmx (Click the image for a larger view)

A [Content_Types].xml part is stored in all packages. This part contains a list of the MIME types and extensions for all of the other parts in the package. In our sample, [Content_Types].xml contains the following information:

<?xml version=”1.0” encoding=”utf-8” ?>

<Types xmlns=

“http://schemas.openxmlformats.org/package/2006/content-types”>

<Default Extension=”xml” ContentType=”text/xml” />

<Default Extension=”htm” ContentType=”text/html” />

<Default Extension=”html” ContentType=”text/html” />

<Default Extension=”rels” ContentType=

“application/vnd.openxmlformats-package.relationships+xml” />

<Default Extension=”jpg” ContentType=”image/jpeg” />

<Default Extension=”png” ContentType=”image/png” />

<Default Extension=”css” ContentType=”text/css” />

</Types>

Within a package, relationships are defined in XML parts with a .rels extension; package-level relationships are defined in a part named /_rels/.rels. A package can have any number of package-level relationships. The relationship with the type /root-html is stored in the part /_rels/.rels. This is the relationship we created to identify the HTML Web page part name. We can use this relationship type in our programs later to return the part name of any HTML Web page we store. In our sample, the /_rels/.rels contains the following XML:

<?xml version=”1.0” encoding=”utf-8” ?>

<Relationships xmlns=

“http://schemas.openxmlformats.org/package/2006/relationships”>

<Relationship Type=

“http://schemas.microsoft.com/opc/2007/relationships/htmx/root-html”

Target=”/packaging.htm” TargetMode=”Internal”

Id=”Rf29a606b57094466” />

</Relationships>

The .htm (or .html) file is an embedded copy of the HTML file that the user selected in the application.

Within a package, part-level relationships are stored in a part with a special name. First the source part’s PartName is divided into its path and filename components. The relationship part name is then formed by appending _rels/ to the path and .rels to the filename. So, if the source PartName is /aaa/bbb/mypage.htm, the path is /aaa/bbb/ and the filename is mypage.htm. The resulting relationship part name is /aaa/bbb/_rels/mypage.htm.rels.

A part can have any number of relationships. In the packaging.htmx package, the packaging.htm part defines its relationships in the part /_rels/packaging.htm.rels.

Reading a Package

The accompanying PackageHtmxRead sample application makes use of a Web browser control to display a Web page packaged in an HTMX file. Since the Web browser doesn’t understand HTMX package files, PackageHtmxRead must first unzip the HTML file and its resources to a local temporary directory. The OpenHtmxPackage method shown in Figure 14 demonstrates how to unzip the HTMX content and then display the Web page.

Figure 14 Unzip Content and Display Web Page

public bool OpenHtmxPackage(string filepath)

{

// Extract the Web page and its local resources to the temp folder.

_tempFolder = GetTempFolder();

// Create a new tempFolder directory.  If the tempFolder

// exists, delete it and create a new empty one.

DirectoryInfo directoryInfo = new DirectoryInfo(_tempFolder);

if (directoryInfo.Exists) directoryInfo.Delete(true);

// Extract the Web page and its local resources to the temp folder.

_htmlFilepath = ExtractPackageParts(filepath, _tempFolder);

// Check that the Web page has a valid path and filename.

if (_htmlFilepath == null)

{

WritePrompt(“  Error: web page not found”);

return false;

}

// Convert the path and filename to a URI for the browser control.

Uri webpageUri;

try

{

webpageUri = new Uri(_htmlFilepath, UriKind.Absolute);

}

catch (System.UriFormatException)

{

string msg = _htmlFilepath + “\n\nThe specified path and “ +

“filename cannot be converted to a valid URI.\n\n”;

System.Windows.MessageBox.Show(msg, “Invalid URI”,

MessageBoxButton.OK, MessageBoxImage.Error);

return false;

}

// Load the Web page.

webBrowser.ScriptErrorsSuppressed = false;

webBrowser.Url = webpageUri;

return true;

}

To extract the individual files, the OpenHtmxPackage method calls the ExtractPackageParts method, which queries the package for the relationship that identifies the root HTML part. The root HTML part is then copied to its relative location in the target directory.

Next, the HTML part is queried for part-level relationships that identify the associated resource parts. The resource part for each part-level relationship is then copied to its relative location in the target directory. Again, all this is done simply through the use of relationships—there’s no parsing of any part content! The sample code in Figure 15 shows the ExtractPackageParts and ExtractPart methods used to unzip the HTML file and its local resource files.

Figure 15 Unzip HTML File and its Resource Files

ExtractPackageParts

private string ExtractPackageParts(

string packageFile, string targetDirectory)

{

Uri uriDocumentTarget = null;

// Open the Package.

using (Package package =

Package.Open(packageFile, FileMode.Open, FileAccess.Read))

{

PackagePart documentPart = null,  resourcePart = null;

// Examine the package-level relationships and look for

// the relationship with the "root-html" RelationshipType.

foreach (PackageRelationship relationship in

package.GetRelationshipsByType(_packageRelationshipType))

{

// Resolve the relationship target URI so

// the root-html document part can be retrieved.

uriDocumentTarget = PackUriHelper.ResolvePartUri(

new Uri("/", UriKind.Relative), relationship.TargetUri);

// Open the document part and write its contents to a file.

documentPart = package.GetPart(uriDocumentTarget);

ExtractPart(documentPart, targetDirectory);

}

// Examine the root part’s part-level relationships and look

// for relationships with "required-resource" RelationshipTypes.

Uri uriResourceTarget = null;

foreach (PackageRelationship relationship in

documentPart.GetRelationshipsByType(

resourceRelationshipType))

{

// Resolve the Relationship Target Uri so the resource part

// can be retrieved.

uriResourceTarget = PackUriHelper.ResolvePartUri(

documentPart.Uri, relationship.TargetUri);

// Open the resource part and write the contents to a file.

resourcePart = package.GetPart(uriResourceTarget);

ExtractPart(resourcePart, targetDirectory);

}

}

// Return the path and filename to the file referenced

// by the HTMX package’s "root-html" package-level relationship.

return targetDirectory + uriDocumentTarget.ToString().TrimStart(‘/’);

}

...

private const string _packageRelationshipType =

"http://schemas.openxmlformats.org/package/2007/" +

"relationships/htmx/root-html";

private const string _resourceRelationshipType =

"http://schemas.openxmlformats.org/package/2007/" +

"relationships/htmx/required-resource";

ExtractPart

private static void ExtractPart(

PackagePart packagePart, string targetDirectory)

{

// Remove leading slash from the Part Uri and make a new

// relative Uri from the result.

string stringPart = packagePart.Uri.ToString().TrimStart(‘/’);

Uri partUri = new Uri(stringPart, UriKind.Relative);

// Create an absolute file URI by combining the target directory

// with the relative part URI created from the part name.

Uri uriFullFilePath = new Uri(

new Uri(targetDirectory, UriKind.Absolute), partUri);

// Create the necessary directories based on the full part path

Directory.CreateDirectory(

Path.GetDirectoryName(uriFullFilePath.LocalPath));

// Write the file from the part’s content stream.

using (FileStream fileStream =

File.Create(uriFullFilePath.LocalPath))

{

CopyStream(packagePart.GetStream(), fileStream);

}

}

Signing and Validating a Package

The PackageHtmxSign sample program shows how digital signatures can be used to sign parts and relationships contained in a package. Using an X.509 certificate, you can identify the entity signing the content elements and create an encrypted hash that can then be used to validate that the content has not changed.

Typically an application would define a policy that lists the specific parts and relationships to be signed. To keep things simple for our .htmx format, our policy is to simply sign all parts and relationships. The SignAllParts method shows how to digitally sign all the parts and relationships in a package. Meanwhile, the ValidateSignatures method could be used to validate that all the signed elements have not been altered.

Designing a File Format

When planning your own package file format, there are several considerations you should keep in mind. First, use a package-level relationship to identify a starting part in the package. While you can use a fixed and mandatory part name, hardcoded part names can limit the flexibility of your package design. Relationships and relationship types enable easy discoverability without relying on predefined part names.

For parts that reference other resources, consider creating a relationship for each target resource. Building a tree of relationships and relationship types makes it easy to discover the structure of the items contained in the package without having to parse the content of each part.

For cases where all resources referenced by a part are defined with relationships, you may want to replace the references in the part content with the corresponding relationship ID values. By using relationship IDs, resource references can be changed simply by updating the appropriate relationship without having to parse and modify the content of the part itself. Note that this may not be practical if a part will be extracted or separated from the relationships that reference the actual target resources.

Try to avoid using relative references to resources outside the package. If the package is moved, the relative references to external resources will likely break.

Decide in advance whether the presence of unknown parts or relationships in your package format should be allowed. It’s easy to simply ignore unknown parts and relationships when they appear, but keep in mind that third-parties could add items into your package for undesirable purposes, such as tracking. To improve security, your application should regard unknown parts or relationships as invalid formats (or at least display an alert). And if the content is not intended to be changed, seriously consider signing the parts and relationships in your package.

Online Resources

- Office Open XML File Formats

- The Addressing Model of the Open Packaging Conventions

- Pack URIs in Windows Presentation Foundation

- Walkthrough: Word 2007 XML Format

Jack Davis is a Program Manager at Microsoft on the Windows Documents and Printing team. In a previous life he worked as a programmer/writer with the Windows Presentation Foundation SDK team. Jack can be reached at jack.davis@microsoft.com.

Andrey Shur is a Program Manager at Microsoft on the Windows Documents and Printing team.

Excel Services

Develop A Calculation Engine For Your Apps

Vishwas Lele and Pyush Kumar

This article discusses:

- Excel as a server-based application

- The Excel Services architecture and APIs

- Creating managed user-defined functions

- Building custom solutions with Excel Services

This article uses the following technologies:

Excel Services

Code download available at: ExcelServices2007_08.exe (226 KB)

Browse the Code Online

Contents

Excel Services Architecture

The Excel Services API

A Custom Solution with Excel Services

The Excel Pre-Compiler

The Excel Web Service Client

Digging into the Code

Performance and Scalability

Organizations use Microsoft® Excel® to perform complex calculations and to visualize information using charts, pivot tables, and the like, and to perform many other custom tasks. But in the past, if you wanted to implement a calculation engine, you needed to enlist the services of a developer who would use algorithms provided by your business analysts to design the code. Now, with the Excel Services technology in Office SharePoint® Server 2007, business analysts themselves can implement the calculation engine formulas they need, reducing the cost of implementation and making maintenance of the calculation algorithms easier than before. In addition, with Excel Services the custom algorithms in an Excel workbook can run on a Web server, allowing users to access them remotely. As you might imagine, this means many more users can take advantage of the software from many more locations.

Excel Services Architecture

Let’s see how the Excel Services architecture enables such flexibility. Excel Services consists of three tiers—a Web front end, an application server, and a database (see Figure 1). The SharePoint content database forms the database tier. To enable the server-side Excel behavior, you place the workbook at a trusted SharePoint location or on a network file share. Some functionality (such as additional security features) is only available through SharePoint.

Figure 1 Excel Web Services Architecture

The application server consists of Excel Calculation Services, which is responsible for loading a given workbook and performing any required calculations. Workbook instances can be connected to external data sources.

The Web front end is responsible for rendering the relevant portions of the workbook in HTML via a SharePoint Web Part. The Web front end is also responsible for exposing the Web service endpoints that allow remote access to the Excel Calculation Services.

An important aspect of the Excel Services architecture is that it is integrated with SharePoint 2007. As noted earlier, to enable some of the server-side behavior the workbook needs to be stored inside a SharePoint content database. This makes it possible to take advantage of SharePoint content management features such as versioning, check-in/check-out, and security roles and permissions in the context of Excel workbooks.

Similarly, the Excel Calculation Services is based on the SharePoint Shared Service Provider (SSP) model. SSP is a mechanism for packaging SharePoint functionality as a service that makes it easy to administer and use across different sites. As a result, it is possible to reuse an Excel Calculation Service instance across SharePoint sites, as well as manage it via a SharePoint administration site.

You should also note that Excel Services imposes some restrictions over Excel. Macros and unmanaged code-based add-ins, such as Visual Basic for Applications (VBA) code are not supported by Excel Services. Instead, Excel Services supports managed server-side user-defined functions (UDF), an interface that allows invocation of custom calculations from inside a server-side workbook.

Later in this article, we will look at a UDF code example. The restriction related to unmanaged code add-ins can be overcome by building a managed UDF to wrap unmanaged code. The restriction on the use of VBA and macros is hard, but it may be a boon in disguise as it prevents the server-side calculation logic from becoming unwieldy.

The Excel Services API

Now let’s look at the Web service-based Excel Services API used to interact with a server-side workbook, using the code snippet in Figure 2 as the basis for the discussion that follows. Note that code has been elided for clarity.

Figure 2 Open a New Workbook Session

string sheetName = “Input”;

string targetWorkbookPath = “workbook.xlsx”;

Excel.Status[] outStatus;

// #1 Excel Service Namespace

// using Excel = Microsoft.Office.Excel.Server.WebServices;

//#2 ExcelService

Excel.ExcelService service = new Excel.ExcelService();

//#3 Open a session with the workbook

string sessionID = service.OpenWorkbook(targetWorkbookPath,”en-US”,

“en-US”, out outStatus);

// #4 Prepare the range

Excel.RangeCoordinates rangeCoordinates = new Excel.RangeCoordinates();

rangeCoordinates.Column = 8;

rangeCoordinates.Row = 7;

rangeCoordinates.Height = 1;

rangeCoordinates.Width = 9;

object[] rangeValues = new object[rangeCoordinates.Width];

// Populate rangeValues

...

// #5 Set range values

service.SetRange(sessionID, sheetName, rangeCoordinates,

rangeValues, out outStatus);

// #6 Calculate the formulas in the workbook

service.CalculateWorkbook(sessionID,

Excel.CalculateType.CalculateFull, out outStatus);

// #7 Get the calculated values

object[] rangeResult = service.GetRange(sessionID,

sheetName, rangeCoordinates, false, out outStatus);

Console.WriteLine(“Total Rows: “ + rangeResult.Length);

// #8 Close the session

service.CloseWorkbook(sessionID, out outStatus);

To begin, we need to make the workbook accessible to the clients. Any workbook saved to a location on the server can be accessed through the Excel Service API, a part of Microsoft.Office.Excel.Server.WebServices. The Excel 2007 client makes publishing the workbook easier through publish functionality. The benefit of using the publish mechanism is that you can control which parts of a workbook (sheets, views, pivot tables, and so on) are accessible via the Excel Services API. The primary class in the API is the ExcelService class, shown in Figure 2. This class represents an in-memory, server-side instance of a workbook. To enable multiple users to interact concurrently with a workbook, a session-based access model has been implemented. Each user opens a separate session with a workbook using the OpenWorkbook method of the ExcelService class. The OpenWorkbook method returns a unique session ID associated with the opened session. This session identifier needs to be supplied when invoking any subsequent methods to interact with the opened workbook. To set a named range inside a workbook, you can use the RangeCoordinates class to define the boundaries of the named range. The SetRange takes RangeCoordinates and the corresponding array containing values to be passed in as parameters. A variation of the SetRange method is the SetRangeA1 method, which uses Excel range specification "A1" instead of the range coordinates used by SetRange. Once all the required range values have been specified, CalculateWorkbook can be invoked to force the workbook to compute the formulas. It is possible to cancel the most recent CalculateWorkbook method by invoking a CancelRequest method. You can use the GetRange method to obtain calculated values from a range in the open workbook. Once all the calculated values have been retrieved, you close the workbook session using the CloseWorkbook method.

Excel Services can be extended by adding UDFs, which are accessible as cell formulas similar to the built-in Excel functions. To create a UDF, you need to create a Microsoft® .NET Framework assembly that contains at least one class that is marked with the UdfClassAttribute and at least one method marked with the UdfMethodAttribute. Please refer to the code snippet below. Here we define ConvertToUpper as a UDF method. After appropriately registering the UDF, the ConvertToUpper function can be inside an Excel Services workbook instance:

using Microsoft.Office.Excel.Server.Udf;

[UdfClass]

public class Util

{

[UdfMethod]

public string ConvertToUpper(string name)

{

return name.ToUpper();

}

}

We’ve covered only a small portion of the Excel Services API here. For additional details, refer to the MSDN®documentation.

A Custom Solution with Excel Services

The primary motivation for developing a custom solution is to allow business analysts to author calculations (such as financial models) directly as Excel formulas. Until now, business analysts have mostly relied on documenting algorithms as pseudocode in text. The pseudocode was then translated into code by developers. Using Excel Services, we were able to overcome some of the limitations inherent in the process of creating formulas in Excel and, as a result, eliminated the need for developers to convert the pseudocode into real code.

The key challenge in allowing non-developers to author calculation logic was to strike a balance between flexibility and ease of authoring, with the ability to enforce a robust structure. To provide the structure, we needed a way to define an input and output "interface" that represents the data contract for a calculation algorithm. Business analysts would be limited to the named ranges that are part of the data contract for flowing data in and out of the calculation instance.

The obvious choice was to define the data contract using named cells or ranges constructs within Excel. Named ranges not only accord the required level of granularity for authoring calculations within Excel, they are also the fundamental data structure on which Excel Services API methods such as SetRange and GetRange are based. However, the challenge with using named ranges is that there is no standard format or language, such as XML Schema Definition (XSD) or Web Services Description Language (WSDL), to define the interface. Moreover, named ranges (and consequently Excel Service API methods) are inherently type unsafe. For instance, there is no way to enforce datatype checking on a given named range. Finally, there is no built-in way to enforce the contract across the calculations (inside Excel) and the Excel Services client program.

To overcome these limitations, we developed a custom two-part solution. The first part is an Excel pre-compiler designed to generate the named ranges based on a defined interface. The second part is a generic Excel Web service client that invokes the calculation inside a workbook, while adhering to the interface.

The Excel Pre-Compiler

XML Schema Definition with all its semantic richness and simplicity, seemed the ideal choice for defining the interface. We decided to use the XML Schema constructs to define the input and output contracts. Next, we needed a way to translate the XML Schema into named ranges. We first looked at the possibility of using the XMLMap feature introduced with Excel 2003. XMLMap allows cells inside Excel to be mapped to the elements of an imported XML Schema. Unfortunately, the XMLMap capability is not available to Excel Services. So the alternative was to create the named ranges inside an Excel workbook. We developed a pre-compiler component that generated a template workbook with the required named ranges based on the schema. The generated template workbook has three sheets—one each for input, output, and calculation. The input sheet contains named ranges that correspond to the input for the calculation. Similarly, the output sheet contains named ranges that correspond to the output of the calculation, and the calculation sheet is where the calculations are placed (see Figure 3).

Figure 3 Workbook Input, Calculation, and Output Sheets (Click the image for a larger view)

As we stated earlier, programming constructs such as looping are not available to Excel Services. The pre-compiler compensates for such limitations by transforming the input XML fields into a format that is accessible without the need for complex programming constructs. For example, XSD element collections can be transformed into dimensions of the named range. Figure 4 depicts an XSD snippet that is part of the input data contract for a calculation engine.

Figure 4 Input Data Contract

<?xml version=”1.0” encoding=”utf-8” ?>

<xs:schema id=”Input” ...>

<xs:complexType name=”TypeA”>

<xs:sequence>

<xs:element name=”ValueA1” type=”xs:int”  minOccurs=”0” />

<xs:element name=”ValueA2” type=”xs:int” minOccurs=”0” />

</xs:sequence>

<xs:attribute name=”RangeHeight” type=”xs:int” default=”1” />

<xs:attribute name=”RangeWidth” type=”xs:int” default=”10” />

</xs:complexType>

<xs:complexType name=”InputType”>

<xs:sequence>

<xs:element name=”InputElementA” type=”TypeA”    />

<xs:element name=”InputElementB” type=”TypeB”    />

</xs:sequence>

</xs:complexType>

<xs:element name=”InputDataset” type=”InputType” msdata:IsDataSet=”true” />

</xs:schema>

Elements TypeA and TypeB are part of the input to the calculation. Note the custom attributes RangeHeight and RangeWidth that define the dimensions of the named ranges. The pre-compiler uses this information to generate the named range dimensions. The pre-compiler can also dereference index fields into separate columns—where each column represents an index value.

A noteworthy aspect of the pre-compiler is the ability to preserve the existing calculations while regenerating the input and output sheets. As depicted in Figure 5, developing a calculation algorithm is an iterative process. Business analysts and developers work together to define the initial data contract. During the course of development of the workbook, input and output contracts may need to be modified, and the workbook must be regenerated for these changes to be applied. The pre-compiler supports such iterative development by preserving the calculations sheet while regenerating the workbook.

Figure 5 Generating a Workbook Using the Pre-Compiler

The input sheet shown in Figure 3 has the pre-generated named ranges. The calculations sheet has calculations that reference the named ranges defined on the input sheet. The output sheet, in turn, references the calculations from the calculation sheet.

The Excel Web Service Client

The primary role of the Excel Web service client is to invoke calculations inside the workbook using the Excel Services API. In doing so, it interprets the XSD-based input contract, mapping the schema elements into appropriate named ranges. Once the calculation is complete, the client maps the output named ranges back into an XSD-based output contract. Figure 6 depicts the role of the Excel Web service client. A typed DataSet (based on the input schema contract) is passed in as input. Data contained within the DataSet is mapped to input named ranges. Once the calculation is complete, the output named ranges is used to populate the output DataSet. The Excel Web service client is responsible for applying the rules defined for the pre-compilers. It is also possible to inject custom data transformations to alter the aforementioned mapping between XSD and named ranges. Recall our earlier discussion on the need for compensating the lack of programming constructs available to workbook authors. Custom data transformations allow mapping to be altered to make it easy for business analysts to author the calculation logic.

Figure 6 Invoking the Custom Calculation Engine

Digging into the Code

The solution we’ve built is composed of four projects. The PreCompiler project houses all the code to generate a workbook with the required named ranges based on the input and output schema. It takes into account the aforementioned custom attributes such as RangeHeight while generating the named ranges. The PreCompiler project in turn relies on SpreadsheetML (an XML-based dialect used to represent information inside a spreadsheet) for generating the workbook. The SpreadsheetML project contains simple classes that wrap the SpreadsheetML components such as Workbook, Worksheet, and so forth.

The Client project, as the names suggests, is the Excel Web service client code. It sets the values for the input named ranges, forces the workbook to recalculate, and retrieves the values for output named ranges. You will recall that we talked about the need to transform the data to make it easier for business analysts to develop the calculations. In order to allow data transformation to be customized for each calculation engine, we externalized the data transformation logic by defining an IDataTransformer Interface, as shown here:

public interface IDataTransformer

{

object    getRangeData(string RangeName);

object[]  getRangeData(string RangeName, int width);

object[,] getRangeData(string RangeName, int width, int height);

string getInputSchema();

string getOutputSchema();

string getInputSchemaPrefix();

string getOutputSchemaPrefix();

}

The name of an assembly that contains an implementation of the IDataTransformer interface is passed in as input to the client program. The client program in turn calls back the appropriate methods on the class implementing the IDataTransformer interface. In this way, the client program obtains the values that are used to populate the named ranges. The implementation logic inside the IDataTransformer methods is responsible for converting the data residing inside the input DataSet to appropriate named range values. For example, some rows from a DataTable may need to be filtered before populating the appropriate named ranges. Or the rows within the DataTable might need to be sorted before passing them along to the workbook. All such data transformation needs can be met using the IDataTransformer interface.

One other class that is important to discuss here is the ExcelServiceFacade. This class hides the Excel Service API details from the caller. The other important function of this class is to combine individual SetRange calls into one aggregated SetRange call. This is crucial for reducing the network latency as each invocation of SetRange causes a round-trip to the server. By exposing a local SetRange call that is ultimately converted into one aggregated SetRange call, ExcelServiceFacade can dramatically improve the response times. Figure 7 depicts the relevant ExcelServiceFacade code. An internal buffer is maintained by the ExcelServiceFacade class that is appended each time a "local" SetRange call is invoked. Once all the input named ranges have been populated, the internal buffer is sent up to the server as part of a single call. A similar mechanism is used when retrieving the output named values after the calculation is complete. Rather than retrieve the values from the output named ranges individually, we simply retrieve all the values on the output sheet in one fell swoop.

Figure 7 ExcelServiceFacade Class Methods

public class ExcelServiceFacade

{

...

public void OpenWorkbook()

{

if (this.SessionID.Equals(String.Empty))

{

Status[] status = null;

this.Excel.Credentials =

System.Net.CredentialCache.DefaultCredentials;

this.m_SessionID = this.Excel.OpenWorkbook(

m_Url, CULTURE, CULTURE, out status);

}

}

public void CalculateWookbook()

{

RangeCoordinates coorInput = new RangeCoordinates();

coorInput.Row = 0;

coorInput.Column = 1;

coorInput.Width = 25;

coorInput.Height = m_RowIndex;

// Set Range

object[] inputValues = (object[])

m_Input.ToArray(typeof(object[]));

SetRange(“Input”, coorInput, inputValues);

this.Excel.CalculateWorkbook(

this.SessionID, CalculateType.CalculateFull);

}

}

Performance and Scalability

We found the actual execution of the calculations within the workbook to be very fast. In an unscientific test conducted with a workbook containing a nontrivial set of calculations involving close to one thousand named ranges, the response time was under one second; the majority of that time was spent on round-trips to the Excel server. As the load on the system grows in terms of the complexity of the calculation and the number of concurrent executions, it is possible to scale the solution by leveraging the various topology options offered by Excel Services. Different topology options allow you to select where each of the logical Excel Services layers (presentation, application, and database) are placed. For smaller setups (mainly used for testing purposes), it is possible to deploy all three layers on a single server. For a medium setup, the presentation and application layers can be installed on a single server, and the database layer on a separate server. For large setups, you can install each of the three layers on separate servers. Additionally, you can scale out the presentation layer by adding more servers using a network load balancer. The application layer comprising the Excel Calculation Services can also be scaled out using the load balancing schemes supported by the SSP framework. Figure 8 depicts a large setup where each layer is installed on a separate server. Furthermore, presentation and application layers are scaled out using load-balancing schemes.

Figure 8 Excel Web Services Large Setup

For computationally intensive workloads, it is also possible to combine Excel Services with the Compute Cluster Server to seamlessly distribute work to compute nodes, as shown in Figure 9.

Figure 9 Excel Services High-Performance Setup

As you can see, a custom solution for implementing calculation engine components using Excel Services is a great productivity boon. It allows you to provide your users with anywhere-access to custom workbook functions, eliminates the need for a developer to implement the logic, and lets you scale out your solution as needed. Give it a try. We’re sure you’ll enjoy the increased flexibility and productivity you’ll gain. For more information, see the "Resources" sidebar.

Resources

- Services Technical Overview

- Determine Resource Requirements to Support Excel Services

- Creating XML Mappings in Excel 2003

Vishwas Lele is a CTO at Applied Information Sciences (AIS) in Reston, VA. He assists organizations in envisioning, designing, and implementing enterprise solutions that are based on Microsoft .NET technologies. Vishwas is the Microsoft Regional Director for the Washington, DC, area. He can be reached at vlele@acm.org.

Pyush Kumar is a Lead Systems Architect for Watson Wyatt Worldwide. He’s been working most recently on grid computing and large-scale software design for the .NET Framework. You can reach him at pyush.kumar@watsonwyatt.com.

Mobility

Make Your WPF Apps Power-Aware

Andre Michaud

This article discusses:

- Creating power-aware applications

- Power support on Windows XP

- Power support on Windows Vista

- Custom WPF dependency properties and routed events

This article uses the following technologies:

.NET Framework 3.0, Windows Vista, Windows XP

Code download available at: PowerAware2007_07.exe (360 KB)

Browse the Code Online

Contents

Existing Work

Overview of the Code

Notification of System Power Changes

PowerModeChanged Event and Handler

Trapping the WM_POWERBROADCAST Window Message

Registering for Notifications on Windows Vista

Detecting the Operating System

Custom Power Dependency Properties

Custom Power Events

Windows XP Power Structs

Initializing the Dependency Properties on Windows XP

Windows Vista Power Structs

How the Message Is Handled

Using the Dependency Properties and Events

Controlling an Animation

Conclusion

As mobile computers become increasingly prevalent, battery life has become more important. Hardware manufacturers are aware of this and design mobile devices with battery consumption in mind. But can the same be said for your software? If not, maybe you should begin to think about coding software that is power-aware. Both Windows® XP and Windows Vista™ provide system information to enable your applications to do just that. In this article, I provide a starting point from which developers can understand power awareness in their Windows Presentation Foundation applications.

First, you must understand that Windows Presentation Foundation itself does not inherently make power notifications available to a would-be power-aware application. You’ll see that for this functionality, my sample relies on the operating system—either Windows XP or Windows Vista (the operating systems on which Windows Presentation Foundation is designed to run).

Before I show you how to get this information into a Windows Presentation Foundation application, I should note that the power notification system on Windows Vista is more evolved than that on Windows XP. Windows Vista contains all the functionality of the Windows XP power notification system and then some, which can present quite a challenge. What are the notification systems, how do they differ, and how do you manage these differences to design a successful power-aware application? These are especially important questions in light of the fact that Windows Presentation Foundation applications are meant to run on both Windows XP and Windows Vista.

Existing Work

There are two excellent hands-on labs designed for those who are looking for sample code and hands-on experience with the Windows XP and Windows Vista power notification systems. These labs can be found at (for Windows XP) and (for Windows Vista). The labs cover the power notification systems as they would be used in managed code; however, they don’t represent how the power notification systems would be used specifically in Windows Presentation Foundation. Still, these labs provide an excellent starting point and cover the notification systems as a whole in greater detail than I can do here. I recommend you use them in conjunction with this article.

Another important point is that every application will have its own unique concerns in regard to power consumption. Not every application will perform 3D animation or write data to disk when the system is about to enter a suspended state. So, a single sample or hands-on lab cannot realistically address all the power needs of all applications. I would thus encourage developers to review the details of each notification system, as well as the measures taken in this article, within the context of his or her application’s needs.

Overview of the Code

In this article, I attempt to reconcile the differences between the Windows XP and Windows Vista power notification systems for use in a Windows Presentation Foundation application. The bulk of the work is performed inside a custom class: PowerAwareWindow. This class derives from the Window class (the element in which many applications live) and exposes much of the power-related system information and notifications in a fashion typical of the Windows Presentation Foundation—using custom dependency properties and events. The definition of this custom class can be found in the PowerAwareWindow.cs file (all of the code for this article is available for download from the MSDN® Magazine Web site).

The functions, variables, and associated structures that make power notifications from the operating system possible exist in native code. In order to make the work that is done in the PowerAwareWindow class more readable, these native elements are placed in a separate code file, NativeMethods.cs.

The sample application, which uses PowerAwareWindow, lives within the Window1.xaml and Window1.xaml.cs files. Rather than being hosted within a standard Window, the application is hosted within a PowerAwareWindow; it hooks up to the custom events and makes use of the custom dependency properties declared within the PowerAwareWindow to receive and respond to changes in the system’s power status.

Note that the sample code in the download differs slightly from the code presented here. Many of the native elements of the sample reside within the NativeMethods.cs file. Therefore, in the sample code, when native elements are referenced they are prefixed with NativeMethods. For example, the downloadable code refers to the operating system version as:

NativeMethods.VISTA_OS_VERSION

In this article, however, for clarity I simply refer to it as:

VISTA_OS_VERSION

In addition, the formatting differs slightly for better readability, the comments differ, and the code in the download includes some Debug.Assert calls. These calls are included as an extra precaution—some sections of the code assume that the application is running on Windows Vista and others assume it to be running on Windows XP.

Notification of System Power Changes

The first step is to get notified of power-related system changes. As I mentioned earlier, this article does not provide a complete overview of the Windows XP and Windows Vista notification systems. I present a subset of this functionality specific to its use in the Windows Presentation Foundation. (For a complete overview of the notification systems, you should check out the Mobile PC Development Guides listed in the "Additional Resources" sidebar.)

I discuss two ways in which you can acquire notifications of system power changes: with the PowerModeChanged event on the SystemEvents class in the Microsoft.Win32 namespace in System.dll, and by trapping the WM_POWERBROADCAST window message.

The PowerModeChanged event is actually built on top of the WM_POWERBROADCAST message, but was done so before new power-related features were added to Windows Vista. As such, on Windows Vista the latter approach is more powerful. For example, when using only the PowerModeChanged event, an application would know nothing about the Windows Vista user’s defined power plan (to be discussed shortly).

PowerModeChanged Event and Handler

The PowerModeChanged event passes a PowerModes enumeration to registered event handlers. This enum has three values: Resume, StatusChange, and Suspend. With this event, an application is notified when:

- The OS is about to resume from a suspended state (Resume)

- The power source has transitioned between AC and battery or some other change has occurred in the status of the system power supply (StatusChange)

- The operating system is about to be suspended (Suspend)

In the PowerAwareWindow, this event would normally be hooked up in the Window.Loaded event handler and unhooked in the Window.Closed event handler (see Figure 1). This, of course, is after the Window.Loaded and Window.Closed events are hooked up. This event covers much of the power-related information that an application might be interested in, and it may be the best option for your power-aware application. If an application is interested only in this information, then this method of receiving power events is much more straightforward than the window message trapping method that I’ll discuss momentarily.

Figure 1 Hooking Up PowerModeChanged Event

protected override void OnInitialized(EventArgs e)

{

// Hook up the Window.Loaded and Window.Closed events

this.Loaded += new RoutedEventHandler(PowerAwareWindow_Loaded);

this.Closed += new EventHandler(PowerAwareWindow_Closed);

base.OnInitialized(e);

}

void PowerAwareWindow_Loaded(object sender, RoutedEventArgs e)

{

Microsoft.Win32.SystemEvents.PowerModeChanged +=

new PowerModeChangedEventHandler(

XP_SystemEvents_PowerModeChanged);

}

void PowerAwareWindow_Closed (object sender, RoutedEventArgs e)

{

Microsoft.Win32.SystemEvents.PowerModeChanged -=

new PowerModeChangedEventHandler(

XP_SystemEvents_PowerModeChanged);

}

The XP_SystemEvents_PowerModeChanged function is implemented in the code download and can be viewed there. It isn’t utilized in the sample as the code for hooking up the event is commented out and is included only for demonstration purposes.

Trapping the WM_POWERBROADCAST Window Message

The operating system sends out WM_POWERBROADCAST Window messages to alert listening applications of changes to power-related status or settings. However, in order to receive some of these notifications, an application must first register for them. The sample accompanying this article uses the window messages for power notifications (rather than the PowerModeChanged event) in the interest of thoroughness.

With WM_POWERBROADCAST, an application can receive the full gamut of system power change notifications. In order to receive window messages in a Windows Presentation Foundation application, an event handler must first be declared to receive window messages, like so:

void PowerAwareWindow_Loaded(object sender, RoutedEventArgs e)

{

WindowInteropHelper helper = new WindowInteropHelper(this);

HwndSource source = HwndSource.FromHwnd(helper.Handle);

source.AddHook(new HwndSourceHook(this.MessageProc));

...

}

This code declares the MessageProc function as the window message event handler. That function will check for the WM_POWERBROADCAST message (see Figure 2) and act accordingly.

Figure 2 WM_POWERBROADCAST Message

const int WM_POWERBROADCAST = 0x0218;

IntPtr MessageProc(

IntPtr hwnd, int msg, IntPtr wParam, IntPtr lParam, ref bool handled)

{

switch (msg)

{

case WM_POWERBROADCAST:

// Handle the message here...

break;

// Handle other window messages (if the application needs to) here

}

}

The information about the event is contained in the wParam of the window message. The wParam can take many forms to indicate different power-related system events. For example, the following value indicates that either the power source changed or that the battery percentage remaining has changed:

const int PBT_APMPOWERSTATUSCHANGE = 0x000A;

These indicate that the system is about to enter or is in the process of leaving a suspended state:

const int PBT_APMSUSPEND = 0x0004;

const int PBT_APMRESUMESUSPEND = 0x0007;

Some values are not available on all platforms. For example, the value shown below is available starting with Windows Vista and indicates that one of the following has changed: the power source, the power plan, the battery percentage remaining, or the monitor state.

const int PBT_POWERSETTINGCHANGE = 0x8013;

A complete list of such events can be found in the Power Management documentation. For comparison, the PBT_APMPOWERSTATUSCHANGE, PBT_APMSUSPEND, and PBT_APMRESUMESUSPEND wParams are analogous to the StatusChange, Suspend, and Resume PowerModes in the PowerModeChange event, respectively.

It is important to note the difference between wParam values for PBT_APMPOWERSTATUSCHANGE and the PBT_ POWERSETTINGCHANGE. The latter can be understood as a Windows Vista-only superset of PBT_APMPOWERSTATUSCHANGE. It indicates all the power-related changes that PBT_APMPOWERSTATUSCHANGE indicates as well as other information specific only to Windows Vista, such as power plans and monitor state. Both of these wParams exist and are passed along with the WM_POWERBROADCAST window message on Windows Vista, but processing both would result in redundancy in this sample when running on Windows Vista.

The PBT_POWERSETTINGCHANGE wParam, however, will not be sent to a Windows Vista application until the application registers for power setting notifications (PBT_APMPOWERSTATUSCHANGE is received without registration). This is a very important step for receiving power notifications specific to Windows Vista.

Registering for Notifications on Windows Vista

To register for power setting notifications on Windows Vista, the first step is to make the RegisterPowerSettingNotification function available for use in the application. You can do that with the following P/Invoke function definition:

[DllImport(@”User32.dll”, SetLastError=true,

CallingConvention=CallingConvention.StdCall)]

static extern IntPtr RegisterPowerSettingNotification(

IntPtr hRecipient, ref Guid PowerSettingGuid, Int32 Flags);

This function returns a handle that will be used to unregister for the events when the application is shutting down.

The Flags passed into this function indicate to the operating system how the application would like to receive notifications. For this sample, I declare this as:

const int DEVICE_NOTIFY_WINDOW_HANDLE = 0x00000000;

This indicates to the registration function that notifications are to be sent to the application using the WM_POWERBROADCAST window message with a wParam of PBT_POWERSETTINGCHANGE. The other option for Flags is relevant for OS services and is explained in the RegisterPowerSettingNotification documentation mentioned earlier.

The PowerSettingGuid value passed to the registration function indicates to the operating system the notifications in which the application is interested. Figure 3 shows the four PowerSettingGuid values you’re probably most interested in.

Figure 3 Four Types of PowerSettingGuid

// notify application when percentage of battery remaining changes

static Guid GUID_BATTERY_PERCENTAGE_REMAINING =

new Guid(“A7AD8041-B45A-4CAE-87A3-EECBB468A9E1”);

// notify application when the state of the monitor changes

static Guid GUID_MONITOR_POWER_ON =

new Guid(“02731015-4510-4526-99E6-E5A17EBD1AEA”);

// notify application when the system power source changes

static Guid GUID_ACDC_POWER_SOURCE =

new Guid(“5D3E9A59-E9D5-4B00-A6BD-FF34FF516548”);

// notify application when the user-defined power plan changes

static Guid GUID_POWERSCHEME_PERSONALITY =

new Guid(“245D8541-3943-4422-B025-13A784F679B7”);

Other notification GUIDs are available and can be used to get notified for events such as when the system is entering or exiting an away mode or when the system will be moving into an idle state in the near future and the current time is a good time to perform background or idle tasks.

In the sample, I have grouped all the Windows Vista power registration into one function (see Figure 4). Similarly, I have also grouped all the unregistration into one function. Registration occurs when the PowerAwareWindow loads (the Window.Loaded event is handled by PowerAwareWindow_Loaded function) and unregistration occurs when it closes (Window.Closed handled by PowerAwareWindow_Closed).

Figure 4 Windows Vista Power Registration

IntPtr _hBattCapacity;

IntPtr _hMonitorOn;

IntPtr _hPowerScheme;

IntPtr _hPowerSrc;

void RegisterForVistaPowerNotifications(IntPtr hWnd)

{

_hPowerSrc = RegisterPowerSettingNotification(

hWnd, ref GUID_ACDC_POWER_SOURCE, DEVICE_NOTIFY_WINDOW_HANDLE);

_hBattCapacity = RegisterPowerSettingNotification(

hWnd, ref GUID_BATTERY_PERCENTAGE_REMAINING,

DEVICE_NOTIFY_WINDOW_HANDLE);

_hMonitorOn = RegisterPowerSettingNotification(

hWnd, ref GUID_MONITOR_POWER_ON, DEVICE_NOTIFY_WINDOW_HANDLE);

_hPowerScheme = RegisterPowerSettingNotification(

hWnd, ref GUID_POWERSCHEME_PERSONALITY,

DEVICE_NOTIFY_WINDOW_HANDLE);

}

void UnregisterForVistaPowerNotifications()

{

UnregisterPowerSettingNotification(_hBattCapacity);

UnregisterPowerSettingNotification(_hMonitorOn);

UnregisterPowerSettingNotification(_hPowerScheme);

UnregisterPowerSettingNotification(_hPowerSrc);

}

Registration introduces one of the main issues that I address in this sample—to create a consistent approach to power-awareness in one Windows Presentation Foundation application that accounts for the differences regarding power notifications on Windows XP and Windows Vista. The RegisterPowerSettingNotification and UnregisterPowerSettingNotification functions exported from User32.dll are not available on operating systems prior to Windows Vista; as a result, any code that tries to use these functions should only do so after first ensuring the functions are available.

Detecting the Operating System

It is important for your application to detect the operating system and act accordingly. To this end, my sample has a private variable marked readonly for storing this information:

readonly int _osVersion = Environment.OSVersion.Version.Major;

The operating system major versions for Windows Vista and Windows XP are:

const int VISTA_OS_VERSION = 6;

const int XP_OS_VERSION = 5;

At this point, the PowerAwareWindow_Loaded and PowerAwareWindow_Closed functions should look something like that shown in Figure 5. At this point, the application is receiving power notifications via window messages in the MessageProc function. Once the power notifications are received, the application needs to respond appropriately.

Figure 5 PowerAwareWindow_Loaded and_Closed Functions

void PowerAwareWindow_Loaded(object sender, RoutedEventArgs e)

{

// Optional: If the PowerModeChanged event contains the desired power-

// notification functionality for an application, no OS detection is

// necessary as PowerModeChanged will work with both

SystemEvents.PowerModeChanged += new PowerModeChangedEventHandler(

XP_SystemEvents_PowerModeChanged);

// Use the helper to hook up our Windows Presentation Foundation window

// to receive window messages.

WindowInteropHelper helper = new WindowInteropHelper(this);

HwndSource source = HwndSource.FromHwnd(helper.Handle);

source.AddHook(new HwndSourceHook(this.MessageProc));

if (_osVersion >= VISTA_OS_VERSION)

{

RegisterForVistaPowerNotifications(source.Handle);

// There’s no need to initialize the dependency properties on

// Windows Vista because this application will get notified of the

// power status when it registers for the events.

}

else

{

// Application power initialization should be done here (otherwise

// the application is left waiting for a WM_POWERBROADCAST window

// message that might not come right away).

}

}

void PowerAwareWindow_Closed(object sender, EventArgs e)

{

// Optional: Unregister for PowerModeChanged if registration for it

// was done in PowerAwareWindow_Loaded

SystemEvents.PowerModeChanged -= new PowerModeChangedEventHandler(

XP_SystemEvents_PowerModeChanged);

if (_osVersion >= VISTA_OS_VERSION)

{

UnregisterForVistaPowerNotifications();

}

else

{

// Do Windows XP-specific application cleanup here

}

}

Custom Power Dependency Properties

In many cases it is necessary for a power-aware application to declare and make use of custom dependency properties other than the ones declared in this sample. Note that custom dependency properties is an advanced topic. If you need more background before reading this section, check out the Custom Dependency Properties documentation.

While these custom dependency properties may seem complicated, they provide a lot of power and flexibility for an application that uses the PowerAwareWindow. The custom dependency properties declared in the sample are PowerPlan, MonitorOn, RemainingBattery, and RunningOnBattery.

PowerPlan exposes the current user-specified power plan. This piece of information is specific to Windows Vista. On Windows XP this will always be a default value (Automatic). The associated PowerPlanChanged event does not fire on Windows XP as there is no notion of power plans on that OS.

The PowerPlan enumeration is defined as:

public enum PowerPlan

{

Automatic,

HighPerformance,

PowerSaver,

};

In the Windows Vista UI, these are called "Balanced," "High Performance," and "Power Saver," respectively. The dependency property registration looks like the code shown in Figure 6.

Figure 6 PowerPlan Dependency Property Registration

private static readonly DependencyPropertyKey PowerPlanPropertyKey =

DependencyProperty.RegisterReadOnly(

“PowerPlan”, typeof(PowerPlan), typeof(UIElement),

new FrameworkPropertyMetadata(PowerPlan.Automatic,

FrameworkPropertyMetadataOptions.Inherits |

FrameworkPropertyMetadataOptions.Journal |

FrameworkPropertyMetadataOptions.

OverridesInheritanceBehavior,

new PropertyChangedCallback(OnPowerPlanChanged)));

public static readonly DependencyProperty PowerPlanProperty =

PowerPlanPropertyKey.DependencyProperty;

private static void OnPowerPlanChanged(

DependencyObject d, DependencyPropertyChangedEventArgs e)

{

PowerAwareWindow myPowerAwareWindow = d as PowerAwareWindow;

if (myPowerAwareWindow != null &&

myPowerAwareWindow.PowerPlanChanged != null)

{

myPowerAwareWindow.PowerPlanChanged(myPowerAwareWindow, e);

}

}

public event DependencyPropertyChangedEventHandler PowerPlanChanged;

public PowerPlan PowerPlan

{

get { return (PowerPlan)GetValue(PowerPlanProperty); }

private set

{

SetValue(PowerAwareWindow.PowerPlanPropertyKey, value);

}

}

MonitorOn exposes the current state of the monitor (display)—either on or off. This piece of information is specific to Windows Vista. On Windows XP, it will always have a default value (True). The associated MonitorOnChanged event does not fire on Windows XP as there is no notion of Monitor state in that OS.

The dependency property registration looks very similar to the one described for PowerPlan, except that the property is of type Boolean, the property is named MonitorOn, and the corresponding event is MonitorOnChanged. The full implementation for this and the rest of the dependency properties can be found in the code download.

RemainingBattery exposes the percentage of the battery currently remaining—ranging from 0 to 100 percent. For this case, the dependency property registration is very similar to that of the previous two dependency properties.

RunningOnBattery is true if the system is running on a battery and false if it is running on AC power.

Custom Power Events

As with the custom dependency properties, it is also often necessary for a power-aware application to declare and make use of custom events. (If you need more background on custom events, check out the Routed Event Overview). The custom events in my sample are PreviewSystemResuming, SystemResuming, PreviewSystemSuspending, and SystemSuspending.

PreviewSystemResuming occurs when the system is resuming from a suspended state. The Preview part of the event indicates that the event’s routing strategy is Tunnel as opposed to Direct or Bubble. The event registration looks like the code shown in Figure 7.

Figure 7 PreviewSystemResuming Event Registration

static readonly RoutedEvent PreviewSystemResumingEvent =

EventManager.RegisterRoutedEvent(“PreviewSystemResumingEvent”,

RoutingStrategy.Tunnel, typeof(RoutedEventHandler),

typeof(PowerAwareWindow));

public event RoutedEventHandler PreviewSystemResuming

{

add { AddHandler(PreviewSystemResumingEvent, value); }

remove { RemoveHandler(PreviewSystemResumingEvent, value); }

}

//  This is called when the PowerAware window wants to raise the event

void RaisePreviewSystemResumingEvent()

{

RaiseEvent(new RoutedEventArgs(

PowerAwareWindow.PreviewSystemResumingEvent, this));

}

SystemResuming occurs when the system is resuming from a suspended state. This uses the Bubble routing strategy version of the PreviewSystemResuming event. The event registration looks like the code shown in Figure 8.

Figure 8 SystemResuming Event Registration

static readonly RoutedEvent SystemResumingEvent =

EventManager.RegisterRoutedEvent(“SystemResumingEvent”,

RoutingStrategy.Bubble, typeof(RoutedEventHandler),

typeof(PowerAwareWindow));

public event RoutedEventHandler SystemResuming

{

add { AddHandler(SystemResumingEvent, value); }

remove { RemoveHandler(SystemResumingEvent, value); }

}

//  This is called when the PowerAware window wants to raise the event

void RaiseSystemResumingEvent()

{

RaiseEvent(new RoutedEventArgs(

PowerAwareWindow.SystemResumingEvent, this);

}

PreviewSystemSuspending occurs when the system is about to enter a suspended state. The Preview part of the event indicates that the event’s routing strategy is Tunnel, as it is with PreviewSystemResuming.

SystemSuspending occurs when the system is about to enter a suspended state. As with SystemResuming, it uses the Bubble routing strategy.

Windows XP Power Structs

The custom dependency properties aren’t much use unless they contain the latest system information. Similarly, the custom power events need to be raised appropriately. Let’s look at this now, as well as some of the trickier things involved in getting Windows Presentation Foundation power-awareness to function well for a single application that runs on both Windows XP and Windows Vista.

Let’s start with a look at important Windows XP power structures. First there’s the SYSTEM_POWER_STATUS (see Figure 9). As you can see, this sample uses the ACLineStatus and BatteryFlag enumerations.

Figure 9 SYSTEM_POWER_STATUS

class SYSTEM_POWER_STATUS

{

public ACLineStatus ACLineStatus;

public BatteryFlag BatteryFlag;

// can be a value in the range 0 to 100, or 255 if status is unknown

public byte BatteryLifePercent;

public byte Reserved1;            // reserved; must be zero

// number of seconds of battery life remaining,

// or –1 if remaining seconds are unknown

public Int32 BatteryLifeTime;

// number of seconds of battery life when at full charge,

// or –1 if full battery lifetime is unknown

public Int32 BatteryFullLifeTime;

}

The SYSTEM_POWER_STATUS is returned from the GetSystemPowerStatus function. It is made available to the application with the following import:

[DllImport(“Kernel32.DLL”, CharSet=CharSet.Auto, SetLastError=true)]

[return: MarshalAs(UnmanagedType.Bool)]

static extern bool GetSystemPowerStatus(

[In, Out] SYSTEM_POWER_STATUS SystemPowerStatus);

This function is called whenever the WM_POWERBROADCAST message is received with the PBT_APMPOWERSTATUSCHANGE wParam. This wParam indicates that something in the system power has changed and the GetSystemPowerStatus function gives the application access to that information. That information is returned inside the SYSTEM_POWER_STATUS that is passed in.

Initializing the Dependency Properties on Windows XP

As discussed in the Detecting the Operating System section, the dependency properties need to be initialized when the application is loading in order for them to be up-to-date (see Figure 5). This is not necessary in Windows Vista, as the operating system makes the application aware of the current system power status through the window message when the application registers for power notifications. This, however, does not happen on Windows XP and therefore initialization is necessary. This initialization is done in the PowerAwareWindow_Loaded function when the operating system is detected and appears as follows:

void InitializeDependencyPropertiesForXP()

{

if (GetSystemPowerStatus(this._xpPowerStatus))

{

this.RunningOnBattery =

(ACLineStatus.Battery == _xpPowerStatus.ACLineStatus);

this.RemainingBattery = _xpPowerStatus.BatteryLifePercent;

}

this.MonitorOn = true;

this.PowerPlan = PowerPlan.Automatic;

}

At this point it is important to point out that there isn’t a Windows XP monitor or power plan notification analogous to the Windows Vista notification. Therefore, when a Windows XP application attempts to get these values, default values will be returned. And even if the application hooks up the MonitorOnChanged event or the PowerPlanChanged event, after initialization nothing will trigger the firing of the events.

Windows Vista Power Structs

Next up are the Windows Vista power structures. POWERBROADCAST_SETTING is the main Windows Vista power struct:

struct POWERBROADCAST_SETTING

{

public Guid PowerSetting;

public UInt32 DataLength;

}

In the official definition of the struct, there is a third member present: Data. As indicated in the struct, this can vary in size, being either a GUID or a DWORD. Thus, declared within the application is a custom variation of this struct that can hold either of these data types (note that this struct doesn’t mimic the layout of the original native structure and thus can’t be used for marshaling; it’s simply used so that the values can continue to be passed around within the application in a strongly typed fashion):

struct my_POWERBROADCAST_SETTING

{

public IntPtr wParam;

public Guid PowerSetting;

public int PowerChangeData;

public Guid PowerPlanGuid;

}

This struct also includes the wParam so that with this custom variation, all relevant data can be passed from the window message handler to the application’s Windows Vista message handler (defined later) in one struct.

So when receiving the WM_POWERBROADCAST window message on Windows Vista along with a wParam of PBT_POWERSETTINGCHANGE, the lParam associated with the message is a pointer to a POWERBROADCAST_SETTING struct that contains information about the change in the system’s power status.

One last important bit of information concerns how Windows Vista defines Power Plans. Maximum Power Savings involves very aggressive power-savings measures to help stretch battery life:

static Guid GUID_MAX_POWER_SAVINGS =

new Guid(“a1841308-3541-4fab-bc81-f71556f20b4a”);

No Power Savings uses almost no power savings measures:

static Guid GUID_MIN_POWER_SAVINGS =

new Guid(“8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c”);

Finally, the typical Power Savings power plan uses relatively aggressive power savings measures:

static Guid GUID_TYPICAL_POWER_SAVINGS =

new Guid(“381b4222-f694-41f0-9685-ff5bb260df2e”);

These correspond to PowerPlan.PowerSaver, PowerPlan.HighPerformance, and PowerPlan.Automatic, respectively. As mentioned earlier, they appear as Power Saver, High Performance, and Balanced in the Windows Vista UI, respectively.

How the Message Is Handled

The code in Figure 10 provides a high-level look at the MessageProc function. There are two important points to explain here. The first point is how the work is sent to each of the handlers. Second is how (and when) data is grabbed in the event of a WM_POWERBROADCAST message on Windows Vista.

Figure 10 MessageProc Function

IntPtr MessageProc(

IntPtr hwnd, int msg, IntPtr wParam, IntPtr lParam, ref bool handled)

{

switch (msg)

{

case WM_POWERBROADCAST:

if (_osVersion >= VISTA_OS_VERSION)

{

// Grab the relevant power data from the message here.

// Then send the data and work to the Windows Vista

// handler here.

}

else

{

//  Send work to the Windows XP handler here.

}

break;

// Other (non-power) window messages in which the

// application is interested would be processed here

// (if the application needs to do so).

}

return IntPtr.Zero;

}

Normally, sending the work to each of the handlers would be as simple as calling functions defined in the sample code. These functions would handle the message appropriately when the application is running on either Windows XP or Windows Vista. In this case, though, a bit more precaution has been taken to prevent system hangs. If, for example, the function that handles the message is complicated and processor-intensive, then control wouldn’t be returned to the window message handle immediately and this could result in noticeable system hangs. To avoid hangs like this, the work is sent to the Windows Vista and Windows XP handlers via a post to the Dispatcher thread (see Figure 11).

Figure 11 Asynchronous Invocation

delegate void VoidXPDelegate(/*power data arguments*/ );

delegate void VoidVistaDelegate(/*power data arguments*/ );

// A call to the Windows XP handler would look like this:

this.Dispatcher.BeginInvoke(DispatcherPriority.ApplicationIdle,

new VoidXPDelegate(/*XP handler function here*/),

/*power data here*/);

// A call to the Windows Vista handler would look like this:

this.Dispatcher.BeginInvoke(DispatcherPriority.ApplicationIdle,

new VoidVistaDelegate(/*Vista handler function here*/),

/* power data here*/);

The message will be handled asynchronously at the specified priority on the thread with which the Dispatcher is associated; control is returned immediately to the MessageProc function after the work is posted to the Dispatcher. More on the Dispatcher and Windows Presentation Foundation threading can be found in the WPF Fundamentals Threading Model documentation.

Now for the topic of grabbing data for Windows Vista power events. The wParam portion of the WM_POWERBROADCAST window message is crucial to understanding the nature of the message, as explained previously. On Windows XP, this is all that is necessary for successfully handling the power message. On Windows Vista, however, this wParam can indicate that the lParam in the message is a pointer to data necessary for determining the nature of the message. Figure 12 shows how I copy that data out of the lParam pointer. There you can see the MessageProc function with these changes.

Figure 12 Copying Data out of the lParam Pointer

// First grab the wParam, that’s important information to the

// Windows Vista handler no matter what

my_POWERBROADCAST_SETTING myPBS = new my_POWERBROADCAST_SETTING();

myPBS.wParam = wParam;

if (wParam == (IntPtr)PBT_POWERSETTINGCHANGE)

{

// Grab the power change information from the pointer passed

// in the window message

POWERBROADCAST_SETTING ps = (POWERBROADCAST_SETTING)

Marshal.PtrToStructure(lParam, typeof(POWERBROADCAST_SETTING));

// Copy the PowerSetting in the POWERBROADCAST_SETTING to

// our own structure

myPBS.PowerSetting = ps.PowerSetting;

// lParam points to the POWERBROADCAST_SETTING, but we need to

// calculate a pointer to where the data about the setting lives.

IntPtr pData = (IntPtr)((int)lParam + Marshal.SizeOf(ps));

// Use the pointer calculated to grab the relevant data. Note that

// the relevant data can be either a DWORD or a GUID, depending on

// the datalength indicated in the POWERBROADCAST_SETTING struct.

if (ps.DataLength == Marshal.SizeOf(typeof(Int32)))

{

myPBS.PowerChangeData = (Int32)Marshal.PtrToStructure(

pData, typeof(Int32));

}

else if (ps.DataLength == Marshal.SizeOf(typeof(Guid)))

{

myPBS.PowerPlanGuid = (Guid)Marshal.PtrToStructure(

pData, typeof(Guid));

}

}

// At this point ‘myPBS’ contains all the data the Windows Vista handler

// needs for this application to process the system power change

Once you have the relevant data for the message, you need to update the relevant properties and fire the relevant events. Considering all the work that’s been done thus far to get notifications and set up custom dependency properties, this step is actually very straightforward. All it involves is either assigning the data provided by the operating system to the appropriate application variables or raising the appropriate custom events.

Using the Dependency Properties and Events

Now let’s take a look at an application that is hosted within a PowerAwareWindow. At this level, power-awareness fits very well into the Windows Presentation Foundation model and creating a power-aware application that runs on both Windows XP and Windows Vista is quite straightforward.

The first thing to examine is the user interface, as defined in Window1.xaml. Note that the application lives within a Power- AwareWindow:

<local:PowerAwareWindow x:Class=”WPFPower_Aware.Window1”

xmlns=”http://schemas.microsoft.com/winfx/2006/xaml/presentation”

xmlns:x=”http://schemas.microsoft.com/winfx/2006/xaml”

xmlns:local=”clr-namespace:WPFPower_Aware”

x:Name=”MainWindow”

Title=”WPF Power Aware Application” Height=”480” Width=”550”

Loaded=”Window1_Loaded”

>

...

</local:PowerAwareWindow>

As a result, the XAML file can make use of the custom dependency properties for data binding. My sample, for example, contains a status bar at the bottom of the window. Within the status bar, there are two pieces of power-related data—a text block indicating the current system power source and a progress bar indicating the current percentage of remaining battery power. The values assigned to these are data-bound to their corresponding dependency properties.

You’ll find that data binding is quite simple. Consider the progress bar, for example:

<ProgressBar IsIndeterminate=”False” x:Name=”myBatteryProgressBar”

Width=”200” Height=”20”

Value=”{Binding ElementName=MainWindow,

Path=RemainingBattery, Mode=OneWay}” />

With this little bit of markup, the progress bar will always be up-to-date and no codebehind intervention is necessary. You can even data-bind to your own custom control, such as a battery meter, to display this information!

The power-source text block is somewhat more complicated as it involves converting from a Boolean value (RunningOnBattery) to a string for the text block. If you’d like to view the details of this, take a look in the sample code.

Controlling an Animation

TIf you run the sample application, you’ll see two animation elements. The one on the left is controlled in the codebehind, the other is controlled in XAML. The animation that is controlled by XAML starts and stops when the RunningOnBattery property is false and true, respectively.

The snippet in Figure 13 is for an animation that demonstrates the power of custom dependency properties via their use in XAML. With this piece of XAML, no code intervention is necessary to stop and start the animation. And this is just the beginning—there’s much more you can do with animations and triggers in XAML. For more information, see the Windows Presentation Foundation Animation Overview and the Trigger Class documentation.

Figure 13 Custom Dependency Properties in XAML

<Rectangle Name=”rect” Width=”140” Height=”90” Fill=”Blue” Stroke=”Red”>

<Rectangle.RenderTransform>

<ScaleTransform/>

</Rectangle.RenderTransform>

<Rectangle.Style>

<Style>

<Style.Triggers>

<Trigger Property=”local:PowerAwareWindow.RunningOnBattery”

Value=”False”>

<Trigger.EnterActions>

<BeginStoryboard Name=”myStoryboard”>

<Storyboard>

<DoubleAnimation Storyboard.TargetProperty=

“(Rectangle.RenderTransform).(ScaleTransform.ScaleY)”

From=”1” To=”1.5” RepeatBehavior=”Forever”

AutoReverse=”True”/>

</Storyboard>

</BeginStoryboard>

</Trigger.EnterActions>

<Trigger.ExitActions>

<StopStoryboard BeginStoryboardName=”myStoryboard”/>

</Trigger.ExitActions>

</Trigger>

</Style.Triggers>

</Style>

</Rectangle.Style>

</Rectangle>

As I mentioned, the left element in the sample app has an animation that is controlled in the codebehind. When a relevant system power event occurs, the current power state of the system is used to determine if the animation should be paused to conserve power. First, the animation gets set up and then the custom events are hooked up. The necessary code for this is found in Window1 .xaml.cs (see Figure 14).

Figure 14 Code in Window1.xaml.cs

DoubleAnimation myDoubleAnimation;

Storyboard myStoryboard;

void Window1_Loaded(object sender, RoutedEventArgs e)

{

// Initialize the animation

myDoubleAnimation = new DoubleAnimation();

myDoubleAnimation.From = 0.0;

myDoubleAnimation.To = 360.0;

myDoubleAnimation.Duration = new Duration(TimeSpan.FromSeconds(10));

myDoubleAnimation.RepeatBehavior = RepeatBehavior.Forever;

// Note that yRotate is an AxisAngleRotation3D declared

// in the Window1.xaml file

Storyboard.SetTargetName(myDoubleAnimation, “yRotate”);

Storyboard.SetTargetProperty(myDoubleAnimation,

new PropertyPath(AxisAngleRotation3D.AngleProperty));

myStoryboard = new Storyboard();

myStoryboard.Children.Add(myDoubleAnimation);

// Begin the animation if appropriate

if (!ShouldStopAnimation()) myStoryboard.Begin(this, true);

// Hook up the events from the PowerAwareWindow

this.RemainingBatteryChanged +=

new DependencyPropertyChangedEventHandler(

Window1_RemainingBatteryChanged);

this.RunningOnBatteryChanged +=

new DependencyPropertyChangedEventHandler(

Window1_RunningOnBatteryChanged);

this.SystemSuspending += new RoutedEventHandler(

Window1_SystemSuspending);

this.SystemResuming += new RoutedEventHandler(

Window1_SystemResuming);

this.MonitorOnChanged +=

new DependencyPropertyChangedEventHandler(

Window1_MonitorOnChanged);

this.PowerPlanChanged +=

new DependencyPropertyChangedEventHandler(

Window1_PowerPlanChanged);

}

Deciding how and when the application should take measures to conserve power depends largely on the behavior and needs of your application. In this sample, the decision is made as follows, and the power event handlers make use of this function in the appropriate event handlers:

bool ShouldStopAnimation()

{

// If the monitor is off, or if we’re running on a battery

// and the user hasn’t explicited requested high performance,

// stop the animation

return

!MainWindow.MonitorOn ||

(MainWindow.PowerPlan != PowerPlan.HighPerformance &&

MainWindow.RunningOnBattery);

}

One thing to note in this sample is that power events are reported in a list box in the UI. Here, for example, is the RunningOnBatteryChanged event handler:

void Window1_RunningOnBatteryChanged(

object sender, DependencyPropertyChangedEventArgs e)

{

myEventsListBox.Items.Add(

“RunningOnBattery changed - now it’s {0}”, (bool)e.NewValue);

if (ShouldStopAnimation()) myStoryboard.Pause(this);

else myStoryboard.Resume(this);

}

You can learn more about other measures an application can take to conserve power by taking a look in the Mobile PC Power and Device Awareness documentation.

Conclusion

This article and the accompanying code provide a starting point from which you can learn to make your Windows Presentation Foundation applications power-aware. Note that there is a lot more to power awareness and the Windows Presentation Foundation than I’ve discussed here. Therefore, I highly recommend that you review the resources mentioned throughout this article before you design your power-aware application. You’ll discover many more capabilities and techniques that will help you tailor the best solution for your program.

Additional Resources

For general information regarding power and device management, and ideas on how you can reduce power consumption, check out these MSDN links for helpful documentation:

- Mobile PC User Experience Guidelines for Developers: Power and Device Awareness

- Mobile PC Development Guide: Power Management on Windows Vista

- Mobile PC Development Guide: Power Management on Windows XP

- Win32 and COM Development: Power Management

For an overview of the Windows Presentation Foundation features used in this article, check out these links:

- Routed Event Overview

- Custom Dependency Properties documentation

- WPF Fundamentals: Threading Model documentation

- Windows Presentation Foundation Data Binding Overview

- Windows Presentation Foundation: Animation Overview

And for specific definitions of certain functions, classes, and enumerations used in this article, see these links:

- SystemEvents.PowerModeChanged Event documentation

- WM_POWERBROADCAST documentation

- RegisterPowerSettingNotification documentation

- SYSTEM_POWER_STATUS documentation

- POWERBROADCAST_SETTING documentation

- Trigger Class documentation

Andre Michaud is an SDET on the Tablet PC Platform Team at Microsoft where he works with handwriting recognition. His previous experience includes the Tablet PC components of the Windows Presentation Foundation.

Share Code

Write Code Once For Both Mobile And Desktop Apps

Daniel Moth

This article discusses:

- Windows Mobile and Windows desktop development

- Porting applications between platforms

- Retargetable assemblies

- Sharing code between platforms

This article uses the following technologies:

.NET Framework

Code download available at: NetCfSharing2007_07.exe (246 KB)

Browse the Code Online

Contents

Platform Differences

How Not to Proceed

Retargetable

Share the Code, Not the Binary

Project and Code Files in the Same Directory

Stub It

Platform Invoke

Different Framework Implementations

Partial Types

System.Diagnostics.Conditional

Deploy to My Computer

Use Reflection

Extension Methods

Wrapping It Up

For the last several years, while developers were building Microsoft® .NET Framework client applications for Windows®, many had no idea that they could also have been creating applications for Windows Mobile® using the same skills and toolsets. But Windows Mobile wasn’t as widespread in the enterprise then, so the need to write custom applications targeting mobile devices was not as great. Today, there’s a huge demand and many desktop developers are getting their feet wet with mobile development. Unfortunately, many miss the opportunity to share their .NET code cross-platform even though it is relatively easy to do so.

There are many reasons you might want to share business logic across both platforms, regardless of whether the original application was written for Windows or Windows Mobile. You can start from either, and port to the other. If your application is currently running on laptops in the field, it is probably a good candidate to run on a mobile device, and vice versa. And if your product is similar to your competitor’s, extending the reach to a new platform gives you an edge. As a bonus, by writing your code to execute on the desktop you can take advantage of developer tools that don’t work with smart device projects.

The techniques and principles discussed in this article can be utilized when writing cross-device code—applications that target Windows devices with differing form factors (different screen sizes, orientations, touch screens and so on).

Platform Differences

When I think of the .NET Framework, I think of the tools, the languages, the libraries, and the runtime engine. Differences in these four elements affect the goal of writing cross-platform code. Fortunately, the tool we use for both desktop and mobile apps is the same: Visual Studio®. With each version of Visual Studio, the Visual Studio for Devices team continues to add new features for device development and to ensure that important new desktop development features work as expected for mobile device development. (You need to use Visual Studio 2005 Standard or higher for device development.)

Both Visual Basic® and C# are fully supported on both platforms, from a syntactical perspective. But if you plan to use Visual Basic, be advised that the Microsoft.VisualBasic.Compatibility assembly and implicit late binding are not supported. The former is an assembly used on desktop projects that are migrated from Visual Basic 6.0, and the latter is not an issue if you are following best practices by enabling Option Strict On in Visual Basic projects. Also realize that managed C++ is not supported (although the same IDE can be used for native device projects). Third-party languages can add support for the .NET Compact Framework by referencing the correct assemblies and by ensuring that their compilers don’t generate IL opcodes that are not supported by the .NET Compact Framework common language runtime (CLR). The offending IL opcodes are: jmp, calli, refanyval, refanytype, mkrefany, arglist, localloc, unaligned, and tail.

It’s worth noting that while the implementations of the garbage collector and the just-in-time (JIT) compiler are different in the .NET Compact Framework than they are in the .NET Framework, they still serve the same purpose and offer the same services to managed code.

Like any modern programming framework, the .NET Framework comes with a rich set of class libraries. The .NET Compact Framework achieves 80 percent of the relevant functionality found in the .NET Framework with only 20 percent of the footprint. Major areas of the .NET Framework that are absent from the compact version include: ASP.NET, CLR hosting, code access security (CAS), binary serialization, Reflection.Emit, and CodeDOM.

Windows Presentation Foundation and Windows Workflow Foundation are not going to be supported any time soon, but there will be a compact version of Windows Communication Foundation in version 3.5 of the .NET Compact Framework. Also in version 3.5, there will be support for LINQ to Objects, LINQ to XML, and LINQ to DataSet, but no support for any other LINQ flavors.

Most of the time, however, it isn’t these large missing areas that cause issues. Typically, problems arise from the fact that some of the supported namespaces don’t have all the classes you expect and, more importantly, some classes do not have all the members you’d expect. I’ll show you some workarounds later on.

The reverse is also true. That is, the .NET Compact Framework is not a strict subset of the .NET Framework; it actually adds some members of its own. There are four device-specific assemblies: System.Data.SqlServerCe, System.Net.IrDA, Microsoft.WindowsMobile.DirectX and Microsoft.WindowsCE.Forms.

SqlServerCe contains classes for working with the in-memory database on the device, IrDA contains classes for programming against infrared, DirectX® is used for rich graphics primarily for game programming, and Forms has a number of GUI classes that are specific to devices. In addition to these device-specific assemblies, there are also some device-specific Windows Mobile APIs for Windows Mobile 5.0 and higher that ship as part of the Mobile platform. Later I’ll show you how to cope with types that exist on one framework and not the other.

Of course, input methods on desktop and mobile devices are different, so you don’t have to worry about keyboard shortcuts, tooltips, and so forth on a mobile device. Moreover, mobile forms are not resizable and are always full screen, which also affects the way you navigate between forms of an application. Finally, the most obvious difference is that the screen size of devices is much smaller. The takeaway is that it is best not to share the UI layer of an application between these two platforms, but rather create platform-specific user interfaces and instead just share the business logic.

How Not to Proceed

The first thing desktop developers usually do when trying to port an application to a mobile device is to take an assembly built against the desktop framework and reference it from a device project. This will not work and is definitely not supported. Even if in your desktop project you’ve been careful to use only types and members that are also available on the .NET Compact Framework, you will still get TypeLoadException and MissingMethodException exceptions at run time. So, if you created a Visual Studio project by choosing a non-smart-device project template, the output of that project will simply not execute on the Windows CE platform (which is what Windows Mobile builds on, of course).

Unfortunately, it is not hard to find yourself in the situation just described. When referencing assemblies from a smart-device project, there are no checks to see if it’s a desktop assembly, so you could be using class libraries built from a desktop project template. This situation arises often when using third-party DLLs. In such cases, you’ll only discover the errors at run time.

If you suspect you are facing this problem, there are a few clues to look for. When you deploy the project to the target through Visual Studio (such as when you run your project with debugging), notice in the Visual Studio output window the diagnostic messages that tell you which assemblies are being deployed to the device (see Figure 1). If the deployment process is taking too long, it may be because Visual Studio is copying to the target the full set of .NET Framework assemblies that it detected as dependencies to the desktop assembly you accidentally referenced.

Figure 1 Output Window of Visual Studio (Click the image for a larger view)

If you are lucky, the deployment process will fail with an error informing you that there is not enough disk space. If the desktop dependencies are limited, then the deployment process will succeed. In that case, using a remote file viewer that shows all file types, look into the folder on the target where your application was deployed. You’ll find all the .NET Framework assemblies that were deployed. Note that the public key token for the .NET Compact Framework assemblies starts with 9 whereas the desktop equivalents start with B. You can use that information to quickly determine whether what you are deploying or referencing is a desktop or a device framework assembly. That information also appears in the .NET Compact Framework loader log, which will contain errors if your application fails at run time due to these issues.

Retargetable

Though desktop assemblies cannot execute on the device, the good news is that the reverse is possible. If you create a smart device project and build it, the output (the EXE or DLL) can be executed on the desktop and can be referenced by desktop projects. This works because the .NET Compact Framework assemblies are retargetable. If you open one of them with ILDASM, you can observe that they have the System.Reflection.AssemblyNameFlags.Retargetable attribute (see Figure 2). This means that at run time all references to the .NET Compact Framework assemblies are retargeted to the desktop equivalents. So, to be clear, your assembly uses the desktop implementation, not the device implementation of the .NET Framework.

Figure 2 Retargetable Flag in ILDASM (Click the image for a larger view)

Let’s look at an example you can try at home. Create a new smart device application targeting your favorite Windows Mobile platform. Add some controls to the main form, for example a TabControl, a ProgressBar, and four MenuItems to the MainMenu control (see Figure 3). Generate the Click event handler methods for each of the MenuItems and add the code in Figure 4 in these event handler methods.

Figure 4 Click Event Handler Methods

private void mnuFormShow_Click(object sender, EventArgs e)

{

// TODO later

}

private void mnuMbox_Click(object sender, EventArgs e)

{

MessageBox.Show(“some text”, “random caption”);

}

private void mnuDateTime_Click(object sender, EventArgs e)

{

this.Text = DateTime.Now.TimeOfDay.ToString();

}

private void mnuCalc_Click(object sender, EventArgs e)

{

string pth = @”\Windows\calc.exe”;

Process.Start(pth, null);

}

Figure 3 MainMenu Design

Run the project targeting the emulator or your own device and observe that the behavior is what you expect when you click on each menu item. Back on your desktop, navigate to the build folder (bin/Debug) and notice how you can double-click on the executable and it runs on your desktop machine (see Figure 5).

Figure 5 Device Project

The controls render themselves in the desktop style—the TabControl has the tabs at the top, the menus appear at the top, the ProgressBar has a slightly different appearance, and so forth. Now click on the MenuItem that shows a MessageBox and it works as expected. Click on the MenuItem that changes the caption of the form and notice how we get higher precision here with the addition of milliseconds. This is a further clue that you are now using the desktop implementation that is slightly different. The lesson here is not to completely rely on retargetability and instead test your application thoroughly to be positive that any minor differences in implementation are not important to you. Finally, click on the MenuItem that launches the calculator and observe the application crashing with a "System.ComponentModel.Win32Exception: The system cannot find the file specified" exception. Clearly, the culprit here is that the path to the calculator on the desktop is different from the path of the calculator on the device. To deal with such platform-specific issues and to offer a single implementation that works cross-platform, you need to detect at run time what platform you are on and branch your code accordingly. See the following modifications to the event handler as an example:

private void mnuCalc_Click(

object sender, EventArgs e)

{

string pth;

if (Environment.OSVersion.Platform == PlatformID.WinCE)

{

pth = @”\Windows\calc.exe”;

}

else

{

pth = @”c:\Windows\system32\calc.exe”;

}

Process.Start(pth, null);

}

You can now run the project on either Windows Mobile or Windows desktop and the calculator application corresponding to the platform will be launched correctly.

Remember that there are device-specific assemblies that do not exist on the desktop. Let’s see what happens if you use such an assembly. Add a new form (Form2) to the smart device project, drop an InputPanel control on it, then add two Button controls and a TextBox. Generate the event handlers for the controls and add code as shown in Figure 6.

Figure 6 textBox Event Handlers

private void textBox1_GotFocus(object sender, EventArgs e)

{

inputPanel1.Enabled = true;

}

private void textBox1_LostFocus(object sender, EventArgs e)

{

inputPanel1.Enabled = false;

}

private void button1_Click(object sender, EventArgs e)

{

Cursor.Current = Cursors.WaitCursor;

}

private void button2_Click(object sender, EventArgs e)

{

Cursor.Current = Cursors.Default;

}

Now revisit the empty MenuItem Click event handler method of the first form and add code that shows Form2, as follows:

private void mnuFormShow_Click(object sender, EventArgs e)

{

Form2 f = new Form2();

f.ShowDialog();

}

If you run the project on the device, you’ll notice that when you open Form2 from Form1’s menu item, you can click on Form2’s buttons to show or hide the busy cursor on the device. Giving the focus to the Textbox brings up the Soft Input Panel (SIP) while taking the focus away hides it. Back on the desktop, navigate to your build directory like before and run the executable on the desktop. Clicking on the menu item to show Form2 results in an exception like this:

System.IO.FileNotFoundException: Could not load file or assembly

‘Microsoft.WindowsCE.Forms, Version=2.0.0.0, Culture=neutral,

PublicKeyToken=969db8053d3322ac’ or one of its dependencies.

The system cannot find the file specified. File name: ‘Microsoft.WindowsCE.Forms,

Version=2.0.0.0, Culture=neutral, PublicKeyToken=969db8053d3322ac’

This happens because when you added an InputPanel control to the project, a reference to Microsoft.WindowsCE.Forms was added automatically. That’s a device-only assembly so there’s nothing on the desktop to retarget to, thus the exception is thrown when the form gets constructed and attempts to create a Microsoft.WindowsCE.Forms.InputPanel from that assembly. Let’s make some modifications to better observe this. First delete the InputPanel control from the form’s designer view and then modify the textBox event handlers as you see in Figure 7.

Figure 7 Checking Platform

private void textBox1_GotFocus(object sender, EventArgs e)

{

if (Environment.OSVersion.Platform == PlatformID.WinCE) //naive

{

Microsoft.WindowsCE.Forms.InputPanel inputPanel1 =

new Microsoft.WindowsCE.Forms.InputPanel();

inputPanel1.Enabled = true;

}

}

private void textBox1_LostFocus(object sender, EventArgs e)

{

// in the interest of space, let’s ignore this method

//inputPanel1.Enabled = true;

}

What this does is isolate the use of the InputPanel to one method, and it also attempts a naïve fix for the exception by checking the platform at run time. If you run this project on the device, it still works as expected, but if you run it on the desktop it fails with the same exception as before. It fails only when you give the focus to textBox1 rather than as soon as the form is shown, but it still fails nonetheless.

A runtime check is not enough in this case because of JIT compiling. At run time when the method call is made, the CLR checks to see if any native code exists for the method being called (textBox1_GotFocus). If not, then the JIT compiler will generate native opcodes from the IL code that the method entry points to. At that point, the JITer does not "care" what paths get executed at run time, it simply needs to find all the types used in the method. So it tries to load the Microsoft.WindowsCE.Forms.InputPanel type and corresponding assembly, but it can’t because the assembly doesn’t exist on the desktop. Then the exception gets thrown. You could extract the use of the InputPanel into a separate method and ensure with a run-time check from the calling method that the extracted method never gets called, but that’s really messier than most developers would like. The correct way to resolve this issue is by exploring an alternative approach to sharing code cross-platform. (If you are interested in pursuing the previously mentioned approach, see the "Platform Detection" section of msdn2.microsoft.com/ aa480686, which has an example of a factoring approach similar to the one noted here.)

Share the Code, Not the Binary

The previous section explored how you could create smart device projects and use the output binaries on the desktop via retargeting. However, there are two limitations with the retargetable approach: First, you can’t use any desktop-only types or methods in your code. They simply will not compile. Second, you must be careful not to load any device-specific assemblies on the desktop as they will fail with run-time exceptions.

Both of these problems can be solved by using conditional branching of the code at compile time rather than at run time, and by creating two projects, with two separate outputs to achieve platform-specific assemblies that both use a single set of source code files. The latter is an important point since ultimately the goal is to have all modifications to the business logic propagate to both platforms simultaneously without any copy/paste error-prone overhead. Let’s try another approach.

First, create a new desktop Windows project. Delete the code files added by default (the Form1 and Program files). From the project menu select "Add Existing Item..." and browse to the folder of the smart device project created earlier, where the code files reside. Finally, select all the code files only ( *.cs) and add them to the project as links. This is important because if you select the default Add option of the dialog, you will be making copies rather than linking to the files (see Figure 8).

Figure 8 Add Existing Files

Build the project and notice that you now find at compile time the error that previously was only found at run time (see Figure 9). Of course, in this case there’s no point adding a reference to the device-specific assembly, so this is where I will introduce conditional compilation.

Figure 9 Finding Problems at Compile Time (Click the image for a larger view)

Go to the desktop project properties and under the Build tab add a new "Conditional compilation symbol" named FULL_FRAME (short for "full framework") as in Figure 10.

Figure 10 Conditional Compilation Constant (Click the image for a larger view)

Next, fix the compilation error in Form2.cs by using the newly introduced compilation constant. Modify the offending method as follows and then rebuild the desktop project:

private void textBox1_GotFocus(object sender, EventArgs e)

{

#if !FULL_FRAME

Microsoft.WindowsCE.Forms.InputPanel inputPanel1 =

new Microsoft.WindowsCE.Forms.InputPanel();

inputPanel1.Enabled = true;

#endif

}

Now that this (desktop) project builds, you can run it and observe how everything works as expected. If you switch to the device project, note how it warns you that a source file has been changed outside of that instance of Visual Studio and asks if you want to reload it; click on "Yes."

Since this is a separate project using a separate set of conditional compilation constants, the project will compile and run fine on the device—you will not be running the output of this project on the desktop anymore! This means that you can change the runtime conditional introduced earlier when launching the calculator, to a compile-time conditional thus reducing the size of the method.

Now that you’ve seen the mechanics of using conditional compilation, let’s reflect on the benefits. Having two separate projects means you can have a different set of project references in each. You can also include code files in one project and not in the other. On the desktop side, you can take advantage of members of the full framework in order to create a good experience for the desktop user; simply ensure that usage of those desktop members is enclosed in an #if preprocessor directive or is separated out into source files that exist only in the desktop project. There are many examples in which you can use full framework members to enhance the experience on the desktop side, so let’s take a look at one.

In the Program.cs file, modify the Main function as follows:

static void Main()

{

#if FULL_FRAME

Application.EnableVisualStyles();

#endif

Application.Run(new Form1());

}

Now run the project and compare the appearance of the original control with the one resulting from the retargetable approach (see Figure 11).

Finally, the astute among you will have observed that the code that shows and hides the cursor will not work on the desktop (even though it produces no compilation or runtime error). Also note that device projects, starting with Visual Studio 2005, have certain conditional compilation constants already defined: PocketPC, Smartphone, Windows CE as appropriate. You can choose to use only those built-in constants or your own, such as the FULL_FRAME that was introduced, or a combination of both—it doesn’t really matter.

Figure 11 Original and Retargeted Controls (Click the image for a larger view)

Now that you understand both approaches to sharing code assets cross-platform, it is time to examine some tips and tricks. Some of these apply to the retargetable approach, some to the conditional compilation, and some to both.

Project and Code Files in the Same Directory

When sharing code, I created two separate projects and added as links the source code files of the first to the second. I prefer having all source code files reside in the same directory as the project files, thus making it very easy in Visual Studio Solution Explorer to include or exclude files as appropriate. To achieve this, after creating say the device project, then create a Windows desktop project, and delete the code files as before. Rather than adding any more files, close Visual Studio and navigate to the directory where the desktop project file was created. Cut the project file and paste it in the same directory where the device project file resides. For example, if your device project was at "C:\Temp\", then cut and paste the desktop project file (csproj) into that folder (ensure that desktop and device projects have different names).

Now open the desktop project and "Show All Files" in Solution Explorer. You can see all the device source code files and include in the project whichever you think are applicable—and vice versa if you open the device project (see Figure 12).

Figure 12 Choosing Files

With this approach, when adding a new file to one of the projects, it becomes very easy to spot when you open the other project type and it is easier to include the file if you wish.

Note that the project and output assembly names are different, but the namespace and type names should remain the same. If you prefer having the same assembly name for both platforms, then in project properties you need to rename the build folders, for example, binCF and binFF. I also recommend having only one project open at any time as this avoids confusion and it works very well with source control.

Stub It

Many times it is best to avoid littering the code with conditional compilation and instead to add dummy empty stubs of the code that is not applicable on the platform. For example, if you make heavy use of a class throughout the project and that class does not apply on the other platform, you can add the class with the same (but empty) interface. Here’s an example using the InputPanel class:

namespace Microsoft.WindowsCE.Forms

{

public class InputPanel

{

public bool Enabled;

}

}

Now, wherever the InputPanel is used, the code can stay the same with no conditional branching; it will simply have no effect on the desktop platform.

Platform Invoke

Many .NET Compact Framework applications achieve additional functionality by using P/Invoke services to access functionality residing in native DLLs. Typically the names of those DLLs are different on Windows CE and desktop versions of Windows. There are a couple of ways to cope with that, depending on which of the earlier approaches you chose. If you are using conditional compilation, the code in Figure 13 illustrates how to write cross-platform P/Invoke declarations. The technique in Figure 14 is useful if you have chosen the retargetable approach.

Figure 14 Run-Time P/Invoke Decision

class PInvokesRuntime

{

[DllImport(“coredll.dll”, EntryPoint = “QueryPerformanceCounter”)]

internal static extern int QueryPerformanceCounterCE(

out Int64 perfCounter);

[DllImport(“kernel32.dll”, EntryPoint = “QueryPerformanceCounter”)]

internal static extern int QueryPerformanceCounterFull(

out Int64 perfCounter);

private void UseIt()

{

long p;

int i;

if (Environment.OSVersion.Platform == PlatformID.WinCE)

{

i = QueryPerformanceCounterCE(out p);

}

else

{

i = QueryPerformanceCounterFull(out p);

}

}

}

Figure 13 Compile-Time P/Invoke Decision

class PInvokesCompileTime

{

#if FULL_FRAME

private const string DllName = “kernel32.dll”;

#else

private const string DllName = “coredll.dll”;

#endif

[DllImport(DllName)]

internal static extern int QueryPerformanceCounter(

out Int64 perfCounter);

private void UseIt()

{

long p;

int i;

i = QueryPerformanceCounter(out p);

}

}

Different Framework Implementations

Earlier you saw how System.DateTime.TimeOfDay returns a value of different precision depending on which framework you use. So after testing you can decide how to deal with that in your code if it matters to you. There are other examples where methods behave differently and in particular there are a few cases where methods on one framework throw different exceptions than methods on the other framework, for example:

private void button1_Click(object sender, EventArgs e)

{

Type t = this.GetType();

MethodInfo m = t.GetMethod(“DoIt”);

m.Invoke(this, null); //throws

}

public void DoIt()

{

throw new InvalidOperationException(“my message”);

}

This code results in an exception on both platforms on the line that invokes the MethodInfo. On the device, the exception thrown is the InvalidOperationException. On the desktop, however, the exception thrown is a TargetInvocationException (that has the original InvalidOperationException set as the InnerException property). Therefore, during testing you should test the exceptional cases as well and ensure that you have a catch block for both Exception types.

Another example of different framework implementations is what you saw earlier with displaying the busy cursor. What works for the device has no effect on the desktop and what is required on the desktop is not supported on the device. If you want consistent behavior across both platforms, you could extract a single method and isolate there the conditional behavior that encapsulates the display of the cursor as follows:

public static class TheCursor

{

public static void CursorCurrent(Control c, Cursor defaultOrWait)

{

#if FULL_FRAME

if (c != null) c.Cursor = defaultOrWait;

#else

Cursor.Current = defaultOrWait;

#endif

}

}

Then the calling code can remain the same across both the device and the desktop platforms:

private void button1_Click(object sender, EventArgs e)

{

TheCursor.CursorCurrent(this, Cursors.WaitCursor);

}

private void button2_Click(object sender, EventArgs e)

{

TheCursor.CursorCurrent(this, Cursors.Default);

}

Partial Types

Earlier I mentioned that you can include source code files in one project and not in the other if they contain platform-specific functionality. A similar scenario occurs when a class is applicable to both platforms but some members of it are not. Instead of wrapping large sections of the class with a #if directive you can use the partial classes feature introduced with Visual Studio 2005. In other words, split the class across two files using the partial keyword, thus separating out the functionality that is applicable to just one platform into a different file. Then simply do not include that file in the inapplicable project.

System.Diagnostics.Conditional

A tip worth noting is to use the System.Diagnostics.ConditionalAttribute for methods you don’t want to be called at run time. The result of applying the attribute to a method is that any calls to the method will be left out of the compiled binary. The method that has the attribute applied must still be able to compile, of course:

[Conditional(“FULL_FRAME”)]

public void SomeDesktopOnlyMethod()

{

//TODO code that compiles in both projects

}

If you enclosed the entire method with a #if, you would have to find all places where the method was called and wrap the call with a #if as well. Instead here, you simply apply the attribute to the undesired method and ignore all the call sites; they will not be compiled in the binary.

Deploy to My Computer

If you choose the retargetable approach, you know that you can only find desktop platform errors at run time. The question of how to debug the application on the desktop arises, since a smart device project only deploys to the emulator or physical device. There are a couple of approaches you can take.

The easiest, albeit most tedious approach, is to run the application from the file system and then attach the debugger to it via the Visual Studio Tools | Attach to Process menu (see Figure 15). You can also semiautomate this process by creating a desktop project to act as the launcher. Add the desktop project to the solution with the smart device project, set it as the startup project, and reference the smart device project. Ensure the desktop project has just a single Program file with a single static Main function and from there pass to the Application.Run method a new instance of the startup form of the smart device project.

Figure 15 Attach to Process Dialog (Click the image for a larger view)

It would be ideal if in the deployment options of the smart device project there was a "Deploy to My Computer"choice. In fact, during an early Community Technology Preview of Visual Studio 2005, that option existed (but it was dropped from the final product, and the option is not coming back in the Visual Studio "Orcas" release).

You can re-enable the "Deploy to My Computer" functionality by following the steps described on my blog, but please note that is not recommended or supported by Microsoft.

Use Reflection

If you have chosen the retargetable approach, then you can’t use desktop-only methods since they will not compile in the device project. However, if you really want to, you can invoke such methods using reflection. Revisiting the earlier example where I wanted to enable themes for the application, I could have achieved this as follows:

static void Main()

{

//Application.EnableVisualStyles();

if (Environment.OSVersion.Platform != PlatformID.WinCE)

{

Type t = typeof(Application);

MethodInfo m = t.GetMethod(“EnableVisualStyles”);

m.Invoke(null, null);

}

// As before

Application.Run(new Form1());

}

Extension Methods

While using .NET Compact Framework classes, the occasion often arises that a particular class has members missing compared to the full Framework. Typically, developers have to write the missing functionality themselves and then decide whether to use inheritance if possible to add the missing members or add them to a utility helper class. For example, assume you want to perform a Cut operation on a TextBox in a device project that you are sharing with the desktop. The Cut method is missing from the TextBox class in the .NET Compact Framework. Based on what you’ve learned in this article, you might add a code file to the device project that looks like Figure 16.

Figure 16 TextBoxExtensions

using System;

using System.Windows.Forms;

using Microsoft.WindowsCE.Forms;

static class TextBoxExtensions

{

const int WM_CUT = 0x0300;

const int WM_COPY = 0x0301;

const int WM_PASTE = 0x0302;

public static void Cut(TextBox t)

{

Message msg = Message.Create(

t.Handle, WM_CUT, IntPtr.Zero, IntPtr.Zero);

MessageWindow.SendMessage(ref msg);

}

}

The file in Figure 16 needs no conditionals as it is only present in the device project. You could then write the calling code (that will exist in both projects) as follows:

private void SomeMethod()

{

#if FULL_FRAME

textBox1.Cut();

#else

TextBoxExtensions.Cut(textBox1);

#endif

}

There is a new feature called Extension Methods in C# 3.0 and in the upcoming version of Visual Basic. I don’t have space here to fully describe the feature, however, so if you are not familiar with it please visit my blog or see Anson Horton’s article in the June 2007 issue of MSDN® Magazine. The feature is also supported in .NET Compact Framework 3.5 that ships with Visual Studio code-named "Orcas" and it helps address the situation described above.

In the device project add a reference to System.Core.dll and then modify the device-only file by adding the this keyword before the first parameter of the helper method, thus making it an extension method, like so:

public static void Cut(this TextBox t)

With that change in place, the calling code can be the same cross-platform with no conditionals:

private void SomeMethod()

{

textBox1.Cut();

}

On the desktop, the framework method will be called, and on the device the extension method will be called. If in the future the .NET Compact Framework team adds the method, then the code will seamlessly call the real .NET Framework method (as instance methods take precedence over extension methods), and at some point you can remove the superfluous code file.

Wrapping It Up

In this article I looked at the differences between developing for the desktop and developing for Windows Mobile, concluding that most of the differences are in the Framework libraries rather than anywhere else. I offered examples of two approaches for sharing code assets cross-platform, indicated my preference for the conditional compilation approach, and suggested some handy tips. While I personally recommend that you use this knowledge to share business logic contained in class libraries, you may wish to share your GUI as well, especially if the application is targeting devices such as the Ultra Mobile PC (UMPC) or other touch-screen devices running Windows desktop rather than Windows Mobile.

Daniel Moth works for Microsoft in the Developer and Platform group in the UK. He is a coauthor of Microsoft Mobile Development Handbook (Microsoft Press, 2007) and can be reached via his blog at www.danielmoth.com/Blog.

NTFS

Enhance Your Apps With File System Transactions

Jason Olson

This article discusses:

- The Kernel Transaction Manager

- The Distributed Transaction Coordinator

- Using Transactional NTFS from native code

- Using Transactional NTFS from managed code

This article uses the following technologies:

Windows Vista, .NET Framework

Code download available at: TxF2007_07.exe (159 KB)

Browse the Code Online

Contents

The Benefits of Transactional NTFS

When to Avoid Transactional NTFS

Getting Started

Secondary Resource Managers

Management

What Happened to the Transacted Command Line?

In Closing

I t’s a revolutionary step for Windows®-based applications. And it can help you develop applications that are more robust than they could ever have been before. It’s Transactional NTFS, also known as TxF. In this article, I explain Transactional NTFS, why it matters to developers, and how you can use this new technology in your applications.

What is TxF? Conceptually, it is quite simple and can be expressed in a logical equation: Transactional NTFS = Transactions + NTFS. A new feature in Windows Vista™ and the next version of Windows Server®, code-named "Longhorn," TxF introduces the concept of transacting file operations.

When I say transacted file operations, I am talking about transactions with fully ACID semantics that can be exposed directly to the developer. This means the ability to perform actions that are fully atomic, consistent, isolated, and durable is now enabled directly for file operations. These ACID capabilities are something that database developers have benefited from for years, and having this power for file operations is a very welcomed addition.

You’re probably wondering if all these transactional capabilities have been built into NTFS, what is the performance overhead of Transactional NTFS? Well, TxF has a strictly pay-to-play model. If you aren’t using transacted file operations, there is no overhead. And even if you are using transacted file operations, the overhead is pretty minimal (in the 1 to 2 percent range). For an application that is not I/O-bound, this performance impact probably would not even be noticeable.

Here’s another aspect of performance to be aware of: Transactional NTFS is optimized for commit. When you write new pages to disk within a transaction, they are written directly in place while the previous pages are saved for retrieval by other readers if necessary. Because of this optimization, when the transaction is committed, the altered pages are already in place on disk. As a result, the overhead of committing the transaction is minimal.

The Benefits of Transactional NTFS

Transactional NTFS sounds nice, in theory. But why should you, as a developer, use TxF? There are a number of distinct reasons. Specifically, TxF achieves three goals that help you develop powerful applications:

- Improved application stability

- Improved platform stability

- Increased innovation

Let’s take a closer look at how TxF achieves these goals.

Transactional NTFS enables better application stability by reducing or eliminating the amount of error-handling code that needs to be written and maintained for a given application. This ultimately reduces application complexity and makes the application easier to test. Say, for instance, you are developing a document management system where a SQL data source needs to be kept consistent with a file store on disk. Ensuring this consistency can be tricky and non-trivial in a non-transactional system. Without transactional file operations, it would be nearly impossible to account for every possible failure scenario, up to and including the operating system crashing at any imaginable point during the process.

One of the ways this was handled in the past was by storing the new version of the file with a temporary file name, writing the new data to the SQL database, and then renaming the temporary file to the real file name when committing the SQL transaction. But consider what happens if the application crashes or there is a power outage right after committing to the SQL database but before the file is renamed. Not only would you have an inconsistent data set, but you would also have an artifact on the file system that you would have to clean up at some point. As usual, the extremely difficult part lies in the details of how many different ways the process can fail.

Some solutions will implement their own entire two-phase commit protocol, which commits changes to the database and the file system at the same time. While this gets you a step closer to achieving the goal of application stability, implementing a custom two-phase commit system in your own application can be very difficult. And this requires code that you have to write, test, and maintain yourself.

Wouldn’t it be better if transaction management was built into the platform rather than having to be implemented by each developer who needs this type of behavior? With the introduction of TxF in Windows Vista and Windows Server "Longhorn," these capabilities are now built directly into the platform. And because Transactional NTFS is embedded in the system itself, it is capable of providing a level of integration that would not otherwise be possible for your applications.

One of the coolest parts about Transactional NTFS is that it can work with a large number of other transactional technologies. Because TxF uses the new Kernel Transaction Manager (KTM) features, and because the new KTM can work directly with the Microsoft® Distributed Transaction Coordinator (DTC), any technology that can work with DTC as a transaction coordinator can use transacted file operations within a single transaction. This means that you can now enlist transacted file operations within the same transaction as SQL operations, Web service calls via WS-AtomicTransaction, Windows Communication Foundation services via the OleTransactionProtocol, or even transacted MSMQ operations. You can even enlist transacted file operations directly with other technologies that are using XA-Transactions since DTC can also work with XA-Transactions. Simply put, if DTC can work with something, you can incorporate transacted file operations into it.

Using the document management example, how can you ensure consistency within a document management application more easily with TxF? This is where the DTC comes in handy. To absolutely ensure consistency between your SQL database and your file store, you can start a transaction, perform your SQL statements and file operations within that same transaction, and then commit or roll back the complete transaction based on the outcome. If your SQL call fails, your file will never be written. If your file system call fails, your SQL is rolled back. Everything remains consistent, and all of this is handled automatically by the platform since the operations are enlisted within a transaction.

When using this approach in my own application, I no longer have to worry about handling and accounting for every possible exception that could occur. I simply incorporate both operations within the same transaction, guaranteeing that if any exception is thrown, the platform will roll back the operations in question. Less code, more robust, very cool!

That leaves two more goals. As for platform stability, this is achieved by some of the ways that Microsoft is using TxF in its own technologies. There are three core features in Windows Vista and Windows Server "Longhorn" that now make use of Transactional NTFS: Windows Update, System Restore, and Task Scheduler. All of these use TxF to write files to the file system within the scope of a transaction in order to handle rollback/commit in case of any exceptions, such as a system reboot due to a loss of power. If you’ve ever experienced a loss of power right in the middle of Windows Update, you know this can leave your system dead and in need of a fresh installation. But now that Windows Update uses TxF, if this same situation occurs on your Windows Vista system, the computer can recover by rolling back the transaction when the system reboots. By adopting TxF internally, Microsoft has helped make its own operating system more stable.

Finally, TxF drives innovation by providing a framework for using transactions outside of SQL calls. Ultimately, Transactional NTFS can fundamentally change the way developers write applications, allowing them to build more robust code. By incorporating transactions into your design, you can write code without having to account for every single possible failure that can occur. The operating system will take care of those mundane details!

When to Avoid Transactional NTFS

Despite all the benefits offered by TxF, there are some scenarios in which you should think about not using it. These are the same scenarios that are discussed when talking about avoiding traditional database transactional systems.

You shouldn’t use TxF if you have a system that will have long-running transactions. The definition of "long-running" here is relative, however. A long-running transaction is any transaction that has been alive longer than many other transactions within the same log. This could mean a few seconds if there are thousands of transactions happening and lasting for very brief periods of time. On the other hand, a day or two might not be a long time if there are very few transactions happening in the system.

You should also be wary of using TxF if you are going to have multiple writers against the same file. Transacted file operations only allow for one writer at a time (even shared writers), meaning that a second writer won’t be allowed to open the file while it is being held open by another writer, unless that second writer is within the same transaction. Because of this, TxF can introduce contention into a system that has multiple writers against a single file.

Finally, you need to be careful about auto-started services and mixing transaction resources. If the service is using Transactional NTFS and SQL, the files it needs access to could remain locked until recovery completes (since your service has a direct dependency on DTC and SQL, those services need to be started first, so you might not be able to fully recover your files until DTC and SQL finish recovering first).

Getting Started

Now let’s take a look at how you can use TxF. If you are at all familiar with the existing file I/O Win32® APIs, the Transactional NTFS APIs should come naturally. The new Transactional NTFS APIs are unchanged from their non-transacted counterparts aside from the addition of a couple of parameters at the end, which are specifically for transactions. Figure 1 gives some examples of Win32 file APIs and their transacted counterparts. The complete list is much longer; nearly every file API in Win32 has a corresponding transacted version.

Figure 1 Win32 APIs and Their Transacted Counterparts

Non-Transacted API

Transacted API

CreateFile

CreateFileTransacted

CopyFileEx

CopyFileTransacted

MoveFileWithProgress

MoveFileTransacted

DeleteFile

DeleteFileTransacted

CreateHardLink

CreateHardLinkTransacted

CreateSymbolicLink

CreateSymbolicLinkTransacted

CreateDirectoryEx

CreateDirectoryTransacted

RemoveDirectory

RemoveDirectoryTransacted

From an application perspective, there are three primary vehicles you can use when developing a TxF-aware application. You can use the new KTM directly as the transaction coordinator. This lets you develop applications that are solely using TxF file operations within their transactions.

The second option is to use DTC as the transaction coordinator. When you use this method, you need to go through DTC to get the handle for the kernel-level transaction object that you can then pass into the Transacted APIs.

The third way is actually an extension to the second approach where you are using System.Transactions from managed code to make use of DTC (or just the Lightweight Transaction Manager if there is only a single durable enlistment). In a moment, I’ll look at the challenges of this approach and what it really means to managed developers.

Right now, let’s take a closer look at these three approaches. Figure 2 illustrates all the major moving parts in the transaction platform in Windows Vista and Windows Server "Longhorn."

Figure 2 The Transaction Platform (Click the image for a larger view)

At the API level, there are three main interfaces into TxF. The first way is to go directly to KTM via the new ktmw32.h header file. At the most basic level, this contains the transaction coordinator Win32 calls for KTM: CreateTransaction, CommitTransaction, and RollbackTransaction. In this approach, you use these calls directly and have KTM itself do the transaction coordination if you are only doing transacted file operations within the single transaction (see Figure 3).

Figure 3 TxF from Native Code with KTM

hTx = CreateTransaction(NULL, NULL, 0, 0, 0, 0, L”MyTxn”);

hFile = CreateFileTransacted(myFileName,

GENERIC_READ | GENERIC_WRITE,

FILE_SHARE_READ | FILE_SHARE_WRITE,

NULL, OPEN_ALWAYS, FILE_ATTRIBUTE_NORMAL, NULL,

hTx, NULL, NULL);

... // work with the file using its file handle hFile

CommitTransaction(hTx);

The second interface is via DTC. With this approach, you use DTC as the transaction coordinator. To use TxF through DTC, you make use of COM and QueryInterface to grab the IKernelTransaction interface directly from DTC’s ITransaction interface. Once you have a reference to the new IKernelTransaction COM interface, you can call its GetHandle method in order to retrieve the kernel-level transaction handle that can then be used to pass to the various Transacted APIs in Win32 (see Figure 4).

Figure 4 TxF from Native Code with DTC

hr = DtcGetTransactionManagerEx( NULL, NULL,

IID_ITransactionDispenser, OLE_TM_FLAG_NONE,

NULL, (void**) &pITransactionDispenser);

hr = pITransactionDispenser->BeginTransaction(

NULL, ISOLATIONLEVEL_READCOMMITTED,

ISOFLAG_RETAIN_BOTH, NULL, &pITransaction);

hr = pITransaction->QueryInterface(

IID_IKernelTransaction,

(void**) &pKernelTransaction);

hr = pKernelTransaction->GetHandle(&hTransactionHandle);

hAppend = CreateFileTransacted(TEXT(“test.txt”),

FILE_APPEND_DATA, FILE_SHARE_READ,

NULL, OPEN_ALWAYS,

FILE_ATTRIBUTE_NORMAL, NULL,

hTransactionHandle,    NULL, NULL);

... // work with the file using its file handle hAppend

hr = pITransaction->Commit(FALSE, XACTTC_SYNC_PHASEONE, 0);

If you are a managed developer, the solution isn’t quite as elegant. Currently, to use TxF from managed code, you have to use P/Invoke and COM interop. Don’t let that scare you away, though. For a large number of uses, the P/Invoke and COM interop code isn’t too bad. If you’re still too timid, you can grab some sample code from the MSDN® Magazine Web site. The sample has a simple managed wrapper around Transactional NTFS that you can freely use in your own applications. (While you’re playing with this sample code, you should also check out the screencast. It demonstrates using TxF with just KTM, with SQL, and with Windows Communication Foundation.) In the short term, a more robust version of this wrapper will be included in an upcoming release of the Windows VistaBridge samples as part of the Windows SDK. In the longer term, the product team is currently working on a plan to integrate these APIs directly into the Microsoft .NET Framework.

To use TxF from managed code, you can continue to use the functionality of the System.Transactions namespace. To dig down into TxF, you are essentially following the same path that you would when using DTC in native code (see Figure 4), but with the managed code equivalents. From an object level, it looks something like Figure 5.

Figure 5 TxF from Managed Code

The first thing you need to do is grab the IDtcTransaction COM interface from the current Transaction (this can be done using the TransactionInterop helper class in System.Transactions). Once you have the IDtcTransaction, you can cast it to an IKernelTransaction COM interface, which under the hood will QueryInterface for the IKernelTransaction COM interface. Then, once you have a reference to the currently running kernel-level transaction via IKernelTransaction, you call GetHandle on it to get a handle that you can pass to the various new Transacted Win32 APIs. For example, you can pass this transaction handle to CreateFileTransacted in order to obtain a SafeFileHandle for a new file. This SafeFileHandle can then be passed to the constructor of the StreamWriter class, which is then used to write to the file (see Figure 6).

Figure 6 TxF from Managed Code with System.Transactions

NativeMethods.cs

// IKernelTransaction COM Interface

[Guid(&quot;79427A2B-F895-40e0-BE79-B57DC82ED231&quot;)]

[InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]

public interface IKernelTransaction

{

int GetHandle(out IntPtr pHandle);

}

[DllImport(KERNEL32,

EntryPoint = &quot;CreateFileTransacted&quot;,

CharSet = CharSet.Unicode, SetLastError = true)]

internal static extern SafeFileHandle CreateFileTransacted(

[In] string lpFileName,

[In] NativeMethods.FileAccess dwDesiredAccess,

[In] NativeMethods.FileShare dwShareMode,

[In] IntPtr lpSecurityAttributes,

[In] NativeMethods.FileMode dwCreationDisposition,

[In] int dwFlagsAndAttributes,

[In] IntPtr hTemplateFile,

[In] KtmTransactionHandle hTransaction,

[In] IntPtr pusMiniVersion,

[In] IntPtr pExtendedParameter);

...

Program.cs

...

using (TransactionScope scope = new TransactionScope())

{

// Grab Kernel level transaction handle

IDtcTransaction dtcTransaction =

TransactionInterop.GetDtcTransaction(managedTransaction);

IKernelTransaction ktmInterface = (IKernelTransaction)dtcTransaction;

IntPtr ktmTxHandle;

ktmInterface.GetHandle(out ktmTxHandle);

// Grab transacted file handle

SafeFileHandle hFile = NativeMethods.CreateFileTransacted(

path, internalAccess, internalShare, IntPtr.Zero,

internalMode, 0, IntPtr.Zero, ktmTxHandle,

IntPtr.Zero, IntPtr.Zero);

... // Work with file (e.g. passing hFile to StreamWriter constructor)

// Close handles

}

You should also know that TxF has read-committed consistency. This has impact on both writing and reading a file transactionally. When writing a file, only a single transaction can be writing to the file (note that we are talking about a single transaction, not a single writer). Non-transacted writers are blocked by the transacted writer, even if opened with share-write access. Additionally, a non-transacted writer prevents any transacted reader or writer from being opened on the file.

When reading from a file transactionally, there is an isolated view of the file for the life of the file handle. This isolated view of the file is based on the most recent contents of the file when the file handle was created. If a transacted writer is in the middle of modifying the file, those modifications won’t be picked up by a new transacted reader until the changes are actually committed. In addition, because of the read-committed consistency of Transactional NTFS, this isolated view of the transacted reader remains even if a transacted writer changes a file and commits those changes anytime during the life of the transacted reader. These transacted readers block non-transacted writers but only while their file handle is open. They do not block modification of the file for the life of the transaction. If you need a consistent view of the file throughout the life of the transaction, you need to keep the transacted reader’s file handle open.

Finally, developers should be aware that when using file handles for transacted file operations, there is a distinct difference between transacted and non-transacted file operations with regard to file handles. If you get a file handle from one of the new Transacted APIs, the file handle could become invalid at any time, whether from a transaction committing or a transaction rolling back.

Secondary Resource Managers

Secondary resource managers allow applications to manage their own resources running within a transaction, particularly their own transaction logs, as well as other possible file operations within a directory (and all subdirectories) specific to the application.

I have a good reason for discussing secondary resource managers in this article. When TxF was developed, there was an emphasis placed on availability over consistency. Since it is now possible within a transaction to modify files that are necessary to boot, the operating system has to guarantee that these files are always available. In order to guarantee these files are always available, the system drive’s transaction manager is required to be the transaction coordinator. Because of this, you cannot use a different transaction coordinator, like DTC for instance, as the "root" or superior transaction coordinator for these files. As a result, a transaction that needs to use DTC with other resource managers (such as SQL) cannot use TxF on the system drive. This means that if you want to use Transactional NTFS on the system drive within the same transaction as operations against SQL Server™, for example, you must use a different strategy.

The simplest way to get around this limitation is to place all of your data on a separate partition. If that is not possible, or you cannot guarantee that the data will be on a non-system partition, you can use a secondary resource manager. When using a secondary resource manager to manage transactional file operations for a given directory (and any subdirectories), you are informing KTM that the application will control resource management for any files residing there. This allows the default resource manager on the system volume to continue to be available and, hence, allows a transaction coordinator other than KTM to become the superior transaction coordinator.

At the command line, you can create and manage secondary resource managers using the FSUtil tool and the commands shown in Figure 7. However, for applications that need to use a secondary resource manager, it is better to control the secondary resource manager programmatically. To do this, you can use the DeviceIoControl Win32 API with specific dwIoControlCodes, outlined in Figure 8.

Figure 8 Manage Secondary Resource Managers Programmatically

Control Codes

Purpose

FSCTL_TXFS_LIST_TRANSACTIONS

Lists all transactions currently involved in the specified resource manager.

FSCTL_TXFS_LIST_TRANSACTION_LOCKED_FILES

Lists all files currently locked by the specified transaction.

FSCTL_TXFS_GET_METADATA_INFO

Retrieves Transactional NTFS metadata for a file, as well as the ID of the transaction that has the file locked.

Figure 7 Manage Secondary Resource Managers from the Command Line

Command

Purpose

fsutil resource create

Creates a secondary transactional resource manager.

fsutil resource info

Displays information relating to a transactional resource manager.

fsutil resource setautoreset

Sets whether a transactional resource manager will clean its transactional metadata the next time it starts.

fsutil resource setlog

Changes characteristics of a running transactional resource manager.

fsutil resource start

Starts a transactional resource manager.

fsutil resource stop

Stops a transactional resource manager.

When using secondary resource managers, the startup and recovery of the resource manager must be done by your application. The act of starting the resource manager creates the KTM objects to manage transactions, and opens up the Common Log File System (CLFS) log and metadata that is stored locally with your directories; the act of recovering the resource manager recovers all files to a consistent state within the secondary resource manager. The resource manager will stop when the last handle to the resource manager root goes away. As a result, files within a secondary resource manager are not automatically made consistent at system startup. Instead, that is done when your application starts and recovers the secondary resource manager. In addition, files are not automatically made consistent when the system is backed up with any software that uses the volume shadow copy service (VSS) unless you provide a VSS writer. Of course, if you restore an inconsistent secondary resource manager and then start and recover it, the files will be made consistent. Make sure that your backup software also backs up the TxF metadata using the FSCTL_TXFS_READ_BACKUP_INFORMATION and FSCTL_TXFS_WRITE_BACKUP_INFORMATION dwIoControlCodes.

As I mentioned, the use of secondary resource managers is only required if the files participating in the transaction are on the system drive and you want a technology other than KTM to be the superior transaction coordinator or you want to control the consistency and availability of the resource manager. Secondary resource managers become very powerful in this regard if you wish to have more direct control over the behavior of the resource manager in your application. If the files you are operating on are on a non-system drive, you do not need to worry about the use of secondary resource managers. We are currently working to lessen this restriction in Windows Server "Longhorn" so this requirement won’t be applied to the entire system drive, but only to the parts of the system drive that Windows absolutely needs to boot.

Management

For any application that is going to live in a production environment, it is important to think about how the application will be managed. Therefore, you need to know how you can monitor TxF itself. Figure 9 shows a list of control codes that can be used with DeviceIoControl to help you do just that.

Similar to the information on resource managers provided previously, you can get the same information as these control codes from the FSUtil command-line tool. The commands are as follows:

- fsutil transaction list

- fsutil transaction fileinfo

- fsutil transaction query

What Happened to the Transacted Command Line?

If you were involved in the early betas for Windows Vista, you may recall seeing a transaction command available from the command line. This allowed an administrator to start a transaction from the command line and run an application, and then any file operations within that application would implicitly be transacted.

While testing application compatibility in Windows Vista, the team found that this implicit model was difficult for COM+ and managed developers to use safely. As a result, the model was changed to be opt-in and explicit to avoid this confusion. The downside to this change is that if you wish to use TxF in your app, your code will have to change to call the new APIs.

In Closing

Transactional NTFS is a truly innovative technology. It helps developers create more reliable applications on a more powerful platform. With TxF, developers can have atomic file operations that result in consistent file data, easing the test burden by reducing the number of error conditions to test for, and assisting in multi-user environments by providing isolation via ACID transaction semantics. Consider using TxF when developing apps that target Windows Vista and Windows Server "Longhorn." For more information, see the "Transaction Resources" sidebar.

I would like to thank Dana Groff, Christian Allred, and Jon Cargille for their help with this article.

Transaction Resources

Here’s a list I’ve compiled of resources relating to Transactional NTFS. I’ll be updating this list over time to reflect new resources, so be sure to watch for updates on my blog.

Web Sites:

- Transactions Programming Forum

- Transaction Management in Windows

- Transactional NTFS (TxF)

Blogs:

- Jim Johnson

- Florin Lazar

- Jason Olson

Specific Blog Posts:

- An MSDTC Proxy Enhancement (MLTM)

- Integration between System.Transactions and TxF

- What Is TxF and Why Do I Care?

Media:

- Transactional Vista: KTM and Friends

- Vista Transactional File System

- The Revolutionary Vista TxF Infrastructure

- Developing Applications for Windows Vista with Transactional NTFS

- Developer, Meet Server #1—Transactional NTFS

- Developer, Meet Server #2—Transactional NTFS and WCF

Jason Olson is a Technical Evangelist at Microsoft working on Windows Server "Longhorn" in the Developer and Platform Evangelism division. Outside of work, he is a loving and dedicated husband, hobbyist game developer, and a blogger at his site, managed-world.com. He can be reached at jason.olson@microsoft.com.

C# 3.0

The Evolution Of LINQ And Its Impact On The Design Of C#

Anson Horton

This article is based on a prerelease version of Visual Studio code-named "Orcas." All information herein is subject to change.

This article discusses:

- C# and LINQ

- The evolution of LINQ

- SQL querying from code

This article uses the following technologies:

LINQ, C#

Contents

Lambda Expressions

Extension Methods

Anonymous Types

Implicitly Typed Local Variables

Object Initializers

Query Expressions

I was a huge fan of the Connections series, hosted by James Burke, when it aired on the Discovery Channel. Its basic premise: how seemingly unrelated discoveries influenced other discoveries, which ultimately led to some modern-day convenience. The moral, if you will, is that no advancement is made in isolation. Not surprisingly, the same is true for Language Integrated Query (LINQ).

In simple terms, LINQ is a series of language extensions that supports data querying in a type-safe way; it will be released with the next version Visual Studio, code-named "Orcas." The data to be queried can take the form of XML (LINQ to XML), databases (LINQ-enabled ADO.NET, which includes LINQ to SQL, LINQ to Dataset and LINQ to Entities), objects (LINQ to Objects), and so on. The LINQ architecture is shown in Figure 1.

Figure 1 LINQ Architecture (Click the image for a larger view)

Let’s look at some code. A sample LINQ query in the upcoming "Orcas" version of C# might look like:

var overdrawnQuery = from account in db.Accounts

where account.Balance < 0

select new { account.Name, account.Address };

When the results of this query are iterated over using foreach, each element returned would consist of a name and address of an account that has a balance less than 0.

It’s immediately obvious from the sample above that the syntax is like SQL. Several years ago, Anders Hejlsberg (chief designer of C#) and Peter Golde thought of extending C# to better integrate data querying. Peter, who was the C# compiler development lead at the time, was investigating the possibility of making the C# compiler extensible, specifically to support add-ins that could verify the syntax of domain-specific languages like SQL. Anders, on the other hand, was conceiving a deeper, more specific level of integration. He was thinking about a set of "sequence operators" that would operate on any collection that implemented IEnumerable, as well as remote queries for types that implemented IQueryable. Ultimately, the sequence operator idea gained the most support, and in early 2004 Anders submitted a paper about the idea to Bill Gates’s Thinkweek. The feedback was overwhelmingly positive. In the early stages of the design, a simple query had the following syntax:

sequence<Customer> locals = customers.where(ZipCode == 98112);

Sequence, in this case, was an alias for IEnumerable<T>, and the word "where" was a special operator understood by the compiler. The implementation of the where operator was a normal C# static method that took in a predicate delegate (that is, a delegate of the form bool Pred<T>(T item)). The idea was for the compiler to have special knowledge about the operator. This would allow the compiler to correctly call the static method and create the code to hook up the delegate to the expression.

Let’s suppose that the example above would be the ideal syntax for a query in C#. What would this query look like in C# 2.0, without any language extensions?

IEnumerable<Customer> locals = EnumerableExtensions.Where(customers,

delegate(Customer c)

{

return c.ZipCode == 98112;

});

This code is frightfully verbose, and worse, it requires significant digging to find the relevant filter (ZipCode == 98112). And this example is simple; imagine how much more unreadable this would be with several filters, projections, and so forth. The root of the verbosity is the syntax required for anonymous methods. In the ideal query, the expression would require nothing but the expression to be evaluated. The compiler would then attempt to infer the context; for example, that ZipCode was really referring to the ZipCode defined on Customer. How to fix this problem? Hardcoding the knowledge of specific operators into the language didn’t sit well with the language design team, so they started looking for an alternate syntax for anonymous methods. They wanted it to be extremely concise, and yet not necessarily require more knowledge than the compiler currently needed for anonymous methods. Ultimately they devised lambda expressions.

Lambda Expressions

Lambda expressions are a language feature that is similar in many ways to anonymous methods. In fact, if lambda expressions had been put into the language first, there would have been no need for anonymous methods. The basic idea is that you can treat code as data. In C# 1.0, it is common to pass strings, integers, reference types, and so on to methods so that the methods can act on those values. Anonymous methods and lambda expressions extend the range of the values to include code blocks. This concept is common in functional programming.

Let’s take the example above and replace the anonymous method with a lambda expression:

IEnumerable<Customer> locals =

EnumerableExtensions.Where(customers, c => c.ZipCode == 91822);

There are several things to notice. For starters, the brevity of the lambda expression can be attributed to a number of factors. First, the delegate keyword isn’t used to introduce the construct. Instead, there is a new operator, =>, which tells the compiler that this isn’t a normal expression. Second, the Customer type is inferred from the usage. In this case, the signature of the Where method looks something like:

public static IEnumerable<T> Where<T>(

IEnumerable<T> items, Func<T, bool> predicate)

The compiler is able to infer that "c" refers to a customer because the first parameter of the Where method is IEnumerable<Customer>, such that T must, in fact, be Customer. Using this knowledge, the compiler also verifies that Customer has a ZipCode member. Finally, there is no return keyword specified. In the syntactic form, the return member is omitted but this is merely syntactic convenience. The result of the expression is still considered to be the return value.

Lambda expressions, like anonymous methods, also support variable capture. For example, it’s possible to refer to the parameters or locals of the method that contains the lambda expression within the lambda expression’s body:

public IEnumerable<Customer> LocalCusts(

IEnumerable<Customer> customers, int zipCode)

{

return EnumerableExtensions.Where(customers,

c => c.ZipCode == zipCode);

}

Finally, Lambda expressions support a more verbose syntax that allows you to specify the types explicitly, as well as execute multiple statements. For example:

return EnumerableExtensions.Where(customers,

(Customer c) => { int zip = zipCode; return c.ZipCode == zip; });

The good news is that we’re much closer to the ideal syntax proposed in the original paper, and we were able to get there with a language feature that is generally useful outside of query operators. Let’s take a look at where we are again:

IEnumerable<Customer> locals =

EnumerableExtensions.Where(customers, c => c.ZipCode == 91822);

There is an obvious problem here. Instead of thinking about the operations that can be performed on Customer, the consumer currently has to know about this EnumerableExtensions class. In addition, in the case of multiple operators, the consumer has to invert his thinking to write the correct syntax. For example:

IEnumerable<string> locals =

EnumerableExtensions.Select(

EnumerableExtensions.Where(customers, c => c.ZipCode == 91822),

c => c.Name);

Notice that the Select is the outer method, even though it operates on the result of the Where method. The ideal syntax would look more like the following:

sequence<Customer> locals =

customers.where(ZipCode == 98112).select(Name);

So, would it be possible to move closer to the ideal syntax with another language feature?

Extension Methods

Much better syntax, it turns out, was to come in the form of a language feature known as extension methods. Extension methods are basically static methods that are callable through an instance syntax. The root of the problem for the query above is that we want to add methods to IEnumerable<T>. However, if we were to add operators, such as Where, Select, and so on, every existing and future implementer would be required to implement those methods. The vast majority of those implementations would be the same, though. The only way to share "interface implementation" in C# is to use static methods, which is what we’ve done with the EnumerableExtensions class used previously.

Let’s suppose we were to write the Where method as an extension method instead. The query could then be rewritten as:

IEnumerable<Customer> locals =

customers.Where(c => c.ZipCode == 91822);

For this simple query, this syntax is very close to the ideal. But what exactly does it mean to write the Where method as an extension method? It’s actually fairly straightforward. Basically the signature of the static method changes such that a "this" modifier is added to the first parameter:

public static IEnumerable<T> Where<T>(

this IEnumerable<T> items, Func<T, bool> predicate)

In addition, the method must be declared within a static class. A static class is one that may contain only static members and that is denoted by the static modifier on the class declaration. That’s all there is to it. This declaration instructs the compiler to allow Where to be called with the same syntax as an instance method on any type that implements IEnumerable<T>. The Where method must, however, be accessible from the current scope. A method is in scope when the containing type is in scope. Therefore, it’s possible to bring extension methods into scope through the Using directive. (See the sidebar "Extension Methods" for more information.)

Extension Methods

It’s clear that extension methods help simplify our example query, but are they a generally useful language feature outside of that scenario? It turns out that there are many uses for extension methods. One of the most common will probably be to provide shared interface implementations. For example, suppose you have the following interface:

interface IDog

{

// Barks for 2 seconds

void Bark();

void Bark(int seconds);

}

This interface requires that every implementer write an implementation for both overloads. With the "Orcas" version of C#, the interface could simply be:

interface IDog

{

void Bark(int seconds);

}

An extension method could be added in another class:

static class DogExtensions

{

// Barks for 2 seconds

public static void Bark(this IDog dog)

{

dog.Bark(2);

}

}

Now the implementer of the interface need only implement a single method, but the clients of the interface may freely call either overload.

We now have a syntax that is very close to the ideal for the filter clause, but is that all there is to the "Orcas" version of C#? Not quite; let’s extend the example a bit by projecting out only the customer’s name, as opposed to the entire customer object. As I mentioned earlier, the ideal syntax would take the following form:

sequence<string> locals =

customers.where(ZipCode == 98112).select(Name);

With just the language extensions we’ve discussed, lambda expressions and extension methods, this could be rewritten as:

IEnumerable<string> locals =

customers.Where(c => c.ZipCode == 91822).Select(c => c.Name);

Notice that the return type is different for this query—IEnumerable<string> instead of IEnumerable<Customer>. This happens because we are only returning the name of the customer from the select statement

That works really well when the projection is only a single field. However, suppose that instead of just the Name of the customer, we also want to return the customer’s address. The ideal syntax might look like this:

locals = customers.where(ZipCode == 98112).select(Name, Address);

Anonymous Types

If we were to continue using our existing syntax to return the name and address, we’d quickly run into the problem that there is no type that contains only a Name and Address. We could still write this query, however, by introducing that type:

class CustomerTuple

{

public string Name;

public string Address;

public CustomerTuple(string name, string address)

{

this.Name = name;

this.Address = address;

}

}

We could then use that type, here CustomerTuple, to construct the result of our query:

IEnumerable<CustomerTuple> locals =

customers.Where(c => c.ZipCode == 91822)

.Select(c => new CustomerTuple(c.Name, c.Address));

That sure seems like a lot of boilerplate code to project out a subset of the fields. It’s also often unclear what to name such a type. Is CustomerTuple really a good name? What if we had projected out Name and Age instead? That could also be a CustomerTuple. So, the problems are that we have boilerplate code and it doesn’t seem that there are any good names for the types that we create. Plus, there could also be many different types required, and managing them could quickly become a headache.

This is exactly what anonymous types are for. This feature basically allows the creation of structural types without specifying the name. If we rewrite the query above using anonymous types, here’s what it looks like:

locals = customers.Where(c => c.ZipCode == 91822)

.Select(c => new { c.Name, c.Address });

This code implicitly creates a type that has the fields Name and Address:

class

{

public string Name;

public string Address;

}

This type can’t be referenced by name, since it has none. The names of the fields can be explicitly declared in the anonymous type creation. For example, if the field being created is derived from a complicated expression, or the name simply isn’t desirable, it’s possible to change the name:

locals = customers.Where(c => c.ZipCode == 91822)

.Select(c => new { FullName = c.FirstName + “ “ + c.LastName,

HomeAddress = c.Address });

In this case, the type that is generated has fields named FullName and HomeAddress.

This gets us closer to the ideal, but there is a problem. You’ll notice that I strategically omitted the type of locals in any place where I used an anonymous type. Obviously we can’t state the name of anonymous types, so how do we use them?

Implicitly Typed Local Variables

There’s another language feature known as implicitly typed local variables (or var for short) that instructs the compiler to infer the type of a local variable. For example:

var integer = 1;

In this case, integer has the type int. It’s important to understand that this is still strongly typed. In a dynamic language, integer’s type could change later. To illustrate this, the following code does not compile:

var integer = 1;

integer = “hello”;

The C# compiler will report an error on the second line, stating that it can’t implicitly convert a string to an int.

In the case of the query above, we can now write the full assignment as shown here:

var locals =

customers

.Where(c => c.ZipCode == 91822)

.Select(c => new { FullName = c.FirstName + “ “ + c.LastName,

HomeAddress = c.Address });

The type of locals ends up being IEnumerable<?> where "?" is the name of a type that can’t be written (since it is anonymous).

Implicitly typed locals are just that: local within a method. It is not possible for them to escape the boundaries of a method, property, indexer, or other block because the type cannot be explicitly stated, and "var" is not legal for fields or parameter types.

Implicitly typed locals turn out to be convenient outside of the context of a query. For example, it helps simplify complicated generic instantiations:

var customerListLookup = new Dictionary<string, List<Customer>>();

We’re now in a good place with our query; we’re close to the ideal syntax and we’ve gotten there with general-purpose language features.

Interestingly, we found that as more people worked with this syntax, there was often a need to allow a projection to escape the boundaries of a method. As we saw earlier, this is possible by constructing an object by calling its constructor from within Select. However, what happens if there is no constructor that takes exactly the values you need to set?

Object Initializers

For this case, there is a C# language feature in the upcoming "Orcas" version known as object initializers. Object initializers basically allow the assignment of multiple properties or fields in a single expression. For example, a common pattern for object creation is:

Customer customer = new Customer();

customer.Name = “Roger”;

customer.Address = “1 Wilco Way”;

In this case, there is no constructor of Customer that takes a name and address; however, there are two properties, Name and Address, that can be set once an instance is created. Object initializers allow the same creation with the following syntax:

Customer customer = new Customer()

{ Name = “Roger”, Address = “1 Wilco Way” };

In our earlier CustomerTuple example, we created the CustomerTuple class by calling its constructor. We can achieve the same result via object initializers:

var locals =

customers

.Where(c => c.ZipCode == 91822)

.Select(c =>

new CustomerTuple { Name = c.Name, Address = c.Address });

Notice that object initializers allow the parentheses of the constructor to be omitted. In addition, both fields and settable properties can be assigned within the body of the object initializer.

We now have a succinct syntax for creating queries in C#. However, we also have an extensible way to add new operators (Distinct, OrderBy, Sum, and so on) through extension methods and a distinct set of language features useful in their own right.

The language design team now had several prototypes to get feedback on. So we organized a usability study with many participants who had experience with both C# and SQL. The feedback was almost universally positive, but it was clear there was something missing. In particular, it was difficult for the developers to apply their knowledge of SQL because the syntax we thought was ideal didn’t map very well to their domain expertise.

Query Expressions

The language design team then designed a syntax that is closer to SQL, known as query expressions. For example, a query expression for our example might look like this:

var locals = from c in customers

where c.ZipCode == 91822

select new { FullName = c.FirstName + “ “ +

c.LastName, HomeAddress = c.Address };

Query expressions are built on the language features described above. They are literally syntactically translated into the underlying syntax that we’ve already seen. For example, the query above is translated directly into:

var locals =

customers

.Where(c => c.ZipCode == 91822)

.Select(c => new { FullName = c.FirstName + “ “ + c.LastName,

HomeAddress = c.Address });

Query expressions support a number of different "clauses," such as from, where, select, orderby, group by, let, and join. These clauses translate into the equivalent operator calls, which in turn, are implemented via extension methods. The tight relationship of the query clauses and the extension methods that implement the operators makes it easy to combine them if the query syntax doesn’t support a clause for a needed operator. For example:

var locals = (from c in customers

where c.ZipCode == 91822

select new { FullName = c.FirstName + “ “ +

c.LastName, HomeAddress = c.Address})

.Count();

In this case the query now returns the number of customers who live in the 91822 ZIP Code area.

And with that, we’ve managed to end just about where we started (which I always find rather satisfying). The syntax of the next version of C# evolved over the past few years through several new language features to ultimately arrive very close to the original syntax proposed in the winter of 2004. The addition of query expressions builds on the foundations provided by the other language features in the upcoming version of C# and makes many query scenarios easier to read and understand for developers with a background in SQL.

Anson Horton has been a Program Manager at Microsoft for almost six years. He has been working on the C# team since its creation, and was on the C++ team prior to that. He has been involved in the design of the C# language and compiler, the C# project system, the C# IDE (IntelliSense), and the C# Expression evaluator and debugger. Anson maintains a blog at blogs.msdn.com/ansonh which he updates as infrequently as possible.

OFFICE UI

New VSTO Features Help You Customize Word And Outlook

Steve Fox and Paul Stubbs

This article is based on a prerelease version of Microsoft Visual Studio Tools for the Microsoft Office System. All information herein is subject to change.

This article discusses:

- New features in VSTO “Orcas” Beta 1

- Using the Ribbon designer for Office documents

- Customizing the Ribbon with XML

- Data binding with Word content controls

- Using VSTO to build custom Outlook form regions

This article uses the following technologies:

VSTO, Office, Visual Studio

Contents

Overview of VSTO

Customizing the Office UI with the Ribbon Designer

Customizing Advanced Ribbons Using XML

VSTO Word Content Controls

Outlook Form Regions

Wrap Up

The 2007 Microsoft® Office system has evolved into a robust business application platform that you can use to build and deploy a range of Office Business Applications (OBAs). OBAs are an emerging class of applications that address real-world and critical business problems. You can build them by seamlessly integrating existing Office software and services (Microsoft Office Word, Excel® Services and SharePoint® Server) with line-of-business (LOB) systems such as SAP. Couple the power and extensibility of the Office system with the upcoming release of Visual Studio® later this year (code-named "Orcas") and you have an extremely strong marriage of technology. In this article, we focus on how professional developers can use Microsoft Visual Studio Tools for the Microsoft Office System (VSTO) to build powerful custom applications against the 2007 Microsoft Office system.

Overview of VSTO

VSTO is a powerful set of tools and features that let developers extend and customize Microsoft Office applications using Visual Basic® and C#. The "Orcas" release of VSTO will bring even more power to the developer. It includes integration of managed code development with Microsoft Office and Microsoft Office SharePoint in unique and exciting ways. It also provides a Ribbon visual designer that lets you customize the Office Ribbon and program against it using a simple Windows® Forms-like programming model.

Furthermore, VSTO provides the ability to easily extend Microsoft Office Outlook® forms with a visual designer that supports the use of Windows Forms controls and Windows Presentation Foundation controls, and also supports the ability to import forms composed of Outlook built-in controls. In addition, SharePoint workflow in C# or Visual Basic is now a rapid application development (RAD) experience in Visual Studio. VSTO includes visual workflow designers and new support for quickly deploying and debugging a custom workflow on a SharePoint server, which can reduce the labor-intensive process from 15 steps and several days to a few clicks in a wizard.

VSTO also has visual designers for Word 2007 and Excel 2007 that appear inside the Visual Studio IDE. This allows the document to act like a visual design surface, which makes the creation of powerful document-based solutions easy. The ability to construct Word 2007 documents using the new Word content controls, with full support for ADO.NET-style data binding, drag and drop from the data sources window, and property grid support are all available in VSTO as well.

In this article, we focus on a subset of the above features. Specifically, we illustrate how you can customize the Office Ribbon, add Word content controls to Microsoft Office Word 2007, integrate a custom Ribbon with the content controls, and build Outlook form regions. Obviously, these are just a few of the features in Visual Studio "Orcas," but we discuss them here because they will be a part of the Beta 1 release, which should be available by the time you read this.

Customizing the Office UI with the Ribbon Designer

In VSTO "Orcas," there are two ways in which you can customize the Office Ribbon. The first is similar to Microsoft Visual Studio 2005 Tools for Office Second Edition (VSTO 2005 SE); that is, through the use of an XML class and definition you can define the elements of the Ribbon and then hook the Ribbon elements to event handlers (through callback code). Since you’re probably familiar with this method, we are going to focus more on the second method of customizing the Office Ribbon in VSTO "Orcas": the Ribbon Designer. (If you’re interested in customizing the Ribbon with VSTO 2005 SE, see Ken Getz’s Advanced Basics column in this issue of MSDN® Magazine.)

The Ribbon Designer, which includes a visual designer and an extensibility object model, makes it straightforward for developers to create, configure, and debug the Ribbon UI. As with other visual designers, you can drag and drop controls onto the design surface, set properties through the Properties window, and easily create event handlers for your new Ribbon items by double-clicking the control. Further, Office callbacks are mapped to the events on the VSTO Ribbon objects so developers write event handlers instead of callback methods. Let’s go ahead and customize a sample Ribbon for Word 2007 for illustration.

To begin, create a new Word 2007 project in either the C# or Visual Basic Office node and call it WordRibbonExample. After you create the project, Visual Studio automatically creates a number of project resource files. On the root project file (WordExample.cs), right-click and select Add New Item | Ribbon (Visual Designer). You can either provide a name for the Ribbon or accept the default name. This will create a default custom Ribbon on which you can add Ribbon elements.

The Ribbon Designer facilitates drag and drop functionality, so you can drag controls from the Office Ribbon Controls tab of the toolbox. Tabs, groups, and buttons are the Ribbon controls that are supported in the Beta 1 release of VSTO "Orcas." Supported Office applications are Word 2007, Excel 2007, PowerPoint® 2007, and Outlook 2007.

In the following example, we have added three groups (you must have groups as containers for the Ribbon controls) to the Ribbon tab—called My Custom Tab. Within each of the groups, we have added buttons and images. Figure 1 shows how to edit a control’s properties.

Figure 1 Ribbon Designer (Click the image for a larger view)

A typical VSTO Ribbon customization consists of one or more Ribbon objects, each of which contains one or more Tab objects that contain one or more Group objects, and so on. Thus, a VSTO customization is constructed within a specific hierarchical structure of Ribbon objects. For example, the conceptual diagram in Figure 2 illustrates that you must have a tab as the highest level control. You can then add one or more group controls to the tab and one or more buttons, toggle buttons, and menus to the group. Within any given menu, though, you can only go five levels deep.

Figure 2 Control Hierarchy

With a custom Ribbon now built, you can add event handlers to the Ribbon elements. You add an event handler to an individual Ribbon control basically the same way you do it with other controls: double-click the control and Visual Studio automatically adds a click event and opens the source-code view for you to enter your event handler code. For example, if you have a button named testButtonOne, you can double-click the button in the designer and the code view will open to the testButtonOne_click event where you can enter some event-handling code, like so:

private void testButtonOne_click(

object sender, RibbonControlEventArgs e)

{

// Simple message box to illustrate click event.

testButtonOne.label = “New Label”;

MessageBox.Show(testButtonOne.label);

}

Customizing Advanced Ribbons Using XML

In VSTO 2005 SE, you customized the Office Ribbon through an XML file, by selecting Ribbon Support when adding a new item to your project. This automatically created a Ribbon class, which is used to load the custom UI, and a Ribbon XML file, which defines the custom UI. Figure 3 illustrates the default XML code in the Ribbon Support item.

Figure 3 Default XML Ribbon Definition

<customUI xmlns=”http://schemas.microsoft.com/office/2006/01/customui” onLoad=”OnLoad”>

<ribbon>

<tabs>

<tab idMso=”TabAddIns”>

<group id=”MyGroup”

label=”My Group”>

<toggleButton id=”toggleButton1”

size=”large”

label=”My Button”

screentip=”My Button Screentip”

onAction=”OnToggleButton1”

imageMso=”HappyFace” />

</group>

</tab>

</tabs>

</ribbon>

</customUI>

There are also default elements that are added to your project that help you customize the UI, such as a toggle button that calls a message box when the user clicks it, callback code that provides you with a sense for how you can build event handlers for the custom UI, and helper code.

While the new Ribbon Designer enables common Ribbon customization tasks, it does not provide support for the more advanced scenarios (such as repurposing built-in commands or working with some of the less commonly used controls). In the event that you want to customize the default Office UI or use control types not available in the designer, you can fall back to using Ribbon XML for your customization. You can either choose to start with the Ribbon (XML) item or you can use the designer as a starting point, then export to Ribbon XML and continue to define the custom UI using XML—but note that this is a one-time export. You will then need to manually define the callback signatures and port any event handler code you’ve written against the designer.

VSTO Word Content Controls

While the customization of the Ribbon enables you to create and wrap custom UI at the application level, VSTO Word content controls enable you to customize a document or template very simply to create a more structured and smarter document. Using drag and drop functionality, you can add controls to a Word 2007 template, build business logic into those controls, protect the content controls from user editing and deleting, and add simple data binding to local or enterprise databases. Because VSTO helps you to access powerful customization features (such as task pane customization), you can imagine a scenario where the document surface becomes an interface into enterprise data and the task pane (or a custom Ribbon) becomes the way in which users can populate the content controls within the Word 2007 document.

Using content controls, let’s create a simple Word template that can be built automatically using data from a customer database, and add a custom Ribbon that enables the end user to browse the data within the database. The resulting document will look like Figure 4.

Figure 4 Example Invitation Letter

To start, you must create a solution so your end users can customize the Word template by selecting the content of predefined areas—based on the data in the customer database. In Visual Studio, you create a new project and, in either Visual Basic or C#, select Office and then Word 2007 Document template. This opens a blank Word document within the Visual Studio IDE, along with the content controls in the Toolbox shown in Figure 5.

Figure 5 Word Controls

With the shell of the project, you can now add controls to the surface of the document. While adding controls is great, we also need to supply some structure to the document. To build this document template, you can either create the template within the Visual Studio environment or you can load a pre-defined template. For this example, we’ve first added some default text that will act as the invitation letter template and then added five content controls (by dragging and dropping the controls from the Toolbox to the surface of the document). The controls we added were a RichTextContentControl for the company name, company address, and addressee; and a DatePickerContentControl for the date and response date. After the controls are on the document surface, you can adjust the properties of the controls (similar to other controls within Visual Studio) through the Properties window. If you look back at Figure 4 you’ll see where we’ve added controls that, when hooked to the data browse event handlers, will allow a user to browse through the customer database.

If you’ve followed along, press F5 to build and run your solution. Because we have not hooked the controls to any existing database, there is no data to populate these fields. So close the Word document without saving, and return to the Visual Studio environment.

With the controls now placed in the document, you can reference a specific data source and bind the controls to it. For this sample, we created a very simple Microsoft Office Access™ customer database called CustomerDatabase. The database comprises one table with six columns: ID, CompanyName, CompanyAddress, ContactName, CurrentDate, and ResponseDate. Creating the reference is quite straightforward: you simply use the Data Source Configuration Wizard to bind the controls to specific records within the customer database.

To begin, click the Data menu and select Add a New Data Source. The Data Source Configuration Wizard will prompt you for a choice. Select Database and click Next. In the next dialog, click New Connection, and then in the Add Connection dialog, accept the default Data Source (which is Access) and then browse to your Access database. The wizard will then prompt you to select the tables and views that you want to associate with the project. If you created the same database that we did, select the CustomerInfo table and leave the Views option unchecked. Figure 6 illustrates this dialog.

Figure 6 Choose Your Database Objects Dialog (Click the image for a larger view)

After you’ve completed this step, click Finish. Based on the information that you’ve provided, Visual Studio automatically creates a data set, table adapter, and binding source—the elements you will need to bind your controls to the data and programmatically manipulate the records in your database. For this example, Visual Studio has automatically assigned names for these as follows: customerDatabaseDataSet, customerInfoTableAdapter, and customerInfoBindingSource.

Now that you have the database connected to the project, you must bind the data to the specific controls. To do this, you use the Data binding property in the Properties window (see Figure 7). Select Text, and then select the specific database record you want to associate with each of the content controls. (Note that you can also drag the specific data elements from the Toolbox and drop them onto the control with which you want to associate the data.)

Figure 7 Data Binding Property (Click the image for a larger view)

With the specific records of the customer database now connected to each of the content controls, let’s customize the Ribbon and add event handlers to the new custom buttons to enable the end user to browse through data in the customer database. You can use the Ribbon Designer to create two buttons (a Next and a Previous button) and the Properties window to edit the properties of the tab, button group, and buttons, and also to map bitmap-format images to the buttons. The Figure 8 shows what this custom Ribbon might look like.

Figure 8 Data Browse Buttons

The last step is to add event handlers to each of the buttons so the end user can browse the data in the customer database. To do this, double-click each of the buttons and add code that will allow the user to move either forward or backward across the data. When you double-click a button, the code view for the Ribbon class opens and enables you to add the event handler code. The code below uses the Globals object to access the document, the binding source for the customer database, and eventually the MoveNext and MovePrevious methods.

private void button1_OnAction(

object sender, ControlEventArgs e)

{

Globals.ThisDocument.customerInfoBindingSource.MoveNext();

}

private void button2_OnAction(

object sender, ControlEventArgs e)

{

Globals.ThisDocument.customerInfoBindingSource.MovePrevious();

}

With the custom buttons now hooked into the content controls, you can click F5 to build and run the solution and test the integration of the custom Ribbon and the content controls. When you run the solution, click the Customer Data tab to view the custom Ribbon buttons you just added. When you click the buttons, as you see in Figure 9, in the upper-right-hand corner of the document, they allow you to browse through the data—loading the specific records into the content controls in the document.

Figure 9 Browsing the Customer Data in the Letter

This scenario is simple yet practical. It illustrates that with the help of the new Ribbon Designer and the Word content controls, you can create an automated application that can load and create personalized invitation letters for specific customers. As you learn more about how to develop against the 2007 Office system using VSTO, you’ll find ways to extend this scenario and build more sophistication into your customized applications. Now, let’s turn to Outlook form regions.

Outlook Form Regions

Outlook 2007 enables you to create form regions that you can use to display information to the user about a mail message, contact, or any other Outlook item. For example, when e-mail comes in from a customer, you can look up that customer in your system and display his order history in the form region. When you open a contact item, you can display a map of the contact’s address in the form region. In this section, you will create a form region that enables you to add notes about mail messages. The notes are entered and displayed in a form region below the message and persisted as a hidden attachment to the message (see Figure 10).

Figure 10 Outlook 2007 Form Region (Click the image for a larger view)

To create an Outlook 2007 form region, you can again start with either a Visual Basic or a C# project. Adding a form region is as simple as adding a new Outlook Form Region item. Right-click the project, and choose Add New Item from the context menu. Choose Outlook Form Region from the Add New Item dialog and set the name as NotesFormRegion.cs. Click Add to launch the Outlook Form Region Wizard, which will guide you through the process of adding the form region. The first thing you need to choose is the type of form region to create. There are four types to choose from: Separate, Adjoining, Replacement, and Replace All.

Choose Adjoining, which will make the form region show at the bottom of the inspector. Click Next to continue. Set the name to VSTO Notes. Note that the title and description are not available for the Adjoining form region types. You can also configure how the form region is displayed in compose mode, reading mode, or the reading pane. Leave all of the defaults checked and then click Next to continue.

Choose which Outlook inspectors your form region is attached to. This example uses the mail message to show the form region only when a mail item is shown. You have now supplied all of the information needed for the wizard to create the form region.

After you click Finish, you will see a new NotesFormRegion.cs item added to your project and the Form Region Designer is opened. You can now design your form region as you would design a Windows Form, by dragging controls from the Toolbox onto the design surface. For this example, all you need to add is a single rich-text box, which is used to display the mail notes. Drag a RichTextBox control from the Toolbox onto the form region design surface. Set it to dock in the parent container so that it fills the entire form region. Set the background color to yellow so that it looks like a note. At this point you have a fully functional Outlook form region. You can press F5 to build and launch Outlook. Open a mail message to see your yellow textbox at the bottom of the message.

Now let’s write some code to make the note-taking form region functional. Right-click on the form region designer and choose View Code. In the code view, you can see that the form region derives from the Microsoft.Office.Tools.Outlook.FormRegionControl class. The FormRegionControl base class contains all of the plumbing to embed your control onto an Outlook form region. You will also notice that there are three event handlers. The FormRegionInitializing event is called when the form region is first created. The event allows you to cancel the creation of the form region, and through it you are also passed a reference to the Outlook item to which the form region is attached. The other two events, FormRegionShowing and FormRegionClosed, are called when the form region is displayed and closed. These events are where you will add your code. When the form region is closed, you want to save the text from the rich textbox as a hidden attachment in the message. In order to attach the text you must first save it to a temporary file. When you display the form region, you want to read the text from the hidden attachment and show it in the rich textbox. Again you will need to write the attachment out to a temporary file first. Figure 11 illustrates how this might be done.

Figure 11 Enabling Text Editing in Outlook 2007 Form Region

Outlook.MailItem mailItem;

string NotesPath = “C:\\VSTO Note.txt”;

string NoteDisplayName = “VSTO Note”;

private void NotesFormRegion_FormRegionShowing(

object sender, System.EventArgs e)

{

mailItem = (Outlook.MailItem)this.OutlookItem;

foreach (Outlook.Attachment attachment in mailItem.Attachments)

{

if (attachment.DisplayName == NoteDisplayName)

{

attachment.SaveAsFile(NotesPath);

richTextBox1.Text = System.IO.File.ReadAllText(NotesPath);

}

}

}

private void NotesFormRegion_FormRegionClosed(object sender,

System.EventArgs e)

{

if (richTextBox1.Modified)

{

System.IO.File.WriteAllText(NotesPath, richTextBox1.Text);

foreach (Outlook.Attachment attachment in mailItem.Attachments)

{

if (attachment.DisplayName == NoteDisplayName)

{

attachment.Delete();

}

}

if (richTextBox1.TextLength > 0)

{

long position = 0;

mailItem.Attachments.Add(NotesPath,

Outlook.OlAttachmentType.olEmbeddeditem, position,

NoteDisplayName);

}

mailItem.Save();

}

}

After you’ve completed all of the steps above, press F5 and run your project. This will build your project and invoke Outlook 2007. When Outlook starts, open a message from your Inbox and type in some text to test it out!

Wrap Up

This article showed you how to customize the Office Ribbon, add content controls to a Word document, integrate a custom Ribbon with the Word document, and create an Outlook form region. These are just some of the Office development features available to you from VSTO in the Beta 1 release of Visual Studio "Orcas."

Steve Fox is a Program Manager with the Visual Studio Tools for Office (VSTO) team. He divides his time working externally with VSTO customers, providing VSTO training, delivering VSTO sessions at conferences and working internally with the development team to help evolve the VSTO product.

Paul Stubbs is a Program Manager with the Visual Studio Tools for Office and Visual Tools for Applications team. He coauthored VSTO for Mere Mortals with Kathleen McGrath (Addison-Wesley, 2006). He has spoken at TechEd and TechReady and participates in the developer community on the Microsoft forums. Read Paul’s blog at blogs.msdn.com/pstubbs.

Got Game?

Unleash Your Imagination With XNA Game Studio Express

Charles Cox and Michael Klucher

This article discusses:

- Game development basics

- The Spacewar Starter Kit

- Adding features to Spacewar

- Customizing game graphics

This article uses the following technologies:

XNA Game Studio Express, Xbox 360, C#

Code download available at: XNA2007_05.exe (155 KB)

Browse the Code Online

Contents

Game Development Basics

The Spacewar Starter Kit

Understanding the Game Loop

Shields Up!

Rendering Details

Energize Shields!

Bonus Credit: Change Space!

Blast Off!

While playing a video game, have you ever thought to yourself, "I could put something together like this"? XNA™ Game Studio Express is a new game development solution for both Microsoft® Windows® and Xbox 360™ targeted primarily at students, hobbyists, and independent game developers—people like you!

Based on Visual C#® 2005 Express Edition, XNA Game Studio Express is built on the XNA Framework, a managed-code class library designed for game development. The framework streamlines the process of writing code for two very different platforms—Xbox 360 and Windows XP SP2. In fact, much of the code you write will work on either system with no changes at all.

You can immediately start writing games for Windows simply by downloading XNA Game Studio Express from MSDN® at msdn2.microsoft.com/xna. To develop games on your Xbox 360 console, you’ll need to obtain a membership in the XNA Creators Club (for more information, see msdn2.microsoft.com/bb219592.aspx). As a member, you’ll have access to game assets as well as exclusive XNA Creators Club versions of starter kits.

Before we get started walking you through XNA Game Studio Express, you’ll need to download and install it. If you’d like help getting set up on your Xbox 360 console, you can find a video walkthrough at msdn.com/xna/videos. Look for the video titled "Getting Started with the XNA Creators Club."

Game Development Basics

Game development presents several unique challenges. Games must strive to be compelling for users—not only from a design perspective, but technically as well—with eye-catching graphics, all while maintaining reasonable frame rates. Intuitive ways for the user to interact with the game via input as well as quality audio output are additional considerations.

Before the introduction of XNA Game Studio Express, game developers used technologies such as DirectX® and OpenGL. These APIs allow basic access to the hardware, but are not intuitive. As a result, developers often assembled core components such as graphics, audio, and input into a framework called an engine, then layered specific game logic over the top of the framework. This abstraction is often costly and time-consuming to create and use.

Furthermore, managing memory and logic in C++, the language of choice for game development, is technically demanding, as is managing the many pieces of art content present in a game. No wonder game development has been particularly daunting for amateur developers. Similarly forbidding are the complex technical and business requirements of developing for next-generation video game consoles such as the Xbox 360.

Enter XNA Game Studio Express, built atop Visual C# 2005 Express Edition. XNA Game Studio Express is designed for hobbyists, students, and other game development enthusiasts, enabling them to create their own games on both Windows and Xbox 360 using C# and the Visual C# 2005 Express Edition integrated development environment.

XNA Game Studio Express includes the XNA Framework, a cross-platform game development API for both Windows and Xbox 360. The XNA Framework drastically simplifies not only graphics, audio, input, and storage, but also the fundamental timing and drawing loops used in all games.

Also included in XNA Game Studio Express is the XNA Framework Content Pipeline, which is integrated into the build environment of Visual C# 2005 Express Edition. With the Content Pipeline, game developers can now add their art content—such as 3D models, graphical textures, shaders, and audio—into their solution in the IDE, and the content will be built into runtime objects that can be accessed in code. This reduces the overhead associated with art content.

Previous game development solutions typically started from scratch: manually writing timing and render loops that managed each tick of time and ensured the game action was handled properly. XNA Game Studio Express constructs the timing and render loops automatically. You are provided with a Game class that handles its own timing and rendering, exposing an Update method where you can add objects to update each frame and a Draw method where you can add objects to render to the screen.

Furthermore, the provided game framework, called the Application Model, automatically sets up the appropriate graphics device without complicated device enumeration code. With this code in place, an XNA Game Studio Express project can be compiled and run, with a fully functional timing and rendering loop running from the moment it is created.

The Spacewar Starter Kit

In addition to XNA Framework code templates, XNA Game Studio Express comes with the Spacewar Starter Kit—a fully functional game, complete with source code and art assets you can modify. The Spacewar Starter Kit is a top-down space shooter game for two players, based on the original implementation of Spacewar, developed for the DEC PDP-1 in 1962 by Stephen Russell, Peter Samson, and Dan Edwards. The starter kit features two versions of the game: a Retro version that retains the original’s gameplay and graphics, and an Evolved version that features improved 3D graphics and sound.

We’ll open up the starter kit, try out the Evolved version, explore how the game is constructed, and then add a new feature: a shield that each player can use to temporarily block enemy shots.

First, launch XNA Game Studio Express, click the File menu, and then click New Project. When the New Project window opens, click the Spacewar Windows Starter Kit, and then click OK (see Figure 1). If you’ve joined the Creators Club and have set up your Xbox 360 to receive from XNA Game Studio Express, click the Spacewar Xbox 360 Starter Kit instead, and then click OK.

Figure 1 Opening the Spacewar Starter Kit (Click the image for a larger view)

The Spacewar Starter Kit will take a moment to load. When it has fully loaded, you’ll see the code files displayed in the Solution Explorer pane. You’ll also see instructions on how to play the game. For the best experience, plug in two Xbox 360 controllers. You can also use the keyboard; the controls are shown in the instructions page when the Spacewar Starter Kit is loaded.

Compile and run the Spacewar application by pressing F5. Once the game has started, select Evolved mode and try out the gameplay for a few moments.

Understanding the Game Loop

One of the core features provided by XNA is the game loop. This loop continually updates the state of the game (based on user input, in-game conditions, and any other applicable condition) and renders it (which involves drawing to the screen, playing appropriate audio, rumbling a controller, and providing any other form of output to the user). Understanding how the game loop in Spacewar works will better help you to understand some of the modifications we’ll be making to the Starter Kit in this article.

Game execution begins as in any other .NET application, with a Main method. The Main method simply constructs an instance of the SpacewarGame class and calls its base class’s Run method:

static void Main(string[] args)

{

using (SpacewarGame game = new SpacewarGame())

{

game.Run();

}

}

Note that SpacewarGame is derived from the Game class in the Microsoft.Xna.Framework namespace; it exposes key methods, properties, and events for the game lifecycle. You then override methods on Game to provide specific functionality for your game. For example, SpacewarGame overrides the BeginRun method (which is called from Run) to play the game’s title music, change its internal state machine to the logo screen stage, and create the initial camera from which a player views everything.

SpacewarGame’s ChangeState method, as called by the BeginRun method, is responsible for creating many of the objects used in the game. When the BeginRun method calls ChangeState with GameState.LogoSplash as an argument, it reacts by creating a TitleScreen. TitleScreen derives from FullScreenSplash, which in turns derives from the abstract Screen class defined in the Spacewar project. The Screen class represents a unit of rendering for the game, such as a splash screen, selection screen, or an actual game level. It exposes several virtual methods, which its derived types override to provide screen-specific functionality. TitleScreen overrides the Screen.Update method to check whether the user has pressed A, B, or X to indicate where to go next in the game (Evolved, Retro, or Information). TitleScreen is just one of several Screen-derived types used in Spacewar; others include SelectionScreen (for ship selection), EvolvedScreen (for playing the "evolved" version of the game), VictoryScreen (for when a player has won), and so forth.

Each Screen encapsulates the state necessary for that screen’s functionality along with update and render behavior specific to that screen. Both EvolvedScreen and RetroScreen, which represent the main game play for their respective game types, derive from SpacewarScreen:

public class SpacewarScreen : Screen

{

protected Projectiles bullets;

protected Ship ship1;

protected Ship ship2;

protected SceneItem sun;

protected Particles particles;

protected SceneItem backdrop;

protected bool paused = true;

protected int player1Score;

protected int player2Score;

...

}

SpacewarScreen contains items to be rendered on the screen, including SceneItems for both ships (implemented as Ship, which derives from SpacewarSceneItem, itself derived from the base SceneItem class), all of the miscellaneous particles and projectiles to be rendered, the backdrop for the game, the sun, and so forth. It also contains game state such as player scores and whether or not the game is paused.

When a Game is created (in our case, a SpacewarGame), it ensures that a host is created to house the game and to provide application-level services to the game, namely notifications for events such as activation, deactivation, exiting, and idling. The latter is most important. The Game.Tick method is called by this idle handler, and the Tick method in turn calls the Game.Update and Game.DrawFrame methods.The Game.Update method is overridden by SpacewarGame.Update, which calls Update on whatever has been set as the current scene (from the ChangeState method). Game.DrawFrame calls Game.Draw, which is overridden by SpacewarGame to call Render on whatever is the current scene.

Shields Up!

It’s easy to augment Spacewar with new features. Let’s add an energy shield. This shield will protect a ship from damage caused by other items in the scene. We’ll need to edit two code files: Ship.cs and SpacewarScreen.cs. You can download it or walk through it at msdn.microsoft.com/msdnmag/code07.aspx.

The file Ship.cs is where we’ll do most of our coding. In Spacewar, the Ship.cs file defines a Ship object. The Ship object contains methods and data that define a spaceship object controlled by a player. Look in Solution Explorer for Ship.cs and double-click it to open the file.

First, we need to define a few variables to control the logic of the shield. The shield will stay up for five seconds, and can be used only once per round. Go to line 40 in Ship.cs, and add the following code (you’ll be adding this among other private members that store information such as whether the ship is currently invulnerable, inHyperspace, or inRecovery):

private bool inShield;

private bool shieldUsed;

private double exitShieldTime;

private const double shieldTime = 5.0;

This establishes a few variables that will track whether the shield is active, whether the shield has already been used, and how much time is left on the shield’s timer.

Let’s use a property so we can easily access whether the ship’s shield is active. This will be accessed by our modifications in SpacewarScreen.cs shortly. Go to line 188 in Ship.cs (what will now be line 188 after adding the previous snippet), and add the following code:

public bool Shield { get { return inShield; } }

So far, we’ve put a set of variables and an accessor in place. Let’s tie these variables to some game action. We’ll be modifying the Update method inside Ship.cs. As described, the Ship’s Update method is called by the game loop’s Update call, which the XNA Framework application model calls automatically once per frame (or more often, depending on how fast the game is running). Update is where you will move your game objects around, have them interact with each other, and take user input, which is what we’ll be doing in this Update method.

Next, we need to modify the Ship’s Update method to respond to input from the user to enable our new shield. Go to line 248 in Ship.cs, which is inside the Update method just before the check for "if (inHyperspace)", and add the following code:

if (XInputHelper.GamePads[player].XPressed && !shieldUsed)

{

inShield = true;

shieldUsed = true;

exitShieldTime = shieldTime;

Sound.PlayCue( Sounds.PhaseActivate );

}

One line of input, shown in the conditional statement, is all it takes to check for a keystroke or controller button press. XInputHelper, provided in the Spacewar Starter Kit, provides a wrapper around both the Xbox 360 controller and keyboard inputs to simplify input. Using XInputHelper, this if statement checks whether the X button is pressed for the player that the ship is associated with. If you are using a keyboard, the equivalent key is F for player 1, and Page Up for player 2.

You can easily change these keyboard mappings by modifying the settings.xml file in the Spacewar project. Settings.xml is unique to Spacewar (it’s loaded in SpacewarGame’s override of the Game.Initialize method), but XML data files are a good choice in general for game data. Since XNA Game Studio Express is built on the .NET Framework, the Xml namespace is exposed to games.

The code inside the conditional raises the shield, marks the shield as having been used this round, and starts the shield timer. It also uses the XNA Framework audio system to play a cue, which in turn plays a sound for the user. Spacewar abstracts the audio layer of the XNA Framework using enumeration values such as Sounds.PhaseActivate to identify the audio cues to play. Generally, you call cues using a string value. If the number of cues in your game is limited, such as in Spacewar, an enumeration is a good solution, especially if cue names are changing during development.

You add sounds to a game by using the Microsoft Cross-Platform Audio Creation Tool (XACT), provided with XNA Game Studio Express. All of the audio files and their associated cues are kept in an XACT project, which is displayed in a graphical interface that allows you to drag and drop sounds into the project. When you build your game, the XNA Framework Content Pipeline loads the XACT project and builds the necessary audio files. If you want to add a sound to Spacewar, you’ll have to take the extra steps of adding the cue name to the array of cue names and providing an enumeration value to the Sounds index into the cue name array. Then you can call your sound by using the Sounds enumeration.

In the Update call, you’ll want to add the necessary logic to run the shield timer if the shield is active. Go to line 348 (at the end of the "if (!Paused)" block), and add the following code:

if (inShield)

{

exitShieldTime -= elapsedTime.TotalSeconds;

if (exitShieldTime <= 0)

{

inShield = false;

Sound.PlayCue( Sounds.PhaseExpire );

}

}

This code simply decrements the time left on the shield by the elapsed time. Elapsed time is a measure of how much time has passed between calls to Update. This amount, applied to game logic such as motion or timers, simulates the passing of time, and is passed into Update by the XNA Framework. Once the shield time runs out, the shield is deactivated and an appropriate sound is played.

Rendering Details

While we’ve handled input and sound handily, the question remains: what will the shield look like? We’ll need to put some code into the Render method. So using some math and drawing functionality provided in the XNA Framework, we’ll draw a sprite—a 2D texture with transparency—on top of the ship’s 3D position.

In Solution Explorer, find the Content folder in the solution tree, open it, and then open the Textures folder beneath it. You’ll see a variety of .tga files. We’ll be using one of these as our shield texture. A .tga file represents a 2D graphical image, a sprite.

Go to line 433 in Ship.cs and make a new line immediately following it—after the end of the block for "if (showThrust && evolved)". In that new line, add the code shown in Figure 2.

Figure 2 Rendering a Sprite

if (inShield && evolved)

{

Texture2D shieldtexture =

SpacewarGame.ContentManager.Load<Texture2D>(

SpacewarGame.Settings.MediaPath + @”textures\circle” );

batch.Begin( SpriteBlendMode.AlphaBlend );

// Move into screen space.

Vector3 location = device.Viewport.Project(

position, SpacewarGame.Camera.Projection,

SpacewarGame.Camera.View, Matrix.Identity );

batch.Draw(

shieldtexture,

new Vector2( location.X - shieldtexture.Width / 2.0f,

location.Y - shieldtexture.Height / 2.0f ),

new Rectangle( 0, 0, shieldtexture.Width, shieldtexture.Height ),

new Color( new Vector4( 0.0f, 1.0f, 0.0f, 0.5f ) ), 0.0f,

new Vector2( shieldtexture.Width / 2.0f,

shieldtexture.Height / 2.0f ),

(3.0f * (float)(Math.Abs( Math.Cos( exitShieldTime ) ))) + 2.0f,

SpriteEffects.None, 0.0f);

batch.End();

}

Let’s walk through what’s happening in this code. First, using the XNA Framework and the XNA Framework Content Pipeline, the ContentManager class loads the texture called circle from the Content\Textures folder. The ContentManager class is an XNA Framework class that loads content from the Content Pipeline at run time. It is part of the prepopulated Application Model code in each new XNA Game Studio Express project. The name "circle" is associated with the circle.tga file in the Content\Textures folder. The XNA Framework Content Pipeline assigns a friendly name to each content file, which is used to load the content at run time.

Next, the texture is loaded into a Texture2D class provided by the XNA Framework. The Texture2D object will be drawn on the screen using a SpriteBatch (named batch in this code block), which represents a batch of textures that will be drawn on the screen at once.

We call batch.Begin to signal that sprite drawing is about to begin, and note that we want to use alpha blending. This will allow the circle texture to be transparent around its edges and partially transparent throughout, because we want to still be able to see the ship below the shield.

Before we draw, we must know where on the screen to draw. Spacewar Evolved is a 3D game, so the objects move around in three dimensions and are projected onto a 2D plane only when they are drawn. We must find the 2D position of the 3D ship before we know where we should draw the shield.

The XNA Framework allows us to get this information from the graphics device. We call device.Viewport.Project and pass in the 3D coordinates of the ship, as well as view and projection matrices. The view matrix defines the linear algebra transformation applied to each 3D object to place the object correctly relative to the player’s viewpoint. The projection matrix transforms the 3D coordinates for each object to 2D screen coordinates that allow the objects to be drawn to the screen. These matrices are often precomputed and stored. In Spacewar, they are kept in a Camera class that also defines the player’s viewpoint. Calling device.Viewport.Project with these values simulates transforming the ship’s location to 2D coordinates without actually drawing it at those coordinates. This call returns a vector that holds the 2D draw coordinates we will use to draw our shield.

The next step is to call batch.Draw, passing in the following:

- The Texture2D object.

- Our projected coordinates minus half the width and height of the sprite; this centers the sprite right on the ship.

- The source size of the texture.

- A color to multiply with the sprite’s existing colors. (We construct it as a Vector4, with three components representing red, green, and blue, and a fourth for alpha, or opacity. These particular values define a partially transparent green; the transparency allows us to see the ship underneath the shield, rather than obscuring it.)

- A scaling factor. (We’re being tricky here by taking the cosine of the remaining shield time, which makes the size of the shield fluctuate over time, just for graphical kicks.)

- A sprite effect flag that we set to None, but which could be used to flip the sprite horizontally or vertically.

- A screen depth. A value of zero makes the sprite draw above everything else, so it’s guaranteed to render on top of the ship instead of underneath it.

Energize Shields!

At this point a player can raise a shield, visualize it, and have the shield disappear after an appropriate period of time. Unfortunately, the shield doesn’t yet provide any protection. To get that functionality, let’s open SpacewarScreen.cs and add two simple blocks of code, one for each player. Go to line 94 in SpacewarScreen.cs, and add the following code to the beginning of HitPlayer2:

if (ship2.Shield)

{

// Make a bouncy sound rather than an explosion.

Sound.PlayCue( Sounds.WeaponPickup );

return;

}

And add the following at line 122, to the beginning of HitPlayer1:

if (ship1.Shield)

{

// Make a bouncy sound rather than an

// explosion.

Sound.PlayCue( Sounds.WeaponPickup );

return;

}

As described earlier, SpacewarScreen.cs represents the update logic for the Spacewar game loop. Many games, including Spacewar, utilize separate game logic loops for the various states of a game, such as the main menu, weapon selection, and the actual game itself. As the players progress through the games, control is handed off from state to state. In Spacewar, these states are called screens and are represented by the Screen class. SpacewarScreen is derived from Screen and handles the actual gameplay state, in which the two players control their spaceships and do battle.

In the SpacewarScreen class, the Update loop manages the game objects, such as the spaceships, weapons, and asteroids that float around the screen. When these objects collide with the spaceships, the logic calls HitPlayer1 or HitPlayer2, which calculates the damage to be dealt to the spaceships.

This code modifies the game logic in HitPlayer1 and HitPlayer2 so that when a ship is dealt damage, the shield is checked before any damage is resolved. If the shield is up, a deflection sound is played, and the damage function exits before taking any more action.

That’s it. Compile and run with F5, and try it out! From the Main Menu, select the Evolved game mode. Once you start the game, press the X button or the keyboard equivalent (F for player 1, Page Up for player 2) to raise the shield, as shown in Figure 3. Both players can use the shield only once per round, but the shield is very strong: it will absorb bullets of every type, asteroid hits will cause no damage—even the sun can’t hurt you!

Figure 3 Shields Energized! (Click the image for a larger view)

Take a spin around the block with the shield, but be sure not to get yourself too close to the sun when the shield runs out—it only lasts five seconds!

Some ideas about where to go from here: using some vector math, you could modify bullets to bounce off the shield rather than being absorbed. It could mean a painful surprise for an opponent when his own shots return to him! Or modify the damage system to make a shielded ship deal double damage when it strikes an opponent! Ramming speed!

The fun of writing games is in experimentation. With XNA Game Studio Express, it’s easy to get down to the good stuff and truly make a game your own creation.

Bonus Credit: Change Space!

XNA Game Studio Express doesn’t only help people worried about how to best code new features into their games. It also incorporates a feature set, mentioned earlier, called the XNA Framework Content Pipeline. The XNA Framework Content Pipeline helps you create 2D, 3D, and even audio content for your XNA Framework–based game.

Part of what the XNA Framework Content Pipeline does is to integrate the content building and compiling process into Visual C# Express. The added benefit of this is that you can launch certain types of content directly from Visual C# Express: you edit the content and then recompile to see the changes in your game. By compiling your content at the same time you compile your code, you can unearth content errors before you run your game.

The XNA Framework Content Pipeline supports various 2D graphics file formats, including .bmp, .jpg, .png, .tga, and .dxt. Two 3D file formats are also supported: .x and .fbx. Both of these 3D formats have file format exporters for a large number of both free and professional 3D graphics creation tools. The XNA Framework Content Pipeline is also designed to be extremely extensible, which means you may be able to find support for more of your favorite file formats by searching online.

Let’s explore the XNA Framework Content Pipeline using Spacewar. Back in XNA Game Studio Express, go to the Solution Explorer. Open the Content folder, then the Textures folder. Here, you’ll see many of the textures that Spacewar uses. Click the file B1_nebula01.tga; this is one of the backgrounds used when playing the game. The property grid at the bottom of the Solution Explorer should then show some relevant properties (see Figure 4). If you cannot see the property grid, right-click the B1_nebula01.tga file and click Properties.

Figure 4 Textures Properties

In the property grid, you’ll see several unique properties dedicated to the XNA Framework Content Pipeline. The Content Importer property is being used here. An importer is responsible for taking the source data—in this case a .tga file—and transforming it to the XNA Framework content DOM. The content DOM holds data that has been transformed and normalized by the importers, so that processors can work independently of file formats. The next property is the Content Processor property. The processor is responsible for converting data from the content DOM to an object suitable for use at run time—in this case, a sprite. The Asset Name property is used for loading the asset at run time; you can turn off processing of the asset by setting the XNA Framework Content property to false.

Let’s try editing the B1_nebula01.tga graphic file to see how it changes in Spacewar. You’ll need an image editor capable of working with a .tga file. If you don’t have one, you can download Paint.NET from getpaint.net.

Double-click the file to launch the associated editor for that file. Then go ahead and edit the file a bit (see Figure 5). One easy way in Paint.NET would be to change the hue and saturation by clicking the Adjustments menu, then clicking Hue/Saturation. Move the sliders to change the hue, saturation, and lightness of the graphic, then click OK to commit the changes. After you’ve changed the file, save it, and return to XNA Game Studio Express. Compile and run the Spacewar project by pressing F5. XNA Game Studio Express will detect that the file has been changed and will rebuild and redeploy the new asset (even to the Xbox 360 console). When you start an evolved game, you should see your modified background!

Figure 5 Editing a Texture in Paint.NET (Click the image for a larger view)

Blast Off!

These simple examples demonstrate only a fraction of the power of XNA Game Studio Express. The cross-platform combination of the XNA Framework and XNA Framework Content pipeline means that you’ll be able to focus on what really makes game development fun: great content and inventive gameplay.

Furthermore, we recommend that you join the XNA Creators Club so that you’ll be able to take your game development skills to the next level. Becoming a member of the club not only opens the door to developing your own games on the Xbox 360 platform, but also gives you access to exclusive content, samples, starter kits, and a lot more resources that will help you take your game from your imagination into reality.

Charles Cox is a Developer Educator in the Microsoft Game Platform Documentation Group. Previously a Technical Writer for DirectX and Microsoft Game Studios, Charles now develops and delivers XNA Game Studio Express tutorials and educational workshops.

Michael Klucher is a Program Manager on the XNA Game Studio team at Microsoft, working on the XNA Framework Content Pipeline and XNA Game Launcher for the Xbox 360 console. Contact Michael via his blog at klucher.com.

WPF

Customizing Controls For Windows Presentation Foundation

Shawn Wildermuth

Code download available at:WpfControls2007_05.exe(173 KB)

Browse the Code Online

This article discusses:

- Composite controls

- Styles and templates

- Custom properties

- Custom controls

This article uses the following technologies:

Windows Presentation Foundation

Contents

Using Composition

Using Styles

Using Templates

Authoring Controls

Custom Properties

Where Are We?

The breadth of the control model in Windows Presentation Foundation is quite staggering, but it is impossible to supply every control that you could ever need. That is where control authoring comes to the rescue. In this article, I show you how to customize existing controls using Windows® Presentation Foundation and how to create entirely new controls (or elements) for use in your projects.

Before you develop a custom control, you need to ask yourself if you really need one. In Windows Presentation Foundation, composition, styling, and templating enable you to customize existing controls to an extent unprecedented with past technologies. Before you decide to create a new control, let’s quickly review these three methods for customizing controls.

Using Composition

A common requirement is to create a composite control—a control made up of more than one control. Suppose you have a Play button to start playback of a video. The XAML and control are shown in Figure 1.

Figure 1 A Simple Play Control

<StackPanel> <Button Height="50" Width="50" Content="Play" /> <Polygon HorizontalAlignment="Center" Points="0,0 0,26 17,13" Fill="Black" /> </StackPanel>

You need to be able to take the play icon and put it on the button. You can use composition to actually embed XAML elements inside other XAML elements. For example, you can change the XAML to create the label and icon as the contents of the button. Placing these elements inside a container (StackPanel in this case) inside the button assigns them to the Content property of the Button class, as seen in Figure 2. This results in a button that works like any other button but has your content inside it.

Figure 2 All of the Content Inside the Button

<StackPanel> <Button Height="50" Width="50"> <StackPanel> <TextBlock>Play</TextBlock> <Polygon Points="0,0 0,26 17,13" Fill="Black" /> </StackPanel> </Button> </StackPanel>

Using composition to create controls like this is simple. Unlike in presentation technologies like Windows Forms, Visual Basic® 6.0, and MFC, most controls are containers for other controls. It’s unnecessary to write a custom control when all you really need is a composite control.

Using Styles

What if all you need is to change the appearance of the control? Styles are the answer. You can specify a style of button that has a red border around it by creating a Style like this one.

<StackPanel> <StackPanel.Resources> <Style TargetType="Button" x:Key="RedButton"> <Setter Property="BorderBrush" Value="Red" /> </Style> </StackPanel.Resources> ... </StackPanel>

Now you can change the border of specific buttons by assigning a style to them, as seen in Figure 3. The first button is the standard appearance, while the second one binds itself to a shared style.

Figure 3 Applying a Style to a Button

Figure 3

<Button Height="50" Width="50"> <StackPanel> <TextBlock>Play</TextBlock> <Polygon Points="0,0 0,26 17,13" Fill="Black" /> </StackPanel> </Button> <Button Height="50" Width="50" Style="{StaticResource RedButton}"> <StackPanel> <TextBlock>Play</TextBlock> <Polygon Points="0,0 0,26 17,13" Fill="Black" /> </StackPanel> </Button>

You can even use styles to change the appearance of all instances of a certain type of XAML element across a container. For example, instead of creating a reusable style to change the button, you could create a style that specifies the look of all buttons, as seen in Figure 4. This example sets the background of all buttons to a gray/green/gray gradient. The Style in this example omits the Key of the style. This causes it to affect all the elements specified in the TargetType attribute.

Figure 4 Applying a Style Across a Container

Figure 4

<StackPanel> <StackPanel.Resources> <Style TargetType="Button"> <Setter Property="Background"> <Setter.Value> <LinearGradientBrush> <GradientStop Color="#DDDDDD" Offset="0" /> <GradientStop Color="#88FF88" Offset=".6" /> <GradientStop Color="#EEEEEE" Offset="1" /> </LinearGradientBrush> </Setter.Value> </Setter> </Style> </StackPanel.Resources> <Button Height="50" Width="50"> <StackPanel> <TextBlock>Play</TextBlock> <Polygon Points="0,0 0,26 17,13" Fill="Black" /> </StackPanel> </Button> <Button Height="50" Width="50"> <StackPanel> <TextBlock>Play</TextBlock> <Polygon Points="0,0 0,26 17,13" Fill="Black" /> </StackPanel> </Button> </StackPanel>

Using Templates

Styles are limited to setting default properties on XAML elements. For example, when I set the BorderBrush in the earlier examples, I could specify the brush but not the width of the border. For complete freedom of a control’s appearance, you need to use templates. To do this, you create a style and specify the Template property (see Figure 5). The Value of the Template property becomes a ControlTemplate element that specifies how to compose the control itself. In this example, I specify a button that is a circle with the play icon in the center. I do this by layering the play icon over an Ellipse element. The new template button can be seen next to a normal button.

Figure 5  Using a Template

Figure 5

<StackPanel> <StackPanel.Resources> <Style TargetType="{x:Type Button}" x:Key="PlayButton" > <Setter Property="Template"> <Setter.Value> <ControlTemplate TargetType="{x:Type Button}"> <Grid> <Ellipse Width="{TemplateBinding Width}" Height="{TemplateBinding Height}" Stroke="DarkGray" VerticalAlignment="Top" HorizontalAlignment="Left" Fill="LightGray" /> <Polygon Points="18,12 18,38 35,25" Fill="Black" /> </Grid> </ControlTemplate> </Setter.Value> </Setter> </Style> </StackPanel.Resources> <Button Height="50" Width="50">Normal Button</Button> <Button Height="50" Width="50" Style="{StaticResource PlayButton}" /> </StackPanel>

In the end, styles and templates still only allow you to change the appearance of a control. To add behavior and other features to the button, you’ll need to create a custom control.

Authoring Controls

The first step you should take before writing your own control is to decide which method you will use for creating the control. There are two main ways to create controls in Windows Presentation Foundation: user controls and custom controls. There are benefits to both approaches.

With user controls, you get a simple development model that is similar to Windows Presentation Foundation application development. User controls are preferred when you need to compose a control out of existing components and do not need complex customization (as with templates and styles). Custom controls are a better choice when you want full control over appearance, need special rendering support, or want your control to be a container for other controls.

If you can’t decide which type of control to choose, pick a user control. If you run into a functionality wall with a user control, you can switch later to custom with little pain.

The first thing to do when creating a user control is to add a new item to your project. If you right-click your project and click Add, you might be tempted to pick the User Control option from the context menu. Unfortunately, this attempts to create a new Windows Forms user control. Instead, pick the Add New Item option. In the Add New Item dialog, pick the User Control (WPF) item.

Creating the new user control creates a new XAML file and backing code file. The XAML file is similar to the main file that is created with new Windows Presentation Foundation projects; the difference is that the root element of the new XAML file is a UserControl element. Inside the UserControl element, you create the content that composes your control.

For this example, continue with the same XAML used earlier to create a template for a PlayButton control. This new control will tie itself to a MediaElement to control playing or pausing some digital media. Figure 6 shows the PlayButton’s XAML.

Figure 6 PlayButton XAML

Figure 7 PlayButton in a WPF Window

<!-- PlayButton.xaml --> <UserControl x:Class="CustomWPF.PlayButton" xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation" xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"> <Grid> <Ellipse Width="50" Height="50" Stroke="DarkGray" VerticalAlignment="Top" HorizontalAlignment="Left" Name="ButtonBack" Fill="LightGray" /> <Path Name="PlayIcon" Fill="Black" Data="M18,12 18,38 35,25"/> <Path Name="PauseIcon" Fill="Black" Opacity="0" Data="M15,12 15,38 23,38 23,12z M27,12 27,38 35,38 35,12" /> </Grid> </UserControl>

From the template example, I have added a new Path, PauseIcon. Because the icon for pausing of the media could not be represented as a Polygon, it was easier to simply change PlayIcon to a Path so you can deal with each icon as a Path object in the codebehind. I want to be able to control a MediaElement element by pausing or playing the media when the button is clicked as well as changing the icon to correctly represent the action (pause or play) when the button is clicked.

Before adding that logic, let’s make sure that my button is displayed correctly and that it can be shown in a window. In this case, I want to show the new control on a window. Before you can use any custom element (user control or custom control) in a XAML document, you must create a reference to it. If the custom element is in the same project as the rest of your XAML, you can just refer to it by adding an XML namespace declaration. In the following lines of code, I created a namespace declaration (xmlns:cust) that specifies a common language runtime (CLR) namespace that the control is in.

<!-- MainWindow.xaml --> <Window x:Class="Tester.MainWindow" xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation" xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml" xmlns:cust="clr-namespace:CustomWPF" Title="Control Viewer" Height="100" Width="200"> <!-- ... --> </Window>

The clr-namespace (CustomWPF) specified in the XML namespace declaration matches the actual CLR namespace of the control (CustomWPF). If the control you want to use is in another assembly, you must also note the assembly name in the namespace declaration. The XML namespace declaration does not import the assembly into your project automatically; you must also add a reference to the assembly to your project manually.

<!-- MainWindow.xaml --> <Window x:Class="Tester.MainWindow" xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation" xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml" xmlns:cust="clr-namespace:CustomWPF;assembly=CustomWPF" Title="Control Viewer" Height="100" Width="200"> <!-- ... --> </Window>

Once you have the reference to the XML namespace, you can use it to create instances of your new user control. You do this using the name of the namespace declaration. This means the XML namespace alias, not the CLR namespace. After the XML namespace alias, you would specify the actual name of the control. This ensures that the name used in the XAML file matches the name of the class. To add an instance of the PlayButton class to the XAML you would specify cust:PlayButton as the element name.

<StackPanel> <TextBlock HorizontalAlignment="Center">User Control:</TextBlock> <cust:PlayButton /> </StackPanel>

Now you can see the PlayButton control hosted in a typical Windows Presentation Foundation Window as in Figure 7.

Custom Properties

As you author controls, you will find it necessary to implement properties to manage both the appearance of the control as well as its runtime behavior. For example, the PlayButton control will need the ability to get and set the color of the icon. To do this you can create a simple CLR property as seen in Figure 8 (note that Visual Basic sample code is available in the download for this article).

Figure 8 Getting the Icon’s Color

// PlayButton.xaml.cs public partial class PlayButton : System.Windows.Controls.UserControl { // ... Brush _iconColor = Brushes.Black; public Brush IconColor { get {return _iconColor; } set { _iconColor = value; PlayIcon.Fill = _iconColor; PauseIcon.Fill = _iconColor; } } }

Simple CLR properties like the IconColor property work well enough. You can set them in the XAML just using the name of the property:

<cust:PlayButton IconColor="Black" />

Simple properties are not sufficient for most controls, though, because they do not support advanced features like data binding or animation support. For example, if you want to specify the IconColor of your control by data binding to the fill color of a rectangle in the XAML, it does not work, as you can see in Figure 9.

Figure 9 This Data Binding Won’t Work

<!-- MainWindow.xaml --> <Window x:Class="Tester.MainWindow" xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation" xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml" xmlns:cust="clr-namespace:CustomWPF;assembly=CustomWPF" Title="Control Viewer" Height="100" Width="200"> <StackPanel> <Rectangle Name="theRect" Fill="Red" /> <TextBlock>User Control:</TextBlock> <!-- Simple Assignment Works --> <cust:PlayButton IconColor="Blue" /> <!—Data Binding Does Not --> <cust:PlayButton IconColor="{Binding ElementName=theRect, Path=Fill}" /> </StackPanel> </Window>

To exploit all the features available, you must use a Dependency Property instead of a simple CLR property. Dependency properties allow the value of an element to be set by a variety of means, including animations and data binding. To implement one, create a static (shared in Visual Basic) DependencyProperty field on the control by calling DependencyProperty.Register method. This method registers your property and returns an instance of the created DependencyProperty.

Once you have the DependencyProperty field, you can use it to set and get the property by using the GetValue/SetValue methods of your control. For example, you can change the IconColor property of the PlayButton to a DependencyProperty as seen in Figure 10.

Notice that I removed the Brush field from the class. The DependencyProperty stores the value for each instance as well as the metadata about the property. This means that each instance of the PlayButton does not need to have its own field to store the data about the property.

Now that you have a DependencyProperty, you can use it in data binding or animation. Currently there is no good way of determining that the property has changed, nor is there a default value. Because setting the value of the property is not routed through the public property (the CLR property is a wrapper for the DependencyProperty, not vice versa), you cannot just use the set accessor to change the icon color when that value is changed. You need to add an event to be called when the property changes. To do this, specify a static or shared method to be called back when the property changes when you register the DependencyProperty. You can amend the registration to include a FrameworkPropertyMetadata object that specifies both the default value and a change callback, as shown here:

public static readonly DependencyProperty IconColorProperty = DependencyProperty.Register( "IconColor", typeof(Brush), typeof(PlayButton), new FrameworkPropertyMetadata(Brushes.Black, new PropertyChangedCallback(OnIconColorChanged )));

Finally, you need to implement the callback. This callback is a static (or shared) method of your control that accepts the object that was changed as well as arguments that specify the old and new values. Typically, you would call a method on the changed object to update it. For example, if the IconColor has changed, you will want to set the Fill of both icons. Both the callback method and the update method are shown here:

private static void OnIconColorChanged(DependencyObject obj, DependencyPropertyChangedEventArgs args) { // When the color changes, set the icon color PlayButton control = (PlayButton)obj; control.PlayIcon.Fill = control.IconColor; control.PauseIcon.Fill = control.IconColor; }

To complete the control, you will also want a DependencyProperty to allow assigning of a MediaElement to control (see Figure 11).

Now that you have the property, you can add a MediaElement to the XAML and data bind that element to the new MediaPlayer property. When you add the MediaElement to the XAML, you need to set the LoadedBehavior to Manual, which allows you to manually control the playback. The new XAML is shown in Figure 12.

Figure 12 Revised MediaElement XAML

<!-- MainWindow.xaml --> <Window x:Class="Tester.MainWindow" xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation" xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml" xmlns:cust="clr-namespace:CustomWPF;assembly=CustomWPF" Title="Control Viewer" Height="100" Width="200"> <StackPanel> <MediaElement Width="150" Height="100" Name="theMedia" Source="http://download.microsoft.com/.../ctorrec9billg.wmv" LoadedBehavior="Manual" /> <TextBlock>User Control:</TextBlock> <cust:PlayButton MediaPlayer="{Binding ElementName=theMedia}" /> </StackPanel> </Window>

Next, implement a click event on the PlayButton user control. First, add a MouseLeftButtonUp event on the control’s main Grid.

<!-- PlayButton.xaml --> <UserControl x:Class="CustomWPF.PlayButton" xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation" xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"> <Grid MouseLeftButtonUp="PlayButton_Clicked"> <!-- ... --> </Grid> </UserControl>

This allows you to implement the behavior of your control as well as changing the icon. This event handler implementation is seen in Figure 13. The completed control and video can be seen in Figure 14.

Figure 14 Completed User Control

Custom Controls

The PlayButton control you just created works pretty well, but it lacks full template and theme support. If your control needs this support, you will need to build it as a custom control. Custom controls derive from other classes in the control hierarchy. For example, you can refactor the PlayButton control into a MediaButton control that is more reusable.

To create a MediaButton control, first pick Custom Control (WPF) in the Add New Item dialog of Visual Studio. This adds a new custom control class (MediaButton.cs) file to the project and also adds a theme folder with a generic.xaml file. The generic.xaml file contains a template for the new control class. It uses this XAML file to allow different themes for the control. The generic.xaml file is used as a fallback; for most controls this is the only theme file you will create. If you want to write a control that changes its appearance depending on the current theme, you can create theme files in this directory. Figure 15 shows the standard theme files and when they are used.

By specifying multiple themes, you can change the appearance of your controls based on the theme selection of the user. To develop a generic theme for my sample control, I can take the user control’s XAML file and place it inside the ControlTemplate tag. Once the XAML is in the template, you will want to use template binding to set properties of the XAML based on properties of the control. Until now, the PlayButton’s width and height were always set to fifty logical units. For a simple control that makes sense, but if you really want the control to be reusable, you should make the template resizable. To do this, replace the height and width of the template with the control’s height and width by marking the height and width in the template to {TemplateBinding Width} and {TemplateBinding Height}.

In the PlayButton, simply change the Opacity of the two icons based on whether it needs to show play or pause. Reaching down into the template to change the Opacity would work, but it would be rather clumsy. A better solution is to have a single icon Path object and change the drawing data to draw the right icon. This way, the size of the visual tree directly influences the performance of the XAML document. To make this work, introduce a new DependencyProperty to store the current icon to use. Create an enumeration that specifies which icon to use, then expose it as a DependencyProperty. With the new Icon property, you can modify the template to include triggers for when the Icon property changes.

The icon is using a Path element to define the appearance of each icon used by the button. This works fine if the button is always a fixed size, but since it is not, you need to find a way to resize the icon with the button. One solution is to create an overlay ellipse over the background circle and use a VisualBrush to paint the icon. The VisualBrush allows the background to size with the control. The completed generic.xaml template is shown in Figure 16.

This MediaButton should control a MediaElement like the one the PlayButton did. Copy the MediaPlayer DependencyProperty from the PlayButton to the MediaButton. Make sure to change any references in the copied code from PlayButton to MediaButton (especially in the DependencyProperty registration).

Unlike PlayButton, you do not need to handle the mouse click event in the template. Instead, you can override the OnMouseLeftButtonUp event to react to clicks. Inside this method, you can change the icon and play or pause media. The final custom control code can be seen in Figure 17. Now that the new control supports resizing and setting the icon property, you can give users more flexibility in changing the size and icon of the control. Figure 18 shows the control with different sizes and icons.

Figure 18 All Shapes and Sizes

In this control example, I have directly derived from the System.Windows.Controls.Control class, but the nature of custom controls allows deriving from anywhere in the class hierarchy. You can use custom controls to override and change the behavior of built-in controls or build your own controls completely from scratch. For example, deriving from FrameworkElement would allow you to create a control with very little built-in layout mechanics. Deriving from Panel allows you to create your own specialized containers for other objects. Determining the right class to derive from is not easy; it depends on the requirements for your control.

Where Are We?

When you need specialized control functionality, you have several options, including composition, styles, and templates. By using componentization, you can often create compound controls, obviating the need for authoring a new control. When the appearance of a control is all you need to change, use styling. Lastly, templates allow complete control over the composition of an existing control. For more information on using templates to customize controls, see the Foundations column by Charles Petzold.

When you do decide to author a new control, you’ll still get the simplified programming model that appears to be just like writing your own Window or Page. Being able to create templates for your controls that look and even act differently in different operating system themes is a distinct advantage of custom controls.

Migrating an existing user control to a custom control is not particularly difficult, but because you are working with a template, instead of direct access to XAML objects, you will need to change how you engineer your custom control.

With Windows Presentation Foundation, the necessity for writing custom controls is the exception rather than the rule. Only when you are really creating custom behavior should you ever need to delve into the control authoring. For more information on controls in Windows Presentation Foundation, please see the SDK at msdn2.microsoft.com/ms754130.aspx.

Shawn Wildermuth is a Microsoft MVP, MCSD.NET, MCT, and the founder of Wildermuth Consulting Services. Shawn is also the author of Pragmatic ADO.NET (Addison-Wesley, 2002), and the coauthor of four Microsoft Certification Training Kits as well as the upcoming Prescriptive Data Architectures. He can be contacted through his Web site at www.wildermuthconsulting.com.

Migration

Convert A Java Web Application To ASP.NET Using JLCA

Brian Jimerson

This article discusses:

- Resource locating

- I/O streams

- Logging and collections

- Refactoring

This article uses the following technologies:

ASP.NET, JLCA, and C#

Code download available at: JLCA2007_05.exe (157 KB)

Browse the Code Online

Contents

About the JLCA

Locating Resources

Handling Input/Output APIs

Logging

Collections

Filters and HTTP Handlers

Source Tree and Naming Conventions

When to Refactor for Conventions

Directory Layout and Namespaces

Properties

Pascal-Casing Method Names

Conclusion

The typical software development cycle follows a straightforward model: gather requirements, design the application, write the code, test the software, and deploy it. Once in a while, though, a new development project is incepted simply based on the platform to which the customer wants to deploy the app. In this case, an existing application’s code base can be converted, or ported, to the desired platform.

In this article, I walk you through converting a Java Web application to an ASP.NET application implemented in C#. This article is based on an actual project I worked on. In that project, we had an existing Java-based application and a customer who wanted an ASP.NET version of it. I begin by introducing you to the Microsoft® Java Language Conversion Assistant (JLCA) and I demonstrate common development paradigms that do not have direct counterparts in the two platforms, such as:

- Input/output

- Resource resolution

- Source tree layout and naming conventions

- Leveraging the executing environment

This application is implemented as a SOAP-compliant Web service, with a traditional relational database persistent store. I won’t discuss the actual Web service presentation layer and SOAP interfaces, but rather the application that powers it. Sample code for this article is available for download.

About the JLCA

The JLCA is a tool used to convert a Java application into a C# application. The tool has shipped with Visual Studio® since Visual Studio .NET 2003. Currently in version 3.0, the JLCA is included with Visual Studio 2005 and is also available as a free download from the JLCA home page.

Version 3.0 includes enhancements to convert Java artifacts, such as Servlets and Java Server Pages (JSPs), as well as rich-client applications that use Swing or the Abstract Windowing Toolkit (AWT). In practice, the JLCA provides a very good place to start a conversion, but it will not successfully complete the entire process. So don’t expect this to be completely hands-free—you will still need to do some manual resolution of converted items after using the tool.

To get started with the JLCA in Visual Studio 2005, launch the wizard by clicking File | Open | Convert. The wizard’s screens are quite self-explanatory. The key piece of information you’ll need to enter, as shown in Figure 1, is the root directory of your existing Java project.

Figure 1 Enter the Root Java Directory in the JLCA Wizard (Click the image for a larger view)

The JLCA then proceeds to convert the Java source code to C#. The process doesn’t take too long. Anecdotally, our code base with approximately 100 class files took less than 10 minutes for the conversion to complete—just enough time to get a cup of coffee. Of course, this number will vary for different projects and systems.

After the conversion process is finished, the JLCA creates an HTML report of errors and warnings. These items are also entered into the generated C# code as comments by the offending members. To help you, each item also contains a hyperlink for more information on resolving the problem.

Many of the warnings can be safely ignored. They simply note differences in behavior between Java and C#, such as the warning that states: "Type castings between primitive types may have different behavior." Still, you should look at each warning to ensure these different behaviors won’t affect your application.

When you first look at the conversion report, there may seem to be an overwhelming number of issues reported. In this case, there were 816 errors and 16 warnings. Most of the errors, however, could be categorized into one of three categories and addressed pretty easily. The three categories they fell into were:

- Members for which there are simply no C# or Microsoft .NET Framework equivalents.

- Popular third-party Java libraries that don’t have direct equivalents in the .NET Framework (such as Hibernate and log4j).

- Items that deal with class loading or resource resolution.

It is also worth noting that the JLCA does not seem to attempt to resolve imported packages (or using namespace statements) that it can’t find. Instead, it just passes them through to the generated C# code. If you were to try to compile your new C# application, you would probably get quite a few more compiler errors than the conversion report indicates.

Not to worry, though. As stated previously, the bulk of these errors falls into the same recurring patterns and can be addressed en masse. I look at resolutions to these common errors in the following sections, and I cover the other necessary tasks to make your converted app a true C# application.

Locating Resources

The process of resource locating—specifically, finding and loading file resources—in Java differs significantly from resource locating in C#. Java uses a class loader to load and parse a class definition at run time. Part of the responsibility of a class loader is to manage the context of a class and provide environmental facilities for classes loaded by it. In Java, a special type of environment variable, called a classpath, is used by a class loader to locate resources. A classpath is similar to a path environment variable in that it defines where a class loader should look for other classes and resources. An application that wishes to load another class or resource can do so by telling the class loader the location of the file relative to the classpath.

A very common approach to resource resolution in Java is to use a special type of file called a properties file for configurable information, such as connection or host information, paths to other resources, localization strings, and credentials to use for authentication. A properties file contains name=value pairs, which are separated by a new line. This format is very similar to INI files, but without the sections.

Furthermore, resource locating is always performed using file system notations—whether they are actually located in the file system. This alleviates the developer’s burden of having to know how the application is being deployed. Other types of resources, such as images and binary files, are located and loaded in this same manner. Here’s an example of locating and using a properties file in Java:

InputStream is = this.getClass().getResourceAsStream(

“/application.properties”);

Properties properties = new Properties();

properties.load(is);

String myValue = properties.getProperty(“myKey”);

On the contrary, resources in the .NET Framework can be deployed and loaded in two different ways: as an embedded, binary resource with an assembly or as a file in the local file system.

The appropriate technique for accessing a resource depends on its location. For example, if it is a file system resource, you might do something like the following:

string fileName = Path.Combine(Path.GetFullPath(

@”..\config\”), “properties.xml”);

Stream fileStream = File.Open(fileName, FileMode.Open);

//Do something with the stream

If, however, the resource is embedded in an assembly, your approach will be more like this:

Assembly assembly = Assembly.GetExecutingAssembly();

Stream fileStream = assembly.GetManifestResourceStream(

GetType(), “properties.xml”);

//Do something with the stream

My team’s application was written with the assumption that it would not have to know how the resource was deployed to be able to resolve. Since there are many resources being loaded throughout the application, it would have taken a lot of effort to analyze each resource being loaded, determine how the resource is deployed, and then modify the code to load it appropriately. So instead we created a utility class called ResourceLocator.

ResourceLocator was designed to approximate a Java class loader’s ability to resolve a resource based on its classpath. Since all of the calls to load resources were written with this approach, it seemed like the least intrusive method of conversion. All we would need to do after ResourceLocator was written would be to change calls from Class.getResourceAsStream to ResourceLocator.LocateResource. This could be done with a simple find and replace in Visual Studio.

Basically, ResourceLocator takes the name and relative path of the resource to find and then attempts to find it by walking through available assemblies and the local file system. There are also overloaded methods that provide more granular control, such as specifying the order in which areas should be searched and choosing to search only assemblies or the file system. The source for ResourceLocator is included in the code samples on the MSDN® Magazine Web site.

You might be thinking that it is very costly to look at all of the available locations to find a resource, and this is certainly true. But all of the resources located and loaded in the application were then cached. This means that each resource is only loaded once during the execution of the application, and thus this overhead is mitigated (though it does increase memory consumption, which could be an issue if there are a large number of resources being loaded). Therefore, we decided the trade-off was acceptable, considering how many code changes that it saved us. These types of changes can also be modified over time, starting with the simple solution to get the port up and running quickly, and then slowly but surely changing the implementation to better match design and implementation guidelines for .NET-based applications.

Handling Input/Output APIs

There are several differences between the I/O APIs in Java and those in .NET that need to be dealt with after conversion. One significant difference is that .NET I/O streams are bidirectional, whereas Java I/O streams are unidirectional. This means that in .NET programming, you can, in theory, read from and write to the same stream. In Java, however, you can only read from a stream or write to a stream, but not both to the same stream. This difference doesn’t pose much difficulty during the conversion since this is a widened difference and the .NET streams provide at least as much functionality as their Java counterparts.

If the safety of unidirectional streams needs to be maintained, then the .NET I/O readers and writers can be utilized. These will wrap an underlying stream to provide reading or writing functionality. The result will be I/O operations that are programmatically similar to Java I/O operations.

For our application, direct access to streams is sufficient. The JLCA properly converts I/O operations, so no changes are necessary for compilation. However, we did carefully review the converted code since there was a significant opportunity for logic errors in these low-level operations.

Logging

Logging concerns the ability to write messages to a destination at particular points of execution in the code, such as caught exceptions, areas in logic where debugging information might be useful, and configuration information being loaded. Java and the .NET Framework both provide powerful frameworks for logging information, but their designs and implementations are very different.

In Java, application logging is usually accomplished through the Apache Software Foundation (ASF) log4j framework or the Java Logging APIs distributed with recent versions of the Sun Microsystems Java Development Kit (JDK). The Java Logging API is very similar in execution to log4j and, therefore, the two frameworks can be applied interchangeably for the purposes of this discussion. For .NET-based applications, the Microsoft Enterprise Library provides a robust application block for logging, called the Logging Application Block (see the Microsoft Logging Application Block homepage).

The standard logging frameworks in both Java and .NET provide strong capabilities, with support for design-time configuration, a host of destinations for logs (such as databases, files, and e-mail recipients), and so on. Note, however, that the API designs are different and thus need some manual intervention on your part during conversion.

To better understand the differences in the APIs, consider the two code snippets shown in Figure 2. These demonstrate a common logging scenario performed in Java and in C#. The snippet using log4j creates a static instance of a Log object from the getLog factory method and assigns it to the MyClass category. The code then prints some information to the Log at a debug level. The C# snippet, which uses the Logging Application Block, creates a new instance of a LogEntry class. This represents an entry in the destination log. The code then assigns the log entry a priority of 2, sets the category as Debug, and provides a message. It is then written to the Logger class.

Figure 2 Logging in Java and in C#

Java with log4j

private static final Log log = Logger.getLog(MyClass.class);

...

//Somewhere else in the class

log.debug(“Printing some debug information.”);

C# with the Logging Application Block

Logger.Write(”Printing some debug information.”, “Debug”);

It is important to note that logging in both platforms is controlled by external configuration. Information such as logging destinations, selective filtering of log messages, and formatting of log entries is contained in this configuration. I’ve intentionally left the configuration out of this example, as it isn’t pertinent to the discussion.

Looking at these two examples, you can see that they perform very similar functions, but they are achieved in different ways. The log4j example uses the category to determine if a message is logged (based on its level) and where it is logged to. The Logging Application Block, meanwhile, uses a combination of filters for priorities and categories to determine what is logged.

The Logger class provides several overloaded Write methods, each with varying levels of flexibility (including an overload that allows you to provide a LogEntry instance which has lots of knobs and controls for tweaking how the information is logged). However, given the simple overload that’s used in Figure 2, we were able to accomplish the bulk of the conversion process with searching and replacing, in combination with regular expressions.

Logging is a very resource-intensive process and care needs to be taken so that it doesn’t affect your application’s performance. In the end, it only took a few hours to search and replace the logging invocations in the application.

Collections

Both the .NET Framework and Java offer a strong collections API. Both are very extensible and cover most of the scenarios you might encounter when working with collections. Let’s look closely at two frequently used types of collections: lists and dictionaries.

Lists are collections that can be accessed by an index. They are ordered collections and can be thought of as a kind of one-dimensional array. Dictionaries, on the other hand, are collections of name and value pairs. The name is the key used to access values in the collection. Note that there is not necessarily guaranteed ordering in dictionaries.

Figure 3 lists the equivalent C# and Java implementations for list and dictionary. The JLCA does a very good job of converting these Java collection classes to their .NET equivalents and there isn’t a lot left to be done after the conversion. At that point, warnings are generated stating that behaviors for these common collections are different on the two platforms, so the conversion should be verified. However, the vast majority of the conversion process should be successful and valid.

Figure 3 Equivalent Collection Classes in .NET and Java

.NET

Java

List interface

IList, IList<T>

List

Common list classes

ArrayList, List<T>

ArrayList, Vector

Dictionary interface

IDictionary, IDictionary<TKey,TValue>

Map

Common dictionary classes

Hashtable, Dictionary<TKey,TValue>

HashMap, HashTable

That said, we encountered a problem with converting collections where we had used specialized collections in the original Java application. An example of this issue is the use of the Java LinkedHashSet class. According to the Java API documentation, the LinkedHashSet class ensures consistent ordering of entries in the HashSet (which is a type of dictionary), without the overhead imposed by other ordered dictionaries. While the definition of a LinkedHashSet is straightforward, the intent of its usage in the application was unclear. (The code was written by someone no longer on the team, and there wasn’t documentation as to why it was used.) Furthermore, the context of its usage didn’t give us anything, so it was unclear whether this class was used to fix a problem or perhaps for some other reason.

In looking through the code, we found no justification for its usage, and thus we had three options: assume it wasn’t actually needed in the original application, write our own implementation for our .NET app, or pick the closest appropriate collection in the .NET Framework. We assumed the specialized implementation of a LinkedHashSet was unnecessary, since there was no indication that ordered name value pairs were used anywhere else. As such, we substituted the basic .NET Hashtable class, and we verified the correct behavior with our existing unit and integration tests. However, if we had discovered performance or functional issues, we could have substituted the SortedDictionary<TKey, TValue> class in the .NET Framework 2.0 which represents a collection of key/value pairs that are sorted on the key; internally its implementation uses a set based on a red-black tree data structure.

In our project, there were only four uses of specialized Java collections classes, and they all presented similar circumstances. The additional functionality they provided was unused, and using the more generic .NET counterparts accomplished the task at hand.

I should note that our Java application was written using Java version 1.4. Generic collections, which allow strong typing of collection members, were not introduced to Java until version 1.5. Therefore, we did not have to delve into converting generics. Presumably converting collections in Java version 1.5 will thus add another layer of complexity to the conversion process since not only do the collections need to be converted, but so do their typed entries.

Filters and HTTP Handlers

A filter is a common paradigm used in J2EE Web applications to selectively intercept requests and responses to perform pre- and post-processing. Some common usages for filters are logging, usage auditing, and security.

Java filters implement the Filter interface, which defines particular lifecycle events. Filters are invoked by the application server by using a URL mapping to map Filter classes to full or partial URLs. When a match is made, the application server implements the lifecycle events of the mapped filter, passing a handle to the request and the response.

ASP.NET provides similar functionality in the form of an interface called IHttpHandler. Both filters and HTTP handlers have very simple interfaces, but are very powerful in what they can do. In our application, we had two different types of filters—one that used GZip compression to compress responses and one that intercepted requests to see if the request was from a known user agent.

The compression filter was implemented in the Java application to improve performance. It took every response stream, checked to see if the client supported GZip compression, and if so, compressed the response. Typically, this is unnecessary, as most modern HTTP servers provide this functionality without custom code. However, our application was contained as a Servlet application, which doesn’t necessitate an HTTP server, so the compression functionality was a value-added module.

The second filter, which rejected requests based on a user agent header value, is of more interest. Many of our clients wished to implement a level of authentication that compared the HTTP user agent header against a list of allowed agents. If the incoming request’s user agent value wasn’t an allowed user agent, the request should be rejected (usually by returning an HTTP unauthorized return value).

This type of request/response filter can be accomplished many ways. Most solutions, though, require a lot of injection of code or attributes. Luckily, J2EE and .NET both provide very simple mechanisms for intercepting, modifying, and adjusting requests and responses at an application level. Additionally, HTTP interceptors like these are not code-pervasive, meaning they are not lines of code in every class. Rather, they are separate classes that are managed and injected via the application server, so it is much easier to modify the functionality rather than perform a global search and replace.

The JLCA 3.0 does provide helper classes that aid in the migration of filters from Java to ASP.NET applications. Figure 4 shows a sample filter in Java that times the server processing, and it also shows how the JLCA attempts to convert this filter for ASP.NET. While in an ideal world the way to port to a .NET implementation probably requires rewriting the filter from scratch as an HTTP handler, JLCA provides the support class SupportClass.ServetFilter which gets you most of the way through a reimplementation. ServletFilter emulates the life cycle Java provides. While it doesn’t solve all problems, it can ease a port implementation.

Figure 4 JLCA Conversion Using ServletFilter

Java

import javax.servlet.*;

import java.io.*;

public final class TimerFilter implements Filter

{

public void doFilter(ServletRequest request,

ServletResponse response,

FilterChain chain)

throws IOException, ServletException

{

long startTime = System.currentTimeMillis();

chain.doFilter(request, response);

long stopTime = System.currentTimeMillis();

System.out.println(“Time to execute request: “ +

(stopTime - startTime) + “ milliseconds”);

}

public void destroy() {}

public void init(FilterConfig fc) {}

}

C#

using System;

using System.Web;

// UPGRADE_TODO: Verify list of registered servlet filters.

public sealed class TimerFilter : SupportClass.ServletFilter

{

public override void DoFilter(HttpRequest request,

HttpResponse response, SupportClass.ServletFilterChain chain)

{

long startTime =

(System.DateTime.Now.Ticks - 621355968000000000) / 10000;

chain.doFilter(request, response);

long stopTime =

(System.DateTime.Now.Ticks - 621355968000000000) / 10000;

Console.Out.WriteLine(“Time to execute request: “ +

(stopTime - startTime) + “ milliseconds”);

}

public void  destroy() {}

// UPGRADE_ISSUE: Interface ‘javax.servlet.FilterConfig’

// was not converted.

public void  init() {}

}

That said, our team took the path of manually converting the Java filter functionality to an HTTP handler. In order to walk you through this, I should compare the two interfaces. A J2EE filter has three methods: init, destroy, and doFilter. The init and destroy methods are lifecycle methods that have some significance, but are outside of the scope of this discussion. The doFilter method, however, does the lion’s share of the work in a filter.

In our scenario, the doFilter method gets the user agent header variable from the incoming request and inspects it against a list of configurable, known user agents. The most difficult thing in conversion is the difference in objects passed as arguments to the actionable method.

In the Java Filter.doFilter method, there are three arguments passed: a request object, a response object, and a filter chain object. Conversely, the IHttpHandler class’s ProcessRequest method has a single argument: an HttpContext variable. The HttpContext variable references an HttpRequest and an HttpResponse object, and most of the same functionality can be accomplished by accessing the HttpContext’s members.

The Java FilterChain object is interesting. This represents a chain of filters in an HTTP request. A filter may pass on the request to the next filter in the chain, allowing responsibilities to be delegated in a sequential fashion. The FilterChain is not used very often, however, but when it is, similar behaviors can be achieved with IHttpModule implementations.

The FilterChain aside, converting Java filters to .NET IHttpHandlers is straight-forward. First, an implementation of IHttpHandler needs to be created. Then the logic within the filter’s doFilter method needs to be reconstructed into the implementation’s ProcessRequest method, utilizing the HttpContext’s members instead of the passed request and response objects. Finally, the IsReusable method needs to be defined in the implementation; for simplicity, it can just return false (another case where later on you can write more code to determine whether the same handler instance can, in fact, be reused by later requests for performance gains).

Source Tree and Naming Conventions

While the Java and C# languages are similar, there are differences in lexicography, source tree layout, and naming conventions. Here, I’d like to detail some of these differences.

Java packages (which are the equivalent of namespaces in C#) follow a convention of a reversed domain name, followed by application or module name, followed by functionality. These packages are usually deeply nested (typically four or five levels deep). C# namespaces, on the contrary, are usually grouped by functionally descriptive names and are typically shallow (usually one to four levels). In addition, Java packages are usually lower or camel-cased, while C# namespaces are usually Pascal-cased. Likewise, Java methods are usually camel-cased, while C# methods and properties are usually Pascal-cased.

C# interfaces usually begin with a capital I, denoting an interface (this is purely a convention and is in no way required for correct functionality). Old Java convention said that interface names should end with "able", denoting the ability to do something. This convention is rarely followed any longer, and now there usually isn’t any distinction between an interface’s name and a class’s name.

C# uses properties to implement accessing and mutating private members. Properties are metadata wrappers for get and set accessor methods. But Java doesn’t have properties. Accessors and mutators for a private member are typically implemented as methods called getters or setters. In other words, an accessor for a private field called "name" would be getName.

Finally, Java classes must be located in a directory that matches its declared package, relative to the root of the source files (or classpath). C# does not impose this restriction.

When to Refactor for Conventions

Although a converted C# application will compile and run without addressing these differences, it is always best practice to follow conventions. Choosing when to refactor your converted code to conform to conventions is a difficult decision. However, there are two factors that suggest this should be done sooner rather than later.

Since refactoring code to conform to conventions is not absolutely critical for the application to function, not to mention the fact that it is tedious at times, it may be deemed as unnecessary later in the project. A variety of things can cause it to become a low-priority task and it will therefore run the risk of never getting done. However, if there is a team of developers working on the project, refactoring code to make it appear familiar can help to increase the team’s efficiency and productivity. Don’t underestimate the importance of these tasks. Refactoring of this sort is as important as the other tasks involved in the conversion process to ensure a successful result.

Directory Layout and Namespaces

Naming and directory conventions are a subjective process, but there are general guidelines. For our project, assume that there is a class for custom XML parsing, and it is located in the com.mycompany.myapplication.xml.util namespace (and directory, after conversion). C# convention suggests that this class should be located in the Xml.Util namespace. Before refactoring, our directory tree would look similar to Figure 5. Dragging and dropping files in Visual Studio will allow you to accomplish the physical movement of the file, so that your directory tree now looks like Figure 6.

Figure 5 Java Source Tree

Figure 6 C# Source Tree

However, C# does not dictate that the directory location of a file match its declared namespace. Therefore, the namespace of the class is not updated to match the file system location. There is no automatic way of moving multiple classes to different namespaces in Visual Studio—the best way that I’ve found to do this is to perform a Find and Replace for the whole solution, as shown in Figure 7. This, of course, assumes that you’re moving all of the classes in one namespace to the same destination.

Figure 7 Find and Replace Namespace Declarations (Click the image for a larger view)

Properties

The construction of properties in C# is significantly different from Java. Properties can be thought of as public fields that store the state of a class or, in UML terms, attributes of class. But Java does not have a property construct—instead, properties are represented in methods called getters and setters.

Consider a class that has an instance variable called "name". You don’t want to make this variable public because all control would be lost to modification of this variable. In Java, the standard method of providing access to this variable is through getters and setters, named so because the convention is to prepend a "get" or "set" to the variable name as the method name. Thus, if the name variable is a string, the Java getter and setter might look like this:

public String getName() {

return this.name;

}

protected void setName(String name) {

this.name = name;

}

C# provides property constructs to accomplish the same thing. Although they can be used for application logic, their intent is to provide protected access to private implementation details, as described above. Therefore, a C# implementation of the same access would look like this:

public String Name

{

get {return this.name;}

protected set {this.name = value;}

}

While C# properties accomplish the same objective, I find them to be clearer—in Java, methods that don’t modify a class’s state may begin with a get or set, leading to ambiguity. Therefore, I recommend that you refactor Java getters and setters to be C# properties.

That said, there isn’t an easy way to convert Java getters and setters to C# properties. A regular expression search and replace would be extremely complex, and the JLCA simply migrates the Java getter and setter methods as-is since there is no way to tell if the method is modifying a class’s state or performing some other functionality. Our approach to this problem was to use a more practical solution. Visual Studio provides a wizard for encapsulating private members with properties. This generates the desired properties without deleting the converted Java getters and setters.

As our team worked on code for other reasons, they also generated the C# properties in Visual Studio and deleted the corresponding getter and setter methods. After this step, developers would use the Visual Studio 2005 Find References functionality which provides a list of all call sites for a particular method. This gave them the references to the old getter and setter methods, so they could easily change the references to the new properties. This wasn’t an elegant solution, but it worked surprisingly well.

I should stress that this was considered a very important step in the porting process since properties are ingrained core aspect of C# and .NET and our goal was to create a C# application. (Remember, this application would eventually need to be supported by another team, which was expecting a C# application.)

Pascal-Casing Method Names

As I’ve already noted, Java method names are conventionally camel-cased. In other words, these names start with a lower-case letter, and each subsequent word boundary is upper-cased. C# convention says that methods and other members are Pascal-cased. Pascal-casing is similar to camel-casing, except that the first letter in the name is also upper case.

What does this mean to you? You should probably rename all of your methods to start with an upper-case letter instead of a lower-case letter. For example, the Java method

public void getResponseCode()

should be refactored as the C# method:

public String GetResponseCode()

As with converting getters and setters into properties, there is no easy way to go through all of the converted code and update members to conform to C# naming convention. However, we considered this a very important task since the members that were converted did not resemble C# members and, again, our team and our customer wanted this to be a proper C# application.

In theory, a code converter could be written to perform this task, and we considered doing this ourselves. However, we decided to take the same approach as we did with properties, updating member names piecemeal. We had several reasons for taking the manual approach.

For starters, our developers were already analyzing all of the code to update other elements, such as the getters and setters. The process could be incremental, since not changing the names of members did not affect the ability to compile or function. And, if for some reason one or two members were missed, the application wouldn’t break. The process didn’t take as much time as you might think—the main problem was that it was a tedious task.

Visual Studio 2005 has built-in refactoring support. Part of the refactoring support is safely renaming members, whereby a member can be renamed, and all references to that member are updated. It takes a little bit of time to go through all members you want to update, but it is an effective solution.

Conclusion

I have described to you a real case of converting a Java Web application to ASP.NET. Much of the heavy lifting was done by the JLCA, but there were several items that needed manual intervention on our part. Nevertheless, the JLCA is a powerful tool and allowed our application to be quickly ported to a production-quality .NET application.

My intent here is to demonstrate the viability of converting a Java application to .NET and to point out some of the issues that typically need to addressed beyond the JLCA’s capabilities. Every port of an application will present unique problems and I couldn’t address all of them in this article. However, I have offered some techniques and approaches that should help with any problems you might encounter.

Brian Jimerson is a senior technical architect for Avantia, Inc. (www.avantia-inc.com), a custom solutions provider based in Cleveland, Ohio. Brian has recently been helping clients define and implement infrastructure and architecture solutions for both Java and the .NET Framework.

Collaborate

Help Teams Work Together With Web Services And Groove 2007

John C. Hancock

This article discusses:

- An introduction to Groove 2007

- Using Groove forms for gathering data

- Groove Web services API

This article uses the following technologies:

Microsoft Office, .NET Framework

Code download available at: Groove2007_05.exe (524 KB)

Browse the Code Online

Contents

Building Solutions with Groove

Groove 2007 Architecture

Using Groove Forms to Collect Data

Accessing Groove Data from an Application

Using the Groove Web Services API

Registry Settings for Groove Web Services

Accounts and Identities

Getting a List of Workspaces and Tools

Fetching Data from Groove Forms

Loading Forms Data into a Database

Conclusion

In the real world, users often need to collect information while they are out of the office and unable to access their company networks. For example, a consumer research organization might send employees out to the field to ask people to fill in a paper survey, and then manually enter this data into a database. If you wanted to write a custom application that could be used in the field to automate this process, you would need to handle issues like managing a local data store and synchronizing it with a server database.

In this article, I show how the unique collaboration capabilities in Microsoft® Office Groove™ 2007 let you solve these problems as I integrate them into a custom Microsoft .NET Framework application. Groove provides the rich client for collaborative data gathering and secure synchronization of data between team members, and my custom .NET Framework application will be used to connect the data to a corporate database.

Groove is an environment for team collaboration that allows members to share data across organizational boundaries, regardless of the network they are connected to. In other words, participants don’t need to be part of the same network, making Groove a great collaboration tool when you need to include employees, partners, and customers without requiring the cooperation of IT departments. Groove is also excellent for mobile scenarios where users have to be able to go offline and still access and update the shared data, and also need to automatically synchronize data when they are back in connected mode.

Groove 2007 is part of the Ultimate and Enterprise editions of the 2007 Microsoft Office system, and can be purchased as a separate product. You can also install a free trial of Groove and download the Groove SDK from microsoft.com/downloads.

Building Solutions with Groove

You can build custom collaboration solutions using the out-of-the-box Groove features like file sharing and discussions just by creating workspaces with the relevant tools and then saving the workspace as a reusable template. If you need more flexibility, you can also build custom applications that take advantage of the secure collaboration facilities in Groove through the Web services-based API.

For the consumer research application solution I discuss here, the researchers who will use the application use Tablet PCs to fill in surveys while they are out in shopping malls. Ideally, the solution should work while the users are offline and then allow them to easily consolidate the information from the different researchers. In practice, it would be great if a researcher could simply walk into a café with a public Internet connection and have the solution automatically and securely synchronize the information with all participants.

The first step in designing the solution is for the research coordinator to create a new workspace in Groove. Every workspace can include multiple tools for particular requirements, such as a Files tool that allows participants to share documents with each other, and a Discussion tool so that people can ask and respond to questions from other participants.

Once the workspace is created, the research coordinator sends an invitation to all of the researchers to join the workspace. This invitation shows up as a notification to each participant, allowing them to download the workspace to their local machines. Everyone is now ready to collaborate—any change that a participant makes to his local copy of the workspace, such as adding a file or discussion topic, will be automatically and securely propagated to all the other participants when they are connected to a network.

Groove 2007 Architecture

Groove has an innovative architecture that is the source of its unique capabilities. First, Groove stores all the data on participants’ PCs rather than on a central server. This means that there is no need to devise a secure way for all participants to access a shared resource; whether the researchers are employees of the company or independent users, the research coordinator can easily create a shared workspace and invite them to participate.

Second, all of the data is always encrypted—both the local workspaces on disk and all network communications. One of the benefits of this is that researchers can synchronize their changes simply by connecting to through a public Internet connection. There is no need for all participants to have virtual private network (VPN) access to corporate resources, because the automatic synchronization is already encrypted.

Although most of Groove is client-based, there are also a few server components. Groove uses a relay server that acts as a store-and-forward service so that clients can communicate across firewalls using standard protocols; the relay server also stores temporary encrypted copies of workspace updates for offline users. In addition, there is a management server so that administrators can manage licenses and policy settings for all the users.

Standalone Groove users can create their own accounts when they install the client, which will automatically use relay servers hosted by Microsoft. Large companies with many Groove users might choose to deploy Office Groove Server 2007 as part of their own infrastructure so that they have more control over license management and can do things like integrate with Active Directory®. Small to mid-size enterprises can purchase a subscription to Office Groove Enterprise Services, which gives them access to a hosted version of the management tools for centralized control without requiring them to deploy their own infrastructure.

Using Groove Forms to Collect Data

For my solution, the workspace must include a survey form that the researchers can fill out for every consumer that they interview. Groove includes a Forms tool for designing custom forms that allows me to specify fields including data types, captions and other properties. Then the participants can add data to the workspace as shown in Figure 1.

Figure 1 Groove Workspace with Forms Tool (Click the image for a larger view)

Forms can also include JavaScript or VBScript to extend their capabilities for building custom solutions. Also, there is some flexibility when it comes to UI styles and formatting because the forms use DHTML and CSS. If you are already using InfoPath® forms in your organization, you can use the Groove InfoPath Forms tool to import XML-based InfoPath forms definitions instead of using built-in forms in Groove.

When the workspace with the custom form has been downloaded to the researchers’ tablet PCs, they can disconnect and still use the form to add records while they are in the field. When they connect back to the network, all of the records they have added will be securely synchronized with the other participants. The research coordinator will get a popup notification from Groove so that she knows that there is new unread data in the workspace.

Collecting the data from the researchers is only part of the story though. Once all the data from the individual surveys is synchronized, the research coordinator needs to load all the data into the company’s database so that it can be analyzed further. The company’s analysts will be using a SQL Server™ database, an Analysis Services cube, and Excel® business intelligence capabilities to understand the survey results and make some recommendations.

Accessing Groove Data from an Application

To get data out of the Groove workspace and into a corporate database, the research coordinator could manually export the data from the Groove forms tool in a format such as CSV (comma-separated values) and then import it into the database. However, this manual process would quickly become unmanageable if she needed to load the survey data into the database for analysis as it became available, rather than all at once.

A better solution would be to build a custom .NET Framework application that uses the Groove Web services-based API to read the data entered by the field researchers. Install this application on the research coordinator’s PC and she can access both the Groove workspace and the SQL Server database because she is connected to the corporate network as shown in Figure 2, unlike the researchers in the field who may only be connected to the public Internet.

Figure 2 Groove Workspace Users and SQL Server Database (Click the image for a larger view)

This approach is similar to the way Groove is integrated with Windows® SharePoint® Services (WSS) 3.0. Teams can use a Groove workspace for collaborating on documents, and then publish the completed documents to a SharePoint document library so everyone in the company can take advantage of SharePoint’s capabilities such as search and workflow. The way this works is that someone with access to both Groove and the SharePoint site adds a SharePoint Files tool to the workspace and connects it to a SharePoint document library. All workspace participants now have copies of all the documents and can submit changes, which the connected user can then synchronize with the SharePoint document library when necessary.

Using the Groove Web Services API

Even though I am developing a client application, the programming model for Groove uses Web services rather than a managed API. It feels a little strange programming this way because the client application is calling Web services that are actually running on the same machine. These services will still be available even when the machine is disconnected from the network—the Groove client itself is answering the requests. For larger enterprise solutions, a server-based product called Groove Server Data Bridge with a similar programming model is available, but I am going with the client-side approach for this solution.

To create the client application shown in Figure 3, I start by adding Web references to my project for each of the Groove services that I want to access. The Groove SDK includes a WSDL file for each service, so I add the Web reference by specifying the path to the relevant .wsdl file, renaming the Web reference from the default "Web Reference" to something more useful and clicking the Add Reference button. In this application, I will be using the GrooveAccounts, GrooveForms2, GrooveSpaces, and GrooveTools services.

Figure 3 Load Groove Data into a SQL Server Database (Click the image for a larger view)

Registry Settings for Groove Web Services

I will need to know the port that Groove is using before I can call the Web service—this defaults to 9080 but I should retrieve it from the GrooveLocalHTTPPort registry setting for the current user. I will also need to supply a value called a request key for every call, which the Web services use to authenticate the caller. I can retrieve this key from the LocalRequestKey registry setting, as shown in Figure 4. The request key is randomly regenerated every time Groove is restarted, so I should retrieve it from the registry before every call in case it has been regenerated since the last time I checked.

Figure 4 Registry Settings for Groove Web Services

using (RegistryKey grooveSettingsRegKey = Registry.CurrentUser.OpenSubKey(

“Software\\Microsoft\\Office\\12.0\\Groove”))

{

// Get the URL for web services including the port

string grooveHost = string.Format(“http://localhost:{0}”,

grooveSettingsRegKey.GetValue(“GrooveLocalHTTPPort”, 9080));

// Get the LocalRequestKey for authentication

string localRequestKey = (string)grooveSettingsRegKey.OpenSubKey(

“WebServices”).GetValue(“LocalRequestKey”);

...

}

The other registry setting that is used for Web service authentication is the response key. Every time I make a call to a Groove Web service, the response includes a key that I can check against the LocalResponseKey registry setting to make sure they match.

Accounts and Identities

Every Groove user has an account that is configured the first time he starts up Groove. The account is stored as a file on their computer (or on multiple computers), and includes information such as cryptographic keys and contact lists. Every account also has at least one identity that contains the contact information that is shared with other Groove users, such as an e-mail address and phone numbers. An account can also contain multiple identities if required so that, for example, a person can use one identity for business purposes and another for personal use such as sharing files with friends.

I can use the GrooveAccounts service to get the list of accounts and identities on the computer. After setting up the Web service call using the request key and port as shown in Figure 5, I can use the Read2 method of the GrooveAccounts service to return an array of Account2 objects. Each account has a collection of identities, so I can load up a list of identities to allow the user to select which one has the consumer research workspace as shown in Figure 3. (Most users only have one account with a single identity.)

Figure 5 List All Groove Accounts

// Set up the web service call for GrooveAccounts

GrooveAccounts.GrooveAccounts svc = new GrooveAccounts.GrooveAccounts();

svc.GrooveRequestHeaderValue = new GrooveAccounts.GrooveRequestHeader();

svc.GrooveRequestHeaderValue.GrooveRequestKey = localRequestKey;

svc.Url = grooveHost + “/GWS/Groove/2.0/Accounts/”;

// Get a list of all the accounts

GrooveAccounts.Account2[] accounts = svc.Read2();

// Display all identities

foreach (GrooveAccounts.Account2 account in accounts)

foreach (GrooveAccounts.Identity2 identity in account.Identities)

Console.WriteLine(account.Name + “, “ + identity.Name);

Getting a List of Workspaces and Tools

Once I know which identity I am working with, I have enough information to get a list of the user’s workspaces using the Read method of the GrooveSpaces service. The identity object has a URI property that is unique for this identity and can be used for most other Groove Web service calls as shown in Figure 6, and also a Spaces property that contains the URL for the user’s workspaces.

Figure 6 List All Groove Workspaces

private void LoadSpaces(GrooveAccounts.Identity2 identity)

{

// Set up the web service call for GrooveSpaces

GrooveSpaces.GrooveSpaces svc = new GrooveSpaces.GrooveSpaces();

svc.GrooveRequestHeaderValue =

new GrooveSpaces.GrooveRequestHeader();

svc.GrooveRequestHeaderValue.GrooveRequestKey =

GrooveSettings.RequestKey;

svc.GrooveRequestHeaderValue.GrooveIdentityURL = identity.URI;

svc.Url = GrooveSettings.GrooveHost + identity.Spaces;

// Get the list of spaces

_spaces = svc.Read(“”);

// Load the list into a combobox

comboWorkspaces.DisplayMember = “Name”;

comboWorkspaces.DataSource = _spaces;

}

As you saw when I created the Groove workspace earlier, each workspace can contain multiple tools such as files, discussions and forms. When the user has selected the workspace in the client application, I can find the forms tool that contains the market research data using the Read method of the GrooveTools service. The workspace only has one forms tool so I can just look for a tool named "Forms", but if I wanted to get more sophisticated, I could display a list of all the tools of the right type and allow the user to pick the right one.

Fetching Data from Groove Forms

Now that I have written code to retrieve the Groove settings from the registry, and to get a list of accounts, identities, workspaces and tools, I am ready to tackle the original requirement of getting the data out of the Groove tool and into my SQL Server database for analysis. Fetching the data is actually pretty easy; I can just set up a query that describes what records I would like to fetch using a RecordQuery object and then pass it to the QueryRecords method of the GrooveForms2 service, as shown here:

// Query ALL records in the ALL view:

GrooveForms2.RecordQuery query = new GrooveForms2.RecordQuery();

query.ViewURI = “”;

query.UnreadRecordsOnly = false;

query.QueryMetadataOnly = false;

query.IncludeFileAttachmentContent = false;

// Issue the query

GrooveForms2.Forms2RecordDataSet recordDataSet =

svc.QueryRecords(query);

The fun starts when you take a look at the Forms2RecordDataSet object that is returned by QueryRecords. Forms2RecordDataSet contains both the schema and the data for the form in XML format, so I am going to have to do some further manipulation before I can use it effectively. The easiest way of working with forms data is to convert the Forms2RecordDataSet object into a more familiar .NET DataSet object.

The Groove SDK documentation has a whole section devoted to this topic, and the code is also included in the sample download for this article, so I will just give you a high-level overview here (especially since this is the type of code that you write once and then reuse all over the place). In principle, you create a new DataSet and use the ReadXmlSchema method to set up the tables and column definitions by creating a string reader for the schema and prepending the appropriate XML version declaration to the string. When the schema has been created, you can use the ReadXml method to load the actual data.

One minor issue that you have to take into account relates to date-time columns. Groove returns all date-time fields in the Forms2RecordDataSet as universal time, while the .NET DataSet has them as local time by default. I can avoid problems by looping through all the columns that are created in the DataSet from the ReadXmlSchema method, and setting their DateTimeMode to Utc before I load the data.

Loading Forms Data into a Database

I can now loop through the DataSet and add a new row to the SQL Server database for each new record from the Groove form. The user could run the process any number of times, so I need to have some mechanism to avoid loading duplicate records into the database.

Every record in a Groove form has a system-generated field called _RecordID that uniquely identifies that record. If I include this field when I load records into the database, I can use it to check whether a record has already been added to the database and if so, update the existing database row rather than inserting a duplicate row. This will ensure that any survey records that were modified since the last time the process was run are correctly reflected in the SQL Server database.

Conclusion

Groove is a great environment for building solutions for distributed teams that would be difficult and costly to develop from scratch. Whether you stick to the out-of-the-box features like file sharing and forms or build entirely new applications that build upon the API, Groove is a great addition to your arsenal.

John C. Hancock is a senior consultant for Microsoft, specializing in business intelligence and .NET Framework development. He is the coauthor of Practical Business Intelligence with SQL Server 2005 (Addison-Wesley Professional, 2006), and can be contacted at www.johnchancock.net.

Identity

Secure Your ASP.NET Apps And WCF Services With Windows CardSpace

Michèle Leroux Bustamante

This article discusses:

- Windows CardSpace and information cards

- Web browsers and identity selectors

- Security tokens

- Integration with WSFederationHttpBinding and ASP.NET membership

This article uses the following technologies:

.NET Framework 3.0

Code download available at: CardSpace2007_04.exe (462 KB)

Browse the Code Online

Contents

Windows CardSpace and Information Cards

Personal Cards

Managed Cards

Browsers and Identity Selectors

Object and XHTML Parameters

Processing the Security Token

Associating Cards with an Account

Integrating Windows CardSpace with WCF

Conclusion

In May 2005, Microsoft introduced its vision for an identity metasystem that would reduce the complexities and risks associated with managing and exchanging digital identities. This vision-discussed in the article "Microsoft's Vision for an Identity Metasystem" explains interactions based on interoperability standards such as WS-Security, WS-Trust, WS-MetadataExchange, WS-SecurityPolicy, and other WS-* protocols. These protocols enable applications and services to communicate their security requirements via security policy and to transfer digital identities safely and securely. The identity metasystem vision describes a simple and consistent experience for users to identify themselves to applications and services while also identifying the target site to the user in order to reduce risks associated with sending private information to unauthorized or malicious sites.

Windows CardSpace™ (formerly known as "InfoCard"), released with Windows Vista™ and the Microsoft® .NET Framework 3.0, plays an important role in this identity metasystem. Windows CardSpace replaces traditional username and password authentication with a tool that helps users better manage their digital identities and helps shield users from various forms of identity attack such as phishing.

In this article, I will start by reviewing the roles of each party in the identity metasystem, introducing Windows CardSpace, and discussing both personal and managed information cards. Then I'll dig into some scenarios involving both ASP.NET Web applications and Windows Communication Foundation services-explaining how to integrate Windows CardSpace as an authentication mechanism for each scenario.

This article assumes you have some experience with ASP.NET and the Windows Communication Foundation, as well as familiarity with Windows CardSpace features. For a detailed overview of Windows CardSpace, I recommend reading the article "Introducing Windows CardSpace" by David Chappell. You can get additional background from Keith Brown's two columns on the subject titled "A First Look at InfoCard" and "A Deeper Look at InfoCard".

Windows CardSpace and Information Cards

In the identity metasystem, there are three key participants: the relying party (RP), the subject, and the identity provider (IP). The RP can be anything (though typically it is an application or a service) that accepts tokens to execute a given task, such as signing-in and presenting membership details. For the purpose of this article, I will demonstrate how a subject can utilize the Windows CardSpace identity selector to choose and present the appropriate credentials to be used for authentication.

The subject is a person who wishes to safely present information (such as identification information) to the RP. This information is communicated in the form of a security token which is issued by an IP such as a bank, employer, government, and so forth.

The IP is responsible for issuing the security token on behalf of the subject, vouching for the subject's information described by the claims held in the token. The IP is normally implemented as a Security Token Service (STS), a Web service endpoint that implements WS-Trust protocol to issue, renew, validate, or cancel security tokens describing a subject.

Figure 1 Identity Metasystem Participants (Click the image for a larger view)

Figure 1 illustrates the relationship between the three parties. The RP relies on a particular type of security token to authorize calls. The IP authenticates the subject and issues to the subject a security token in the format required by the RP. The subject then relays that token, which contains claims on behalf of the subject, to the RP. The separation of these roles, and the interoperability protocols used to communicate between them, are part of the identity metasystem credo. Each participant can be implemented on a different technology or platform, so long as they comply with the appropriate Web and WS-* protocols to facilitate interoperable communications.

In the context of this identity metasystem, Windows CardSpace has two roles. First, it is a client technology used for creating, managing, and selecting digital identities in a secure and consistent manner. (In the latter capacity, Windows CardSpace serves as an identity selector.) Second, it has a local IP for personal digital identities and can generate security tokens for those identities.

A digital identity is represented as an information card. Windows CardSpace allows users to create personal cards or to import managed cards issued by other IPs. Users can later select a card when an RP requires that users authenticate themselves with one. The user can only select cards that satisfy the requirements of the RP-or, alternatively, select a personal card and add the necessary claims on the fly-and the associated IP issues the security token containing the appropriate subject claims.

Cards are issued by an IP to describe the claims that the IP is able to vouch for, on behalf of an authenticated subject. Each card represents important information about the IP that issued it, including the following:

- The URL (or URLs) of the IP's STS, which issues security tokens. This way, when the user selects a card to authenticate to an RP, Windows CardSpace knows where to request the security token containing the appropriate claims.

- The types of claim associated with the card. Since the IP issues the card, it gets to define what claims the card represents. The RP specifies the claims that it requires and only cards representing these claims will be enabled for a user to select.

- The types of security token that the IP can issue. The RP indicates which tokens it supports, so only cards that support these tokens will be enabled for selection.

A card issued by an IP can be serialized as a signed XML document with a .crd file extension. The .crd file contains an <InformationCard> section with this and other pertinent information for Windows CardSpace. Figure 2 shows an example of a signed, managed card imported into Windows CardSpace.

Figure 2 Serialized Version of a Managed Information Card

<dsig:Signature xmlns:dsig="http://www.w3.org/2000/09/xmldsig#">

<dsig:SignedInfo>...</dsig:SignedInfo>

<dsig:SignatureValue>...</dsig:SignatureValue>

<dsig:KeyInfo>...</dsig:KeyInfo>

<dsig:Object Id="_Object_InfoCard">

<ic:InformationCard xmlns:ds="http://www.w3.org/2000/09/xmldsig#"

xmlns:ic="http://schemas.xmlsoap.org/ws/2005/05/identity"

xmlns:mex="http://schemas.xmlsoap.org/ws/2004/09/mex"

xmlns:wsa="http://www.w3.org/2005/08/addressing"

xmlns:wsid="http://schemas.xmlsoap.org/ws/2006/02/addressingidentity"

xmlns:wst="http://schemas.xmlsoap.org/ws/2005/02/trust" xml:lang="en-us">

<ic:InformationCardReference>

<ic:CardId>http://www.thatindigogirl.com/card/

ED18504F-40B9-5A58-C200-761DC886DF29</ic:CardId>

<ic:CardVersion>1</ic:CardVersion>

</ic:InformationCardReference>

<ic:CardName>thatindigogirl</ic:CardName>

<ic:CardImage MimeType="image/png">[base64 encoded image]

</ic:CardImage>

<ic:Issuer>http://www.thatindigogirl.com/tokenissuer.svc</ic:Issuer>

<ic:IssuerName>That Indigo Girl</ic:IssuerName>

<ic:TimeIssued>2006-12-29T22:32:17Z</ic:TimeIssued>

<ic:TimeExpires>9999-12-31T23:59:59.9999999Z</ic:TimeExpires>

<ic:TokenServiceList>

<ic:TokenService>

<wsa:EndpointReference>

<wsa:Address>http://www.thatindigogirl.com/sts/tokenissuer.svc

</wsa:Address>

<wsa:Metadata><mex:Metadata><mex:MetadataSection>

<mex:MetadataReference><wsa:Address>

http://www.thatindigogirl.com/sts/mex</wsa:Address>

</mex:MetadataReference>

</mex:MetadataSection></mex:Metadata></wsa:Metadata>

<wsid:Identity>

<ds:KeyInfo><ds:X509Data>

<ds:X509Certificate>[base64 encoded

certificate of the IP]</ds:X509Certificate>

</ds:X509Data></ds:KeyInfo>

</wsid:Identity>

</wsa:EndpointReference>

<ic:UserCredential>

<ic:UsernamePasswordCredential>

<ic:DisplayCredentialHint>Please enter your

password</ic:DisplayCredentialHint>

<ic:Username>thatindigogirl</ic:Username>

</ic:UsernamePasswordCredential>

</ic:UserCredential>

</ic:TokenService>

</ic:TokenServiceList>

<ic:SupportedTokenTypeList>

<wst:TokenType>urn:oasis:names:tc:SAML:1.0:assertion

</wst:TokenType>

</ic:SupportedTokenTypeList>

<ic:SupportedClaimTypeList>

<ic:SupportedClaimType Uri=

"http://schemas.xmlsoap.org/ws/2005/05/identity/

claims/personalprivateidentifier">

<ic:DisplayTag>PPID</ic:DisplayTag>

<ic:Description>Unique identifier for the card</ic:Description>

</ic:SupportedClaimType>

...other claim types

</ic:SupportedClaimTypeList>

<ic:PrivacyNotice>http://www.thatindigogirl.com/PrivacyPolicy.txt

</ic:PrivacyNotice>

</ic:InformationCard>

</dsig:Object>

</dsig:Signature>

There are two types of card: personal and managed. Personal cards (sometimes referred to as self-issued cards) are created by the user within Windows CardSpace and are issued by the local Windows CardSpace IP. Managed cards can be issued by any other IP and must be explicitly imported into Windows CardSpace.

The interaction between the subject, the RP, cards and associated IPs, and Windows CardSpace is shown in Figure 3. When a Web site or service (the RP) requires a token issued by Windows CardSpace, it specifies the claims it needs. The client application (a browser or a Web service client) invokes Windows CardSpace (1), passing this information along. Windows CardSpace uses this information to present the appropriate cards that can satisfy those claims to the user (2). When the user selects a card (3), Windows CardSpace authenticates the user using a method defined by the IP, calling the appropriate IP as referenced by the card (4) to have a security token issued containing the claims requested by the RP (5). However, it is up to the IP to determine which claims are included in the token (this could be a superset or subset of those in the RP's policy). Ultimately, a signed and encrypted security token is passed back to the application or Web browser (6), which forwards it to the RP (7).

Figure 3 How Cards Are Selected and Used to Issue Security Tokens (Click the image for a larger view)

Cards are stored and secured on the local machine in a file that is encrypted twice, protected by a machine key, your Windows login, and optionally a pin number you supply to secure access to the card. If you use multiple computers, you can export your cards to an encrypted data file and then import them on another machine that you trust.

Personal Cards

Personal cards are created through the Windows CardSpace Control Panel. When you create a personal card, you are in effect creating two things: a set of claims representing some personal information, and a card that lists those claims. Essentially the local Windows CardSpace IP issues personal cards representing the claims you entered and safely stores your claims waiting for a request for a security token. Figure 4 illustrates how a personal card indicates to the Windows CardSpace IP which claims to put into a requested security token.

Figure 4 Generate a Security Token from a Personal Card (Click the image for a larger view)

When you create a personal card, you can only enter information for some or all of a predefined set of 15 claims, such as first and last name, e-mail address, and mailing address information. Claims are represented by a URI from the WS-Identity namespace, schemas.xmlsoap.org/ws/2005/05/identity. Figure 5 lists the URI for the claims that a personal card can store (the URIs are abbreviated for clarity).

Figure 5 Claims Supported by Personal Cards

http://schemas.xmlsoap.org/.../identity/claims/privatepersonalidentifier

http://schemas.xmlsoap.org/.../identity/claims/name

http://schemas.xmlsoap.org/.../identity/claims/givenname

http://schemas.xmlsoap.org/.../identity/claims/surname

http://schemas.xmlsoap.org/.../identity/claims/emailaddress

http://schemas.xmlsoap.org/.../identity/claims/streetaddress

http://schemas.xmlsoap.org/.../identity/claims/locality

http://schemas.xmlsoap.org/.../identity/claims/stateorprovince

http://schemas.xmlsoap.org/.../identity/claims/postalcode

http://schemas.xmlsoap.org/.../identity/claims/country

http://schemas.xmlsoap.org/.../identity/claims/homephone

http://schemas.xmlsoap.org/.../identity/claims/otherphone

http://schemas.xmlsoap.org/.../identity/claims/mobilephone

http://schemas.xmlsoap.org/.../identity/claims/dateofbirth

http://schemas.xmlsoap.org/.../identity/claims/gender

http://schemas.xmlsoap.org/.../identity/claims/webpage

Creating personal cards is not much different from creating a username and password for a site. Web sites and Web services can associate personal cards with your user account so you can use Windows CardSpace to select your identity with a few mouse clicks, instead of typing in your username and password. A unique identifier, based on the card's personal private identifier (PPID), is used to associate a card with an account. This identifier is unique to each card and RP combination. That is, if the same card is used for multiple RPs, each RP will receive a distinct PPID to protect the privacy of the user's usage patterns.

Managed Cards

Unlike personal cards, managed cards can represent any claims the IP wishes to assert about a subject (the actual card data is stored within the IP's systems, not on the local machine). You can't create managed cards in the Windows CardSpace interface because the local Windows CardSpace IP can only issue cards for a fixed set of claims. Any IP can issue managed cards representing the claims for which they can generate security tokens. In fact, since a relationship must exist between the RP and the IP, it may be that the RP and IP are owned by the same enterprise.

For example, if you have an account with a bank that knows your account number and type, the IP might issue a card that indicates a security token can be issued with those claims. To avoid the risks associated with username and password login to the account, the managed card can be installed into Windows CardSpace so that the user can select it when authenticating to banking sites. This implies that the bank associated the card with your account when it issued the card, or that an additional step to associate the card once installed is required.

When the user selects this card to authenticate to the banking RP, only compatible managed cards that can satisfy these claims can be selected. Managed cards do not contain the actual claim values, but Windows CardSpace knows where to find the IP to issue the encrypted token containing these claims. Figure 6 illustrates how the managed card indicates to the managed IP what to put into the requested security token.

Figure 6 Generate a Security Token from a Managed Card (Click the image for a larger view)

Each RP and IP that have a relationship should also agree on the claims to be included in security tokens. These claims can be identified by unique URIs. Consider these Create/Read/Update/Delete (CRUD) claims for example:

http://www.thatindigogirl.com/2006/06/claims/create

http://www.thatindigogirl.com/2006/06/claims/read

http://www.thatindigogirl.com/2006/06/claims/update

http://www.thatindigogirl.com/2006/06/claims/delete

An application or service could potentially use these claims to do claims-based authorization against features and resources.

Though managed cards are secured by the local machine and can be exported to other machines, the actual claims are stored at the IP and never leave the IP without being wrapped in an encrypted security token. The IP encrypts the security token with the public key of the RP so that nothing-not even Windows CardSpace-can view the claims inside the token. (If the user wants to preview the claims that are about to be sent to the RP, an extra and optional display token can be requested. This is encrypted in such a way that Windows CardSpace can decrypt it and display it to the user.)

Browsers and Identity Selectors

Other identity selectors are being built to perform the same function that Windows CardSpace does in Windows. Windows CardSpace or any other identity selector can be used to select personal or managed cards to satisfy an authentication request.

For Web applications to support personal or managed card authentication, they must first provide a Web page with an object tag or an XHTML binary behavior describing their information card requirements. Browsers that support these tags and have an information card extension will be able to launch the appropriate identity selector on the client machine for users to select a card. As Figure 7 illustrates, there are no specific ties to Windows CardSpace as the identity selector, to Internet Explorer® as the browser, or to ASP.NET as the Web site technology. In fact, other browsers have their own extensions for handling these and other object tags and behaviors, and each operating system can have its own identity selector that supports information cards. All of these elements are platform-neutral.

Figure 7 Trigger an Identity Selector from an Object Tag or Binary Behavior (Click the image for a larger view)

Any Web application platform, including ASP.NET, can expose pages that include the required object tag or XHTML binary behavior to trigger identity selection. The object tag uses the MIME type application/x-informationCard as shown here:

<object id="informationCard" name="informationCard"

type="application/x-informationCard">

...

</object>

The information card binary behavior can be specified in XHTML by using a #default#informationCard behavior in the <informationCard> element.

<ic:informationCard name="xmlToken"

style="behavior:url(#default#informationCard)" ... >

...

</ic:informationCard>

Within these <object> and <informationCard> elements are parameters that describe a Web site's requirements for information card authentication. I'll get to those details shortly.

Although earlier versions of Internet Explorer support object tags and XHTML, only Internet Explorer 7.0 supports information cards using a MIME handler extension called Microsoft Information Card IE Helper, available from icardie.dll. (An extension adding Windows CardSpace support to Mozilla Firefox is available.) The following registry key links the MIME type for information cards to the handler:

HKEY_CLASSES_ROOT\MIME\Database\Content Type\application/x-informationCard

This MIME handler is then linked to object tag and behavior configurations for the browser. This handler is responsible for calling the identity selector, passing parameters specified by the object tag or XHTML binary behavior. For Internet Explorer 7.0, the identity selector defaults to Windows CardSpace, indicated by the following registry entry:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Internet Explorer\InformationCard Token Provider

For the object tag syntax, the MIME handler is triggered through the InformationCardSigninHelper Class, enabled by default in the Internet Explorer 7.0 add-ons dialog as shown in Figure 8. (You can find this dialog by opening Internet Options and from the Programs tab selecting "Manage add-ons.") If this add-on is disabled, the object tag will not trigger identity selection, but this setting has no effect on XHTML syntax. The XHTML information card behavior is configured in a string entry labeled INFORMATIONCARD under the registry key:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Internet Explorer\Default Behaviors

Figure 8 Information Card MIME Handler in Internet Explorer 7.0 (Click the image for a larger view)

This behavior also triggers the MIME handler to activate the identity selector.

Unfortunately, consistently detecting browser support for information cards is not an exact science and differs across browsers. While not recommended, for Internet Explorer you can perform a preliminary check of the UserAgent looking for Internet Explorer 7.0 and .NET common language runtime (CLR) 3.0 support. Here's an example of a UserAgent string that indicates this support:

Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; SLCC1;

.NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.04506)

In ASP.NET, you can check possible support at the server using the UserAgent property of the Request object as follows:

string userAgent = Request.UserAgent;

if (userAgent.Contains("MSIE 7.0") &&

userAgent.Contains(".NET CLR 3.0"))

{ Response.Write(

"Information cards are supported, but might not be enabled."); }

else

{ Response.Write("Information cards may not be supported."); }

This approach has many shortcomings. Other browsers support Windows CardSpace besides Internet Explorer, and even in browsers that support Windows CardSpace, the MIME handler could be disabled. A better solution is to use client-side script to detect whether the browser supports Windows CardSpace and whether that support is enabled. For an example of this, see www.fearthecowboy.com/2006/12/detecting-cardspace-support-including.html.

Internet Explorer 7.0 and the appropriate MIME handler configurations are central to triggering Windows CardSpace from object tags and XHTML. In addition to this, the Web site must be SSL-enabled to secure communications. Extended validation (EV) certificates issued by a trusted root certification authority such as VeriSign or Thawte won't require any special configuration, but if you are using test certificates, you'll need to install the test SSL certificate as a trusted root and add the site to the trusted sites list in Internet Explorer 7.0. Instructions for this are included with the code samples for this article.

Object and XHTML Parameters

As I mentioned, you can use object tags or XHTML binary behaviors to trigger an identity selector from supporting browsers. With either syntax, you must provide a base set of properties to the information card browser extension, listed in Figure 9.

Figure 9  Information Card Activation Properties

Property

Description

requiredClaims (mandatory)

You can indicate any of the personal claims listed in Figure 5 or custom claims from a managed IP. If any of the claims cannot be satisfied by a personal card, a managed card must be installed to satisfy the claims.

issuer (optional)

Indicates the IP from which to request a security token. This defaults to the URI for personal tokens: schemas.xmlsoap.org/ws/2005/05/identity/issuer/self. Leave this setting as the default to trigger Windows CardSpace.

issuerPolicy (optional)

Indicates the URL where the security policy for the IP can be found. This does not need to be set for the local Windows CardSpace IP.

tokenType (optional)

If omitted, this defaults to the URI for SAML 1.0 tokens, urn:oasis:names:tc:SAML:1.0:assertion. For personal cards, you can specify either SAML 1.0 or SAML 1.1 token format: http://docs.oasis-open.org/wss/oasis-wss-saml-token-profile-1.1#SAMLV1.1. Anything else implies a managed card.

optionalClaims (optional)

You can indicate any personal or custom claims as optional. This causes Windows CardSpace to give the user the option of sending the information to the site.

privacyVersion (optional)

Specifies the privacy policy version. If this changes, the user is notified and given a chance to review any changes to the privacy policy.

Using the object tag, you configure these properties as in Figure 10. Each <param> represents one of the information card properties by name. The values for requiredClaims and optionalClaims are space-delimited lists of claim URIs. You could optionally omit the issuer and tokenType parameters if the defaults are acceptable to your application.

Figure 10 Object Tag Syntax

<object id="informationCards" name="informationCards"

type="application/x-informationCard" >

<param name="issuer"

value="http://schemas.xmlsoap.org/ws/2005/05/identity/issuer/self"/>

<param name="tokenType" value="urn:oasis:names:tc:SAML:1.0:assertion"/>

<param name="requiredClaims" value="http://schemas.xmlsoap.org/ws/

2005/05/identity/claims/privatepersonalidentifier

http://schemas.xmlsoap.org/ws/2005/05/identity/claims/emailaddress"/>

<param name="optionalClaims" value="http://schemas.xmlsoap.org/ws/

2005/05/identity/claims/dateofbirth"/>

</object>

Using XHTML you configure information card properties as shown in Figure 11. In this case, attributes are used to specify the behavior style, issuer, and tokenType. Each claim is specified with an <add> instruction indicating the claim URI and whether it is optional. At least one claim must be specified.

Figure 11 Binary Behavior Syntax

<ic:informationCard name="informationCards"

style="behavior:url(#default#informationCard)"

issuer="http://schemas.xmlsoap.org/ws/2005/05/identity/issuer/self"

tokenType="http://docs.oasis-open.org/wss/

oasis-wss-saml-token-profile-1.1#SAMLV1.1">

<ic:add claimType="http://schemas.xmlsoap.org/ws/2005/05/

identity/claims/privatepersonalidentifier" optional="false" />

<ic:add claimType="http://schemas.xmlsoap.org/ws/2005/05/

identity/claims/emailaddress" optional="false" />

<ic:add claimType="http://schemas.xmlsoap.org/ws/2005/05/

identity/claims/dateofbirth" optional="true" />

</ic:informationCard>

In any case, when a Submit button is selected on a page that includes an information card trigger, the browser invokes the MIME handler and passes the parameters to the identity selector. If a card is selected, the signed and encrypted token is posted to the site in a form parameter named for the <object> or <informationCard> identifier. In Figures 10 and 11, the identifier is informationCards.

If, for some reason, information cards are disabled by the browser, or if the site is not SSL-enabled, the form parameter collection will not include an informationCards item. If Windows CardSpace is launched and the user cancels for any reason, including if the parameters are invalid and cause an error loading Windows CardSpace, the informationCards form parameter is posted as an empty string. In other words, there is nothing to do unless the informationCards form parameter includes an <encryptedData> element.

Using either syntax for <object> or <informationCard>, you can trigger Windows CardSpace from any ASP.NET page of your choice, but there are a few additional steps required to ensure that Windows CardSpace is only invoked for specific Submit buttons on the page. ASP.NET pages have a single active <form> object, so if you slap the <object> or <informationCard> tag inside the only <form> object, any button that posts back to the server will trigger Windows CardSpace, like it or not.

To avoid this, you can include the information card object in the HTML header section and explicitly invoke it using script when the appropriate button is clicked. Figure 12 shows a simple ASP.NET page that includes the <object> tag in the header, along with a SelectInformationCard function that activates the object and puts the return value into a hidden input field. Inside the <form> tag is a standard ASP.NET Login control and an ImageButton.

Figure 12 Explicitly Invoke the Object Tag

<%@ Page Language="C#" AutoEventWireup="true"

CodeFile="Login.aspx.cs" Inherits="Login" %>

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"

"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

<html  >

<head runat="server">

<title>Login</title>

<!--Declaration of information card object -->

<object declare id="informationCards" name="informationCards"

type="application/x-informationCard" > ... </object>

<script type="text/javascript" language="javascript">

function SelectInformationCard()

{

var infoCardObject =document.getElementById("informationCards");

var hiddenToken = document.getElementById("hiddenXmlToken");

hiddenToken.value = infoCardObject.value;

}

</script>

</head>

<body>

<form id="standardLogin" name="standardLogin" method="post"

runat="server">

<asp:Login ID="Login1" runat="server" />

<asp:ImageButton ID="cardSpaceSubmit" runat="server"

ImageUrl="cardspacelogin.jpg"

OnClientClick="SelectInformationCard()"

OnClick="CardSpaceLogin" />

<input id="hiddenXmlToken" type="hidden" name="hiddenXmlToken"

value="empty"/>

</form>

</body>

</html>

When the Login control submits, Windows CardSpace is not invoked since the object tag is in the header section. When the ImageButton submits, Windows CardSpace is invoked because the OnClientClick attribute calls SelectInformationCard. You could also create a custom ASP.NET control that emits the object tag and associated script. The important thing is to distinguish each Submit button so that Windows CardSpace is only activated at the appropriate time.

Processing the Security Token

As I mentioned earlier, if Windows CardSpace is invoked, a signed and encrypted token is posted. In the case of Figure 12, the hiddenXmlToken input field will contain this <encryptedData> section if a token is successfully issued. If Windows CardSpace was activated but the user canceled, the field will contain an empty string. If Windows CardSpace could not be activated, it will contain its initial value "empty". The Windows CardSpaceLogin method is invoked if the Windows CardSpace login option is selected on this page; thus you can check the value of the hidden field and proceed from there.

The first problem you'll encounter when an encrypted token is posted back to your Web site is that ASP.NET disallows posting XML and HTML data in order to reduce vulnerability to scripting attacks. You'll receive an HttpRequestValidationException like this one:

System.Web.HttpRequestValidationException: A potentially dangerous

Request.Form value was detected from the client

(informationCards="<enc:EncryptedData T...").

You can disable this feature for the entire site by adding the following to the <system.web> section of your web.config:

<pages validateRequest="false"/>

This is one of those cases where I would recommend you set this only for those pages that will receive a postback with a security token. Fortunately, you can set this per page as follows:

<%@ Page Language="C#" AutoEventWireup="true"

CodeFile="Login.aspx.cs" Inherits="Login"

ValidateRequest="false" %>

The next problem you'll run into is that token-processing code is not built into ASP.NET. You must manually process the <encryptedData> element, decrypt the token, validate the token signature, and pull out the claims before you can perform any authentication or authorization. Fortunately, Microsoft provided an example TokenProcessor class with the Windows CardSpace samples that ship with the Windows SDK for Windows Vista and the .NET Framework 3.0 (referenced at msdn2.microsoft.com/aa967562.aspx). I have included a slightly modified version of this TokenProcessor in the code sample for this article.

To use the TokenProcessor class with your ASP.NET applications, all you have to do is construct the type with the <encryptedData> element and access the claims that were passed in the security token. For example, this code looks for the PPID, emailaddress, and dateofbirth claims required by the object tag discussed earlier:

using Microsoft.IdentityModel.TokenProcessor;

string token = Request.Form["hiddenXmlToken"] as string;

if (!String.IsNullOrEmpty(token)) {

Token tokenProcessor = new Token(token);

string ppid = tokenProcessor.Claims[ClaimTypes.PPID];

string emailaddress = tokenProcessor.Claims[ClaimTypes.Email];

string dateofbirth = tokenProcessor.Claims[ClaimTypes.DateOfBirth];

}

From a very high level, the TokenProcessor works as follows. The token is encrypted using the Web site SSL certificate. The TokenProcessor uses the thumbprint of this certificate to find the associated private key, defaulting to the My/LocalMachine store. The Web application host identity (typically ASPNET or NETWORK SERVICE) must be granted access to this private key or the decryption will fail. The decrypted result is then converted into a SamlToken and validated to ensure that the contents of the token have not been tampered with since the token was created. Once validated, a reference to the token and its claims are available to your code as shown earlier.

For personal cards, the key used to sign the SAML token is not known to your application since it is generated on the fly when the card is used for the first time at your site. Thus, you can't check the signature for personal cards against a trusted public key. You can, however, rely on a hash of the PPID if you previously associated the card with a user account.

For managed cards, since a relationship exists between the RP and IP, you should add an additional step to the RP's token processing code to validate that the token has been signed by an IP you accept. You can do this by using the public key of the IP to validate the signature.

Associating Cards with an Account

Now that I've explained how to trigger Windows CardSpace from ASP.NET, and how to process the resulting security token, you might be wondering what you should do with the claims inside the token. This is influenced by the type of card you require at the Web site: personal or managed.

Sites that support personal cards largely do so to provide users with an improved login experience. The types of claims supported, as I've discussed, are limited-although the associated demographic information could be useful to the site. That said, these claims won't remove the need for role-based authorization. One way you can get the best of both worlds is to allow users to associate a personal card with their existing account and use the card to look up the user's roles when they log in with it.

Figure 13 illustrates the interaction between the user, the Web site, and Windows CardSpace to achieve this goal. The user can create the personal card ahead of time and then log in to an existing account to associate the card. It is also possible to create the card during the association step if a card that satisfies the site's requirements doesn't already exist. Once a personal card is selected to associate with the site, the resulting security token is posted to the site for processing. The claims contained in the security token must be sufficient to look up the user account when a user logs in with the card.

Figure 13 Associate a Personal Card with a Web Site Account (Click the image for a larger view)

The following code associates a card with the current user account only if the e-mail address claim supplied with the token matches the e-mail of the logged-in user:

MembershipUser user = Membership.GetUser(this.User.Identity.Name);

if (user.Email == emailaddressClaim) {

user.Comment = ppidClaim;

Membership.UpdateUser(user);

}

This code uses the Comment field of the user record to associate the card's PPID with the user account. The PPID claim is unique to the card and the RP it is sent to, thus uniquely identifying the user. (Note, though, that the PPID can only be relied on if the token's signature is first verified and validated cryptographically.)

When the user logs in using the personal card, the e-mail address claim can be used to look up the user account using the ASP.NET Membership API. A matching PPID in the comment field indicates the correct user has been found, as shown here:

MembershipUserCollection matchingUsers =

Membership.FindUsersByEmail(emailaddressClaim);

if (matchingUsers.Count > 0) {

foreach (MembershipUser user in matchingUsers) {

if (user.Comment == ppid) {

authenticatedUser = user;

break;

}

}

}

Having found the user account, the usual steps to create an authentication ticket for the user can be completed. A simple call to RedirectFromLoginPage can facilitate this:

FormsAuthentication.RedirectFromLoginPage(

authenticatedUser.UserName, true);

Interestingly, this also authenticates the user in the traditional way for the duration of the ASP.NET session, which means that traditional role-based security checks will also work across the application.

Managed cards introduce a slightly different workflow for associating cards to accounts. The distribution of managed cards to end users can be handled in a number of ways, one of which could be to allow users to request a card after logging in to the site. The card is then imported to Windows CardSpace for later use from the site's login page. Figure 14 illustrates this flow of communication.

Since managed cards can contain literally any set of claims deemed valuable to the IP and RP, they may even have more meaning to the authorization process. A set of claims received by an RP might indicate a list of rights to the application. In this case, the ASP.NET membership and role provider may be bypassed altogether since it relies on role-based security and has no notion of claims.

Figure 14 Request and Import a Managed Card to Log In (Click the image for a larger view)

Integrating Windows CardSpace with WCF

Browser-based applications are natural candidates for integrating with Windows CardSpace authentication. Web services can also require Windows CardSpace authentication such that client applications invoke Windows CardSpace to issue security tokens in a similar way.

In the case of Windows Communication Foundation, this is done by configuring service endpoints to use WSFederationHttpBinding. This generates a security policy for the service, included in the Web Service Description Language (WSDL) document that indicates it requires personal tokens. When SvcUtil generates a Windows Communication Foundation proxy from this WSDL, matching client endpoints are generated that use an equivalent WSFederationHttpBinding configuration. Client applications use this proxy to construct a channel to call the service, and the proxy handles calls to Windows CardSpace to gather a security token satisfying the claims of the target service. The resulting interaction looks much like that in Figure 3, where the RP is the Windows Communication Foundation service instead of a Web application, and the client application is the Windows Communication Foundation client and proxy instead of a browser.

WSFederationHttpBinding allows you to expose Windows Communication Foundation services over HTTP that rely on an issued token from an IP. The IP can be Windows CardSpace, which implies a personal or managed card must be supplied to find the appropriate IP to issue the actual token. The IP can also be explicitly configured, which means that the token request does not go through Windows CardSpace. For the purpose of this discussion I'll focus only on the Windows CardSpace configuration.

The parameters required of WSFederationHttpBinding will look very similar to the requirements of the object tag or XHTML configuration used to trigger Windows CardSpace from a Web page. You must specify an issuer, a token type, and a list of required or optional claims. The issuer must be set to schemas.xmlsoap.org/ws/2005/05/identity/issuer/self to trigger Windows CardSpace; the token format can be any format supported by a personal or managed card. The claims can likewise be personal or managed claims. These settings will determine whether a personal or managed card can satisfy the request for a security token.

Figure 15 shows a complete <system.serviceModel> section for a simple Windows Communication Foundation service that exposes a WSFederationHttpBinding endpoint. The <wsFederationHttpBinding> section describes the properties I just discussed for the binding configuration. In this case, SAML 1.1 tokens are required to contain a PPID and e-mail address claim. The service must also provide a certificate to facilitate secure exchanges with clients, and this certificate is specified in the <serviceBehavior> section within the <serviceCertificate> element.

Figure 15 Service Config to Trigger Windows CardSpace

<system.serviceModel>

<services>

<service name="HelloIndigo.HelloIndigoService"

behaviorConfiguration="serviceBehavior">

<endpoint address="http://localhost:8000/HelloIndigo"

contract="HelloIndigo.IHelloIndigoService"

binding="wsFederationHttpBinding"

bindingConfiguration="wsFed" />

</service>

</services>

<bindings>

<wsFederationHttpBinding>

<binding name="wsFed">

<security mode="Message">

<message issuedTokenType="http://docs.oasis-open.org/wss/oasis-

wss-saml-token-profile-1.1#SAMLV1.1" negotiateServiceCredential="false" >

<claimTypeRequirements>

<add claimType="http://schemas.xmlsoap.org/

ws/2005/05/identity/claims/privatepersonalidentifier"

isOptional="false"/>

<add claimType="http://schemas.xmlsoap.org/

ws/2005/05/identity/claims/emailaddress" isOptional="false"/>

</claimTypeRequirements>

<issuer address=

"http://schemas.xmlsoap.org/ws/2005/05/identity/issuer/self" />

</message>

</security>

</binding>

</wsFederationHttpBinding>

</bindings>

<behaviors>

<serviceBehaviors>

<behavior name="serviceBehavior" >

<serviceCredentials>

<issuedTokenAuthentication

allowUntrustedRsaIssuers="true" />

<serviceCertificate findValue="RPKey"

storeLocation="LocalMachine" storeName="My"

x509FindType="FindBySubjectName" />

</serviceCredentials>

</behavior>

</serviceBehaviors>

</behaviors>

</system.serviceModel>

The allowUntrustedRsaIssuers setting in the <issuedTokenAuthentication> element is important when relying on personal cards. By default this setting is false. However, it must be set to true for personal cards since the resulting token is signed with an unknown private key. For managed cards, this isn't required since the service will likely install the public key of any IP you accept into the Trusted People certificate store. In the case of managed cards, the signature of the issued token is checked against this list of trusted public keys. Custom authorization options also exist.

The generated client configuration used to initialize the proxy is equivalent to the service configuration with a few exceptions. The client requires access to the service public key to encrypt the token and its communications to the service. Fortunately, SvcUtil generates a base64-encoded version of the public key with the client endpoint configuration under <identity> to avoid the step of installing the public key into the client machine certificate store. This configuration is shown in Figure 16. The <wsFederationHttpBinding> section is collapsed in this illustration because it is equivalent to Figure 15. The client proxy is initialized with this information when constructed and uses it to determine how to acquire the security token to authenticate to the service. In fact, the actual code to invoke the service can be as simple as this:

HelloIndigoContractClient proxy = new HelloIndigoContractClient();

string s = proxy.HelloIndigo("Hello");

Figure 16 Client Configuration for WSFederationHttpBinding

<system.serviceModel>

<bindings>

<wsFederationHttpBinding>...</wsFederationHttpBinding>

</bindings>

<client>

<endpoint address="http://localhost:8000/HelloIndigo"

binding="wsFederationHttpBinding"

bindingConfiguration="wsFed"

contract="Client.localhost.HelloIndigoContract"

name="wsFed">

<identity>

<certificate encodedValue="base64 encoded public key" />

</identity>

</endpoint>

</client>

</system.serviceModel>

Windows CardSpace is triggered by Windows Communication Foundation at the client because the issuer setting tells it to use the personal token provider. The user is presented with the identity selection interface with only cards satisfying the required claims in the binding presented. Managed cards are required when the token type or required claims cannot be satisfied by personal cards.

One area where Windows Communication Foundation significantly differs from ASP.NET in regards to Windows CardSpace authentication is in how claims are processed at the server. Unlike with ASP.NET, the security token issued through Windows CardSpace is packaged as part of the security headers in the SOAP message sent to the service. Before the service operation is invoked, this security token is decrypted and validated, and its claims extracted into the service security context. These extracted claims are added to an abstraction known as a ClaimSet, which contains the list of claims and information about the issuer of the claims. In the case of personal cards, the claims are self-issued, which means that you won't be authorizing the public key of the issuer. That's why it is important to associate the card's PPID claim (which is unique and kept secret) with an account in the system. You can check the claim sets attached to the AuthorizationContext and look for a PPID claim to find a match, and then check the values of other expected claims for further authorization, as shown in Figure 17.

Figure 17 Access Claims through the AuthorizationContext

AuthorizationContext authContext =

ServiceSecurityContext.Current.AuthorizationContext;

DateTime? birthDate = null;

string ppid = null;

foreach (ClaimSet cs in authContext.ClaimSets)

{

IEnumerable<Claim> claims = cs.FindClaims(ClaimTypes.PPID,

Rights.PossessProperty);

ppid = ValidatePPID(claims);

if (ppid!=null)

{

claims = cs.FindClaims(ClaimTypes.DateOfBirth,

Rights.PossessProperty);

foreach (Claim c in claims)

birthDate = Convert.ToDateTime(c.Resource);

}

}

if (birthDate == null)

throw new FaultException("Missing date of birth claim.");

if (birthDate.Value.AddYears(13) > DateTime.Now)

throw new FaultException(

"User is too young to access this operation.");

The code in Figure 17 checks for PPID and date-of-birth claims from any ClaimSet-assuming that the token was issued by a personal card-then checks for a date-of-birth claim to verify the age of the caller. One or more sets of claims may be attached to the AuthorizationContext as different security tokens are extracted from the message and processed by service authorization policies. If the issuer is known and can be identified, it would be possible to select the correct ClaimSet from a particular issuer before processing claims.

Managed cards yield an extended set of possibilities since the claims have an identifiable issuer. The issuer is also described by a ClaimSet that often includes an identity claim with the public key of the IP you accept. You can search for an identity claim and check that it is a valid RSA claim as follows:

ClaimSet csIssuer = cs.Issuer;

IEnumerable<Claim> issuerClaims = csIssuer.FindClaims(

ClaimTypes.Rsa, Rights.Identity);

foreach (Claim c in issuerClaims)

{

System.Security.Cryptography.RSACryptoServiceProvider rsa =

c.Resource as RSACryptoServiceProvider;

if (rsa != null)

{

... // claim exists; check the public key to identify the IP

}

}

Once you're sure you have the right ClaimSet, you can perform claims-based security checks. For example, if the claims are based on the CRUD claims mentioned earlier, you can check for create, read, update, and delete claims for appropriate operations. In this way, managed cards not only make it easier for a user to select his identity, but they also facilitate the generation of a rich set of claims identifying the user's rights. Service developers can focus on claims-based authorization checks instead of traditional role-based security checks. A complete sample that illustrates claims-based security can be downloaded from msdn.microsoft.com/msdnmag/code07.aspx.

Conclusion

Integrating Windows CardSpace with your ASP.NET applications and Windows Communication Foundation services improves the end-user experience by providing a simple and consistent identity selection process for authentication to trusted applications and services. While personal and managed cards provide an equivalent login experience, managed cards have the advantage of enabling an IP to supply a specific set of claims that applications and services can use to better identify the user's rights in their system. The topics discussed in this article and implemented in the sample code for this article will help you on your way to triggering the Windows CardSpace experience from ASP.NET or Windows Communication Foundation. Enjoy!

Michèle Leroux Bustamante is a Chief Architect for IDesign Inc., a Microsoft Regional Director and MVP for Connected Systems, and an internationally known speaker and author. Michele contributed to the design and implementation of Windows CardSpace during its early beta stages. She recently completed the book Learning WCF (O'Reilly, 2007; book blog: www.thatindigogirl.com). Reach her at www.idesign.net or visit her main blog at www.dasblonde.net.

Aero Glass

Create Special Effects With The Desktop Window Manager

Ron Fosner

This article discusses:

- Understanding the DWM

- Programming the DWM

- Drawing on glass and incorporating live thumbnails

This article uses the following technologies:

Windows Vista, .NET Framework

Code download available at: DWM2007_04.exe (166 KB)

Browse the Code Online

Contents

Technical Overview of the DWM

The Benefits of a Compositing Desktop

Some Common Compositing Functions

Getting Ready for Aero Glass

Programming the DWM

Drawing on Glass

Thumbnails

Wrap-Up

Ialways look forward to new releases of Windows. It's fun to poke around MSDN® and the SDK documentation looking for the latest innovations to play with and build upon, to amaze friends and coworkers and-if you're lucky-your employer. Windows Vista™ contains a lot of goodies in this regard. I've been particularly looking forward to getting my hands on this release ever since I heard that it integrates a 3D/compositing layer into the desktop. I've written a ton of 3D applications over the years, and one thing I always found annoying was being able to provide cool user-interface eye-candy in a 3D application but not in a non-3D app. With Windows Vista and the Desktop Window Manager (DWM), this is starting to change (see Figure 1).

Figure 1 DWM Enables Features Like Flip 3D Task Switching (Click the image for a larger view)

The DWM is the new interface that manages how the various windows that are running and rendering are merged onto the Windows Vista desktop. Windows® Presentation Foundation (WPF) provides a higher-level layer that controls the rendering to the desktop layer, and the Windows Display Driver Model (WDDM) handles the actual low-level rendering to the display. I'm only going to talk about using the DWM interfaces. Read the MSDN article "Introducing Windows Presentation Foundation" by David Chappell for more on that subject (msdn2.microsoft.com/aa663364.aspx). WDDM is only of interest in the context of this article because of the new effects it makes possible through the DWM interfaces, and the problems it was designed to fix.

Technical Overview of the DWM

The DWM interface, available on all versions of Windows Vista except Windows Vista Home Basic, is housed in dwm.exe. All applications in the system benefit from the DWM without any modifications or recompilations.However, applications that choose to take advantage of specific DWM features can call to interfaces in dwmapi.dll (the public interface of DWM), which will then pass them along to dwm.exe. The interface declarations are found in dwmapi.h and you can get up-to-date API information online from windowssdk.msdn.microsoft.com.

Windows Vista was designed to use graphics accelerators for every window, not just for 3D DirectX® applications. In order to do this, DWM talks to WDDM, which is the ultimate owner of the graphics processor and video memory. (DWM relies on milcore.dll, a component shared with Windows Presentation Foundation, for output and rendering to DirectX.) Rendering is handled by a separate thread in DWM, inaccessible to the user, that owns the DirectX device. In addition, most applications have their own threads that handle rendering and UI (such as the USER message pump thread in a typical Win32® application), but which don't conflict with DWM's rendering thread. The DWM gets a list of windows and manages their bitmaps in a tree structure, which it then composites onto the final desktop.In other words, each application renders its own bitmaps that are then composited by the DWM.

An application's main window thread renders its scene, the scene is accessed by the DWM render thread, and the render thread updates the desktop through its DirectX interface. The information passed is compacted to just changes (deltas) from the previous render, and large data, like images, are placed in shared memory. This can potentially allow the scene generation to occur on one machine while the final rendering can be done on another machine. 3D programmers who have used OpenGL will be familiar with this architecture, which allows a server to manage a 3D scene and just send deltas to the client machines. You can have a distributed rendering of a 3D scene and have it running with full hardware acceleration on any number of client machines.This architecture allows the DWM to provide first-class support for Remote Desktop scenarios.

While Windows Vista will run with a legacy Windows XP-compatible driver, a WDDM video driver is required to get all the DWM features. Contrary to some speculation, DWM doesn't require DirectX 10, but does require some more video/texture memory and a video card that supports Shader Model 2.0 or better. The biggest change with using WDDM is that it introduces the Video Memory Manager (VidMM), which can swap video memory allocations between system memory and video memory. This means that WDDM can virtualize the video card's resources so that it's possible to do a better job of sharing and swapping video memory, and of context-switching the graphics processor among different threads in disparate applications. It used to be nearly impossible to run multiple 3D applications with any kind of stability, due to drivers that were unable to handle context switches. And prior to WDDM, there was no formal scheduling available, so often one DirectX application would starve others.With WDDM, it's much harder for that to happen. Drivers are also on a much tighter leash for Windows Vista and are forced to be much more robust than for previous versions of Windows.

As an aside, note that DirectX 10 is a Windows Vista-only API. Applications designed for previous version of DirectX will run on a legacy DirectX API implementation expected to be called DirectX 9 L.This will be the last version supported by pre-WDDM drivers. DirectX 9 L apps are expected to run on Windows XP with DirectX 9 L installed as well as on Windows Vista. DirectX 10 contains no legacy interfaces.

The Benefits of a Compositing Desktop

All these new subsystems get you the ability to render windows independently and perform a compositing step on them prior to rendering them onto the desktop. Some cool uses of this are visible in Windows Vista and some of the updated applications that ship with it. The two I'm going to show you how to use are the Aero™ glass effect and thumbnails. The glass effect is available only when running the Aero Glass scheme and compositing is turned on. It isn't available for Aero Basic.

Since each window is created in its own section of video memory, the DWM gets to do the final compositing of that window onto the desktop. This means the DWM has access to the image that's on the desktop and can blend it with your window's rendering, creating a rendering that's a composition of both. This is most noticeable in any areas of a window that are blended with the existing desktop image to create a frosted glass effect. Since each window is rendered to an intermediate off-screen surface, this means the DWM is the only program involved in updating the glass effect. When you move a window with a glass effect, no underlying window needs to be invalidated. The DWM handles updating the visible translucent image to the new coordinates. It's possible to tell the DWM to add some of the client area of a window to be rendered with glass-thus letting you create glass areas for your own use.

This off-screen compositing makes the desktop more responsive. Since each window is now rendered separately from the desktop, the problem you often see in slow-to-update applications like Web browsers is eliminated. In previous version of Windows, it wasn't uncommon to see a display like Figure 2. You could move a window around on top of another application and get this tearing effect because the window underneath took too long to update itself. With desktop composition, you'll no longer see this artifact.

Figure 2 Slow Rendering Causes Tearing (Click the image for a larger view)

Some of the applications that ship with Windows Vista take advantage of the ability to render glass into the client area. One of the nicest examples is Windows Media® Player, which extends the frame into the bottom of the client area where it draws some custom controls. The minimal version looks like Figure 3.

Figure 3 Glass Effects in Windows Media Player (Click the image for a larger view)

When I move the window around my desktop, the image I can see through the controls is made up of the part of the desktop that's under the player. If I move the window over some animation, I can see the animation through the window that's on top of it! This is the power of a composited desktop.

If you turn the composition effect off, you'll just get the "aurora" effect on the windows and the window will render opaquely with the default window color as in Figure 4.

Figure 4 Windows Media Player with Glass Effects Turned Off (Click the image for a larger view)

In essence, the Aero Basic and Aero Glass interfaces are the demarcation between the pre-DWM UI standards and the new. The Aero Basic interface presents the same API to programs to maintain backwards compatibility, but running the Basic interface means you are using the legacy window manager, and DWM is not active. The legacy interface means that the UI layout will behave as expected by programs written prior to Windows Vista. The DWM controls the Aero Glass interface and restricts access to it. If an application draws in the non-client area (the glass frame), the DWM detects this and will switch over to the Aero Basic frame.

Other new features in Windows Vista-available because of a compositing desktop-are the new Windows Flip (Alt+Tab) and Flip 3D (Alt+Windows Key) task switchers. The Flip 3D is of particular interest because its functionality depends upon some code in the DWM that takes each top-level window in the scene graph and renders it onto a series of skewed windows you can scroll through with your keyboard or mouse (see Figure 1).

The DWM controls how windows interact with the desktop composition engine. Being able to integrate your program into the DWM functionality requires that you gain an understanding of how the DWM works and how you can interact with it. While it does much more, the DWM has four primary functional areas in its public API:

- Basic desktop composition settings

- Rendering the glass effect in a client window

- Rendering thumbnails

- Tuning rendering for interaction with multimedia programs

I'll cover the first three sections in this article. The last one is provided for DirectX and video playback applications because the DWM is running asynchronously, which can lead to sampling artifacts if not tightly controlled.

Some Common Compositing Functions

If you want to use the desktop composition functionality in your program, you need to query and set various DWM parameters. For example, if some application switches to full-screen display and then DWM turns off composition and renders in the desktop background color with no transparency, your application should recognize this situation and disable composition-specific features. Here are the basic functions for integrating your program with the DWM:

DwmEnableComposition enables or disables DWM composition. The DWM will maintain this setting for either the duration of the current process or until it is reset. Changing the setting causes a WM_DWMCOMPOSITIONCHANGED notification. Most applications have no need to call this function, but you will likely want to monitor for the resulting Windows message.

DwmIsCompositionEnabled gets the DWM composition-enabled state for the desktop.

DwmSetWindowAttribute sets the value of the specified DWM attribute for a window, controlling how DWM transitions are handled, whether non-client rendering is allowed, and how Flip 3D will treat the window. For example, if non-client rendering is turned off for a window, then later calls to extend the frame or blur behind the window will fail.

DwmGetWindowAttribute retrieves the current value of the specified DWMWINDOWATTRIBUTE for the specified window.

DwmGetColorizationColor retrieves the current color that is being used for DWM glass composition. This value is based on the current color scheme. Changing the setting causes a WM_WMCOLORIZATIONCOLORCHANGED notification.

DwmDefWindowProc enables DWM hit-testing within the non-client area when called with the WM_NCHITTEST notification as well as in certain situations where you might need to process WM_NCCALCSIZE and the like because you have extended the client frame.

Rendering the glass effect in your program is pretty straightforward. The DWM provides two functions for this:

DwmExtendFrameIntoClientArea , a simple function, extends the non-client frame edge into your window.

DwmEnableBlurBehindWindow , a more complicated function, gives you a lot more control over how the glass effect is rendered.

Since all composited windows are rendered to an off-screen window by the DWM for later composition onto the desktop, it's a simple matter to take those images and provide a live thumbnail representation of the application. The DWM provides four functions to let you control how thumbnails get rendered:

DwmQueryThumbnailSourceSize returns the source size of the DWM thumbnail.

DwmRegisterThumbnail creates a thumbnail relationship between the destination and source windows.

DwmUnregisterThumbnail removes a DWM thumbnail relationship created by DwmRegisterThumbnail.

DwmUpdateThumbnailProperties updates the properties for a given thumbnail.

The DWM has five functions for fine-tuning how the DWM renders, but these functions are beyond the scope of this article.

Getting Ready for Aero Glass

To program the DWM interface, you need to be running a version of Windows Vista capable of displaying Aero Glass.While it's easiest to call these new functions from C++ code, I like to write user-interface code in C# if I can. All the code for this article is written in C#, but that does mean that you have to jump through a few hoops. To use the functions discussed in this article, you'll either need to use C++ and link in the correct library, or you'll have to write P/Invoke wrappers for the functions and structures in C#. In the download for this article, I've included a library that provides wrappers for the functions and structures required by the DWM so you can call it from your C# program. Basically, it's just a set of instructions to load the interface from dwmapi.dll. In order to use the DWM functions for the glass effect and thumbnails used in this article, you'll need to create C# declarations of the DWM functions and data structures.The ones I created for this article look like Figure 5.

Figure 5 C# Declarations for DWM

internal class DwmApi

{

[DllImport("dwmapi.dll", PreserveSig = false)]

public static extern void DwmEnableBlurBehindWindow(

IntPtr hWnd, DWM_BLURBEHIND pBlurBehind);

[DllImport("dwmapi.dll", PreserveSig = false)]

public static extern void DwmExtendFrameIntoClientArea(

IntPtr hWnd, MARGINS pMargins);

[DllImport("dwmapi.dll", PreserveSig = false)]

public static extern bool DwmIsCompositionEnabled();

[DllImport("dwmapi.dll", PreserveSig = false)]

public static extern void DwmEnableComposition(bool bEnable);

[DllImport("dwmapi.dll", PreserveSig = false)]

public static extern void DwmGetColorizationColor(

out int pcrColorization,

[MarshalAs(UnmanagedType.Bool)]out bool pfOpaqueBlend);

[DllImport("dwmapi.dll", PreserveSig = false)]

public static extern IntPtr DwmRegisterThumbnail(

IntPtr dest, IntPtr source);

[DllImport("dwmapi.dll", PreserveSig = false)]

public static extern void DwmUnregisterThumbnail(IntPtr hThumbnail);

[DllImport("dwmapi.dll", PreserveSig = false)]

public static extern void DwmUpdateThumbnailProperties(

IntPtr hThumbnail, DWM_THUMBNAIL_PROPERTIES props);

[DllImport("dwmapi.dll", PreserveSig = false)]

public static extern void DwmQueryThumbnailSourceSize(

IntPtr hThumbnail, out Size size);

[StructLayout(LayoutKind.Sequential)]

public class DWM_THUMBNAIL_PROPERTIES

{

public uint dwFlags;

public RECT rcDestination;

public RECT rcSource;

public byte opacity;

[MarshalAs(UnmanagedType.Bool)]

public bool fVisible;

[MarshalAs(UnmanagedType.Bool)]

public bool fSourceClientAreaOnly;

public const uint DWM_TNP_RECTDESTINATION = 0x00000001;

public const uint DWM_TNP_RECTSOURCE = 0x00000002;

public const uint DWM_TNP_OPACITY = 0x00000004;

public const uint DWM_TNP_VISIBLE = 0x00000008;

public const uint DWM_TNP_SOURCECLIENTAREAONLY = 0x00000010;

}

[StructLayout(LayoutKind.Sequential)]

public class MARGINS

{

public int cxLeftWidth, cxRightWidth,

cyTopHeight, cyBottomHeight;

public MARGINS(int left, int top, int right, int bottom)

{

cxLeftWidth = left; cyTopHeight = top;

cxRightWidth = right; cyBottomHeight = bottom;

}

}

[StructLayout(LayoutKind.Sequential)]

public class DWM_BLURBEHIND

{

public uint dwFlags;

[MarshalAs(UnmanagedType.Bool)]

public bool fEnable;

public IntPtr hRegionBlur;

[MarshalAs(UnmanagedType.Bool)]

public bool fTransitionOnMaximized;

public const uint DWM_BB_ENABLE = 0x00000001;

public const uint DWM_BB_BLURREGION = 0x00000002;

public const uint DWM_BB_TRANSITIONONMAXIMIZED = 0x00000004;

}

[StructLayout(LayoutKind.Sequential)]

public struct RECT

{

public int left, top, right, bottom;

public RECT(int left, int top, int right, int bottom)

{

this.left = left; this.top = top;

this.right = right; this.bottom = bottom;

}

}

}

You'll need to create something similar to this in your code if you're using C#. Then, assuming you're running Windows Vista, you'll be able to make DWM calls. Of course, your application shouldn't just assume it's running on Windows Vista.In order to be sure, you need to verify that Environment.OSVersion.Version.Major is at least 6.0. Alternatively, you can catch the exceptions that result from trying to call nonexistent functions through P/Invoke.

If you want to use the glass effects, the computer you're using will need to meet three requirements. First, you need to be running the Premium, Business, or Ultimate editions of Windows Vista. Second, you need hardware capable of running the Aero interface (see microsoft.com/windowsvista/getready/capable.mspx for details). Finally, you need to select the Windows Aero color scheme in Windows Vista.Be willing to use it sparingly, as the effect can be taxing on your computer's GPU and, if overdone, on your users.

Figure 6 Window Color and Appearance Options (Click the image for a larger view)

You enable the Aero scheme by opening the Personalization control panel and clicking the Window Color and Appearance option. On this screen (see Figure 6), make sure the Enable Transparency option is selected, then click the "Open classic appearance properties." link. In the Appearance Settings dialog box, under Color scheme, click Windows Aero (see Figure 7). When you click OK, the Aero interface and glass effect will be visible. If you like, you can also customize the window color and the opacity level.

Figure 7 Windows Aero Glass Effects (Click the image for a larger view)

Programming the DWM

You can check to see if the Aero scheme is enabled in your program by calling DwmIsCompositionEnabled. Be aware, though, that not only can the user change the current Aero scheme at any time, it's also possible for other applications to programmatically enable or disable it.Thus, checking the result of this function once may not be robust enough.

The DwmEnableComposition function allows a program to turn the Aero scheme on or off. For example, if you're writing an application that may encounter compatibility problems, you might want to disable composition while you app is running (if you're writing a full-screen DirectX exclusive application, composition will be automatically disabled). This setting is only maintained for the duration of the process that sets it; when the process ends, the composition flag will be reset to its original value. In general, unless for application compatibility reasons, applications shouldn't use this and should let the system or user make the decision instead.

When the status of desktop composition is changed, a WM_DWMCOMPOSITIONCHANGED message is broadcast. There are no parameters telling you if it's being enabled or disabled, so it's up to you to call DwmIsCompositionEnabled if you're interested. The code to do the check is straightforward-the tricky part is deciding how you want your window to look if composition is disabled.

// Check to see if composition is Enabled

if (DwmIsCompositionEnabled())

{

// enable glass rendering

}

else

{

// fallback rendering

}

Finally, even if the Aero scheme is enabled, the user might have changed the glass color and made the composition opaque. I wrote a small app that creates a window that's entirely glass and then I changed the glass properties in the control panel (see Figure 8). The first image shows the default setting for the window color and transparency. Then I turned transparency off, leaving an opaque window. I changed to the red window color and the default transparency where you can still make out some of the underlying window image. On a large enough opaque window, you'll observe the highlighting placed in the glass rendering-it's that streaky diagonal "aurora" effect visible on the window.

Figure 8a Changing Color and Transparency

Figure 8b

Figure 8c

You can check to see the composition color and opacity by calling the DwmGetColorizationColor function. If this function succeeds, it will set a GDI+ ARGB color value and a Boolean indicating whether the color is opaque. Just like changing the Aero scheme in the control panel, there's a message broadcast when the composition color has changed. WM_DWMCOLORIZATIONCOLORCHANGED is sent when this happens, but in this case the parameters tell you what the new color and opacity are.

protected override void WndProc(ref Message msg)

{

switch (msg.Msg)

{

case WM_DWMCOLORIZATIONCOLORCHANGED:

// The color format of currColor is 0xAARRGGBB.

uint currColor = (uint)msg.WParam.ToInt32();

bool opacityblend = (msg.LParam.ToInt32() != 0);

...

break;

}

}

The WM_DWMNCRENDERINGCHANGED message is sent when DWM rendering has changed for the non-client area. The wParam will be true if non-client rendering by the DWM is enabled. You'll also get notification when a DWM composited window is maximized or un-maximized when you get a WM_DWMWINDOWMAXIMIZEDCHANGE message. The wParam will be true if the window has been maximized.

Earlier we noted the two functions you can use to get the glass effect into your program. The first is DwmExtendFrameIntoClientArea. A window with the Aero scheme has glass in the title bar area and in a border around the edges of the window-essentially all of the non-client area of the window. This function allows you to extend each of the sides of the non-client area rendering into the client area, rendering it with the glass effect. In other words, you can extend the top, left, right and bottom edges of the glass window frame into your window seamlessly.

The second function is DwmEnableBlurBehindWindow, which lets you render an arbitrarily shaped region with the glass effect and specify more parameters with greater control over the effect, though I suspect that most users of the glass effect will simply extend the glass from the edge into the client area. With either function, you'll need to closely track the composition status to see if you should render with the glass effect enabled. This means tracking the four WM_DWM* messages or calling DwmIsCompositionEnabled to see if you should render with the glass effect turned on or off.

Let's look at the simpler call first. This function is intended for use on windows without a frame (such as the taskbar, Sidebar, tablet pen input window, and the Start menu); behavior on framed windows is undefined.

The DwmExtendFrameIntoClientArea function takes a window handle and a MARGINS structure. The window handle is the window for which the frame is to be extended from the edge into the client area. You'll need to set up a MARGINS structure that contains the number of pixels to extend the frame into the client area. A C# implementation of the MARGINS structure is shown in Figure 5.

It's a bit confusing at first since there are no other Win32 functions that work like this, but, basically, you control each side independently from the others. Pick the sides you want to extend and specify how far into the client area the effect should be rendered (see Figure 9). If you want more than one side to be extended, they can overlap. If you want the effect to dynamically track the window size, then you'll need to call the DwmExtendFrameIntoClientArea function every time the window size changes. A special case is to set one or more margins to -1, which will then extend the glass effect to your entire window. To reset the margins, just set all of the margin values to 0 and call DwmExtendFrameIntoClientArea again.

Figure 9 Glass Margins in Client Area (Click the image for a larger view)

So what do you do if you don't want to extend the glass effect from the frame into the client area? The DwmEnableBlurBehindWindow function gives you a bit more control over how the glass effect is added to your window. Again, it takes a window handle of the window you want to add the glass effect to, but it also takes a DWM_BLURRBEHIND structure that lets you set various parameters regarding how to use the blur effect on your window. The most important one of the parameters is a region, which is a GDI term that describes an arbitrarily shaped area that is usually constructed out of a series of lines and curves.

The DWM_BLURRBEHIND structure, shown in Figure 5, contains the parameters that control how the blur effect is presented.

If you want to turn on the glass effect in the client area, you set the fEnable flag to true. To turn it off you'd set the flag to false. The hRgnBlur parameter is a handle to a region you create that the glass effect appears on. Just like setting a margin value to -1 for Dwm-ExtendFrameIntoClientArea, setting the hRgnBlur parameter to null in the blur structure informs the DWM to apply the glass effect to the entire window.

The final parameter, fTransitionOnMaximized, is a bit misleading. Since the glass effect is turned off on windows that are maximized, you'd think this flag would have something to do with that. This flag actually controls whether or not the window transitions to the maximized color when there is a maximized window on the desktop. Unfortunately, if you set this parameter to true, when you render the window you'll get a region without the transparency, just the aurora effect.

The dwFlags parameter is how you tell the interface which parameters you are setting; when you want to set a parameter, you'll need to turn on the corresponding bit in the dwFlags parameter. This is consistent throughout the DWM interface.

Don't forget that you'll have to render the glass color into the region. Using the same black brush that was used in DwmExtendFrameIntoClientArea will work fine for creating a glass effect on the specified region.

It's easy to see that DWM has provided two interfaces for you-the more complicated one that lets you construct an arbitrarily shaped region, and the simple one that just lets you extend the window frames glass effect into your client area so you can draw additional controls and the like onto something that looks like it's part of the title bar. In both cases, you have to worry about what happens to the region's shape when the window is resized and update the area to render the glass effect in if it's not the entire client area. The source code that accompanies this article will allow you to set both an extended client frame or a region and to toggle the composition flag as well.

Drawing on Glass

Using glass as a background on your window is a bit tricky. If you render anything naturally opaque (such as GDI functions), you'll get your item rendered on glass, though sometimes with unexpected results. If you want to actually blend rendering into the glass surface, you'll need to take advantage of functionality that utilizes the alpha channel of colors, such as GDI+, Windows Presentation Foundation, or the Windows XP Theme API.

One particular gotcha is that rendering a GDI item in black uses the bit pattern 0x00000000-which also happens to be a completely transparent black if you are using an alpha channel. This means that if you draw with a black GDI brush or pen you'll get a transparent color, not a black one. The biggest problem this presents is when you try to use the default text color in a control of a text label that sits on the glass area. Since the default text color is usually black, the DWM will consider this to be transparent and the text will be written in the glass incorrectly. An example can be seen in Figure 10. The first line is written with GDI+, the second is a text label control using the default color. As you can see, it's nearly illegible because it's actually incorrectly rendered text that shows up as gray, not black.

Figure 10 Transparent Dialog Box

Happily, there are a number of ways around this problem. Using owner-draw controls is one. Rendering to a bitmap that has an alpha channel is another. Fortunately, the easiest way to get text on controls is to let the .NET Framework 2.0 use GDI+ for you. This is easily accomplished by setting the UseCompatibleTextRendering property on your controls. By default, this property is set to false so that controls written for previous versions of the .NET Framework will render the same. But if you set it to true, your text will come out looking correct. You can set the property globally with the Application.SetUseCompatibleTextRenderingDefault method. If you're using Visual Studio® 2005, the template code will include a call to set compatible text-rendering to false in the main routine before your form is created. You can just edit this to set it to true as shown below and all your controls will look correct when written on a glass window.

static void Main()

{

Application.EnableVisualStyles();

Application.SetCompatibleTextRenderingDefault(true);

Application.Run(new GlassForm());

}

You can find more information about this and using the TextRenderer class in the March 2006 MSDN Magazine article, "Build World-Ready Apps Using Complex Scripts In Windows Forms Controls", by Miguel A. Lacouture.

You should enable the glass effect just before you start rendering your window. The composition engine will look at the alpha values of your window and will apply the blur effect to those regions that are not opaque. This can be a problem when using some GDI functions because they don't preserve alpha values. You can use GDI+ when you have to, but you should be careful because GDI+ calls are rendered in software, not hardware, so a high window refresh frequency mixed with GDI+ calls can cause a significant hit to system resources.

Getting the glass effect in DirectX applications is done the same way. All you need to do is to control the alpha value of the render target in addition to using one of the two glass-enabling DWM functions. Wherever you've told the DWM to use glass, it'll use the alpha value of the render target. Anywhere else, the render target should be opaque or you'll get undefined behavior.

Thumbnails

Thumbnails are live display-only windows on open applications rendered by the DWM. Thumbnails are used by the Flip and Alt+Tab task switchers. You can essentially request a thumbnail of an application's window and have it rendered in your application. The thumbnail API will provide you with a live representation of an application's window.

Thumbnails are simple to use, as most of the hard work is done for you by Windows. The hard part is getting hold of an app's HWND. Once you've got the HWNDs you want, you simply register a thumbnail to associate that HWND with the HWND where you want the thumbnail rendered and the location in that window. The OS takes care of updating from then on. Whenever the source window is changed, the change is reflected in the target window.

To use a thumbnail, you must first register a thumbnail using the DwmRegisterThumbnail function. You provide two window handles, the source HWND (that is, the window you want the thumbnail view of) and the target HWND (the window where you want the thumbnail rendered). When you are finished using a thumbnail, you have to let the DWM know that the relationship is ending by calling DwmUnregisterThumbnail. After you have created a thumbnail, the DwmRegisterThumbnail function returns a thumbnail handle and all further thumbnail functions will take this handle as an argument. You'll have to call DwmUpdateThumbnailProperties after you register a thumbnail to get the thumbnail updating. Example code for a form that renders a live thumbnail of another window is shown in Figure 11.

Figure 11 Basic Thumbnail

public partial class Thumbnail : Form

{

private IntPtr m_hThumbnail;

public Thumbnail() { InitializeComponent(); }

public void CreateAndShow(IntPtr sourceWindow)

{

m_hThumbnail = DwmApi.DwmRegisterThumbnail(

Handle, sourceWindow);

DwmApi.DWM_THUMBNAIL_PROPERTIES m_ThumbnailProperties =

new DwmApi.DWM_THUMBNAIL_PROPERTIES();

m_ThumbnailProperties.dwFlags =

DwmApi.DWM_THUMBNAIL_PROPERTIES.DWM_TNP_VISIBLE +

DwmApi.DWM_THUMBNAIL_PROPERTIES.DWM_TNP_OPACITY +

DwmApi.DWM_THUMBNAIL_PROPERTIES.DWM_TNP_RECTDESTINATION +

DwmApi.DWM_THUMBNAIL_PROPERTIES.

DWM_TNP_SOURCECLIENTAREAONLY;

m_ThumbnailProperties.opacity = 255;

m_ThumbnailProperties.fVisible = true;

m_ThumbnailProperties.rcSource =

m_ThumbnailProperties.rcDestination = new DwmApi.RECT(0, 0,

ClientRectangle.Right, ClientRectangle.Bottom);

m_ThumbnailProperties.fSourceClientAreaOnly = false;

DwmApi.DwmUpdateThumbnailProperties(

m_hThumbnail, m_ThumbnailProperties);

Show();

}

protected override void Dispose(bool disposing)

{

if (disposing && (components != null)) components.Dispose();

base.Dispose(disposing);

if (m_hThumbnail != IntPtr.Zero)

{

if (DwmApi.DwmIsCompositionEnabled())

DwmApi.DwmUnregisterThumbnail(m_hThumbnail);

m_hThumbnail = IntPtr.Zero;

}

}

}

In addition to the two functions for registering and unregistering a thumbnail, there are two other functions that work with thumbnails. DwmQueryThumbnailSourceSize returns the source size of the specified thumbnail. DwmUpdateThumbnailProperties allows you to update the properties for a given DWM thumbnail. It takes a DWM_THUMBNAIL_PROPERTIES structure, for which a C# implementation is shown in Figure 5.

The DWM_THUMBNAIL_PROPERTIES structure allows you to specify a number of properties such as the destination rectangle in the target window (the rcDestination member) and the rectangular region of the source window to use (the rcSource member) in the event that you don't want to use the entire source window.

You can also specify the opacity of the thumbnail if you don't want it to be totally opaque by using the opacity member, where 0 is transparent and 255 is opaque. If you want the thumbnail to be invisible, you can set the fVisible flag to false. If you want to use just a windows' client area for the thumbnail instead of the entire source window (which will include the non-client areas such as the frame and title bar), you can set the fSourceClientAreaOnly Boolean to true. The dwFlags parameter is how you tell the interface which parameter(s) you are setting. When you set a parameter, you'll need to turn on the corresponding bit in the dwFlags parameter.

Finally, there's no restriction as to the size of the target window-it's perfectly legal to use the thumbnail interface to magnify the source window or to shrink it. There is a restriction about maintaining the aspect ratio. The aspect ratio of the source window is always maintained. If the source window changes size, the thumbnail will change size as well to maintain itself within the boundaries specified.

In the source code that accompanies this article, there's a button that will create a small live thumbnail of the main application window, as seen in Figure 12.

The thumbnail rendering is live, which you can easily see. If you change the main application window, you'll see the thumbnail update. You will find that with a little ingenuity, it's pretty easy to create your own task switcher using thumbnails and the FindWindow and GetWindow Win32 functions.

Figure 12a Create a Live Thumbnail

Figure 12b

Wrap-Up

That was a whirlwind tour of the DWM interfaces. I think you'll be able to find some nifty applications for these APIs. In particular, I think we'll be seeing some very slick application of the glass effect in the future as folks get used to applying it to their windows. For more information, I recommend Greg Schechter's blog at blogs.msdn.com/greg_schechter.

Ron Fosner has been writing 3D applications on Windows for 20 years and is starting to get the hang of it. He runs DirectX.com and loves to write fast OpenGL and Direct3D applications. You can reach him at Ron@directx.com. Thanks to Greg Schechter and Jevan Saks of Microsoft for their help with this article.

Event Tracing

Improve Debugging And Performance Tuning With ETW

Dr. Insung Park  and Ricky Buch

This article discusses:

- Understanding the ETW architecture

- What's new for ETW in Windows Vista

- Programming with the event provider APIs

This article uses the following technologies:

Windows Vista

Contents

Event Tracing for Windows

The Unified Event Provider Model and APIs

Design and Implementation Guidelines

Tools and Support

Conclusion

The increasing size and complexity of today's software systems make software development and management a tough challenge. Accounting for all execution states is nearly impossible, and applications often exhibit behaviors developers did not expect. Furthermore, an enormous number of hardware combinations and constantly varying workload characteristics add to the difficulty of diagnosing various software problems. It is not surprising then that reliability and manageability are becoming important features themselves. These features, in turn, create a need for instrumentation.

Clever instrumentation added for a few key error states in software execution can drastically reduce the time spent debugging problems. Instrumentation is also useful in other cases. Certain undesirable situations, such as software and hardware failures and low resource conditions, must be monitored and acted upon in a managed business environment for a large group of computers. Moreover, instrumentation can be very helpful for resolving performance problems, which can be hard to diagnose since they are sensitive to external workload, configuration parameters, and underlying hardware and software status. Traces from a production machine during a period of performance degradation may enable developers and administrators to pinpoint poorly performing components or services or identify bottlenecks that were not foreseen during development. Finally, IT professionals use various management tools to compute resource usage statistics from transaction traces for capacity planning and trend analysis.

Event Tracing for Windows® (ETW) is a general-purpose, high-speed tracing facility provided by the operating system. Using a buffering and logging mechanism implemented in the kernel, ETW provides a tracing mechanism for events raised by both user-mode applications and kernel-mode device drivers. Additionally, ETW gives you the ability to enable and disable logging dynamically, making it easy to perform detailed tracing in production environments without requiring reboots or application restarts. The logging mechanism uses per-processor buffers that are written to disk by an asynchronous writer thread. This allows large-scale server applications to write events with minimum disturbance.

ETW was first introduced on Windows 2000. Since then, various core OS and server components have adopted ETW to instrument their activities, and it's now one of the key instrumentation technologies on Windows platforms. A growing number of third-party applications are using ETW for instrumentation as well, and some take advantage of the events provided by Windows itself. ETW has also been abstracted into the Windows preprocessor (WPP) software tracing technology, which provides a set of easy-to-use macros for tracing "printf" style messages for debugging during development.

On Windows Vista™, ETW has gone through a major upgrade, and one of the most significant changes is the introduction of the unified event provider model and APIs. In short, the new unified APIs combine logging traces and writing to the Event Viewer into one consistent, easy-to-use mechanism for event providers. At the same time, several new features have been added to improve developer and end user experience. In this article, we will introduce the new ETW provider model and describe how developers can adopt the new model in their Windows Vista-based applications.

We start by presenting an overview of ETW architecture and usage model, then describe the new event model and the APIs. We'll follow with a short guide to event instrumentation design and implementation and, finally, look at in-box tools for controlling ETW sessions, processing logged events, and analyzing them for higher-level report generation.

Event Tracing for Windows

The core architecture of ETW is illustrated in Figure 1. As shown, there are four main types of components in ETW: event providers, controllers, consumers, and event trace sessions. Buffering and logging take place in event tracing sessions, which accept events and create a trace file. There are a number of logging modes available for ETW sessions. For instance, a session can be configured to deliver events directly to consumer applications or to overwrite old events in a file by wrapping around when a certain size is reached. A separate writer thread created for each session flushes them to a file or to real-time consumer applications. To enable high-performance, per-processor buffers are used to eliminate the need for a lock in the logging path.

Figure 1 ETW Architecture (Click the image for a larger view)

An event provider is a logical entity that writes events to ETW sessions. Any recordable activity of significance can be an event, and each is represented by an event logged to ETW. An event provider can be a user-mode application, a managed application, a driver, or any other software entity. The only requirement is that the event provider must register a provider ID with ETW through the registration API. A provider first registers with ETW and writes events from various points in the code by invoking the ETW logging API. When a provider is enabled dynamically by the ETW controller application, calls to the logging API send events to a specific trace session designated by the controller. Each event sent by the event provider to the trace session consists of a fixed header that includes event metadata and additional variable user-context data. Due to the growing event instrumentation in many OS components, even a simple application for Windows Vista will already contain several components that are event providers.

When an event is logged to a session, ETW adds a few extra data items along with the user-provided data. They include timestamp, process and thread ID, processor number, and CPU usage data of the logging thread. These data items are recorded in the ETW event header and passed on to event consumers along with the variable event content given by the provider. Many trace consumers find these data fields to be essential in their analysis.

A controller starts and stops ETW sessions and enables providers to them. In some scenarios, such as debugging and diagnosis, a controller tool is invoked as needed to collect in-depth traces. In contrast, for events such as admin-targeted events (we will define these in a later section) that need to flow to the Event Viewer at all times, providers are enabled automatically by the event log service when they register. A controller must have ETW permission on Windows Vista to control sessions, which is given only to a small group of privileged users by default.

Lastly, a consumer is an application that reads log files or listens to a session for real time events and processes them. Event consumption is callback-based; a consumer registers an event callback, which ETW calls with one event at a time. Events are delivered to the ETW consumer in chronological order. There are general-purpose event consumer tools that dump the events into various formats. Figure 2 shows an XML dump of a "Process" event logged by the kernel provider as generated by the tracerpt.exe tool on Windows Vista. This event indicates the start of a Notepad process. Since events contain custom user content logged by the provider, some type of metadata is needed to decode them correctly. The providers using the new APIs are expected to supply an event manifest-an XML file-that defines all events that providers write along with their layout information. A general-purpose consumer application uses Trace Data Helper (TDH) APIs to retrieve the event metadata, decode the events, and display them.

Figure 2 XML Dump of Process Start Event

<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">

<System>

<Provider Guid="{9e814aad-3204-11d2-9a82-006008a86939}" />

<EventID>0</EventID>

<Version>2</Version>

<Level>0</Level>

<Task>0</Task>

<Opcode>1</Opcode>

<Keywords>0x0</Keywords>

<TimeCreated SystemTime="2006-12-18T12:26:27.887309500Z" />

<Correlation

ActivityID="{00000000-0000-0000-0000-000000000000}" />

<Execution ProcessID="3396" ThreadID="3260" ProcessorID="0"

KernelTime="390" UserTime="195" />

<Channel />

<Computer />

</System>

<EventData>

<Data Name="UniqueProcessKey">0xFFFFFA800143FA80</Data>

<Data Name="ProcessId">0x10EC</Data>

<Data Name="ParentId">0xD44</Data>

<Data Name="SessionId">1</Data>

<Data Name="ExitStatus">0</Data>

<Data Name="UserSID">guest</Data>

<Data Name="ImageFileName">notepad.exe</Data>

<Data Name="CommandLine">notepad</Data>

</EventData>

<RenderingInfo Culture="en-US">

<Opcode>Start</Opcode>

<Provider>MSNT_SystemTrace</Provider>

<EventName xmlns=

"http://schemas.microsoft.com/win/2004/08/events/trace">

Process</EventName>

</RenderingInfo>

<ExtendedTracingInfo xmlns="

http://schemas.microsoft.com/win/2004/08/events/trace">

<EventGuid>{3d6fa8d0-fe05-11d0-9dda-00c04fd7ba7c}</EventGuid>

</ExtendedTracingInfo>

</Event>

To many, tracing means collecting events from certain providers of interest. In this way of thinking, an event trace session is tied to one or more providers in a conceptual collection as a whole, and a session itself (the logging engine) is often overlooked. The ETW architecture allows for more dynamic and flexible trace and event management. Here, sessions and providers exist in different spaces. A controller is the one that starts and stops ETW sessions and enables providers to sessions dynamically. Thus, a controller can choose to enable a group of providers to a session, disable some of them after a while, and enable another provider to that same session later. Sessions operate in the kernel and are not statically tied to providers. Likewise, providers typically are not aware of which sessions their events are being logged to. There are large scale applications and services that are providers, controllers, and consumers all at the same time. APIs are provided for all operations for providers, controllers, and consumers, and applications may assume any combination of roles. In general, however, developers implement only event providers and use in-the-box tools to collect traces and view them.

One of the advantages of the separation of providers and trace sessions is that tracing becomes immune to application problems such as crashes or hangs. Events logged by providers before a crash will be in the kernel memory, if not in a trace file already, which makes this particularly useful in debugging application anomalies.

As mentioned earlier, events are used by developers, IT administrators, and management tool developers for debugging, monitoring, diagnosis, and capacity planning. The usual analysis methodologies based on events can be categorized into the following techniques.

Scanning Users scan through the event dump to find a single important event or a small pattern of known events. This is typically done when events are used to debug for failure cases corresponding to end-user problems or to search for significant failures in the event log.

Delta Analysis Since ETW captures timestamp and CPU usage numbers for each event, such as simple delta analysis of the form

Property (Event B)-Property (Event A)

allows for response time and CPU usage statistics of application activities. If two events mark a beginning and end of an activity, a large set of events collected from production mode applications can be processed in this manner to produce a summary of response time and CPU usage statistics.

Statistical Analysis Sometimes, just counting certain events provides extra insight into software behavior.

State Machine and Resource Tracking A sufficient set of events enables the construction of a state machine and in turn, a simulation based on traces. For instance, since the majority of core OS activities are instrumented with ETW events, OS traces can be used to build a state machine that keeps track of scheduler, memory, I/O activities, and so forth.

End-to-End Tracing Large applications often consist of a number of distributed components integrated via complicated interconnections. They frequently encompass multiple machines, each serving a different role. Instrumentation geared toward request tracking is one of the approaches to overcoming the increasing difficulty of diagnosing problems in such environments. In this framework, instrumentation points are added throughout applications that record activities along with the unique ID for the request currently being served. After traces are collected, events that correspond to the same request are correlated during event consumption such that its activity and progress can be tracked. Later, specific requests of interest can be looked at individually for problems in different service stages, or a group of requests can be summarized through a statistical analysis. Server Performance Advisor (SPA), available from the Microsoft Download Center, is a server management and diagnosis tool that employs server instrumentation designed for end-to-end tracing.

The Unified Event Provider Model and APIs

On Windows Vista, we introduce a new set of event provider APIs that are easier to use while offering more features and enhanced security options. The new APIs are also used to write to the Event Viewer, uniting trace and event logs into one consistent API set. This section describes the model and the APIs in more detail. The differences between the new APIs and the existing ones will be pointed out as necessary.

In order to become an ETW provider, a software component needs to register with ETW through the EventRegister API. EventRegister requires a GUID called ProviderId that uniquely identifies the provider. Any software entity (application, shared DLL, or driver) can register as a provider since a provider does not have to be bound to any OS entity. Registration is typically done at the entry point into the component: a DLL attach routine or a driver entry, for example. A registration handle is returned to be used in subsequent logging API calls. Finally, an EventUnregister call is made at the end of the provider execution.

With the existing provider APIs, providers must supply a callback for enable/disable notification. That is, if a controller enables the provider, the registered callback function is invoked with enable settings. One of the parameters to the callback is a handle to the session that the provider is enabled to. Upon receiving an enable callback, the provider sets a global variable (such as TracingOn) to indicate whether tracing is on or off and stores the session handle as well. Then it uses this session handle, which it obtained from the callback, in logging API calls (conditionally based on the value of TracingOn).

In the new ETW provider model, ETW remembers the enable settings on behalf of providers. In other words, providers register and invoke logging calls without having to check whether they are currently enabled or not. Within the logging API, ETW quickly looks at the enable settings and sends the events to sessions only if they are enabled. If they're not, the logging call is discarded. Hence, enable/disable callback is optional in the new model. However, an enable callback may still be necessary in some scenarios. For instance, instrumentation intended for a state machine construction often requires a snapshot or state rundown events at the beginning and end of traces.

The logging API, EventWrite, takes a registration handle (as opposed to a session handle in the old model). Using a registration handle in the logging calls allows enable settings to be transparent to providers. Since the handle used in logging is an opaque handle in the new model, ETW is now capable of multicasting events. That is, a provider can be enabled to more than one ETW session, which was not possible in the old model where a session handle is used.

ETW provides separate APIs, EventEnabled and EventProviderEnabled, that test if the provider is enabled. Although the logging APIs already check the enable setting before writing events, sometimes providers may require additional work when tracing is enabled. One such scenario involves gathering and constructing informational event data that is not necessarily required for program execution. With these APIs, a provider can find out at any time whether tracing is enabled or not, if there is a need.

As stated previously, users can add variable context data to each event. The logging API employs a scatter/gather mechanism for picking up event-specific data items. Callers pass in additional event data items by constructing an array of data descriptors. A data descriptor is a struct with pointer and size fields. Thus, users add one data descriptor for each data item to log. A macro (EventDataDescCreate) is provided for easy construction of a data descriptor. ETW then copies user-provided content into its session buffers during logging.

In the corresponding event manifest, the layout of an event should be specified through a <Template> tag. A template describes user-specified context data that each event includes. The template can define the layout, which may contain individual data fields, such as integers and strings, or complex data structures such as an array of structs. A template is not required for all events; if no template is specified, that event is expected to have no user-provided data. In the manifest, a template may be shared by multiple events, such as a Start and Stop event having the same context information. When a consumer application encounters an event, it locates an event template through the TDH APIs and decodes the variable event data accordingly. Providers using the old APIs supply layout information through Windows Management Instrumentation (WMI) Managed Object Formats (MOFs).

Figure 3 contains sample provider code that uses the new event APIs for registration and logging. This is a user-mode provider that employs user-mode provider APIs, but a corresponding set of kernel-mode provider APIs are also available. The first event in Figure 3 writes two user-provided data items. One is of type ULONG and the other a NULL terminated WCHAR string. EventDataDescCreate is invoked to construct the appropriate data descriptor array. In addition to the EventWrite API shown here, there are two more logging APIs: EventWriteString and EventWriteTransfer. EventWriteString allows simple logging of a non-manifested string. When EventWriteString is called, ETW marks the header denoting that the event data is a single NULL terminated WCHAR string. If a consumer sees this in the header, it treats the user data as a string, without having to search for the event schema through TDH. EventWriteString enables a quick logging of strings without changing the manifest.

Figure 3 ETW Provider

#include <myevents.h>   // Header generated from manifest.

// Contains MyProviderId and event descriptors.

REGHANDLE MyProvRegHandle;

ULONG MyInteger;

PWCHAR MyString;

ULONG MyStringLength;

EVENT_DATA_DESCRIPTOR DataDescriptor[2];

...

// Register the ETW provider.

Status = EventRegister(&MyProviderId,      // ProviderId (GUID)

NULL,               // Optional Callback

NULL,               // OPtioanl Callback Context

&MyProvRegHandle);  // Registration Handle

...

// Construct DataDescriptor and write an event with

// MyInteger and MyString.

EventDataDescCreate(&DataDescriptor[0],    // DataDescriptor

&MyInteger,            // Pointer to the data

sizeof(ULONG));        // Size of data

EventDataDescCreate(&DataDescriptor[1], &MyString, MyStringLength);

Status = EventWrite(MyProvRegHandle,       // Registration Handle

MyEventDescriptor1,    // EventDescriptor

2,                     // DataDescriptor array size

DataDescriptor);       // DataDescriptor array

...

// Write another event with no user data.

if (EventEnabled(MyProvRegHandle, MyEventDescriptor2)) {

// Do extra work if enabled and write event.

...

Status = EventWrite(MyProvRegHandle, MyEventDescriptor2, 0, NULL);

}

...

// Unregister the ETW provider.

Status = EventUnregister(MyProvRegHandle);

The EventWriteTransfer and EventActivityIdControl APIs cater to the need of end-to-end tracing instrumentation. As described earlier, end-to-end tracing is an instrumentation methodology targeting server applications that perform different activities simultaneously for a number of user requests. For instance, a request for a script execution on a Web page leaves a client machine and arrives at the network layer on a server. Then it travels through the HTTP driver, IIS, the ASP.NET engine, and maybe an Exchange Server on another machine. The goal of end-to-end tracing is to record all the relevant activities regarding this request through ETW events for later debugging and performance analysis. This requires a unique ID that can identify individual requests. Correlation can be performed during consumption with the help of this unique activity ID.

ETW addresses this need by introducing an ActivityId in each event. Every event logged with the new API automatically picks up the current activity ID stored in the executing thread. The activity ID is displayed in the XML dump in <Correlation ActivityId> tag in the <System> section. The provider can get, set, and create an activity ID for the executing thread using the EventActivityIdControl API. The activity ID can travel with the request across multiple components. Unfortunately, it may not be possible to propagate the activity ID in some cases due to public protocol restriction, design restriction, and so on. The EventWriteTransfer API writes a transfer event, indicating a transfer of activity IDs. In addition to all the parameters of EventWrite, EventWriteTransfer takes two more arguments-ActivityId and RelatedActivityId.

Every event is stamped with the provider ID and assigned an entity called the event descriptor, which defines standard event information and provides further identification and semantics to it. Developers determine the event descriptors for instrumentation points during the instrumentation design phase and write corresponding entries in the event manifest. Then the Message Compiler in a development environment generates event descriptors in a header file from the given event manifest, which in turn are included and used in source files. Programmatically, event descriptor is a struct consisting of the following fields: Id, Version, Channel, Level, Opcode, Task, and Keywords:

typedef struct _EVENT_DESCRIPTOR {

USHORT Id;

UCHAR Version;

UCHAR Channel;

UCHAR Level;

UCHAR Opcode;

USHORT Task;

ULONGLONG Keyword;

} EVENT_DESCRIPTOR, *PEVENT_DESCRIPTOR;

An event ID is used to uniquely identify an event in a provider. When an event is defined in the manifest, the event ID is the only mandatory entry. Upon encountering an event, a consumer uses its provider ID (GUID) and event ID (USHORT) to locate the manifest for it. Likewise, a version offers events to be changed and augmented in later releases while retaining the same semantics and event ID. Therefore, the event ID and version together with the provider ID can uniquely identify an event.

A channel defines a group of events for a target audience. A channel belongs to one of the four types: admin, operational, analytic, and debug. Events raised into an admin channel are actionable events; upon receiving an event, the administrator should immediately know why the event was raised and what to do about it. Events raised into an operational channel are targeted at high-level monitoring tools and support staff; they offer more detailed context and are more frequent than admin channel events. Events grouped in admin and operational channels are sent to the event log automatically and displayed in the Event Viewer. An analytic channel is for traditional traces that are targeted at expert-level support professionals or detailed diagnosis and troubleshooting tools. A debug channel is to be used for debug messages and contains events that are meant to be consumed by developers. Analytic and debug channel events are not enabled by default. Through channels, events intended for different purposes and audiences can be added with one set of APIs.

When enabling a provider, a controller can specify a level (1-byte integer) and keywords (8-byte bit masks). The level and the keywords are used to add dimensions to ETW instrumentation. The level is designed to enable filtering based on the severity or verbosity of events. The keyword is designed to indicate sub-components in a provider. For instance, developers may divide events into informational events and critical-error events. Also, they can assign different keywords to sub-components of the application. By enabling selectively with different level and keywords, the trace controller can enable providers to log only error events from sub-component B, or all events from sub-components A and C, and so forth. For providers using the old APIs, a keyword is 4 bytes, and filtering by level and keyword has to be done explicitly in the provider code. It is important to note that when a controller enables a particular level, all events with a level value less than or equal (higher or equal severity) to what the controller specified are also enabled. Although levels and keywords can be custom designed and assigned to events by developers, there are predefined levels, as shown in Figure 4.

Figure 4 Severity Levels

<levels>

<level name="win:LogAlways" symbol="WINEVENT_LEVEL_LOG_ALWAYS"

value="0" message="$(string.level.LogAlways)"> Log Always

</level>

<level name="win:Critical" symbol="WINEVENT_LEVEL_CRITICAL" value="1"

message="$(string.level.Critical)"> Only critical errors </level>

<level name="win:Error" symbol="WINEVENT_LEVEL_ERROR" value="2"

message="$(string.level.Error)"> All errors, includes win:

Critical </level>

<level name="win:Warning" symbol="WINEVENT_LEVEL_WARNING" value="3"

message="$(string.level.Warning)"> All warnings, includes

win:Error </level>

<level name="win:Informational" symbol="WINEVENT_LEVEL_INFO"

value="4"

message="$(string.level.Informational)"> All informational

content, including win:Warning </level>

<level name="win:Verbose" symbol="WINEVENT_LEVEL_VERBOSE" value="5"

message="$(string.level.Verbose)"> All tracing, including

previous levels </level>

</levels>

Task and opcode are used to attach additional information to each event. A task specifies a common logical component or task being instrumented. The task often represents the key high-level steps that a component takes to achieve its purpose. An opcode signifies the specific operation being performed at the time the event is written. For example, the Windows kernel provider groups all file I/O operation events into a "FileIO" task. Opcode indicates what the operation was, such as Create, Open, Read, and Write. Unlike ID, version, channel, level, and keyword, task and opcode are only used for adding information; they do not have any impact on controlling instrumentation or locating metadata.

The event descriptors and layouts are specified in the event manifest. Developers write the event manifest when instrumentation is designed. An event manifest is written in XML, in which user-defined channels, tasks, opcodes, levels, and keywords are specified in appropriate XML tags. ETW predefined channels, levels, and opcodes can also be used; the predefined channels, however, refer to global channels and should only be used for admin-targeted events. The different metadata fields are combined for each event and defined in an <Event> tag that is uniquely associated with an event ID. An event can also have a message string that will be displayed with substituted values from the event data when the event is being read by a consumer. Figure 5 shows an XML fragment from a sample event manifest. Later we'll present a step-by-step guideline to event instrumentation design and instrumentation.

Figure 5 Event Manifest Fragment

<provider name="Microsoft-Windows-Kernel-Registry"

guid="{70eb4f03-c1de-4f73-a051-33d13d5413bd}"

symbol="RegistryProvGuid"

resourceFileName="%SystemRoot%\System32\advapi32.dll"

messageFileName="%SystemRoot%\System32\advapi32.dll">

<channels>

<channel name="Microsoft-Windows-Kernel-Registry/Analytic"

chid="RegistryEvents" symbol="REG_Events" type="Analytic"

isolation="System">This channel contains registry

events.</channel>

</channels>

<opcodes>

<opcode value="32" name="CreateKey" symbol="" />

...

</opcodes>

<keywords>

<keyword name="CreateKey" symbol="" mask="0x1000" />

...

</keywords>

<templates>

<template tid="tid_RegOpenCreate">

<data name="BaseObject" inType="win:Pointer"

outType="win:HexInt64" />

<data name="KeyObject" inType="win:Pointer"

outType="win:HexInt64" />

<data name="Status" inType="win:UInt32"

outType="win:HexInt32" />

<data name="Disposition" inType="win:UInt32" />

<data name="BaseName" inType="win:UnicodeString"

outType="xs:string" />

<data name="RelativeName" inType="win:UnicodeString"

outType="xs:string" />

</template>

...

</templates>

<events>

<event value="1" symbol="ETW_REGISTRY_EVENT_CREATE_KEY"

template="tid_RegOpenCreate" opcode="CreateKey"

channel="RegistryEvents" level="win:Informational"

keywords="CreateKey"

message="$(string.Registry.RegOpenCreate)"/>

...

</events>

</provider>

...

<localization>

<resources culture="en-US">

<stringTable>

<string id="Registry.RegOpenCreate"

value="Registry key %6 was created with status %3." />

</stringTable>

</resources>

</localization>

Managed ETW provider APIs will also be available in the Microsoft® .NET Framework 3.5, code-named "Orcas." Managed consumer and controller APIs are currently in the planning stage. System.Diagnostics.Eventing contains the EventProvider class. EventProvider offers methods for all the functionality described above for the native applications. Users need to instantiate the class with a provider GUID and use the class instance to log events. EventDescriptor is a struct equivalent to the event descriptor in the native APIs. Unlike the native case, EventDescriptors are not generated for managed applications. However, a tool capable of generating code aimed at better performance and validation is under consideration. A sample usage of EventProvider is shown in Figure 6.

Figure 6 Managed ETW Provider

using System.Diagnostics.Eventing;

...

static void Main(string[] ArgRead)

{

int MyInteger;

string MyString;

...

// Construct event descriptor.

EventDescriptor Event1 = new EventDescriptor(5, 0, 0, 2, 0, 0, 0);

// Instantiate event provider.

EventProvider etwProvider = new EventProvider(

new Guid("d58c126f-b309-11d1-969e-0000f875a5bc"));

...

// Write an event with MyInteger and MyString.

etwProvider.WriteEvent(ref Event1, MyInteger, MyString);

...

}

In addition, ETW on Windows Vista offers improved security options for providers. By default, any provider can register and write events. However, developers can place restrictions on a GUID so that only authorized users can register a provider with that GUID. A provider can also specify who can enable it. Furthermore, ETW allows a controller to declare a session to be secure, meaning that it receives events only from a certain group of users.

Once the binary with the event provider instrumentation is compiled, the provider is installed. Then users can collect events from the provider using the logman tool, an in-the-box controller application. The logman.exe command below issues two ETW control API calls that start an ETW session called mysession and enable a provider:

> logman start mysession -p <provider name> -o mytest.etl -ets

Session mysession writes events to a file called mytest.etl. There are a number of logman.exe options that customize logging modes, buffer configuration, and so on. Here, <provider name> can be the provider name in the manifest or the GUID that the provider uses to register with using the EventRegister API. If the provider GUID is 11223344-5566-7788-99aa-bbccddeeff00, the actual command line becomes the following:

> logman start mysession

-p {11223344-5566-7788-99aa-bbccddeeff00}

-o mytest.etl -ets

ETW allows a controller to pre-enable a provider before it registers and executes. This lets users collect startup traces for dlls and drivers for which the loading of binary images is not easily controlled. A controller starts a session and enables a provider before the provider begins execution. As soon as the provider registers, it is enabled.

After events are collected as needed, the session can be stopped by using the stop option in logman.exe.

> logman stop mysession -ets

A trace file called mytest.etl should now exist in the same directory in which the logman command was executed. Tracerpt.exe can be used to get the dump of events from this file:

> tracerpt mytest.etl

This command creates a dump file (dumpfile.xml by default) and a text summary file (summary.txt by default). The summary file contains a short summary of event statistics. The dump file contains XML dumps of events, such as the one shown in Figure 2. The data fields from the event header are presented in the <System> section. Process ID, thread ID, processor ID, and CPU usage are given in the <Execution> tag within <System>. Also, the <System> section shows the values of the event descriptor fields of the event. Output file names can be specified by the user as options to the tracerpt.exe command. There are other GUI tools on Windows Vista capable of collecting and viewing traces, which we'll discuss later.

There are already hundreds of event providers installed on Windows Vista. Users can see the list of these providers from PerfMon or the Event Viewer or through a logman.exe command:

> logman query providers

Design and Implementation Guidelines

In this section, we discuss the steps to create the manifest file that defines the events, instrument the code with ETW API calls, and successfully build and install the event provider.

Design the Instrumentation Manifest The instrumentation manifest is an XML-format file that defines the event provider and the events that the event provider logs to ETW. Each event has a standard metadata and a variable event data section. The manifest can either be created by hand or using the manifest generator tool available in the SDK (ecmangen.exe). Here are the steps:

1. Create an event provider and determine its provider ID, which is a GUID.

2. Define the logical tasks of the component and the specific instrumentation points within those logical tasks. This defines the tasks and opcodes for the events that the provider writes.

3. Define the subcomponents that will be logging events and assign a keyword for each subcomponent. This allows controllers to selectively enable events.

4. Assess the target audiences of events being designed and define channels for each target audience.

5. Create the different constructs that will define the variable user context data. This is done in the <templates> section.

6. Put everything together and define events in the <events> section of the manifest. Assign to each event the symbolic identifier, severity, keywords, channel, task, opcode, and template. A localizable event-specific message can be added as well.

Instrument the Code with the ETW API After the events have been defined in the manifest, use the unified APIs to write events to ETW. For native applications, first generate a header file that contains the event definitions from the manifest so it can be included in the provider code. Then use the ETW APIs to write the events designed previously. This is the process:

1. Run the Message Compiler (mc.exe) on the manifest created above. This generates the header file that contains event descriptor structs for the events, with each struct instance having the same name as the event symbolic identifier. The Message Compiler also produces the resource string file. The generated header file needs to be included in the provider code.

2. Before writing code to raise events, a provider needs to register with the provider ID defined in the manifest by calling the EventRegister function. A registration handle returned from EventRegister should be stored in order to be used for logging API calls and the corresponding EventUnregister call.

3. Call EventWrite to log the events defined in the manifest. The event-specific data is passed in through a data descriptor array as depicted in Figure 3. (Developers must make sure that the data fields exactly match the corresponding event templates specified in the manifest.) Pass in the right event descriptor struct defined in the header file with the event.

4. After logging all the necessary events, call EventUnregister and pass in the registration handle.

Compiling the Instrumented Binary The manifest information that defines the events should be compiled during the build process and attached as a resource into the provider executable binary. All the localizable strings will be taken out of this compiled manifest and placed in a separate resource file that must be compiled and linked separately. When the provider binary is deployed, the manifest information needs to be installed on the system so that both controllers and consumers can find the event provider and event definitions. All the required tools are available natively on Windows Vista or from the SDK. The steps involved follow:

1. If there are localizable message strings for the events, it may be necessary to run rc.exe to compile the resource file into a .res.

2. If Visual Studio® is used, the developer can include a pre-build command to run rc.exe to generate the .res compiled resource and include this compiled resource file in the project.

3. Link the compiled resource file using either cl.exe or link.exe (this is done in Step 2 for users of Visual Studio) and compile the application or driver.

4. When installing the provider, run the wevtutil.exe tool with the -im option to install the manifest that defines the events. This allows controllers to discover the event providers and also lets consuming tools find the information needed to decode events properly.

Tools and Support

So far we have used logman.exe, a command-line controller, and tracerpt.exe, a general consumer tool. Logman.exe also has an option to schedule collections on remote machines. And Tracerpt.exe is capable of dumping events in CSV format, and it can generate resource-consumption reports using the -report option for some of the known core OS events. Figure 7 shows the hot file table generated from tracerpt.exe after processing and correlating some known core OS events based on a state machine-building technique.

Figure 7 Hot File Report Generated by Tracerpt (Click the image for a larger view)

Reliability and Performance Monitor (RPM) (which includes PerfMon on Windows Vista) offers a graphical interface for collecting traces. Using the Event Trace Sessions interface, users can set up a trace session, pick providers to enable, and start and stop ETW sessions. Figure 8 shows RPM when a user is looking at the list of providers to enable. RPM introduces the Data Collector Set concept, in which all the necessary information regarding event traces, performance counters, and configuration (registry and WMI classes) data collection is combined into a single collection set. Users can create a data collection set with providers enabled for trace and counter data. Once created, the collection set settings are stored in RPM so that users can easily start and stop data collection without having to specify sources every time. This feature supports common diagnostic scenarios, where users can specify the providers to collect traces and counters from for some known diagnostic scenarios.

Figure 8 Event Trace Providers in RPM (Click the image for a larger view)

The Event Viewer offers another means for collecting and viewing events. The new event APIs were designed for events directed to the event log as well, and admin and operational channel events are automatically routed to and displayed in the Event Viewer, to be monitored for important software status. The Event Viewer also expands its functionality for analytic and debug events. The "View" option allows displaying all available analytic and debug channels, and users can right-click to enable trace collection. In Figure 9, a user right-clicked an analytic channel of the registry event provider to enable trace collection.

Sample code is available in software development kits for provider, controller, and consumer APIs. Sample providers using the new provider APIs include manifests for the events that they log. A driver sample is available as well.

Figure 9 Event Viewer Interface for Analytic Event Collection (Click the image for a larger view)

Conclusion

We have presented the unified event provider model and APIs for ETW newly available on Windows Vista. Our goal is to introduce the APIs to developers and provide them with a usage example and general guidelines. Tracing is still an unfamiliar technology to many developers, and we hope this article provides insight into one of the key instrumentation and diagnostic infrastructures on Windows platforms. Within a few years since its introduction as a general tracing technology on the Windows 2000 operating system, a number of components and applications have employed ETW as a basis for their tracing and manageability features, and this trend is likely to continue.

We anticipate a greater number of developers will begin to adopt event instrumentation technology in order to deliver more reliable, better performing, and more easily manageable applications. Instrumentation will not only help these developers debug their applications, but will also help the people who deploy and manage the applications. Furthermore, instrumentation will aid in capacity planning, trend analysis, and finding performance bottlenecks as well. ETW provides a useful tracing infrastructure to enable applications to provide instrumentation to address all of the above scenarios, and we expect this technology to play a fundamental role as the core building block in the endeavor to make applications more manageable and more diagnosable.

Dr. Insung Park is a Development Lead for the Windows Instrumentation Platform Team. He has published a dozen papers on performance analysis, request tracking, instrumentation technology, and programming methodology and support. His e-mail address is insungp@microsoft.com.

Ricky Buch is a Program Manager for the Windows Instrumentation Platform Team. He works on both Event Tracing for Windows and the Performance Counter Library technologies. He can be reached at ricky.buch@microsoft.com.

IIS 7.0

Explore The Web Server For Windows Vista And Beyond

Mike Volodarsky

This article discusses:

- Modular Web server functionality

- Simplified deployment and configuration

- Extensibility and ASP.NET integration

- Improved security, performance, and compatibility

This article uses the following technologies:

Windows Vista, IIS

Contents

Modular Web Server

Simplified Deployment and Configuration

Improved Administration

.NET Framework and Scripting

Building Web Server Features

ASP.NET Integration

Improved Security

Improved Diagnostics

Diagnosing Errors without Debugging

Improved Performance

Backward Compatibility

Conclusion

I often hear people-both inside and outside of Microsoft-refer to the new IIS 7.0 Web server as one of the most important developments coming out of Microsoft over the past several years. This is a rather significant statement, given the impressive lineup of technologies released by Microsoft recently, including Windows Vista™!

The release of IIS 7.0 coincides with the ten-year anniversary of the release of the first version of IIS, in Windows NT® 4.0. In 2001, four versions later, IIS 5.0 became the most prevalent Web server on the Internet, though months later it fell victim to the infamous Code Red and Nimbda worms. IIS 6.0, released in Windows Server® 2003, was a major rewrite of the server, focused entirely on improving security, reliability, and performance. Since then, IIS 6.0 has proven to be a rock-solid Web server, achieving high reliability and security track records with only a single critical security bulletin since its release (which was not exploitable remotely).

In this article, I want to take the opportunity to present the major reasons why the next-generation IIS 7.0 Web server makes such a big difference to developers and administrators alike-and give you a head start in using many of its new features.

The vision for IIS 7.0 was to take the speed, reliability, and security of the IIS 6.0 codebase and turn it into a highly extensible and manageable Web server platform powerful enough to run Web applications of the future. The result is the most ambitious Microsoft Web server yet, delivering the largest number of architectural improvements in the history of IIS.

At the core of the IIS 7.0 release is a completely modular Web server, comprised of more than 40 features that can be assembled into small-footprint Web servers optimized for the desired role in your application topology. These features are built on top of a new extensibility layer that allows developers to extend or replace virtually any aspect of the server, in native code or with the Microsoft® .NET Framework. IIS 7.0 offers extensibility throughout its runtime, management, and operational features to help you build end-to-end solutions for your specific needs. On top of the core platform, IIS 7.0 tackles many of the problems associated with the manageability and operation of the server. It features a brand new configuration system that enables fully delegated management of sites and finally makes xcopy deployment of Web applications a reality. The new management APIs and diagnostic features make deployment, administration, and troubleshooting of the server significantly easier and more convenient than ever before.

But why should you start thinking about IIS, a server application, before the next version of Windows Server, code-named "Longhorn," is even near final release? It's important to start thinking about it now because Windows Vista ships with the same full-featured IIS 7.0 bits that are expected to be released in Windows Server "Longhorn." This means you can immediately take advantage of the new IIS 7.0 features to build your personal Web site and host it on Windows Vista. What's more, you can get a head start in developing and testing your production Web applications and Web server infrastructure on the same IIS platform you will be deploying them on when Windows Server "Longhorn" ships.

Intrigued? Let's dive into the details.

Modular Web Server

IIS 7.0 breaks up the Web server into a lightweight server core and more than 40 feature modules that can be plugged into this core. These modules -like StaticFileModule, which allows downloads of static Web content, or WindowsAuthModule, which supports integrated NTLM authentication -can be independently installed on the server to provide the exact functionality you need.

These modules can be completely uninstalled from the server at any time (see Figure 1) or specifically disabled for a particular application where they aren't needed. This allows server administrators to deploy minimal footprint servers quickly with dramatically reduced attack surface area and to significantly improve performance by executing only the required code.

Figure 1 Use Only the Features You Want (Click the image for a larger view)

The componentized architecture is a critical property of IIS 7.0, leading to reduced security risks and minimized patching requirements. It also enables specialized server deployments, which combine select IIS features and custom components optimized for a specific server role in your application topology, such as reverse proxies and cache servers, HTTP protocol load balancers, or SSL and security sentinel servers.

All of the server features that ship with IIS 7.0 are built on top of new public extensibility APIs. As a developer, you have the ability to replace any of the existing server features with your own or build a new module to add to the IIS 7.0 feature set. Want to replace the built-in authentication mechanisms with a custom authentication module or provide a new form of response compression? Go ahead.

The new extensibility APIs are a fundamental improvement over the previous ISAPI extensibility model, enabling you to enhance the server with much more flexibility and ease. Virtually every single aspect of the server, starting with the core server and continuing to configuration, management, and diagnostics, provides extensibility to allow you to extend and tailor the server to your own needs. More about extensibility later in the article.

Simplified Deployment and Configuration

The centralized configuration store of the previous IIS releases, known affectionately as the metabase, is gone. IIS 7.0 features a new delegated configuration system based on a hierarchy of distributed XML configuration files. This hierarchy is comprised of a global applicationHost.config file, which contains server-level configuration defaults, and distributed web.config files within the application's directory structure. These are the same web.config files that are used by the ASP.NET application framework to store application settings in a portable way. This allows the side-by-side storage of IIS and ASP.NET configuration, using clean and strongly structured XML directives. Here's an example:

<?xml version="1.0" encoding="UTF-8"?>

<configuration>

<system.web>

<customErrors mode="Off" />

</system.web>

<system.webServer>

<directoryBrowse enabled="true" />

</system.webServer>

</configuration>

In the past, IIS application settings had to be explicitly configured in the machine-level metabase repository before the application could function correctly. With distributed web.config files, applications encapsulate the required server configuration within their directory structure. This dramatically simplifies deployment, allowing self-contained applications to be simply copied to the target server's application directory and thus be up and running immediately with the desired settings.

The new configuration system also gives comprehensive control to the server administrators, allowing them to delegate certain configuration options to the application while maintaining control over others for security or business reasons. In this way, applications on hosted servers can set essential configuration directly in their application without requiring the calling of the server administrator for help or using an external configuration panel.

In the true spirit of IIS 7.0, the configuration system is completely extensible. New modules can add their own configuration schema, enabling applications to configure their features side-by-side with the IIS and ASP.NET configuration:

<configuration>

<system.webServer>

<directoryBrowse enabled="true" />

</system.webServer>

<myBandwidthThrottler enabled="true" />

</configuration>

The custom configuration sections use the same configuration schema as the configuration for the IIS 7.0 features, taking advantage of the strongly typed attribute values, collection syntax, and hierarchical overriding and locking semantics.

IIS 7.0 continues to support existing setup code that uses the Admin Base Object (ABO) APIs to write to the legacy metabase or scripts that use the higher-level Active Directory® Service Interfaces (ADSI) and Windows Management Instrumentation (WMI) objects to configure IIS. It does this by providing a compatibility layer that emulates the ABO APIs, on top of which all other legacy configuration APIs are based, allowing such scripts to read and change configuration just like they did in the previous releases of IIS. (For more information about metabase compatibility, see the "Improved Performance" and "Backward Compatibility" sections towards the end of this article.) While the new structured XML configuration format makes it easier to work with configuration in your favorite text editor, IIS also provides a host of management tools and APIs for administrators to simplify server management and enable automated configuration and deployment.

Improved Administration

IIS 7.0 offers a rich set of administration features that make managing the server possible in a wide range of scenarios. The new graphical IIS Manager administration tool, which replaces the InetMgr.exe MMC snap-in, makes manual server administration very simple with its task-based management interface (see Figure 2).

Figure 2 IIS Manager Provides Graphical Administration Tools (Click the image for a larger view)

IIS Manager allows you to administer most IIS 7.0 features and monitor the server's operation. The tool supports remote administration over a firewall-friendly HTTP/SSL connection, with an option to support both Windows-based and other credentials for authentication.

What's more, the tool enables delegated management, letting application owners manage their applications remotely without having administrative access to the server computer. Armed with this capability, users of hosted services can run the administration tool on their home desktop and connect remotely to manage their applications on the hosted server. Server administrators, of course, have complete control over what management features are delegated to application owners.

Finally, the Administration tool is completely extensible, building on top of configuration system extensibility to allow custom management UI to be added to the tool. You can learn more about the IIS Manager tool and how to add your own administration plug-ins at iis.net/default.aspx?tabid=7&subtabid=73.

For more flexible command-line administration, IIS 7.0 offers the appcmd.exe command-line tool (see Figure 3). This tool provides a comprehensive set of management functionality and better support for bulk operations than the UI. This powerful utility makes it easy to read and write configuration, access site and application pool state information, and perform almost any other management task, all from the command prompt.

Figure 3 Appcmd.exe Command-Line Administration for IIS 7.0 (Click the image for a larger view)

With appcmd.exe, you can create and configure sites, apps, application pools, and virtual directories. You can start and stop sites, recycle application pools, list running worker processes, examine currently executing requests, and search Failed Event Request Buffering (FREB) trace logs. You can also search, edit, export, and import IIS and ASP.NET configuration data.

The tool is built to allow flexible searches for supported server objects, for example, letting you quickly find sites with a particular setting set or application pools that are stopped. When performing a search, you can use any number of conditions on any of the object's properties, including using numeric ranges and simple wildcard string matching.

Appcmd also supports linked operations like those found in Windows PowerShell™, allowing multiple operations on a related set of objects to be performed together from a single command line. For example, you can find and recycle all application pools that host applications for a certain site in a single command. To learn how to manage IIS with AppCmd, check out iis.net/default.aspx?tabid=2&subtabid=25&i=954&p=1.

.NET Framework and Scripting

In addition to the manual server administration with IIS Manager or the appcmd.exe command-line tool, IIS 7.0 provides a wealth of options for programmatic administration. First, you can utilize the Microsoft.Web.Administration API to manage the server from .NET applications. Or you can use the new COM API to manage the IIS configuration system directly, or access it from a scripting environment such as ASP or Windows® Script Host (WSH). There is also a new WMI provider and support for legacy WMI and ADSI providers via the metabase compatibility layer.

Microsoft.Web.Administration, the new .NET Administration API, makes it easy for managed code applications to programmatically provision IIS sites and applications, access important state and diagnostic information, and otherwise configure the server. Enabling .NET Framework-based applications to easily access IIS configuration and state information opens up a world of possibilities for writing .NET-based setup and management applications or even carrying out management tasks directly from ASP.NET pages.

As an example, Figure 4 shows a small C# program that uses Microsoft.Web.Administration to create a new Web site from the command line.

Figure 4 Provisioning Sites with Microsoft.Web.Administration

using System;

using Microsoft.Web.Administration;

class CreateASite

{

static void Main(string[] args)

{

ServerManager serverManager = new ServerManager();

Site mySite = serverManager.Sites.Add(

"MySite", "d:\\inetpub\\mysite", 8080);

mySite.ServerAutoStart = true;

serverManager.CommitChanges();

}

}

Microsoft.Web.Administration makes IIS operations and configuration tasks a breeze, right inside your application in a .NET-compliant language of your choice. It also makes it easy to access runtime state information about the server, such as the running worker processes or currently executing requests.

The Microsoft.Web.Administration API serves as the foundation for accessing custom configuration inside custom .NET server modules and UI plug-ins for the IIS Manager tool. For an example of an end-to-end server package, including an image copyright handler to enhance the Web server and associated configuration and management components, take a look at iis.net/default.aspx?tabid=2&subtabid=25&i=1076.

In the Windows Server "Longhorn" timeframe, the IIS team is creating a unified extensibility model for adding custom management objects or extending existing ones, which will enable custom management functionality to be automatically exposed through the different management features including scripting and the Microsoft.Web.Administration API. While you cannot add or extend management objects in Windows Vista, you can use Microsoft.Web.Administration and other APIs to access and manipulate your custom configuration sections just like you can do with existing IIS configuration sections.

Building Web Server Features

IIS 7.0 enables you to mold the server into whatever you need it to be, letting you add or replace any feature in the server in order to provide the functionality you need. At the heart of this capability is a brand new Web server extensibility API on top of which all existing IIS 7.0 HTTP features are built. This API is public, which means that you can implement any of the features that ship with IIS 7.0. This is a first for IIS and a fundamental improvement over the previously limited ISAPI extensibility model.

The new extensibility API is a set of intuitive C++ classes that define the Web server object model and enable a module to provide request processing services on IIS. These classes are defined in the \inc\httpserv.h header file in the Windows Vista SDK.

Compared to ISAPI, these APIs are both more powerful and a lot easier to use. How is this possible? First, the new API features a type-safe, well-encapsulated object model. Development is much easier with the new server object model, which provides specialized interfaces for all of the basic server objects and tasks. This includes:

- Inspecting the request with the IHttpRequest class

- Manipulating the response with the IHttpResponse class

- Consuming useful utility functionality from the IHttpServer class

- Providing authentication with the IHttpUser class

- Accessing your module's custom configuration section with the configuration APIs

These classes expose a much greater set of the server's functionality than before (more than needed to build all of the features that ship with IIS), but are also much easier to use then the loosely typed ISAPI interfaces.

Developers will also benefit from improved patterns for memory and state management. Most of the IIS 7.0 server APIs use server-managed memory for data they return, instead of requiring you to allocate and manage the buffers as does ISAPI and the majority of existing Win32® APIs. In the past, this has been one of the most error-prone and tedious areas for ISAPI development. The new API also simplifies many complex request processing tasks, such as response buffering, authentication, and preparing response data for the client. A few months ago, I started a series of posts of my blog that explain the critical improvements and patterns in the new programming model. If you are considering C++ development for IIS, check it out at mvolo.com/blogs/serverside/archive/2006/10/07/10-reasons-why-server-development-is-better-with-IIS7.aspx.

IIS 7.0 also provides a fully integrated .NET Framework API for extending the server. What's more, this is the same API that's been provided by ASP.NET for building ASP.NET modules and handlers since the days of ASP.NET 1.0 on Windows 2000. But don't be fooled-while the familiar ASP.NET model allows existing ASP.NET modules and handlers to keep working on your IIS 7.0 server, it is actually far from the same old technology.

In IIS 7.0, ASP.NET comes in two versions: Classic mode and Integrated mode. Classic mode works exactly the same way as it did in previous versions of IIS. Integrated mode, the new platform default, uses a brand new engine to provide unparalleled integration with the IIS Web server. In Integrated mode, ASP.NET APIs can be used to develop IIS 7.0 modules that integrate directly with the Web server and are capable of providing virtually all of the services that are possible with the underlying C++ API.

This is basically the best of both worlds-the familiar interfaces and convenience of the .NET Framework and ASP.NET 2.0 application services like Membership and Role Management, plus the raw power to extend the server that was previously only available to C-based ISAPI components.

In addition to being able to write new ASP.NET modules that build on the specific benefits of the Integrated mode, many legacy ASP.NET modules can be made a lot more powerful just by changing a few configuration options in your web.config file.

ASP.NET Integration

With IIS 7.0, ASP.NET 2.0 becomes more than just an excellent framework for building dynamic applications. It becomes a platform for extending the IIS Web server, enabling ASP.NET components to become full members of the IIS request processing pipeline. Here's how it works.

In IIS releases up to version 6.0, ASP.NET connected to the Web server as a standalone application framework. It was responsible for processing the request extensions that were registered to it, typically .aspx and few others, and for those requests it offered powerful features like Forms Authentication, Response Output Caching, and others including services provided by custom ASP.NET modules. Because of this, only content types registered to ASP.NET could benefit from these services. Others, including ASP pages, PHP pages, images, and CGI applications could not. In addition, even for ASP.NET resources, certain Web server functions were not possible in ASP.NET because of runtime limitations. For example, it was not possible to inspect the set of outgoing HTTP response headers and modify them before being sent to the client.

When running in Integrated mode in IIS 7.0, ASP.NET modules run in the unified request processing pipeline side-by-side with native C++ IIS modules (see Figure 5). This means that existing ASP.NET services like Output Caching, URL Rewriting, and any others provided by your custom ASP.NET modules can now apply to any content type. Better runtime integration also enables ASP.NET modules to access previously unavailable server functionality, removing the need to write native IIS extensibility in most cases.

Figure 5 Integration with ASP.NET in IIS 6.0 and IIS 7.0 (Click the image for a larger view)

Finally, in Integrated mode, ASP.NET provides a small number of new APIs that expose additional functionality available due to tighter integration with IIS. These include the ability to review all of the response headers regardless of who generated the response and the ability to fully rewrite request execution to another URL, among other features.

An existing application can often take advantage of Integrated mode without requiring new ASP.NET modules that use Integrated mode-specific functionality. Simply by changing configuration, an application can do things such as using ASP.NET Forms Authentication and URL Authorization to protect your entire Web site with user security or using ASP.NET URL Mapping to rewrite URLs in your application. For an example of leveraging Intergrated mode to stop Web leeches from hot-linking to your site's images, check out the sample ASP.NET module to do this on mvolo.com/2006/11/10/stopping-hotlinking-with-iis-and-aspnet.aspx. This is a good example of how you can get more bang out of existing third-party ASP.NET modules by using them in Integrated mode.

For a detailed walkthrough on taking advantage of Integrated mode for existing applications, check out my article at iis.net/default.aspx?tabid=2&subtabid=25&i=1081&p=1.

Improved Security

IIS 7.0 builds upon the IIS 6.0 code base, which has a proven security track record thanks to careful coding practices and secure-by-default design principles. On top of this, IIS 7.0 introduces a few architectural changes to provide even stronger security and a number of features to help you build secure Web applications.

Reducing the attack surface area is one of the fundamental principles of designing and deploying secure systems. Taking the lockdown-by-default approach of IIS 6.0 to the next level, IIS 7.0 installs fewer features by default, yielding an even more locked down server. By further taking advantage of the modular nature of the server to remove all unused features, you can reduce your server's attack surface to a minimum-significantly reducing the risk of your server being compromised by an attacker.

If a vulnerability is found in any of the components not in use on your server, you do not need to take the server out of rotation in order to prevent exploits or patch the vulnerable component immediately. This can result in increased availability of your application and lower patch management costs.

In addition to the core security improvements, IIS 7.0 offers a number of security features you can use to further lock down and deploy secure apps on your server. IIS has always provided strong support for protecting application content with authentication. Now, with ASP.NET Integrated mode, you can use the popular ASP.NET security features such as Forms Authentication, Membership, and Login controls to provide a complete authentication and access control solution for your entire app. You can often do this in minutes and without writing a single line of code.

The new URL Authorization feature, inspired by the ASP.NET URL Authorization feature, can be used to configure declarative access control rules for your entire application. The access rules can be used to allow or deny access to URLs within your application based on user names and roles. URL Authorization integrates seamlessly with the ASP.NET 2.0 Membership and Role Management features and can be used effectively with the ASP.NET Forms Authentication and Login controls to quickly enable user security for your applications.

The new request filtering feature provides powerful lockdown functionality, part of which was available in the popular URLScan tool. You can use request filtering to further lock down your site by rejecting requests containing suspicious data, protecting sensitive resources, or enforcing aggressive request limits.

IIS 7.0 also introduces a number of changes meant to make deployment and management of security settings easier. The new IIS_IUSR anonymous account is built in, which means that it is not affected by password expirations and does not require password synchronization between machines. The new IIS_IUSRS group, which replaces the IIS_WPG group, is automatically injected into the worker process' identity at run time, which alleviates the need to manually add the worker process identity to the group when using custom accounts.

Because of the built-in IIS_USR account and IIS_USRS group, application content that specifies access control lists (ACLs) for the anonymous IIS account and group can be simply copied from one IIS server to another without any extra steps required to preserve the security settings. This dramatically simplifies application deployment across the development-test-production cycle.

The distributed configuration system discussed earlier allows application owners to manage the necessary Web server settings directly in their application without having administrative access to the server. Application administrators can specify the required configuration in the web.config files inside their application content when uploading their applications to the server or use the IIS Manager tool in order to configure their applications remotely.

The IIS Manager tool offers secure remote administration via firewall-friendly HTTPS connections. With the ability to authenticate application administrators as either Windows users or custom user accounts via the Membership service, the Administration tool allows remote application management without the owner having any Windows access on the server.

As a server administrator, you have complete control over what settings are configurable by the application via the flexible locking support in the configuration system. Likewise, you can control which IIS Manager tool features are available to the application administrators managing their applications remotely.

Improved Diagnostics

Between all the new features supported in Windows, IIS 7.0, and your Web application, a Web server is a very a complicated system that often takes a lot of effort to troubleshoot. IIS 7.0 introduces a number of new features to help you monitor the operation of the server and debug application problems.

First, IIS 7.0 lets you peer into the real-time state of the server. This feature, called the Runtime State and Control API, or RSCA (pronounced "reeska"), exposes the active state of sites and application pools, running worker processes, and even allows you to view the currently executing requests on the server! It also enables you to control the state of the server, such as starting and stopping sites, or recycling application pools. In Windows Vista, you can access this information in the IIS Manager, via the appcmd.exe command-line tool, or programmatically using the Microsoft.Web.Administration API.

For example, you can view currently executing requests and the server stages where they are stuck. This lets you quickly resolve hanging request problems and track down which script is spinning your CPU (see Figure 6).

The RSCA capabilities come in very handy when you are investigating server issues or tuning server performance, both for being able to quickly see what's going on in the system and to control the server when performing troubleshooting. I often end up using appcmd.exe to view the state of application pools, examine worker processes, and start or stop offending applications pools to quickly zero in on the problem when investigating bugs in the office.

Figure 6 Tracking Down Stuck Scripts in IIS Manager (Click the image for a larger view)

When errors occur in your Web application, they can be due to incorrect server configuration, application errors, or various environmental factors. The status code and standard error messages provide little clue to what went wrong and can make troubleshooting the server a nightmare. IIS 7.0 provides detailed error information about most errors, telling you exactly what went wrong, why, and how to fix it (see Figure 7).

Figure 7 Error Details Indicate the Problem and the Solution (Click the image for a larger view)

The detailed errors follow a security scheme similar to the ASP.NET detailed errors. By default, you will only get the detailed information when browsing the Web site from the local machine. As before, you can also configure custom error pages for different error codes or redirect to a custom URL. The detailed error pages are also now localized, offering the error description in the preferred language of the client, if the language pack for the corresponding language is installed.

Diagnosing Errors without Debugging

What if the error condition you are experiencing is unknown or caused by a complex interplay of multiple Web server components? Not to worry-IIS 7.0 provides a comprehensive tracing mechanism that generates a verbose paper trail for each request, which can be used to track down the problem quickly.

IIS 7.0 adds more informational events on top of the Event Tracing for Windows (ETW) events added to IIS 6.0 in Windows Server 2003 Service Pack 1 (SP1). These events contain useful information about each stage of server processing you can examine to back-trace through request execution and pinpoint the place where things went wrong. These events can be routed to the Windows tracing infrastructure, which allows multiple Windows components including ASP.NET and SQL Server™ to link their tracing information into a single, logical execution trace for the request.

They can also be routed to the new Failed Request Tracing feature (also called FREB), which saves the trace logs into XML log files that can be viewed with the provided XSLT stylesheet (see Figure 8) or consumed programmatically.

Figure 8 Viewing XML Log Files (Click the image for a larger view)

The coolest thing about Failed Request Tracing is that it can be left enabled on the server. It allows you to capture trace logs for requests that have encountered a configurable failure condition automatically while avoiding the performance penalty of saving the trace logs for requests that have completed successfully. For example, you can turn it on for requests that result in server errors or take more than a certain amount of time to complete.

With Failed Request Tracing, you can always capture the valuable tracing information when errors occur, even if they are intermittent or hard to reproduce. This can help diagnose and fix difficult problems that previously required hardcore debugging.

The underlying tracing infrastructure is exposed to IIS modules via the server extensibility model, allowing all server components, whether they ship with IIS or are developed by third parties, to emit detailed tracing information during request processing. IIS 7.0 tracing integrates with ASP.NET tracing via the System.Diagnostics API and the ASP.NET page trace, allowing managed modules to take advantage of the unified tracing model. To take things even further, you can write your own tracing modules that provide new ways to process and output the tracing information. For example, you can be the first one to write a module to save IIS tracing information to SQL Server or to a text file.

Improved Performance

While Windows Vista is a client operating system release and not intended for high-throughput production deployment (IIS on Windows Vista is limited to 10 concurrent requests at a time), it already showcases some of the major architectural improvements aimed at significantly increasing Web application performance. Coupled with the extensive performance work we are doing in the Windows Server "Longhorn" timeframe, these improvements will help IIS 7.0 elevate the performance of your server.

The first one is, of course, componentization. The modular nature of the server allows administrators to remove unneeded server features, saving memory and CPU usage during request processing. This can result in significant improvements in throughput and capacity on the machine. The ability to enable features in a granular way (turning them on and off for each application on your server) further enables applications to achieve performance gains in the case where only certain parts of your site require a particular feature.

The other notable performance feature in IIS 7.0 is the new IIS Output Cache. This feature provides support for re-using responses to expensive dynamic pages on the server, alleviating the need to perform the expensive rendering and database transactions in order to return the response to the client. The IIS Output Cache is a faster alternative to the existing rich output caching feature in ASP.NET, supporting a smaller set of caching functionality but providing enough flexibility to cache dynamic content in a performance-enhancing way.

By output caching your dynamic content, whether it is ASP.NET pages, PHP scripts, or CGI applications, you can frequently get a 5-10 times performance boost with much less load on your disk and database.

Backward Compatibility

IIS 7.0 should be able to run most existing applications without modification. This was a major feat given the extent of the architectural changes that were needed to support the innovation in this release. The configuration system has undergone the most changes, moving away from a centralized loosely typed configuration store to a delegated XML configuration file hierarchy. Both the structure and the storage of the configuration information are completely different from the IIS 6.0 metabase and do not enable access via the legacy configuration APIs.

IIS 7.0 solves this problem by providing an emulation layer for the metabase, which does on-the-fly translations between the underlying data in the configuration system and the interfaces exposed by the metabase ABO APIs. This, in turn, allows code written for the metabase to work correctly when accessing it via ABO, or higher-level WMI or ADSI scripts. Be sure to install the compatibility setup components to get this functionality, though.

While IIS 7.0 provides a new extensibility model for developing IIS components, ISAPI components are still supported. If you install the ISAPI Extensions and ISAPI Filters setup components, you will be able to run your extensions and filters as before. However, if you are developing new components, you should make sure to use the new extensibility model for a much more powerful and improved development experience.

A small percentage of ASP.NET applications that have runtime incompatibilities with the Integrated mode may have to be moved to an application pool running in Classic mode. In this case, you can run multiple applications in both modes side-by-side on the same server by placing them in separate application pools. For a complete list of ASP.NET breaking changes and general ASP.NET compatibility information on IIS 7.0, be sure to check out the ASP.NET compatibility whitepaper at iis.net/default.aspx?tabid=2&subtabid=25&i=1223.

Conclusion

The IIS 7.0 release in Windows Vista aims to provide the best architectural foundation for the next-generation Web application platform, focusing on the right core architecture, extensibility, and management platform for the Web server. Windows Vista gives you the ability to develop and test your applications on the same server platform as will be used to deploy it when the server version of Windows Vista becomes available.

As IIS 7.0 is released in Windows Vista, the Web Platform and Tools team's focus is shifting to getting the Web server ready for production environments and improving stability and performance for production scenarios. However, the core development and management features that ship in Windows Vista will remain the same-and when the server version of IIS 7.0 is complete, it's improvements are expected to become available for Windows Vista via a service pack. At that point, your client and server machines will once again be running the same exact version of IIS so you can continue to develop and test Web applications on your desktop running Window Vista.

To get started with IIS 7.0, be sure to check out the large number of exceedingly useful resources available on the Web, starting with the iis.net Web site, which is the new home of the IIS team. Our new site contains a portal for all things IIS 7.0, including detailed articles and walkthroughs for all IIS 7.0 features. Be sure to use the forums to ask questions and discuss problems with the IIS team and the IIS community.

You can also find in-depth coverage of IIS 7.0 and inside information on my blog, www.mvolo.com. Be sure to stop by. And do let me know what your favorite IIS 7.0 topics are and I will do my best to cover them in the blog.

Mike Volodarsky is a technical Program Manager on the Web Platform and Tools Team at Microsoft. He owns the core server infrastructure for ASP.NET and IIS. Michael is now focusing on improving the Web application platform in the next-generation Web server, IIS 7.0.

.NET Security

Support Certificates In Your Applications With The .NET Framework 2.0

Dominick Baier

This article discusses:

- The Windows Certificate Store

- Certificate classes in .NET

- Validation, SSL, Web services, and code signing

- Signing and encrypting data

This article uses the following technologies:

.NET Framework 2.0

Code download available at: Certificates2007_03.exe (160 KB)

Browse the Code Online

Contents

How to Get a Certificate

The Windows Certificate Store

Working with Certificates

Accessing Certificates

Display Certificate Details and the Certificate Picker

Validating Certificates

SSL Support

Web Service Security

Security Policy and Code Signing

ClickOnce Manifests

Signing and Encrypting Data

Decrypting Data and Verifying Signatures

Putting It All Together

Certificates are used in many places across the Microsoft® .NET Framework, from secure communication to code signing to security policies. The .NET Framework 2.0 introduced revamped support for certificates and it added a completely new namespace for standards-compliant cryptographic operations with certificates. In this article, I will discuss the background for certificates and the Windows® Certificate Store. I'll also show you how to work with the certificate APIs and how they are used by the Framework to implement security features.

A "certificate" is really an ASN.1 (Abstract Syntax Notation One) encoded file that contains a public key and additional information about that key and its owner. In addition, a certificate has a validity period and is signed with another key (the so-called issuer) which is used to provide an authenticity guarantee of those attributes and, most importantly, the public key itself. You can think of ASN.1 as a sort of binary XML. Like XML, it also has encoding rules, strong types, and tags; however, these are binary values that often don't correspond to any printable character.

For such a file to be interchangeable between systems, a standard format is needed. This is X.509 (currently version 3), which is described in RFC 3280 (tools.ietf.org/html/rfc3280). X.509 doesn't dictate the type of key embedded in the certificate, but the RSA algorithm is currently the most popular asymmetric cryptographic algorithm in use.

I'll start with a little history. The name RSA is an acronym for the surnames of three inventors of this algorithm: Ron Rivest, Adi Shamir, and Len Adleman. They formed a company, RSA Security, which published several standard documents called Public Key Cryptography Standards (PKCS). These documents describe several aspects of cryptography.

One of the most popular of these documents, PKCS #7, defines a binary format for signed and encrypted data called the Cryptographic Message Syntax (CMS). CMS is now used in many popular security protocols, including secure sockets layer (SSL) and Secure Multipurpose Internet Mail Extensions (S/MIME). Since it is a standard, it is also the format of choice for when applications need to exchange signed and encrypted data between several parties. The PKCS documents are available on the RSA Laboratories Web site (www.rsasecurity.com/rsalabs/node.asp?id=2124).

How to Get a Certificate

There are several ways to acquire a certificate. When files are being exchanged, certificates will usually appear in one of two formats. Files with the .cer extension are signed ASN.1 files in the X.509v3 format. They contain a public key and the extra information I mentioned earlier. This is what you give to business partners or friends so they can use the public key to encrypt data for you.

You may also encounter files with a .pfx (Personal Information Exchange) extension. A .pfx file contains a certificate and the corresponding private key (the format is described in the PKCS #12 standard). Such files are highly sensitive and are typically used to import key pairs on a server or for backup purposes. When exporting key pairs, Windows offers to encrypt the .pfx file with a password; you have to provide this password again when importing the key pair.

You can also generate your own certificates. How you generate them usually depends on how they will be used. For normal Internet scenarios, where you don't know who your peers are, you typically request a certificate from a commercial certification authority (CA). This approach has the advantage that these known CAs are already trusted by Windows and any other OS (and browser) that supports certificates and SSL. As a result, you don't have to do a CA key exchange.

For B2B and intranet scenarios, you can use an internal CA. Certificate Services are included in Windows 2000 and Windows Server® 2003. Combined with Active Directory®, this functionality lets you easily distribute certificates across an organization. (I will show you how to request certificates from a private CA in a moment.)

Sometimes during development you might be in a situation where the approaches just described don't work. For instance, if you need a certificate quickly for testing purposes, you can use makecert.exe. Included in the .NET Framework SDK, this tool generates certificates and key pairs. There is a similar tool, called selfssl.exe, in the IIS Resource Kit; it is specialized for creating SSL key pairs, and it can also configure IIS with such a key pair in a single step.

The Windows Certificate Store

Certificates and their corresponding private keys can be stored on a variety of devices, such as hard disks, smartcards, and USB tokens. Windows provides an abstraction layer, called the certificate store, to unify how you access certificates regardless of where they are stored. As long as the hardware device has a Windows-supported cryptographic service provider (CSP) you can access the data stored on it using the Certificate Store API.

The certificate store is buried deep in the user profile. This allows use of ACLs on the keys for a specific account. Every store is partitioned into containers. For instance, there's a container called Personal where you store your own certificates (the ones that have an associated private key). The Trusted Root Certification Authorities container holds the certificates of all CAs that you trust. The Other People container holds the certificates of people you securely communicate with. And so on. The easiest way to get to your certificate store is to run certmgr.msc.

There is also a machine-wide store, which is used by the Windows machine accounts (NETWORK, LOCAL SERVICE, and LOCAL SYSTEM) or if you want to share certificates or keys across accounts. ASP.NET applications always use the machine store; for desktop applications, you typically install certificates in the user store.

Only administrators can manage the machine and service account stores. For this purpose, you have to start the Microsoft Management Console (mmc.exe) and add the Certificates snap-in. There you can choose the store to administer. Figure 1 shows a screenshot of the MMC snap-in.

Figure 1 Certificates MMC Snap-In (Click the image for a larger view)

Besides allowing you to import, export, and search for certificates, the snap-in also lets you request certificates from an internal enterprise CA. Just right-click the personal container and select All Tasks | Request Certificate. The local machine then generates an RSA key pair and sends the public key portion to the CA for signing. Windows adds the signed certificate to the certificate store and the corresponding private key to a key container. The certificate gets linked to the key container via a storage attribute.

The private key containers are strongly ACL'd for either the corresponding account or LOCAL SYSTEM. This is a problem when you want to access keys stored in the machine profile from ASP.NET or from other user accounts. I wrote a tool that you can use to modify the ACLs of the container file (it's available at www.leastprivilege.com/HowToGetToThePrivateKeyFileFromACertificate.aspx).

Commercial and Windows CAs also have Web interfaces for requesting certificates. For these scenarios, typically an ActiveX® control in Internet Explorer® generates the keys and imports them into the store of the current user. As a general rule, when you want to make a certificate accessible to a user or service, you have two choices: either import it into his store or request it while being logged on as that user.

Working with Certificates

Certificates are used in various places in the .NET Framework, and at some level all of this functionality relies on the X509Certificate class from the System.Security.X509Certificates namespace. If you take a closer look, you'll also find a certificate class ending with a 2. This is because the .NET Framework 1.x had a representation of X.509 certificates called X509Certificate. This class had limited functionality and no support for cryptographic operations. In version 2.0, a new class was added called X509Certificate2. This is derived from X509Certificate and adds many capabilities. You can convert back and forth between them as necessary, but whenever possible you should use the latest version.

Accessing Certificates

You can retrieve certificates from the file system directly. However, it is better to retrieve them from the certificate store. To create an X509Certificate2 instance from a .cer file, simply pass the file name to the constructor:

X509Certificate2 cert1 = new X509Certificate2("alice.cer");

You can also load certificates from .pfx files. However, as I mentioned earlier, .pfx files can be password protected, and you should supply this password as a SecureString. SecureString encrypts the password internally and tries to minimize exposure of it in Memory, page files, and crash dumps. For this reason, you can only add a single (value type) char at a time to the string. The code in Figure 2, which disables the console echo and returns a SecureString, is useful if you want to ask your users for a password from the console.

Figure 2 Requesting a Password from the Console

private SecureString GetSecureStringFromConsole()

{

SecureString password = new SecureString();

Console.Write("Enter Password: ");

while (true)

{

ConsoleKeyInfo cki = Console.ReadKey(true);

if (cki.Key == ConsoleKey.Enter) break;

else if (cki.Key == ConsoleKey.Escape)

{

password.Dispose();

return null;

}

else if (cki.Key == ConsoleKey.Backspace)

{

if (password.Length != 0)

password.RemoveAt(password.Length - 1);

}

else password.AppendChar(cki.KeyChar);

}

return password;

}

In the article "Credential Management with the .NET Framework 2.0" (available at msdn.microsoft.com/library/en-us/dnnetsec/html/credmgmt.asp), Kenny Kerr included code to convert the result of the usual Windows credentials dialog into a SecureString. Regardless of how you obtain it, the SecureString can then be passed to the X509Certificate2 constructor to load the .pfx file, like so:

X509Certificate2 cert2 = new X509Certificate2("alice.pfx", password);

To access the Windows certificate store, you use the X509Store class. In its constructor you provide the store location (current user or machine) and the store name. You can use either a string or the StoreName enumeration to specify the container you want to open. Be aware that the internal names don't always match the names you find in the MMC snap-in. The Personal container maps to the name My, whereas Other People becomes AddressBook.

Once you have a valid instance of X509Store, you can search for, retrieve, delete, and add certificates. With the exception of deployment scenarios, you will probably use the search functionality most often. You can search for certificates on a variety of criteria, including subject name, serial number, thumbprint, issuer, and validity period. If you programmatically retrieve certificates in your applications from the store, you should use a unique property-the subject key identifier, for instance. The thumbprint is also unique, but keep in mind that this is a SHA-1 hash value of the certificate and will change if, for example, the certificate gets renewed. The code in Figure 3 shows a generic way to search for certificates.

Figure 3 Searching for Certificates

static void Main(string[] args)

{

// search for the subject key id

X509Certificate2 cert = FindCertificate(

StoreLocation.CurrentUser, StoreName.My,

X509FindType.FindBySubjectKeyIdentifier,

"21f2bf447298e83056a69eb02ebe9085ed97f10a");

}

static X509Certificate2 FindCertificate(

StoreLocation location, StoreName name,

X509FindType findType, string findValue)

{

X509Store store = new X509Store(name, location);

try

{

// create and open store for read-only access

store.Open(OpenFlags.ReadOnly);

// search store

X509Certificate2Collection col = store.Certificates.Find(

findType, findValue, true);

// return first certificate found

return col[0];

}

// always close the store

finally { store.Close(); }

}

Once you have an instance of X509 Certificate2, you can inspect the various properties of the certificate (such as the subject name, expiration dates, issuer, and the friendly name). The HasPrivateKey property tells you if there is an associated private key. The PrivateKey and PublicKey properties return the corresponding key as an RSACryptoServiceProvider instance.

To import a certificate, you call the Add method on the X509Store instance. When you specify a store name that doesn't exist in the constructor of the store, a new container will be created. Here's how you would import a certificate in a file named alice.cer into a new container called Test:

static void ImportCert()

{

X509Certificate2 cert = new X509Certificate2("alice.cer");

X509Store store = new X509Store("Test", StoreLocation.CurrentUser);

try

{

store.Open(OpenFlags.ReadWrite);

store.Add(cert);

}

finally { store.Close(); }

}

Display Certificate Details and the Certificate Picker

Windows offers two standard dialogs for working with certificates: one for showing certificate details (properties and certification path) and one for letting users pick a certificate from a list. You can access these dialogs from the two static methods of the X509Certificate2UI class: SelectFromCollection and DisplayCertificate.

To show a list of certificates you have to populate an X509Certificate2Collection and pass that to SelectFromCollection. It is very common to let a user choose from one of his personal certificates in the store. For this, you simply pass in the Certificates property of an opened X509Store. You can also control the dialog caption, a message, and whether multiple selections are allowed. The DisplayCertificate method shows the same dialog you see when double-clicking on a .cer file in Windows Explorer. Figure 4 shows the dialog used for picking a certificate and Figure 5 provides the corresponding code.

Figure 5 Code for Choosing a Certificate

private static X509Certificate2 PickCertificate(

StoreLocation location, StoreName name)

{

X509Store store = new X509Store(name, location);

try

{

store.Open(OpenFlags.ReadOnly);

// pick a certificate from the store

X509Certificate2 cert =

X509Certificate2UI.SelectFromCollection(

store.Certificates, "Caption",

"Message", X509SelectionFlag.SingleSelection)[0];

// show certificate details dialog

X509Certificate2UI.DisplayCertificate(cert);

return cert;

}

finally { store.Close(); }

}

Figure 4 Dialog for Choosing Certificates (Click the image for a larger view)

Validating Certificates

There are a few criteria to consider when validating a certificate, especially the issuing party (generally, you only trust certificates that were issued by a CA in your trusted CA list) and its current validity (certificates can become invalid, such as when they expire or are revoked by the issuing CA). You can use the X509Chain class to check these various properties. Using this class, you can specify a policy for validity checking-for example, you can demand a trusted root CA or specify whether to check online or local revocation lists. If you need to check certificates that were used to sign data, it is important to check whether the certificate was valid when the signature was computed-for this, X509Chain allows you to change the verification time.

After constructing a policy, you call the Build method to get information about the validation outcome on the ChainStatus property. If there are multiple validation errors, you can iterate over the ChainElement collection to get more details. Figure 6 shows how to perform a strict validation of a certificate and its issuer against offline and online revocation lists.

Figure 6 Strict Certificate Validation

static void ValidateCert(X509Certificate2 cert)

{

X509Chain chain = new X509Chain();

// check entire chain for revocation

chain.ChainPolicy.RevocationFlag = X509RevocationFlag.EntireChain;

// check online and offline revocation lists

chain.ChainPolicy.RevocationMode =

X509RevocationMode.Online | X509RevocationMode.Offline;

// timeout for online revocation list

chain.ChainPolicy.UrlRetrievalTimeout = new TimeSpan(0, 0, 30);

// no exceptions, check all properties

chain.ChainPolicy.VerificationFlags = X509VerificationFlags.NoFlag;

// modify time of verification

//chain.ChainPolicy.VerificationTime = new DateTime(1999, 1, 1);

chain.Build(cert);

if (chain.ChainStatus.Length != 0)

Console.WriteLine(chain.ChainStatus[0].Status);

}

SSL Support

The SSL authentication protocol relies on certificates. Support for SSL in the .NET Framework consists of two parts. The special (but most widely used) case of SSL over HTTP is implemented by the HttpWebRequest class (this is also ultimately used for Web service client proxies). To enable SSL, you don't have to do anything special besides specify a URL that uses the https: protocol.

When connecting to an SSL secured endpoint, the server certificate is validated on the client. If validation fails, by default the connection is immediately closed. You can override this behavior by providing a callback to a class called ServicePointManager. Whenever the HTTP client stack does certificate validation, it first checks if a callback is provided-if that's the case, it executes your code. To hook up the callback, you have to provide a delegate of type RemoteCertificateValidationCallback:

// override default certificate policy

// (for example, for testing purposes)

ServicePointManager.ServerCertificateValidationCallback =

new RemoteCertificateValidationCallback(VerifyServerCertificate);

In your callback, you get the server certificate, an error code, and a chain object passed in. You can then do your own check and return true or false. It can be helpful to turn off one of these checks if, for instance, your certificate has expired during development or testing. On the other hand, this also allows you to implement stricter validation policies than provided by default. Figure 7 provides a sample validation callback.

Figure 7 Validation Callback

private bool VerifyServerCertificate(

object sender, X509Certificate certificate,

X509Chain chain, SslPolicyErrors sslPolicyErrors)

{

if (sslPolicyErrors == SslPolicyErrors.None) return true;

foreach (X509ChainStatus s in chain.ChainStatus)

{

// allows expired certificates

if (string.Equals(s.Status.ToString(), "NotTimeValid",

StringComparison.OrdinalIgnoreCase))

return true;

}

return false;

}

SSL also supports client authentication using a certificate. If the Web site or service you want to access mandates a client certificate, both the Web service client proxy and HttpWebRequest provide a ClientCertificates property of type X509Certicate:

proxy.Url =

"https://server/app/service.asmx";

proxy.ClientCertificates.Add(

PickCertificate(...));

In addition, the .NET Framework 2.0 introduces a new class called SslStream. This lets you layer SSL on top of any stream, not just HTTP, which makes it possible to SSL-enable a custom socket based protocol. SslStream makes use of the standard .NET certificates support in multiple ways, for example, using the validation callback mechanism I discussed:

public SslStream(Stream innerStream, bool leaveInnerStreamOpen,

RemoteCertificateValidationCallback ValidationCallback) {...}

And to start an SSL authentication with SslStream, you pass an X509Certificate to its AuthenticateAsServer method:

ssl.AuthenticateAsServer(PickCertificate(...));

Web Service Security

The WS-Security standard specifies client and server authentication and secure communication using certificates. Toolkits like the Web Services Enhancements (WSE) for the .NET Framework and technologies like the Windows Communication Foundation fully support this. Again, this boils down to supplying a certificate either in code or through configuration. The following snippet shows how to add a client certificate to a Web service proxy using WSE3:

X509SecurityToken token = new X509SecurityToken(PickCertificate(...));

proxy.RequestSoapContext.Security.Tokens.Add(token);

With Windows Communication Foundation, you typically provide a reference to a certificate store in a configuration file (see Figure 8). As you can see, all configuration attributes map directly to the enums used earlier in code.

Figure 8 Providing Certificate Reference in WCF

<system.serviceModel>

<behaviors>

<serviceBehaviors>

<behavior name="ServiceBehavior">

<serviceCredentials>

<serviceCertificate storeLocation="LocalMachine"

storeName="My" x509FindType="FindBySubjectKeyIdentifier"

findValue="1a7b..." />

</serviceCredentials>

</behavior>

</serviceBehaviors>

</behaviors>

</system.serviceModel>

Security Policy and Code Signing

Certificates are also used in Authenticode® code signing. By signing a binary, you can add information about the publisher and make sure the signed file can be reliably validated after it has been signed. You can use the signtool.exe tool from the .NET Framework SDK to sign .exe and .dll files. Afterwards, you can verify the signature and view the certificate using the properties dialog in Windows Explorer. Note that if both Authenticode and strong name signatures are going to be used, the strong name signature needs to be applied first. Additionally, Authenticode signed assemblies can experience delays at load time, which translates to a longer application startup time if it's the entry point executable that's been signed.

Signed files can also be used for security policies. Using software restriction policies, you can restrict execution of unmanaged executables based on signatures or the absence of signatures (see microsoft.com/technet/prodtechnol/winxppro/maintain/rstrplcy.mspx). And the .NET Framework code access security (CAS) policy supports code groups based on a publisher certificate.

To create a CAS policy, you use mscorcfg.msc to create a new code group based on a publisher membership condition. You can then assign a permission set to all applications signed by that publisher (see Figure 9).

Figure 9 Assigning Permissions to a Publisher (Click the image for a larger view)

ClickOnce Manifests

Another technology that uses certificates for publisher information is ClickOnce. When you publish a ClickOnce application, you have to sign the deployment and application manifest. This again adds publisher information to the application and ensures that sensitive information in the manifests (such as the security policy and application dependencies) cannot be modified without invalidating the signature. ClickOnce makes the publisher information available to clients during the installation so they can make intelligent decisions about the trustworthiness of the application. Depending on the certificate (and its validation outcome) the ClickOnce installer also uses different visual cues. Figure 10 shows the Visual Studio® manifest signing dialog.

Figure 10 Visutal Studio Manifest Signing Dialog (Click the image for a larger view)

Signing and Encrypting Data

So far, I've focused on the fundamental certificate-related APIs and how other technologies make use of them. Now I want to discuss cryptographic operations, like encrypting and signing data with certificates, and the new PKCS #7 implementation found in the .NET Framework 2.0.

Protecting data is always a two-step process. First you sign the data to make it tamper-proof. Then you encrypt the data to protect it from disclosure. Before you can perform any cryptographic operation with the PKCS #7 classes, however, you first have to wrap the data in a ContentInfo object, representing a CMS data structure. From there you can transform the data into signed or encrypted data, represented respectively by the SignedCms and EnvelopedCms classes.

Technically, a digital signature is the hash of the data that's then encrypted with your private key. This means you need a certificate with an associated private key or a .pfx file. Based on such a certificate, you can create a CmsSigner object, which represents the signer of the data. The SignedCms class in turn computes the signature and outputs a PKCS #7, CMS-compliant byte array. Figure 11 shows the corresponding code. The encoded byte array contains your data, the signature, and the certificate used to sign the data.

Figure 11 Outputting CMS-Compliant Byte Array

byte[] Sign(byte[] data, X509Certificate2 signingCert)

{

// create ContentInfo

ContentInfo content = new ContentInfo(data);

// SignedCms represents signed data

SignedCms signedMessage = new SignedCms(content);

// create a signer

CmsSigner signer = new CmsSigner(signingCert);

// sign the data

signedMessage.ComputeSignature(signer);

// create PKCS #7 byte array

byte[] signedBytes = signedMessage.Encode();

// return signed data

return signedBytes;

}

This might not be critical if you are signing large amounts of data, but if the amount of data is small, this adds some overhead. For example, signing a 10 byte array with a 2KB public key results in a roughly 2,400 byte array. Keep that in mind if you want to store the signed data in, say, a database. An alternative approach is to use a so-called detached signature. This lets you remove your data from the signature and store it separately. You could, for example, first combine multiple small pieces of data and sign them altogether. To create a detached signature you have to pass an additional true to the SignedCms constructor, as shown in Figure 12.

Figure 12 Creating Detached Signature

byte[] SignDetached(byte[] data, X509Certificate2 signingCert)

{

// create ContentInfo

ContentInfo content = new ContentInfo(data);

// pass true to the constructor to indicate

// we want to sign detached

SignedCms signedMessage = new SignedCms(content, true);

// these steps are the same

CmsSigner signer = new CmsSigner(signingCert);

signedMessage.ComputeSignature(signer);

byte[] signedBytes = signedMessage.Encode();

// return only the signature (not the data)

return signedBytes;

}

Once you've signed the data, you can encrypt it. You'll need the public keys of the recipients that should be able to decrypt the data. You usually get these from your Other People store (or from a .cer file if you don't want to use the certificate store). This time the EnvelopedCms class does all the heavy lifting. You specify the public keys used for encryption in a CmsRecipientCollection, which you pass into the Encrypt method. As with SignedCms, here the Encode method creates the PKCS #7, CMS-compliant byte array (see Figure 13).

Figure 13 Encode Method

byte[] Encrypt(byte[] data, X509Certificate2 encryptingCert)

{

// create ContentInfo

ContentInfo plainContent = new ContentInfo(data);

// EnvelopedCms represents encrypted data

EnvelopedCms encryptedData = new EnvelopedCms(plainContent);

// add a recipient

CmsRecipient recipient = new CmsRecipient(encryptingCert);

// encrypt data with public key of recipient

encryptedData.Encrypt(recipient);

// create PKCS #7 byte array

byte[] encryptedBytes = encryptedMessage.Encode();

// return encrypted data

return encryptedBytes;

}

Internally, EnvelopedCms generates a random session key with which the data is symmetrically encrypted. Afterwards, the session key is encrypted with the public key of each recipient. Thus, you don't need a separate encrypted version of the data for each of your recipients. In addition, some extra information is embedded, allowing the recipient to find the matching private key for decryption in his certificate store.

Decrypting Data and Verifying Signatures

At the receiving end, the whole process is reversed. That means you first decrypt the data and then validate the signature and the signing certificate. In code, you first have to call the Decode methods of the SignedCms and EnvelopedCms classes to deserialize the CMS byte array back to an object representation. Then you can call Decrypt and CheckSignature, respectively.

The process looks in the encrypted package to see if the session key can be decrypted by searching for a corresponding private key in the certificate store. Afterwards, the decrypted session key is used to decrypt the actual data. You can also supply a list of additional certificates that should be taken into consideration during decryption in case the private key is not stored in the certificate store:

static byte[] Decrypt(byte[] data)

{

// create EnvelopedCms

EnvelopedCms encryptedMessage = new EnvelopedCms();

// deserialize PKCS#7 byte array

encryptedMessage.Decode(data);

// decryt data

encryptedMessage.Decrypt();

// return plain text data

return encryptedMessage.ContentInfo.Content;

}

Verifying the data is a two-step process. You first make sure the signature is valid, which means the data has not been tampered with. You then check the signing certificate. The CheckSignature method of the SignedCms class lets you do both steps at once. In this case, the certificate is validated against the default system policy. If you want more control over that process, you can do your own checking using an X509Chain object and code like that in Figure 6. And Figure 14 shows the code used to check and remove a signature, while Figure 15 provides the code used for detached signature validation.

Figure 15 Detached Signature Validation

static bool VerifyDetached(byte[] data, byte[] signature)

{

ContentInfo content = new ContentInfo(data);

// pass true for detached

SignedCms signedMessage = new SignedCms(content, true);

// deserialize signature

signedMessage.Decode(signature);

try

{

// check if signature matches data

// the certificate is also checked

signedMessage.CheckSignature(false);

return true;

}

catch { return false; }

}

Figure 14 Verify and Remove a Signature

byte[] VerifyAndRemoveSignature(byte[] data)

{

// create SignedCms

SignedCms signedMessage = new SignedCms();

// deserialize PKCS #7 byte array

signedMessage.Decode(data);

// check signature

// false checks signature and certificate

// true only checks signature

signedMessage.CheckSignature(false);

// access signature certificates (if needed)

foreach (SignerInfo signer in signedMessage.SignerInfos)

{

Console.WriteLine("Subject: {0}",

signer.Certificate.Subject);

}

// return plain data without signature

return signedMessage.ContentInfo.Content;

}

Putting It All Together

When working with security policies or communication protocols, you will encounter certificates in all kinds of situations. In this article I explained the basic APIs used to retrieve and search certificates and showed you how to use them for encryption and digital signatures. In addition, I provided some examples of higher level application services that require you to understand the Windows certificate store and the relationship between public and private keys.

The source code for this article, available for download from the MSDN®Magazine Web site, includes a small Windows Forms application that supports signing and encrypting files. It uses many of the techniques I've discussed in this article, like selecting certificates from different stores and protecting/verifying data using encryption and signatures.

Dominick Baier is an independent security consultant in Germany. He helps companies with secure design and architecture, content development, penetration testing, and code auditing. He is also the security curriculum lead at DevelopMentor, a Developer Security MVP, and author of Developing More-Secure Microsoft ASP.NET 2.0 Applications (Microsoft Press, 2006). His blog is located at www.leastprivilege.com.

Digital Media

Add Video To Controls And 3D Surfaces With WPF

Lee Brimelow

This article discusses:

- Windows Presentation Foundation digital media basics

- Using MediaElement and MediaPlayer

- Video embedded in WPF controls

- Mapping video onto 3D surfaces

This article uses the following technologies:

.NET Framework 3.0

Contents

Technical Overview

Using the MediaElement Control

Using the MediaPlayer Class

Embedding Video inside WPF Controls

The Wet Floor Technique

Mapping Video onto 3D Surfaces

Rendering Bitmaps from Video Frames

Additional Audio Capabilities in WPF

Summary

When the buzz surrounding Windows Presentation Foundation (WPF) began circulating throughout the Adobe Flash community, my initial reaction was one of skepticism. That Microsoft had introduced a competing technology led many Flash developers, including myself, to lash out against the company in support of our beloved platform. Then I received an e-mail message saying that Microsoft was sending a technical evangelist to the frog design studio in San Francisco to demo Windows® Presentation Foundation, and I saw it as a perfect opportunity to show everyone why Flash was superior in every way.

Halfway through Karsten Januszewski's presentation, he showed off the now-famous North Face demo (channel9.msdn.com/Showpost.aspx?postid=116327), which features a 3D carousel in which curved 3D mesh objects are mapped with high-quality Windows Media® Video (WMV) clips. The demo was created jointly by Fluid, an interactive studio also located in San Francisco, and members of the Microsoft Windows Presentation Foundation team. My prejudices were obliterated as reality set in. This type of presentation would be difficult or impossible to accomplish with Flash-or any other platform for that matter.

Not long after that session, I swallowed my pride and downloaded the Microsoft® .NET Framework 3.0 runtime, thus beginning my career as a Windows Presentation Foundation interactive designer. Since that time, the whole Flash versus Windows Presentation Foundation debate has dissipated due to the fact that there aren't many areas in which the two technologies actually compete. The debate may start raging again when Microsoft releases its latest solution for delivering cross-platform Web pages that include graphics, video, animation, and audio. This upcoming release is codenamed "WPF/E" and a preview is available on the "WPF/E" developer center at msdn2.microsoft.com/bb187358.aspx.

The ability to map video to the 3D surfaces certainly provided the necessary eye candy to grab the attention of interactive designers. But this feature only scratches the surface of what's possible when integrating audio and video into Windows Presentation Foundation. This article will get you up to speed with Windows Presentation Foundation media integration and give you the knowledge you need to add media files and that optional visual interest to your own applications.

Technical Overview

The two most common ways of adding digital media files to Win32® applications today are by using the DirectShow® API or by embedding the Windows Media Player ActiveX® control. Using DirectShow gives developers ultimate control over media integration, but is much more complex than simply embedding the Windows Media Player ActiveX control.

Using DirectShow is still a viable option for Windows Presentation Foundation applications that need fine-grained control over media display and management. You might also implement DirectShow in a Windows Presentation Foundation application when an existing Win32 app is being ported to Windows Presentation Foundation and the .NET Framework 3.0.

Playing media using the Windows Media Player ActiveX control is still an option in Windows Presentation Foundation, but it makes much more sense to use the new MediaElement control. It allows you to do essentially the same thing, as you'll see in a few moments; however, it provides better integration with Windows Presentation Foundation. Specifically, it gives you the ability to composite media with alpha-blending, and it also lets you render onto 3D surfaces.

Audio and video integration with Windows Presentation Foundation gives developers multiple options to choose from, each with its own levels of complexity and control over media files. Regardless of the control or class used, what happens behind the scenes remains pretty much the same. The general idea is that when your application is told to play a media file in Windows Presentation Foundation, it launches the Windows Media Player runtime in the background to play the file. The fact that Windows Presentation Foundation uses Windows Media Player and does not contain its own media rendering engine presents some advantages and disadvantages. Piggybacking on Windows Media Player is somewhat analogous to using the ActiveX control.

Since Windows Media Player is responsible for Windows Presentation Foundation media playback, it's easy to talk about which media formats are compatible with Windows Presentation Foundation. Basically, any formats an end user can play through an existing Windows Media Player installation are the formats available to Windows Presentation Foundation applications. In order to add support for an additional media type, the user would have to install a new video codec.

Be aware that since the Windows Media Player runtime has to launch to play media files, there will be a delay in the media's initial playback as the runtime is loaded in the background. This can make precise control over media difficult, especially when trying to do things like syncing audio or video to animations or other events in Windows Presentation Foundation. One way to get around this is by preloading media files and keeping them in a paused state until you are ready to use them. Another options is that by using slip in the animation engine, you can make other animations dependent on the MediaElement, so at least everything will delay equally.This is not an optimal workflow and the Windows Presentation Foundation media team is working on a solution for the next version. Luckily, for audio playback there's an alternative-the System.Media.SoundPlayer class that allows for simple playback of WAV files without the need for Windows Media Player. We'll look at using this class later in this article.

Using the MediaElement Control

By far the easiest way to add media to a Windows Presentation Foundation application is by using the new MediaElement control. This control, like all others in Windows Presentation Foundation, can be added using either XAML or in the codebehind file using either C# or Visual Basic®. MediaElement enables basic playback and control over media files and can be run in one of two modes: Independent or Clock.

Independent mode, which is the default, works most closely like a traditional media player:

<MediaElement Source="myMedia.wmv" Width="320"

Height="240" LoadedBehavior="Manual" />

You set the Source property, which will be a URI to a media file. What happens when the media file is loaded depends on the LoadedBehavior property. This property must be set to a value from the MediaState enumeration: Close, Manual, Pause, Play, or Stop. In order to control media playback using the Play, Pause, or Stop methods of the MediaElement control, the LoadedBehavior property must be set to Manual.

Figure 1 shows an example of a simple media player using the MediaElement control running in Independent mode. Figure 2 shows the XAML and C# code used to implement this control.

Figure 2 Implementing the MediaElement Control

XAML

XAML

<Window x:Class="VideoProject.Window1"

xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"

xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"

Title="VideoProject" Height="800" Width="600">

<Grid>

<StackPanel HorizontalAlignment="Center" VerticalAlignment="Center">

<MediaElement Name="myMedia" Source="myMedia.wmv"

LoadedBehavior="Manual" Width="320" Height="240" />

<StackPanel Orientation="Horizontal" Margin="0,10,0,0">

<Button Content="Play" Margin="0,0,10,0"

Padding="5" Click="mediaPlay" />

<Button Content="Pause" Margin="0,0,10,0"

Padding="5" Click="mediaPause" />

<Button x:Name="muteButt" Content="Mute"

Padding="5" Click="mediaMute" />

</StackPanel>

</StackPanel>

</Grid>

</Window>

C#

C#

using System;

using System.Windows;

using System.Windows.Controls;

using System.Windows.Data;

using System.Windows.Documents;

using System.Windows.Input;

using System.Windows.Media;

using System.Windows.Media.Imaging;

using System.Windows.Shapes;

namespace VideoProject

{

public partial class Window1 : System.Windows.Window

{

public Window1()

{

InitializeComponent();

myMedia.Volume = 100;

myMedia.Play();

}

void mediaPlay(Object sender, EventArgs e)

{

myMedia.Play();

}

void mediaPause(Object sender, EventArgs e)

{

myMedia.Pause();

}

void mediaMute(Object sender, EventArgs e)

{

if (myMedia.Volume == 100)

{

myMedia.Volume = 0;

muteButt.Content = "Listen";

}

else

{

myMedia.Volume = 100;

muteButt.Content = "Mute";

}

}

}

}

Figure 1 A Simple MediaElement Control

Running the MediaElement control in Clock mode enables you to control media playback using the robust Windows Presentation Foundation animation engine. Clock mode can be enabled either by targeting the MediaElement as part of a MediaTimeline or by explicitly creating a MediaClock from a MediaTimeline and assigning it to the MediaElement (though that's not a recommended approach). This may sound confusing, but essentially it just means you can attach the MediaElement to a Storyboard and animate its position using the Windows Presentation Foundation animation engine. This allows you to sync up media playback to other animations in your application.

Here's an example of the XAML code needed to playback media in Clock mode using a Storyboard:

<MediaElement Name="myMedia" Width="320" Height="240" />

<Grid.Triggers>

<EventTrigger RoutedEvent="Grid.Loaded">

<EventTrigger.Actions>

<BeginStoryboard>

<Storyboard Storyboard.TargetName="myMedia">

<MediaTimeline Source="myMedia.wmv"

BeginTime="00:00:00" Duration="00:05:00" />

</Storyboard>

</BeginStoryboard>

</EventTrigger.Actions>

</EventTrigger>

</Grid.Triggers>

Note that the source of the media file is set in MediaTimeline rather than in MediaElement. MediaTimeline targets the MediaElement control via the Storyboard.TargetName attached property. Clock mode can be confusing if you're not familiar with the Windows Presentation Foundation animation engine, so make sure you're comfortable with basic Windows Presentation Foundation animation before venturing down this path.

After you've set up MediaElement and configured its mode, you can then read and write to a whole slew of properties to further enhance the media playback experience. To get or set the position of the media file, you can simply set the Position property with a valid TimeSpan object. You can display the buffering and playback progress to the user by reading the BufferingProgress and DownloadProgress properties. For audio control, the Balance and Volume properties can be set to customize or animate a media's audio track. Now think about if these values were databound to a couple of sliders-you could give the user complete control over the media's aural experience!

MediaElement also fires a number of useful events. MediaOpened and MediaEnded are handy for triggering other parts of your application based on whether the media has started or ended. The BufferingStarted and BufferingEnded events allow you to display feedback to the user when the video freezes during a buffering operation. If used in conjunction with the BufferingProgress property, they let you display a buffering percentage to the user similar to what is seen when viewing streaming video with Windows Media Player.

Based on my own experience-and reinforced by conversations with the Windows Presentation Foundation media team-I recommend you normally use the MediaElement control in Independent mode unless Clock mode is absolutely necessary. For standard media playback and control, there's no need to deal with Storyboards or MediaTimelines. And MediaElement should be your default choice for media integration since the ability to add it using XAML lets designers implement media, leaving the developer free to do other things.

Using the MediaPlayer Class

If the MediaElement control doesn't provide enough functionality and control over playback for your application, the next option for you to consider is using the MediaPlayer class located in the System.Windows.Media namespace. Since this is not a Windows Presentation Foundation control element, the only way to implement a MediaPlayer is to create it in the codebehind file using either C# or Visual Basic. Here's a simple example that shows how to implement a MediaPlayer using C#:

MediaPlayer mp = new MediaPlayer();

Try {

mp.Open(new Uri("myMedia.wmv"));

}

catch(Exception ex) {

MessageBox.Show(ex.Message);

}

The properties and methods available in the MediaPlayer class are strikingly similar to those of the MediaElement control. This is not merely a coincidence-the MediaElement is essentially a high-level wrapper of the MediaPlayer class. The major caveat with using MediaPlayer is that it doesn't have any direct visual representation and can't be added directly to the visual tree. To play video you need to draw the MediaPlayer onto a visual surface and then add that surface to the visual tree. This makes MediaPlayer a good choice for playing audio as, obviously, no visual representation is needed. For straight video playback, the MediaElement control is a much better option since it inherits from the UIElement class and thus can be directly added to the visual tree in your applications.

Just like the MediaElement control, you can run a MediaPlayer instance in either Independent or Clock mode. As for playback control, again this is essentially a carbon copy of the MediaElement control. So what's different about the MediaPlayer? Well, for one, there are no LoadedBehavior or UnloadedBehavior properties to deal with, as these are unique to the MediaElement control. If you are planning to draw media to 3D surfaces using the VideoDrawing class, then you must use MediaPlayer because the VideoDrawing's Player property won't accept a MediaElement control as its value. (Don't worry-I'll show you how to draw video to 3D and other surfaces a little later.)

Another advantage (or disadvantage, depending on your situation) is that since the MediaPlayer is not implemented via XAML, it doesn't become part of the visual tree. Not being a part of the visual tree can be beneficial if your media file doesn't need visual representation. There are also some differences in how you load a media file into your application. In the MediaElement control you set its Source property, but when using the MediaPlayer class you call the Open method and pass in a valid Uri object.

So if your application doesn't require media files to be implemented using XAML or to have an automatic visual representation, the Media Player is probably your best option since it gives you all the capabilities of the MediaElement control without the dreaded additional overhead.

Embedding Video inside WPF Controls

Since the MediaElement control inherits from the System.Windows.UIElement class, it can be placed anywhere any other UIElement control can be. One such example would be nesting a MediaElement control inside a Button control, thus creating a video button. This can be accomplished using this XAML code:

<Button Name="myVideoButton" Click="playVideo"

Width="320" Height="240">

<MediaElement Name="myVid" Source="myMedia.wmv"

LoadedBehavior="Manual" Width="320"

Height="240" />

</Button>

In this code I'm embedding MediaElement inside of a Button and setting its LoadedBehavior to Manual. The click event of the Button calls a method (not shown) that tells the MediaElement to begin playing. With a few lines of code you've created an interactive video thumbnail. All kinds of creative possibilities open up due to the fact that Windows Presentation Foundation treats the MediaElement control just like any other UI element in the visual tree. As you'll see in the next section, you can paint the video playing in the MediaElement onto other Windows Presentation Foundation objects to create incredibly rich user experiences.

The Wet Floor Technique

In recent times, glass and reflections have taken over the interfaces of everything in sight. Without a doubt, someday you will be asked to deliver such a technique in Windows Presentation Foundation, and fortunately it's easy. The extensive brush system in Windows Presentation Foundation allows you paint one control onto another with ease. In this example, I'll paint the video playing in a MediaElement control onto a Rectangle using a VisualBrush, creating a perfect reflection of the video. In the design world, this effect is known as a wet floor.

The easiest way to create a wet floor effect is by placing your MediaElement control in the first slot of a StackPanel control. Below that you then create a Rectangle with the same dimensions as the video. Next, use a VisualBrush as the fill for the Rectangle and databind its Visual property to the MediaElement control. Finally, flip the reflection to create a mirror image and apply an OpacityMask to fade out the reflection.

Figure 3 shows the completed effect, while Figure 4 shows the XAML code. This type of visual manipulation would have been incredibly complicated in traditional Win32 programming. And, thanks to the powerful databinding features in Windows Presentation Foundation, this is a live reflection that automatically updates whenever your video does.

Figure 4 XAML Used to Create the Wet Floor

<Grid x:Name="myGrid">

<StackPanel VerticalAlignment="Center">

<MediaElement Name="myVid" Source="myMedia.wmv"

LoadedBehavior="Play" Width="320" Height="240" />

<Rectangle Width="320" Height="240">

<Rectangle.Fill>

<VisualBrush Visual="{Binding ElementName=myVid}" />

</Rectangle.Fill>

<Rectangle.OpacityMask>

<LinearGradientBrush StartPoint="0.5,0" EndPoint="0.5,1">

<GradientStop Color="#AA000000" Offset="1" />

<GradientStop Color="#00000000" Offset="0" />

</LinearGradientBrush>

</Rectangle.OpacityMask>

<Rectangle.RenderTransform>

<TransformGroup>

<ScaleTransform ScaleY="-1" />

<TranslateTransform Y="242" />

</TransformGroup>

</Rectangle.RenderTransform>

</Rectangle>

</StackPanel>

</Grid>

Figure 3 The Completed Wet Floor

Mapping Video onto 3D Surfaces

If I had to pick the one area that seems to have created the biggest buzz around Windows Presentation Foundation, it would be the ability to map video files onto 3D mesh objects. The North Face demo showed just how powerful this could be in helping to create a visually stunning user interface. While many people have written off this effect as either too gimmicky or too resource-intensive, I feel that, when done in moderation, it can turn an ordinary application into one that provides an outstanding experience for users.

As for performance, of course there's quite a bit of overhead involved with displaying and animating 3D meshes. If you add to that displaying full-motion video that is being mapped onto the 3D meshes, you are talking about a ton of overhead. Just as with drinking mocha frappucinos, the key here is moderation. Overall, I've found the 3D performance in Windows Presentation Foundation excellent, so long as the complexity (polygon count) of the 3D meshes is kept to the minimum necessary. The Windows Presentation Foundation media team is committed to reducing the expense of 3D media in future versions, although it will always be more expensive than 2D media.

Before looking at how to map video onto 3D objects, let's first review what it takes to set up a 3D scene in Windows Presentation Foundation. First and foremost, you need a 3D object to place into the scene. There are no 3D primitive classes that are a part of the .NET Framework 3.0. But someone at Microsoft created a class called Mesh3DObjects that can create a wide range of primitive objects. (You can find more information about this class in the tutorial by Karsten Januszewski at msdn.microsoft.com/library/en-us/dnlong/html/avalon2d-3d.asp.)

There are certain sets of elements required in order to create a 3D scene in Windows Presentation Foundation. First, you need the 3D object, and this object needs to have some kind of material in order to be viewable in the scene. Materials are created using brushes like SolidColorBrush, LinearGradientBrush, ImageBrush, or a VisualBrush. You'll see in a few moments how a VisualBrush can be used to map video, or any other visual element for that matter, onto a 3D surface. The technique is essentially the same as the wet floor technique that I discussed earlier.

The 3D mesh and materials need to be grouped inside of a GeometryModel3D element, and doing so creates a valid Windows Presentation Foundation 3D object. For a 3D scene to be visible, you'll need to add one or more lights to the scene. Then, just as a photograph isn't possible without a camera, a 3D scene in Windows Presentation Foundation isn't either. Finally, to complete the scene, all of these items need to be wrapped in a Viewport3D control in order to seal the deal.

Once you have a functional 3D scene in place, you can start having fun mapping video and other media to the 3D meshes. This is accomplished using a VisualBrush to draw the video onto the 3D surface. In the following code snippet, I am applying a VisualBrush to a 3D object's DiffuseMaterial:

<GeometryModel3D.Material>

<DiffuseMaterial>

<DiffuseMaterial.Brush>

<VisualBrush>

<VisualBrush.Visual>

<MediaElement Source="myMedia.wmv" />

</VisualBrush.Visual>

</VisualBrush>

</DiffuseMaterial.Brush>

</DiffuseMaterial>

</GeometryModel3D.Material>

Figure 5 shows a single frame of the video mapped onto the Windows Presentation Foundation 3D surface.

Figure 5 Mapped to a WPF 3D Surface

There's another alternative you can use to map video onto 3D surfaces in Windows Presentation Foundation. The VideoDrawing class, which is a part of the System.Windows.Media namespace, does precisely what its name implies. It can draw video coming from a MediaPlayer instance via a DrawingBrush onto any brush-compatible surface. In order to achieve this, you need to set the Drawing property of the DrawingBrush to an instance of the VideoDrawing class. This VideoDrawing instance can render video by setting its Player property to a MediaPlayer instance containing a video file. Finally, the DrawingBrush can be applied to a 3D object's DiffuseMaterial's Brush property. The following code snippet shows an example implementation using C#:

MediaPlayer mp = new MediaPlayer();

mp.Open(myValidMediaUri);

VideoDrawing vd = new VideoDrawing();

vd.Player = mp;

DrawingBrush db = new DrawingBrush();

db.Drawing = vd;

diffuseMat.Brush = db;

vd.Player.Play();

One of the major differences between using a VisualBrush or the VisualDrawing class is that a VisualDrawing must be created and implemented in the codebehind file (using C# or Visual Basic). Furthermore, the VideoDrawing class will only work with MediaPlayer instances and not with the MediaElement control. When it comes to performance, there isn't much difference between the two methods since both require Windows Presentation Foundation to create an intermediary render surface in order to correctly map the video to 3D.

Rendering Bitmaps from Video Frames

A video application may need to essentially take a screen grab of the video stream and render it to a bitmap. One such example is seen in some commercial DVD playback software where the user can click on a camera icon at any time during playback to take a screen grab from the video. Another use could be for creating thumbnails of a set of video files if you didn't want all of them actually connected to MediaElement controls. There is also some potential for creating runtime video effects, although for this type of thing you'd be much better served by diving into the DirectShow API.

Like most things in Windows Presentation Foundation, there are multiple ways of accomplishing a task, and rendering bitmaps of video frames is no different. No matter what method is used, the core functionality for rendering bitmaps of visual elements in Windows Presentation Foundation is handled by the RenderTargetBitmap class, which is a part of the Windows.Media.Imaging namespace. This amazing class can create bitmaps from any item in the visual tree of a Windows Presentation Foundation application.

To render a bitmap using this class, first you need to create a new DrawingVisual object, which gets passed to the RenderTargetBitmap.Render method. Next you create a new DrawingContext object that you'll pass to the DrawingVisual.Render method. With this DrawingContext instance, you can draw all kinds of graphics into the DrawingVisual object. To draw a frame of video, you can call the DrawingContext.DrawVideo method and pass in a MediaPlayer instance that is currently playing the video file. You'll also need to pass in a Rect value that determines the area of the video file that will be captured.

At this point, you can actually render a bitmap from the DrawingVisual instance that now contains your captured video frame. To do this, you just need to pass it to the RenderTargetBitmap.Render method. Now your RenderTargetBitmap instance contains the video frame, but it won't be a bitmap until you pass it to the BitmapFrame.Create method. To see the bitmap frame, you can simply set this method call as the value of an Image control's Source property. The following snippet shows a C# implementation of this frame-capture technique:

RenderTargetBitmap rtb = new RenderTargetBitmap(320, 240, 1 / 200,

1 / 200, PixelFormats.Pbgra32);

DrawingVisual dv = new DrawingVisual();

DrawingContext dc = dv.RenderOpen();

dc.DrawVideo(myMediaPlayer, new Rect(0, 0, 320, 240));

dc.Close();

rtb.Render(dv);

Image im = new Image();

im.Source = BitmapFrame.Create(rtb);

The target bitmap's format is set by passing a valid PixelFormats enumeration value to the RenderTargetBitmap constructor. The ability to render bitmaps from video frames may at first seem like a niche feature, but there are actually many uses for this type of functionality beyond the simple video screen capture tool.

Additional Audio Capabilities in WPF

Most everything I've discussed so far about Windows Presentation Foundation media integration holds true for both video and audio. There are, however, several additional classes available for handling audio playback in Windows Presentation Foundation. As I mentioned earlier, media played in Windows Presentation Foundation is actually driven by a set of Windows Media Player runtimes that need to be loaded before the file can be played back. This can cause major issues when you're trying to synchronize media files to user events or animations, though it can be mitigated with slip behavior in timelines. When it comes to audio, there are some alternatives that can bypass this problem.

The SoundPlayer class, which is a part of the System.Media namespace, provides a simple interface for loading and playing back audio WAV files. These WAV files can be external, streaming, or embedded resources. This method of audio playback has much lower overhead than playing an audio file through a MediaElement control since it does not at all depend on Windows Media Player for playback.

To load a WAV file into a SoundPlayer instance, you first need to set the SoundLocation property to the location of the WAV file and then call the Load method to start the loading process. Once the file is fully loaded you can play it using the SoundPlayer.Play method. The file won't start playing until it is fully loaded into memory.

This lightweight audio playback is also available in XAML using the SoundPlayerAction control, which lets you play a WAV file inside of an EventTrigger. This is very useful for things like interface sounds for button rollovers, for example. To use it, you simply place it into the EventTrigger.Actions block of the EventTrigger. Following is a code snippet of SoundPlayerAction being called in response to a button's MouseEnter event:

<EventTrigger RoutedEvent="Button.MouseEnter" SourceName="myButton">

<EventTrigger.Actions>

<SoundPlayerAction Source="media/overSound.wav"/>

</EventTrigger.Actions>

</EventTrigger>

Summary

I've explored here the available methods for integrating audio and video into Windows Presentation Foundation applications. If Windows Presentation Foundation does not provide the level of control needed for your project, you can always revert to using the DirectShow API for media integration. But the ability to add audio and video using XAML allows designers to handle simple media integration so that developers can focus on more complex coding. I am no longer the sceptic I once was. The amazing visual richness available in Windows Presentation Foundation makes media integration not only possible, but incredibly exciting!

Lee Brimelow, a Senior Design Technologist at frog design, is an award-winning interactive designer who specializes in Flash and WPF. He also runs the popular WPF technology blog at www.thewpfblog.com.

WiX Tricks

Automate Releases With MSBuild And Windows Installer XML

Sayed Ibrahim Hashimi

This article discusses:

- An overview of Windows Installer XML

- Creating WiX packaging instructions

- Integrating WiX and MSBuild

- Automating builds and packaging

This article uses the following technologies:

Visual Studio, Windows Installer XML (WiX), MSBuild

Code download available at: WiX2007_03.exe (498 KB)

Browse the Code Online

Contents

Introduction to WiX

Creating WiX Files

WiX and MSBuild

MSBuild Batching

Automating Builds and Packaging

Anatomy of the MSBuild Script

Putting It All Together

Customizing the Process

Conclusion

During the course of development, it's important to have an automated build process. Equally important is having an automated means of creating releases. Unfortunately, in many organizations-especially smaller ones-this doesn't happen. Typically, you'll find that the release is simply cobbled together at the last minute. However, if you take the time to set up an automated build and release plan, you'll save countless hours that could be better spent on tasks other than building and releasing your project.

In this article, I'll show you how to achieve an automated and repeatable build and release process in your organization using the Microsoft® Build Engine (MSBuild) and the Windows® Installer XML (WiX) toolset. This article covers WiX v2 (note that when WiX v3 is released, some of the syntax examples won't translate directly). The techniques described here will be relevant whether or not you are using WiX to create your releases, though WiX does simplify the process. They can also be applied, with some modifications, to applications that are being developed without using the Microsoft .NET Framework 2.0.

I'll assume you're familiar with MSBuild (if you need to refresh, look at my June 2006 MSDN®Magazine article "Inside MSBuild: Compile Apps Your Way with Custom Tasks for the Microsoft Build Engine"). I will provide an overview of the WiX toolset for those who are not familiar with it. See the "MSBuild and WiX Resources" sidebar for references to more related articles and tools. In this article I'll use my Sedodream MSBuild project for demonstration purposes. You can get the latest sources at www.codeplex.com/Sedodream.

Introduction to WiX

When creating an application, the end result is usually to install and run it on production machines. The WiX toolset can help you do this. In this section, I will describe WiX and give an introduction to creating installers using WiX.

WiX describes what the installation looks like on the target machine. Windows Installer is used in many Microsoft applications, from Visual Studio® to Microsoft Office. You might be surprised to learn that WiX is actually an open source project and is hosted at sourceforge.net. You can download the latest binaries and sources at wix.sourceforge.net. When you download and install WiX, you'll find a number of executables installed as summarized in Figure 1. I will focus on using the Candle.exe and Light.exe tools.

Figure 1 WiX Components

Name

Description

Candle.exe

Transforms the WiX source file into an intermediate representation. This is actually another XML file, but you should never manually change those generated files.

Dark.exe

Converts an MSI file into an appropriate WiX source file-can be thought of as "decompiling" the installer.

Light.exe

Generates the Windows installer from the intermediate representation of the WiX source file(s).

Lit.exe

Generates WiX libraries that can be used to build other installer packages.

Tallow.exe

Used to create the WiX source XML to replicate its files and folders in an installation directory or file.

WixCop.exe

Checks a WiX source file for potential problem areas, similar to FxCop.

WiX uses a declarative language, not a procedural one, which means you describe what your installation will look like, not what steps need to be followed in order to achieve it. This may be different from what you are accustomed to, but it is surprisingly easy to learn. Typically, your WiX source files will be populated by describing files that will be installed on the target machine. I will focus on those components here.

In a WiX source file, there are three main elements with respect to the files you intend to install: File, Component, and Feature. A File element is a reference to an individual file. Files must be contained in a Component element, which is the smallest unit to be installed. That is, if you have a component containing 100 files and you install that component, all of its contained files are installed. Conversely, if that component is not installed, none of the files are installed. Creating components that contain a large number of files is not recommended.

Components are always contained in a Feature element and can be contained in more than one feature. A feature is a set of components and potentially of sub-features. If your installer has a graphical interface that allows the user to select which items to install, they are actually selecting features.

Creating WiX Files

You can author WiX source files in any text or XML editor you choose. You can also use Visual Studio to author WiX source files with IntelliSense®. If IntelliSense is not enabled when you are editing a WiX source file using Visual Studio, all you have to do is copy the wix.xsd file into the Visual Studio schemas directory. Typically, this directory resides at %Program Files%\Microsoft Visual Studio 8\Xml\Schemas. As you create WiX source files you will also need to generate many GUIDs. Visual Studio has a tool that you can use for this purpose, and there are Visual Studio macros available so you can assign a shortcut to directly insert a new GUID as well.

Now let's start by creating a new WiX source file. The first element is always going to be a Wix element. Child elements for the Wix element include Product, Fragment, Module, and PatchCreation. The type of desired output will drive which of these you will use. In this case, I want the end result to be an installer database (MSI) for my project so I'll use the Product element. The beginning of the WiX v2 source file, Sedodream.wxs, for the sample project is shown here:

<?xml version='1.0' encoding='UTF-8'?>

<Wix xmlns='http://schemas.microsoft.com/wix/2003/01/wi'>

<Product Name='Sedodream MSBuild Project'

Id='C9D25926-FCE0-4EB6-8FF5-4686EE5AB089'

Language='1033' Codepage='1252' Version='1.0.0'

Manufacturer='SedoTech'

UpgradeCode='9C5E4073-EFDE-419B-935D-CE2632BC560E'>

<Package Id='????????-????-????-????-????????????'

Keywords='Installer'

Description='Sedodream MSBuild Project Installer'

InstallerVersion='100' Languages='1031'

Compressed='yes' SummaryCodepage='1252' />

...

As you can see ,the Product element has many attributes, the most important of which are Id and Name. The Id is used to uniquely identify the product. You must change this for a new major release. You should also make a note of its value for later reference. The value of the Name attribute is what will be displayed in the Add/Remove programs panel, and if your installer has a UI it will be shown on the introduction page.

The first child of the Product element, the Package element, is shown here as well. Notice that the Id attribute contains a series of ? characters in the format of a GUID. By using this syntax, the code specifies that the Id should be generated at build time. When you are creating installers, you want the Package Id to be different for each installer created, even from one build to the next. This is the only GUID you can have auto-generated; all other GUIDs need to stay the same and should be recorded for future reference.

Almost every installer will place at least one file onto the target machine, and this installer is no different. Since WiX uses a declarative approach, I'll declare what the directory structure is and that will be recreated on the target machine. I do this using a series of Directory elements:

<Media Id='1' Cabinet='Sedodream.cab' EmbedCab='yes'/>

<Directory Id='TARGETDIR' Name='SourceDir'>

<Directory Id='ProgramFilesFolder' Name='PFiles'>

<Directory Id='MSBuildDir' Name='MSBuild'>

<Directory Id='INSTALLDIR' Name='Sedodrea'

LongName='Sedodream'>

The Media element in this sample specifies that a cabinet file will be generated and placed into the installer database. This is a required element, and if you have installations spanning more than one medium you can have more than one. For example, if you are distributing your application on a CD, but it needs to span over multiple discs. In most cases, you won't have to worry about this issue.

Following Media is the series of Directory elements. The first Directory element will be a virtual element that simply encapsulates the other entries. The entry under the TARGETDIR directory element has the Id value of ProgramFilesFolder. As you may have guessed, this is a well-known location and Windows Installer will set its value when launched. There are other system folders you can use; a complete list is available from msdn.microsoft.com/library/en-us/msi/setup/system_folder_properties.asp. These system properties always resolve to their full paths.

The next elements provide names of custom directories. The directory will be named using the LongName attribute if possible; otherwise the Name attribute is used. If those directories don't exist, Windows Installer will create the appropriate one at install time. The complete tree of what is declared in the previous code fragment ends up being C:\Program Files\MSBuild\Sedodream.

After you declare your directory structure, it's time to start creating Component declarations. As noted earlier, the smallest unit that can be installed is the component, which can consist of many different items including files, shortcuts, registry keys, and certificates. Design your components so they are independent of each other. That is, when you install or uninstall a component there should be no adverse effect on other components. Inherently this means that items should only be contained in a single component. If that's not the case, you may want to reconsider how your components are organized. It's not unusual for a component to contain only a single file.

Each Component declaration must have an Id and a GUID attribute. The Id is the name that you use to refer to it, and the GUID is a unique identifier for Windows Installer to use. It is very important to ensure that none of your component GUIDs repeat.

Figure 2 shows a component from the sample WiX source file. You can see that it contains four File elements and two XmlFile elements. The File element is simply a reference to a file that needs to be placed on the target device. The File element can have many different attributes, but the ones you'll mostly use are Id, Name, LongName, Source, and DiskId, described in Figure 3.

Figure 3 File Attributes

Attribute Name

Description

Id

An identifier that can be used to refer to the file.

Name

The short name of the file in 8.3 format.

LongName

The long name of the file; this is the name that will be used on the target machine if it supports long file names. Otherwise the Name value will be used.

Source

The relative path to the file on the machine that is creating the installer database.

DiskId

The identifier of the media item that this file will be contained in.

Figure 2 Defining Components

<Component Id='TaskBinFiles' Guid='38736E2E-BEB7-48A9-A2B3-138A57A69D45'>

<File Id='MSBCommonDLL' Name='CommoDLL'

LongName='Sedodream.MSBuild.Common.dll'

Source='Sedodream.MSBuild.Common.dll' Vital='yes' DiskId='1'/>

<File Id='SedodreamLoggersDLL' Name='Logger'

LongName='Sedodream.MSBuild.Loggers.dll'

Source='Sedodream.MSBuild.Loggers.dll' Vital='yes' DiskId='1'/>

<File Id='SedodreamTasksDLL' Name='Tasks'

LongName='Sedodream.MSBuild.Tasks.dll'

Source='Sedodream.MSBuild.Tasks.dll' Vital='yes' DiskId='1'/>

<File Id='SedodreamTASKS' Name='SeTasks' LongName='Sedodream.tasks'

Source='Sedodream.tasks' Vital='yes' DiskId='1'/>

<!-- Merge in the Xsd for the custom tasks we have created to have

IntelliSense -->

<XmlFile Id='SedoSche'

File='C:\Program Files\Microsoft Visual Studio 8\

Xml\Schemas\1033\Microsoft.Build.xsd'

Action='createElement' ElementPath='//xs:schema'

Name='xs:include' Permanent='no' Sequence='0'/>

<XmlFile Id='SedoSch2'

File='C:\Program Files\Microsoft Visual Studio 8\

Xml\Schemas\1033\Microsoft.Build.xsd'

Action='setValue'

ElementPath='//xs:schema/xs:include[\[]not(@schemaLocation)[\]]'

Name='schemaLocation' Value='MSBuild\Sedodream.MSBuild.xsd'

Permanent='no' Sequence='1' />

</Component>

The XmlFile element actually modifies an existing XML file on the target machine. It inserts a new element into the Microsoft.Build.xsd file, which allows IntelliSense to be enabled for the custom MSBuild tasks contained in the sample project.

Features are the items that the user will select for installation if your installer shows a UI. Here are the Feature declarations from the sample project:

<Feature Id='Complete'

Title='Sedodrem Core' Description='MSBuild libraries'

Display='expand' Level='1' ConfigurableDirectory='INSTALLDIR'>

<ComponentRef Id='TaskBinFiles'/>

<ComponentRef Id='NUnitFiles'/>

<Feature Id='Samples' Title='Samples'

Description='Contains samples of Sedodream usage'

Display='expand' Level='100' ConfigurableDirectory='INSTALLDIR'>

<ComponentRef Id='Samples'/>

</Feature>

</Feature>

Like many other WiX elements, the Feature element has many possible attributes, some of the more commonly used ones are described in Figure 4.

Figure 4 Feature Attributes

Attribute Name

Description

Id

An identifier that can be used to refer to the feature.

ConfigurableDirectory

Allows you to set the installation location from a property, which can be determined by a user with a UI.

Description

A description for the feature, which will be displayed to a user with a UI.

Level

Specifies the install level for the feature. A value of zero means the feature will not be installed. Non-zero values that are less than or equal to the INSTALLLEVEL mean the feature is installed.

Title

A title for the feature whose value will be displayed in the tree of features if a UI is being used.

In this WiX snippet the main feature, Complete, contains a reference to two components and a sub-feature. The sub-feature references are achieved by a child Feature element. The WiX source file can contain as many of these as necessary. The component reference uses a ComponentRef element. When using the ComponentRef element you have to use the same ID value that is used in the Component element you are trying to reference.

WiX and MSBuild

In the latest releases for WiX v2 you'll find a wix.targets file, which contains definitions that will assist you in creating WiX releases from MSBuild. The WiX team has created a set of MSBuild tasks that are referenced by the wix.targets file. Those tasks are all contained in the WiXTasks.dll assembly and summarized in Figure 5.

Figure 5 MSBuild Tasks for WiX

Name

Description

Candle

Creates the intermediate representation of the WiX source files by calling the WiX compiler.

Lit

Creates a WiX library from the intermediate representation.

Light

Creates the final installer from the intermediate representation by calling the WiX linker.

The way in which you'll build installers is very similar to how managed projects are built now. For example, when creating a C# project, your project file will contain all the necessary properties and items to build the project, but all the steps to build the project are contained in the Microsoft.CSharp.targets file. This file is included into the project file with the MSBuild Import element. For WiX, your project file will define all necessary properties and items and then include the wix.targets file to define the logistics of how to build the final packages. Figure 6 provides a list of the minimum declarations required to successfully build an installer using wix.targets.

Figure 6 Declarations Needed to Build an Installer

Name

Description

ToolPath

Contains the path to the WiX installation directory. This is where the wix.targets file resides.

OutputName

Path and name of the output that will be created. The name should not include any extension because it will be used for both the intermediate representation and the final package file. This property is analogous to the OutputFile argument for Candle.exe and Light.exe.

OutputType

Represents what the final file should be. The possible values are package, module, library, and object. This will determine whether the Light or Lit task will be invoked and the extension of the final file created. Since we want to create an installer, we will provide the value package for this property.

Compile

MSBuild item that contains the WiX source files to be compiled.

BaseInputPath

Serves as the input path for source files. Although technically not required, most times this should be employed. Otherwise you'll have to make sure to invoke msbuild.exe from the correct directory.

To create the installer you need to create an MSBuild file to define the required properties and items. In this case, I'll name the file SedodreamMSI.wproj. This file is shown in Figure 7.

Figure 7 MSBuild File SedodreamsMSI.wproj

<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003" DefaultTargets="Build">

<!--===============================================================

These must be declared BEFORE the statement

that imports the wix.targets file

==================================================================-->

<PropertyGroup>

<!-- The location pointing where WiX is installed -->

<ToolPath>C:\Data\Development\WiX\</ToolPath>

<!-- Required Property by WiX -->

<OutputName Condition="$(OutputName)==''" >

$(Configuration)Sedodream</OutputName>

<!-- Required property by WiX -->

<OutputType

Condition="$(OutputType)==''" >package</OutputType>

<!-- Input path to source files  -->

<BaseInputPath  Condition="$(BaseInputPath)==''">

$(PackageRoot)$(Configuration)</BaseInputPath>

</PropertyGroup>

<ItemGroup>

<!-- Required WiX item.

Files in this item are sent to the Candle tool.

-->

<Compile Include="$(BaseInputPath)\Sedodream.wxs"/>

</ItemGroup>

<Import Project="$(ToolPath)wix.targets"/>

</Project>

As you can see, there's not much to this MSBuild file, just a few properties and a lone item defined. The file assumes that a WiX source file, Sedodream.wxs, has been created and is residing in the BaseInputPath directory. The WiX file also makes some assumptions about where files are placed. For now let's assume that all the files are where they need to be.

To build the installer I will need to invoke MSBuild on the project file. I open the Visual Studio 2005 command prompt, navigate to where the file resides, and execute the following command:

msbuild Sedodream.wproj

The output is shown in the Figure 8.

Since I didn't define a target to execute, the DefaultTargets were executed, and this was defined to be Build in the Sedodream.wproj file. From the output you can see that a file named ReleaseSedodream.msi was created in the bin\Release folder. At this point you can execute the installer to ensure that it works as expected. Before continuing with our main goal, I will cover an advanced MSBuild topic, batching, in the next section. This is necessary because batching is used throughout the process that will build the product and its release.

Figure 8 WiX Installer Creation Output from MSBuild (Click the image for a larger view)

MSBuild Batching

When using MSBuild you'll discover that there is no looping construct to be found. In place of a loop you can employ batching, which uses item metadata to break items into different categories called batches or buckets that consist of one or more items. Once the items are separated you can iterate through each batch. This is used throughout the build process for managed projects. There are two broad categories of batching: task batching and target batching. In task batching, when you execute a task, the MSBuild engine will determine what buckets need to be created and execute the task over those buckets. To use task batching, you provide an item's metadata to the task.

To demonstrate batching, imagine you are faced with the following scenario. You have to copy a set of files from one location to one or more other locations and you have two options: you can copy the files using individual Copy tasks to each location, or you can use batching to achieve this in one copy element. The first approach, although simple, is not a good solution because it is difficult to maintain. To illustrate the second approach, have a look at this very simple MSBuild file, batching01.proj:

<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003"

DefaultTargets="CopyFiles">

<ItemGroup>

<SourceFiles Include="*.txt"/>

<Dest Include="One;Two;Three;Four;Five"/>

</ItemGroup>

<Target Name="CopyFiles">

<Copy SourceFiles ="@(SourceFiles)"

DestinationFolder="%(Dest.FullPath)"/>

<Message Text="Fullpath: %(Dest.FullPath)"/>

</Target>

</Project>

In this file, I declare an item SourceFiles to include any file that ends with .txt, and I also declare a Dest item that includes five locations. In the target CopyFiles, I use the Copy task to copy the SourceFiles to each destination. Since I pass the FullPath metadata into the Copy task, the MSBuild engine will create a batch of all the Dest items that have distinct FullPath values, then the Copy task is invoked for each batch. This is task batching. To execute this I can invoke the following command from the directory that contains the batching01.proj file:

msbuild.exe batching01.proj

Now I'll demonstrate the same result with target batching. For target batching, the batches are created based on the Inputs and Outputs of the target so I have to pass an item's metadata for these values:

<Project xmlns="http://schemas.

microsoft.com/developer/msbuild/2003"

DefaultTargets="CopyFiles">

<ItemGroup>

<SourceFiles Include="*.txt"/>

<Dest Include="One;Two;Three;Four;

Five"/>

</ItemGroup>

<!-- These targets demonstrate target

batching -->

<Target Name="CopyFiles"

Inputs="@(SourceFiles)"

Outputs="%(Dest.FullPath)">

<Copy SourceFiles="@(SourceFiles)"

DestinationFolder=

"%(Dest.FullPath)"/>

</Target>

</Project>

Target batching is driven off of the targets Inputs and Outputs. Since Outputs contains the Dest.FullPath declaration the target will be invoked once per bucket. This is similar to task batching, except that the contents of the entire target will be repeated once per batch, as shown in Figure 9.

Figure 9 MSBuild Output for Target Batching (Click the image for a larger view)

From this output you can see that the target, CopyFiles, was invoked once per batch for the Dest item. In the example of automating the build/package process, I'll use batching to create an installer for each combination of configuration platform and flavor.

Automating Builds and Packaging

For each organization the steps involved to build and package are going to be slightly different, in some cases due to the technologies used, and in others because of organizational requirements. I have put together a set of steps that I think will serve many organizations well, though it may require some fine tuning based on your particular scenario. The goal of an automated build-and-package process is twofold: to create a repeatable public build process and to create an automated repeatable package process. There are three core steps: get latest, build, and package, but this is broken down into many smaller steps as shown in Figure 10.

Figure 10 Build Process (Click the image for a larger view)

A very important step is labeling the sources. This is a requirement because without this step you would not be able to repeat the process. When you create a build that will be sent for production, you need to be able to recreate that exact same build in order to maintain the deployed code. This step will be different based on the tool you are using for your source control provider. There are many different MSBuild tasks available for download to achieve this for various source control providers. See the "MSBuild and WiX Resources" sidebar for more detail.

Anatomy of the MSBuild Script

In this section, I'll introduce the MSBuild script that can be used to build and package the sample application. The overall structure is very similar to how managed projects are built. That is, you'll define a project file that contains what should be processed, and you import another project file, Sedodream.Package.targets, that defines the process flow. Think of these processes as sentences. Your packaging project file defines the nouns and the imported file defines the verbs. I will build the project by using the MSBuild task on the Solution file. To build the installer, the previous project file, SedodreamMSI.wproj, will be used.

When you have a process that has segregated elements, they somehow need to be brought together. This is done by defining a common set of properties, items, and targets (see Figure 11).

Figure 11 Properties and Items Used to Connect Elements

Name

Description

SolutionFilePath

Property that defines the location of the solution file that will be used to build the product.

PackageRoot

Property that defines where the projects are built and serves as the temporary build location for WiX.

WiXSourceFiles

Item that contains all of the WiX files to be sent to the Candle tool for compiling.

AllConfigurations

Item that contains all the configurations you want this process to be built for. Each configuration should contain metadata values for both FlavorToBuild and PlatformToBuild. An example declaration is:

<AllConfigurations Include="Release|Any CPU">

<FlavorToBuild>Release</FlavorToBuild>

<PlatformToBuild>Any CPU</PlatformToBuild>

</AllConfigurations>

OtherFiles

Optional item that contains other files that should be copied to the package directory. These items should contain Destination metadata. This metadata defines the relative path of the destination to the package root. Here is a sample declaration:

<OtherFiles Include=

"..\Sedodream.MSBuild.Tasks\SampleTargets\**\*">

<Destination>Samples\</Destination>

</OtherFiles>

In this process, the solution will be built and the WiX source files will be copied into that same directory. You should define your WiX source files keeping this path in mind. Regardless of where your WiX files are contained in source control, they will be built from the same directory that the product is built to. Other required files are also copied to this same directory by the CopyFilesForPackaging target.

Also in this process, the PackageRoot property will define where the projects are built and where WiX will build the installers. In the sample, the directory is at the same level as the solution file, but it could be anywhere on the build machine-though I recommend not using a network share for this location.

The main targets declared in the Sedodream.Package.targets file are shown in Figure 12. All of these targets (and many others in the targets file) have their dependencies defined in properties so you can completely change the sequence of events in the process. For example, if you needed to inject a step before the Build target executed your project file, you could simply insert the following snippet after the Import statement for the Sedodream.Package.targets file:

<BuildDependsOn>

CustomBeforeBuild;

$(BuildDependsOn);

</BuildDependsOn>

Figure 12 Build Targets in the Sample Project

Name

Description

Package

Only target in the DefaultTargets list; will be called to execute the entire process.

Build

Builds the solution file. The output of this build is routed to the correct directory by overriding the OutputPath property when invoking the MSBuild task.

CopyFilesForPackaging

Copies all necessary files that are not built to the build directories.

DeployPackage

Called near the end of the process. In the sample this target is empty, but in your implementation you may decide to override the target, to send your packages to the QA gate or to your deployment team.

Clean

Cleans up the mess made by the other targets.

Because the BuildDependsOn property was redefined, you effectively inject the CustomBeforeBuild step into the current process without modifying the existing targets file. For a more detailed discussion of this issue, you should take a look at my June 2006 article (referenced previously).

Before creating the installer, let's have a quick look at the CoreBuild target that builds the solution (see Figure 13). This will demonstrate how target batching is used to build the solution for each defined configuration.

Figure 13 CoreBuild Target

<Target Name="CoreBuild"

Inputs="%(AllConfigurations.PlatformToBuild);

%(AllConfigurations.FlavorToBuild)"

Outputs="%(AllConfigurations.PlatformToBuild);

%(AllConfigurations.FlavorToBuild)" >

<Message Text="Building for Flavor/Platform:

%(AllConfigurations.FlavorToBuild)/

%(AllConfigurations.PlatformToBuild)"/>

<MSBuild Projects="@(SolutionFile)"

Targets="Build"

Properties="OutputPath=$(PackageRoot)

%(AllConfigurations.FlavorToBuild);

Configuration=%(AllConfigurations.FlavorToBuild);

Platform=%(AllConfigurations.PlatformToBuild)">

<Output ItemName="OutputFiles" TaskParameter="TargetOutputs"/>

</MSBuild>

</Target>

Specifically, the first thing to notice is the Inputs and Outputs; these are the metadata values for the AllConfigurations item that will cause the target to be executed once per platform/flavor pair. Then, the MSBuild task is called to build the solution file with all the appropriate properties. The OutputPath is provided because I want to redirect the output to another directory.

Putting It All Together

At this point, I have discussed how to create an installer using WiX, some advanced MSBuild concepts, and the structure of the MSBuild packing script. The only remaining issue with regard to creating the installer for the sample project is creating the MSBuild project file to define the required properties. This is surprisingly simple. Finally, I'll show you how to customize the process to suit your needs.

Figure 14 shows the MSBuild file, Sedodream.Package.dproj, that will drive the process for the sample project. As discussed earlier, the main purpose of this project file is to describe what needs to be built. The imported file, Sedodream.Package.targets, knows all the details about how to create the final product, and you can also customize the process by injecting steps into the build process.

Figure 14 Sedodream.Package.dproj MSBuild Project

<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003"

DefaultTargets="Package">

<!-- Required properties by the deployment.targets file -->

<PropertyGroup>

<SolutionFilePath>..\Sedodream.MSBuild.sln</SolutionFilePath>

<PackageRoot>..\Package\</PackageRoot>

</PropertyGroup>

<ItemGroup>

<WiXSourceFiles Include="Sedodream.wxs"/>

<OtherFiles Include="..\Sedodream.MSBuild.Tasks\

SampleTargets\**\*">

<Destination>Samples\</Destination>

</OtherFiles>

<!-- Copy the license to the BaseSearchPath so it can

be included into the installer -->

<OtherFiles Include="License.rtf">

<Destination></Destination>

</OtherFiles>

<!-- Copy the custom installer bitmaps -->

<OtherFiles Include="bitmaps\dlgbmp.bmp">

<Destination>bitmaps\</Destination>

</OtherFiles>

</ItemGroup>

<!-- Define all the configurations that you want to build here -->

<ItemGroup>

<AllConfigurations Include="Debug|x86">

<FlavorToBuild>Debug</FlavorToBuild>

<PlatformToBuild>Any CPU</PlatformToBuild>

</AllConfigurations>

<AllConfigurations Include="Release|Any CPU">

<FlavorToBuild>Release</FlavorToBuild>

<PlatformToBuild>Any CPU</PlatformToBuild>

</AllConfigurations>

</ItemGroup>

<PropertyGroup>

<DropLocation>C:\Data\Drops\MSBuild-Wix\</DropLocation>

</PropertyGroup>

<!-- Import the deployment target to do all the work for us -->

<Import Project="Sedodream.Package.targets"/>

</Project>

In this project file, you see the required SolutionFilePath property defined. This is the path to the solution file that will be used to build the product. Also, you see the declaration of the WixSourceFiles item, which includes the set of files that will be used to compile your installer. These are the files that will be sent to the Candle.exe tool for compiling.

Along with this is the declaration of the OtherFiles item, which contains files that should be copied to the location that will be used to create the installer. These files will be copied to the BaseInputPath that will be used to create the installer. You can have them copied to any location underneath that by using the Destination metadata value. In this example, I am copying custom bitmaps to be used to generate the installer into the bitmaps folder. When the Light.exe tool is invoked to create the MSI file, it will pick up files that are in this directory before any default files that are located in the local directory.

Note the declaration of the DropLocation property. By declaring this property, all the files in the PackageRoot will be copied to a directory in the DropLocation. It's helpful to have the binaries and installer files in one location, in case you need to make some quick modifications.

To see the build and package executed for the sample project, open a Visual Studio 2005 command prompt and navigate to the projects deployment folder. Then execute the following command:

msbuild SedodreamPackage.dproj

Lots of output is generated to the command prompt, and you will find two directories, Debug and Release, under the Package directory. Each of these directories will contain the generated installer and its associated files.

Customizing the Process

When you set up this process for your product, you will need to create a file similar to the SedodreamPackage.dproj file. You can place your customizations directly inside that file. It shouldn't be necessary to change the Sedodream.Package.targets file itself. If you want to inject steps, you can follow the process described previously. If you want to redefine targets, you can simply override them by redeclaring the target after the import statement. For example, the Sedodream.Package.targets file defines a target, GetLatest, but it is empty. In this target you should place the necessary tasks to get the latest sources from your repository. Here is an outline of what it might look like in your project:

<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003"

DefaultTargets="Package">

<!-- Your properties and items defined here -->

<!-- Import the deployment target to do all the work for us -->

<Import Project="Sedodream.Package.targets"/>

<!-- Place overriding customizations after this point -->

<Target Name="GetLatest">

<Message Text="Getting latest sources"/>

<!-- Insert tasks to get latest from source control -->

</Target>

</Project>

Figure 15 provides a list of important targets that are defined in the Sedodream.Package.targets file. You can customize these targets to suit your needs.

The remaining outstanding topic is how to automate this process, which is now extremely simple. All you have to do is automate the execution of MSBuild.exe on the deployment file, in this case SedodreamPackage.dproj. How this is accomplished will depend on how you are currently generating public builds. If you are using Visual Studio Team Foundation Server, you can use Team Build to perform this for you. You can use the MSBuild task in your TFSBuild.proj file. If you are using some other technologies, it is likely that they have native support for MSBuild. If you are not employing any of these types of technologies, you can use the Windows Scheduler to execute this on a timed basis.

Conclusion

In this article, I've shown how to create an automated build and package process based on WiX toolset. Once you have completed this integration you can build and package your products in a reliable and repeatable manner. This is very important for applications that are being sent into deployment.

If WiX is not your installer technology, you can still use the ideas and files, provided here with some modifications. All installer technologies that I know of support some form of a command-line execution. You would employ this in order to create your installers using these other technologies. Some third-party installer technologies have also shipped with MSBuild tasks and targets, which may assist you in recreating this process.

MSBuild and WiX Resources

- Inside MSBuild: Compile Apps Your Way with Custom Tasks for the Microsoft Build Engine

- Using the WiX Toolset to Integrate Setup into Your Development Process

- WiX Project Home

- Windows Installer SDK

- rRob Mensching’s Blog (WiX Project Lead)

- Justin Rockwood’s Blog (WiX Developer)

- Sayed Hashimi’s Blog

- MSDN MSBuild Forum

- MSBuild Team Blog

- MSBuild MSDN Documentation

- MSBuild MSDN Reference

- MSBuild WiKi

Sayed Ibrahim Hashimi has a computer engineering degree from the University of Florida. He is a developer and architect in Jacksonville, Florida. He is an expert in the financial, education, and collection industries. He works with .NET technologies at Latitude Software.

ASP.NET 2.0

Manage Web Users With Custom Profile Providers

Jason N. Gaylord

This article discusses:

- ASP.NET 2.0 profile providers

- Building a custom provider

- Authentication and authorization

This article uses the following technologies:

ASP.NET, XML

Code download available at: ProfileProviders2007_03.exe (268 KB)

Browse the Code Online

Contents

Using a Custom Profile Provider

Customizing the CreateUserWizard Control

Creating a Registration Procedure

Checking the Profile Version

Stepping through the Application

Where to Go from Here

With ASP.NET 2.0, you can add authentication, authorization, and profiles to your Web site without writing a single line of code. That's quite a step forward from the way things used to be. Like most of the new features in ASP.NET 2.0, authentication, authorization, and profiles each have their own built-in providers. You can think of providers as modules that contain the methods for a particular task. These providers are quite flexible-they can be customized by specifying attributes that pass information into the provider to allow it to execute different behavior. For extreme customization, though, it may be necessary to replace a built-in version with a custom or third-party provider.

The Profile feature in ASP.NET allows developers to specify per-user settings or data. All this data can be stored in an anonymous profile so that settings can be retrieved without requiring the user to log into the site. However, if the user creates an account, these settings can be migrated to the logged in profile. The anonymous profile feature can also be turned off.

To store and retrieve a user's profile properties, simply name each profile property and add them to the profile element in the web.config file, as shown in Figure 1. This puts the profile properties into the profile's SettingsPropertyValueCollection. You may also specify the data type and a default value for each property.

Figure 1 Profile Section in web.config

<profile>

<properties>

<add name="FirstName" />

<add name="LastName" />

<add name="Address1" />

<add name="Address2" />

<add name="City" />

<add name="State" />

<add name="Zip" />

<add name="Phone" />

<add name="ProfileVersion" type="int" defaultValue="0" />

</properties>

</profile>

ASP.NET reads the web.config file in search of the profile section group. If one is found, ASP.NET then looks for a type or connection string attributes (not included in Figure 1). If neither is found, the machine.config file is examined. In most scenarios, the machine.config file would not have been altered since ASP.NET was installed. By default, the machine.config file has the System.Web.Profile.SqlProfileProvider registered as the default provider for the profile feature. The SqlProfileProvider is defined to use the LocalSqlServer connection string name to connect to the target database for profile information (all connection string names and attributes are stored in a group called connectionStrings, found in the web.config file). Again by default, the LocalSqlServer connection string is defined as a SQL Server™ 2005 Express database that can be found at |DataDirectory|aspnetdb.mdf (translated in ASP.NET as \App_Data\aspnetdb.mdb).

The database used by the SqlProfileProvider to store profile information must be configured using a known schema. If a SQL Server 2005 Express database is being used, ASP.NET can dynamically create the database file and create the necessary tables, views, stored procedures, and so forth on demand; otherwise, the aspnet_regsql tool from the Microsoft® .NET Framework SDK can be used. One of the tables, the aspnet_Profile table, is used for storing the profile information for each user. When a user assigns a profile property to his profile, a new row is added that contains the user's unique UserID, the names of each of the profile properties in a single column, and the data that relates to each of the profile names in another column. In Figure 2, you can see that the data is stored as a BLOB. The property names and values can be in a variety of data formats including a continuous string, an XML dataset, or a binary serialization of the data. As a result, storing profile data using the built-in profile provider can make it difficult for a developer or database administrator to query or modify the data using SQL. If the default provider's functionality doesn't completely meet your needs, a custom provider can be used.

Figure 2 Property Names and Values (Click the image for a larger view)

Using a Custom Profile Provider

Even while developing a new version of ASP.NET, the ASP.NET team is hard at work enhancing the existing versions. The useful resources they produce are called sandbox projects and can be downloaded from the official ASP.NET Web site at sandbox.asp.net. One of the cool things about many of these projects is that they can be used in production environments. Some of the best add-ons for ASP.NET can be found here, including the Web Deployment Project and the Table Profile Provider Samples. In this project, I'll use the Table Profile Provider Samples, which you'll find at asp.net/sandbox/samp_profiles.aspx?tabid=62.

Unlike the built-in ASP.NET Profile provider, the Table Profile Provider Samples allow profile data to be stored in an easily readable format. The download contains two sample providers and instructions. The SqlTableProfileProvider allows data to be stored in individual columns within a profile table. The table must include the UserID unique identifier as the primary key and the LastUpdatedDate date time field. The UserID field will allow each profile record to be associated with the User account contained in the aspnet_Membership and aspnet_Users tables. You'll notice that neither of these fields need to be specified in the web.config file. The other profile sample, the SqlStoredProcedureProfileProvider, passes each profile property value into a separate parameter in a SQL stored procedure. This option will allow a stored procedure to store the profile properties in a location up to the stored procedure and to use business logic to transform the data before storing it.

Before using these samples in an ASP.NET project, you must copy the code for them to the App_Code folder. The code is written in C#, so if the application is being developed in Visual Basic®, the samples will need to be in a separate folder under the App_Code folder. The folder must be added as a code subdirectory in the web.config file as shown here:

<compilation debug="true" strict="false" explicit="true">

<codeSubDirectories>

<add directoryName="ProfileProvider" />

</codeSubDirectories>

</compilation>

By adding the folder as a code subdirectory, all C# files in the folder will be compiled separately from the rest of the Web application project. At run time, two DLLs will be generated in the ASP.NET temporary folder for the application.

For this application to be used with the SqlTableProfileProvider, I'll create a new table called CustomProfile that will contain columns to store the user's first and last name, full address, and phone number, along with a version number used to determine the current version of the user's profile. The table will also include the user's unique identifier as its primary key and the last updated date in the last column. For example, Figure 3 shows how the provider will be added to the profile section. Also note how each property will be added to the properties section group. In addition to a .NET type and default value, a custom provider data attribute will also need to be specified on each property. This attribute will include the column name from the SQL table and the SQL type of the column.

Figure 3 Profile Section

<profile defaultProvider="MyCustomProfileProvider">

<providers>

<add name="MyCustomProfileProvider"

type="Microsoft.SqlTableProfileProvider"

connectionStringName="LocalSqlServer"

table="CustomProfile"

applicationName="/"

profileVersion="1"

/>

</providers>

<properties>

<add name="FirstName" type="string" defaultValue="[null]"

customProviderData="FirstName;nvarchar" />

<add name="LastName" type="string" defaultValue="[null]"

customProviderData="LastName;nvarchar" />

<add name="Address1" type="string" defaultValue="[null]"

customProviderData="Address1;nvarchar" />

<add name="Address2" type="string" defaultValue="[null]"

customProviderData="Address2;nvarchar" />

<add name="City" type="string" defaultValue="[null]"

customProviderData="City;nvarchar" />

<add name="State" type="string" defaultValue="[null]"

customProviderData="State;nvarchar" />

<add name="Zip" type="string" defaultValue="[null]"

customProviderData="Zip;nvarchar" />

<add name="Phone" type="string" defaultValue="[null]"

customProviderData="Phone;nvarchar" />

<add name="ProfileVersion" type="int" defaultValue="[null]"

customProviderData="ProfileVersion;int" />

</properties>

</profile>

Customizing the CreateUserWizard Control

Not only does ASP.NET 2.0 have built-in providers to support its authentication features, it also has built-in Web controls for dealing with authentication and authorization. These include Login, LoginView, PasswordRecovery, LoginStatus, LoginName, CreateUserWizard, and ChangePassword. The CreateUserWizard control allows users to create an account within an ASP.NET application. Much like the providers, the controls require little if any coding. To use the CreateUserWizard control, or any of these controls for that matter, simply drag it from the Toolbox onto any ASP.NET page or user control.

CreateUserWizard is a two-step wizard control. The first step, called "Sign Up for Your New Account," asks the user for information such as username, password, e-mail address, and a security question and answer. In some instances, the Membership provider defined in the web.config may be configured where the requiresQuestionAndAnswer property is set to false. In such cases, the security question and security answer fields are hidden at run time. The second step, called "Complete," confirms that the user account was created and displays a Finish button. Clicking the button redirects the user to the URL specified in the ContinueDestinationPageURL property or runs an event handler for the button's click event if one was provided. Usually, a developer will redirect the user to the homepage or to the referring page using the ContinueDestinationPageURL property. For more detailed information on the CreateUserWizard control, see msdn.microsoft.com/msdnmag/issues/05/10/ExtremeASPNET and msdn.microsoft.com/msdnmag/issues/04/11/CuttingEdge.

Creating a Registration Procedure

There are a few ways to create a custom user registration process. First, you can use the built-in CreateUserWizard control and modify it with an additional action in front of the Sign Up for Your New Account step. Second, you can convert each step to a navigation template and customize each control step as you see fit. This lets you move the user information and profile information to different steps or to leave all of the fields in the same step. Finally, you can create your own registration control using the Membership API's CreateUser method. The solution you choose depends on the design of the application and the level of customization you'd like to add. In this case, you'll add a step in front of Sign Up for Your New Account as this will let you add your profile form with minimal coding.

The new step will ask the user to submit values for each of the profile properties mentioned earlier, with the exception of the ProfileVersion property. As a reminder, the UserID and LastUpdatedDate will be populated by the provider and don't need values supplied by the user. The ProfileVersion property will be populated when the form is submitted by retrieving the value from an application setting or from another section in web.config. You can find out more about creating your own configuration sections at msdn.microsoft.com/msdnmag/issues/06/06/ConfigureThis.

Figure 4 shows the property being added to the SqlTableProfileProvider that has been provided by Microsoft. We will use the Profile sectionGroup to specify the value of the profile version. The Profile will be updated immediately after the user's account has been created. An object defined as the ProfileCommon will load the user's current profile properties and values. Since the profile and user account are new to the system, all of the values will be null. All of the profile properties will be updated to contain non-null values. Finally, the ProfileCommon will be saved so the data can be written back to the database using the provider. If the ProfileCommon isn't saved, there is no way for ASP.NET to know that a data write is needed.

Figure 4 SqlTableProfileProvider Class

public class SqlTableProfileProvider : ProfileProvider

{

...

private string _profileVersion;

public override void Initialize(

string name, NameValueCollection config)

{

...

// Check to see that a profile version exists

_profileVersion = config["profileVersion"];

if (string.IsNullOrEmpty(_profileVersion))

{

throw new ProviderException("No profile version specified");

}

// Validate the profile version

int testProfileVersion;

if (!Int32.TryParse(_profileVersion, out testProfileVersion))

{

throw new ProviderException(

"Profile version must be an Integer");

}

...

config.Remove("profileVersion");

}

// Public property to obtain the ProfileVersion from the web.config

static public Int32 ProfileVersion

{

get

{

ProfileSection profileConfig = (ProfileSection)

WebConfigurationManager.GetSection("system.web/profile");

return Int32.Parse(profileConfig.Providers[1].

ElementInformation.Properties["profileVersion"].

Value.ToString());

}

}

Checking the Profile Version

One of the benefits of using ASP.NET is that it's modular. Since ASP.NET 1.0, developers could create an HTTP handler or module for ASP.NET. You can write an HTTPModule to see if a profile exists for a user and if it is the correct version. If a profile does not exist or if it's the wrong version, you could force the user to go to a centralized update-profile page.

To determine the profile version that a user is currently registered with, you need to handle the application's AuthorizeRequest method. In doing so, your own custom method will run when the user is authenticated and before the user is granted access to the page he wants (see Figure 5). Since you won't be able to pull the profile on users not logged in, you'll need to exclude those requests from your examination. You'll also want to exclude users who are currently on your editprofile.aspx page so they are not caught in a cycle of continuous redirection. You'll then obtain the value for the profileVersion property on the second registered profile provider, which will be the SqlTableProfileProvider. You don't want to pull the first provider in the provider collection because that will always be the built-in profile provider unless you explicitly removed it from the providers collection. Once you've pulled the current profile version on the application, load the user account's profile and check the profile version in the profile. To do this, create a new ProfileCommon object and load the user's profile based on his name. If the user's profile version is less than the application's profile version, the user will be redirected to the editprofile.aspx page. If it is the same, the page the user was attempting to access will load.

Figure 5 ProfileHttpModule Class

Public Class ProfileHttpModule

Implements IHttpModule

Public Sub Dispose() Implements System.Web.IHttpModule.Dispose

End Sub

Public Sub Init(ByVal context As System.Web.HttpApplication) _

Implements System.Web.IHttpModule.Init

AddHandler context.AuthorizeRequest, AddressOf Me.CheckProfile

End Sub

Public Sub CheckProfile(ByVal s As Object, ByVal e As EventArgs)

Dim objApp As HttpApplication = CType(s, HttpApplication)

Dim objContext As HttpContext = objApp.Context

'User is authenticated, check profile

If objApp.User.Identity.IsAuthenticated And _

objContext.Request.Path <> "editprofile.aspx" Then

Dim ProfileVersion As Integer = _

SqlTableProfileProvider.ProfileVersion

Dim UserProfile As New ProfileCommon

If CInt(UserProfile.GetProfile(objApp.User.Identity.Name). _

ProfileVersion.ToString) < ProfileVersion And _

!objContext.Request.Path.EndsWith("editprofile.aspx") _

Then

objContext.Response.Redirect("~/editprofile.aspx")

End If

End If

End Sub

End Class

Before you can build the application, make sure that your web.config file is configured for Forms authentication. If you don't, the application will assume that you are using Windows® authentication as defined in the machine.config file. The web.config file will also need a location section group to deny any users that are not authenticated access to the edit profile page. You must also add a login control so that your users can log into the site. You can add the control to your main page or to a separate login page. It may be a good idea to add a register link to the login control so that your users can find the registration page.

Stepping through the Application

To review, when a user visits the site and clicks on the link asking him to create a user account, he'll be redirected to the registration page and asked to fill in his profile information. When he clicks Next, he'll be asked to enter a username and password and will then be redirected to the default page. During each page load, the HTTPModule will check to see if the profile version specified in the web.config file is greater than the user's profile version. If it is, the user will be redirected to the edit profile page. If not, he'll continue browsing.

To ensure that the module is working properly, add a column to the custom profile table. The column must allow null values or you'll receive an error message when a user attempts to browse a page. This is because that property is not established yet for all users and has not been added to the web.config file. Add it and change the profile from version 1 to 2. When you save the web.config file and log onto the site, you can't browse to any page without updating the profile first, as expected.

The one catch with using the web.config file to store the profile version is that if you modify it when users are on the Web site, the application will reset and the users will be kicked off. Then they'll be redirected to the edit profile page on their next login attempt. With this in mind, you may want to create a simple text file or XML file to read the version number of the profile.

Where to Go from Here

If you need some help getting started, you can download a Visual Studio® Installer (VSI) project that I've put together. It includes full source code written in Visual Basic and ASP.NET and provides a great foundation for beginning your application. It can be downloaded from the MSDN®Magazine Web site. I've explored only a very small portion of the customization possibilities with the ASP.NET 2.0 Profile provider; I'm sure you'll discover lots more if you look. That's what's great about .NET-it's like having your own personal sandbox.

Jason N. Gaylord is the Project Manager and Senior Application Developer at a leading risk management information services company in Wilkes-Barre, Pennsylvania. He also runs the .NET Valley User Group. For more information, visit www.jasongaylord.com or contact him at jason@jasongaylord.com.

SQL Server 2005

Regular Expressions Make Pattern Matching And Data Extraction Easier

David Banister

This article discusses:

- Efficient SQL querying using regular expressions

- Support in SQL Server 2005 for regular expressions

- Using .NET Regex classes from SQL Server

- Effective uses for regular expressions in a database

This article uses the following technologies:

SQL Server 2005, .NET Framework

Code download available at: Regex2007_02.exe (154 KB)

Browse the Code Online

Contents

CLR User-Defined Functions

Pattern Matching

Data Extraction

Pattern Storage

Matches

Data Extraction within Matches

Conclusion

Although T-SQL is extremely powerful for most data processing, it provides little support for text analysis or manipulation. Attempting to perform any sophisticated text analysis using the built-in string functions results in massively large functions and stored procedures that are difficult to debug and maintain. Is there a better way?

In fact, regular expressions provide a much more efficient and elegant solution. It is easy to see how they would be helpful for comparing text to identify records, but there is so much more they can do. I'll show you how to perform a variety of tasks, some simple and some amazing, that were considered impractical or impossible in SQL Server™ 2000, but are now possible in SQL Server 2005 due to its support for hosting the common language runtime (CLR).

Regular expressions are not new to SQL. Oracle introduced built-in regular expressions in 10g, and many open source database solutions use some kind of regular expressions library. Regular expressions could actually be used in earlier versions of SQL Server, but the process was inefficient.

Using the sp_OACreate stored procedure, any OLE automation object that implemented regular expressions could be used, but you had to create a COM object first, then make at least one IDispatch call, then destroy the object. For most purposes this was too inefficient and caused too many performance problems. The only alternative was to create an extended stored procedure. However, now there's SQLCLR, a CLR user-defined function (UDF) that lets you create an efficient and less error-prone set of functions using the Microsoft® .NET Framework.

CLR User-Defined Functions

CLR user-defined functions are simply static methods (shared functions in Visual Basic) defined within a .NET assembly. To use the SQLCLR objects, you must register the assembly with SQL Server using the new CREATE ASSEMBLY statement and then create each object pointing to its implementation within the assembly. For functions, the CREATE FUNCTION statement has been extended to support the creation of CLR user-defined functions. To make things easier, Visual Studio® 2005 takes care of all of the registration processes on your behalf when you use a SQL Server Project. This kind of project is different than most Visual Studio projects because when you attempt to debug (or start without debugging), the project is recompiled and the resulting assembly, as well as all of the SQLCLR objects defined within it, are then deployed to and registered with SQL Server. The IDE then runs the test script designated for the project. Breakpoints can be set in both the SQL script and in your .NET code, making debugging a simple process.

Adding a function is just like adding a new class to any other project type. Simply add a new item to the project and select User-Defined Function when prompted. The new method is added to the partial class that contains all of your functions. Your new method will also have a SqlFunction attribute applied to it. It is used by Visual Studio to create the SQL statements necessary to register the function. The IsDeterministic, IsPrecise, DataAccess, and SystemDataAccess fields on SqlFunction are also used by SQL Server for a variety of purposes.

Pattern Matching

Determining if a string matches a pattern is the simplest use of regular expressions and, as you see in Figure 1, it's, easy to do.

Figure 1 String Matching

public static partial class UserDefinedFunctions

{

public static readonly RegexOptions Options =

RegexOptions.IgnorePatternWhitespace |

RegexOptions.Singleline;

[SqlFunction]

public static SqlBoolean RegexMatch(

SqlChars input, SqlString pattern)

{

Regex regex = new Regex( pattern.Value, Options );

return regex.IsMatch( new string( input.Value ) );

}

}

First I use the Options field to store the regular expression options for the functions. In this case, I've selected RegexOptions.SingleLine and RegexOptions.IgnorePatternWhitespace. The former specifies single-line mode, and the latter eliminates unescaped whitespace from the regular expression and enables comments marked with a pound sign. Another option you might want to use after careful thought and analysis is RegexOption.Compiled. As long as there aren't too many, you will see significant performance gain if you use Compiled on expressions that are heavily used. Expressions that are used over and over again should definitely be compiled. However, for regular expressions that are rarely used, do not use Compiled as it causes increased startup costs and memory overhead. As such, you might want to augment my general purpose RegexMatch function with an additional parameter that specifies whether you want the expression to be compiled; that way, you can decide on a case by case basis whether the additional overhead will be worth the resulting performance gains.

After specifying the RegexOptions to be used, I define the RegexMatch function, using the SqlChars data type instead of SqlString. The SqlString data type translates into nvarchar(4,000) while SqlChars translates into nvarchar(max). The new max size functionality allows strings to extend beyond the 8,000 byte limit of SQL Server 2000. Throughout this article, I use nvarchar(max) to be as generic as possible and to allow for the most flexibility. However, performance can be significantly better using nvarchar(4,000) if all the relevant strings contain less than 4,000 characters. You should examine your specific needs and code to them appropriately.

The remaining code in the method is simple. A Regex instance is created with the defined options and the provided pattern and the IsMatch method is then used to determine if the specified input matches the pattern. Now you need to add a simple query to the test script:

select dbo.RegexMatch( N'123-45-6789', N'^\d{3}-\d{2}-\d{4}$' )

The pattern in this statement is a simple test for a US Social Security Number. Set a breakpoint on the new query and then start debugging to step through the function. This function lets you do many different tests, but I'll show you some things most people don't consider. For example, it's very important to maintain consistent naming conventions within a database, and writing a query to validate that all of your stored procedures meet your organization's guidelines is difficult. The RegexMatch function makes this task much simpler. For example, the following query test performs this task:

select ROUTINE_NAME

from INFORMATION_SCHEMA.ROUTINES

where ROUTINE_TYPE = N'PROCEDURE'

and dbo.RegexMatch( ROUTINE_NAME,

N'^usp_(Insert|Update|Delete|Select)([A-Z][a-z]+)+$' ) = 0

This query tests that every stored procedure is prefixed with "usp_" followed by "Insert", "Update", "Delete", or "Select", followed by at least one entity name. It also verifies that each word in the entity begins with a capital letter. Compare those four lines with this oversimplified version using only built-in functions:

select ROUTINE_NAME

from INFORMATION_SCHEMA.ROUTINES

where ROUTINE_TYPE = N'PROCEDURE'

and ( LEN( ROUTINE_NAME ) < 11

or LEFT( ROUTINE_NAME, 4 ) <> N'usp_'

or SUBSTRING( ROUTINE_NAME, 5, 6 ) not in

( N'Insert', N'Update', N'Delete', N'Select' ) )

Even though this is more code, this query is actually missing several features present in the regular expressions version. First, it is not case-sensitive, and using collations within the query to perform the tests would make it unruly. Second, it does not perform any tests on the actual entity name contained within the procedure name. The third problem is the four strings tested in the query are all six characters long, which allowed me to simplify the code by pulling out a single substring of six characters that I could then compare against each of the acceptable operations. This is not problematic specifically in this example, since all of the operation names are six characters long, but imagine a standard that specifies more complex verbs like "Get", "List", or "Find". Those verbs are easily handled by the RegexMatch function, because they are just additional alternatives in the list.

Validation is a very common use of regular expressions, for anything from a phone number to a ZIP code to a custom account number format. The CHECK constraint is perfect for this, as the following table definition shows.

CREATE TABLE [Account]

(

[AccountNumber] nvarchar(20) CHECK (dbo.RegexMatch(

[AccountNumber], '^[A-Z]{3,5}\d{5}-\d{3}$' ) = 1),

[PhoneNumber] nchar(13) CHECK (dbo.RegexMatch(

[PhoneNumber], '^\(\d{3}\)\d{3}-\d{4}$' ) = 1),

[ZipCode] nvarchar(10) CHECK (dbo.RegexMatch(

[ZipCode], '^\d{5}(\-\d{4})?$' ) = 1)

)

The AccountNumber column is validated against an arbitrary convention that begins with three to five letters followed by five numbers, then a dash, and finally three more numbers. Both the phone number and ZIP codes are validated against standard US phone number and ZIP code formats. The RegexMatch function provides many features to SQL Server, but the regular expressions implementation in .NET provides much more, as you'll see next.

Data Extraction

The grouping features of regular expressions can be used to extract data from a string. My RegexGroup function provides that functionality to T-SQL:

[SqlFunction]

public static SqlChars RegexGroup(

SqlChars input, SqlString pattern, SqlString name )

{

Regex regex = new Regex( pattern.Value, Options );

Match match = regex.Match( new string( input.Value ) );

return match.Success ?

new SqlChars( match.Groups[name.Value].Value ) : SqlChars.Null;

}

This function creates a Regex object just as the RegexMatch function did. Instead of testing for a match, however, a Match object is created for the first match found in the input string. The Match object is used to retrieve the group that is specified. If no match is found within the input, a null value is returned. If you prefer to use numbered groups instead of named groups, this function will still work for you. Just pass an integer value to the function within your SQL code and it will be implicitly cast to an nvarchar and the appropriate group will be returned.

You can use the RegexGroup function within a SELECT list to extract specific pieces of information from some other piece of data. For example, if you had a column that stored a URL, you can now easily parse the URL to determine individual pieces. This query uses grouping to determine every distinct server stored in the Url column of the UrlTable table.

select distinct dbo.RegexGroup( [Url],

N'https?://(?<server>([\w-]+\.)*[\w-]+)', N'server' )

from [UrlTable]

You can also use this function within computed columns. The following table definition divides e-mail addresses into the mailbox and the domain.

CREATE TABLE [Email]

(

[Address] nvarchar(max),

[Mailbox] as dbo.RegexGroup( [Address],

N'(?<mailbox>[^@]*)@', N'mailbox' ),

[Domain] as dbo.RegexGroup( [Address], N'@(?<domain>.*)', N'domain' )

The mailbox column will return the mailbox or username of the e-mail address. The domain column will return the domain of the e-mail address.

Pattern Storage

All of the patterns used by these functions are just strings, which means that any of them can be stored in a table within your database. Most databases that store international data have a table representing countries. By adding a few extra columns to that table, you could store country-specific validation patterns. That would allow the constraint applied to an address row to vary based on the country for that row.

In databases that store data on behalf of clients, there is typically already a table representing a client. That table can be used to store grouping patterns that let you describe the way raw client data is stored within the database, and this allows you to create computed columns to pull the data you actually need from the client data. For example, if each of your clients has unique schemes for account numbers and you only need specific pieces of that account number, you could easily create an expression that pulls the correct piece of information for each client.

Matches

Rather than determining if a string matches a pattern, it is sometimes desirable to extract every match. Previously this kind of extraction would require cursors iterating over sections of a string. Not only is that process slow, but the code is also difficult to understand and maintain. Regular expressions are a much better means of performing this operation. The problem at hand is how to return all of the required data within a SQL construct. Table-valued functions are the answer.

Table-valued functions are somewhat similar to the previous functions, but vary in two distinct ways. First, the attributes applied to the method must fully declare the structure of the table that is returned. Second, there are two methods involved. The first returns an enumerable object instead of the actual result of the function. The second method is passed the enumerated objects to populate the fields of each row. Each value retrieved via the enumerator should correspond with one row of the resultset. The ICollection interface in the .NET Framework implements IEnumerable which means that any collection can be returned by the first method. The Regex class contains a Matches method that returns a MatchCollection that you could use. The problem with the MatchCollection is that the entire string must be processed prior to the Matches method returning. SQL Server includes optimizations that depend on processing occurring as needed, so instead of returning the entire collection up front, I prefer to write my own enumerator that returns each match as requested. This decision really depends on how the function is used and should be heavily tested prior to optimizing the enumerator.

The code in Figure 2 shows the enumerator. The MatchNode class wraps an individual match in the string while tracking its position within the set of matches returned. The MatchIterator class is enumerable and handles the regular expression processing. It uses the new yield keyword to create the enumerator much more easily than previous versions of the framework. It will return each match detected within the input string as requested.

Figure 2 Custom Enumerable Object for Matches

internal class MatchNode

{

private int _index;

public int Index { get{ return _index; } }

private string _value;

public string Value { get { return _value; } }

public MatchNode( int index, string value )

{

_index = index;

_value = value;

}

}

internal class MatchIterator : IEnumerable

{

private Regex _regex;

private string _input;

public MatchIterator( string input, string pattern )

{

_regex = new Regex( pattern, UserDefinedFunctions.Options );

_input = input;

}

public IEnumerator GetEnumerator()

{

int index = 0;

Match current = null;

do

{

current = (current == null) ?

_regex.Match( _input ) : current.NextMatch( );

if (current.Success)

{

yield return new MatchNode( ++index, current.Value );

}

}

while (current.Success);

}

}

The code in Figure 3 defines the table-valued CLR UDF. The RegexMatches method returns a new MatchIterator. The SqlFunctionAttribute on the RegexMatches method also includes some additional properties. The TableDefinition property is set to the table definition of the function. The FillRowMethodName is set to the name of the method to call for each iteration of the enumerable object returned. In this case, that method is FillMatchRow.

Figure 3 Table-Valued CLR UDF for Matches

[SqlFunction( FillRowMethodName = "FillMatchRow",

TableDefinition = "[Index] int,[Text] nvarchar(max)" )]

public static IEnumerable RegexMatches(SqlChars input, SqlString pattern)

{

return new MatchIterator( new string( input.Value ), pattern.Value );

}

[SuppressMessage( "Microsoft.Design", "CA1021:AvoidOutParameters" )]

public static void FillMatchRow( object data,

out SqlInt32 index, out SqlChars text )

{

MatchNode node = (MatchNode)data;

index = new SqlInt32( node.Index );

text = new SqlChars( node.Value.ToCharArray( ) );

}

For each iteration of the MatchIterator, a MatchNode is passed to the FillMatchRow method as its first argument. The remaining parameters of the FillMatchRow method must be declared as out parameters and must match the table definition defined in the first function. The FillMatchRow function simply uses the properties of the MatchNode to populate the field data.

With this function you can finally extract multiple pieces of data from a string with ease. To illustrate the use of the RegexMatches function, let's process a string to determine how many distinct words are contained within it using this query:

declare @text nvarchar(max), @pattern nvarchar(max)

select

@text = N'Here are four words.',

@pattern = '\w+'

select count(distinct [Text])

from dbo.RegexMatches( @text, @pattern )

This example is rather straightforward. It shows some potential for using the function but by removing the distinct keyword, it returns the total word count of a string. There are many Web sites that limit text entry to what seems like an arbitrary length. With this kind of test combined with the new nvarchar(max) notation, it becomes possible to limit input to a word count instead. This kind of query can be used for various analytic processing needs, but the RegexMatches function can also be used for more common tasks. Unfortunately, this kind of query also represents an overzealous use of regular expressions. The splitting operation accomplished by the "\w+" expression in this case could be just as easily accomplished with the String.Split method, which would be much faster. Regular expressions are a very powerful tool, but do make sure when you use them that you're using them for a good reason: there might be simpler tools you can use for specific cases that would yield better performance.

I often see questions in the MSDN® forums about how to pass a list of values to a stored procedure. I have also seen various convoluted methods of parsing such a list into an actual list to determine the correlated records. The RegexMatches function provides a much cleaner approach.

declare @pattern nvarchar(max), @list nvarchar(max)

select @pattern = N'[^,]+', @list = N'2,4,6'

select d.* from [Data] d

inner join dbo.RegexMatches( @list, @pattern ) re

on d.[ID] = re.[Text]

The pattern matches any group of characters not containing a comma. Given a table named Data with an integer column named ID, this query will return each record identified in the list. This becomes more useful when considering the implicit casting features within SQL Server. The same query can be used for integer, date/time, GUID, or floating-point data types. Other methods of processing a list of values would require multiple functions or stored procedures to be this flexible. This function can also be used for lists that are not comma-delimited. A list separated by spaces, semicolons, tabs, carriage returns, or any other identifiable character can be processed.

Data Extraction within Matches

Similar to returning matches, we can also extract data from each match. Attempting to do this using SQL is very difficult. Normally, this kind of task would be implemented within an application instead of the database, which causes problems because each application using the database would have to implement the required processing. In such scenarios, a reasonable approach might be to implement this functionality within stored procedures.

As with the RegexMatches implementation, I prefer to use a custom enumerable object to return group information. Grouping is only slightly more complicated because we also have to iterate over groups within each match. In Figure 4, the GroupNode class is just like the MatchNode class except that it also includes the name of group it represents. The GroupIterator class is similar to the MatchIterator class but includes an additional loop to return each group. Now that I have an enumerable object, I define a table-valued function just as I did with the RegexMatches function.

Figure 4 Custom Enumerable Object for Groups

internal class GroupNode

{

private int _index;

public int Index { get { return _index; } }

private string _name;

public string Name { get { return _name; } }

private string _value;

public string Value { get { return _value; } }

public GroupNode( int index, string group, string value )

{

_index = index;

_name = group;

_value = value;

}

}

internal class GroupIterator : IEnumerable

{

private Regex _regex;

private string _input;

public GroupIterator( string input, string pattern )

{

_regex = new Regex( pattern, UserDefinedFunctions.Options );

_input = input;

}

public IEnumerator GetEnumerator()

{

int index = 0;

Match current = null;

string[] names = _regex.GetGroupNames();

do

{

index++;

current = (current == null) ?

_regex.Match( _input ) : current.NextMatch( );

if (current.Success)

{

foreach(string name in names)

{

Group group = current.Groups[name];

if (group.Success)

{

yield return new GroupNode(

index, name, group.Value );

}

}

}

}

while(current.Success);

}

}

In Figure 5, the RegexGroups function is defined like the RegexMatches function except that it returns an additional column of data containing the name of the group within the match. With this function we can now find multiple matches within a string and extract specific pieces of the information from within each match.

Figure 5 Table-Valued CLR UDF for Groups

[SqlFunction( FillRowMethodName = "FillGroupRow", TableDefinition =

"[Index] int,[Group] nvarchar(max),[Text] nvarchar(max)" )]

public static IEnumerable

RegexGroups( SqlChars input, SqlString pattern )

{

return new GroupIterator( new string( input.Value ), pattern.Value );

}

[SuppressMessage( "Microsoft.Design", "CA1021:AvoidOutParameters" )]

public static void FillGroupRow( object data,

out SqlInt32 index, out SqlChars group, out SqlChars text )

{

GroupNode node = (GroupNode)data;

index = new SqlInt32( node.Index );

group = new SqlChars( node.Name.ToCharArray( ) );

text = new SqlChars( node.Value.ToCharArray( ) );

}

Importing data in various formats is a common task when dealing with databases. Importing files in a comma-delimited format is much more of a chore than it should be. Most developers create an application that processes each line, extracts the data, and then executes a stored procedure for each line. Although that process works, I'd like to propose another solution. What if you could pass the entire file to a stored procedure and let the stored procedure handle the entire process? This idea is usually considered too complicated to implement, but with the RegexGroups function you can actually perform this insert with a single query. For example, consider the following customer data.

2309478,Janet Leverling,J

2039748,Nancy Davolio,N

0798124,Andrew Fuller,M

4027392,Robert King,L

There are three different pieces of information you need from each line: the seven-digit customer number, the customer name, and the single character customer type. With the following expression you can extract all three pieces of information.

(?<CustomerNumber>\d{7}),(?<CustomerName>[^,]*),(?<CustomerType>[A-Z])\r?\n

The problem you now have is that the results returned by the RegexGroups function are not directly usable. Instead of using a cursor to iterate over the results, you can use the pivot functionality in SQL Server 2005. Putting all of this together into a stored procedure, you have everything you need. The stored procedure in Figure 6 accepts the text of an entire comma-delimited file containing up to 2GB of Unicode data. It processes the entire file and inserts each line of the file as a row into the Customer table. Any delimited text file could be processed the same way. With some small changes to the pattern, escape sequences could be added to support commas within strings.

Figure 6 Processing a Comma-Delimited File

create proc ImportCustomers

(

@file nvarchar(max)

)

as

declare @pattern nvarchar(max)

set @pattern = N'(?<CustomerNumber>\d{7}),

(?<CustomerName>[^,]*),(?<CustomerType>[A-Z])\r?\n'

insert [Customer]

(

[CustomerNumber],

[CustomerName],

[CustomerType]

)

select

f.[CustomerNumber],

f.[CustomerName],

f.[CustomerType]

from dbo.RegExGroups( @file, @pattern ) regex

pivot

(

max([Text])

for [Group]

in ( [CustomerNumber], [CustomerName], [CustomerType] )

) as f

Again, however, this procedure also demonstrates that there are multiple ways to do the same task, and sometimes regular expressions aren't always the best option. In this example, using a pivot is, in effect, undoing all the work that RegexGroups did to return the data in the special grouped format. One could insert the data directly into the table using a much simpler and faster TVF that just read each line, String.Split on commas, and returned each row.

Conclusion

Although these matching functions are very powerful, they are not complete. There are many possible options that determine the exact way matches are performed. If your database collation is case-insensitive, you may want the functions to perform their matches in a case-insensitive manner as well. The explicit capture option may be required to reduce some resultsets. The multi-line option can allow you to create more precise patterns for some tasks. You may even want to create a user-defined type to pass the exact required options to each of the functions, which would allow each execution of a function to use a different set of options.

You should also be aware that there are localization issues when processing text. For example, the .NET Framework Regex class is aware of many more characters than the Latin ones used in my examples, so care should be taken when developing patterns for databases that use international data.

And of course, as mentioned several times in the article, while regular expressions are immensely powerful, make sure you actually need that power. Some tasks can be performed faster and more simply with more basic tool sets.

For simplicity, the examples I provided lack validation and error handling, which should be included in any production system. Each of the inputs to the function should be validated and your requirements should determine the responses to null or empty string inputs. The Regex class can throw exceptions when the pattern can't be parsed or the options are invalid. Those exceptions should be handled gracefully.

Combining regular expressions with SQL provides many alternative means of processing data. Using these functions can reduce the amount of time required to add functionality to your database as well as make the system more maintainable. Any database could use regular expressions, and I recommend that you experiment with these functions to find new and even more creative uses.

David Banister is a Senior Software Developer for a top-4 accounting firm in Atlanta. He has been writing software for many years. In his spare time he enjoys reading ECMA language specifications, playing tennis, and helping local bands.

Windows Vista and Office

View Data Your Way With Our Managed Preview Handler Framework

Stephen Toub

Code download available at: PreviewHandlers2007_01.exe (276 KB)

Browse the Code Online

This article discusses:

- Writing managed add-ins for Windows and Office

- Implementing a managed preview handler framework

- Developing custom preview handlers for any file extension

- COM interop

This article uses the following technologies:

.NET Framework, Windows Vista, Outlook

Contents

Hosting a Preview Handler

The PreviewHandler Framework

Implementing PreviewHandler

Using the COM Interfaces

Registering Handlers

Debugging

Conclusion

Every version of Microsoft Windows and Office brings new methods and approaches for improving your ability to see, interact with, and understand data. Likewise, each version also presents new and improved extensibility points for plugging in custom functionality. In Windows Vista™ and in the 2007 Microsoft® Office system, these two areas of advancements have merged, giving you the ability to write custom preview handlers for Windows Vista and for Outlook® 2007.

Outlook 2003 provided a reading pane for e-mail that made it easy to view the contents of a message without having to open it. You simply selected the message in the mail folder's list view, and the message was rendered in a side or bottom window. Outlook 2007 extends this concept by allowing you to view message attachments in that same preview pane, without having to double-click an attachment to open it in the appropriate viewer.

Out of the box, Outlook includes preview handlers for Word documents, PowerPoint® presentations, Excel spreadsheets, font files, video and audio files, and a variety of other file types that are commonly sent as attachments. But that's only the beginning. Windows Vista supports a similar preview pane that is accessible from any folder in the shell. You can enable the preview pane by selecting Organize | Layout | Preview Pane from the folder's menu (see Figure 1).

Figure 1 Enabling Previews in Windows Vista Folders

Both Outlook and Windows Vista subscribe to the same underlying preview mechanism, and they allow developers to implement custom preview handlers for any file type, register them, and instantly gain preview capabilities for those file types in Outlook 2007 and Windows Vista.

In this article, I'll explain what is required to implement a preview handler and discuss how to do so using managed code (the Windows® SDK for Vista includes a sample preview handler written in native C++). The code download for this article includes a framework that makes it a snap to implement your own preview handlers, and it provides several sample previewers (including previewers for PDF, XML, ZIP, MSI, BIN, CSV, XPS, and XAML files).

Hosting a Preview Handler

Those of you who have attempted to implement managed add-ins for the Windows shell in the past might be a bit uneasy about this concept. After all, Microsoft strongly advises that shell add-ins not be implemented in managed code, and such add-ins are not considered a supported scenario. This is because add-ins are loaded in-process into the shell (explorer.exe), only one version of the common language runtime (CLR) can be loaded into a given process, and managed code built against one version of the runtime may not run in a process that is running an earlier version of the runtime. What happens, then, if you have two shell add-ins that are both written in managed code-one targeting the .NET Framework 1.1 and one targeting the .NET Framework 2.0? If the 2.0-targeted add-in is loaded first, you probably won't notice any problems; the 1.1 add-in should load and run successfully against the CLR 2.0. But if the 1.1-targeted add-in loads first, the .NET Framework 1.1 CLR will load into explorer.exe. Assemblies targeting the .NET Framework 2.0 cannot be loaded by the .NET Framework 1.1, and therefore the 2.0-targeted add-in will fail when loaded after the 1.1-targeted add-in.

The same versioning problems exist with Windows Vista, so Microsoft still advises that shell add-ins shouldn't be implemented in managed code-even for new extensibility points within the shell, such as thumbnail providers and property handlers (which are used out-of-process by the Windows Vista search indexer but in-process by the Windows Vista shell).

There is good news, however, regarding preview handlers: they are always loaded out-of-process, at least as far as the shell is concerned. Preview handlers are implemented as COM components, but they're not hosted by the Windows Vista shell. Instead, a preview handler is either hosted by the preview handler surrogate host (prevhost.exe) or implemented as a local COM server. For managed code, implementing the latter is well beyond the scope of this article (for an overview of what's involved, see the "Preview Handlers and Windows XP" sidebar by Ryan Gregg). Besides, using prevhost.exe is the preferred, Microsoft-recommended service model.

The PreviewHandler Framework

To be a valid preview handler, several interfaces must be implemented (all of which are documented at windowssdk.msdn.microsoft.com/aa361576.aspx). This includes IPreviewHandler (shobjidl.h); IInitializeWithFile, IInitializeWithStream, or IInitializeWithItem (propsys.h); IObjectWithSite (ocidl.h); and IOleWindow (oleidl.h). There are also optional interfaces, such as IPreviewHandlerVisuals (shobjidl.h), that a preview handler can implement to provide extended support.

If you were to write a preview handler in native code, you could dive right in, given that all of these interfaces are already defined and ready for inclusion from the header files I just noted parenthetically. However, to write a preview handler in managed code, you must first write or obtain COM interop definitions for each of these interfaces. My definitions are shown in Figure 2.

Figure 2 COM Interfaces Interop

[ComImport]

[InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]

[Guid("8895b1c6-b41f-4c1c-a562-0d564250836f")]

interface IPreviewHandler

{

void SetWindow(IntPtr hwnd, ref RECT rect);

void SetRect(ref RECT rect);

void DoPreview();

void Unload();

void SetFocus();

void QueryFocus(out IntPtr phwnd);

[PreserveSig]

uint TranslateAccelerator(ref MSG pmsg);

}

[ComImport]

[InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]

[Guid("8327b13c-b63f-4b24-9b8a-d010dcc3f599")]

interface IPreviewHandlerVisuals

{

void SetBackgroundColor(COLORREF color);

void SetFont(ref LOGFONT plf);

void SetTextColor(COLORREF color);

}

[ComImport]

[InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]

[Guid("b7d14566-0509-4cce-a71f-0a554233bd9b")]

interface IInitializeWithFile

{

void Initialize([MarshalAs(UnmanagedType.LPWStr)] string pszFilePath,

uint grfMode);

}

[ComImport]

[InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]

[Guid("b824b49d-22ac-4161-ac8a-9916e8fa3f7f")]

interface IInitializeWithStream

{

void Initialize(IStream pstream, uint grfMode);

}

[ComImport]

[InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]

[Guid("fc4801a3-2ba9-11cf-a229-00aa003d7352")]

public interface IObjectWithSite

{

void SetSite([In,MarshalAs(UnmanagedType.IUnknown)] object pUnkSite);

void GetSite(ref Guid riid,

[MarshalAs(UnmanagedType.IUnknown)] out object ppvSite);

}

[ComImport]

[Guid("00000114-0000-0000-C000-000000000046")]

[InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]

public interface IOleWindow

{

void GetWindow(out IntPtr phwnd);

void ContextSensitiveHelp(

[MarshalAs(UnmanagedType.Bool)] bool fEnterMode);

}

In my managed framework for implementing preview handlers, most of the functionality for these interfaces is wrapped up into an abstract base class, called PreviewHandler:

public abstract class PreviewHandler :

IPreviewHandler, IPreviewHandlerVisuals, IOleWindow, IObjectWithSite

{ ... }

This hides all of the gory implementation details (which, as you'll see, aren't really gory at all). Note, however, that neither IInitializeWithFile nor IInitializeWithStream are implemented by PreviewHandler. Those honors are bestowed upon two abstract classes that derive from PreviewHandler:

public abstract class StreamBasedPreviewHandler :

PreviewHandler, IInitializeWithStream { ... }

public abstract class FileBasedPreviewHandler :

PreviewHandler, IInitializeWithFile { ... }

To implement a custom preview handler, you derive a new class from StreamBasedPreviewHandler or FileBasedPreviewHandler and override one method. Here's the implementation from my XmlPreviewHandler class:

public sealed class XmlPreviewHandler : FileBasedPreviewHandler

{

protected override PreviewHandlerControl

CreatePreviewHandlerControl()

{

return new XmlPreviewHandlerControl();

}

}

The CreatePreviewHandlerControl method returns an instance of a custom type that you write derived from either StreamBasedPreviewHandlerControl or FileBasedPreviewHandlerControl. Both derive from my PreviewHandlerControl abstract base class:

public abstract class FileBasedPreviewHandlerControl :

PreviewHandlerControl { ... }

public abstract class StreamBasedPreviewHandlerControl :

PreviewHandlerControl { ... }

public abstract class PreviewHandlerControl : Control

{

public abstract void Load(FileInfo file);

public abstract void Load(Stream stream);

public virtual  void Unload() { ... }

}

As its name implies, the Load method is called when a file or stream should be loaded and previewed. Likewise, the Unload method is called when the current preview should be torn down. A custom PreviewHandlerControl is then responsible for deriving from the appropriate type (either FileBasedPreviewHandlerControl or StreamBasedPreviewHandlerControl), overriding the Load method, and creating whatever Windows Forms controls are needed to render the file or stream. For my XML preview handler, I simply create a WebBrowser control and load the XML document into it, giving Windows Vista shell users the same pretty-printing of XML that is provided by Internet Explorer®:

public class XmlPreviewHandlerControl : FileBasedPreviewHandlerControl

{

public override void Load(FileInfo file)

{

WebBrowser browser = new WebBrowser();

browser.Dock = DockStyle.Fill;

browser.Navigate(file.FullName);

Controls.Add(browser);

}

}

Figure 4 ZIP Preview in Outlook 2007 (Click the image for a larger view)

The base PreviewHandlerControl's Unload implementation disposes and clears all controls from the Controls collection. If that functionality is appropriate for your control, there's no need to override it.

Aside from applying a few attributes to the derived PreviewHandler class, that's all there is to writing a custom preview handler. Figure 3 shows the complete implementation of a ZIP preview handler, the results of which are shown in Figure 4, running in Outlook 2007 (the download includes a richer ZIP preview handler that sports a tree view of the files and directories within the .zip, file icons, and double-click support for viewing the files contained in the archive). This class uses the Visual J#® 2005 ZIP library (included in the .NET Framework 2.0) to provide a list of all the files contained in the ZIP file. To demonstrate that these preview handlers do work in both Windows Vista and Outlook, Figure 5 shows the XmlPreviewHandler running in the Windows Vista shell.

Figure 3 ZIP Preview Handler

[PreviewHandler("ZIP Preview Handler", ".zip",

"{c0a64ec6-729b-442d-88ce-d76a9fc69e44}")]

[ProgId("MsdnMag.ZipPreviewHandler")]

[Guid("853f35e3-bd13-417b-b859-1df25be6c834")]

[ClassInterface(ClassInterfaceType.None)]

[ComVisible(true)]

public sealed class ZipPreviewHandler : FileBasedPreviewHandler

{

protected override PreviewHandlerControl CreatePreviewHandlerControl()

{

return new ZipPreviewHandlerControl();

}

private sealed class ZipPreviewHandlerControl :

FileBasedPreviewHandlerControl

{

public override void Load(FileInfo file)

{

ListView listView = new ListView();

listView.Dock = DockStyle.Fill;

listView.BorderStyle = BorderStyle.None;

listView.FullRowSelect = true;

listView.HeaderStyle = ColumnHeaderStyle.Nonclickable;

listView.MultiSelect = false;

listView.View = View.Details;

listView.Columns.Add("File Name", -2);

listView.Columns.Add("Size", -2);

listView.Columns.Add("Comment", -2);

ZipFile zip = new ZipFile(file.FullName);

Enumeration entryEnum = zip.entries();

while (entryEnum.hasMoreElements())

{

ZipEntry entry = (ZipEntry)entryEnum.nextElement();

if (!entry.isDirectory())

{

listView.Items.Add(new ListViewItem(new string[] {

entry.getName(), entry.getSize().ToString(),

entry.getComment() }));

}

}

zip.close();

Controls.Add(listView);

}

}

}

Figure 5 XML Preview in Windows Vista (Click the image for a larger view)

Implementing PreviewHandler

Now that you know how to implement a managed preview handler using my framework, let's take a look at the guts of the PreviewHandler base class to better understand how preview handlers work.

At its core, PreviewHandler is simply a container for a Windows Forms control, which is stored in a private member variable, _previewControl. It is initialized to the control returned from the GetPreviewHandlerControl method implemented by the custom preview handler implementation (for example, XmlPreviewHandlerControl or ZipPreviewHandlerControl):

protected PreviewHandler()

{

_previewControl = CreatePreviewHandlerControl();

IntPtr forceCreation = _previewControl.Handle;

}

While this is a very small constructor, there are several subtle issues worth pointing out. The first is that I'm violating an important .NET Framework design guideline, which at a high level says that constructors should not call virtual methods (this will be caught by the FxCop rule ConstructorsShouldNotCallBaseClassVirtualMethods). In .NET, unlike in ISO C++, the target of a virtual call will be on the most derived type (the override) rather than on the base type currently being constructed (the virtual). The problem here is that at the time when the base type's constructor is executing, the derived type's constructor has not yet executed. This means the method override will be called on the derived instance before that instance's construction has been completed, hence the rule that constructors should not call virtual methods that haven't been sealed in the same class. Keep this in mind, as it means you shouldn't do anything in your override of CreatePreviewHandlerControl that relies on anything set up in the constructor (that construction code will not have run yet). Based on the framework I've created, however, where CreatePreviewHandlerControl should simply be instantiating and returning the correct control type, this rule shouldn't be difficult to follow.

Of course, this begs the question, if I knew about this guideline in the first place, why did I break it by calling CreatePreviewHandlerControl from the constructor? Because I need to create the Control on the VI thread. This leads to the second issue I want to address. At the end of the constructor shown previously, you'll notice that I've retrieved the value of _previewControl.Handle, but I'm not doing anything with the results. I've done this in order to invoke the get accessor of the Handle property on the control. In other words, I didn't want the result; I simply wanted the accessor to execute. Invoking the Control.Handle get accessor forces the creation of the underlying window for the control. This is important, because the thread that instantiates the preview handler component and calls its constructor is a single-threaded apartment (STA) thread, but the thread that calls into the interface members later on is a multithreaded apartment (MTA) thread. As you may know, Windows Forms controls are meant to run on STA threads and sometimes die a horrible death if an attempt is made to use them from MTA threads. So, the best time to create the Windows Forms preview window is in the preview handler's constructor. Later calls to the other interface methods that require interaction with the preview control will need to be marshaled back to that STA main thread. This is easily accomplished using the control's ISynchronizeInvoke interface. Note that this problem of two threads does not exist when implementing a native preview handler and is most likely an artifact of how the CLR handles COM interop.

(It's worth pointing out that although I am discussing implementations that rely on Windows Forms, there may be times when the easiest way to render a preview is to use an ActiveX control. For a discussion on this, see the "Using ActiveX Controls" sidebar.)

Using the COM Interfaces

Believe it or not, the constructor is the most difficult part of the whole PreviewHandler implementation. The rest of the implementation simply serves to route calls made on the COM interfaces to the Windows Form controls and to handle some basic bookkeeping. I'll start by discussing the easiest interfaces first.

IOleWindow IOleWindow allows a hosting application to get a handle to the window that participates in in-place activation-in other words, a handle to our control (and for preview handlers, ContextSensitiveHelp is never called). This makes the interface extremely easy to implement, as shown here:

void IOleWindow.GetWindow(out IntPtr phwnd)

{

phwnd = _previewControl.Handle;

}

void IOleWindow.ContextSensitiveHelp(bool fEnterMode)

{

throw new NotImplementedException();

}

IObjectWithSite IObjectWithSite is just as simple. The interface is used to provide an object with a pointer to the site associated with its container. The site provided to SetSite is actually the IPreviewHandlerFrame that contains the preview handler's window. As such, the SetSite method stores the provided IUnknown interface pointer into a private member (which GetSite can return on request) and casts this to an IPreviewHandlerFrame (which it also stores). Under the covers, this cast results in a QueryInterface call:

private object _unkSite;

private IPreviewHandlerFrame _frame;

void IObjectWithSite.SetSite(object pUnkSite)

{

_unkSite = pUnkSite;

_frame = _unkSite as IPreviewHandlerFrame;

}

void IObjectWithSite.GetSite(ref Guid riid, out object ppvSite)

{

ppvSite = _unkSite;

}

IInitializeWithFile and IInitializeWithStream Moving on, IInitializeWithFile and IInitializeWithStream are implemented in FileBasedPreviewHandler and StreamBasedPreviewHandler, respectively. They are called to provide the preview handler with the full path or IStream to the file to be previewed. Implementing these methods takes only a few lines of code:

// in FileBasedPreviewHandler

private string _filePath;

void IInitializeWithFile.Initialize(string pszFilePath, uint grfMode) {

_filePath = pszFilePath;

}

// in StreamBasedPreviewHandler

private IStream _stream;

void IInitializeWithStream.Initialize(IStream pstream, uint grfMode) {

_stream = pstream;

}

The sole method on these interfaces, Initialize, is provided with a path to the file to be previewed or with an IStream representing the file, along with a file mode indicating how the file can be opened. The file path or stream is stored in a member variable so it can be accessed later. Preview handlers should be read-only, so I've ignored the second parameter. (In fact, all preview handlers should ignore the second parameter and open files read-only. And they should allow for subsequent deleting, reading, and writing of files). IInitializeWithFile and IInitializeWithStream are easy to implement, but they do deserve some further discussion.

If you read the MSDN® documentation on implementing preview handlers, you'll find that the documentation strongly advocates implementing IInitializeWithStream instead of IInitializeWithFile. The main advantage IInitializeWithStream has over IInitializeWithFile is that IInitializeWithStream allows a preview handler to preview data that isn't stored in an independent file, such as a text file stored in a ZIP file. (If the shell tries to preview data available only as a stream using a previewer that only implements IInitializeWithFile, the shell will save a copy of the stream to a local file and then preview that file. Clearly that's not as efficient as previewing the stream directly.) Whenever possible, follow this advice. Admittedly, however, this isn't always possible or practical. This is because many previewers are for file types most easily loaded and rendered using APIs that only support file paths or URLs. For example, the MsiOpenDatabase function I use in my sample MsiPreviewHandler to open an MSI file expects a file path to the target MSI file. It doesn't accept an IStream, which is the only data I would have if I implemented IInitializeWithStream instead of IInitializeWithFile.

I've chosen to support this model in my framework using the two PreviewHandler subclasses mentioned earlier: FileBasedPreviewHandler and StreamBasedPreviewHandler. Each implements only one of the interfaces, not both. If PreviewHandler itself implemented both, the shell would always use the IInitializeWithStream implementation, which it prefers for reasons having to do with security and isolation. (Outlook would actually favor the IInitializeWithFile implementation, for legacy reasons). In your derived class, you can choose which interface to use by deriving from the appropriate PreviewHandler subclass.

Another important consideration is that the data from the file or stream should not be loaded in Initialize. Rather, as I've done here, the path to the file or the reference to the stream should be stored, and only when the preview handler is actually asked to render the preview should it load the contents of the file. Additionally, it should not in any way lock the file for exclusive access while the preview is being displayed. A user should be able to look at the preview and open the target file for editing at the same time. Most importantly, a user should still be able to delete the file while it is being previewed.

IPreviewHandler This brings me to the core interface a preview handler must support, which is aptly named IPreviewHandler. (My implementation of this interface is shown in Figure 6.) IPreviewHandler exposes seven methods that must be implemented: SetWindow, SetRect, DoPreview, Unload, SetFocus, QueryFocus, and TranslateAccelerator.

Figure 6 IPreviewHandler Implementation

private void InvokeOnPreviewThread(MethodInvoker d)

{

_previewControl.Invoke(d);

}

private void UpdateWindowBounds()

{

if (_showPreview)

{

InvokeOnPreviewThread(delegate()

{

NativeWin32.SetParent(_previewControl.Handle, _parentHwnd);

_previewControl.Bounds = _windowBounds;

_previewControl.Visible = true;

});

}

}

void IPreviewHandler.SetWindow(IntPtr hwnd, ref RECT rect)

{

_parentHwnd = hwnd;

_windowBounds = rect.ToRectangle();

UpdateWindowBounds();

}

void IPreviewHandler.SetRect(ref RECT rect)

{

_windowBounds = rect.ToRectangle();

UpdateWindowBounds();

}

protected abstract void Load(PreviewHandlerControl c);

void IPreviewHandler.DoPreview()

{

_showPreview = true;

InvokeOnPreviewThread(delegate()

{

Load(_previewControl);

UpdateWindowBounds();

});

}

void IPreviewHandler.Unload()

{

_showPreview = false;

InvokeOnPreviewThread(delegate()

{

_previewControl.Visible = false;

_previewControl.Unload();

});

}

void IPreviewHandler.SetFocus()

{

InvokeOnPreviewThread(delegate() { _previewControl.Focus(); });

}

[DllImport("user32.dll")]

private static extern IntPtr GetFocus();

void IPreviewHandler.QueryFocus(out IntPtr phwnd)

{

IntPtr result = IntPtr.Zero;

InvokeOnPreviewThread(delegate() { result = GetFocus(); });

phwnd = result;

if (phwnd == IntPtr.Zero) throw new Win32Exception();

}

uint IPreviewHandler.TranslateAccelerator(ref MSG pmsg)

{

if (_frame != null) return _frame.TranslateAccelerator(ref pmsg);

return 1; // S_FALSE

}

When a file is about to be previewed, the host passes information about the file or stream to the preview handler using one of the initialization interfaces discussed previously. The handle for the view window that will contain the preview window is then passed to the preview handler using the SetWindow method-this allows the preview handler to set that view window as the parent of the preview window. The host may then set the size of the preview window appropriately, using the SetRect method. At some point after that, the host calls the DoPreview method to cause the preview to be displayed. While the preview is displayed, the host may call SetRect again, whenever the view window is resized. Finally, when the preview is closed, the host tells the preview handler to tear down the preview by calling the Unload method. This refers to the unloading of any resources loaded from the item being previewed; it doesn't mean the preview handler itself is being unloaded. The same instance of the preview handler can be reused for multiple previews.

To support my implementation, I am using a couple of helper functions. First, my InvokeOnPreviewThread helper accepts a MethodInvoker delegate (a void and parameterless delegate defined in the System.Windows.Forms namespace). This is executed on the main UI thread with the help of the preview Control's ISynchronizeInvoke.Invoke method (remember that the control was created in the preview handler's constructor so as to be instantiated from an STA thread). Second, I use my UpdateWindowBounds helper to set the parent window for the preview control, move it to the correct position, and display it. This functionality is useful in a helper method, as several IPreviewHandler interface methods modify this information and require it to affect the preview.Using ActiveX Controls

The preview handlers shown in this article were implemented with Windows Forms controls, such as ListView. But for some file formats, the easiest way to render a preview is to use an ActiveX® control with which the file format is associated. Through COM interop, Windows Forms allows ActiveX controls to be hosted like any other controls. The easiest way to get a managed wrapper for an ActiveX control is to use the Windows Forms ActiveX Control Importer, aximp.exe (see msdn.microsoft.com/library/en-us/cptools/html/cpgrfWindowsFormsActiveXControlImporterAximpexe.asp for more information). This will generate interop assemblies that you would then need to manually add references to in your project. If you instead add the ActiveX control to the Visual Studio Toolbox and drag-and-drop an instance of the control onto one of your design surfaces, Visual Studio will both generate the assemblies and automatically add the references for you.

An alternate approach is to manually create a wrapper for the control. For simple controls or for controls where you don't need to access much functionality, this technique may be more appealing. This is the approach I used to create a preview handler for the Adobe PDF format.

I receive a lot of PDF attachments in e-mail messages, and it would be very handy to be able to view the PDFs right within the attachment preview pane. While Office 2007 can save documents to the PDF file format (see microsoft.com/downloads/details.aspx?FamilyID=4d951911-3e7e-4ae6-b059-a2e79ed87041 for the download), it doesn't open this format for reading. So I created my PdfPreviewHandler, which allows me to preview PDF files by hosting the ActiveX control included with Adobe Reader. In order to host the control, I wrap it with a class derived from AxHost:

public class PdfAxHost : AxHost {

public PdfAxHost() :

base("ca8a9780-280d-11cf-a24d-444553540000") {}

object _ocx;

protected override void AttachInterfaces(){_ocx = base.GetOcx(); }

public void LoadFile(string fileName) {

_ocx.GetType().InvokeMember(

"LoadFile", BindingFlags.InvokeMethod, null,

_ocx, new object[] { fileName });

}

}

Aximp.exe does this, but it also does a whole lot more (providing high-fidelity wrappers for all methods and events). I've done the bare minimum necessary to wrap the control and to expose the one method (LoadFile) I need to be able to invoke on it. LoadFile accepts as an argument the location of the file to be displayed and renders it in the control. I can then use this control like any other:

public override void Load(FileInfo file)

{

PdfAxHost viewer = new PdfAxHost();

Controls.Add(viewer);

IntPtr forceCreation = viewer.Handle; // creates the OCX

viewer.Dock = DockStyle.Fill;

viewer.LoadFile(file.FullName);

}

SetWindow and SetRect are two such methods. The former provides the parent window handle and the new bounds for the preview window. The latter provides just the new bounds. SetFocus simply translates into a call to the preview control's Focus method, and QueryFocus returns the result of calling user32.dll's GetFocus function from the preview window's thread.

That leaves TranslateAccelerator, Unload, and DoPreview. My implementation of PreviewHandler currently doesn't provide much support for accelerator keys. As such, TranslateAccelerator simply delegates to the TranslateAccelerator method on the IPreviewFrameHandler stored in SetSite. If your preview handler displays multiple controls that can be tabbed through, you will want to augment TranslateAccelerator to provide more robust tab handling.

IPreviewHandlerFrame has another method on it, GetWindowContext, that allows the previewer to get a table that can be used to filter down the accelerators that should be forwarded to IPreviewHandlerFrame::TranslateAccelerator. This is meant to enable performance gains. However, even in native code, it doesn't provide huge performance improvements, and considering the complications involved in managing an accelerator table, doing so is probably not worthwhile. Preview handlers don't need to use the method-just forwarding everything to TranslateAccelerator is fine. In short, don't worry about GetWindowContext.

The Unload method hides the preview window and then delegates to the PreviewHandlerControl.Unload virtual method shown earlier in this article. And DoPreview, which is arguably the most important method on the interface, is no harder to implement. DoPreview is called to perform the actual rendering of the preview. My implementation calls to the abstract Load method, passing into it the PreviewHandlerControl. The derived implementation then calls either the FileBasedPreviewHandler's Load method with the path stored from IInitializeWithFile or the StreamBasedPreviewHandlerControl's Load method with the stream provided in IInitializeWithStream:

// in FileBasedPreviewHandler

protected override void Load(PreviewHandlerControl c)

{

c.Load(new FileInfo(_filePath));

}

// in StreamBasedPreviewHandler

protected override void Load(PreviewHandlerControl c)

{

c.Load(new ReadOnlyIStreamStream(_stream));

}

(Note that it encloses the COM IStream with a wrapper I derived from System.IO.Stream, allowing the derived class to consume the IStream in a read-only manner just as it would with any other .NET stream.) Once the data has been loaded, the DoPreview method calls UpdateWindowBounds to ensure that the preview is properly displayed.

Registering Handlers

Both the Windows Vista shell and Outlook 2007 look to the Windows registry to determine what preview handlers are available and to see what file types they are associated with.

As with any other COM class, a preview handler needs to be assigned a class ID. Referring back to Figure 2, this is the job of the GuidAttribute applied to the class:

[Guid("853f35e3-bd13-417b-b859-1df25be6c834")]

You'll notice that each of my preview handlers has a different GUID. Any custom preview handlers you create must also be assigned unique IDs. This ID is used as part of the preview handler's registration. The uuidgen.exe and guidgen.exe utilities included with Windows provide quick ways to generate GUIDs.

Figure 7 List of Registered Preview Handlers (Click the image for a larger view)

First, add a REG_SZ value to the PreviewHandlers key at HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\PreviewHandlers, where the name of the value is the GUID assigned to the preview handler. While it's not actually used, the value of this entry should be the display name of the preview handler. All of the built-in preview handlers do this, making it easy to see what preview handlers are registered by simply scanning this list in the registry (see Figure 7). Later on, after everything has been registered correctly, you can also see a list of all of the registered preview handlers within the Outlook Trust Center (see Figure 8).

Figure 8 List of Preview Handlers in the Outlook Trust Center (Click the image for a larger view)

Next, you need to create a special registry key under the shellex key for the target extension. This special key is named with a specific identifier, "{8895b1c6-b41f-4c1c-a562-0d564250836f}", which tells the system that the data it contains represents a preview handler. The default value for that key is set to the GUID for the preview handler used for that extension:

using (RegistryKey extensionKey =

Registry.ClassesRoot.CreateSubKey(extension))

using (RegistryKey shellexKey = extensionKey.CreateSubKey("shellex"))

using (RegistryKey previewKey = shellexKey.CreateSubKey(

"{8895b1c6-b41f-4c1c-a562-0d564250836f}"))

{

previewKey.SetValue(null, previewerGuid, RegistryValueKind.String);

}

If a specific preview handler can be used with multiple file extensions, you can simply repeat this step for as many extensions as appropriate. My framework looks for a PreviewHandlerAttribute on the preview handler. This attribute specifies the display name of the handler, the extensions the handler supports, and an AppID (which I'll discuss in a moment). I allow the supported extensions to be a simple string, as is the case with the ZipPreviewHandler:

[PreviewHandler("ZIP Preview Handler", ".zip",

"{c0a64ec6-729b-442d-88ce-d76a9fc69e44}")]

I also allow it to be a semicolon-delimited list of extensions, as is the case with my BinPreviewHandler:

[PreviewHandler("Binary Preview Handler", ".bin;.dat",

"{e92d3c10-89c8-4543-91b9-7a74305e9df4}")]

In such a scenario, my registration function (which you can see in the download) loops through each of the listed extensions and registers the handler for each of them. (Note that there are some preview handlers in Windows Vista that can handle more file types than they're registered to support. For info on previewing more file types, see the "If You've Got It, Flaunt It" sidebar.)

If You've Got It, Flaunt It

There are a handful of preview handlers built into Windows Vista. However, some of them can actually handle more file types than they're registered to support by default. Take, for instance, the Microsoft Windows TXT Preview Handler. As you can guess, this preview handler renders .txt files. But nothing about this preview handler restricts it to working with just .txt. As a developer, I frequently receive C#, Visual Basic®, and C++ code files as e-mail attachments. And I'd love to be able to preview these attachments within Outlook, rather than having to open Visual Studio® as the default viewer. Well, I can. All I have to do is register .cs, .vb, .cpp, and .h files as extensions to be previewed with the Microsoft Windows TXT Preview Handler. The following .reg file does just that:

Windows Registry Editor Version 5.00

[HKEY_CLASSES_ROOT\.cs\shellex\{8895b1c6-b41f-4c1c-a562-0d564250836f}]

@="{1531d583-8375-4d3f-b5fb-d23bbd169f22}"

[HKEY_CLASSES_ROOT\.vb\shellex\{8895b1c6-b41f-4c1c-a562-0d564250836f}]

@="{1531d583-8375-4d3f-b5fb-d23bbd169f22}"

[HKEY_CLASSES_ROOT\.cpp\shellex\{8895b1c6-b41f-4c1c-a562-0d564250836f}]

@="{1531d583-8375-4d3f-b5fb-d23bbd169f22}"

[HKEY_CLASSES_ROOT\.h\shellex\{8895b1c6-b41f-4c1c-a562-0d564250836f}]

@="{1531d583-8375-4d3f-b5fb-d23bbd169f22}"

If you want to be really slick, you can write a simple application or script that enumerates all of the file extensions under HKEY_CLASSES_ROOT. You can check to see if each has a Content Type value set, and if it does, whether that Content Type's value starts with "text/". If it does and if the extension doesn't already have a preview handler configured, you can set it to be previewed with the TXT handler—just as I did in the previous registry script. With only a few minutes work, you can dramatically expand the number of file types that can be previewed.

Of course, the TXT Preview Handler isn't the only handler capable of previewing a variety of file types. Some handlers you write might also exhibit this flexibility. Consider the XmlPreviewHandler described in this article. If you look in the code download, you'll see that the preview handler is actually named InternetExplorerPreviewHandler. I changed its name since it's not really specific to the XML file format—it's more specific to the control used to render the XML (the WebBrowser control). Since the preview handler uses the WebBrowser control to navigate to the file being previewed, this handler can be used with quite a variety of file types. Many of these file types are already covered by other preview handlers, but some aren't. For example, the new XML Paper Specification (XPS) file format delivered with Windows Vista doesn't have a built-in preview handler, but Internet Explorer is capable of rendering XPS documents. Therefore, I augmented the InternetExplorerPreviewHandler by adding .xps to the list of supported file extensions, and with that simple effort, I now have a preview handler for XPS.

In addition to registering on file types, you can register on Prog IDs. For example, the default value in HKEY_CLASSES_ROOT\.xml is "xmlfile", so my XML preview handler is actually registered under HKEY_CLASSES_ROOT\xmlfile\ShellEx\{8895b1c6-b41f-4c1c-a562-0d564250836f}:

[PreviewHandler("XML Preview Handler", "xmlfile",

"{88235ab2-bfce-4be8-9ed0-0408cd8da792}")]

That way, if you have multiple file extensions that are really the same type, you can use the same Prog ID for each and register the previewer only once. (For example, you could register on HKEY_CLASSES_ROOT\jpegfile instead of on both HKEY_CLASSES_ROOT\.jpeg and HKEY_CLASSES_ROOT\.jpg). In fact, the Windows shell team recommends that new file types should define new Prog IDs and register previewer handlers on those, rather than on actual extensions, even if there's currently a 1:1 mapping between the Prog ID and the extension.

The next registry change (for managed handlers) involves AppIDs. The reason relates back to my discussion about writing managed add-ins for the Windows Vista shell. As I mentioned, preview handlers are loaded out-of-process from the shell, by default in a surrogate host (prevhost.exe). For efficiency, the system wants as few instances of prevhost.exe running as possible, so multiple types of preview handlers are loaded into the same external process. This means that while the shell won't be attempting to load managed preview handlers targeting different versions of the CLR, prevhost.exe might. And this brings us back to the problem we initially had. The solution is to tell the system that instances of your preview handler must be loaded into a surrogate host dedicated to preview handlers of that same type. This guarantees you won't have multiple preview handlers targeting different versions of the CLR loaded into the same surrogate process. The COM class registration for the preview handler (which I'll get to momentarily) specifies an ID for configuration details about exactly what surrogate host to use, and that ID is referred to as an AppID. When a managed preview handler is registered, I create a new AppID registration under HKCR\AppID that will only be used by that managed preview handler:

using (RegistryKey appIdsKey = Registry.ClassesRoot.OpenSubKey(

"AppID", true))

using (RegistryKey appIdKey = appIds.CreateSubKey(appID))

{

appIdKey.SetValue("DllSurrogate",

@"%SystemRoot%\system32\prevhost.exe",

RegistryValueKind.ExpandString);

}

Please note that this AppID still points to prevhost.exe as the surrogate host.

The only thing left to do is to add the preview handler's COM component registration. Most of this is easily done by using the regasm.exe tool included with the .NET Framework SDK:

regasm /codebase ManagedPreviewHandler.dll

(You can do this from an installer as well, with a bit of code that uses the RegisterAssembly method of the RegistrationServices class in the System.Runtime.InteropServices namespace of mscorlib.)

As mentioned previously, however, I need to modify the component registration to point to the new AppID I've created. And I need to add to it a display name that will be used by hosts like Outlook 2007 to list registered handlers:

using (RegistryKey clsidKey = Registry.ClassesRoot.OpenSubKey("CLSID"))

using (RegistryKey idKey = clsidKey.OpenSubKey(previewerGuid, true))

{

idKey.SetValue("AppID", appID, RegistryValueKind.String);

idKey.SetValue("DisplayName", name, RegistryValueKind.String);

}

For a production preview handler, DisplayName should really be set to a REG_SZ value that points to a localized Win32 binary resource string ("@myhandler.dll,-101", for example). Unfortunately, the IDE for Visual C#® 2005 does not provide a built-in way to include a Win32 resource file in a managed assembly. To do that, you need to drop down to the command line, since csc.exe supports the /win32res switch that allows for the incorporation of Win32 resource files. For the purposes of this article and my registration code, I've resorted to a non-localized value.

All of this registration logic is wrapped up into a single method on PreviewHandler:

protected static void RegisterPreviewHandler(

string name, string extensions, string previewerGuid, string appID)

There is a corresponding unregistration method, as well.

The regasm.exe tool supports ComRegisterFunctionAttribute from the System.Runtime.InteropServices namespace in mscorlib. You can apply this attribute to a static method in an assembly to be registered with regasm, and when types in that assembly are found to be ComVisible, their type info is passed to the method marked with the ComRegisterFunction attribute (after the type has had its COM class registered in the registry). This makes it very easy to write a method that handles all of the previously mentioned registration goo when the assembly is registered for COM interop (see Figure 9). I've included a counterpart unregistration function that is marked with the ComUnregisterFunctionAttribute.

Figure 9 COM Registration

[ComRegisterFunction]

public static void Register(Type t)

{

if (t != null && t.IsSubclassOf(typeof(PreviewHandler)))

{

object[] attrs = (object[])t.GetCustomAttributes(

typeof(PreviewHandlerAttribute), true);

if (attrs != null && attrs.Length == 1)

{

PreviewHandlerAttribute attr =

attrs[0] as PreviewHandlerAttribute;

RegisterPreviewHandler(attr.Name, attr.Extension,

t.GUID.ToString("B"), attr.AppID);

}

}

}

With this in place, installing and registering my preview handlers requires just two lines at the command prompt:

gacutil -i MsdnMagPreviewHandlers.dll

regasm /codebase MsdnMagPreviewHandlers.dll

Similarly, unregistering the installed preview handlers requires just two lines at the command prompt:

regasm /unregister MsdnMagPreviewHandlers.dll

gacutil -u MsdnMagPreviewHandlers

If you write your own custom preview handlers that use this framework, and you write them in the same assembly as mine, you won't have to do anything else to get them to work. If you write them using my framework, but you write them in a separate assembly, you just need to add the following class to your assembly:

internal sealed class PreviewHandlerRegistration

{

[ComRegisterFunction]

internal static void Register(Type t) {

PreviewHandler.Register(t);

}

[ComUnregisterFunction]

internal static void Unregister(Type t) {

PreviewHandler.Unregister(t);

}

}

First install and register MsdnMagPreviewHandlers.dll, as I showed previously. Then install and register your assembly. When regasm sees the ComRegisterFunction in the PreviewHandlerRegistration class in your assembly, it will call the ComRegisterFunction for each of the preview handlers in your assembly. The function will delegate to all of the previously described functionality included in MsdnMagPreviewHandlers.dll.

Figure 10 XamlPreviewHandler in Outlook 2007 (Click the image for a larger view)

To exemplify how to do this, I've included a preview handler for XAML files in the download for this article. It's implemented in its own assembly, which contains my XamlPreviewHandler class and the PreviewHandlerRegistration class. Figure 10 is a screenshot of the handler in action in Outlook 2007 (the XAML file being previewed, which is from the Windows Presentation Foundation SDK team blog, is available at blogs.msdn.com/wpfsdk/archive/2006/05/23/Animating_XAML_Clip_Art.aspx). Note that XamlPreviewHandler also derives from StreamBasedPreviewHandler rather than FileBasedPreviewHandler. The XamlReader class provided with the Windows Presentation Foundation supports loading from a Stream, and thus it makes sense to implement IInitializeWithStream rather than IInitializeWithFile:

public override void Load(Stream stream)

{

Frame f = new Frame();

XamlReader reader = new XamlReader();

f.Content = reader.LoadAsync(stream);

ElementHost xamlHost = new ElementHost();

xamlHost.Child = f;

xamlHost.Dock = DockStyle.Fill;

Controls.Add(xamlHost);

}

Debugging

Since in-process preview handlers run under a surrogate hosting process (prevhost.exe by default), to debug a preview handler, you need to attach the debugger to the host process. There are a few ways to do this. The first is to wait for the host process to start and then use the debugger's ability to attach to an existing process. However, there may be multiple instances of prevhost.exe, especially if you follow my suggestion of creating a new AppID for each of your managed preview handlers. When there are multiple instances of prevhost.exe running, you need to know which instance to connect to. You can use a tool like Process Explorer (www.sysinternals.com/Utilities/ProcessExplorer.html) from Sysinternals (recently acquired by Microsoft) to examine the command-line arguments used when the appropriate instance of prevhost.exe was launched. As a surrogate host executable, prevhost.exe needs to be told at startup what COM component to host, and thus it will be passed the GUID for your preview handler on the command line:

prevhost.exe {8FD75842-96AE-4AC9-A029-B57F7EF961A8} -Embedding

You can look at the command-line arguments for each instance of prevhost.exe to find the one with the GUID that matches the preview handler you want to debug. You can then use the process ID to attach the debugger to the correct process.

If the process is already running, the preview handler will have already been constructed and some of its code executed. To debug its startup, you need a different solution. One approach is to place a call to System.Diagnostics.Debug.Break at the beginning of the constructor. This causes the debugger to launch and attach to the process at that moment, allowing you to debug your entire component.

But attaching a debugger isn't always the best solution. Sometimes you just want classic "printf-style" debugging, where you output a bunch of information that you can watch as the preview handler runs. In this case, it's best to use the System.Diagnostics.Trace class. By default, the TraceListenerCollection that Trace outputs to contains a TraceListener that writes the traced data to OutputDebugString, enabling any listening debuggers to display the output text. The Windows SDK includes Debug Monitor (DbMon.exe) which makes it easy to view these traces.Preview Handlers and Windows XP

By Ryan Gregg

With the framework provided in this article, writing an in-process preview handler is a straightforward exercise in managed code. However, creating a preview handler that works with Office Outlook 2007 on both Windows XP and Windows Vista is a more complicated matter. The proxy application (called prevhost.exe) that hosts in-process preview handlers on Windows Vista is not available on Windows XP. Preview handlers for Windows XP need to be written as local COM servers, without relying on another application to do the hosting.

Unfortunately, the .NET Framework 2.0 doesn't provide most of the glue necessary to register a .NET application as a local COM server. As a developer of a managed preview handler, you need to write code that looks and acts a lot like a native code implementation of a preview handler. Here's a very high-level view of what it is you need to write.

A typical local COM server is compiled as a Windows Application, instead of as a Class Library. When COM requests a new instance of the local server, it launches the local server application with a special set of flags to indicate that the application should start up in embedding mode. This causes the application to launch without a UI and to register class factories with COM. This same server likely has two additional logical components over the in-process preview handler implementation: a COM server implementation and a class factory.

The COM server implementation provides the running application framework for the executable launched by COM when the preview handler is called. Inside the Main method for the application, you need to parse the command-line parameters, initiate each class factory's registration with COM, monitor when it is safe to shutdown the process, and keep the message loop pumping so that the application hosting the preview handler won't hang. Most local COM servers also support command-line switches to register and unregister the server in the Windows registry. Since regasm.exe does not provide a method to register as a local server, you also need to write methods to create the proper registration keys for local COM server registration. These keys are well documented in the COM documentation.

The class factory component implements the IClassFactory interface provided by COM. Additionally, this component is responsible for registering class objects with COM and creating instances of class objects for COM callers. In particular, the class factory should make calls to CoRegisterClassObject and CoRevokeClassObject to indicate to the system that a class factory is available (or revoked) for the particular GUID used to identify the preview handler. When requested, the class factory creates a new instance of the preview handler class and marshals it back to the COM caller. These instances need to be tracked and garbage collection may need to be forced so the server application can determine when all external references have been released allowing it to shutdown properly.

Writing a managed preview handler that works with Outlook 2007 on both Windows XP and Windows Vista is a considerably involved effort. While this sidebar covers the required pieces, I haven't gotten into the important details. Anyone developing a local COM server needs to understand the way COM works, the difference between threading apartment styles, and how to work with Win32® APIs using managed code. Without this knowledge, frustration will likely follow.

While writing a local COM server preview handler in managed code may not be as simple as writing an in-process handler, this is a detail you shouldn't overlook if you want to ensure your preview handler will work for every Office 2007 user, regardless of the OS they are running on. For more details and working examples on how these pieces fit together, be sure to check out these helpful resources:

- COM Fundamentals (windowssdk.msdn.microsoft.com/ms694505.aspx) is MSDN documentation that describes the Microsoft Component Object Model.

- PInvoke.net (www.pinvoke.net) is an excellent site dedicated to helping users define Win32 API calls in managed code.

- Building COM Servers in .NET (www.codeproject.com/useritems/BuildCOMServersInDotNet.asp) provides a useful example of a local COM server implementation.

Ryan Gregg works for Microsoft as a Program Manager on Office Outlook, primarily on the extensibility platform and custom forms. He blogs semi-regularly about Outlook extensibility at blogs.msdn.com/rgregg.

For testing purposes, I recommend that you add a trace statement to the beginning of each of the interface methods I discussed earlier. Alternatively, you can use Debugger.Break in concert with the new tracepoint feature in Visual Studio 2005. By doing this, you'll gain a much clearer understanding of how the shell and Outlook 2007 interact with your preview handler.

Conclusion

Preview handlers are an awesome addition to both the Windows Vista shell and Outlook 2007. Throw in the productivity advantages of managed code (using my existing framework, it took me only seven minutes to create the XAML preview handler and get it up and running) and you've got yourself a very powerful platform for displaying existing file types as well as your own custom file types. I believe preview handlers have the potential to make us more productive in our daily routines-and the more supported file types there are, the more productive we'll all be. So get out there and start coding a preview handler for your favorite extension.

Stephen Toub is the Technical Editor for MSDN Magazine.

SideShow Gadgets

Get Started Writing Gadgets For Windows SideShow Devices

Jeffrey Richter

This article discusses:

- SideShow and its associated APIs

- Developing, installing, and configuring gadgets

- Device cache management

- Gadget events and notifications

This article uses the following technologies:

Windows Vista

Code download available at: SideShow2007_01.exe (169 KB)

Browse the Code Online

Contents

What Are SideShow Devices?

Use and Configuration

SideShow Gadget and Device Architecture

Gadget Installation

Device Cache Management

Gadget Lifetime

Content

Capabilities, Events, and Notifications

Conclusion

Today's applications have lots of information to share with end users. However, screen real estate is limited and organizing all of the possible information is taxing for software developers as well as end users. Furthermore, people prioritize information in different ways and at different times. For example, when a meeting is imminent, I need to know the time and place right now. When I'm driving, I need to know how soon to make the next turn. In general, I might want to always be able to check newly received e-mail messages. Likewise, you can imagine scenarios where I might want information on traffic, flights, weather, financial transactions, and so on. Trying to get the right information to the user at the right time is a difficult problem.

The Microsoft gadgets initiative is committed to alleviating this problem. A gadget is an application (or part of an application) that can present users with the data they want when and where they want it. Currently, Microsoft offers three different kinds of gadgets: gadgets for Windows® Sidebar present a user's information on the user's desktop or in the Windows Vista™ Sidebar; gadgets for Windows Live™ present a user's information on any computer via a Web browser; and gadgets for Windows SideShow™ present a user's information on an auxiliary hardware device via its SideShow-enabled device driver. In this article, I will focus specifically on SideShow hardware devices and how to write a SideShow gadget to communicate with these devices using the managed API.

What Are SideShow Devices?

Windows Vista supports the ability to communicate with one or more auxiliary display devices. These auxiliary display devices can come in all kinds of shapes and sizes with varying capabilities. Many of these devices have not yet been manufactured, but should be available soon.

Some units will be powered when the computer is turned on while others can be powered whether the computer is on or off. Some units may even have their own separate rechargeable batteries. Some units will be in color and some will be grayscale or monochrome. Some will allow bitmaps to be displayed while some may only allow text. Some units will offer varying amounts of on-board memory for caching data while others may have no on-board memory at all. Some will offer buttons allowing navigation of the cached data. Some may offer touch-sensitive displays.

Some may offer a built-in wireless technology (such as Bluetooth) so that the computer can communicate with the device without it being physically connected to the computer. Some may offer a headphone jack allowing you to listen to music or video that has been downloaded into its memory.

Figure 1 shows a prototype notebook computer with a SideShow device mounted in its cover and a close-up picture of the SideShow device itself. This particular SideShow device has a 220×176 resolution display with each pixel offering 16 bits of color depth. The unit also offers Menu, Left, Right, Up, Down, and Enter buttons. This SideShow device is self-powered, meaning that it is powered using the computer's battery; however, the computer doesn't have to be turned on for the user to interact with the device. When on, this particular device consumes about 4 milliwatts of power, compared to the 75 or so watts of power consumed by the whole computer when it is turned on.

Figure 1 Prototype Notebook with a Mounted SideShow Device (Click the image for a larger view)

Figure 1b(Click the image for a larger view)

So, with the computer's lid closed and the computer's power off, the user can still read e-mail, examine his calendar, look at some digital photos, get mapping directions, see the current date/time, and so on. And of course, you can modify your own application to export its data to the SideShow device too. I'll show you how to do this later in this article.Figure 2 shows a prototype for a remote control device for controlling Windows Media® Center. The auxiliary display allows the user to select a song and have Media Center play it. What's cool about this is that the user's television or monitor doesn't have to be turned on in order to listen to a song. Also, while watching a movie, program information can be sent from the Media Center PC to the Windows SideShow device, allowing you to read it without affecting the playback of the movie on the monitor. Furthermore, you can use the remote control in order to view the TV guide data and schedule recordings.

Figure 2(Click the image for a larger view)

Other examples of SideShow devices include keyboards with LCD displays, digital picture frames, and even mobile phones. A single computer can have multiple SideShow devices connected to it simultaneously. For example, the prototype notebook computer shown in Figure 1 could have both a keyboard and a remote control talking to it. Each of these displays could show similar or different kinds of information.

Use and Configuration

End users configure their SideShow devices with the Windows SideShow Control Panel applet. When the user opens this applet, she is greeted by a window similar to the one in Figure 3.

Figure 3 SideShow Applet Controls Devices and Gadgets (Click the image for a larger view)

The leftmost panel shows the SideShow hardware devices currently installed, sorted by the date and time of their installation. In the main panel is a grid. The first column shows the installed applications that know how to communicate with SideShow devices; these are the SideShow gadgets. By default, Windows Vista ships with two gadgets: Inbox—Windows Mail and Windows Media Player. Figure 3 also shows another gadget that I've installed (Office Outlook® 2007 Calendar, which ships with Microsoft® Office Outlook 2007) and a gadget that I've written (called Jeff's SideShow Gadget). Each additional grid column represents a SideShow device attached to the computer. Figure 3 shows two SideShow devices: Windows SideShow Simulator—0 and Logitech G-15 Keyboard.

By selecting the checkboxes in the grid, the user indicates which SideShow devices a gadget can talk to. The user can also configure a specific SideShow device by selecting it at the top of the grid. For example, if I select the Windows SideShow Simulator—0 item, I get a window as shown in Figure 4. Here you can configure screen appearance and behavior (theme, language, font size, brightness, backlight, and so on), configure security and privacy settings (use a PIN to unlock the device, delete all data when the user logs off the computer), configure gadgets (turn gadgets on or off, change order of appearance), and uninstall the device.

Figure 4 Configuring a SideShow Device (Click the image for a larger view)

Many SideShow devices will have their own on-board memory that is used to cache gadget data. A gadget must be running in order to send data to a SideShow device, meaning that the computer must be powered on. So, Windows Vista allows the user to configure when and how often the computer should wake itself up and run the gadget. The user configures this by selecting the "Set my computer to wake automatically" option shown in Figure 3. When selecting this option, the window shown in Figure 5 appears. After being awakened, the computer will stay awake until the machine has been idle for the amount of time specified using the machine's Power Options Control Panel applet. Note that some computers do not support auto-wake and some applications can prevent a machine from going back to sleep.

Figure 5 Configuring How a Gadget Wakes Up The Device (Click the image for a larger view)

SideShow Gadget and Device Architecture

Figure 6 shows an architectural diagram of how SideShow gadget applications communicate with SideShow hardware devices. Each hardware device must come with a user-mode device driver that will have to be installed into Windows Vista. When Windows communicates with the device, the operating system creates a WUDFHost.exe process and loads the device driver into it; there is an instance of WUDFHost.exe for each device driver. There are many benefits to user-mode device drivers, including easier development, easier debugging, greater system stability, and improved security. For more info about the Windows User-Mode Device Driver Framework, see "User-Mode Driver Framework (UMDF)"

Figure 6 SideShow Gadget Communication Architecture (Click the image for a larger view)

A SideShow gadget application can be written in native code or in managed code. A native application will control the devices that the user has connected it to by using COM components and methods implemented in AuxiliaryDisplayApi.dll. (The original name for SideShow was Auxiliary Display.) This DLL ships with Windows Vista and is guaranteed to be available on every Windows Vista machine; however, it does not function on the Starter and Home Basic versions of Windows Vista. The main goal of the COM components is to abstract away from the developer the fact that different devices are or are not connected and that the devices connected support different capabilities (screen size, color, and so on).

A managed SideShow gadget will control the devices connected to it using types defined in the Microsoft.SideShow namespace. These types are implemented in the Microsoft.SideShow.dll assembly file. (I designed and implemented this for the Windows Vista team and it is downloadable via Microsoft Download Center.) Internally these types talk directly to the COM components in AuxiliaryDisplayApi.dll. Unfortunately, due to some timing issues with the release, the assembly is not guaranteed to exist on all installations of Windows Vista. If you write a managed SideShow gadget using this assembly, you will have to ensure that this DLL gets on to the end user's machine.

To help developers test their gadget code, the Windows Vista SDK ships with a SideShow Simulator application. This application runs on Windows Vista and emulates a SideShow hardware device. To get the simulator working, you must first run the following from a command prompt:

WindowsSideShowVirtualDevice.exe /regserver

This registers the COM components required by the simulator. You only need to register the COM components once on a machine. Then, whenever you want to test your gadget, you run the VirtualSideShow.exe app that also comes with the SDK.

The Microsoft.SideShow.dll assembly defines a number of types in the Microsoft.SideShow and the Microsoft.SideShow.SimpleContentFormat namespaces, making it easy to build SideShow gadgets or to integrate SideShow functionality into an already existing application. The most important and most-used classes are described in Figure 7.

Figure 7 Important SideShow-Related Classes

Class

Description

GadgetRegistration

A static class to aid in setting up the gadget in the registry.

SideShowGadget

An abstract base class representing a connection between application code and one or more SideShow hardware devices that the user has connected. This class is derived from Component, allowing it to be dropped on a Windows Form.

ScfSideShowGadget

Derived from SideShowGadget, this class allows the application to send Simple Content Format (SCF) XML strings and images down to a SideShow device.

Scf

A static class offering static methods, making it easy to create SCF XML strings in source code.

DeviceCapabilities

An object of this class represents a hardware device's capabilities. This can be used by a gadget application to fine-tune the data it chooses to send to a device.

Gadget Installation

When developing a SideShow gadget, you must first create a GUID for it; this becomes the gadget's ID and Windows Vista uses this ID to interact with the gadget. When you install a gadget, your setup program should create a new registry subkey based on the GUID you've created for your gadget:

HKEY_CURRENT_USER\SOFTWARE\Microsoft\SideShow\Gadgets\{<GUID>}

If you want to install the gadget for all users on the machine, use HKEY_LOCAL_MACHINE instead of HKEY_CURRENT_USER. Under this registry subkey you then need to add a few registry values, as described in Figure 8.

Figure 8 Registry Values for Gadget Installation

Registry Value

Description

Icon

The path of the file (and optional Win32® icon resource ID number) that refers to an icon that should appear in Control Panel and on the device itself. If not specified, a default icon is shown. It is recommended that you supply 16x16, 32x32, and 48x48 pixel images using 32-bit color (with alpha channel).

FriendlyName

A string, or a pathname and Win32 string resource ID (for a localized string), that should appear as the name of the gadget in Control Panel. If you don't provide this, no string will be shown to the user.

StartCommand

The path of the executable file containing the gadget's code. Windows Vista uses this to invoke the application automatically when a user logs on, during fast user switches, when the computer resumes from sleep, or when the user enables a gadget for a device. (There is very little intelligence here; Windows Vista invokes gadget applications fairly frequently,so gadget applications should be kept small. Furthermore, a gadget application should terminate itself if another instance of it is already running or when no devices are connected to it.)

Endpoints

The list of supported endpoints, which can be either iCal format or SCF. iCal (see www.ietf.org/rfc/rfc2445.txt) is a text representation designed for publishing calendar information. The Office Outlook 2007 Calendar gadget uses this endpoint to send iCal data to a SideShow device, allowing the device to render the user's calendar data. Few gadgets and devices support the iCal endpoint. In contrast, SCF is a series of XML content pages that reference each other. Since most applications will use the SCF format, it is the focus of this article.

CacheAlgorithm

This value gives a hint to the device as to how the gadget would like its content managed (devices tend to have very limited cache memory and many gadgets could be connected to a single device). The gadget can tell the device to keep newest items (0, the default), keep oldest items (1), keep frequently accessed items (2), or keep recently accessed items (4).

OnlineOnly

If the value is 1, the user can only work with the gadget on the device if Windows is running and if the gadget application is also running.

PropertyPage

A gadget can offer a configuration settings dialog box to its users that can be launched via the Windows Control Panel. To do this, the gadget must define a COM component that implements the IPropertyPage interface. Then, during setup, the gadget would add this registry value indicating the GUID of the COM component. If specified, Control Panel will underline the gadget application, allowing the user to click it to invoke the gadget's configuration dialog box.

Your gadget's setup program should create its own registry subkey and set any of the registry values it desires. Typically, an installation program will place these registry additions in a .reg file and invoke this file upon application setup. When setup is near completion, you should execute the following command, which will cause Windows Vista to display a tooltip alerting the user that your gadget application has been installed:

SchTasks.exe /run /tn Microsoft\Windows\SideShow\GadgetManager

The GadgetManager task will not show the tooltip if no devices are installed, if it detects that no new gadgets have been installed since it ran last, or if the user turned it off by unchecking the "Notify me when new gadgets are installed" checkbox (see Figure 3).

Once the gadget is installed, users can open Control Panel to configure the gadget and connect it to whatever hardware devices they desire. For security reasons, however, by default a newly installed gadget is not allowed to communicate with any devices.

When a gadget is running, it usually sends data to its connected devices periodically. This way, when the user turns off the computer, the devices have cached some data, making it available whenever the user wants it without having to power up the computer. While a gadget is running, it will receive notifications from Windows indicating when the user has added or removed a device from the gadget via the control panel.

A gadget can also be marked as online-only, meaning that the user can only interact with the gadget on the device if both the computer and the gadget on Windows are running. If the computer is sleeping or in hibernation when the user enters this gadget on the device, the device asks the user whether the computer should be turned on. If the user responds yes, the computer powers up. When the computer is running and the user is logged in, the device enters the gadget. If the computer is in full shutdown, the device tells the user to manually power up the computer and log in, which allows the gadget to be entered on the device.

Device Cache Management

Since devices will have limited amounts of cache memory and users can have several gadgets connected to a single device, devices need to manage memory efficiently. Different devices handle this in various ways, but I'll explain how it works for devices using the Microsoft firmware.

The current Microsoft firmware divides the memory on the device equally among all the gadgets connected to the device. So if three gadgets are connected to a device, then each gadget is guaranteed to get a third of the memory. Individual gadgets are allowed to use more than their portion if other gadgets are using less than their portion. If a gadget that is under its minimum wants more memory, it can take that memory from any gadget that is over its minimum.

Similarly, if a gadget that is over its minimum wants more memory, it can take memory away from the lowest-priority gadget that is over its minimum. In Figure 3, you'll notice that there is an option allowing the user to change the order in which gadgets appear. Gadgets that the user puts at the top are high priority and are likely to get more memory; gadgets towards the bottom are lower priority and have memory taken away from them more quickly. When a gadget has memory taken away from it, the device uses the cache policy set by the gadget via its CacheAlgorithm registry value.

For gadget developers, the important thing to realize is that there is no guarantee that any content pushed down to the device will actually remain on it. If the user wants to examine content that has been purged, the device must notify the gadget on Windows so that it can push down the discarded content again. If the gadget does not send the missing content within an acceptable timeframe (which varies by device—on the simulator it is three seconds), then the device will notify the user that the content is missing. Unlike the native SideShow API, the managed SideShow library caches all content on Windows and will automatically send it back to the device as long as the gadget process stays up and running on Windows Vista.

Gadget Lifetime

Windows must be running a gadget in order for the gadget to download information to a SideShow device. There are basically two kinds of gadgets: UI gadgets and non-UI gadgets. A UI gadget is an application the user runs and interacts with. For example, imagine a Windows Forms application that the user invokes via Windows Explorer. As the user manipulates the application, it can download data to the device. If the user terminates the application, the information remains on the device and can be accessed when the computer is turned off. The lifetime of this application is determined completely by the user. The Microsoft Office Outlook 2007 gadget is an example of a UI gadget.

A non-UI gadget doesn't show any user interface at all. It runs in the background, perhaps monitoring something, and periodically sends data to the device, allowing the data to be viewed when the computer is turned off. To allow the user to remain unaware of non-UI gadgets, Windows offers a mechanism that can automatically invoke non-UI gadgets without the user being involved. Whenever the user logs onto his account, resumes the computer from a sleep state, or uses the SideShow control panel applet to connect or disconnect a device from a gadget, Windows automatically invokes all gadgets present in the registry by executing their StartCommand registry values.

Windows doesn't employ any sophisticated algorithm here; it simply invokes all the registered gadgets. This means that a gadget application could already be running and if the user connects or disconnects a device, Windows will simply run the application again. For this reason, non-UI gadgets should immediately check whether another instance is currently running and, if so, additional instances should just kill themselves. Also, a non-UI gadget would normally terminate itself when the user has disconnected all devices from it. See the Events section later in this article for more about this.

UI gadgets may also want to ensure that only one instance is executing to prevent each application from clobbering data sent to the device by another instance. Furthermore, you typically do not want Windows to invoke UI gadgets automatically (as the UI would appear) and therefore, most UI gadgets should not have a StartCommand registry value at all.

Content

All gadgets push content down to a device. The device has a home display that shows all the gadgets that are connected to the device. In Figure 9, the upper-left image shows a sample home screen. As you can see, each gadget appears with an icon, a name, and optionally some text. The icon and name come from the Icon and FriendlyName registry values. The text shown here is called the glance content and is sent from the gadget to the device by calling SideShowGadget's AddGlanceContent method when the gadget is running.

Navigating through Gadget Screens

Figure 9a Device Home Page (Click the image for a larger view)

Figure 9b Gadget Home Page (Content ID 1) (Click the image for a larger view)

Figure 9c Text Page (Content ID 2) (Click the image for a larger view)

Figure 9d Image Page (Content ID 3 Showing Image (Content ID 1000) (Click the image for a larger view)

When the user enters the gadget on the device, the device renders the content previously downloaded from the gadget while it was running on Windows. Currently, SideShow supports two different kinds of content: Simple Content Format (SCF) and iCal; the latter is not discussed in this article. SCF content is a set of XML strings sent from the gadget application to the device. The device parses the XML strings in order to render the content on the device.

The new SCF format allows the gadget application to define three types of content: text and images, menus and context menus consisting of items that initiate navigation to other content, and dialogs that present the user with questions resulting in navigation to other content. Details about the SCF XML schema can be found at windowssdk.msdn.microsoft.com/ms744133.aspx.

The managed SideShow assembly (Microsoft.SideShow.dll) includes a public, static Scf class that contains a bunch of static methods allowing you to create the XML string in code easily. The DownloadDataToDevices method, shown in Figure 10, demonstrates how to open an ScfSideShowGadget and send glance string content. The sample also shows how to call ScfSideShowGadget's AddContent method to send SCF menu, SCF text, and SCF image content down to a device. As you can see, each piece of content is identified by a content ID number. The first page of the gadget (the lower-left image in Figure 9) is its home page which always has a content ID of 1 (represented by the ScfSideShowGadget's HomePageContentId constant). When the user selects the See Text menu item, the device will navigate to the content page with an ID of 2 and when the user selects the "See Image" menu item, the device will navigate to the content page with an ID of 3. Page 3, the Image page, contains XML describing how the image should be shown (centered and auto fitting). This page also refers to the actual image, which I've assigned a content ID of 1000. In my code, I've extracted the image from an assembly resource.

Figure 10 DownloadDataToDevices Method

private static void DownloadDataToDevices() {

// Construct a SCF SideShow gadget for our gadget's Guid

using (ScfSideShowGadget gadget = new ScfSideShowGadget(s_gadgetId))

{

// Add Glance content showing date/time of last update

gadget.AddGlanceContent(

String.Format("Content updated on {0}{1:D}.",

Environment.NewLine, DateTime.Now));

// The home page is a menu with an item that goes to some

// text and another item that goes to an image

gadget.AddContent(

Scf.Menu(ScfSideShowGadget.HomePageContentId,

"Simple Gadget Menu", ScfSelectAction.Target,

Scf.Item(2, "See text"), Scf.Item(3, "See image")));

// This is the text page

gadget.AddContent(

Scf.Content(2, "This is the Text Page",

Scf.Txt(ScfAlign.Left, true, Color.Red,

"This is some text")));

// This is the image page

gadget.AddContent(

Scf.Content(3, "This is the Image Page",

Scf.Img(1000, ScfAlign.Center, ScfImageFit.Auto,

"alt picture text")));

// This is the image itself (that appears on the image page)

Stream imageStream = Assembly.GetEntryAssembly().

GetManifestResourceStream(

"SimpleSideShowGadget.Aidan And Jeff.jpg");

gadget.AddContent(1000, ImageContentTransforms.ReduceColorDepth |

ImageContentTransforms.StretchToFit,

Image.FromStream(imageStream));

}

}

Capabilities, Events, and Notifications

A gadget application can query Windows, asking it how many devices are connected to it and what the capabilities are of these devices. Examples of capabilities include DeviceId, ScreenType (Bitmap or Text), ScreenWidth, ScreenHeight, ClientAreaWidth, ClientAreaHeight, ColorDepth, and SupportedImageFormats.

A gadget application could use the device capabilities when deciding what content to push down to a device. The sample application associated with this article demonstrates how a gadget application can query the number of devices connected to it and can also query the capabilities of these devices.

When running, a gadget application can receive events (described in Figure 11) from a device by registering with several events. The SideShowGadget type (the base class of the all gadget types) offers events that apply to all gadgets regardless of endpoint. The ScfSideShowGadget type (derived from SideShowGadget) offers additional events specific to the SCF gadget endpoint. An application can respond to these events in order to add or remove content. A sophisticated gadget may require that it always be executing in order to properly respond to these events. If this is required, the gadget should set its OnlineOnly registry value to 1.

Figure 11 Device Events to which Your Gadget Can Respond

Event

Description

DeviceAdded

Raised when the user connects a device to the gadget application. The receiving method is passed the device capabilities of the device.

DeviceRemoved

Raised when the user disconnects a device from the gadget application. The receiving method is passed the device capabilities of the device.

AllDevicesRemoved

Raised when the user has disconnected all devices from a gadget. Usually a non-UI gadget will terminate itself when it receives this event.

ContentMissing

Raised when the user attempts to navigate to content on the device that is missing. The receiving method is passed the ID of the missing content. As mentioned previously, the managed wrapper saves all content in a collection. If a device discards content due to the gadget's caching policy (as set by its CacheAlgorithm registry value), then the managed wrapper automatically sends the content back to the device and your code will not receive this event. This is one of the big benefits of using the managed API over the native API.

GadgetEnter

Raised when the user enters the gadget on the device.

GadgetExit

Raised when the user exits the gadget on the device.

ContentNavigate

For SCF gadgets only. Raised when the user navigates from one content page to another. The receiving method is passed the previous page ID, the target page ID, and the button that caused the navigation to occur.

MenuSelect

For SCF gadgets only. Raised when the user selects an item from a menu. The receiving method is passed the previous page ID, the target page ID, the menu item ID, and the button that caused the menu selection to occur.

ContextMenuSelect

For SCF gadgets only. Raised when the user selects an item from a context menu. The receiving method is passed the previous page ID, the target page ID, the previous item ID, the menu page ID, and the menu item ID.

When running, a gadget can send a notification to a SideShow device by calling SideShowGadget's ShowNotification method. A gadget application would show a notification for things such as meeting reminders, e-mail alerts, an incoming instant message, or system alerts. A notification is similar to a message box except that it appears on the hardware device. A notification consists of an ID number, an icon, a string caption, a string message, and an expiration time. The device will then show the notification until it is dismissed by the user or until the expiration time has elapsed. An application can also revoke a notification by calling SideShowGadget's RevokeNotification method passing it the notification's ID number.

Conclusion

In this article, I have covered the goals of the Windows SideShow feature and explained in depth how an end user interacts and configures the gadget. I've also shown how developers can integrate SideShow support into their new or existing managed applications. By using these managed wrappers, you should easily be able to integrate SideShow functionality into your own applications.

Personally, I find the new Windows SideShow feature very exciting as it offers new ways for users to access and interact with their data. I expect a slew of new hardware devices that support SideShow to become available in the next few years. In particular, I can't wait until Media Center remote controls and mobile phones support SideShow. It shouldn't take long for these incredibly useful devices to be as commonplace as a PDA or a cell phone.

Jeffrey Richter is a cofounder of Wintellect (www.Wintellect.com), a training and consulting firm. He is the author of several books, including CLR via C# (Microsoft Press, 2006). Jeffrey is also a contributing editor to MSDN Magazine and has been consulting with Microsoft since 1990.

Debug Leaky Apps

Identify And Prevent Memory Leaks In Managed Code

James Kovacs

This article discusses:

- Understanding memory leaks in managed apps

- Unmanaged memory used in .NET apps

- Helping the .NET garbage collector do its job

This article uses the following technologies:

.NET Framework

Code download available at: MemoryLeaks2007_01.exe (163 KB)

Browse the Code Online

Contents

Memory in .NET Applications

Checking for Leaks

Leaking Stack Memory

Leaking Unmanaged Heap Memory

"Leaking" Managed Heap Memory

Conclusion

The first reaction many developers have to the idea of memory leaks in managed code is that it's not possible. After all, the garbage collector (GC) takes care of all memory management, right? The garbage collector only handles managed memory, though. There are a number of places where unmanaged memory is used in Microsoft® .NET Framework-based applications, either by the common language runtime (CLR) itself, or explicitly by the programmer when interoperating with unmanaged code. There are also occasions where the GC seems to be shirking its duties and not efficiently handling managed memory. Usually this is caused by subtle (or not so subtle) programming errors that hinder the GC from performing its job. As good memory citizens, we still have to profile our applications to ensure they are leak-free and make efficient use of the memory they require.

Memory in .NET Applications

As you probably know, .NET applications make use of several types of memory: the stack, the unmanaged heap, and the managed heap. Here's a little refresher.

The Stack The stack is where local variables, method parameters, return values, and other temporary values are stored during the execution of an application. A stack is allocated on a per-thread basis and serves as a scratch area for the thread to perform its work. The GC is not responsible for cleaning up the stack because the space on the stack reserved for a method call is automatically cleaned up when a method returns. Note, however, that the GC is aware of references to objects stored on the stack. When an object is instantiated in a method, its reference (a 32-bit or 64-bit integer depending on the platform) is kept on the stack, but the object itself is stored on the managed heap and is collected by the garbage collector once the variable has gone out of scope.

The Unmanaged Heap The unmanaged heap is used for runtime data structures, method tables, Microsoft intermediate language (MSIL), JITed code, and so forth. Unmanaged code will allocate objects on the unmanaged heap or stack depending on how the object is instantiated. Managed code can allocate unmanaged heap memory directly by calling into unmanaged Win32® APIs or by instantiating COM objects. The CLR itself uses the unmanaged heap extensively for its data structures and code.

The Managed Heap The managed heap is where managed objects are allocated and it is the domain of the garbage collector. The CLR uses a generational, compacting GC. The GC is generational in that it ages objects as they survive garbage collections; this is a performance enhancement. All versions of the .NET Framework have used three generations, Gen0, Gen1, and Gen2 (from youngest to oldest). The GC is compacting in that it relocates objects on the managed heap to eliminate holes and keep free memory contiguous. Moving large objects is expensive and therefore the GC allocates them on a separate Large Object Heap, which does not compact. For more information on the managed heap and GC, see Jeffrey Richter's two part series, "Garbage Collection: Automatic Memory Management in the Microsoft .NET Framework" and "Garbage Collection-Part 2: Automatic Memory Management in the Microsoft .NET Framework". Although the articles were written based on the .NET Framework 1.0, the core concepts have not changed in versions 1.1 or 2.0, although the .NET GC has improved since then.

Checking for Leaks

There are a number of telltale signs that an application is leaking memory. Maybe it's throwing an OutOfMemoryException. Maybe its responsiveness is growing very sluggish because it started swapping virtual memory to disk. Maybe memory use is gradually (or not so gradually) increasing in Task Manager. When a memory leak is suspected, you must first determine what kind of memory is leaking, as that will allow you to focus your debugging efforts in the correct area. Use PerfMon to examine the following performance counters for the application: Process/Private Bytes, .NET CLR Memory/# Bytes in All Heaps, and .NET CLR LocksAndThreads/# of current logical Threads. The Process/Private Bytes counter reports all memory that is exclusively allocated for a process and can't be shared with other processes on the system. The .NET CLR Memory/# Bytes in All Heaps counter reports the combined total size of the Gen0, Gen1, Gen2, and large object heaps. The .NET CLR LocksAndThreads/# of current logical Threads counter reports the number of logical threads in an AppDomain. If an application's logical thread count is increasing unexpectedly, thread stacks are leaking. If Private Bytes is increasing, but # Bytes in All Heaps remains stable, unmanaged memory is leaking. If both counters are increasing, memory in the managed heaps is building up.

Leaking Stack Memory

Although it is possible to run out of stack space, which results in a StackOverflowException in the managed world, any stack space used during a method call is reclaimed once that method returns. Therefore, there are only two real ways to leak stack space. The first is to have a method call that consumes significant stack resources and that never returns, thereby never releasing the associated stack frame. The other is by leaking a thread, and thus that thread's entire stack. If an application creates worker threads for performing background work, but neglects to terminate them properly, thread stacks can be leaked. By default, the stack size on modern desktop and server versions of Windows® is 1MB. So if an application's Process/Private Bytes is periodically jumping in 1MB increments with a corresponding increase in .NET CLR LocksAndThreads/# of current logical Threads, a thread stack leak is very likely the culprit. Figure 1 shows one example of improper thread cleanup caused by (purposely bad) multithreaded logic.

Figure 1 Buggy Thread Cleanup

using System;

using System.Threading;

namespace MsdnMag.ThreadForker {

class Program {

static void Main() {

while(true) {

Console.WriteLine(

"Press <ENTER> to fork another thread...");

Console.ReadLine();

Thread t = new Thread(new ThreadStart(ThreadProc));

t.Start();

}

}

static void ThreadProc() {

Console.WriteLine("Thread #{0} started...",

Thread.CurrentThread.ManagedThreadId);

// Block until current thread terminates - i.e. wait forever

Thread.CurrentThread.Join();

}

}

}

A thread is launched, which displays its thread ID and then tries to Join on itself. Join causes the calling thread to block waiting on the other thread to terminate. So the thread is caught in a chicken-or-egg scenario-the thread is waiting for itself to terminate. Watch this program under Task Manager to see its memory usage increase by 1MB, the size of a thread stack, every time <Enter> is pressed.

The reference to the Thread object is being dropped every time through the loop, but the GC does not reclaim the memory allocated for the thread stack. A managed thread's lifetime is independent of the Thread object that creates it, a very good thing given that you wouldn't want the GC to terminate a thread that was still doing work simply because you lost all references to the associated Thread object. So the GC is collecting the Thread object, but not the actual managed thread. The managed thread does not exit (and the memory for its thread stack is not released) until its ThreadProc returns or it is explicitly killed. So if a managed thread is not properly terminated, the memory allocated to its thread stack will leak.

Leaking Unmanaged Heap Memory

If total memory use is increasing, but logical thread count and managed heap memory is not increasing, there is a leak in the unmanaged heap. We will examine some common causes for leaks in the unmanaged heap, including interoperating with unmanaged code, aborted finalizers, and assembly leaks.

Interoperating with Unmanaged Code One source of memory leaks involves interoperating with unmanaged code, such as when C-style DLLs are used through P/Invoke and COM objects through COM interop. The GC is unaware of unmanaged memory, and thus a leak here is due to a programming error in the managed code using the unmanaged memory. If an app is interoperating with unmanaged code, step through the code and examine memory usage before and after the unmanaged call to verify that memory is being reclaimed properly. If it isn't, look for the leak in the unmanaged component using traditional debugging techniques.

Aborted Finalizers A very insidious leak occurs when an object's finalizer does not get called, and it contains code to clean up unmanaged memory allocated by the object. Under normal conditions, finalizers will get called, but the CLR does not make any guarantees. While this may change in the future, current versions of the CLR use only one finalizer thread. Consider a misbehaving finalizer trying to log information to a database that is offline. If that misbehaving finalizer erroneously tries over and over again to access the database, never returning, the "well-behaved" finalizer will never get a chance to run. This problem can manifest itself very sporadically because it depends on the order of finalizers on the finalization queue as well as the behavior of other finalizers.

When an AppDomain is torn down, the CLR will attempt to clear the finalizer queue by running all finalizers. A stalled finalizer can prevent the CLR from completing the AppDomain tear down. To account for this, the CLR implements a timeout on this process, after which it will stop the finalization process. Typically, this isn't the end of the world, as most applications only have one AppDomain, and its teardown is due to the process being shut down. When an OS process is shut down, its resources will be recovered by the operating system. Unfortunately, in a hosting situation such as ASP.NET or SQL Server™, the teardown of the AppDomain doesn't mean the teardown of the hosting process. Another AppDomain can be spun up in the same process. Any unmanaged memory that was leaked by a component because its finalizer didn't run will still be sitting around unreferenced, unreachable, and taking up space. This can be disastrous as more and more memory is leaked over time.

In .NET 1.x, the only solution was to tear down the process and start again. The .NET Framework 2.0 introduces critical finalizers, which indicate that a finalizer will be cleaning up unmanaged resources and must be given a chance to run during AppDomain teardown. See Stephen Toub's article, "Keep Your Code Running with the Reliability Features of the .NET Framework" for more information.

Assembly Leaks Assembly leaks are relatively common and are caused by the fact that once an assembly is loaded, it can't be unloaded until the AppDomain is unloaded. In most cases, this is not a problem unless assemblies are being dynamically generated and loaded. Let's now look at dynamic code generation leaks, and specifically XmlSerializer leaks, in more detail.

Dynamic Code Generation Leaks Sometimes code needs to be generated dynamically. Maybe the application has a macro scripting interface for extensibility similar to Microsoft Office. Maybe a bond-pricing engine needs to load the pricing rules dynamically so end users can create their own bond types. Maybe the application is a dynamic language runtime/compiler for Python. In many cases, it is desirable to compile the macros, pricing rules, or code to MSIL for performance reasons. System.CodeDom can be used to generate MSIL on the fly.

The code in Figure 2 dynamically generates an assembly in memory. It can be called repeatedly without a problem. Unfortunately if the macro, pricing rule, or code changes, the dynamic assembly must be regenerated. The old assembly will no longer be used, but there is no way to evict it from memory, short of unloading the AppDomain in which the assembly was loaded. The unmanaged heap memory, which is used for its code, JITed methods, and other runtime data structures, has been leaked. (Managed memory has also been leaked in the form of any static fields on the dynamically generated classes.) There is no magic formula to detect this problem. If you're dynamically generating MSIL using System.CodeDom, check whether you regenerate code. If you do, you're leaking unmanaged heap memory.

Figure 2 Dynamically Generating an Assembly in Memory

CodeCompileUnit program = new CodeCompileUnit();

CodeNamespace ns = new

CodeNamespace("MsdnMag.MemoryLeaks.CodeGen.CodeDomGenerated");

ns.Imports.Add(new CodeNamespaceImport("System"));

program.Namespaces.Add(ns);

CodeTypeDeclaration class1 = new CodeTypeDeclaration("CodeDomHello");

ns.Types.Add(class1);

CodeEntryPointMethod start = new CodeEntryPointMethod();

start.ReturnType = new CodeTypeReference(typeof(void));

CodeMethodInvokeExpression cs1 = new CodeMethodInvokeExpression(

new CodeTypeReferenceExpression("System.Console"), "WriteLine",

new CodePrimitiveExpression("Hello, World!"));

start.Statements.Add(cs1);

class1.Members.Add(start);

CSharpCodeProvider provider = new CSharpCodeProvider();

CompilerResults results = provider.CompileAssemblyFromDom(

new CompilerParameters(), program);

There are two main techniques for solving this problem. The first is to load the dynamically generated MSIL into a child AppDomain. The child AppDomain can be unloaded when the generated code changes and a new one spun up to host the updated MSIL. This technique works on all versions of the .NET Framework.

Another technique introduced in .NET Framework 2.0 is lightweight code generation, also known as dynamic methods. Using a DynamicMethod, MSIL op codes are explicitly emitted to define the method body, and then the DynamicMethod is invoked either directly via DynamicMethod.Invoke or via a suitable delegate.

DynamicMethod dm = new DynamicMethod("tempMethod" +

Guid.NewGuid().ToString(), null, null, this.GetType());

ILGenerator il = dm.GetILGenerator();

il.Emit(OpCodes.Ldstr, "Hello, World!");

MethodInfo cw = typeof(Console).GetMethod("WriteLine",

new Type[] { typeof(string) });

il.Emit(OpCodes.Call, cw);

dm.Invoke(null, null);

The main advantage of dynamic methods is that the MSIL and all related code generation data structures are allocated on the managed heap. This means that the GC can reclaim the memory once the last reference to the DynamicMethod goes out of scope.

XmlSerializer Leaks Portions of the .NET Framework, such as the XmlSerializer, use dynamic code generation internally. Consider the following typical XmlSerializer code:

XmlSerializer serializer = new XmlSerializer(typeof(Person));

serializer.Serialize(outputStream, person);

The XmlSerializer constructor will generate a pair of classes derived from XmlSerializationReader and XmlSerializationWriter by analyzing the Person class using reflection. It will create temporary C# files, compile the resulting files into a temporary assembly, and finally load that assembly into the process. Code gen like this is also relatively expensive. So the XmlSerializer caches the temporary assemblies on a per-type basis. This means that the next time an XmlSerializer for the Person class is created, the cached assembly is used rather than a new one generated.

By default, the XmlElement name used by the XmlSerializer is the name of the class. Thus, Person would be serialized as:

<?xml version="1.0" encoding="utf-8"?>

<Person xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

xmlns:xsd="http://www.w3.org/2001/XMLSchema">

<Id>5d49c002-089d-4445-ac4a-acb8519e62c9</Id>

<FirstName>John</FirstName>

<LastName>Doe</LastName>

</Person>

Sometimes it is necessary to change the root element name without changing the class name. (The root element name might be required for compatibility with an existing schema.) So Person may have to be serialized as <PersonInstance>. Conveniently, there is an overload of the XmlSerializer constructor that takes the root element name as its second parameter, like this:

XmlSerializer serializer = new XmlSerializer(typeof(Person),

new XmlRootAttribute("PersonInstance"));

When the app starts serializing/deserializing Person objects, everything works until an OutOfMemoryException is thrown. This overload of the XmlSerializer constructor does not cache the dynamically generated assembly, but generates a new temporary assembly every time you instantiate a new XmlSerializer! The app is leaking unmanaged memory in the form of temporary assemblies.

To fix the leak, use the XmlRootAttribute on the class to change the root element name of the serialized type:

[XmlRoot("PersonInstance")]

public class Person {

// code

}

If the attribute is applied directly to the type, the XmlSerializer caches the generated assemblies for the type and there is no leak. If root element names need to be dynamically switched, the application can perform the caching of the XmlSerializer instances itself by using a factory to retrieve them:

XmlSerializer serializer = XmlSerializerFactory.Create(

typeof(Person), "PersonInstance");

XmlSerializerFactory is a class I created that checks whether a Dictionary<TKey, TValue> contains an XmlSerializer for Person using the PersonInstance root element name. If it does, the instance is returned. If not, a new one is created, stored in the hash table, and returned to the caller.

"Leaking" Managed Heap Memory

Now let's turn our attention to "leaking" managed memory. When dealing with managed memory, the GC takes care of most of the work for us. We do need to provide the GC with the information it needs to do its job. However, there are a number of scenarios that prevent the GC from doing its job efficiently and result in higher managed memory use than would otherwise be required. These situations include large object heap fragmentation, unneeded rooted references, and a midlife crisis.

Large Object Heap Fragmentation If an object is 85,000 bytes or larger, it is allocated on the large object heap. Note that this is the size of the object itself and not any children. Take the following class as an example:

public class Foo {

private byte[] m_buffer = new byte[90000]; // large object heap

}

Foo instances would be allocated on the normal generational managed heap as it only contains a 4-byte (32-bit Framework) or 8-byte (64-bit Framework) reference to the buffer, plus some other housekeeping data used by the .NET Framework. The buffer would be allocated on the large object heap.

Unlike the rest of the managed heap, the Large Object Heap is not compacted due to the cost of moving the large objects. So as large objects are allocated, freed, and cleaned up, gaps will appear. Depending on usage patterns, the gaps in the large object heap can result in significantly more memory usage than is required by the currently allocated large objects. The LOHFragmentation application that is included in this month's download demonstrates this by randomly allocating and freeing byte arrays in the Large Object Heap. Some runs of the application result in the newly created byte arrays fitting nicely into the gaps left by freed byte arrays. On other runs of the application, this is not the case and the memory required is much larger than the memory required for the currently allocated byte arrays. To visualize fragmentation of the large object heap, use a memory profiler, such as the CLRProfiler. The red regions in Figure 3 are allocated byte arrays whereas white regions are unallocated space.

Figure 3 The Large Object Heap in CLRProfiler (Click the image for a larger view)

There is no single solution for avoiding Large Object Heap fragmentation. Examine how the application uses memory and specifically the types of objects that are on the large object heap using tools like the CLRProfiler. If the fragmentation is due to re-allocating buffers, maintain a fixed set of buffers that are reused. If the fragmentation is being caused by concatenation of large numbers of strings, examine whether the System.Text.StringBuilder class can reduce the number of temporary strings created. The basic strategy is to determine how to reduce the application's reliance on temporary large objects, which are causing the gaps in the large object heap.

Unneeded Rooted References Let's consider how the GC determines when it can reclaim memory. When the CLR attempts to allocate memory and has insufficient memory in reserve, it performs a garbage collection. The GC enumerates all rooted references, including static fields and in-scope local variables on any thread's call stack. It marks these references as reachable and follows any references these objects contain, marking them as reachable as well. It continues this process until it has visited all reachable references. Any unmarked objects are not reachable and hence are garbage. The GC compacts the managed heap, tidies up references to point to their new location in the heap, and returns control to the CLR. If sufficient memory has been freed, the allocation proceeds using this freed memory. If not, additional memory is requested from the operating system.

If we forget to null out rooted references, the GC is prevented from efficiently freeing memory as quickly as possible, resulting in a larger memory footprint for the application. The problem can be subtle, such as a method that creates a large graph of temporary objects before making a remote call like a database query or call to a Web service. If a garbage collection happens during the remote call, the entire graph is marked reachable and is not collected. This becomes even more costly because objects surviving a collection are promoted to the next generation, which can lead to a midlife crisis.

Midlife Crisis A midlife crisis does not cause an application to go out and buy a Porsche. It can, however, cause an overuse of managed heap memory and excessive amounts of processor time spent in the GC. As mentioned previously, the GC uses a generational algorithm, which is predicated on the heuristic that if an object has lived a while, it will probably live a while longer. For example, in a Windows Forms application, the main form is created when the application starts and the application exits when the main form closes. It is wasteful for the GC to continually verify that the main form is being referenced. When the system requires memory to satisfy an allocation request, it first performs a Gen0 collection. If sufficient memory is not available, a Gen1 collection is performed. If the allocation request still can't be satisfied, a Gen2 collection is performed, which involves an expensive sweep of the entire managed heap. Gen0 collections are relatively inexpensive because only recently allocated objects are considered for collection.

A midlife crisis occurs when objects tend to live until Gen1 (or worse, Gen2), but die shortly thereafter. This has the effect of turning cheap Gen0 collections into much more expensive Gen1 (or Gen2) collections. How can this occur? Take a look at the following code:

class Foo {

~Foo() { }

}

This object will always be reclaimed in a Gen 1 collection! The finalizer, ~Foo(), allows us to implement cleanup code for our objects that, barring a rude AppDomain abort, will run before the object's memory is freed. The GC's job is to free up as much managed memory as possible as quickly as possible. Finalizers are user-written code and can do absolutely anything. Although not recommended, a finalizer could do something silly such as logging to a database or calling Thread.Sleep(int.MaxValue). So when the GC finds an unreferenced object with a finalizer, it places the object on the finalization queue and moves on. The object has survived a garbage collection and hence is promoted a generation. There is even a performance counter for this: .NET CLR Memory-Finalization Survivors, which is the number of objects during the last garbage collection that survived due to a finalizer. Eventually the finalizer thread will run the object's finalizer and it can subsequently be collected. But you have turned a cheap Gen0 collection into a Gen1 collection, all by simply adding a finalizer!

In most cases, finalizers are not necessary when writing managed code. They are only needed when a managed object holds a reference to an unmanaged resource that needs cleanup, and even then you should use a SafeHandle-derived type to wrap the unmanaged resource rather than implementing a finalizer. Additionally, if you're using unmanaged resources or other managed types that implement IDisposable, implement the Dispose pattern to allow users of the object to aggressively clean up the resources and avoid any related finalization.

If an object only holds references to other managed objects, the GC will clean up unreferenced objects. This is in stark contrast to C++, where delete must be called on child objects. If a finalizer is empty or simply nulling out references to child objects, remove it. It is hurting performance by needlessly promoting the object to an older generation, making them more expensive to clean up.

There are other ways to get into a midlife crisis, such as holding onto objects before making a blocking call like querying a database, blocking on another thread, or calling a Web service. During the call, one or more collections can occur and result in cheap Gen0 objects being promoted to a later generation, again resulting in much higher memory usage and collection costs.

There is an even more subtle case that occurs with event handlers and callbacks. I will use ASP.NET as an example, but the same type of problem can occur in any application. Consider performing an expensive query and wanting to cache the results for 5 minutes. The query is page-specific and based on query-string parameters. To monitor caching behavior, an event handler logs when an item is removed from the cache (see Figure 4).

Figure 4 Logging Items Removed from Cache

protected void Page_Load(object sender, EventArgs e) {

string cacheKey = buildCacheKey(Request.Url, Request.QueryString);

object cachedObject = Cache.Get(cacheKey);

if(cachedObject == null) {

cachedObject = someExpensiveQuery();

Cache.Add(cacheKey, cachedObject, null,

Cache.NoAbsoluteExpiration,

TimeSpan.FromMinutes(5), CacheItemPriority.Default,

new CacheItemRemovedCallback(OnCacheItemRemoved));

}

... // Continue with normal page processing

}

private void OnCacheItemRemoved(string key, object value,

CacheItemRemovedReason reason) {

... // Do some logging here

}

This innocuous-looking code contains a major problem. All of these ASP.NET Page instances just became long-lived objects. The OnCacheItemRemoved is an instance method and the CacheItemRemovedCallback delegate contains an implicit this pointer, where this is the Page instance. The delegate is added to the Cache object. So there now exists a dependency from the Cache to the delegate to the Page instance. When a garbage collection occurs, the Page instance remains reachable from a rooted reference, the Cache object. The Page instance (and all the temporary objects it created while rendering) will now have to wait for at least five minutes before being collected, during which time they will likely be promoted to Gen2. Fortunately this example has a simple solution. Make the callback function static. The dependency on the Page instance is broken and it can now be collected cheaply as a Gen0 object.

Conclusion

I have discussed a variety of problems in .NET applications that can lead to memory leaks or overconsumption of memory. Although .NET reduces the need for you to be concerned with memory, you still must pay attention to your application's use of memory to ensure that it is well-behaved and efficient. Just because an application is managed doesn't mean you can throw good software engineering practices out the window and count on the GC to perform magic. You must continue to monitor your application's memory performance counters during the development and testing process. But it's worth it. Remember, a well-behaved application means happy customers.

James Kovacs is an independent architect, developer, trainer, and jack-of-all-trades living in Calgary, Alberta specializing in the .NET Framework, security, and enterprise application development. He is a Microsoft MVP for Solutions Architecture and received his Masters degree from Harvard University. James can be reached at jkovacs@post.harvard.edu or www.jameskovacs.com.

Table of Contents

MSDN Magazine

All articles

WPF: Flexible Content Display With Flow Documents

OPC: A New Standard For Packaging Your Data

Excel Services: Develop A Calculation Engine For Your Apps

Mobility: Make Your WPF Apps Power-Aware

Share Code: Write Code Once For Both Mobile And Desktop Apps

NTFS: Enhance Your Apps With File System Transactions

C# 3.0: The Evolution Of LINQ And Its Impact On The Design Of C#

OFFICE UI: New VSTO Features Help You Customize Word And Outlook

Got Game?: Unleash Your Imagination With XNA Game Studio Express

WPF: Customizing Controls For Windows Presentation Foundation

Migration: Convert A Java Web Application To ASP.NET Using JLCA

Collaborate: Help Teams Work Together With Web Services And Groove 2007

Identity: Secure Your ASP.NET Apps And WCF Services With Windows CardSpace

Aero Glass: Create Special Effects With The Desktop Window Manager

Event Tracing: Improve Debugging And Performance Tuning With ETW

IIS 7.0: Explore The Web Server For Windows Vista And Beyond

.NET Security: Support Certificates In Your Applications With The .NET Framework 2.0

Digital Media: Add Video To Controls And 3D Surfaces With WPF

WiX Tricks: Automate Releases With MSBuild And Windows Installer XML

ASP.NET 2.0: Manage Web Users With Custom Profile Providers

SQL Server 2005: Regular Expressions Make Pattern Matching And Data Extraction Easier

Vista and Office: View Data Your Way With Our Managed Preview Handler Framework

SideShow Gadgets: Get Started Writing Gadgets For Windows SideShow Devices

Debug Leaky Apps: Identify And Prevent Memory Leaks In Managed Code