Next Section

• | Table of Contents
XML: A Manager's Guide, Second Edition
By Kevin Dick
Publisher | : Addison Wesley
Pub Date | : August 28, 2002
ISBN | : 0-201-77006-7
Pages | : 320

Extensible Markup Language (XML) has revolutionized distributed computing. By providing a standard means for specifying the structure of information, XML enables sophisticated e-commerce systems and facilitates interoperable enterprise software. Knowing how to leverage XML's technical capabilities into business value has become an important asset for managers.

Fully updated and expanded to incorporate the latest in XML technology advances and its application, XML: A Manager's Guide, Second Edition serves as a concise guide for managers as well as a starting point for developers. It helps managers build a working knowledge of XML's capabilities so they can communicate intelligently with XML developers and make informed decisions about when to use the technology. This book also provides manager-specific information about software acquisition, staffing, and project management.

The author provides practical experience-based information on the following key topics:

  • How XML fills a critical gap in Internet standards

  • Emerging standards that complete the XML paradigm

  • XML messaging and Web services

  • The software infrastructure needed to deploy XML applications

  • Staffing and project management advice for XML development teams

  • Preventing the problems commonly encountered on XML development projects

  • Applications for enterprises and vendors

Designed to let you quickly and easily access exactly the information you require, this book clearly delineates different paths through the chapters based on your needs, provides executive briefings for every chapter, and includes fast-track summaries of major points in the margins.

With this book in hand, you will have a firm grasp of the fundamental technology, competitive advantages, and potential best uses of the XML standard.

Next Section


• | Table of Contents
XML: A Manager's Guide, Second Edition
By Kevin Dick
Publisher | : Addison Wesley
Pub Date | : August 28, 2002
ISBN | : 0-201-77006-7
Pages | : 320

Copyright
Foreword
Preface
 | Purpose of This Book
 | Who Should Read This Book
 | Organization of this Book
Acknowledgments
Chapter 1. The Internet Crisis: Exchanging Information
 | Connections Without Understanding
 | Convergence of Information Exchange Problems
 | Metadata Standard Is a First Step
 | Shared Context Standards Deliver True Understanding
 | The XML Approach
Chapter 2. XML Basics
 | Executive Summary
 | Jumping In
 | XML Conceptual Model
 | Introducing Elements
 | Introducing Attributes
 | Creating an "Order" in XML
 | Introducing Document Types
 | Introducing Entities
 | Technical Summary
Chapter 3. Related Standards
 | Executive Summary
 | Overview of Namespaces
 | Overview of XPath, XPointer, and XQuery
 | Overview of XSLT
 | Overview of XSL
 | Overview of XLink
 | Overview of XML Schema
 | Technical Standards
 | Infrastructure Standards
 | Technical Summary
Chapter 4. XML Messaging and Web Services
 | Executive Summary
 | Motivation
 | Protocol Design Problem
 | Messaging Architectures
 | Major Web Services Initiatives
 | Other Web Services Initiatives
Chapter 5. XML Software Infrastructure
 | Executive Summary
 | Fundamental Components
 | Storage Systems
 | Server Infrastructure
 | Data-Oriented Components
 | Content-Oriented Components
 | Design Tools
 | Infrastructure Selection Strategy
Chapter 6. Processes and People
 | Executive Summary
 | XML Applications and Change
 | Content Documents
 | Business Documents
 | Data Documents
 | Common Issues
Chapter 7. Five XML Applications for Enterprises
 | Executive Summary
 | Workforce Automation
 | Knowledge Management
 | Trading Partner Coordination
 | Application Integration
 | Data Integration
Chapter 8. Five XML Applications for Vendors
 | Executive Summary
 | Flexible Content Provisioning
 | Information Aggregation
 | Application Services
 | Configuration and Logging Files
 | Distributed Protocol
Glossary


Copyright

Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in this book, and Addison-Wesley was aware of a trademark claim, the designations have been printed with initial capital letters or in all capitals.

The author and publisher have taken care in the preparation of this book, but make no expressed or implied warranty of any kind and assume no responsibility for errors or omissions. No liability is assumed for incidental or consequential damages in connection with or arising out of the use of the information or programs contained herein.

The publisher offers discounts on this book when ordered in quantity for bulk purchases and special sales. For more information, please contact:

U.S. Corporate and Government Sales

(800) 382-3419

corpsales@pearsontechgroup.com

For sales outside of the U.S., please contact:

International Sales

(317) 581-3793

international@pearsontechgroup.com

Visit Addison-Wesley on the Web: www.awprofessional.com

Library of Congress Cataloging in Publication Data

Dick, Kevin, 1969–

XML : a manager's guide / Kevin Dick.—2nd ed.

p. cm.

ISBN 0-201-77006-7

1. XML (Document markup language) I. Title.

QA76.76.H94 D534 2003

005.7'2—dc21 2002071656

Copyright © 2003 by Pearson Education, Inc.

All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form, or by any means, electronic, mechanical, photocopying, recording, or otherwise, without the prior consent of the publisher. Printed in the United States of America. Published simultaneously in Canada.

For information on obtaining permission for use of material from this work, please submit a written request to:

Pearson Education, Inc.

Rights and Contracts Department

75 Arlington Street, Suite 300

Boston, MA 02116

Fax: (617) 848-7047

Text printed on recycled paper

1 2 3 4 5 6 7 8 9 10—MA—0605040302

First printing, August 2002

Dedication

For my wife, Jane. I couldn't accomplish anything meaningful without your love, encouragement, and humor to support me. Also for my son, Shep. Thanks for helping me keep everything in perspective.


Foreword

David A. Taylor, Ph.D.

When I wrote the foreword to the first edition of this book, I stressed the importance of keeping up with emerging technologies in a time of rapid innovation. Today, those words are no longer appropriate to this book. In just three short years, XML has gone from a promising technology to the new standard for Internet communications, which makes it all the more urgent for managers to understand what XML is and how it can be used to gain maximum leverage from the Internet.

The problem with using the Internet to conduct business is that, even as the 'Net accelerates the flow of data, it strips away the structure that gives the data meaning. For example, suppose you are searching the Web for the price of a product you'd like to buy. Typing in the name of the product together with the word "price" *may* produce a Web page that gives you the answer you want, but that answer will be buried under an avalanche of pages that happen to include the name of the product and the word "price" but don't put the two together. The information may be out there somewhere, but you can't find it because all the meaning has been stripped away by the time it hits your screen.

XML—the extensible markup language—is the solution to this problem. Rather than just formatting text for display as HTML does, it communicates information as meaningful patterns. To do this, XML uses named attributes, and it organizes those attributes in a way that reflects the structure of real-world objects. In the search example, the price of a product would be defined as a property of that product, and a request for its price would either return the correct value or no value at all.

This example illustrates the potential of XML to improve the Web, but the real impact of XML is being felt on business communications, which typically bypass the Web and ride directly on the Internet. Here, XML provides a common language for transmitting such vital business documents as orders, queries, quotes, and notices. Instead of requiring a person to read these documents and interpret their meaning, computers can communicate with each other directly and understand each other in business terms. This kind of communication—which might have been viewed as science fiction a decade ago—now happens countless times every day.

Exciting as this development is, it's already yesterday's news. The newest breakthrough is Web Services, in which computers exchange Internet messages directly, without human intervention, and request services from each other. These requests automatically call up the required functionality and return not just data but meaningful responses, including business commitments. Thanks to an unusual level of cooperation among standards bodies, software companies and corporate customers, the standards to make Web services a reality are already in place. In addition to XML, Web Services use SOAP—the Simple Object Access Protocol—and a set of registration services to let programs discover each other and invoke each others' capabilities.

At present, Web services are being used primarily for specific, isolated requests, such as requesting a price or scheduling an activity. As the use of Web Services becomes accepted practice, a new generation of software will emerge that operates entirely over the Internet, without being constrained to the capabilities of a single host machine. With software such as this, entire business communities, such as alliances and supply chains, can begin to function like a single enterprise.

In the midst of exciting developments such as these, no manager can afford to be ignorant of the concepts and applications of XML. Unfortunately, the widespread adoption of XML has made the search for a good management book on the subject as frustrating as the search for a price on the Web. As I write this, a query on Amazon nets 360 books explaining XML. By the time you read this, there will be many more.

Let me simplify your search. If you are a manager and you want to understand XML, look no further. This book offers up the clearest, most understandable explanation of XML you are ever likely to read, and it provides just the right level of information to help you make solid decisions without bogging you down in details. And the new edition just makes a good book better, with a new chapter devoted to Web services, updated advice based on lessons learned, and new applications to illustrate the way XML is changing the way companies do business.

If you are going to read one book on XML, this is the book.


Preface

    Purpose of This Book

    Who Should Read This Book

    Organization of this Book


Purpose of This Book

Extensible Markup Language (XML) is an important technology for exchanging structured information over intranets, extranets, and the Internet. As with many new software technologies, information flows into the developer community first. This flow starts electronically with e-mail lists, newsgroups, and technical Web sites. Then technology references, developer guides, and tutorials appear on bookstore shelves.

After developers use a technology to create some inspiring prototypes, the computing press usually latches on to the trend. Articles hail it as the solution to a wide variety of application development problems. Executives take notice of the press coverage. They may even hear about internal "skunkworks" projects. Quickly, they want assessments of how the technology will impact their organizations.

Managers get caught in the middle. They are at an information disadvantage when it comes to assessing the benefits of the technology and managing its use. Developers are pushing from the bottom to use the technology on projects, whereas executives are pushing from the top for formal technology planning. Unfortunately, information resources targeted specifically at managers are extremely limited. They must resort to a time-consuming process of scanning volumes of developer-oriented details and dissecting executive summaries to synthesize a manager's perspective. The need for this synthesis continues as a technology evolves because every new advance follows the information path from developers to executives to managers.

XML: A Manager's Guide addresses this problem for XML. It includes the following features.

  • An introduction to XML technology and software at a level that will allow managers to communicate with developers without having to become one.

  • Information about the processes and people managers will need for successful XML projects.

  • Inspiration for how to deliver value through XML, including an analysis of market adoption and the types of applications where it offers the most benefit.

There are two views of XML: as human readable content and as machine readable data. One of XML's advantages is that it enables developers to move information easily between perspectives. In traditional enterprise applications, however, XML as machine readable data usually influences more of the development effort. The book is somewhat biased toward the view of XML as data because this type of application is the author's primary area of expertise, but it covers both perspectives.


Who Should Read This Book

Obviously, this book targets managers. More specifically, it targets software development managers in (1) information systems (IS) organizations within enterprises and (2) product development groups within software vendors. To a great extent, however, the needs of these two managerial audiences intersect. They both need a basic understanding of the technology software, as well as guidance in the processes and people necessary for success, but their organizations require different XML applications. This book accommodates that difference.

Even within these two audiences, managers differ significantly in their individual backgrounds and managerial goals. Different managers will require different levels of detail for each of the three basic XML topics listed previously. To a certain extent, the level of detail required correlates with job responsibility, as shown in Table Preface-1.

As Table Preface-1 shows, technology planners and project managers will find this book the most helpful. Executives will also find portions of the book useful, but they may want to skip much of the detailed information. Application designers can benefit from the overview of XML technology, but they should also consult more technical resources such as developer guides.

Table Preface-1.
Major Book Topics
Example IS Titles Example Vendor Titles XML Technology Processes and People Inspiration
Executive CIO Director, IS

VP, Engineering

VP, Marketing

 Low Detail Low Detail Low Detail
Technology Planning

Director, Technology

Director, Architecture

CTO

Product Manager

 Medium Detail Medium Detail High Detail
Project Management

Program Manager

Project Manager

Director, Development

Development Manager

 Medium Detail High Detail Medium Detail
Application Design

Systems Architect

Systems Analyst

Chief Architect

Technical Lead

 High Detail Low Detail Medium Detail


Organization of this Book

This book is designed for either consecutive reading of chapters or picking out certain chapters for topics of interest. All chapters after Chapter 1 begin with an Executive Summary section that will help you determine if you should read the entire chapter. You identify relevant topics. The Fast Track paragraph summaries in the margins will also help. This book has eight chapters.

  • Chapter 1—The Internet Crisis: Exchanging Information. Motivates the need for XML by examining the requirements left unfilled by existing of Internet technologies.

  • Chapter 2—XML Basics. Introduces fundamental XML concepts at a level that enables you to understand XML applications without going into the details necessary to develop such applications. The Technical Summary section at the end of the chapter reviews basic XML concepts.

  • Chapter 3—Related Standards. Gives an overview of standards and emerging standards closely related to XML, including an analysis of their benefits and some simple example code. For established standards, it covers practical uses and experiences. It also has the Technical Summary section at the end.

  • Chapter 4—XML Messaging and Web Services. Introduces the motivation behind and concepts of XML messaging. Compares the different architectures for XML messaging and their respective requirements. Discusses some of the leading initiatives and examines how Web services have evolved into a complete distributed computing paradigm built on top of XML messaging. This chapter assumes familiarity with distributed computing concepts.

  • Chapter 5—XML Software Infrastructure. Examines the basic types of software infrastructure you will need to deploy XML-based applications. Discusses when different pieces of software are necessary and how to choose among alternatives. Briefly discusses major vendors and products.

  • Chapter 6—Processes and People. Analyzes the software development processes and staff necessary to deploy XML applications. The first part uses three general application categories to abstract common process and staff requirements. You probably want to concentrate on the category that matches the applications you plan to build. The second part discusses common problems on XML projects and how to prevent them.

  • Chapter 7—Five XML Applications for Enterprises. Presents an architectural overview of five XML applications important for enterprises and discusses the benefits of using XML in these applications.

  • Chapter 8—Five XML Applications for Vendors. Presents an architectural overview of five XML applications important for vendors of software products and services and discusses the benefits of using XML in these applications.

As you can see, Chapters 6 through 8 focus on the needs of project managers, enterprise technology planners, and vendor technology planners, respectively. If you don't fall into the primary audience for one of these chapters, you may wish to read only the Executive Summary. If you apply this advice to all chapters, Table Preface-2 suggests which parts of which chapters each target audience should read.

The Glossary at the end of the book defines many of the XML and Internet terms used in the book. Refer to it if you come across an unfamiliar term or simply want to refresh your memory of its definition. The first time a word defined in the Glossary appears, it is set in color. Terms specific to XML appear in italics, while general terms appear in roman typeface.

Table Preface-2.
Executive | Technology Planning | Project Management | Application Design
Chapter 1 | Entire | Entire | Entire | Entire
Chapter 2 | Summary | Entire | Entire | Entire
Chapter 3 | Summary | Entire | Summary | Entire
Chapter 4 | Summary | Entire | Selected Sections | Entire
Chapter 5 | Summary | Entire | Entire | Entire
Chapter 6 | Summary | Summary | Entire | Summary
Chapter 7 | Summary | Selected Sections | Summary | Selected Sections
Chapter 8 | Summary | Selected Sections | Summary | Selected Sections


Acknowledgments

This book would not be possible without the patience and hard work of Mary O'Brien, Jacquelyn Doucette, and the rest of the team at Addison-Wesley.

I'd like to thank my formal reviewers of the second edition:

Dan Applequist

Mark Baker

Mike Champion

Frank Coyle

Paulo Rios

Much gratitude for comments on portions of the manuscript to Patrick Anderson, David Orchard, and Eric Rescorla.

The second edition obviously rests on the solid foundation of the first edition. Thanks to its reviewers: Dan Applequist, David A. Epstein, Tim Kientzle, Alfie Kirkpatrick, Liam Quin, Lauren Wood. Those who took the time to review two drafts: Eve Maler, David A. Taylor, and David Turner. Last but not least, those who donated their friendly comments: Dad, Mom, Adam, Goutam, Matt, Rafe, Todd, and Walter.


Chapter 1. The Internet Crisis: Exchanging Information

    Connections Without Understanding

    Convergence of Information Exchange Problems

    Metadata Standard Is a First Step

    Shared Context Standards Deliver True Understanding

    The XML Approach


Connections Without Understanding

Without question, the Internet is a revolutionary phenomenon. In just a few years, it has completely transformed the landscapes of both personal and enterprise computing. The word processor used to be the most important application on a personal computer. Now it's rivaled by the Web browser. The database used to be the most important piece of software in an enterprise computing architecture. Now it's rivaled by the Web application server. What makes the Internet so compelling?

Connectivity Is the Key

The revolutionary benefit of the Internet is connectivity. From a technical standpoint, the Internet is a networking protocol, TCP/IP, plus a suite of higher-level application protocols, such as FTP, HTTP, and SMTP. These protocols are all standardized. Anyone can access information on the Internet using the standard protocols. It's easy to get a connection, and that connection provides access to a broad range of resources.

This easy connectivity creates a positive feedback loop and has motivated a large number of people and organizations to connect themselves to the Internet. As more people and resources become connected, the value of connecting additional people and resources increases.

If the only resource on the Internet were an online bookstore, a few prolific readers might invest the necessary time and money to get a connection. But the fact that people can buy books, music, and computer hardware—as well as find information about companies, hobbies, and long-lost friends—makes hundreds of millions of people willing to connect themselves. The existence of all these users creates a market for even more products and services.

Similarly, if intranets provided enterprises with only the ability to connect their employees to human resource applications, enterprises might invest a few million dollars in the technology. But because intranets can connect all employees to all applications, and connect these resources to customers and suppliers, enterprises are willing to spend tens or hundreds of millions of dollars on intranets.

Some people believe that Internet connectivity will snowball to the point where virtually everything containing a computer chip is connected. For example, sensors in your car's fuel pump could detect a problem and send a message to the car's central computer, which would then inform the computer at the auto repair shop. The repair shop's computer would automatically order new parts from the manufacturer. The car's central computer would then query your electronic datebook for a good time to bring your car to the shop, enter the appointment, and inform you via a screen on the dashboard. And you don't have to worry about forgetting your appointment because your electronic datebook would automatically alert you.

Unfulfilled Requirements

Achieving this vision where information seamlessly moves from device to device and server to server requires that the information transferred over the Internet have a great deal of structure. Without the necessary structure, the opportunities for misunderstandings accumulate to make chains of transfers impossible. Before the emergence of Extensible Markup Language (XML), the Internet did not have a structuring mechanism that was flexible enough to accommodate a wide variety of uses. Even relatively simple information transfers such as the following presented challenges.

  • Interchangeable files. Connecting everyone to the Internet naturally motivates people to try to share the information they've created. You've probably experienced the frustration of attempting such a file transfer only to discover that your audience can't read the file format you used. This problem is particularly frustrating in software development. Developers tend to have different preferences in the particular tools they like to use. Moreover, a given development project will often employ a variety of different types of tools. Getting all these tools to exchange files seamlessly is so difficult that most organizations force developers to use the same tools and restrict the types of tools used in the organization. These measures hurt developer productivity. The situation can be equally frustrating for other types of knowledge workers.

  • Customized page layout. Often you'd like to apply a customized layout to a Web page. Perhaps you're accessing the page from a small device such as a cellular phone and want to format it for the smaller display, or you have poor eyesight and would like a larger point size. Maybe you want to filter the Web page to display only recently updated information. With traditional technology, the Web page author must provide completely different pages to support each mode of access.

  • Downloadable product comparisons. The Web has a wealth of information about the features and prices of various products and services, making comparison shopping services popular. But suppose you use one of these services to construct a comparison matrix and then want to import it into a spreadsheet so that you can add your own information. You'd simply like to download the matrix and open it in your spreadsheet application. However, with early Web technology, you had to cut and paste each cell of the table manually into the spreadsheet.

  • Business integration. Many enterprises want to improve the efficiency of internal business processes by integrating different applications and databases. The Internet makes it possible for enterprises to connect these systems physically. Unfortunately, different applications have their own programming interfaces so they cannot understand each other. Furthermore, these applications access databases with different ways of representing data, so it's difficult to create a coherent view of all information related to a particular topic. Together, the lack of common structures for application interfaces and data representation makes it extremely challenging for enterprises to improve their global business processes continuously.

  • Trading partner coordination. Many enterprises also want to improve the efficiency of external business processes by integrating their systems with those of their trading partners. While the Internet offers an enormous advance in enabling these partners to exchange information electronically, they must still agree on how to interpret this information. In addition to all the problems inherent in business integration, trading partner coordination involves another set of issues regarding agreement on the terms of the trading relationship and the policies to apply during trading. Without mechanisms for establishing these agreements, partners cannot automatically process the information they receive from each other.

Whereas earlier Internet technologies have difficulty accommodating these tasks, XML does not. XML makes it easier for different people and different pieces of software to agree on the structure of information exchanged over the Internet.

Roots of the Connection Problem

Understanding how XML can help you overcome the challenges of universal connectivity begins with analyzing the root cause of this problem. The underlying issue is that each party that accesses a piece of information has a different perspective on what that information means. The more parties involved, the greater the opportunity for confusion.

Consider the case of spoken language. You probably find it easy to communicate with your immediate family and coworkers. Now expand the number of people with whom you communicate. As you expand outside your immediate geographical location, you will find that people use different colloquialisms. As you expand outside your profession, you will encounter unfamiliar jargon. And if you expand the number of people with whom you communicate far enough, you will encounter completely different languages. The more people with whom you speak, the greater the opportunity for miscommunication.

Now let's look at an example of different perspectives on the Internet. Suppose you plan to buy a CD-RW drive online. This is an easy enough task if you plan to visit just one online store, but what if you want to compare prices at several different Web sites selling computer hardware? Each Web site has a top-level page with the different categories of products it offers. The first site you visit has a "CD-RW" category. The second one does not, but it does have a "Storage Devices" category that contains CD-RW drives. The third one also has a "Storage Devices" category, but this category has "Fixed," "Removable," and "Optical" subcategories. It turns out that CD-RW drives are in the "Optical" subcategory. Yet a fourth site has a "Storage Devices" category with no CD-RW drives. This particular site put them in the "Multimedia" category. Once you find a list of CD-RW drives at each site, you face another problem. Some list them by product description and others by manufacturer serial number. How do you compare prices if you're not sure they apply to the same item? The more stores compared, the greater the range of representations for the same information.

Given enough time, you would probably surmount these obstacles to online comparison shopping for CD-RW drives, but what about an automated shopping agent? It should be easy to create a software program that visits each retail site and finds the price for a particular product. Unfortunately, with different organizational and product listing schemes, the shopping agent will probably be unable to gather pricing information from every site, making it difficult to find the lowest price. People compensate for the Web's current shortcomings in representing information by spending more time in their searches. But we are denied much of the Internet's potential for automatically gathering and processing information. Delivering on this potential is difficult with ad hoc information representation.

The problem of automated information processing is particularly acute for enterprises. One of the great promises of Internet computing is connecting enterprises to their suppliers, partners, and customers so that they can achieve manufacturing, inventory, and distribution efficiency. However, each company's information systems define common business concepts such as product, order, and invoice differently. Even within one company, two divisions often have their own definitions of these business concepts. Imagine the gulf that exists between different companies in different industries located in different parts of the world. How could an enterprise make its information systems cooperate with those of a foreign supplier? Solving the problem for one supplier is easy. The company could just build a one-to-one translation program. Unfortunately, most enterprises deal with hundreds of suppliers, requiring the construction and maintenance of hundreds of translators. The resulting business environment is hardly a picture of free-flowing global electronic commerce.


Convergence of Information Exchange Problems

The problem of achieving understanding among many parties is endemic to distributed computing. Over the years, people have developed many applications for distributed networks. As the reach of these networks has grown, the applications have encountered the information exchange problem. The heart of the problem is spontaneity. If people were able and willing to negotiate the format and interpretation of each message with each new party from the ground up, there would be no problem. However, the cost of such an arrangement is prohibitive on any but the smallest scale. People need a common infrastructure for describing how to conduct information exchanges that is flexible enough to serve many different domains but constrained enough so that parties can quickly agree on a limited set of parameters. The convergence of problems from multiple sources makes such a balanced solution critical in several areas.

Web Documents

The number of documents accessible via the Web is staggering. Unfortunately, it's hard to figure out what any of the documents means without actually reading them. Search engines are one example. Suppose you wanted to find online versions of works by William Shakespeare. A search on "William Shakespeare" would return documents he has written, documents written about him, and documents describing people's preferred authors. In all probability, the number of documents in the latter two categories would be much greater than the number in the first. What's the problem? There is no agreement on how to indicate what a particular word or phrase means. Therefore, documents where "William Shakespeare" means document author, document subject, and preferred author list element all look the same. Consequently, either people are unable to find what they need, or it takes much longer than they want.

The problem lies with HTML, by far the most widely used means of delivering information over the Web. The way most people use it, HTML serves as an online page layout language. Authors place information in specific parts of the screen and control the format of information. HTML is an adequate page layout language. Web page authors have used it to create some nice looking pages. However, it offers inadequate features for describing the information itself.

Another area where HTML is inadequate is information analysis. Suppose you use a Web-based home search service to develop a list of homes that meet your criteria. Once you have the list, you might want to analyze it a number of different ways. You might want to sort the houses on the list by price, square footage, or school district. The Web site may be able to perform the transformations you want, but all the processing occurs on the Web server, which sends you formatted text. This approach limits you to the filtering options provided by the Web site and places a heavy processing load on the Web server, causing response delays. A more flexible and efficient approach would be to send the data itself to the browser and let a built-in spreadsheet give you the power to slice and dice the data any way you wished. Unfortunately, the Web server has no way to send the data to browsers in a format that it knows they can understand.

Electronic Commerce

Electronic commerce is perhaps the best example of the information exchange problem. It is clear why parties previously unknown to each other want to exchange information, and there is an obvious monetary value to the exchange. There are two fundamental types of electronic commerce: business-to-consumer and business-to-business.

In business-to-consumer commerce, the information exchange problem imposes a time penalty. As we saw in the CD-RW drive example, people can compensate for different information representations at different electronic commerce sites by manually developing knowledge of the site structure. This process requires time and reduces the effectiveness of time-saving automated shopping agents. It also eliminates the possibility of higher-level automated tasks. What if you wanted to find CD-RW drives compatible with your existing SCSI controller card? This information is available at manufacturer sites. However, there is no standard way for you to describe the characteristics of your SCSI controller card or for manufacturers to describe compatibility information for their drives. This information is buried in paragraphs of text so you have to read every CD-RW drive data sheet. Other barriers to information exchange make it difficult to eliminate mundane tasks such as online product registration. You have to type in your name, address, phone number, e-mail address, and other information for every product that you buy.

In business-to-business commerce, the information exchange problem imposes money and choice penalties. If two businesses want to conduct electronic commerce, they have to agree beforehand on formats for all messages and the order in which they can process these messages. Developing software code to implement every such agreement costs money. Requiring prior agreements also restricts choice. If an innovative new company that offers substantially lower prices and higher quality comes along, the enterprise is out of luck until it can reach an agreement on message processing with this new party.

The situation for business-to-business commerce becomes even more complex when more than two businesses must coordinate their actions. Suppose a company that designs clothing receives an order from a chain of retail stores. To fill the order, it needs to contract with at least one raw materials supplier, at least one manufacturing plant, and at least one shipping provider. Adding these three participants to the designer and retailer results in at least five companies participating in the process. They all have to exchange information, and they must use the same overall plan for the completing the process.

Database Access

Enterprises have accumulated large volumes of data in various databases over the years. The current problem is delivering it to people in a way that increases productivity. Certainly, it is easy for any one person to access any one database by installing the appropriate software on a client machine and executing queries against the database. However, in many cases, the information that a particular person needs to do a job resides in several different databases. Take the typical customer service representative as an example. The representative may need to help customers place orders, check on shipments, correct account information, configure products, and get replacement parts. Each of these tasks may require accessing a different database. In many cases, the representative simply cannot help customers with certain tasks. In others, it may be necessary to switch back and forth between different database access programs, writing notes on pieces of paper, to assemble the information necessary to assist the customer.

The problem is not really that the enterprise uses different database products, although it is a compounding factor. The real problem arises when employees have different models of the business than those encoded in database schemas. There is simply no way for a database designer to predict and accommodate the different uses for even a single database. With multiple databases involved, the problem gets exponentially worse. There are many examples where the difference between the user information model and the database data model hurts productivity, including sales representatives trying to check manufacturing schedules for multiple products, marketing analysts trying to assemble historical sales information for multiple products, and executives trying to analyze sales trends across divisions.

The database access problem becomes even more acute as the enterprise expands the number of people who have access to the data. There is a trend toward empowering employees throughout the enterprise by giving them access to the information they need to make good decisions. But if they cannot process this information, it does them no good. Each employee has specific information requirements particular to his job task. Simply throwing data at the employee doesn't help. There is also a trend toward making corporate data available to strategic partners and even customers so that they can integrate their operations more fully with the enterprise. Of course, if the enterprise has trouble providing its own employees with the data they need, delivering it to outsiders is nearly impossible. For years, enterprises have searched for a technology that could help them synthesize data from different databases into different packages, depending on the needs of the particular user. Such a technology would help reduce costs as well as increase the capabilities to deliver innovative products and services.

Knowledge Sharing

Solving ever more complex problems requires vastly improved cooperation among large groups. A major source of difficulty is effectively sharing knowledge of a complex topic. Consider three scenarios: (1) a joint product development effort among five companies, (2) a research partnership among twelve laboratories to create medicines based on gene sequencing, and (3) joint combat exercises that include a dozen ships and a hundred aircraft from three countries. All of these scenarios have a common problem. Individual participants want to share their knowledge with the rest of the group and integrate the group's knowledge into their understanding of the problem.

Sharing the information itself is not too difficult. One approach is for each participant to do a presentation in a teleconference. However, this approach does not make it easy for a participant to integrate the knowledge shared by everyone else. Although presentation slides may be an effective means of organizing high-level concepts, they are not very useful for exchanging detailed information. The product development participants need the three-dimensional models of product concepts and specifications for different prototype parts. The research partnership participants need gene sequences and protein folding simulations. The joint naval exercise participants need detailed sensor deployment plans, rules of engagement contingencies, and weapon delivery assignments.

A further refinement of the presentation approach is to follow up with electronic versions of software files that contain the detailed information. There is the small problem of different participants using different file formats. There is the large problem of relating these files to the original presentation and to each other. Fundamentally, there is no way for one participant to create an information package that includes all the connections among the different elements that make it truly meaningful. Such information packages along with standard file formats could accelerate the pace of innovation and enable the coordination of larger groups in real-time tasks.


Metadata Standard Is a First Step

The convergence of information exchange problems reveals the importance of a general-purpose approach for structuring information exchanges. Such an approach would have to meet two requirements: (1) a party could indicate what each piece of information in a conversation means, and (2) two parties could agree on a limited vocabulary to use for the exchange. The first step in developing such an approach is to meet requirement (1) by adding metadata to documents.

Metadata is data about data. For instance, "author" is metadata about "William Shakespeare." A database schema comprises metadata about the data in the database. It gives all data in a column the same description—such as "Customer Number"—and indicates the datatype of data in that column, such as currency, integer, or string. A distributed object interface definition comprises metadata about the objects that implement that interface. It describes the behaviors an object implements and the arguments necessary to activate each behavior. Metadata is a formal way to describe what a piece of information means.

Metadata addresses requirement (1) for a general-purpose approach to information exchange. However, it does not address requirement (2). Metadata by itself would allow the home search site mentioned previously to describe a set of search results as useful data. But the user's browser would not necessarily understand the organization of that description. If "Home Price" has the metadata "number" attached, what does that mean? What if the metadata were "numeric," "int," or "dollars"? Does the browser need a library of common terms for the same concept? Metadata alone certainly wouldn't solve the problem of business-to-business commerce. What if the metadata "price" attached to "100,000" meant U.S. dollars for one company and pounds sterling for another? Metadata makes it easier to locate interesting data and associate a piece of data with a description, but it still requires the explicit agreement of each party. The parties have to agree on the structure of the metadata.


Shared Context Standards Deliver True Understanding

Shared context is a formal description of the rules metadata must follow. If people had a standard way to express shared context, they could quickly reach agreement on how to conduct an information exchange. In a document-centric architecture, shared context applies to a particular type of document and serves as a contract between the document sender and the document recipient. The document sender agrees that the document conforms to the shared context. The document recipient agrees to interpret the document according to the shared context.

In the case of business-to-business commerce, two companies may agree to a shared context for "Order" documents. Part of this shared context might be that (a) an "Order" can have one or more "Line Items"; (b) each "Line Item" has an "SKU," a "Unit Price," and a "Quantity"; and (c) "Unit Price" is a number with two decimal places and represents U.S. dollars. This approach could help address many of the information exchange problems just discussed.

  • To solve the problem of organizing data on the Web—such as home search results—one could develop a universal shared context for tabular data.

  • One could help solve the database access problem by defining a universal shared context that describes how a database organizes data and the relationship of the data to common business concepts.

  • Business-to-consumer electronic commerce could become much more efficient through shared contexts for catalog structure, orders, and receipts.

  • Business-to-business commerce could become much more efficient through frameworks that enable businesses to describe shared contexts for purchase orders, invoices, and business process descriptions.

  • The knowledge-sharing problem could become easier by enabling collaborators to defining shared contexts for each class of problem they want to solve jointly—for example, special contexts for product development, special contexts for developing medicines from gene sequences, and special contexts for naval task force combat information.

Shared context does not provide complete spontaneity, but it comes close. At least one party must develop the shared context. However, the party can then make the shared context available on the Internet and anyone else can participate in the information exchange it governs. The shared context developer doesn't even have to be a party to these exchanges. One could imagine an organization dedicated to developing such shared contexts as a community service or commercial venture. It might never use the shared context itself; it would simply provide the seed for other parties to begin spontaneous information exchange. The beauty of this approach is that the connectivity of the Internet becomes part of the solution. The developer of a shared context can simply post it to the Internet, where it becomes available to anyone who is interested.


The XML Approach

The metadata and shared context concepts are general, and there are many different ways to implement them. XML is an increasingly popular implementation, and it will almost certainly be the most widely used in the near future. XML is an outgrowth of Standard Generalized Markup Language (SGML), which became a standard of the International Organization for Standardization (ISO) in 1986. SGML had its origins in IBM, which wanted a means of describing document content so that it could publish the same content in a number of different ways. For instance, it would be nice to be able to deliver the same technical information about a new product feature both as a stand-alone bulletin for existing users and as part of a user's guide for new users. Because other organizations were struggling with similar problems, it made sense to create a standard for document markup. The result of the standards process was a rich document markup language that allowed authors to separate the logical content of a document from its presentation. The fundamental approach was to add metadata to denote this logical structure and provide shared context for specifying the metadata rules followed by each type of document.

HTML is also a descendant of SGML, but it was never intended as a general means of structuring information exchanges. It predefines a narrow set of metadata specific to online publishing. SGML and HTML leave the requirement for a standard means of defining metadata on the Internet unfulfilled. However, when people began to develop such a standard, they naturally looked to SGML as a starting point. The World Wide Web Consortium (W3C) formed a working group to study the issue in July 1996. This group's goal was to produce a simplified subset of SGML suitable for use on the Web. Why a simplified version? SGML is extremely complex and poses some problems for automated processing of large volumes of Internet documents. It seemed that a subset of SGML could be simple enough for people to understand in its entirety, yet expressive enough to meet the need for shared context on the Internet. The result was the XML 1.0 specification, released as a Recommendation, the highest level of W3C endorsement, on February 10, 1998.

The XML approach to metadata and shared context is simple. Authors add metadata through tags. The syntax for adding these tags is very similar to that of HTML. To indicate that William Shakespeare is the author of a work, one could write "<author>William Shakespeare</author>." Document designers add shared context through document type definitions (DTDs). A DTD specifies the allowable order, structure, and attributes of tags for a particular type of document. A document references the DTD that governs its structure with an Internet uniform resource locator (URL). Therefore, any party can spontaneously access the DTD, interpret its rules, and process the document.

While DTDs are useful, they are insufficiently precise in certain cases. The more sophisticated the application, the more precise the shared context must be to avoid misunderstandings. In many cases, the level of precision must match that of the programming languages and databases used to process the information. A new W3C standard for XML Schema addresses the need for more precise shared context and will likely supplant DTDs for many applications. It became a Recommendation on May 2, 2001.

While available for only a few years, XML has experienced rapid adoption. Because of the convergence of information exchange problems, the application of XML in one area has led to the application of XML in other areas. As a critical mass of solutions has emerged, much of the software and networking infrastructures have developed direct support for XML, including the following.

  • Development tools. To assist programmers in creating application code that works with XML, development tools have evolved XML-specific features that improve productivity. These features go beyond the simple inclusion of low-level software components for processing XML data. They include utilities for generating XML processing code, manipulating information extracted from XML documents, and distributing this information through different channels.

  • Application servers. Because XML facilitates information exchange in distributed systems, organizations often deploy XML-based solutions on three-tiered architectures, with an application server performing much of the processing. Therefore, application servers have evolved features for processing XML-like frameworks for user presentation, interfacing between business documents and business objects, and assembling documents from database data.

  • Databases. Clearly, organizations need to store mission-critical XML data in an industrial-strength database. While developers can use the application layer to assemble and disassemble XML documents, it is even more convenient for the database to handle them directly. Toward this end, databases are evolving features for mapping documents to tables, directly storing and searching XML data, and performing sophisticated operations on this data without ever taking it out of the database.

  • Networking software. Because many XML-based solutions involve sending and receiving documents across networks, developers need networking software that supports such operations. A family of emerging distributed XML messaging standards has provided a starting point that ensures compatibility among such software. This seed has led to the widespread availability of components that implement these standards and provide high-level interfaces to functions critical to the development of distributed XML applications.

  • Networking hardware. With the potential explosion of important data flowing over the Internet in XML format, there is the opportunity to improve the quality of service by putting XML support directly in the network hardware. The self-describing nature of XML makes it possible for this hardware to take actions based on the message content, so routers can expedite the flow of important messages and even determine their eventual destination based on rules describing which machines can process which types of XML information.

This widespread support for XML has created a network effect similar to the one that catapulted the Web to prominence. It seems as if everyone is using XML for some purpose. In some sense, the W3C greatly exceeded expectations. It developed a technology that not only solved the information exchange problem on the Web but has the potential to solve information exchange problems in every area of distributed computing. Unfortunately, the generality of the solution poses a problem. Because the potential applications are so broad and the support from computing infrastructure is so widespread, there is the opportunity for confusion. Organizations have a hard time sifting through all the possibilities to identify how XML can best help them and then selecting the XML-based technologies that best match their situation. It is therefore important for managers charged with solving information exchange problems to have a solid understanding of how XML works, how to identify opportunities for its use, and how to overcome the inevitable barriers to deployment.


Chapter 2. XML Basics

    Executive Summary

    Jumping In

    XML Conceptual Model

    Introducing Elements

    Introducing Attributes

    Creating an "Order" in XML

    Introducing Document Types

    Introducing Entities

    Technical Summary


Executive Summary

It's important to differentiate between the narrowly defined XML specification and the broadly defined XML paradigm. The XML specification itself has two parts: one for XML documents and one for XML DTDs. The document portion governs the use of tagged markup to indicate the meaning of data. The DTD portion governs how to describe the allowable structure for a class of XML documents. This chapter focuses on the narrow XML specification.

Often, however, the term XML is used to refer to a broader Internet software paradigm in which XML documents serve as the foundation. This paradigm encompasses a wide variety of standards that both enhance the document processing platform and add domain-specific capabilities. Chapter 3 introduces some of the important platform standards, and Chapter 4 focuses on the platform and domain standards related to sending XML business messages over the Internet.

Figure 2-1 shows how XML supports this larger paradigm. As you can see, the paradigm is somewhat modular, where you can use each part for a particular application. Some applications may use only XML documents, while others may include XML DTDs. Still others may take advantage of processing standards or messaging standards.

Figure 2-1. XML Supports a Larger Paradigm

Because the XML specification is a foundation for the entire paradigm, understanding the basics of XML syntax is fundamental to comprehending the power of the paradigm. The process of learning these mechanisms will also help you develop an intuition for where XML can be the most beneficial. Although you probably won't create many XML documents, a command of the syntax improves your ability to collaborate effectively with developers.


Jumping In

XML is intended to be human-readable, so a common example effectively illustrates its primary features. Consider the possibility of an electronic business card. Every day people exchange physical business cards that have a common structure. XML makes it easy to do this electronically. The document in Example 2-1a contains much of the data found on a business card.

Example 2-1a
 <BusinessCard>
 <Name>
  <GivenName>Kevin</GivenName>
  <MiddleName>Stewart</MiddleName>
  <FamilyName>Dick</FamilyName >
 </Name>
 <Title>
  Software Technology Analyst
 </Title>
 <Author/>
 <ContactMethods>
  <Phone>650-555-5000</Phone>
  <Phone>650-555-5001</Phone>
 </ContactMethods>
</BusinessCard>

As you can see, tags in XML documents are similar to those in HTML documents. Familiar open and close tags define the beginnings and ends of elements. These elements appear in a stricthierarchy: "GivenName" is a child of "Name," which is a child of "BusinessCard." The most important difference from HTML is that you can use any types of elements you choose rather than relying on a predefined set.

DTDs describe the allowable structure of XML documents. A document does not need a DTD, but using a DTD is a convenient way for two parties to ensure that they are using the same data format. A DTD can constrain the types of data that may occur in a document, the hierarchy of data items, and the number of times each item of data may appear. Example 2-1b shows a DTD for our simplified business card document.

Example 2-1b
 <!ELEMENT BusinessCard (Name, Title, Author?,
  ContactMethods)>

<!ELEMENT Name (GivenName, MiddleName?, FamilyName)>
<!ELEMENT GivenName (#PCDATA)>
<!ELEMENT MiddleName (#PCDATA)>
<!ELEMENT FamilyName (#PCDATA)>

<!ELEMENT Title (#PCDATA)>
<!ELEMENT Author EMPTY>

<!ELEMENT ContactMethod (Phone*)>
<!ELEMENT Phone (#PCDATA)>

A DTD lists the types of elements within a document, the types of child elements for these elements, and so on. Special characters such as the "?" and the "*" constrain the number of times an element may appear—in this case, 0 or 1 times and 0 or more times, respectively. So our DTD says that a business card may have 0 or 1 "Author" elements, 0 or 1 "MiddleName" elements, and 0 or more "Phone" elements.


XML Conceptual Model

Our business card example is extremely simple. XML's designers intended it to support much more sophisticated documents. When they defined the standard and mapped out the broader paradigm, they used an implicit conceptual model based on their experiences with HTML and SGML, as well as their beliefs about how the use of documents on the Internet would evolve. This model has subtly influenced the development of the XML paradigm, and reconstructing it yields insight into both XML's power and limitations. It has five parts.

  1. Human and machine readability

  2. Defining content

  3. Defining structure

  4. Separation of content from relationships

  5. Separation of structure from presentation

This model is an ideal. The realities of system design and the practical usage of XML have forced compromises and changes. Subsequent chapters address these issues, but understanding the vision behind XML sheds light on why the technology evolved the way it did.

Human and Machine Readability

The XML design strives to achieve both easy human reading and easy machine processing. This goal results in two different views of any particular document: the human view and the machine view. These views are subtly different, and accommodating both views accounts for much of the XML syntax.

Like HTML, XML relies on tag-based markup. From the human perspective, tag-based markup is a readable way to integrate metadata with content. From the machine perspective, tag-based markup is simply one of many possible parsing tactics. Tag-based markup is a compromise that makes human reading easier without imposing too much of a burden on machine processing. Con versely, XML requires a strict hierarchy of tagged data. A strict data hierarchy is a compromise that makes machine processing easier without imposing too much of a burden on human reading. From the machine perspective, the use of a strict hierarchy makes it easy to create programming data structures from document content. From the human perspective, hierarchy is simply one of many possible organizational strategies. These two trade-offs have their roots in the human-machine duality. The human view is of a marked-up document; the machine view is of a tree of data.

In addition to being convenient from the machine perspective, hierarchy is a powerful organizational strategy. It can represent both business concepts and programming data structures. Business process models, decision trees, and geographical models are examples of business concepts that fit the hierarchical model. Trees, linked lists, and tables are examples of programming data structures that can fit into a hierarchy.

Defining Content

To see how machine processing constrains the XML syntax, consider what a machine needs to know to understand a simple order for produce. A human reader may be able to decipher the implicit relationships among the words of an order for "100 10-pound bags carrots at $1 per pound." However, a machine would have great difficulty with the same task. The human and machine readability goal necessitates a standard means of explicitly declaring the relationships among the words. Metadata enables a document author to specify these relationships in the document. Without metadata, the content of a document looks the same; it's just a jumble of individual words. Because the content is homogeneous, it is meaningless.

In XML, authors define content as a set of elements. An element is a self-contained unit of content with a description of what the content means. For example, you could separate the preceding document into chunks such as "Quantity = 100," "Size = 10," "Size Unit = pound," "Product = carrots," "Price Currency = $," "Price = 1," and "Pricing Unit = pound." Breaking the document into these elements would allow a human reader to interpret the order unambiguously and a machine to calculate the total price easily.

Defining Structure

With very simple documents, breaking content into elements may be sufficient by itself to make the document easy to interpret. However, even a modest document can be confusing without some additional structure. Consider an expanded produce order such as, "100 10-pound bags carrots $1 per pound and 25 limes 50¢ per lime." With just a list of elements, how would you know which quantity and price applied to which product? How would you account for prices in different units? The solution is to structure the content so the relationships among elements are clear. As mentioned previously, XML uses a hierarchical structuring model. By organizing the expanded document into a hierarchy, you can now clearly apply the correct quantity and price to each product as shown in Example 2-2a.

Example 2-2a
 Order
 Line Item
  Quantity = 100
  Product = carrots
   Size = 10
   Size Unit = pound
  Price = 1
   Pricing Currency = dollars
   Pricing Unit = pound
 Line Item
  Quantity = 25
  Product = limes
   Size = 1
   Size Unit = each
  Price = 50
   Pricing Currency = cents
   Pricing Unit = each

Unfortunately, a generic hierarchy is not specific enough to provide the shared context described in Chapter 1. Clearly, a produce order and a patient record follow different rules. XML DTDs provide a means for specifying the rules that the hierarchy in a particular document must follow. These rules include the allowable element types at each level of the hierarchy. An abstract description of the rules governing the hierarchy for our example produces an order document that might look like Example 2-2b.

Example 2-2b
 Top-level elements = exactly 1 "Order"
 Sub-element of "Order" is 1 or more "Line Item"
  Sub-element of "Line Item" is exactly 1 "Product"
   Sub-element of "Product" is exactly 1 "Size"
   Sub-element of "Product" is exactly 1 "Size Unit"
  Sub-element of "Line Item" is exactly 1 "Price"
   Sub-element of "Price" is exactly 1 "Price Currency"
   Sub-element of "Price" is exactly 1 "Price Unit"

Separation of Content from Relationships

Although hierarchical relationships among elements within a document are important for many applications, associative relationships among elements in different documents are also important. HTML links are an example of these associative relationships. As the success of the Web illustrates, there is often informational value in these associative relationships as well as document content. For example, a set of five Product, five Order, and five Customer documents is much more useful if you know which Orders relate to which Products and which Customers. Unfortunately, HTML embeds these important relationships within the document structure, making them difficult to maintain. The results are broken links and frustrating "HTTP 404—Object Not Found" errors.

In the XML paradigm, document content and document relationships can be separated. In fact, the XML specification ignores relationships. A different standard specifies the appropriate syntax. This syntax enables relationships to exist outside the documents they connect, making it possible to maintain the integrity of relationships and even to introduce new relationships without affecting the referenced documents. This independence theoretically allows authors to create valuable information by relating documents in sophisticated ways without ever changing the content of the related documents.

Separation of Structure from Presentation

Another drawback of HTML is the primacy of presentation over structure. For example, to distinguish sections of a document, many authors have forsaken heading specifiers such as <H1> for direct control over font attributes such as <FONT SIZE=16>. This approach results in two problems. First, all content looks exactly the same. Second, readers are limited to the presentation imposed by the author.

In XML, document structure and document presentation can be separated. As with relationships, the XML specification ignores presentation, relying on another standard. For a given XML document, authors may suggest a presentation using this other standard, but readers are free to use whatever presentations they choose. Different readers may use different presentations, and the same reader may use different presentations under different circumstances. Another advantage of this approach is that theoretically it enables a clear separation of responsibility between the author of structured information and the designer of the presentation layout.


Introducing Elements

The fundamental unit of XML content is the element, which is an author-specified chunk of information. An element consists of an element name and element content. XML is case sensitive, so you must pay attention to case when assigning element names and creating element content. Start and end tags denote the boundaries of the element and contain the element name. The element content may consist of character data or other elements. An element may also be empty of content. Consider Example 2-3, an annotated version of our business card document to see examples of these content types.

Example 2-3
 <BusinessCard>                       document element
 <Name>                              element content
  <GivenName>Kevin</GivenName>       data content
  <MiddleName>Stewart</MiddleName>   data content
  <FamilyName>Dick</FamilyName>      data content
 </Name>
 <Title>
  Software Technology Analyst        data content
 </Title>
 <Author/>                           empty content
 <ContactMethods>                    element content
  <Phone>650-555-5000</Phone>        data content
  <Phone>650-555-5001</Phone>        data content
 </ContactMethods>
</BusinessCard>

In Example 2-3, "BusinessCard" is the top-level element. In XML, there can be only one element at the top level. This element is called the document element or sometimes root element. Think of this element as the trunk of the tree from which all other elements branch. Figure 2-2 shows the corresponding tree for Example 2-3 with each node representing an element and identified with the element name. Conceptually the element content resides within the node.

Figure 2-2. Business Card Element Tree

The annotations in Example 2-3 indicate the content model for each element. There are four allowable content models for elements.

  1. Data content. These elements contain only data. Note that in the 1.0 version of the XML specification, there is no way to enforce typing restrictions on data content such as integer, floating point, and date. Chapter 3 discusses a related standard, XML Schema, which addresses this potentially serious shortcoming.

  2. Element content. These elements contain only other elements.

  3. Empty. These elements contain neither elements nor data.

  4. Mixed content. These elements contain both data and other elements. None of the elements in Example 2-3 has this type of element content because many XML experts feel using elements with mixed content is poor design practice for data-oriented formats.

Notice that, except for empty elements, all elements in Example 2-3 have a start tag and an end tag. The start tag is bounded by angle brackets—for example, <ElementName>. The end tag is bounded by angle brackets and has a leading slash, as in </ElementName>. All content, whether data or element, must occur between the start and end tags. An empty element may use this syntax by simply providing no content between the start and end tags—for example, <ElementName></ElementName>. An empty element may also use an abbreviated syntax, with a single tag bounded by angle brackets with a trailing slash—for example, <ElementName/>.

A document that obeys all the XML syntax rules is well formed. There are several technical criteria for well-formedness, but the primary ones are the following.

  • There is one document element.

  • All nonempty elements have start tags and end tags that match exactly.

  • All empty elements have the correct empty tag syntax.

  • Elements are strictly nested; there are no overlapping elements.

An XML processor can process a well-formed XML document unambiguously, building a tree data structure in which each node is an element that contains either data content or references to its subelements or both or neither. You could use such documents to represent many different kinds of content. Example 2-4 shows a document that represents the schema for a simple contact database. Figure 2-3 shows the corresponding tree.

Example 2-4
 <Database>
 <Table>
  <Column>Name</Column>
  <Column>Phone Number</Column>
 </Table>
 <Table>
  <Column>Date</Column>
  <Column>Person</Column>
 </Table>
</Database>
Figure 2-3. Contact Database Element Tree

Although the document in Example 2-4 captures the basic structure of the contact database, there is not enough information for a software application to process the document, establish a connection to the database in question, and perform queries. The element names "Database," "Table," and "Column" are insufficiently descriptive. Clearly, you need a richer syntax for describing the metadata associated with an element.


Introducing Attributes

Attributes provide the solution for adding richer metadata to elements. An attribute consists of an attribute name and an attribute value. The attribute value is bounded by quotation marks. This attribute specification appears in the start tag of the element, and you can specify as many attributes for an element as you consider necessary. As with all XML constructs, attribute names and attribute values are case sensitive. As a stylistic convention, some authors use upper camel case text for element names and lower camel case text for attribute names so that they can easily distinguish between names of elements and names of attributes. Example 2-5 shows how adding attributes to the contact database could make it possible to use the database.

Example 2-5
 <Database dbType="Oracle" address="192.168.1.1">
 <Table name="Person">
  <Column dataType="String">Name</Column>
  <Column dataType ="String">Phone Number</Column>
 </Table>
 <Table name="Conversation">
  <Column dataType ="Date">Date</Column>
  <Column dataType ="String">Person</Column>
 </Table>
</Database>

Figure 2-4 shows the element tree of this document. Note that it is exactly the same as the tree in Figure 2-3. From Example 2-4 to Example 2-5, the element hierarchy did not change. Even the element content did not change. What did attributes add? They added further metadata. They added information that will allow consumers of the document—in this case database access programs—to use the content effectively.

Figure 2-4. Contact Database Element Tree with Attributes

In most cases, attributes aren't strictly necessary. You could put the same information into a subelement, but often it seems more convenient and cleaner to use attributes for small pieces of contextual information rather than subelements. Each attribute in Example 2-5 provides the context necessary to interpret the element content. The "dbType" attribute tells it what database networking protocol to use. The "location" attribute tells it where to find the database. The "name" attribute tells it how to select the appropriate table. The "dataType" attributes tell it how to form valid queries for each column. Obviously, a real version of this application might require even more attributes, but with XML, you could easily provide them.


Creating an "Order" in XML

The earlier examples in this chapter have used somewhat simplified XML documents. To get a proper feel for XML, it helps to see a realistic and complete document. Example 2-6 is a complete order for PC hardware in XML.

You can see immediately that it has some features not seen in previous examples. First, it starts with a prolog. The prolog is header information that appears before the document element. The first line of the prolog must be the XML declaration that identifies the document as an XML document and specifies the XML version. This particular prolog also contains a processing instruction. A processing instruction gives to software special information that may process the document. It starts with "<?" and ends with "?>". It has a target—in this case, any software that wants to apply a stylesheet to the document—and one or more attribute-value pairs—in this case, a reference to the stylesheet file and a MIME type for the file. A document may contain as many processing instructions as necessary, including different instructions for different applications that may process it.

At the end of the prolog, you may optionally declare the type of the document. A document type declaration indicates the DTD to which the document complies and where to find it. It starts with "<!DOCTYPE" and ends with ">". It has the document element, which must match an element defined in the DTD. It also has a place to look for the DTD—in this case, the local filesystem—and a handle—in this case a filename. The next section discusses how DTDs specify the rules that a document's elements must follow.

The second new feature of Example 2-6 is that the body of the document starts with a comment, always a good idea with any software code. Comments can occur anywhere after the prolog. They start with "<!—" and end with "—>".

Example 2-6
 <?xml version="1.0"?>
<?xml-stylesheet href="order.xsl" type="text/xsl" ?>
<!DOCTYPE Order SYSTEM "Order.dtd">
<!-- Example Order Form from _XML: A Manager's Guide_ -->
<Order source="web" customerType="consumer"    currency="USD">
 <Addresses>
  <Address addressType="billship">
   <FirstName>Kevin</FirstName>
   <LastName>Dick</LastName>
   <Street lineOrder="1">123 Anywhere Lane</Street>
   <Street lineOrder="2">Apt 1B</Street>
   <City>Palo Alto</City>
   <State>CA</State>
   <Postal>94303</Postal>
   <Country>USA</Country>
  </Address>
 </Addresses>

 <LineItems>
  <LineItem ID="line1">
   <Product category="Mboard">
    815i Motherboard
   </Product>
   <Quantity>1</Quantity>
   <UnitPrice>200</UnitPrice>
  </LineItem>

  <LineItem ID="line2">
   <Product category="RAM">
    128 MB PC-133 DIMM
   </Product>
   <Quantity>2</Quantity>
   <UnitPrice>175</UnitPrice>
  </LineItem>
  <LineItem ID="line3">
   <Product category="CDROM">
    40x CD-ROM
   </Product>
   <Quantity>1</Quantity>
   <UnitPrice>50</UnitPrice>
  </LineItem>
 </LineItems>

 <Payment>
  <Card cardType="VISA">
   <CardHolder>Kevin S Dick</CardHolder>
   <Number>41280000000000</Number>
   <Expiration>01/2010</Expiration>
  </Card>
 </Payment>
</Order>

After the comment comes the document element. In this case, the document element is "Order." It has attributes that indicate where the order came from, the type of customer, and the currency for the quoted prices. The document element has three subelements corresponding to the three major sections of an order: addresses, line items, and payment information.

Most of the rest of the document is self-explanatory. However, there are a few interesting features. First, the "Address" element has an "addressType" attribute, indicating that the contained address should be used as both the billing and shipping address. As you will see in the next section, the DTD will specify that this attribute must have one of the following values: "bill," "ship," and "billship." Using this approach of an attribute with an enumerated set of possible values is a common technique in designing XML documents. Another common technique is to use attributes to indicate the logical order of multiple elements in a list, such as the "lineOrder" attribute of the "Street" element.

Another feature of note is the "ID" attribute of the "LineItem" element. As you will see in the next section, the DTD will specify that this attribute be of a special type—ID. ID attributes must be unique within the document, allowing software programs to use the attribute as an index when managing a group of the same elements.


Introducing Document Types

From a business perspective, the single order document in Example 2-5 is obviously less useful in isolation than as a collection of many order documents. It would hardly make sense to write a piece of software or design a screen layout for just one document. The true power of XML to improve business processes comes when multiple documents all use the same data format. Then a single piece of software can process them all, and a single screen layout can display them all. Most important, parties can quickly agree on the structure of their information exchange by referring to the appropriate format.

DTDs provide these common, public data formats. A DTD is a collection of rules that specifies the allowable structure of a document class. It serves as a format referee at two important points in the software life cycle. During the design phase, a software developer can look at a DTD and know that as long as the application he builds will output documents that conform to that DTD, other applications can process those documents. During the execution phase, the XML processor can verify that a document conforms to the DTD so the application that processes the document knows that it will receive properly structured content. In essence, the DTD is a contract between the supplier of the document and the consumer. A particular XML document is valid if it obeys all the rules of its parent DTD as well as the criteria for well-formed documents.

Earlier in the chapter, you saw that elements and attributes are the two primary constructs in an XML document. It shouldn't surprise you to learn that the syntax for DTDs deals primarily with specifying allowable element structure and rules that attributes must follow.

Defining Element Structure

In Example 2-1b, you saw a DTD for a simple business card that specified the shape of the document tree—the nodes and allowable branches from each node. In Example 2-7, this DTD is annotated to illustrate how to use DTD syntax to specify the different types of content.

Example 2-7
 <!ELEMENT BusinessCard (Name, Title,
Author, ContactMethods)>              document element

<!ELEMENT Name (GivenName,
MiddleName?, FamilyName)>             element content
<!ELEMENT GivenName (#PCDATA)>        data content
<!ELEMENT MiddleName (#PCDATA)>       data content
<!ELEMENT MiddleName (#PCDATA)>       data content
<!ELEMENT Title (#PCDATA)>            data content

<!ELEMENT Author EMPTY>               empty content

<!ELEMENT ContactMethods (Phone*)>    element content
<!ELEMENT Phone (#PCDATA)>            data content

Each element declaration begins with "<!ELEMENT" and ends with ">". It contains the element name and a content model surrounded by parentheses. As we saw earlier, there are four types of allowable content. The corresponding syntax for each content model is as follows.

  • Data content. These elements contain only data. To indicate this structure, the element declaration specifies a content model of #PCDATA.

  • Element content. These elements contain only other elements. To indicate this structure, the element declaration specifies a content model that lists the element names, separated by commas. Note that this list is order sensitive.

  • Empty. These elements contain neither elements nor data. To indicate this structure, the element declaration uses the keyword EMPTY as the content model.

  • Mixed content. These elements contain both data and other elements. To indicate this structure, the element declaration specifies a content model that includes PCDATA to indicate that data content is allowed and element names to indicate that elements of these types are allowed.

In defining the element model, a document designer starts with the document element. After the document element, the designer moves on to the subelements of the document elements. Then come the subelements of those elements, continuing until there are only leaf elements—those that have only data content or are empty.

In addition to these basic content models, DTD designers may use special characters to encode rules about the number of sub elements that an element may contain. These cardinality rules use the following syntax. Note that the default cardinality is exactly 1.

  • 0 or 1. The ? character indicates an optional subelement. So <!ELEMENT Person (FirstName, MiddleName?, LastName)> indicates that "Person" must have one "FirstName," then may or may not have one "MiddleName," then must have one "LastName."

  • 0 or more. The * character indicates a subelement that may optionally appear one or more times. So <!ELEMENT ContactMethods (Phone*)> indicates that "ContactMethods" may have any number of "Phone."

  • 1 or more. The + character indicates a subelement that must appear at least once. So <!ELEMENT ContactMethods (Phone+)> would indicate that there must be at least one "Phone."

  • Enumerated alternatives. A list of subelements separated by vertical bars indicates that the element must contain one of the subelements in the list. So, <!ELEMENT Payment (Cash | Check | Card)> would indicate that "Payment" must contain either "Cash," "Check," or "Card."

Document designers can combine different rules in the same element declaration, using parentheses to group subelements together. For example, <!ELEMENT WorkId (Passport | (Drivers License, SocialSecurityCard))> indicates that acceptable "WorkId" consists of either a "Passport" or both a "DriversLicense" and a "SocialSecurityCard." The declaration <!ELEMENT EMailList (Name?, (MailServer, To+, CC*)+, (Version | Updated))> indicates that an "EMailList" has (1) an optional "Name"; (2) one or more blocks that include one "MailServer"; one or more "To" and zero or more "CC"; and (3) either a "Version" or an "Updated." As you can see, the syntax for defining element structure is rich enough to make plain English descriptions difficult. These basic building blocks give developers the power to specify sophisticated content models.

Defining Attribute Rules

As you've already learned, elements are only one of the constructs available to XML document authors. Attributes enhance the meaning of element content by providing additional metadata. Not surprisingly, XML DTDs include syntax for defining the rules that attributes must follow. Example 2-8 is a DTD for the database schema definition document from Example 2-5. As you can see, the element declarations specify a database document element with one or more table elements. Each table element must have one or more column elements, each of which has character data.

Example 2-8
 <!ELEMENT Database (Table+)>
<!ATTLIST database
dbType       CDATA            #REQUIRED
address      CDATA            #IMPLIED >

<!ELEMENT Table (Column+)>
<!ATTLIST Table
name         CDATA            #REQUIRED>

<!ELEMENT Column (#PCDATA)>
<!ATTLIST Column
dataType  (String | Int | Float | Date | BLOB) "String">

Remember that in Example 2-5 attributes were added to an existing set of elements. Therefore, in addition to element declarations, Example 2-8 includes attribute declarations that define the rules that these attributes must follow. These declarations begin with <!ATTLIST and end with ">". Internally, they have four parts.

  1. Element type. After the ATTLIST keyword, the declaration specifies the element to which the list applies. In Example 2-8, the "Database," "Table," and "Column" elements have attribute list declarations.

  2. Attribute name. The rule for each attribute in the list appears on a new line. The first part of the rule is the attribute name. For example, the "Database" element has "dbType" and "address" attributes.

  3. Attribute type. After the attribute name, the type specification of the attribute appears. For character values such as "dbType" and "address," the type specification is CDATA. For enumerated values, such as "dataType," the type specification is a list of the possible values, separated by a vertical bar and enclosed in parentheses. Not appearing in Example 2-8 are the ID and IDREF type specifications. ID indicates a character string must be unique. IDREF indicates a character string that corresponds to the value of an ID attribute within the document. Using ID and IDREF attributes, document authors can create links between elements in the same document.

  4. Default value. After the attribute type, the document designer must specify the default value for the attribute. There are a number of options. #REQUIRED indicates that every document must explicitly assign a value to the attribute. "dbType" is an example of a #REQUIRED attribute. #IMPLIED indicates that a document does not have to assign a value to the attribute; the XML processor will tell the application that no value was assigned. "address" is an example of an #IMPLIED attribute. A value in quotation marks indicates an attribute with a specific default value; if the document does not explicitly assign a value to the attribute, the XML processor will automatically assign the default value. "dataType" has the default value "String." Not appearing in Example 2-8 are attributes with #FIXED default actions. After the #FIXED keyword, there is an attribute value in quotation marks. If a document explicitly assigns a value to such an attribute, it must be the same as the value specified after #FIXED.

DTDs give document designers a high degree of control over both the structure of document elements and the rules attributes must follow. This control opens the doors for document designers to create DTDs that apply to vastly different fields, from online catalogs, to database integration tools, to supply chain management systems.

Example DTD

Examples 2-7 and 2-8 demonstrate the syntax for specifying element structure and attribute rules for simple DTDs. Even DTDs for modest applications, such as the order in Example 2-6, can quickly become complex. Illustrating the potential complexity of a DTD for a complete XML application, Example 2-9 consists of a DTD for this order form.

Example 2-9
 <!-- Example Order Form DTD from _XML: A Manager's
  Guide_ -->
<!-- Document Structure -->
<!ELEMENT Order (Addresses, LineItems, Payment)>
<!ATTLIST Order
source          (web | phone | retail )      #REQUIRED
customerType    (consumer | business)        "consumer"
currency        CDATA                        "USD">

<!-- Collection of Addresses -->
<!ELEMENT Addresses (Address+)>

<!-- Address Structure -->
<!ELEMENT Address (FirstName, MiddleName?, LastName,
  Street+, City, State, Postal, Country)>
<!ATTLIST Address
addType        (bill | ship | billship)     "billship">
<!ELEMENT FirstName (#PCDATA)>
<!ELEMENT MiddleName (#PCDATA)>
<!ELEMENT LastName (#PCDATA)>
<!ELEMENT Street (#PCDATA)>
<!ATTLIST Street
lineOrder            CDATA                   #IMPLIED>
<!ELEMENT City (#PCDATA)>>
<!ELEMENT State (#PCDATA)>
<!ELEMENT Postal (#PCDATA)>
<!ELEMENT Country (#PCDATA)>

<!-- Collection of LineItems -->
<!ELEMENT LineItems (LineItem+)>

<!-- LineItem Structure -->
<!ELEMENT LineItem (Product, Quantity, UnitPrice)>
<!ATTLIST LineItem
ID                   ID                  #REQUIRED>
<!ELEMENT Product(#PCDATA)>
<!ATTLIST Product
category             (CDROM | MBoard | RAM)   #REQUIRED>
<!ELEMENT Quantity(#PCDATA)>
<!ELEMENT UnitPrice(#PCDATA)>

<!-- Payment Structure -->
<!ELEMENT Payment (Card | PO)>

<!-- Card Structure -->
<!ELEMENT Card (CardHolder, Number, Expiration)>
<!ATTLIST Card
cardType  (VISA | MasterCard | Amex)        #REQUIRED>
<!ELEMENT Cardholder (#PCDATA)>
<!ELEMENT Number (#PCDATA)>
<!ELEMENT Expiration (#PCDATA)>

<!-- PO Structure -->
<!ELEMENT PO (Number, Authorization*)>
<!ELEMENT Number (#PCDATA)>
<!ELEMENT Authorization (#PCDATA)>

The document element for this DTD is "Order," and it has three required subelements: "Addresses," "LineItems," and "Payment." The "Addresses" and "LineItems" elements are simply collections of one or more "Address" and "LineItem" elements, respectively. Therefore, "Addresses" and "LineItems" are not strictly necessary. The "Order" element could contain the collections of "Address" and "LineItem" elements directly. In this case, the first element declaration would be <!ELEMENT Order (Address+, LineItems+, Payment)>. However, at some point in the future, the application may need additional information that applies to a group of "Address" or "LineItem" elements. Also, many developers find it easier to work with documents structured in this way because it's analogous to collection types used in object-oriented programming languages.

To see how wrapping a collection of "LineItem" elements in a "LineItems" element may improve extensibility, suppose that the company develops a pricing strategy that allows sales representatives to apply different discount policies to different blocks of line items. In this case, you would need to allow multiple "LineItems" elements and add a "DiscountPolicy" subelement. The syntax for these changes would be <!ELEMENT Order (Addresses, LineItems+, Payment)> and <!ELEMENT LineItems (DiscountPolicy?, Line Items+)>, respectively. Notice that because the "DiscountPolicy" element is optional all documents that conform to the original DTD will also conform to the new DTD. Planning for such future enhancements is an important part of DTD design.

Beyond this subtle design point, the rest of the element structure is relatively straightforward. The other interesting features of this structure are in the "Payment" element. This element presents a binary decision between the credit card and purchase order payment methods with the "Card" and "PO" subelements. The "PO" subelement allows for an optional list of "Authorization" elements. This list enables the application to keep track of any employee as well as the customer who authorized the PO. If there is a dispute about payment, maintaining this information could be very important.

The attribute specification reveals the flexibility available to document designers. "currency" and "lineOrder" may both have character data values, but "currency" has a default value, whereas "lineOrder" is optional. "source," "addressType," "category," and "cardType" are all enumerated types, but the document must explicitly assign an option to "source," "cat," and "cardType," whereas "addType" has a default value. The required "ID" attribute of ID type for the "LineItem" element enables an application using an order document to identify uniquely any particular "LineItem" element by this attribute.

Although the DTD in Example 2-9 is somewhat long and modestly complex, it is still relatively simple in comparison with DTDs used in actual applications. You could imagine the necessary complexity of DTDs for documents for financial wire transfers, telecommunications service provisioning, and medical records. Designing DTDs that cover all the possible document cases as well as allowing for future enhancements is one of the crucial steps in deploying effective XML applications.

As organizations have attempted to design DTDs for some of these sophisticated domains, they have found DTDs lacking important features. The inability to specify the datatype of element and attribute values with enough precision to ensure compatibility with programming languages and databases is perhaps the biggest drawback. But the lack of other features such as finer cardinality constraints on elements, range checking on values, and specifying reusable structures also create the need for a more sophisticated alternative. Chapter 3 discusses how XML Schema provides this alternative.


Introducing Entities

Elements and attributes are primarily logical mechanisms. They enable authors to specify the logical meaning of document content. On the other hand, entities are primarily physical mechanisms. They enable authors to manipulate the physical structure of documents. As physical concepts, entities are interesting mostly to programmers. However, they do provide useful features for organizing XML documents, and you should have a high-level understanding of how they work.

An entity associates an entity name with a fragment of content. Authors define entities with an "!ENTITY" declaration at the beginning of a document or DTD. There are four types of entities in XML, three that apply to documents and one that applies to DTDs.

  • Internal parsed entities. These entities allow authors to define an alias for a fragment of text within a document. So the declaration <!ENTITY author "Kevin Dick"> would allow me to type "&author" within a document and have it replaced by "Kevin Dick" when the document was processed. Using internal entities can reduce the size of documents by replacing large pieces of recurring text with a shorter name. It also makes documents more extensible. If I decided that I wanted my name to appear as "Kevin S. Dick," I would have to change only the ENTITY declaration.

  • External parsed entities. These entities allow authors to include an entire file as part of a document. Using external entities, we could decompose the order document from Example 2-6 into three components:

    <!ENTITY addressBlock SYSTEM "./address.xml">
<!ENTITY lineItemsBlock SYSTEM "./lineitems.xml">
<!ENTITY paymentBlock SYSTEM "./payment.xml">
&addressSection;
&lineItemsSection;
&paymentSection;

    Such decomposition would be useful for managing long documents. It would also be useful if different authors were responsible for creating different parts of the document. They could work on these files independently without worrying about overlapping changes.

  • Unparsed entities. These entities allow authors to insert arbitrary data in a document. This data may not be XML. It may not even be text. The XML processor will obviously not attempt to parse such an entity.

  • Parameter entities. These entities work only within DTDs. They enable the DTD designer to create reusable and extensible design elements for document structure. Remember the DTD for our order document in Example 2-9. One of the attributes of the "Card" element was "cardType." This attribute provided a list of acceptable credit cards.

    cardType (VISA | MasterCard | Amex) #REQUIRED

    What if the company using this DTD decided to take Discover cards as well? This company might have many other document types with the same information. A parameter entity would allow DTD designers a means to update the acceptable cards at the beginning of each document.

    <!ENTITY % cardList "VISA | MasterCard | Amex |
  Discover">
CARDTYPE (%cardList;) #REQUIRED>

Although XML documents and DTDs deliver significant benefits in exchanging information, it's only natural to want to enhance these capabilities even further. Much as with peeling an onion, addressing the most visible information exchange problem on the Internet reveals further possibilities for improvement. Chapter 3 introduces some of the new related standards that deliver such improvements.


Technical Summary

  • An element consists of open and close tags containing the element name surrounding the element content— <ElementName>element content</ElementName>.

  • Attributes appear in the open tag and consist of an attribute name and an attribute value—<ElementName attributeName = "attributeValue">.

  • For elements with element content, a DTD specifies the allowable child elements—<!ELEMENT ElementName (Child1*, Child2?, Child3+)>.

  • *, ?, and + indicate the number of times an element may occur—0 or more, 0 or 1, and 1 or more, respectively. The default is exactly 1.

  • For elements with data content, a DTD specifies the type of data it may contain—<!ELEMENT ElementName (#PCDATA)>.

  • For elements with no content, a DTD specifies an empty content model—<!ELEMENT ElementName EMPTY>.

  • For elements with attributes, a DTD specifies the list of allowable attributes—

    <!ATTLIST ElementName
attributeName1 CDATA "default">
attributeName2 CDATA #REQUIRED
attributeName3 (option1 | option2) "option1">

  • Default attribute values appear within quotation marks.

  • DTDs can require documents to assign an attribute value explicitly with the "#REQUIRED" keyword.

  • DTDs can specify an enumerated list of allowable attribute values bounded by parentheses with options separated by vertical bars.


Chapter 3. Related Standards

    Executive Summary

    Overview of Namespaces

    Overview of XPath, XPointer, and XQuery

    Overview of XSLT

    Overview of XSL

    Overview of XLink

    Overview of XML Schema

    Technical Standards

    Infrastructure Standards

    Technical Summary


Executive Summary

XML documents and DTDs provide the foundation for an evolving Internet document paradigm. As people developed XML applications, they naturally identified additional features they wanted to have and troublesome drawbacks they wanted to avoid. Such discoveries inspired further standards efforts built on the foundation of XML documents and DTDs. As the use of XML has evolved, this process has led to a complicated landscape where old specification efforts languish for extended periods of time and new efforts rapidly achieve acceptance. Therefore, identifying the standards relevant to your needs can be something of a challenge.

Categorizing the various specifications offers a potential aid to navigating the landscape. Figure 3-1 presents one potential categorization of XML-related standards. It expands on Figure 2-1 by revealing the detailed structure of both the platform standards and the domain standards. This categorization can help you identify the depth of understanding you need for any given standard.

Figure 3-1. Categories of Related Standards

The line between platform and domain standards is somewhat fuzzy. As a rule of thumb, platform standards potentially add capabilities to XML documents of all types, while domain standards define an XML format to encode information for a narrower purpose. In general, a standard that could ultimately affect a majority of XML developers is probably a platform standard. Within this category, there are three subcategories.

  1. Technical standards. Standards in this category provide a precise technical foundation for XML processing components to work correctly. Without these standards, ambiguity could cause unexpected results when working with documents. The most important is XML Infoset, which provides an underlying data model for XML documents.

  2. Feature standards. Standards in this category add new features to the overall XML document paradigm. Without these standards, people could not do all of the things that they want to do with XML documents, and the paradigm would be much less valuable. Table 3-1 lists the most important feature standards.

  3. Infrastructure standards. Standards in this category provide crucial infrastructure necessary for many XML applications. They are low-level applications of the feature standards that provide services used by higher-level applications. Standards such as XHTML and XForms specify how to integrate XML with Web technologies. Others, such as SOAP and XML Encryption, specify a uniform way to perform common operations. Without these standards, XML applications would be incompatible with existing Web applications, and all developers would have to reinvent the same infrastructure.

It is important to have a thorough understanding of feature standards because they will help you to determine whether XML supplies the capabilities necessary to solve a given business problem. On the other hand, most managers only need to understand technical and infrastructure standards at a very high level. A general knowledge of technical standards could help you understand the source of a defect in a third-party component that delays a project. A general knowledge of infrastructure standards could help you ensure that your developers maximize the use of existing solutions. Detailed comprehension of technical and infrastructure standards is usually necessary only for developers working with software components that implement them.

Figure 3-1 also shows the structure of the domain standards. The XML paradigm includes both horizontal and vertical domain standards. Horizontal domain standards provide a solution for manipulating information specific to a type of application but used in many industries. Examples include portable formats for vector graphics (SVG), multimedia (SMIL), and mathematical expressions (MathML). They generally facilitate common software components across different industries. Their applicability depends on whether your application manipulates the particular type of data they describe.

The vertical domain standards provide formats for describing information specific to a particular industry but used by many types of applications. Examples include interoperability formats for finance, health care, and telecommunications. They generally facilitate compatibility among systems used within the same industry. Their applicability depends on whether your organization works with information from a particular industry.

Table 3-1. XML Feature Standards
Standard | Abbreviation | Purpose
XML Namespaces | Namespaces | Prevent overlap of names used by different software
XML Path Language | XPath | Address data within a document
XML Pointer Language | XPointer | Specify locations within a document
XML Query Language | XQuery | Search for data within a set of documents
XSL Transformations | XSLT | Transform data from one format to another
Extensible Stylesheet Language | XSL | Present data in a paginated format
XML Linking Language | XLink | Link data from different documents
XML Schema | N/A | Specify more extensive format definition rules than DTDs

Table 3-1 lists the most important feature standards. The rest of this chapter reviews the important details of these standards. At the end of the chapter, there is a summary of key technical and infrastructure standards.


Overview of Namespaces

As you saw in the previous chapter, the primary units of content in XML are elements and attributes. Authors distinguish among different types of elements and attributes by assigning them unique names. When applications process a document, they associate element content with the corresponding element name and attribute content with the corresponding attribute name.

As people started to use XML, they realized that organizations and application often use the same name to describe different concepts. When these organizations want to exchange information using XML, conflicts may occur. A single document often combines information intended for multiple organizations and applications. For example, most mail order vendors do not charge customers until they ship the ordered goods. Enforcing this policy requires processing the same information with an Accounting application for billing purposes and a Fulfillment application for shipping purposes.

But what if the Accounting and Fulfillment applications use the term "status" to signify different meanings? Both applications want their respective status data associated with the "Status" element name. Such a naming collision has the potential to make life very uncomfortable for developers responsible for specifying the formats of affected documents.

XML Namespaces, a W3C Recommendation, enables developers to avoid naming collisions by assigning element and attribute names to namespaces. Accounting and Fulfillment departments could each have separate namespaces. Developers can thereby qualify the use of the "Status" element name with the appropriate namespace.

How It Works

Figure 3-2 illustrates how namespaces can resolve the conflict between the Accounting and Fulfillment applications. A Reconciliation Document consolidates billing and shipping information about a given order. When the billing status is "invoiced" and the shipping status is "shipped," the document goes to both the Accounting and Fulfillment applications.

Figure 3-2. Resolving a Conflict with Namespaces

When the Accounting application processes the document, it wants to use "Status" as a keyword to signify billing status. When the Fulfillment application processes the document, it wants to use "Status" as a keyword to signify shipping status. Moreover, the same problem extends to other keywords. Example 3-1 shows how to solve this problem using XML Namespaces.

Example 3-1
 <Reconciliation
 xmlns:acct=http://www.foocompany.com/names/acct-REV10
 xmlns:ful="http://www.foocompany.com/names/ful-REV10">
 <acct:Customer>
  <acct:Name>Bar Corporation</acct:Name>
  <acct:Addressee>
   Accounts Payable—Bar Corporation
  </acct:Addressee>
  <acct:Order acct:reference="5566-1010">
   <acct:Status>invoice</acct:Status>
  </acct:Order>
 </acct:Customer>
 <ful:Customer>
  <ful:Name>Bar Corporation</ful:Name>
  <ful:Addressee>
   Loading Dock—Bar Corporation
  </ful:Addressee>
  <ful:Order ful:reference="A9875656">
   <ful:Status>shipped</ful:Status>
  </ful:Order>
 </ful:Customer>
</Reconciliation>

As you can see, attaching the namespace identifier to an element or attribute name with a colon associates that namespace with that element or attribute. Before you use a namespace, you associate the namespace prefix with the namespace name, using the reserved XML attribute "xmlns." The namespace name is a Uniform Resource Identifier (URI), essentially indicating that the owner of an Internet resource is the owner of the namespace. In this case, "acct" signifies the namespace owned by the Accounting department of Foo Company, and "ful" signifies the namespace owned by the Fulfillment department of Foo Company. The example uses the common convention of declaring all namespaces in the document element. This convention enables human readers and software processors to determine easily all the namespaces that may be used subsequently.

Attaching the appropriate prefix to all element and attribute names can be cumbersome. To avoid this chore, you can also set a default namespace, which applies to all contained names, as shown in Example 3-2.

Example 3-2
 <Reconciliation
 <Customer
  xmlns="http://www.foocompany.com/names/acct-REV10">
  <Name>Bar Corporation</Name>
  <Addressee>
   Accounts Payable—Bar Corporation
  </Addressee>
  <Order reference="5566-1010">
   <Status>invoice</Status>
  </Order>
 </Customer>
 <Customer
  xmlns="http://www.foocompany.com/names/ful-REV10">
  <Name>Bar Corporation</Name>
  <Addressee>
   Loading Dock—Bar Corporation
  </Addressee>
  <Order reference="A9875656">
   <Status>shipped</status>
  </Order>
 </Customer>
</Reconciliation>

In Example 3-2, we first set a default Accounting namespace by specifying the value of the "xmlns" attribute without attaching a namespace prefix. All names within the first "Customer" element automatically use the Accounting namespace. Then we set a default namespace Fulfillment for the second Customer element. You can also set a default namespace and then use the syntax in Example 3-1 to specify a different namespace for particular elements or attributes. The combination of the two mechanisms for specifying namespaces delivers a convenient and flexible way to avoid naming collisions.

Practical Usage

In practice, the most important application of XML Namespaces comes when using the related feature standards. For example, a given document may have wrapper tags from an XML messaging standard, tags for domain-specific XML content, and tags from the XLink standard, specifying links to other documents. Without XML Namespaces, it would be impossible to ensure that the correct type of processing component received the elements intended for it. As you will see in the subsequent sections of this chapter, each of the other feature standards has its own namespace to avoid this problem.

In deciding whether to define your own namespaces, consider that the need for them increases in direct proportion to the number of different groups that will use a document format. In the case of industry-specific standards, this need is obviously quite high. The industry standards body does not want the format it defines to conflict with any of the internally defined formats of its members. An enterprisewide format for a large multinational corporation would also almost certainly need to use XML Namespaces so that it would not conflict with any local formats. However, a format intended for use by two parties, each of which plans to construct a dedicated application, might be able to get away without XML Namespaces. But it would probably be a good idea to use them anyway, in case the parties decide to expand the number of participants. A format used only by a single application for internal purposes almost certainly does not need XML Namespaces.

If you work for a software vendor that uses XML in a product, you will want to define a namespace if your XML features are visible to external developers or applications. Typically, using the Internet domain name for your company, the name of the product, and a version identifier will be sufficient.

If you work for a large enterprise that uses XML in an integration or e-commerce application, the proper approach for constructing namespaces is less clear. Of course, you use the domain name of your company for the first part of the namespace and a version identifier for the last part. But the middle part takes some thought. You can use a path that mirrors your organizational chart, but organizational charts may change rather often. If your enterprise has an organizational directory based on a standard data model such as X.500 or LDAP, you may be able to leverage that. In any case, an officially designated person should coordinate the policy used by all groups developing XML applications.


Overview of XPath, XPointer, and XQuery

An XML document represents a convenient package of related XML data. However, there are many situations where people need to work with a subset of data within a single document or an aggregation of data from several documents. To print a shipping label, they might want to grab just the address information from an order document. To print a fraud-resistant receipt, they might want to reference just the last four digits of the credit card number. To produce an upgrade notification, they might want contact information from every order document that includes the purchase of a specific product. To perform these types of operations, it helps to have the means of specifying such subsets and aggregations of document data.

XML Path Language (XPath) provides a syntax for addressing individual nodes within a document hierarchy. Developers specify a path from the document node to the target node. It includes basic selection criteria so that a particular XPath expression can differentiate among similar nodes. To grab only the billing address from orders, a developer could use XPath to specify the "Address" element within an "Order" document whose "addType" attribute is set to "ship" or "billship." XPath is a widely used W3C Recommendation.

When referring to content within a document, authors sometimes need more precision than XPath provides. XML Pointer Language (XPointer) extends the XPath syntax to include the notions of points and ranges. With points, authors can specify locations within a particular node. With ranges, they can specify a set of information that crosses node boundaries. To create a fraud-resistant receipt, an author could use XPointer to specify the last four characters of the "Number" element within the "Card" element of an "Order" document. XPointer is useful primarily for hypertext applications and is taking a long time to work its way through the W3C standards process.

As with other forms of data, people often want to search among collections of XML documents. XML Query Language (XQuery) provides a framework that enables developers to indicate the data they want, the criteria this data must satisfy, and intermediate processing steps. It uses XPath as the format for specifying many of these parameters. To create a mailing list for a product upgrade notice, a developer could use XQuery to search "Order" documents for "Address" elements with an "addType" attribute set to "ship" or "billship" where the document has a "Product" element whose value is "FooBar Version 5." The expectations for adoption of XQuery as a standard search syntax are very high, but it was still a Working Draft at the time of this writing.

How They Work

With XPath, you specify a particular node or set of nodes by indicating a navigation path. Suppose you wanted to refer to each of the "Description" elements in our example order from Example 2-6. You could use three separate XPath expressions to address each of the three desired elements individually, as shown in Example 3-3.

Example 3-3
 /Order/LineItems/LineItem[1]/Description
/Order/LineItems/LineItem[2]/Description
/Order/LineItems/LineItem[3]/Description

Notice that each expression in Example 3-3 provides a full path from the document element to the target element. Each one uses an index in brackets to indicate a specific "LineItem" element based on the order in which they appear in the document. It's somewhat inconvenient to use a different expression for each "LineItem" element, especially if you don't know beforehand how many there are. Example 3-4 shows how to return all the "Description" elements in a document, regardless of how many there are.

Example 3-4
 //Description

The use of a double slash rather than a single slash indicates that desired "Description" elements may have any path to the document element rather than the specific one indicated in Example 3-3. This single expression returns all the three "Description" elements in our example order document at the same time. When multiple elements meet the same criteria, XPath returns the set of all matching elements. However, you may not want all the elements. Consider the previously discussed scenario of accessing only the "Address" element with an "addType" attribute of either "ship" or "billship." Example 3-5 shows the appropriate XPath expression.

Example 3-5
 //Address[@addType="ship" or @addType="billship"]

The double slash first selects all the "Address" elements. Then the qualification of these elements appears in brackets. But instead of an index corresponding to the document order of the element, Example 3-5 specifies a more dynamic constraint based on the "addType" attribute. Because a billing address could have an "addType" value of either "bill" or "billship," the expression concatenates the two possibilities with "or."

All of these XPath expressions return document nodes. When working with text documents, authors may want to specify a point that marks the beginning or end of a text selection. This feature is what we need to solve the problem of selecting the last four digits of a credit card number. The first step in solving this problem is specifying a point just after the 12th digit of a 16-digit number. Example 3-6 shows how to use XPointer for this purpose. Note that the syntax of any final XPointer Recommendation may differ somewhat from that used here.

Example 3-6
 xpointer (//Payment/Card/Number
/child::text()/point()[position()=12])

The first part of this XPointer expression uses XPath syntax to narrow the scope of the match to the "Number" element of the "Card" element of any "Payment" element within the document. This limitation prevents an unintended match within a document that also uses a "Number" element as part of something other than a credit card number. This expression then goes beyond XPath, using the "child:text" clause to narrow further the scope of the match to the internal text content of a node. The "position()=12" clause places the point just after the 12th character of the text content.

By combining two points to specify a range, authors can use XPointer to achieve the precision necessary to indicate only the last four digits of a credit card number. Example 3-7 specifies two separate points, using the syntax from Example 3-6. The first point is just after the 12th character. The second point is just after the 16th character. Connecting these two points with the "range-to" function returns all text between the points—in this case, the 13th, 14th, 15th, and 16th digits of the credit card number. Points can even cross node boundaries. This capability is useful for quoting passages in text documents. It is possible to grab all of the text from the 2nd paragraph of the 3rd chapter to the 5th paragraph of the 4th chapter using a similar XPointer expression.

Example 3-7
 xpointer(//Payment/Card/Number
/child::text()/point()[position()=12]
/range-to(//Payment/Card/Number
/child::text()/point()[position()=16]))

While XPointer provides precision for specifying matches within a single document, XQuery provides precision for specifying matches across multiple documents. The basic framework of XQuery is the "FLWR" expression—FOR, LET, WHERE, RETURN—although the final syntax may evolve from that used in Working Draft. We can use this framework to create a mailing list to notify customers of an upgrade opportunity from a hypothetical collection of documents that uses the "Order" DTD from Example 2-9.

Example 3-8 wraps the query expression in a "MailingList" element so that the query engine can generate a well-formed XML results document. The "FOR" clause specifies the scope of the query as "Order" documents within the "Orders" collection at Foo Company and assigns these document to the "o" variable. The "WHERE" clause constrains the matched documents to those where a "Description" element has the value "FooBar Version 5."

The "RETURN" clause indicates that the results document should wrap individually returned records in an "Addressee" element. It filters the returned information down to all child elements of billing address elements using an XPath expression like that of Example 3-5. Example 3-8 doesn't use a "LET" clause. This clause is useful for sophisticated queries where developers need to manipulate intermediate results to generate the final results.

Example 3-8
 <MailingList>
 FOR $o IN document
  ("http://www.foocompany.com")/Orders/Order
  WHERE $o//Description = "FooBar Version 5"
  RETURN
  <Addressee>
   $o//Address[@addType="ship" or
   @addType="billship"]/*
  </Addressee>
</MailingList>

At the time of this writing, the W3C was rapidly revising XPath and XQuery. XPath 2.0 and XQuery 1.0 were both Working Drafts. The purpose of these revisions is to resolve two primary issues. The first is to ensure compatibility with the XML Schema data model. It would be nice if XPath expressions could use constraints based on the basic data types defined by XML Schema. The second is to ensure compatibility between XPath and XQuery. It makes a great deal of sense to use XPath as the syntax for expressing many of the XQuery parameters. However, there are several subtle issues concerning XPath 1.0 that inhibit this compatibility. XPath 2.0 will resolve these issues to the point where XQuery 1.0 will include XPath 2.0 as a subset.

Practical Usage

Developers and authors have been using XPath for some time, but they are just beginning to experiment with XPointer and XQuery. XPointer and XQuery incorporate much of the XPath syntax. These factors make XPath the most important feature standard for accessing parts of documents. Managers should ensure that all developers working on XML projects are comfortable using XPath.

The biggest use for XPath is as a means of passing references to specific document nodes among software components. Programming languages make extensive use of pointers or handles to refer to specific locations within internal data structures. XML documents are essentially external data structures. Pointers or handles to XML document data would be meaningless to another component that did not share the same memory space. XPath provides the appropriate external reference mechanism. If one component wants to tell another to use a particular address for sending an invoice, it passes the document location and the appropriate XPath.

XPointer offers advantages over XPath primarily for hypertext applications. In these applications, much of the content is text-oriented rather than data-oriented, and documents naturally tend to refer to content within other documents. This applicability extends to boundary situations where a text-oriented document refers to parts of a data-oriented document generated by an application. XPointer is closely related to XLink, the feature standard focused on XML hypertext links and discussed later. The motivation for XPointer stems from the common need for such links to specify a particular place within a document rather than the entire document.

While XPath and XPointer address the issue of accessing data within an individual document, XQuery handles accessing data across a collection of documents. Although the standard was still under development at the time of this writing, the motivation for it is clear. Any time an application generates a significant number of XML documents containing meaningful data, it's only a matter of time before someone wants to search those documents for a specific piece of information. In cases where XML is only one representation of data actually managed within a relational database model, existing query languages and tools will be adequate. But if XML is the native representation of the data or is the common representation provided by otherwise heterogeneous data models, XQuery should see extensive use.


Overview of XSLT

The three standards covered in the previous section enable you to select relatively small parts of documents in various ways. In some cases, you may want to take this manipulation a step further by completely reorganizing a document. This type of transformation requires selecting different parts of a document and rearranging them. XSL Transformations (XSLT) extends the XPath model of how to address parts of documents with more sophisticated operators that enable developers to specify these rearrangements.

You may be wondering why, as the name implies, XSLT is associated with XSL. Originally, people saw XSL as a generic way to display XML documents for all presentation technologies. Accomplishing this goal naturally required two different types of features: (1) features for rearranging document content so that it made the most sense for display and (2) features for attaching display properties to the content. However, rearranging content turned out to be useful for other purposes, and the appropriate display properties turned out to be a topic of extensive debate. Because people wanted to use the rearranging features and agreement on these features was far ahead of that for display properties, the two standards separated and XSLT achieved W3C Recommendation status ahead of XSL.

One of the greatest demands for XSLT came from trying to avoid the meta-incompatibility problem where two different organizations use two different, so called, standards. XML enhances compatibility by making it possible to exchange information using a standard syntax and to define standard formats that constrain the structure of this information. But what if two different groups define different formats for the same logical type of document? That leads to meta-incompatibility and big headaches for managers who are developing applications that must exchange documents with both groups. In some sense, this is the opposite problem from the one solved by Namespaces. Namespaces solve the problem of groups calling different concepts by the same name. XSLT solves the problem of groups calling the same concept by different names.

Such a scenario is highly likely in applications such as supply chain management where two companies want to exchange conceptually the same information but already have formats that they use internally. Also, industry groups in finance, telecommunications, and transportation have defined formats for transactions in those industries. In some cases, multiple industry groups are working on the same problem, creating the potential for dueling standards. Moreover, with the integration of global supply chains, standards for related industries such as manufacturing and shipping may need to ensure compatibility where they overlap.

The idea behind XSLT is to define a scripting language—using XML syntax, of course—that enables developers to transform one format into another. Wherever two formats overlapped, developers would create a transform that extracts the overlapping information from one format and rearranges it into the other format. These transforms are directional; to rearrange documents in both directions, you would need two separate transforms.

How It Works

Consider the basic problem of automatically placing an order with a trading partner over the Internet. Foo Company has defined a Foo Company Order DTD that it uses internally. Bar Corp has defined a Bar Corp Order DTD that it uses internally. Now Foo Company wants to place orders automatically with Bar Corp.

When Foo Company creates a Foo Company Order Document, it is valid with respect to the Foo Company Order DTD. However, for Bar Company to accept the order, the order document must be valid with Corp to the Bar Corp Order DTD. To achieve this end, Foo Company creates a Foo Bar Transformation document that specifies how to translate a Foo Company Order Document into a Bar Corp Order Document as shown in Figure 3-3.

Figure 3-3. Translating Order Formats with XSLT

To see how XSLT works, let's consider a simple example. Examples 3-9a and 3-9b show parts of order documents in two different formats. Example 3-9a models currency information as an attribute on the "Order" element. Example 3-9b models currency information as a child element of the "Order" element. The choice of modeling information as an attribute or child element is an arbitrary one, so it is likely that two different DTDs for the same concept would choose different modeling techniques for at least one piece of information.

Example 3-9a
 <Order currency="USD">
...
</Order>
Example 3-9b
 <Order>
 <Currency>USD</Currency>
...
</Order>

Example 3-10 is the XSLT code for transforming the attribute model of Example 3-9a to the child element model of Example 3-9b. Note the use of the XSL namespace to denote the elements using XSL-specific constructs. The transformation document selects the "Order" element in the source document. It then begins a new "Order" element with a new "Currency" child element in the translated document. It inserts the value of the "Currency" attribute of the selected order element in the source document as the element content of the "Currency" element in the translated document.

Example 3-10
 <xsl:template match="/">
 <xsl:for-each select="Order">
  <Order>
   <Currency>
    <xsl:value-of select="@currency"/>
   </Currency>
  </Order>
 </xsl:for-each>
</xsl:template>

As you can see, XSLT is itself XML. Every XSLT document is also an XML document. Therefore, all the tools for creating and managing XML documents work with XSLT documents. Of course, they do not necessarily understand the specific XSLT syntax, but they do provide some leverage. Although this example is motivated by valid documents that use different DTDs, XSLT scripts also work with well-formed ones. However, people often use XSLT when they have in one data format many documents that they want translated to another data format—precisely the same conditions under which they use valid documents.

At the time of this writing, the W3C had commenced work on XSLT 2.0. The work was in its beginning stages, focusing on the requirements for the new version. Given the work on XML Schema and XPath 2.0 since XSLT 1.0 appeared as a Recommendation, one of the primary requirements was compatibility with the rest of the XML standards family. The rest of the proposed requirements mostly revolved around making XSLT easier to use. Since the release of XSLT 1.0, a great deal has been revealed about what people want to do with XSLT and what they are finding they can do. So there are a number of proposals to close the most glaring gaps between frequency of need to perform a type of operation and the difficulty of actually performing the operation.

Practical Usage

XSLT is becoming an increasingly important part of the XML family of specifications. XML provides a general grammar for defining data formats, but as data moves through an organization, different consumers of the data will probably want it in different formats. XSLT provides the mechanism for supporting this customized data flow. It fundamentally alters the issue of information exchange from defining common data formats for all applications to defining the transformations necessary to deliver data to each application in the format it desires.

Another common use for XSLT is transforming data documents into presentation documents. There are a number of presentation formats based on tagged markup. HTML is by far the most widely used, but there is also Wireless Markup Language (WML) for small devices and VoiceXML for voice-driven interfaces. The W3C has redefined HTML in terms of XML with its XHTML initiative, while WML and VoiceXML are already defined in terms of XML. Therefore, it is fairly straightforward to create XSLT transforms that take a data-oriented format like an Order and rearrange it into a presentation-oriented format like XHTML. It's even possible to create pages that incorporate advanced features like JavaScript or VBScript.

Figure 3-4 shows the typical architecture of an XSLT-driven Web site. Note that the XSLT transformation from XML data into XHTML presentation takes place on the server rather than the client. While this approach violates the vision of users specifying their own presentation, it is much more convenient to do transforms on the server with the current Internet software infrastructure. Some projects have even combined XSLT with Cascading Style Sheets (CSS). CSS enables authors to specify detailed screen presentation properties for HTML elements. One drawback of CSS is that it can only add formatting information to a document; it cannot rearrange the document information to make more sense to the user. In this sense, XSLT and CSS complement each other. The application reorganizes the data-oriented XML with XSLT to suit the user requirements and then applies CSS to get the final presentation-oriented document.

Figure 3-4. Driving Web Site Display with XSLT

There is some backlash against using XSLT. Some experts observe that XSLT is evolving into a complete programming language even though existing programming languages do almost as good a job of transforming XML documents. They believe that the small benefit offered by XSLT is overwhelmed by the cost of forcing developers to learn yet another special-purpose programming language. Finally, they feel the performance of XSLT is dismal when compared to the same operations in traditional programming languages. Others argue that because of the complexity of the low-level interfaces, using existing programming languages requires a high degree of skill that many Web developers do not have. They point to the success of JavaScript and to HTML itself as evidence that special-purpose scripting languages improve the reach of information exchange technologies. Last, they point out that people also complained about Java's initial performance and that the performance of XSLT will also certainly improve as it matures. Given this debate it's probably worth encouraging your architects and lead developers to examine carefully whether XSLT is right for a particular project.


Overview of XSL

As discussed in the previous section, XSLT was originally part of the effort to create a master presentation language for XML documents. While the ability to rearrange document content turned out to have many applications, it is still a necessary component of document presentation. Extensible Stylesheet Language (XSL) uses the XSLT syntax for modifying the document tree and adds its own syntax for attaching presentation properties to the nodes.

XSL's presentation properties focus on the requirements of paginated output, where the notion of a page has important formatting ramifications. While the most obvious form of paginated output is printing, electronic renditions of pages such as PDF or electronic books also clearly fall in the paginated category. You could even imagine pagination applying in some sense to aural output for books on tape. As you can probably guess, XSL is therefore of primary interest to publishing professionals.

The goal of XSL is to specify a language that allows people to apply paginated formatting to XML documents without contaminating logical document content. There are three primary requirements for a solution.

  1. Applying formatting rules to elements. Authors must be able to specify complete formatting rules for each type of element in a document. These rules include font format, indentation, line spacing, leading and trailing space, table formatting, and so on. Applying formatting rules to elements makes it very efficient to create stylesheets in conjunction with DTDs. Because all documents using a DTD have the same structure, it makes sense that they should have the same formatting rules.

  2. Applying formatting rules to pages. In addition to formatting rules for elements, paginated output requires applying formatting rules to each page. Changing the content of elements in a document changes the position of page breaks. It would be highly cumbersome for authors to figure out the position of these page breaks and insert special elements to provide features such as margins, headers, and footers. Therefore, style sheets for paginated output need general rules for specifying these page-level formatting concepts.

  3. Usable with different display technologies. One of the big advantages of a stylesheet approach is that an organization could render the same document content in different formats by changing stylesheets. A publisher could render the same book in a printing press format, an electronic book format, and a books on tape format. Therefore, the formatting rules must be flexible enough to accommodate the special needs of these different presentation platforms.

How It Works

The fundamental idea behind XSL is to use the same content to generate different paginated presentations. As just discussed, different presentations can accommodate different platforms. They could also accommodate different conventions, such as those used to format product press releases versus product data sheets, and users' physical differences, such as sight impairment.

People may also have logical differences. In Figure 3-5, employees in the Accounting department and Fulfillment department may want to print the same Order Document differently. Accounting employees want emphasis on the billing address for printing invoices, and Fulfillment employees want emphasis on the shipping address for printing packing lists. Each department can each define its own stylesheet, and the XSL processor can generate different page layouts for the same document.

Figure 3-5. Applying Different Stylesheets for Different Users

Generating output occurs in two steps. First, an XSL stylesheet transforms a content document into a presentation document with the same mechanisms as using XSLT to transform one data document into another data document. The resulting presentation document contains XML content elements decorated with XSL formatting objects and is called a formatting object tree. Actually generating the final output requires the use of a formatter as a second step. As Figure 3-5 shows, a given presentation document may be rendered into many different outputs. In this case, each department generates both a Postscript file for printing and a PDF file for electronic storage.

Example 3-11 shows a fragment of a stylesheet you could apply to the "Order" document in Example 2-6 that emphasizes the billing address over the shipping address. With XSL, you define formatting templates that apply to an element and its subelements. This simple example has two templates. Note that instructions for selecting the particular elements are the same as for XSLT and use the "xsl" namespace, whereas instruction for formatting output use the "fo" namespace.

Example 3-11
 <xsl:template match="address" [@addType="bill" or    @addType="billship"]>
 <fo:block
  fo:font-size="large"
  fo:font-weight="bold"
  fo:font-family="Arial"
  fo:line-height="2"
  <xsl:apply-templates/>
 </fo:block>
</xsl:template>
<xsl:template match="address[@addType="ship"]">
<fo:block
  fo:font-size="small"
  fo:font-weight="normal"
  fo:font-family="Times"
  fo:line-height="1"
  <xsl:apply-templates/>
 </fo:block>
</xsl:template>

The first template selects billing address elements. For each child element, it starts a new paragraph, in large, bold, Arial type with double line spacing. The "apply-templates" syntax applies this formatting to all children of the "address" element, including the "firstname," "lastname," "street," "city," "state," "postal," and "country" elements. The second rule selects shipping address elements. For each child element, it starts a new paragraph in small, normal, Times type with single line spacing. So addresses for billing appear in large, bold type. Addresses for shipping appear only in small, lightface type.

As mentioned previously, XSL also includes formatting rules for pages as well as elements. These rules enable you to specify common page-related concepts such as margins, headers, and footers. Example 3-12 shows an XSL fragment that provides a header and a footer for orders.

Example 3-12
 <fo:layout-master-set>
 <fo:simple-page-master master-name="Order"
   page-width="8.5in" page-height="11in"
   margin-top="0.5in" margin-bottom="0.5in"
   margin-left="0.5in" margin-right="0.5in">
  <fo:region-before extent="1.0in"/>
  <fo:region-body margin="1.0in"/>
  <fo:region-after extent="1.0in"/>
 </fo:simple-page-master>
</fo:layout-master-set>

<fo:page-sequence master-name="Order">
 <fo:static-content flow-name="xsl-region-before">
  <fo:block>Foo Company Order</fo:block>
 </fo:static-content>
 <fo:static-content flow-name="xsl-region-after">
  <fo:block> p. <fo:page-number/></fo:block>
 </fo:static-content>
</fo:page-sequence>

The first part of Example 3-12 defines a page master named "Order" and gives it half-inch margins. Besides the main body of the page, it also defines a header with the "fo:region-before" element and a footer with the "fo:region-after" element. The second part assigns content to this header and footer. The header gets the title "Foo Company Order" and the footer gets a page number identifier.

Practical Usage

XSL achieved W3C Recommendation status relatively recently, so there is not much practical experience with its application. At the time of this writing, there were only a handful of XSL processors and formatters for a very few output formats available. Most of the work with XSL so far has been performed by contributors to the standard and other industry experts using it on their own projects, such as book publishing.

However, given the history of XSL, we can make some fairly confident predictions about where it will be most useful. As discussed in the coverage of XSLT, applications that dynamically generate XML content as part of a server-side application are using XSLT to generate markup-oriented presentations such as HTML, WML, and VoiceXML directly. Combine this usage pattern with the focus of XSL on paginated output, and it becomes clear that the primary use of XSL will be for publishing.

Within publishing, XSL has the potential to affect both companies dedicated to the business of publishing and internal publishing groups within companies. For publishing companies, XSL enables them to target different delivery platforms such as printed pages, electronic books, and books on tape efficiently. For internal groups, XSL enables them to repurpose the same content for different uses such as press releases, data sheets, and technical specifications. There is also the possibility that XSL could greatly expand the scope of internal publishing to include capturing application-generated XML content, collating it, and dispersing it in a published form. All of these uses for XSL depend on the adoption by vendors of traditional publishing and content management systems.


Overview of XLink

Many people want to use XML on the Web, where linking is an absolute requirement, so it makes sense to define a companion linking standard for XML. Also, once people can easily exchange and understand structured documents, specifying relationships among them becomes very valuable. These forces have motivated the XML Linking Language (XLink) initiative. However, the advanced features of XLink may have little appeal to mainstream Web users.

Most Web authors are familiar with HTML and its href syntax. Therefore, a goal of the XLink specification is to make it very simple to use this syntax to create a one-way link from a point in an XML source document to a target document. This compatibility lowers the learning barrier for existing Web developers. It also raises the question of whether XLink gives these developers enough incentive to adopt it.

With XLink, it is possible to go beyond simple HTML-like links. Theoretically, once people have access to a wide variety of content structured in XML, they will naturally want to connect different pieces of this content in many different ways. To accommodate this need, XLink offers powerful capabilities for specifying relationships among multiple target documents rather than just between two documents. Of course, this more sophisticated approach requires more sophisticated software infrastructure and more sophisticated user behavior.

Suppose that an attorney has researched a particular esoteric and complex point of the law. Parts of many different court opinions apply in differing degrees to this attorney's particular case. With XLink, the attorney could create an extended link that led to all the different opinions, categorized the opinions by relevance, and pointed to specific passages in the opinions. The attorney could then e-mail this extended link to other attorneys on the case. To accomplish this task, the attorney would need software features for creating the extended link, and his colleagues would need software features for navigating them. Whether there is a need for such features outside of specialized domains such as law and medicine remains an open question.

How It Works

Links with multiple targets are an exciting development, but let's start with the simple case. If you wanted to link just one Order Document to one Customer Document, you would use a simple link. A simple link is very similar to a standard HTML link. Figure 3-6 shows how the link is defined as part of the Order Document and targets the Customer Document.

Figure 3-6. A Simple XLink

The syntax for a link is itself XML. Therefore, extracting the linking information from a document does not require any additional capabilities. However, knowing what to do with this information requires an XLink processor that understands linking information. Document viewers such as Web browsers, therefore, need to include such a component to support links using the XLink syntax. Example 3-13 shows the actual syntax for the simple link represented in Figure 3-6.

Example 3-13
 <Customer
 xlink:form="simple"
 xlink:inline="true"
 xlink:href="http://www.foocompany.com/customers/   barcorp.xml"
 xlink:show="new"
 xlink:actuate="user"
 xlink:title="Bar Corporation"
 xlink:label="Customer">
 Bar Corporation
</Customer>

This XLink comprises a set of attributes on an element. Because these attributes use the XLink namespace, you can attach them to any element without affecting the processing of that element. Only the XLink processor will receive the information contained in these attributes. The "inline" attribute indicates whether the element contents, in this case "Bar Corporation," are themselves part of the link. The "form" attribute specifies that this link is a simple link. The "href" attribute specifies the URI of the target document.

In addition to these basic attributes, there are special attributes to indicate the desired behavior for the link. The "show" attribute instructs the processor to open a new window for the target document. The "actuate" attribute instructs the processor to activate the link when the user clicks on it. The rest of the attributes provide the XLink processor with contextual information about the link. The interpretation of contextual attributes may be specific to a particular XLink processor.

As shown in Figure 3-7, extended links allow multiple target documents. In this example, a Tracking Document associates the corresponding Invoice Document and Shipping Document. This Tracking Document contains only this link. Even though the Tracking Document is a valid XML document, it is logically independent of the content it connects. This separation of relationships from content is a powerful tool for categorizing content on the Web. For a given set of logical documents, there may be dozens of different categorizations, all implemented with links and all independent of each other.

Figure 3-7. Extended XLink

An extended link has more complex structure than a simple link. Because the link includes multiple targets, there will be subelements corresponding to each target. Therefore, extended links have two kinds of attributes: those that appear once in the top-level element of the link and those that appear in each of the subelements corresponding to a target. Example 3-14 shows this syntax for the link represented in Figure 3-7.

Example 3-14
 <OrderTracking xlink:form="extended">
  <Invoice
  xlink:form="locator"
  xlink:href="http://www.foocompany.com/
   BarCorpInvoice.xml"
  xlink:title="Bar Corporation Invoice"
  xlink:label="Invoice"/>
  <Shipment
  xlink:form="locator"
  xlink:href="http://www.foocompany.com/
   customers/BarCorpShipment.xml"
  xlink:title="Bar Corporation Shipment"
  xlink:label="Shipment"/>
</OrderTracking>

The "OrderTracking" element is at the top level for this link, so it has the "xlink:form" attribute set to "extended." The XLink processor then knows to look for additional attributes on the subelements. In this case, there are "Invoice" and "Order" subelements, each corresponding to a target document. The "locator" value of the "xlink:form" attribute indicates that the subelement corresponds to an external document, while the "xlink:href" attribute gives the location of the document. Extended links have the same attributes as simple links for indicating contextual information such as "xlink:label" and "xlink:title." In addition to the attributes shown here, there is also a set of advanced extended link attributes useful for creating sophisticated semantic webs among documents.

As discussed in the coverage of XPointer, you can combine XLink and XPointer to refer from one document to a specific location within another document. In this case, you add a "#" to the end of the URI in the "xlink:href" attribute and then append the XPointer expression.

Practical Usage

As with XSL, the finalized version of XLink is relatively new. The most important factor affecting the adoption of XLink is the availability of XLink-capable Web infrastructure. At the time of this writing, only open source browsers supported XLink, although work on compatibility with major commercial browsers was under way. Moreover, most browsers limit support to simple links. While XLink simple links are somewhat more powerful than HTML links, they still suffer from the problem of being embedded in the source document. Therefore, delivering on the goal of separating relationships from content necessitates support for extended links.

Support for extended links really requires server-side support because an extended link is essentially a virtual document that requires dynamic assembly. The actual implementation of such a server is more difficult than it might seem because there are issues with how to deal with links to documents that themselves contain links. If limited demand for such advanced capabilities does not justify the inclusion of the necessary infrastructure in general purpose Web servers, the use of extended links may be limited to dedicated hypertext systems in specialized applications such as bioinformatics.

Just as XSL will likely be limited to publishing applications, XLink will likely be limited to hypertext applications. People probably won't apply XLink to application-generated XML documents very much, so application developers will not need to be very familiar with it. However, potentially any XML document could become part of paginated output or part of a hypertext archive, so a general awareness of these standards is valuable.


Overview of XML Schema

One of the latest and most eagerly awaited XML-related initiatives is XML Schema. As the name implies, it is intended for use in specifying the structures of document types. DTDs already provide a mechanism for addressing this function, but they have some drawbacks. These drawbacks are especially severe for data-oriented application.

The vocabulary for DTDs is the only one in the XML platform that does not follow XML document syntax. Therefore, applications and tools must be able to process both XML document and XML DTD syntax. Moreover, developers and authors must learn how to use both types of syntax. While not a functional short coming, this difference is annoying.

The functional shortcoming of DTDs is that they specify data types with far less precision than programming languages and DBMSs. In data-oriented applications where the purpose of XML is to facilitate the exchange of data among different pieces of software, the XML processor cannot perform even the most basic validation of whether a piece of data is a string, decimal, or date.

Therefore, programmers must write special code to convert data back and forth between the less precise DTD type system and the more precise programming type systems. This conversion code imposes a significant development burden, but it also raises the possibility of different conversion code treating the same XML document data differently. So despite using the same DTD, such software could still be incompatible due to different conversion policies.

Another concern about DTDs is that they lack flexibility in reusing structures across formats and supporting formats that evolve over time. DTDs work well when a single group controls all of the definitions used in its software and all of the software that uses a particular definition. But the use of XML documents over the Internet has evolved so that organizations want to combine definitions from a variety of formats to suit their business needs and then want to extend definitions without breaking software they have already deployed.

How It Works

Like a DTD, an XML Schema file is a description of the rules that valid documents must follow. However, the process of designing formats with XML Schema is more similar to designing programming data structures or database schemas. Example 3-15 uses XML Schema to specify rules for the address section of our "Order" format defined using a DTD in Example 2-9.

Example 3-15
 <xsd:complexType name="AddressesType">
 <xsd:choice>
  <xsd:element name="BillShip" type="AddressType"/>
  <xsd:sequence>
   <xsd:element name="Ship" type="AddressType"/>
   <xsd:element name="Bill" type="AddressType"/>
  </xsd:sequence>
 </xsd:choice>
</xsd:complexType>

This example enhances the corresponding section of Example 2-9 in two ways. First, it starts by defining the "AddressesType" rather than the "Addresses" element. By defining a more generic type, developers will find it easier to reuse this block of rules across multiple formats such as Orders, Invoices, and Statements. Second, it takes a more explicit approach to differentiating among shipping and billing addresses. It specifies that elements of the "AddressesType" have the choice of having either a single "BillShip" element or a sequence with a "Bill" element, then a "Ship" element. However, all of these elements are of the same "AddressType." By making it easier to define an "AddressType" and reuse its structure across elements, XML Schema raises the distinction between billing and shipping addresses in the element name rather than hiding it in an attribute. Example 3-16 shows the definition for the "AddressType."

Example 3-16
 <xsd:complexType name="AddressType">
 <xsd:sequence>
  <xsd:element name="FirstName" type="xsd:string"/>
  <xsd:element name="MiddleName" type="xsd:string"/>
  <xsd:element name="LastName" type="xsd:string"/>
  <xsd:element name="Street" type="xsd:string"
  minOccurs="0" maxOccurs="4">
   <xsd:attribute name="lineOrder"
   type="xsd:integer" use="implied">
   <xsd:unique name="lineOrderUnique>
    <xsd:selector xpath="/"/>
    <xsd:field xpath="@lineOrder"/>
   </xsd:unique>
  <xsd:element name="City" type="xsd:string"/>
  <xsd:element name="State" type="xsd:string"/>
  <xsd:element name="Postal" type="PostalType"/>
  <xsd:element name="Country" type="xsd:string"/>
 </xsd:sequence>
</xsd:complexType>

This example specifies a structure for an address similarly to Example 2-9, but with three improvements. First, the cardinality constraint on the "Street" elements is more precise than simply 1 or more. Constraining the maximum number of such elements to 4 could be very important for ensuring that addresses fit on envelopes and labels. Second, we've explicitly indicated that the "lineOrder" attribute must be an integer and have a unique value for each "Address" element. Last, we've defined the "Postal" element as having the "PostalType" rather than a generic "string" type.

The definition of this "PostalType" illustrates how XML Schema accommodates additions over time. Example 3-17 shows how to define this type as a string with a maximum length of 10. This definition could be precise enough for the initial version of the schema, but we might want to perform more validation in a subsequent version by having specific formats for US and UK postal codes.

Example 3-17
 <simpleType name="PostalType">
 <restriction base="string">
  <length value="10" fixed="true"/>
 </restriction>
</simpleType>

Example 3-18 shows how to extend Example 3-17 seamlessly to accommodate this change.

Example 3-18
 <element name="USPostal" type="USPostalType"
substitutionGroup="Postal"/>
<element name="UKPostal" type="UKPostalType"
substitutionGroup="Postal"/>

<simpleType name="USPostalType">
 <restriction base="PostalType">
  <pattern value="\ d{ 5} |\ d{ 5} -\ d{ 4} "/>
 </restriction>
</simpleType>
<simpleType name="UKPostalType">
 <restriction base="PostalType">
  <pattern value="[A-Z]{ 2} \ d\ s\ d[A-Z]{ 2} "/>
 </restriction>
</simpleType>

In the new version of the schema represented by Example 3-18, we first define the "USPostal" and "UKPostal" elements as members of a substitution group for "Postal" elements. These definitions make it allowable for conforming documents to substitute a "USPostal" or "UKPostal" element for any "Postal" element. Then we extend the "PostalType" so that it now has the subtypes "USPostalType" and "UKPostalType." We can then use these more specific types in place of less specific "PostalTypes." This combination of substitution groups and type derivation provides something very much like polymorphism in object-oriented programming languages.

Each of these subtypes specifies different restrictions on the pattern of valid postal codes. For the United States, the allowable pattern is five numeric digits plus an optional dash followed by four numeric digits. For the United Kingdom, the allowable pattern is two capital letters followed by a numeric digit, a space, and a numeric digit followed by two capital letters. Any new software that creates "Order" documents can specify a "USPostal" or "UKPostal" element based on the country where the order originates and be assured that this important typing information will be enforced as the document works its way through the supply chain. Note that any document conforming to the original schema with just "Postal" elements is still valid, so this extension maintains backward compatibility.

In addition to improving schema over time, XML Schema also has features that make it easy to reuse the efforts of other groups. Suppose that the developers creating the schema for Order documents want to take advantage of the work done by another group in the area of payment encoding. Example 3-19 shows how to defer the format validation of a specific element to a schema defined elsewhere.

Example 3-19
 <element name="Payment>
 <complexType>
  <any namespace=http://www.foocompany.com/xml/payments
  processContents="strict"/>
 </complexType>
</element>

As with the "Order" DTD in Example 2-9, this snippet defines a "Payment" element as part of the format. However, rather than defining the subelements themselves and forcing developers to maintain these rules independently, this example enables any element from the namespace maintained by another hypothetical development team. By setting the "processContents" attribute to "strict," this example instructs the XML Schema processor to retrieve the other schema and validate the elements against it. This approach makes the very definition of data formats a collaborative exercise well suited to the Internet.

Practical Usage

Most new commercial XML applications that work with server applications or databases use XML Schema. The rapid emergence of support for the standard in development tools and components makes it the most convenient means of ensuring the precise compatibility necessary for software data exchange. The endorsement of the W3C really makes it the only viable choice where a format may be released to external parties as a means of commercial inter operability. In cases where an organization intends to use a format for internal purposes only or the format is part of a noncommercial effort, you could consider alternatives.

XML Schema is not without its detractors. Many developers complain that it is overly complex, failing to follow the 80-20 rule. They say that instead of making it easy to satisfy the needs of most developers that need a relatively small set of features, it forces everyone to deal with a huge feature set. There are even efforts to promote alternative means of specifying XML data formats that are easier to understand. The most visible alternative is RELAX NG from the Organization for the Advancement of Structured Information Standards (OASIS). However, the simple reality is that the XML Schema standard, while imperfect, is usable as it is. Therefore, coming from the premier XML standards body makes its widespread acceptance very likely.

For managers and developers worried about minimizing complexity, a better strategy is to develop formal guidelines about what features to use. As part of an overall XML technology strategy, the senior architects and developers within an organization may agree on the parts of the standard that they will use and then enforce this decision internally. As a proper subset of the standard, this approach retains compatibility. If externally provided formats go outside the designated feature set, they can decide on a case by case basis whether the benefit of the format justifies the additional complexity burden.


Technical Standards

Because developers are using XML for so many different purposes and XML can accommodate significant complexity, there are understandable difficulties covering all the potential cases of XML document processing. Therefore, there are several standards that add rigor to the XML processing model. These standards ensure that XML documents remain portable, interchangeable, and unambiguous. These are some of the more important ones.

  • XML Infoset. The original XML specification focused primarily on defining how to process content from the document perspective. There was no means to construct an abstract model of document information. This made it very difficult to perform operations like determining if two documents contain the same logical information. If they order elements differently, contain different whitespace, or use alternative syntax for empty elements, the document representations are different. Infoset acts much like a data dictionary for documents, cataloging only the logical information they contain, so that processing software can operate on them at a more abstract level.

  • XML Include. Developers commonly want to partition their code into different files for programming convenience. XInclude enables developers to include an external XML file in another. While technically a new feature, all of the issues with delivering it lie in the details of internal processing and treatment of content from the external file.

  • XML Fragment Interchange. The issues of processing content from an external source become even more difficult if the content is only a fragment of a complete file. While conceptually straightforward, treating such fragments appropriately requires a significant amount of work on the processing model. Therefore, it is really more of a technical issue than a feature issue.


Infrastructure Standards

The whole reason for XML is to ensure compatibility during information exchanges across the Internet. In certain cases, existing technologies for this purpose are inadequate or incompatible with XML. In these cases, the W3C or another standards organization has moved in to provide a common approach. While the number of such standards may seem large, in relation to the number of different ways people have tried to use XML, such proliferation is actually somewhat rare. Here are some of the infrastructure standards that are most relevant to electronic commerce.

  • SOAP. Using XML as the basis for distributed communication is a common application. Unfortunately, if different groups develop their own protocols for this purpose, the resulting systems could not exchange messages. SOAP provides a unified approach (discussed more fully in Chapter 4) because it forms the basis for most XML messaging systems. The W3C is the standards body responsible for SOAP, and it was a late-stage working draft at the time of this writing. An earlier version, however, was in wide commercial use.

  • XHMTL. HTML is very close to XML, but HTML documents are not quite XML documents. This slight difference presents a problem because all of the emerging infrastructure for working with XML documents won't work with HTML documents. XHTML is a reformulation of HTML that takes care of these small incompatibilities. A forthcoming benefit of this approach is that XHTML is modular, making it better suited for different access devices and more extensible for customized applications. XHTML is a W3C Recommendation.

  • XForms. Most interactive Web applications use HTML forms to gather data from users. Integrating this data into an XML format requires translating from the HTML form layout to an XML layout. XForms introduces an XML-based form layout that solves the issue. It ensures that the presented forms include a data model for the form data so that applications can treat each piece of a submission appropriately. This more abstract approach also facilitates the use of the samev form-processing code with data submitted from different types of devices. The W3C is the standards body responsible for XForms, and it was a late-stage working draft at the time of this writing

  • XML Signatures. In cases where XML documents represent legal commitments, parties to the exchange need the ability to sign them cryptographically. While the technology and standards exist to achieve a basic form of cryptographic signatures on XML documents, they can result in incompatibilities and do not take advantage of XML document properties. XML Signatures provides both the standardization and the additional features necessary for more fluid XML-based electronic commerce. XML Signatures is a W3C Recommendation.

  • XML Encryption. Certain XML documents may include sensitive information that the sender and receiver do not want revealed to third parties. As with XML Signatures, the technology and standards exist to perform encryption on XML documents. But the XML Encryption specification provides further standardization and features specific to the structure of XML documents. XML Encryption was a W3C Candidate Recommendation at the time of this writing.

  • Security Assertion Markup Language (SAML). The Web facilitates one server handing off user requests to other resources. It is inconvenient when the user must prove his identity and access rights to every resource. SAML enables single sign on by providing a standard protocol that resources can use to verify the user's identity and access rights by querying the referring server. OASIS is the standards body responsible for SAML, and it was a last-call working draft at the time of this writing.

  • Business Transaction Protocol (BTP). Many application and database systems rely on transaction semantics that require an entire group of actions to happen either all at once or not at all. To work effectively over the Internet, these systems need a standard means of enforcing these transaction semantics. BTP provides the necessary transaction model and protocol. OASIS is also the standards body responsible for BTP, and it was a late-stage working draft at the time of this writing.


Technical Summary

  • When a document contains elements generated by more than one author, XML namespaces prevent collisions. For example, order information from the Accounting and Fulfillment departments can be distinguished by "acct:order" and "ful:order," respectively.

  • To avoid having namespace prefixes for every element and attribute name, the XML namespace syntax allows authors to set the default namespace with the reserved "xmlns" attribute.

  • XPath enables developers to address particular element nodes within a document tree, while XPointer enables authors to address content within and across element nodes.

  • When developers have two different DTDs or schemas that govern the same logical content, they can easily transform documents between the two formats using XSLT.

  • Developers can use XSLT to transform XML content directly into tag-oriented display formats such as HTML,WML, and VoiceXML.

  • XSL allows authors to create paginated output from XML content for one or more audiences.

  • XLink allows authors to link one XML document to another XML document or even to many other XML documents.

  • XML Schema is superior to DTDs for data-oriented applications because it uses XML syntax, enforces precise datatype restrictions, and has more flexibility for reuse and extension.

  • Technical standards like Infoset and XML Fragment Interchange improve the processing of XML documents and reduce ambiguities that could lead to misinterpretations.

  • Infrastructure standards like XHTML and XML Signatures ensure that common operations on XML documents remain uniform across applications.


Chapter 4. XML Messaging and Web Services

    Executive Summary

    Motivation

    Protocol Design Problem

    Messaging Architectures

    Major Web Services Initiatives

    Other Web Services Initiatives


Executive Summary

XML messaging is a technique that uses XML as the format for messages sent between software programs over the Internet. This technique has the potential to revolutionize distributed computing. XML, however, does not really enable new types of distributed applications but rather makes important types of existing applications more flexible, interoperable, and cost effective.

Distributed computing is the branch of computer science that is concerned with how pieces of software running on different machines communicate via a network. Some examples are connecting an order entry application running in a New Delhi call center to a product distribution application running in a Los Angeles warehouse or managing the ventilation, elevator, and security systems of an office building from a central control room. The Web is perhaps the best known example; it satisfies requests from user desktops throughout the world for content located on corporate servers. Each of these examples has the same basic characteristics. A program on one machine must send information to a program on another machine. The receiving program must interpret this information and take appropriate action.

EDI and distributed objects are two very successful distributed computing paradigms. EDI enables trading partners to turn the physical flow of document-oriented business processes into an electronic flow, thereby decreasing costs and cycle times and increasing predictability. Unfortunately, overly rigid formats for electronic business documents constrain EDI's ability to accommodate business change.

Distributed objects enable an enterprise to turn physically separate business systems into a single virtual system, thereby decreasing costs, improving efficiency, and improving decision making. Unfortunately, lack of interoperability among different distributed object implementations constrains their ability to integrate business operations. Both technologies are expensive to deploy and maintain.

XML messaging addresses the problems of flexibility, interoperability, and cost effectiveness. However, the EDI and distributed object paradigms are distinctly different in their semantics and infrastructure requirements. The EDI paradigm is about connecting similar systems across organizational boundaries. The distributed objects paradigm is about connecting different systems in the same organization. When EDI software sends information over the network, it sends large, self-contained documents like Request for Quote, Quote, and Purchase Order. The receiving software accepts a document and processes it over minutes, hours, or even days and sends back another large, self-contained document whenever it is ready. When distributed object software sends information over the network, it sends small, detailed instructions like Create New Order, Add Line Item to Order, and Set Payment Terms of Order. The receiving software processes the instruction in milliseconds and immediately returns the results to the waiting sender.

Software designed following the EDI model has relatively few exchanges of large blocks of data, but a series of explicitly related exchanges can take place over days, weeks, or months. Both the sender and the receiver always know the implicit context of what the other party is trying to accomplish. Software designed following the distributed objects model has exchanges that are relatively disconnected from each other but that occur in rapid-fire flurries lasting a few seconds. The sender knows the instructions that the receiver can execute, but the receiver often has no context for how the individual instructions relate to the sender's overall goal. Therefore, implementing these two paradigms with XML creates some confusion. They share much of the same low-level infrastructure, but the way they use this infrastructure differs significantly. This has serious implications for organizations considering building applications on top of an XML messaging foundation.

The situation has become even more confusing with the rise of Web Services. Originally, the term applied to the use of three protocols: Simple Object Access Protocol (SOAP); Web Services Definition Language (WSDL); and Universal Description, Discovery, and Integration (UDDI). The use of these protocols favored the distributed objects paradigm. However, as people began considering systems based on the EDI paradigm, the term began to mean a comprehensive interoperable framework based on XML messaging that could address both paradigms.

Because you can use XML messaging and Web Services as part of either paradigm, you should explictly choose between the business document paradigm of EDI and the remote interface paradigm of distributed objects for each project. Once you have determined the paradigm, much of the apparent complexity of the technology falls away. You can more easily identify which Web Services initiatives match your needs. Table 4-1 summarizes these choices, and the rest of this chapter addresses them in detail

Table 4-1. EDI vs. Distributed Object XML Paradigms
XML Feature | EDI | Distributed Objects
Messaging Protocol | SOAP | SOAP
Framework | BizTalk or ebXML | WSDL + UDDI
Flow Language | ebXML or BPML | XLANG or WSFL


Motivation

Exposure to the many initiatives related to XML messaging often makes the topic appear complex to newcomers. However, this apparent complexity results from many layers of detailed decisions rather than the inherent difficulty of the underlying concepts. Attacking this topic at the conceptual level provides an easily comprehensible framework for examining the details of different initiatives later.

Much of the confusion surrounding XML messaging stems from the fact that it has two distinct motivations that became intertwined while the protocols were being designed. Therefore, from the individual perspective of either motivation, different initiatives appear to address superfluous issues. Separating these motivations quickly clears up much of the confusion.

The first motivation is to enable the automated exchange of XML business documents among software applications. This motivation is easy to understand. It makes little sense for a group of enterprises to agree on XML formats for catalogs, purchase orders, and invoices without a standard means to send these documents over the Internet. Certainly, they could e-mail or FTP them to each other, but processing would require human intervention. A better solution would be a standard protocol that software applications could use to exchange such documents directly. We'll refer to this motivation as business document messaging. It is the same motivation behind traditional EDI systems.

The second motivation is to have a standard means of exposing remote interfaces that other software could use to access application functions. Notice that there is nothing XML-specific about this motivation. In fact, a number of similar attempts have been made in the past, such as the OMG's CORBA and Microsoft's COM. Unfortunately, these past attempts did not result in the widespread interoperability that organizations were looking for. Because XML has achieved broad cooperation among vendors and enterprises, another attempt at achieving this desired interoperability makes sense. We'll refer to this motivation as remote interface messaging. It is the same motivation behind distributed object computing.

The protocol requirements for business document messaging and remote interface messaging have a significant amount of overlap. Both need to get XML data from one point to another over the Internet. Theoretically, you could even argue that business document messaging is actually a subset of remote interface messaging. Simply define a remote interface with one function—"process business document"—that takes a single parameter—"business document." However, this argument ignores the practicalities of how developers architect business document versus remote interface messaging applications.

The desire for common infrastructure further complicates the issue. If many applications use the same infrastructure, it reduces costs and increases robustness in the long run, even though this infrastructure might not provide an optimal feature set for any particular application. XML messaging evolved based on the belief that it would be possible to create a simple protocol that satisfied the requirements of both types of messaging. Only after the practical constraints of developing these systems emerged did it become clear that a single protocol would have to be more complicated than originally envisioned. Nevertheless, the benefits of a common protocol may still outweigh the costs.


Protocol Design Problem

To understand how XML messaging works, it helps to consider how one might go about designing a basic distributed protocol with XML. Figure 4-1 captures the general problem. An instance of a software application running on one machine wants to send some XML data to an instance of another software application running on a different machine. There are three issues. First, what information does the sending application have to wrap around the XML data to enable processing by the receiving machine? Second, what is the low-level mechanism for moving this data across the network? Third, what additional contextual information will the receiving application need to facilitate message processing?

Figure 4-1. XML Messaging Problem

Wrapper Information

At a minimum, the wrapper information must identify the target application, specify what it wants this application to do, and indicate the data this application should use in fulfilling the request. A concrete example in plain English would be "The target application is the Shipping Application at http://www.foocompany.com/Application/Shipping; the application should perform its Get Status function; and it should use the Account Number ABCDEFGHI and the Order Number 123456789." To encode such a request as XML, identifying the target application and specifying the type of request clearly requires adding new information, either as elements or attributes. Indicating the data to use for execution requires a standard method for encoding such parameters in XML elements.

Once the receiving application processes the message, it may want to send back some information. This returned information might be as simple as an acknowledgement that it has received and understood the message. It could be a single value such as "Shipped," in the case of our hypothetical "Get Status" request. It could also be a sophisticated structure such as a complete "Shipment" business document. The protocol designer must decide the level of complexity to allow in the returned information. The designer must also decide where to interpret return values of a given complexity. Simple acceptance codes may belong at the transport level. A transport may also accommodate a single string return value. Returning an entire XML document may require a complete wrapper just like the request.

Transport Binding

Once the designer has a firm plan for formatting requests and responses, he needs to select a means of transporting information between sender and receiver. Typically, people refer to the combination of the message format and a transport as a binding. Different applications may require transports with different characteristics, so a messaging protocol should support multiple transport bindings. This goal starts to make the protocol designer's task very difficult because he must take into account the low-level behavior of many transports. A binding maps the features supported by the transport protocol to the information exchanges possible with the messaging protocol.

The most commonly used transport for Internet applications is HTTP, so it is the obvious binding for an XML messaging protocol, although others are certainly possible. HTTP includes a number of methods such as GET and POST. Requests and responses can be formatted as text, typically using the MIME standard to package different chunks of information. So the binding for an XML messaging protocol to HTTP must specify how to use these methods, how to use MIME for packaging different parts of transmissions, and the meaning of various return codes.

Contextual Information

So far, the designer has enough functionality to send XML data between applications and receive a reply. But many electronic commerce applications require extended features to meet the business needs of the parties. Security is one class of such features. At the very least, applications should authenticate themselves before exchanging messages. The message wrapper may need to reflect this requirement by including the identity of the sender, which the receiver verifies through the authentication process. Often, security concerns also necessitate further work on the transport binding. If exchanges include sensitive information, they may need encryption, a feature provided by HTTPS. While identical to HTTP except for additional security measures, the transport binding must specify how to use these features.

Reliability is another class of extended features. Certain circumstances may rely on guaranteed delivery of each message. Transactions are yet another class of feature. A sending application may want to indicate that a set of messages represents a single logical unit of work that should either succeed or fail together. Satisfying the requirements for these additional features depends on the precise architecture of the application messaging, and addressing different architectures is where complications in protocol design arise.


Messaging Architectures

Obviously, you could argue that each XML messaging application has its own unique architecture. However, much of the variability in protocol features requirements comes from the high-level distinction between applications serving the two motivations for XML messaging discussed previously. While business document architectures and remote interface architectures share the basic need to exchange XML data among applications, the purpose, communication model, and resulting need for extended features differ significantly.

Business document applications naturally reflect the semantics of the business processes that they serve. Consider the process of "Purchasing." In the paper world, this process requires customer and supplier staff to assemble and exchange a certain set of documents such as "Request for Proposal," "Proposal," "Purchase Order," "Shipment Notification," and "Invoice." The exchange of these documents occurs in a well-defined pattern—an "Invoice" requires the previous exchange of a "Purchase Order" and a "Shipment Notice." The sender does not usually expect an immediate response other than the acknowledgment that the document was received. It implicitly understands that the other party may perform a significant amount of internal processing before it's ready to send the next document in the business process. People also use terms like message-oriented architectures and coarse-grained Web Services to describe this type of application.

Remote interface applications naturally reflect the semantics of the applications they encapsulate. Consider an "Order Entry" application. It offers a certain set of functions such as "Create New Order," "Set Billing Address," "Add Line Item," "Set Payment Terms," and "Check Order Status." Using these functions requires some knowledge of the application design. One "Order Entry" application may require that you "Set Payment Terms" before you "Add Line Item," while another may not. The sender expects an immediate response because it usually needs to use the value of that response in the very next request. Because each function performs a relatively small piece of work, the sender implicitly understands that it needs to make a series of related requests. People also use terms like remote procedure call (RPC) architectures and fine-grained Web Services to describe this type of application.

Basic Protocol Requirements

The different semantics of the two architectures lead to different communication models. Business document applications typically require an asynchronous model—the sender does not wait for the receiver to reply. Each document represents an action within a broader business process. The very act of sending the document completes the corresponding action. There is no need for further communication until the receiving party completes the next action. From the networking perspective, the sending party transfers a single, large block of data to the receiving party. The sending program does not wait for an immediate response before continuing on to the next task. Later, the roles may reverse with a new document flowing in the other direction. The asynchronous model works well over the Internet where latencies—delays as the message moves through switches, routers, and servers—are often too high to guarantee a timely response. Figure 4-2 shows the asynchronous communication model of the business document model architecture.

Figure 4-2. Asynchronous Communication Model

On the other hand, remote interface applications typically require a synchronous communication model—the sender waits for the receiver to respond. Remote interfaces mimic the operation of local programming language interfaces. Most programming languages structure interfaces as a set of functions, each with a set of possible request parameters and a set of possible return values. Usually, each function call makes sense only in the context of a series of function calls related to a larger goal. Providing the same structure over a remote connection requires that the sender wait in a synchronous fashion for the recipient to process the request and return an appropriate value. Even if a particular transport does not physically have to wait for the return value, the dependencies of subsequent function calls on the return value impose a semantic requirement to wait for its completion. The synchronous model works well over intranets where lower latencies enable timely response, and the close dependencies among enterprise applications make interaction at the function level desirable. Figure 4-3 shows the synchronous communication model of the remote interface architecture.

Figure 4-3. Synchronous Communication Model

Additional Protocol Requirements

The combination of semantics and communication models influence the necessary protocol features. Because a business document transmission represents the completion of an entire business action, the sending application needs absolute assurance that the receiving application gets the document. However, the asynchronous communication model means that it may require some time and perhaps even intermediate processing by the messaging system to deliver the document. So the sender has no immediate indication that the message was received. Therefore, business document applications need a protocol that is reliable. It can guarantee the delivery of documents to a specific target application.

As for security, these applications may need explicit authorization traceable to a particular staff member of appropriate authority. As a completed action, a business document essentially authorizes the receiving party to take subsequent actions based on the information contained in the document. They often require the security property of nonrepudiation—a provable commitment, equivalent to signing a physical document. However, the ability to define transaction boundaries is unnecessary. Each document represents a whole business action and is therefore implicitly a single transaction. There is rarely a need to specify that multiple messages are part of the same transaction.

Remote interface applications require almost the negative image of extended features from the messaging protocol. Because each function call represents only part of a larger operation, these applications usually need the ability to define transaction boundaries explicitly. If a set of function calls changes several pieces of related information, the sending application needs the absolute assurance that either all or none of these changes will occur.

From the perspective of security, remote interface applications rely on the mutual authentication of sender and recipient identities. There is no need for nonrepudiation of each function call. Once authenticated, the recipient simply verifies that the sender has the authorization to perform that particular function call. However, reliable message delivery is superfluous. Because the function call model requires the successful return of the call, the sender can assume that the recipient got the message if the function returns successfully. If the function takes too long to return, the sender can time out and retry the call.

Using Both Architectures

Neither architecture is inherently better than the other. They have different purposes: Business document architectures work best when the parties need to exchange well-defined work products over the Internet. Remote interface architectures work best when parties need to assemble a set of information flexibly from small pieces over an intranet. Ideally, e-commerce systems use both architectures simultaneously.

An enterprise gateway acts as both a business document application and a remote interface application. The business document half exchanges packages of business data, while the remote interface half assembles these packages through remote function calls. The business document halves of the gateways talk to each other, while the remote interface halves talk to internal applications. Figure 4-4 shows this combination of architectures.

Figure 4-4. Combining the Two Architectures


Major Web Services Initiatives

Various standard and industry initiatives have attempted to tackle the issues of messaging protocol requirements and the two different architectures. The goal of these initiatives is to provide standardization in the name of interoperability. Just as people want to ensure that other parties can understand the data in a document by using a standard XML format, they also want to exchange these documents in a business context by using standard protocols. In determining which of these initiatives is relevant to you, you must consider both the type of architecture you plan to use and the scope of functionality you require.

It would be nice if just knowing whether you will be using a business document or a remote interface architecture told you which standards to use, but, sadly, that is not the case. However, it does help. If you plan to have a remote interface architecture, you know that SOAP is relevant but that neither BizTalk nor ebXML is. You also have to determine if higher-level Web Services protocols are relevant. On the other hand, if you plan to have a business document architecture, you pretty much need to choose among BizTalk, ebXML, and approaches that leverage Web Services protocols within a proprietary business document framework.

While there are certainly philosophical and platform issues that affect this choice, there is an important functional one as well. BizTalk and ebXML have different scopes. BizTalk focuses on delivering a relatively simple document wrapper with higher-level functionality contained in the Microsoft tools. ebXML focuses on delivering a complete methodology and software stack for B2B commerce.

SOAP / W3C XML Protocol

The good news is that all major Web Services initiatives have decided to use SOAP as their low-level XML messaging protocol. There has also been a great deal of cooperation within the developer community to ensure the interoperability of different implementations. Thus XML messaging has, to a certain extent, achieved the goal of using common infrastructure. However, as you will see, these higher-level initiatives use SOAP somewhat differently.

The bad news is that SOAP made itself attractive for a wide variety of uses by deferring many important issues. SOAP provides no extended features. It does provide a mechanism for adding extended features, but such additions aren't necessarily interoperable. The thinking behind this minimalist approach is that it's more important to establish a common low-level XML messaging protocol now rather than figure out how to standardize even a relatively small set of extended features. Eventually, as people begin depending on such features, they could migrate into the SOAP standard if necessary.

Microsoft, DevelopMentor, and Userland Software collaborated on version 1.0 of the Simple Object Access Protocol. The effort rapidly expanded to include a large group of vendors, resulting in the widely used version 1.1. Then these vendors submitted the protocol to the W3C, where it is now evolving under the direction of the XML Protocol Working Group. Hopefully, this working group will issue SOAP 1.2 as a Recommendation by the time you read this.

SOAP works much like our hypothetical messaging protocol. Example 4-1 shows a simple SOAP requests-response pair for the shipping status of an order, using an HTTP binding. Example 4-1a is the request. The request comes in via an HTTP POST method that contains the SOAP message as a file. The SOAP wrapper is called an Envelope, and it consists of two parts. The Header contains information for custom-extended features, and the Body provides standardized encoding of the function call. The location of the target application is encoded in the namespace declaration. The function is indicated with a corresponding element name—in this case, "GetStatus" element. The parameters of the request are encoded with corresponding elements—in this case, "Acct" and "Order" elements.

Example 4-1a
 POST /ShippingStatus HTTP/1.1
Host: www.foocompany.com
Content-Type: text/xml; charset="utf-8"
Content-Length: nnnn
SOAPAction: "http://www.foocompany.com/SOAP/Shipping"

<SOAP:Envelope xmlns:SOAP=
 "http://schemas.xmlsoap.org/soap/envelope/"
SOAP:encodingStyle=
 "http://schemas.xmlsoap.org/soap/encoding/">
<SOAP:Body>
  <m:GetStatus xmlns:m=
   "http://www.foocompany.com/SOAP/Shipping">
   <Acct>ABCDEFGHI</Acct>
   <Order>123456789</Order>
  </m:Status>
 </SOAP:Body>
</SOAP:Envelope>

Example 4-1b is the response. The response comes embedded in the HTTP 200 Return. It uses the same wrapper as the request, but with information about the return values instead of the function call. The outermost body element indicates which function is responding—in this case, "GetStatusResponse." Its child elements contain the return types and their values—in this case, a single "Status" element with a value of "Shipped."

Example 4-1b
 HTTP/1.1 200 OK
Content-Type: text/xml; charset="utf-8"
Content-Length: nnnn

<SOAP:Envelope xmlns:SOAP=
 "http://schemas.xmlsoap.org/soap/envelope/"
SOAP:encodingStyle=
 "http://schemas.xmlsoap.org/soap/encoding/">
<SOAP:Body>
  <m:GetStatusResponse xmlns:m=
   "http://www.foocompany.com/SOAP/Shipping">
   <Status>Shipped</Status>
  </m:GetStatusResponse>
 </SOAP:Body>
</SOAP:Envelope>

There are a couple of interesting points to note about this SOAP exchange. First, the element structures of the function call and the return values are completely dependent on a particular interface. The requester must know this structure before it can construct a valid message. Usually, a programmer will need documentation of this interface and then have to write the requesting code to produce the appropriate SOAP messages. In some cases, it may be possible to use higher-level tools to map external interfaces to internal data structures, but this approach still requires someone with programming skills who understands how these pieces work.

The second interesting point is that the HTTP binding obviously depends on the HTTP interaction model. In this case, the return values are embedded in the HTTP response to the HTTP POST method containing the request. This approach results in an inherently synchronous model—the requester must wait for the HTTP response before proceeding. Of course, the response could just contain an acknowledgment of the request and later send the results in a separate SOAP message. Unfortunately, the SOAP specification does not describe how to identify individual messages. If once piece of software sends two different "GetStatus" messages to another piece of software and then a "GetStatusResponse" message comes back, SOAP has no mechanism to indicate a correpondence with a particular "GetStatus" message. So using SOAP in an asyncrhonous model requires additonal agreements between sender and recipient.

The lack of message identifiers is symptomatic of a larger issue. SOAP does not include any extended features. Developers must define custom elements and put them into the Header if they want security, reliability, or transactions. Of course, different developers may define these tags differently, leading to an interoperability problem. Example 4-2 shows a SOAP Header with security and transaction information. The encoding of this information is completely hypothetical but nevertheless valid from the perspective of the SOAP standard.

Example 4-2
 <SOAP:Envelope>
 <SOAP:Header>
  <s:Authenticate xmlns:m=
  "http://www.foocompany.com/SOAP/Security"
  mustUnderstand="1">
   <Method>SSL</Method>
   <Credential>
    <CN>www.barcorp.com</CN>
   </Credential>
  </s:Authenticate>
  <s:Transaction xmlns:m=
  "http://www.foocompany.com/SOAP/Transaction">
   <TransID
   begin="false" commit="false" rollback="false">
   123456789
   </TransID>
   <SequenceNumber>3</SequenceNumber>
  </t:Transaction>
 </SOAP:Header>
...
</SOAP:Envelope>

Example 4-2 has hypothetical encodings for two extended features: authentication and transactions. The exact elements used in these encodings are completely made up. Developers could certainly use the implied format to implement these features, but the resulting software would be incompatible with any other implementation. What is standardized is the specification that understanding the authentication encoding is mandatory, while understanding the transaction encoding is optional. When a requester wants to use an extended feature, it specifies through an attribute whether the recipient "must support" the feature. This attribute enables developers of proprietary extensions to ensure that their software interacts only with software that subscribes to the same proprietary model.

Specifying "must support" means that the recipient must return an error if it cannot support the feature. If "must support" is not specified, the recipient should use the feature if it can be supported but may continue processing if it does not. Of course, mandating support for extended features means that all the parties must either have agreements about these Header elements or use a higher-level standard. Otherwise, using the extended features simply results in many unprocessable messages.

Support for security and transactions works well with this approach because the only issue is whether the recipient knows how to act on the information contained in the Header. Business document applications, however, need guaranteed delivery. Reliability features affect the transport layer, making them somewhat problematic. For each transport binding, partners must agree on how to achieve reliability for a message. There are currently initiatives to add reliability to SOAP, but the complexity of these issues currently makes using SOAP for business document applications more difficult than for remote interface applications. The best solution is to use on top of SOAP a standard that adds the capabilities necessary to support business document applications.

BizTalk

BizTalk is the Microsoft-backed initiative for business document applications. BizTalk Framework is the open specification for how these applications should behave. BizTalk Server is the Microsoft product that implements this specification. Other vendors may offer implementations as well, but the specification does defer the precise nature of certain optional features, thereby potentially limiting the interoperability of these alternative implementations.

The BizTalk Framework defines a set of header tags that extend SOAP to meet the needs of business document applications. From a manager's perspective, the exact syntax of these tags is not particularly relevant, but the capabilities they deliver are significant. Message properties, endpoint, and service tags address a major shortcoming of SOAP by providing the information necessary for asynchronous communication. Manifest and process management tags further extend the model by allowing for non-XML attachments and automated workflow management.

As you saw in the discussion of SOAP, the lack of message identifiers severely limits the protocol's usefulness in an asynchronous model. The BizTalk Framework makes such identifiers mandatory as well as other useful information such as a timestamp, expiration, and topic. This metadata about the message gives servers the foundation to transport, route, and process all the messages in a complex business interaction properly.

Perhaps the most important feature enabled by the message property tags is the ability to specify the sender and recipient of business documents logically. The transport of the underlying SOAP binding obviously knows the physical endpoints of the message transfer, but this level of addressing merely ensures the successful movement of the message from one machine to another. It does not ensure that the application capable of handling the particular request receives it and processes it. The BizTalk endpoint tags specify the sender and recipient in terms of logical business entities, such as the "Accounts Payable Department of Foo Company." Just as a logical e-mail address enables mail servers to route an e-mail through many hops to the correct desktop, the endpoint tags enable messaging servers to route a message to the application designated to handle the message.

The message property and endpoint tags form the basis for reliable messaging. Suppose the sender wants guaranteed delivery of a given message to a particular recipient. To do its job, the underlying messaging system needs to know the logical recipient and the message identifier. The logical recipient tells the system, which has to acknowledge the message, and the message identifier makes sure a specific recipient gets a specific message. A delivery receipt serves as the tangible proof that the messaging system has done its job. To enable all this processing with the BizTalk Framework, the sender requests a delivery receipt and may optionally request a commitment receipt. The commitment receipt goes a step further by requiring the recipient to specify the date by which it will provide a response.

The primary goal of a business document application is to facilitate the flow of these documents among trading partners. Realistically, a business document may have a set of corresponding supporting documents. Consider the case of a purchase order issued by a semiconductor design firm to an overseas fabrication plant. A number of additional documents such as CAD drawings and process notes would accompany the purchase order to form a precise definition of the expected service. BizTalk accommodates this requirement using a manifest that catalogs all attachments. Each manifest entry provides descriptive information about the attachment and a reference to its location in a larger multipart MIME package that contains the business document and its attachments, similar to the packaging of e-mail attachments.

Another reality of business document applications is that a given business document rarely exists in isolation. It is usually just part of a larger set of interactions among trading partners. Remember the series of document exchanges necessary to complete the "Purchasing" process discussed earlier. An emerging class of products enables the automated processing of such multistep processes based on a shared definition of the process. The BizTalk Framework accommodates such software by providing a means to specify a reference to a process definition. But the format of this process definition is not standardized. The BizTalk Server product uses this mechanism to leverage its built-in workflow capabilities, based on its own process definition format called XLANG.

While the BizTalk tags provide the information necessary to exchange business documents among trading partners, such exchanges naturally require security. BizTalk messages may use a variety of underlying transports, so the BizTalk Framework defers the issue of securing the communications channel to these transports. It does not, however, defer the issue of securing the messages themselves. Because it uses the standard MIME packaging for messages, it uses the S/MIME standard for securing them. This standard covers the encryption and signing of messages and is already in widespread use for applications such as securing e-mail. Note that S/MIME does rely on each party having some degree of public key infrastructure to take advantage of its most useful features.

While Microsoft's BizTalk Server was the only commercial product available that implements the BizTalk Framework at the time of this writing, there were other implementations under development. A number of custom programs demonstrated the ease of extending a SOAP implementation to create a server that could process BizTalk messages. The BizTalk Framework addresses the basic issues of asynchronous, reliable, and secure business document exchange, but it ignores the condition that trading partners should agree on common business processes to execute with this document exchange. Even simple items like the precise format for logical addressing of sender and recipient are currently not part of the standard. Of course, the Microsoft product suite provides a number of fine tools for dealing with these concerns.

ebXML

In contrast to BizTalk, ebXML deals with the function of establishing automated business processing among trading partners. It specifies a much grander architecture that starts with business process modeling and proceeds to the negotiation of shared business processes. This initiative is proceeding under the auspices of the Organization for the Advancement of Structured Information Standards (OASIS) and UN/CEFACT, an international body with a great deal of experience in facilitating electronic business.

Of course, ebXML does need the basic plumbing to move XML business documents from one point to another. ebXML takes a very similar approach to BizTalk in providing this structure by extending SOAP with its own header tags. These tags address the same issues, such as message identity, logical addressing, and reliable delivery. There are some other additions, such as enhanced error reporting and status checking, but these are not the primary differentiators. Rather, ebXML offers an added set of conventions on top of the basic plumbing.

ebXML looks at business document messaging as part of a larger process of negotiating automated business processes. This perspective comes from its roots in traditional EDI. The experience of EDI is that enterprises typically need to fit electronic business methods mostly within their existing business processes. For two trading partners to automate their interaction, they must arrive at a common process that meets these constraints. Each partner must discover how the other is willing to do business and tailor its electronic interactions to accommodate these practices.

Obviously, this process of discovery goes beyond simply inspecting programming interfaces. While not strictly required, ebXML emphasizes the use of business modeling and accompanying documentation to educate potential partners. When using a modeling language, the specification mandates the use of the Unified Modeling Language (UML).

ebXML has a framework that defines the types of modeling constructs that enterprise should define. An enterprise may adopt a Role within a Business Process. So "Manufacturing" would have "Designers," "Raw Materials Suppliers," and "Factories." A set of trading partner interactions to accomplish a set of interdependent Business Processes is a Business Collaboration. So a group of trading partners could link the "Manufacturing," "Shipping," and "Distribution" Business Processes as part of the larger "Deliver Goods to Market" Business Collaboration.

A particular Business Collaboration consists of multiple Business Transactions. Each Business Transaction defines a pattern of Message exchanges that advance the Business Collaboration by a logical unit of progress. The hypothetical "Deliver Goods to Market" Business Collaboration might include such Business Transactions as "Authorize Manufacturing," "Update Production Schedule," and "Select Shipping Carrier." The "Authorize Manufacturing" Business Transaction could require "Specification," "Quote," and "Purchase Order" messages. Each enterprise may then specify the format it supports for the messages it must exchange within the Business Collaboration. Figure 4-5 summarizes the hypothetical model.

Figure 4-5. Abbreviated ebXML Business Model

Obviously, this model-driven approach to electronic business processes generates a substantial number of documents. Enterprises signal their support for various Roles within different Business Processes by submitting documents to an ebXML registry, where they become available to potential trading partners. Each enterprise maintains control over the conditions under which others may access its information. This registry is part of a larger repository for all human and machine information generated during trading partner collaborations.

Within the registry, each enterprise maintains a Collaboration Protocol Profile (CPP). A CPP describes the pattern of interactions an enterprise will use when fulfilling each of its possible Roles within each of its supported Business Processes. A CPP includes the technical interfaces, messaging, and security capabilities supported by the enterprise. When two partners have opposing Roles within the same Business Process, their interaction patterns are compatible for this Business Process. If they have a common subset of technical capabilities, they can automate this business process. They create a Collaboration Protocol Agreement (CPA) that governs this automated process. The CPA references a Business Process Schema Specification (BPSS) that provides an XML description of the workflow rules used to execute the business process. All CPPs, CPAs, and BPSSs reside in the registry.

Reaching agreement on CPPs would clearly be easier if trading partners shared a basic vocabulary of business concepts. To this end, ebXML includes a set of common Process Templates and Core Components. The Process Templates cover interactions in common Business Processes such as finance, product development, product manufacturing, transportation and logistics, and product support. The Core Components include XML Schema definitions for elements that define entities like names, addresses, accounts, product descriptions, and prices. Together, they greatly reduce the modeling and negotiation burden for enterprises that use them. Moreover, by mining the repository for common patterns, industry groups can further expand these templates and components, making the process of implementing ebXML systems easier and easier as adoption grows. Figure 4-6 shows the relationships among the different ebXML components.

Figure 4-6. ebXML Components

ebXML is a very inclusive initiative. Large enterprises can have sophisticated and expensive ebXML gateways that integrate with their backend systems to achieve massive efficiency improvements. But the potential advantages for large enterprises don't preclude the participation of small ones. Because the entire system is designed for human readability, a one-person company could easily work with the largest corporations by manually processing ebXML documents accepted through e-mail.

For those enterprises that want a lighterweight business document messaging model, OASIS has another group working on the Universal Business Language (UBL). The goal of this group is to define a standard set of formats for business documents, such as "Purchase Order," "Shipping Schedule," and "Commercial Invoice." These formats will be compatible with the ebXML Core Components but usable without the rest of the ebXML model. Many enterprises do not initially require all of ebXML's sophistication, but it would be nice if the business document formats they used were compatible with ebXML. It would also be nice if these enterprises could easily upgrade to the full ebXML model should the need arise. Moreover, there are a number of competing business document standards, including commerce eXtensible Markup Language (cXML), RosettaNet, and XML Common Business Library (xCBL). These standards are all somewhat similar because each took the same approach of trying to reformulate preexisting EDI formats as XML, but the existence of multiple options has somewhat fragmented compatibility. Unifying them with UBL will hopefully lead to more universal interoperability.


Other Web Services Initiatives

SOAP, BizTalk, and ebXML are all fairly well-established initiatives, but the emergence of the Web Services movement has inspired a host of new ones. The Web Services label originally applied to the use of three specific protocols: SOAP; Web Services Description Language (WSDL); and Universal Description, Discovery, and Integration (UDDI). Together, they formed a stack very similar to that found in distributed object computing. Both have a wire protocol, an interface description language, and a discovery mechanism. This similarity naturally made Web Services as originally conceived better suited to remote interface architectures. However, as technologists and developers considered how to connect applications over the Internet, the Web Services movement came to embrace both remote interface and business messaging architectures.

Basic Web Services Stack

Initially, the goal of the basic Web Services stack was to lower the barrier to connecting applications using the Internet. Typically, you would decide to connect at the application level rather than the business process level for two reasons. In the first case, you might be attempting to integrate several enterprise applications to support a particular business goal. Bringing together accounting, manufacturing, and shipping applications for automated order processing is one example.

In the second case, you might be trying to assemble a complete Web application from a group of more modular services. Using third-party express shipping price and local sales tax calculators to improve an online shopping application is one example. Developers have a great deal of experience from the world of distributed objects that reveals where the difficulties in these tasks lie. By using XML at ever-higher levels in the software stack, they hope to minimize these barriers.

There is a common misconception that Web Services can eliminate the custom programming associated with application integration. In most cases, this is simply not possible, but Web Services can greatly reduce the amount of programming. The first point of leverage comes in providing a common communications model. Using only one model to program increases productivity. Moreover, once developers can assume they have a common model, they can invest in more sophisticated tools to support this model.

As discussed in Chapter 5, most of the major application server vendors include utilities for automatically publishing application components as Web Services and for generating the client code to access them. Some also provide graphical modeling and lightweight scripting environments for assembling multiple Web Services into a higher-level application. Even with these features, the realities of software integration mean that most applications will still require some amount of traditional programming.

The protocols in the basic stack are very straightforward. As with the other initiatives covered so far, we use SOAP as the standardized plumbing. Unfortunately, as we saw in the SOAP example, using SOAP in a remote interface application requires knowledge of the target application. Developers need to use the correct tags to format the function call and know how to interpret the tags from the return value.

WSDL provides a standardized way for an application to publish its SOAP-accessible functions. It serves the same purpose as an interface definition language for a distributed object system. While the particular syntax is complicated, the basic concepts are relatively standard. A coherent Web Service in WSDL is called a portType and has a group of related functions called operations. So a "Foo Company Shipping" port type may have "New Shipment," "Change Shipment," and "Status" operations. Each operation has input and output messages, whose formats are defined using XML Schema.

The set of operations for a WSDL port type define the logical interface for the service. WSDL also provides for the definition of the communication interfaces. It uses the concept of a binding to represent a potential communications interface, such as a specific transport and a particular URL. A given Web Service may support different bindings for the same port type. This separation gives a Web Service provider a great deal of flexibility, such as offering different levels of service for different users. The free version of a Web Service accessed through one URL would have a higher quality of service compared to the paid version of the service accessed through a different URL.

Of course, even if a provider goes to the trouble of creating a WSDL description of its Web Service, potential users have to find this service. The whole purpose of WSDL is so a user can access a service without a lengthy communication with the provider. But this brings up the likely possibility that the user won't know about every Web Service that could fulfill his needs. The UDDI protocol fills this gap. UDDI is essentially a directory for Web Services. Providers can submit descriptions of their services and accompanying interface descriptions. Users can search for services that deliver what they require. Ariba, IBM, and Microsoft are working together to offer top-level public directories based on UDDI. Other organizations may offer directories specialized by topic or industry. An enterprise would probably have its own private UDDI directory if it used Web Services for application integration. Figure 4-7 shows how a sequence of operations using the basic protocol stack helps establish distributed communication.

Figure 4-7. Web Services Protocols in Action

The goal of this Web Services architecture is to lower the cost of interoperability. Compared to past distributed object systems, Web Services offers the advantages of true interoperability among implementations from different vendors. Like such systems, it provides for interface repositories and interface definitions so that developers can quickly access the information necessary to write software that accesses a particular service. However, capitalizing on these benefits requires widespread adoption that is far from certain. The biggest challenge is that Web Services do not allow us to do anything that we couldn't do before. It simply makes it somewhat easier.

Expanding the Protocol Stack

The basic Web Services stack best serves remote interface applications. Discovering, understanding, and programming to remote interfaces are the primary problems, but adding WSDL and UDDI to SOAP is a solution. Security remains unaddressed, but transport level strategies—such as binding SOAP to HTTPS—provide a quick fix. Transactions also present an issue, but careful structuring of interfaces can avoid serious problems in most cases. While the basic stack has enough functionality to support remote interface architectures, this is not the case for business document architectures. To support business documents from a practical standpoint as well as a marketing standpoint, the Web Services stack needs to address the issues of extended features that SOAP deferred.

Because business document applications absolutely require reliable messaging, reliability is the biggest gap. As we saw for both BizTalk and ebXML, it is possible to plug this gap with application-specific SOAP header tags. Unfortunately, this approach leads to incompatibilities and duplication of effort among the different applications. Adding reliability to the standard Web Services stack solves the problem more tastefully. There are two approaches: Add reliability to specific transports, and add reliability specifications to SOAP. IBM has a proposal for HTTP-R that adds reliability-related headers to the HTTP transport. Special HTTP-R capable servers can interpret these headers to provide guaranteed delivery and delivery receipts. Adding reliability to a specific transport has the disadvantage of limiting the scope of reliable messaging to that transport. Microsoft has a proposal for WS-Routing that adds tags to SOAP and that describe the reliability requirements of the message. While different transports can fulfill these requirements, it does open the door for different implementations to interpret requirements differently. Most likely, a combination of these approaches—SOAP-level specifications and enhancements to standard transports—will fill this gap in the long term.

In addition to reliability, there are a number of efforts to add security, transaction, and inspection features to facilitate the execution of both remote interface and business document applications. For the most part, there are no major technical hurdles; it is simply a matter of agreeing on a common model. More interestingly, there is a trend toward trying to replace existing Internet protocols with those better optimized for Web Services. A good example is Direct Internet Message Encapsulation (DIME). This protocol replaces MIME for packaging information in a message. While perfectly possible to perform XML messaging with MIME, it does have a few drawbacks that DIME attempts to address. However, it is an open question as to whether the relatively minor benefits outweigh the cost of replacing a very well-understood and mature protocol with a brand new one. There are even some discussions of replacing HTTP itself. You should be very cautious of such initiatives. The Internet and Web have evolved over time, and shifting fundamental pieces of the infrastructure could have unintended consequences. You should consider using such technology only after satisfying yourself that the benefits are compelling and testing has been rigorous.

Clearly, the proliferation of Web Services protocols may lead to both confusion and incompatibility. Recently, a number of companies founded the Web Services Interoperability Organization (WS-I). These founders included Accenture, BEA, Hewlett-Packard, IBM, Intel, Microsoft, and SAP. As major providers of Web Services products and services, they have pledged to work together to promote compatibility. They plan to promote a common set of core Web Services principles to educate developers and to conduct interoperability tests to ensure their offerings work together smoothly. Hopefully, this cooperative effort can maintain momentum for standards in XML messaging.

Message Flow Languages

There is a potentially strong motivator for Web Services adoption on the horizon. One of the remaining points of intensive labor in using Web Services is the need for programmers to connect services together by writing glue code. There are a number of initiatives in their early stages that aim at specifying yet another level of XML-based information to supplant the need for such programming. Often known as orchestration technologies, these initiatives are attempting to define the glue as XML rather than as traditional programming code.

This concept is not new. Since the invention of distributed components, there has been a lot of talk about wiring applications together. Most attempts to achieve this goal have failed. But the XML-based approach may have a chance because Web Services blur the distinction between behavior and data. In traditional component systems, the actual execution of remote functions is clearly defined as behavior. However, Web Services make this execution a matter of sending the correctly formatted text message, which looks a lot more like data. The advantage is that data is easily transformed.

Most wiring paradigms work perfectly well as long as the output of one remote function matches the requirements for inputs to the next function. Unfortunately, the world is not usually this well ordered. However, because XML is so data oriented and has a companion data transformation language in XSLT, it may be possible to deal with mismatches without dropping into a traditional programming language. Simply apply an XSLT transform on the output of one function to get the input to the next function. Of course, if the difference between output and input are dramatic, the amount of XSLT necessary for the transformation will look more like a program in its own right than a simple scripting fragment.

Another benefit of the XML approach to wiring components is that the description of component assembly is portable. A disadvantage of previous approaches was that the result of wiring components together was either a proprietary file or generated programming code. These approaches limited the portability of the wiring information. With XML-based descriptions of the wiring, they could easily flow from one tool to another and from one vendor's Web Services implementation to another. Of course, the downside is that the server software that interprets these descriptions must be extremely sophisticated.

There are currently two major competing initiatives for wiring together Web Services built using the remote interface architecture: Web Services Flow Language (WSFL) from IBM and XLANG from Microsoft. Both take a very similar approach: They define an assembly of Web Services as a set of possible states. A given state may send or receive messages. Once a message is sent or received, the system moves to another state. They each have elements for specifying the rules by which these state transitions occur and operations for transforming data as it makes these transitions. These descriptions are much too complicated to create by hand, so they rely on flowcharting tools as a graphical interface.

Both WSFL and XLANG are really intended to work with remote interface applications. The problem is that these applications require many operations on small chunks of data. So graphically expressing the message orchestration for an application of any complexity may require so many steps as to erode its advantage over hand coding.

Business document applications are another matter. They require fewer operations on much bigger chunks of data. There are fewer steps to model, and the advantage of transformations is larger.

A big objection to using orchestration for remote interface applications is that this technique really offers little advantage over the existing technique of custom programming. Orchestrating a remote interface application already requires programmer-level understanding of how distributed systems work. A good programmer can create custom programs almost as fast as orchestration diagrams and achieve more flexibility. Moreover, two organizations are unlikely to use the same set of Web services in the same way, so portability of the assembly description is not that much of an advantage. The situation is different for business document applications. Business analysts already use business process modeling tools to communicate within and across enterprises. They are comfortable treating business processes at the abstract level, so they may be able to specify most of a business document application's behavior in a graphical manner.

The challenge is keeping the business document flow language simple enough that it doesn't require programming expertise to create but expressive enough to cover a large variety of business processes. One potential approach to resolving this conflict would be to have several task-specific flow languages rather than one very general one. So one flow language might apply to transportation logistics, while another would apply to insurance claims processing, while yet another would apply to telecommunications service provisioning. The author has had some success with this approach in manufacturing supply chains.

At the time of this writing, the Business Process Management Language (BPML) was the only independent initiative working on a business document flow language, and it's in the early stages. This initiative does suffer the drawback of attempting to define a completely general-purpose flow language. This language is too complicated for business analysts, but it may serve as the foundation for defining more manageable task-specific vocabularies. There is also work occurring within ebXML to make BPSSs into practical and portable machine executable specifications. Hopefully, the CPA and associated BPSS negotiated between two parties will be directly executable by software from multiple vendors in the near future. Furthermore, if enterprises begin demanding business document orchestration, then other public and proprietary solutions will undoubtedly emerge.


Chapter 5. XML Software Infrastructure

    Executive Summary

    Fundamental Components

    Storage Systems

    Server Infrastructure

    Data-Oriented Components

    Content-Oriented Components

    Design Tools

    Infrastructure Selection Strategy


Executive Summary

The previous chapters examined the theory behind the XML paradigm. But realizing practical benefits requires successfully deploying XML-based systems. As with any software technology, the ability to leverage off-the-shelf infrastructure greatly improves your chances of successful deployment. Luckily, the XML paradigm's high degree of standardization and openness has resulted in rich layers of software infrastructure for both data-oriented and content-oriented applications.

This chapter focuses on the basic infrastructure used in many XML projects. This core functionality supports the rest of the application. Of course, not every application needs every piece of infrastructure, and there are often choices within an infrastructure category. Understanding when different components are necessary and the issues in choosing a particular component will help you ensure that your projects have a strong foundation for success. Figure 5-1 presents a conceptual model of the basic infrastructure.

Figure 5-1. Conceptual Model of XML Software Infrastructure

As you can see, fundamental components provide the basic services for the rest of the categories. Fundamental components, such as XML processors that implement the XML specification and XSLT processors that implement the XSLT specification, are already widely available from open source projects and software vendors. They have had plenty of time to mature and provide a robust infrastructure foundation.

Because XML is about data and content, these fundamental components implicitly assume they have a means of persistent storage. A simple application may use the local filesystem, but one that is more sophisticated requires mechanisms with greater reliability, scalability, and flexibility. There is no single optimal storage solution. The appropriate choice of database management system, content management system, or native XML store depends on the intended purpose and access pattern of your information.

Enterprise and Web architectures depend on various types of servers to manage data, execute behavior, and distribute content. For developers and authors to move XML documents around in this environment, servers must be at least XML-aware and may need XML-specific functions. Data servers must provide interfaces for accessing data in an XML format. Application servers must deliver scaleable execution of fundamental components. Content servers must facilitate the delivery, distribution, and cataloging of XML-packaged content.

Above the server layer, infrastructure clearly diverges along two paths: the data path and the content path. The data path includes XML components related to the generation, processing, and movement of machine-readable XML data. Development tools enable programmers rapidly to create software that moves data between internal data structures and XML documents. Transformation tools enable programmers to convert data between XML formats or between XML formats and other data sources. Web services components help programmers package, process, and interpret XML-encoded messages.

The content path includes components releated to the authoring, presentation, and distribution of human-readable XML content. Authoring tools facilitate the creation and editing of XML documents. Layout tools facilitate the presentation of XML documents in a variety of media. Content management components facilitate collaboration, packaging, and delivery of XML documents.

The two paths of XML infrastructure merge again with design tools. The process of information design is what sets XML apart from other software technologies. This process provides the programmer or author the means to design formats for XML, whether as data or as content. In fact, it is the information design process that determines whether your XML documents are data or content. While this process should actually occur very early in application development, the results drive the choice of other components, which is why design tools appear as the top layer in Figure 5-1.


Fundamental Components

XML defines a standard for encoding documents to facilitate seamless information exchange. Obviously, for such exchanges actually to take place, all parties must be able to process XML documents. If they use related feature standards, such as XSLT or Schema, they must be able to handle those formats. Applications need these basic processing capabilities to realize the benefits of the XML paradigm. The software engines that perform this essential work are the fundamental components.

The biggest workhorse in the XML paradigm is, of course, the XML processor. This piece of software can read an XML-formatted file and provide the encoded data to other software components. Most XML processors also understand DTDs, Schema, and Namespaces. Many can process XPath statements. Other fundamental components, such as those for processing XSLT and XLink, rely on the XML processor because the standards they implement use XML syntax. Therefore, a basic understanding of XML processors is essential to understanding the architecture of XML-based applications.

An XML processor is typically embedded in an application. It reads the physical files associated with a document and converts the document text into programming constructs accessible to the application logic. There are two basic types of processors: tree-based and event-based.

Figure 5-2 shows the operation of a tree-based processor. It accepts an XML document as input and then parses the document to create a hierarchical data structure that is an in-memory representation of the data contained in the document. Optionally, it may also accept an XML DTD or Schema, in which case it verifies that the document follows the specified rules.

Figure 5-2. Tree-based XML Processor

Application logic accesses the hierarchical data structure through an application programming interface (API) provided by the processor. The Document Object Model (DOM) is another W3C Recommendation that defines the characteristics of the document tree and the API for manipulating it. A tree-based processor does not have to support DOM to support XML; it may have a proprietary type of data tree and API. However, DOM support is a good idea because it ensures that developers can learn this one API and use any compliant processor. DOM and proprietary APIs allow developers to write application logic that moves through the document's tree of data, extracting and evaluating the information required to execute application functions. In addition to DOM or other tree API, many tree-based processors also support the use of XPath expression to access nodes within the data hierarchy as described in Chapter 3.

This process can also work in reverse. An application can use the processor's API to create a new tree, add nodes to the tree, and fill them with data. The processor uses this newly created tree of data to create a corresponding XML document. In this case, the application logic and processor work together as a document generator. Optionally, it can check to make sure that the generated document conforms to a specified DTD or Schema.

Tree-based processors are highly effective for applications that need random access to document elements. The application can use the API to navigate the path to any piece of data. However, building a complete tree for every document can consume a substantial amount of memory. Event-based processors avoid this cost because they do not create an in-memory data structure for the entire document.

Instead of creating a data structure and letting the application logic access it, event-based processors send just event descriptions to the application logic. These events include the beginning and end of each element but not its attributes or text content. If the application logic is interested in that particular element, it requests additional data from the processor. This approach is highly effective when an application accesses each element sequentially or needs access only to a known subset of elements. Figure 5-3 shows how this process works. As with tree-based processors, event-based processors can also take an optional XML DTD or Schema against which it validates the input document.

Figure 5-3. Event-based XML Processor

The Simple API for XML (SAX) is a commonly used event API for XML processors. Example 5-1 illustrates the types of events a SAX-based processor would generate when processing the order document in Example 2-6. As you can see, the processor does not include any element content or attribute information. When the application logic receives an event signifying the beginning of an element whose content it wants to access, it must specifically request information about that element.

Example 5-1
 Start document
 Start element: Addresses
  Start element: Address
   Start element: FirstName
   End element: FirstName
   Start element: LastName
   End element: LastName
   ...
  End element: Address
  Start element: Address
   ...
  End element: Address
 End element: Addresses
 Start element: LineItems
  Start element: LineItem
   Start element: Product
   End element: Product
   Start element: Quantity
   End element: Quantity
   ...
  End element: LineItem
  Start element: LineItem
   ...
  End element: LineItem
 End element: LineItems
 ...
End Document

In practice, many XML processors support both tree-based and event-based APIs. This support stems from the fact that it's easy to build a tree-based processor on top of an event-based processor. To build the in-memory data tree, the tree-based processor acts like any other application from the perspective of the event-based processor. The only difference is that it always requests all the information about an element. It then takes this information and puts it in the data tree. Figure 5-4 shows how this process works. Developers that want a tree-based API enable the tree assembly functions of the processor, while developers that want an event-based API suppress them. The advantage of this dual architecture is that both types of developers can use the same component.

Figure 5-4. Dual Mode XMl Parser

Figures 5-2, 5-3, and 5-4 show processors accepting a DTD or Schema as input along with the XML document. In such cases, the processor validates the document against the rules specified in the DTD or Schema. Such processors are called validating processors. Most of the major validating processors now support Schema as well as DTDs. However, applications do not have to use the validating features of such processors. There are also nonvalidating processors designed for applications that require only well-formed documents.

Both open source projects and software vendors provide XML processors. The Apache Software Foundation provides the popular Xerces open source processor for both Java and C++, although the Java version is usually somewhat ahead of the C++ version in terms of features. Microsoft provides an XML processor as part of its operating systems and makes this component easily accessible from its development tools. Oracle has C, C++, and PL/SQL processors for use with its database and middleware products. There are open source processors for many other languages, including JavaScript, Perl, and Python.

After XML processors, the next most important fundamental component is the XSLT processor. XSLT has rapidly achieved widespread acceptance and is an important technology for many data- and content-oriented applications. The Apache Software Foundation provides an XSLT processor called Xalan in both Java and C++ flavors. Microsoft and Oracle also provide XSLT companions to their XML processors. There are other less-known open source and commercial XSLT processors, but the complexity of XSLT has limited their proliferation.

The availability of XSLFO formatters and XLink processors is not as great. These feature standards are just beginning to see substantial use, and their areas of application lend themselves less well to distribution as components. XSLFO is relevant primarily as part of publishing systems, while XLink is primarily relevant as part of hypertext systems. This is not to say that corresponding fundamental components are not available, only that their number and quality are less than for XML and XSLT. At the time of this writing, there were a few open source XSLFO formatters, notably FOP from the Apache Software Foundation and PassiveTeX from the Text Encoding Initiative. Commercially, Antenna House's XSL Formatter and RenderX's XEP Rendering Engine were available. All tend to focus on PDF output. For XLink, empolis's X2X and Fujitsu XLink Processor (XLiP) both handle extended links but require fees for commercial use.


Storage Systems

Once you have acquired the necessary fundamental components, you can create, access, and manipulate data in XML documents. But as with all data, you usually want some form of persistent storage that's more robust than the local filesystem. It is not uncommon for projects using XML to stall while figuring out how to address the storage issue. The confusion stems from the fact that there are three vastly different choices: a database management system (DBMS), a content management system (CMS), or a native XML store. The appropriate choice depends on the characteristics of your XML data.

What if you use XML as a data interchange format? In this case, a source application encodes data from its own native format as XML, and a target application decodes the XML data into its own native format. XML is an intermediate data representation. Both the source and target applications already have persistent storage mechanisms, almost certainly DBMSs of one sort or another. There is really no need to store the XML documents persistently themselves, except perhaps for logging purposes.

In fact, the entire purpose of the interchange format is to combine data from an external source with the rest of the data in the DBMS. If you want to access or search data from these interchange documents along with data already in the DBMS, you need to convert it from XML to the DBMS's native format. You may take this approach even further by making XML the lingua franca among different data sources. The discussion of Data Servers in the Server Infrastructure section that follows addresses this option. But even in this sophisticated case, XML remains an intermediate data format. The data is ultimately translated and stored in an existing DBMS.

What if you use XML as a content format? In this case, authoring tools generate content as XML, and layout tools generate stylesheets for displaying this content. But content production usually requires higher-level features beyond storage, such as collaborative authoring, rendering to different media, and indexing documents. The following discussion of Content Provisioning Components in the Content Components section examines the need for these higher-level features in more detail. Moreover, you may also have in other formats content that you must manage alongside the XML documents.

In this case, you probably want to use a CMS that addresses persistent storage in conjunction with these other needs. Because most commercial CMSs evolved with the use of SGML, vendors have found it fairly easy to add excellent XML support. So if XML is a content and layout representation, use a CMS. CMS products with XML capabilities include BroadVision Publishing Center, Chrystal's Astoria, Documentum4i, Interwoven TeamSite, OmniMark Technologies' OmniMark, Red Bridge Interactive's DynaBase, SiberLogic's SiberSafe, and Vignette's Content Suite. The discussion of Content Management Components in the Content-Related Components section that follows covers the features of CMSs in more depth.

What if you use XML as an operational data format? Operational data is data that directly drives an application or process. Usually, DBMSs maintain operational data, but there are two cases where XML is likely to be the format. In the first case, XML is the format for an important work product of some kind. As discussed in Chapter 4, the business document architecture uses XML in precisely this manner. Each document represents a completed work product exchanged between organizations. Certainly, organizations will break down this document and map certain portions to corresponding DBMSs. However, the XML document is the starting point for driving this downstream processing and the ultimate point of reference for auditing. In the second case, XML is the format for instructions used in executing a process. Chapter 4 also discussed the emerging class of orchestration applications that use an XML format to describe the assembly of software components or the workflow for business processes.

In either of these situations, neither a traditional DBMS or a CMS is appropriate. You need to use the XML document as a single unit but still index its internal contents. Traditional DBMSs do this poorly because they either have to disassemble the document into their internal formats or create special functions for treating documents as Large Objects (LOBs). Traditional CMSs do this poorly because they are not optimized for subsecond response under high request loads. So if XML is an operational data representation, use a native XML store. Such products include Ipedo XML Database, IXIASOFT's TEXTML Server, NeoCore XMS, Software AG's Tamino, and XYZFind Server. The products mentioned in the Data Server discussion of the Server Infrastructure section can also store native XML data and are particularly useful when your application has a combination of native XML and traditional DBMS data.


Server Infrastructure

With an appropriate place to store XML data persistently, the next concern is distributing and manipulating this data. In modern Web application architectures, servers play critical roles in assembling, processing, and distributing information. Adding XML support to your server infrastructure mostly involves making sure that existing servers are XML-enabled, with perhaps the installation of a few XML-specific components. Most important, you must verify that XML capabilities meet the scalability and reliability demands of all server functions.

In general, there are three types of server components: data servers, application servers, and content servers. Data servers access, aggregate, and format data. Application servers execute business logic components and mediate distributed business processing. Content servers facilitate the acquisition of content, enhance its accessibility to users, and apply formatting. Figure 5-5 shows these different types of servers working together in a typical Web application environment. This type of server web provides the conduit for propagating XML documents within an enterprise and throughout the Internet.

Figure 5-5. Types of Server Components

While Figure 5-5 shows each server component as a distinct node, this arrangement isn't necessary. Server software may combine these components in different ways, and, in fact, different combinations lead to distinct product segments. Integration servers combine data server functions to aggregate information from multiple sources with application server functions to control the flow of business processes. Portal servers combine data server functions to access information from multiple sources with content server functions to filter this information based on user requirements. Personalization servers combine application server functions to calculate user needs with content server functions to customize their experiences dynamically. By understanding the roles of the three basic types of server functionality, you can evaluate whether such a combination suits your needs.

Data Servers

DBMSs inherently constrain the use of data. They have to choose a particular paradigm, such as relational or object. Relational DBMSs with normalized tables optimize the combination data in different ways. Object DBMSs with associated instances optimize the traversal of information webs. Within a given paradigm, each individual database has a particular structure limiting the types of information it can store and the access patterns it supports. DBMSs do a wonderful job of managing data when a given database must support only a few types of applications and when each application relies on only a few databases. However, when a given database must support a wide variety of application types or a given application must rely on many different databases, satisfying these demands often tax DBMSs to their limits. In such cases, an XML-enabled data server can improve flexibility and performance.

XML broadens the use of data. The ability to design special purpose data formats quickly encourages the combination of information managed in different databases. So while data servers have existed for some time, XML's emergence as a solution to information exchange problems has elevated their role. Data servers perform three major functions: (1) they unify the data access interface to simplify application development, (2) they aggregate data from different sources to deliver customized packages of information, and (3) they consolidate requests to DBMSs to improve performance. XML requires special support only in the first two functions. Because optimizing performance through consolidation strategies like data caching and connection pooling occurs internally to the data server, the use of XML as the format does not affect this function.

An XML-enabled data server supports XML as the unified data access format. When an application submits a request to the data server, the data server fulfills it with an XML document. Given the rise of XML messaging discussed in Chapter 4, the data server should probably support this interaction over SOAP, using an interface specified in WSDL. Merely retrieving ad hoc bits of data as XML documents that the application then has to translate into programming data structures doesn't add much benefit. Programmatic solutions such as ODBC and JDBC already satisfy this need. The more substantial benefit comes from defining synthetic XML documents that form customized packages of data suited to a particular purpose.

To deliver a synthetic XML document, the data server must have a mapping between the document type and the structures managed by backend DBMSs. A developer defines an XML DTD or Schema for the document type and then maps fields in the various database schemas to element and attribute types. The developer also defines the keys used to select the correct records for populating a document instance. At runtime, an application submits a request for a synthetic document type and the appropriate keys. The data server then looks up the mapping, constructs queries based on the mapping and the keys, and puts the results into an XML document. This results document is valid with respect to the specified DTD or Schema.

In some cases, a DBMS vendor may include some data server capabilities with its DBMS product. For instance, Oracle9i includes XML mapping capabilities. In cases where the need for a data sever stems from a small set of homogenous databases attempting to serve many different applications, this solution is sufficient. But when the need for a data server stems from a set of applications attempting to aggregate data across heterogeneous databases, you probably need a separate data server product.

Such products include eXcelon's eXtensible Information Server and Versant enJin, both of which are based on object persistence engines. Data servers require many of the capabilities of backend databases to provide high availability and transactional integrity. They use their own persistence engine as a staging area between applications and backend DBMSs. Therefore, most of the native XML store products discussed previously can also operate as XML data servers by adding features for synchronizing with backend databases. In fact, many vendors of these products are finding that this approach drives a substantial percentage of their sales. Conversely, data server products like eXcelon and enJin can operate as native XML stores, so distinctions between the two markets are blurring. When evaluating either type of product's suitability as a data server, focus on the facilities for mapping backend data to XML documents and the efficiency of performance optimization strategies like caching and pooling.

Application Servers

Application servers operate in the middle tier, applying business logic to data, then handing off the results for presentation. In this capacity, they have three primary reasons for working with XML documents.

  1. They may need to accept data as XML documents from data servers.

  2. They may need to provide business results as XML documents to content servers.

  3. They may have to exchange XML-formatted business messages with other application servers.

To support these operations, the application server can supply basic and advanced services.

Basic services include the execution of XML and XSLT processors, as well as a SOAP implementation. Whether it extracts data from XML documents, exchanges XML business documents, or produces XML business results, the application server needs the access and creation capabilities of an XML processor. Because many developers use XSLT for pre- and postdocument processing, support for this standard should be part of the basic package. Interaction with XML-enabled data, application, and content servers almost certainly includes SOAP communication, so an implementation of the protocol is essential.

Theoretically, because an application server can execute any code in a language it supports, providing basic services is simply a matter of downloading XML and XLST processors plus a SOAP implementation, then installing them. Practically, assuring the performance and quality of execution requires the vendor at the very least to certify components for use with the application server and probably include the recommended packages in the product distribution. You want to make sure that the vendor has tested the particular components, can provide estimates of how much throughput these components can handle, and knows how to support their use with its application server. For J2EE application servers, most vendors recommend the Xerces XML processor, the Xalan XSLT processor, and either their own or a particular third-party SOAP implementation. Microsoft has its own XML processor, XSLT processor, and SOAP implementation for its application server products.

Advanced services tend to vary significantly across application servers and evolve rapidly over time. Therefore, it's more appropriate to focus on the categories of advanced services rather than particular instances. Most advanced services are delivered in the form of frameworks. There are abstraction frameworks and task frameworks. Abstraction frameworks give developers more flexibility to make future changes by performing operations at a higher level. Two excellent examples are Sun's Java API for XML Processing (JAXP) and Java API for XML Messaging (JAXM). Both of these frameworks provide high-level APIs for performing specific XML-related operations. By programming to these abstract APIs rather than the concrete APIs of specific components, developers make it possible to switch their XML processor or XML messaging protocol easily.

Task frameworks provide additional functionality for building specific types of applications. Personalization is a good example of a task framework used to produce XML documents for content servers. These types of applications use metadata about user preferences and metadata about content topics to generate customized content. Because XML is a convenient format for both types of metadata, there is the opportunity to deliver a package that greatly simplifies the development of such applications. But perhaps the best XML-related example of such an application is B2B messaging. This type of application touches on a host of issues, from specifying the allowable flows of messages, to generating views of executing processes, to integrating with back-end systems. Providing all this functionality would be difficult for a single application development team. By using XML, vendors can deliver a widely applicable framework that puts such applications within the reach of more organizations. All the major application server vendors—including BEA, IBM, Microsoft, Oracle, and Sun—provide their own flavors of both personalization and B2B messaging frameworks.

Content Servers

Content servers combine data from DBMSs, results from business operations, and authored content into presentation formats suitable for different users. XML-based technologies improve every stage of the fulfillment pipeline. At the very end of the pipeline, they enable dynamic layouts that better fit each user's needs. In the middle of the pipeline, they make it easier to connect a user to the exact information he wants. At the beginning of the pipeline, they make it easier to acquire the library of content necessary to satisfy the user base. Most content servers focus on one or two aspects of this pipe line, so implementing a complete XML content strategy may require several types of content servers.

The most common use for XML in content servers is applying dynamic presentation to XML content. This process occurs as described in Chapter 3's discussion of using XSLT to generate pages in XML-based presentation languages such as HTML, VoiceXML, and WML. Based on variables, including the type of client device, the type of content, and the localization settings for the user, the content server selects an XSLT transform and applies it to the XML document. Because most Web servers have programming extensions that support XSLT, you won't need any additional server infrastructure if all you want is dynamic presentation.

Customizing layouts for users is only part of the content delivery equation. Users also need help finding the content that addresses their immediate needs. Traditional search engines suffer from the problem (raised in Chapter 1) of distinguishing between different contexts for the same word. With XML content, a search engine can use the element structure and attribute values to improve search precision. Using an XML-aware search engine helps maximize the benefits of an XML-based content strategy. Usually, employing such a product involves assigning a dedicated server or cluster of servers to perform searches that then refer users to the appropriate content. Such standalone solutions include DocSoft's extend XML and XML Global's GoXML Search. Of course, most of the CMS and native XML store products discussed previously can perform searches on XML document collections, but this approach works only if you store all the content you plan to search in one of these products.

XML-aware search engines leverage metadata at the element and attribute levels. However, metadata can also apply to entire collections of content. The foundation of the Semantic Web is the use of metadata to provide a conceptual map of an entire site or group of sites. Another W3C Recommendation, Resource Definition Framework (RDF), provides a standardized XML vocabulary for describing the types of content offered, the relationships among content, and the conditions under which content might be relevant. Most site creators use an implicit information model in selecting and organizing content. RDF makes it possible to state this model explicitly. The availability of machine-readable models facilitates automated information retrieval, filtering, and visualization capabilities far beyond those of traditional search engines. The Semantic Web is in its early development, and much of the work is in the form of research and open source projects. However, in the near future, RDF may migrate into mainstream content infrastructure. Web servers will offer RDF descriptions. Search engines will use these descriptions as part of the search criteria. Authoring tools will generate these descriptions.

In addition to making it easier to find content, XML also makes it easier to acquire content. Content can come from two sources: You can create it, or you can borrow it. When creating content, the ability of multiple authors to collaborate effectively greatly enhances productivity. Web Distributed Authoring and Versioning (WebDAV), a set of XML-based extensions to HTTP from the IETF, makes it possible for authors to work together to create, enhance, and maintain content. A WebDAV server manages contributions, tracks changes, and enforces permissions. A number of portal servers, including Microsoft's SharePoint Portal Server and Oracle9iAS Portal, use WebDAV to enable the collaborative editing of portal content. Common Web servers such as Apache and IIS also support WebDAV. Any client that speaks the WebDAV protocol can use these servers to collaborate on documents. Such clients include popular content authoring tools such as Adobe Acrobat and Microsoft Office. Taken to an extreme, WebDAV enables the replacement of traditional document management systems with a set of distributed WebDAV-capable servers. Oracle iFS and Xythos's Web File Server use this approach.

It is often more cost effective to borrow content from someone else than to generate it yourself. However, this type of syndication faces two problems. First, it is often difficult to fit third party content into an application because of differences in layout. XML solves this problem by giving both parties a format for exchanging information separate from presentation. The subscriber knows the structure of each publisher's content, so it can use XSLT to integrate content from different sources and apply its preferred layout. There is also the problem of how to negotiate subscriptions, track usage, and update information automatically. Information and Content Exchange (ICE) addresses these issues by providing a standard XML protocol for such interactions between subscribers and publishers. ICE support is available in a wide variety of products that generate and manage content, including Interwoven's OpenSyndicate, Oracle9i, and Vignette's Content Syndication Server.


Data-Oriented Components

Deploying XML-based systems requires that developers write software that accesses, manipulates, and creates XML as if it were data. In some cases, this software runs on top of the server infrastructure discussed previously. In other cases, the software may operate as a standalone program. In all cases, XML offers improved productivity through the wide availability of components for generating, processing, and distributing machine-readable XML data, which developers require.

While there are numerous types of utilities that can assist them, there are three categories crucial to most serious applications. Any serious XML programming requires moving XML data in and out of internal programming data structures. Development tools reduce the amount of coding and improve the quality of code for this repetitious task. In practice, using XML as data also requires transforming documents from one XML format to another, as well as between XML and other formats. Transformation tools simplify the problem of creating XSLT transforms and integrating external data into XML architectures. Moreover, most data-oriented applications of XML involve distributed communication. While application servers and packaged solutions provide basic functionality, the Web services paradigm often requires either integrating a variety of server-provided services or deploying standalone functionality to nonserver devices. In these cases Web services components reduce time to market.

Development Tools

The most common task in developing XML application is ensuring that a piece of code that manipulates an XML document works as desired. Performing this task requires the developer to work simultaneously with a document and the associated code. It helps the developer to view the XML document in two different ways. First, developers need a text view but with all the additional features they've come to expect from a code editor—indenting, highlighting, and syntax checking. Second, they need a browsable tree view, much like they have for class hierarchies. Together these two views allow the developer to maintain a good mental model of the document and thereby understand how the code running against it should work. Many commercial IDEs, especially those for Java and from Microsoft, provide this functionality. There are also open source modules for adding the text viewing capabilities to popular code editors like EMACS.

While IDEs supply features that help when working on code that manipulates XML documents, there is the issue of writing this code in the first place. Many XML applications work with a known set of XML formats. For example, an application for the insurance industry would work with documents conforming to insurance-related XML DTDs or Schemas. Writing the code to move data back and forth between documents using these formats and internal data structure is time-consuming and repetitive. Schema compilers can take a DTD or Schema and generate much of this code, saving time and improving quality. Such tools include Breeze Factor's Breeze XML Studio (Java), Bristol Technology's eXactML (C++), Microsoft's Visual Studio .NET (C++ and Visual Basic), The Mind Electric's Electric XML+, OracleXML Class Generator (C++ and Java), and Sun's Java Architecture for XML Binding (JAXB).

Transformation Tools

Many applications concern the exchange of information about coherent business entities. It is likely that different parties to these exchanges will have different definitions of these entities. These different definitions will probably take the form of different DTDs or Schemas. As you saw in Chapter 3, XSLT can address this problem. However, for large applications that work with many different formats, you may find it difficult and time-consuming to create all the necessary XSLT by hand. Creating a supply chain management system may require writing transforms for hundreds of formats. Moreover, these formats can change regularly. Developers need tools to make this process more efficient. A number of vendors offer visual mapping tools that let the developer indicate how to move element and attribute values from the source format to element and attribute values of the target format. The tools then generate the XSLT code, and many provide debugging features for making sure the generated code works properly. Products in this category include ActiveState's Visual XSLT (Visual .NET plug-in), B-Bop Xfinity Designer, eXcelon's Stylus Studio, IBM's Visual XML Transformation Tool, Induslogic's XSL Stylesheet Editor, and TIBCO XMLTransform.

Unfortunately, not all data exists in an XML format. Many applications must map data from non-XML sources to XML formats and vice versa. Developers need components that can perform this mapping both in bulk and upon request. These tools are similar to those for mapping data from relational databases to object-oriented programming structures and migrating data from operational stores to data warehouses. They include features for browsing the structures of data sources and then specifying how to translate from data fields to elements and attributes. These tools are quite sophisticated because processing data from a non-XML source may require more than simply pulling data from a column and putting in an element. There are also datatype conversions and collapsing or exploding relationships between fields. The two leading products for this purpose are Data Junction's Integration Engine and Mercator's Integration Broker.

Distributed Application Components

As previewed in the discussion of Application Servers in the preceding Server Infrastructure section, many applications use XML to facilitate distributed communication in custom applications. Usually, such applications use SOAP and the other Web services protocols. Task frameworks can help improve productivity, but the issues can extend beyond the boundaries of application servers. It often helps to have tight integration of distributed application development across development tools, frameworks, and servers. The development tools can then quickly lead developers in the use of the advanced framework and server features. This breadth of integration usually comes under the umbrella of a large vendor's overarching distributed XML paradigm. Examples include BEA WebLogic Workshop, IBM WebServices, Microsoft .NET, and Sun ONE.

In many cases, you will already have a substantial commitment to a vendor through existing server infrastructure and development tools. Obviously, the least disruptive choice for Web services components will be that vendor. However, if you plan to make a decision on your preferred supplier based on Web services support, you can go either the integrated solution or best-of-breed route. The integrated solution route offers the benefit of making a single vendor responsible for the success of your applications. It has the drawback of restricting you to that vendor. The best-of-breed route has the advantage of enabling you to choose the best vendor for other infrastructure such as platforms and tools. It has the disadvantage of making you responsible for integrating the entire system. Major vendors offer integration along two dimensions: platforms and tools. In this case, the platform is a composite of hardware architecture, operating system, and mission critical components like DBMSs.

Figure 5-6 presents a conceptual graph of where major vendor offerings fall in the space formed by these two dimensions. Microsoft has the highest platform specificity because it requires that you use its operating system and other server infrastructure. IBM and Sun both offer their solutions on multiple platforms but, of course, they try to drive the sales of their specific platforms. BEA is almost completely platform agnostic. All vendors try to take advantage of a Web service's potential productivity gains to some extent, with a certain degree of development tool integration. BEA does this through a high-level framework but leaves core programming tool choices. IBM and Sun offer their own IDEs based on standard programming and scripting languages. Microsoft offers its own IDEs and languages.

Figure 5-6. Degree of Integration by Major XML Component Vendors

The large vendor initiatives have drawbacks. To achieve the desired breadth of integration, they often sacrifice modularity and performance. Obviously, they want developers to buy the complete packages, so they often require all-or-nothing commitments. Maintaining the synchronization between all the components usually introduces a substantial amount of overhead. These drawbacks are severe for applications not deployed in application servers, especially if targeted at small devices. Smaller products such as CapeClear's CapeStudio, The Mind Electric's GLUE, and Polar Lake fill this gap with components that Web services enable application-specific code in a tight, standalone footprint.


Content-Oriented Components

Delivering XML content to users requires off-the-shelf components for authoring, presenting, and delivering human-readable XML content. The principal benefit of XML as a content standard is that content from all sources uses the same infrastructure. Different groups of authors may contribute content, and it is easy to convert data from applications into content for users. Unfortunately, this capability raises the complexity of the authoring task because authors need to merge static content seamlessly from numerous contributors with dynamically generated content from applications. Well-defined processes become critical to coordinating multiple content production paths.

There are three things authors must do to take advantage of XML's benefits for content delivery: (1) format content as XML documents, (2) create layouts for this content, and (3) follow a rigorous production process to ensure uniform quality. Authoring tools, layout tools, and content management components provide the functionality necessary to deliver high-quality, high-appeal content quickly.

Authoring Tools

Much XML content includes static documents produced by human authors. Creating them with a text editor is a slow and error-prone process. Moreover, many documents have primarily static content with select pieces of information extracted dynamically from an application. Authors certainly do not want to have to enter programming-related tags manually to drive this capture process. Document authoring tools offer three primary features for improving productivity.

First, they offer word processing-like interfaces that enable authors to create ad hoc documents. This feature separates the free text information from the layout. Second, they offer a wizard- or form-based interface that authors can use to populate documents conforming to a more detailed DTD or Schema. This feature speeds data entry for content such as customer contact reports. Finally, they include features for specifying placeholders for application content that a content server uses to insert dynamic values at delivery time. This feature requires integration with a runtime content processing engine. While different products offer different mixes of these three features and are evolving quickly, some of the most popular products include Arbortext's Epic Editor, XMLSpy's Document Editor, and SoftQuad's XMetaL.

Layout Tools

XSL is a highly sophisticated layout description language. A large proportion of layout designers with the necessary graphical design background may not have the technical background necessary to create stylesheets by hand. Therefore, they need stylesheet layout tools for automatically configuring different page regions to display different types of elements and specifying the text formatting based on rules such as element type and attribute values. These tools must also be aware of DTDs and Schema so that authors can match detailed layouts with detailed formats.

Typically, layout tools allow designers to work in either concrete or abstract modes. In a concrete mode, the designer has an example XML content document and applies formatting to that document's information. The tool then abstracts this information to generate a generic stylesheet. By using several example content documents, the author can ensure complete coverage of different possible cases. In an abstract mode, the designer applies formatting rules to a DTD or Schema. This approach gives the author more flexibility in dynamically determining how to present information based on its value, but it is not as intuitive. Before selecting a layout tool, you should work with your designers to decide which mode is more important. The best tools allow designers to switch between modes, but they still tend to emphasize one over the other. Leading products include Arbortext's Epic Editor, eXcelon's Stylus Studio, IBM's XSL Stylesheet Editor, Whitehill <xsl> Composer and XMLSpy's XSLT Designer.

Content Management Components

As an organization adopts XML, more and more people become involved in authoring XML content. As with HTML and SGML, managing this content poses a logistical challenge. Moreover, because software will automatically generate many documents, there is the possibility for new challenges and even greater content volume. Figure 5-7 shows the generic architecture of a CMS that can help address this problem. As shown, CMSs typically have several major components.

Figure 5-7. Generic CMS Architecture

  • Repository. The repository provides a robust and fault-tolerant location for storing XML documents, stylesheets, and DTDs or Schema. It consists of an interface that enables the content management system to store and retrieve information, a manager that controls storage mechanisms, and the storage mechanisms themselves. These mechanisms may include filesystems, relational databases, or object databases.

  • Version control. The version control subsystem performs two functions. First, it prevents multiple authors from simultaneously making changes in the same content. Second, it maintains a version tree of all content. All requests to store or retrieve content must go through the version control system because it maintains the mapping of logical versions to physical data. CMSs that support WebDAV for version control offer the advantage of easier interoperability with different types of authoring clients.

  • Authoring workflow. A content management system needs a component to coordinate the contributions and revision process. Typically, this coordination includes the maintenance of an authoring schedule and assignments for each author. It ensures the routing of documents from one author to another based on content dependencies. As published content reflects on the organization, this routing may also include approval and revision workflow.

  • Content processing. Once authors have submitted content, the CMS may offer a number of processing functions. Foremost among these functions is content indexing. If you commit to managing all your content within a CMS, you rely on the search functions it provides. To perform such searches efficiently, authors need to specify how to index the content. In cases where documents include dynamic information bound to application data, another processing function includes accessing, formating, and distributing this data. Some CMSs offer advanced filtering and transformation processing to create different views of content suitable for different audiences automatically. For example, a filter might specify how to generate a summary view for a particular type of document automatically.

  • Deployment management. When content becomes ready for consumption, an administrator must release it for distribution. Depending on the channel, managing this deployment may take several forms. For static content intended for Web servers, it might create the directory structure and install it in all the Web server machines. For dynamic content intended for Web servers, it may require a substantial amount of configuration information governing network topology, access control, and performance parameters. In cases where you intend to syndicate your content, the CMS will need additional information to manage this process.

There are a wide variety of CMSs, each with its own target use. Content management is a complicated topic, and the choice of product requires considerable analysis. Factors to consider include the primacy of Web over other channels, the use of non-XML formats, and integration with dynamic data sources. Products to consider include Arbortext's Epic E-Content Engine (no repository), BroadVision Publishing Center, Chrystal's Astoria, Documentum4i, Interwoven TeamSite, OmniMark Technologies'/OmniMark, Red Bridge Interactive's DynaBase, SiberLogic's SiberSafe, and Vignette's Content Suite.


Design Tools

The previous two sections divided components into data and content manipulation categories. How do you know if XML is data or content? It depends on the design of the XML formats used by your system. When you plan to use formats that interact primarily with application code or database servers, you design them to reflect the rigid structures and datatype focus used by these systems. When you plan to use formats that primarily drive the organization of information delivered to users, you design them with flexible structures and semantic focus used by people. These decisions should occur early in the development process and guide how the entire team uses XML.

In the case of data-oriented formats, the design process is similar to that for programming data structures or database schemas. In the case of content-oriented formats, the design process is similar to information design or user requirements analysis. In either case, the designer needs to focus on the structure of information rather than the syntax for DTDs or Schema. Design tools facilitate the conceptual design of documents and then generate the appropriate syntax. A graphical designer works much like a data modeling tool. Designers use a tree metaphor to create the basic element structure and then use dialog boxes to configure the allowable element and attributes data. In practice, the generated syntax may need manual tuning, especially for very complex formats, but tools can usually cover most of the cases.

There are currently two primary choices for graphical design tools: Altova's XML Spy Integrated Development Environment and TIBCO TurboXML. On paper, they provide much the same feature set. Both provide tree, inspector, and syntax views of XML formats. Both support DTDs and Schema. In fact, many organizations use these tools to convert their existing DTDs to Schema. Both do a pretty good job of generating the correct syntax. In practice, they provide interfaces that are different enough to yield distinct experiences. Many designers develop a strong preference for one interface over another after using them, so you should thoroughly evaluate both before choosing one.


Infrastructure Selection Strategy

This chapter presents XML software infrastructure in the order that you would bring pieces together during development. To maintain maximum flexibility, you would actually make your product selections in the same order. However, the practicalities of the procurement process will probably force you to make decisions early in the project, or perhaps even before starting a specific project. Also, as you will see in Chapter 6, establishing your infrastructure beforehand can also make it easier to acquire staff with the appropriate skills.

If you do have to make an early decision on XML software infrastructure, it's important to know whether you plan to use XML for data-oriented or content-oriented projects. For data-oriented projects, the choice of application server will usually drive the rest of the decisions. Your existing environment, price, and the availability of applicable features like Web services frameworks should factor most prominently. This choice will probably make the selection of fundamental components for you. It will also influence whether you need other server infrastructure and data-oriented components, as well as the most suitable products if you do.

For content-oriented projects, the choice of CMS will usually drive the rest of the decisions. In nearly all cases, the CMS will include fundamental and other content-oriented components. You may have to choose a data server separately, depending on the degree and volume of dynamic content generation. In cases where you have a balance of data-oriented and content-oriented development, you will have to decide simultaneously on an application server and CMS, taking into account the success of other organizations successfully using different combinations.


Chapter 6. Processes and People

    Executive Summary

    XML Applications and Change

    Content Documents

    Business Documents

    Data Documents

    Common Issues


Executive Summary

As a manager, you will probably have at least some involvement in the management and staffing of an XML development project. In most cases, the addition of XML to the technology mix requires more than developer training and tool acquisition. While the use of XML can fit into almost any well-defined process, you will have to make some adjustments.

The extent of these adjustments depends on the gap between the special needs of XML development and the accumulated experience of your team. If your existing processes and staff are not suited to XML, the cost of implementing the first application will be particularly high. To estimate the gap, you should first categorize your application based on its general use of XML. While large projects may use XML in more than one way, these different applications of the technology usually occur in distinct subsystems. The general applications of XML are the following.

  • Content Documents. Documents used primarily for the exchange of information among people.

  • Business Documents. Documents representing business entities and intended for use by both people and applications.

  • Data Documents. Documents used primarily for the exchange of information among applications.

As discussed in Chapter 5, schema design really defines the intention of XML and determines into which category a particular application falls. However, by now you know enough about XML to predict roughly how a potential application will use it before designing schemas. Table 6-1 summarizes the development process and staffing requirements for each of the categories. The first part of this chapter covers these requirements in detail.

Table 6-1. The Development Process and Staffing Requirements
Content Documents | Business Documents | Data Documents
Resemblance to Existing Processes | Developing interactive Web site | Automating cross-functional business processes | Programming client-server interactions
Key Process Steps

  • Stylesheet design

  • Repository integration

  • Schema adoption

  • Back-end integration

  • Document processing

  • Application integration
Key Staff

  • Producer bearer

  • Information designer

  • Layout designer

  • Standards

  • Business analyst

  • Architect

  • Data designer

  • Library developer

  • Application developer

The recommended process and people changes for these three categories provide general preparation in guiding XML application development. However, there are also a number of specific issues that commonly arise on XML projects. The most prominent include Development Team Cohesion, Version Skew, Open Source Management, Business and Development Focus, and Cooperation with Other Parties. Particular preventive techniques offer another form of preparation. The second part of this chapter describes these five common problems and the recommended remedies.


XML Applications and Change

XML has potential benefits in almost every area of software development. However, many software development organizations lack experience with structured document technologies such as XML. In the past, introducing new software development technologies like client-server architectures, object-oriented programming, and the Internet has resulted in significant disruptions to adopting organizations. You can mitigate the XML disruption by addressing the need for process and people changes before starting a project. This strategy helps minimize the impact of XML on the software development process in three important areas.

  • Budgeting. All software development organizations face a scarcity of resources. Therefore, you must efficiently allocate resources to application development projects and accurately estimate the cost impact of specification changes. New software development technologies usually have important effects on the cost of application development. For example, XML can reduce the cost of developing a given set of information exchange functionality. However, XML can also deliver completely new information exchange capabilities. Implementing such capabilities adds to the cost. New software development technologies often have an important impact on the timing of costs. For example, designing schemas can increase the costs during the design phase while decreasing the costs during the implementation and maintenance phases. Accounting for such impacts enables you to allocate software development resources across an application's life cycle better.

  • Staffing. Using new software development technologies typically requires new or different skills. For example, XML developers must be able to design schemas, manipulate the data structures produced by XML processors, code with XSLT, and work within domain-specific XML frameworks like ebXML. Understanding the set of required skills allows you to make appropriate staffing decisions by finding new developers with these skills or providing these skills to existing developers through training programs.

  • Project management. Delivering an application on time and on budget requires the use of a software development process that matches the natural dependences in the application design. Such a process enables project managers to measure progress accurately and to coordinate the need for different resources at different stages of the process smoothly. New software development technologies typically result in application designs with dependencies that are different from those of previous technologies. For example, in many XML applications, the set of schemas employed will drive much of the development work. However, much of the benefit from using schemas comes from making them an explicit contract for interaction among several software applications. Therefore, schema design often requires working with other organizations or at least considering their requirements. This cooperative process can often consume a considerable amount of time. But with schemas in place, the work of document processing and stylesheet design proceeds more rapidly than you might otherwise expect.

Clearly, the breadth of potential uses for XML means its effect on the software development process will vary greatly from project to project. However, by abstracting the similarities among XML applications into the three categories introduced previously, we can perform a high-level analysis of the software development process impacts. For each of these categories, this chapter examines two crucial areas.

  • Development process. We examine the XML-specific steps in the development process and the dependencies among the steps. The particular steps provide a general framework for structuring timelines and budgets as well as revealing opportunities for parallel development. The processes presented do not replace your existing methodology but rather supplement it to account for the addition of XML. You usually repeat each XML-specific step as you progress through the standard design, implementation, and unit testing phases.

  • Required staff. We examine the different types of staff and their required skills, tying their activities to the development process presented in the preceding section. Each type of staff has a label for convenience, although the actual employee titles will obviously vary from organization to organization. The important points are the roles of the different staff types and their respective skill sets.

Because there is a significant amount of variation among applications within each category, the recommended XML-specific steps are generalizations. Clearly, the process for a particular application will deviate somewhat from the general models presented. Moreover, differences among software development organizations will probably cause further deviations. However, the three general models can guide you in updating your development processes to accommodate XML.


Content Documents

In this application category, the XML paradigm is something of a replacement for HTML on its own. The end delivery format in most cases will still be HTML, but XML-formatted content and XSLT style sheets drive the process. Usually, the document author wants an XML solution instead of plain old HTML for one of these three reasons.

  1. The author wants to generate content from an existing electronic source. Generating personalized news documents from news feeds and databases is an excellent example. In this case, the document author wants to define a common structure for all the documents and populate the structure automatically based on certain parameters. An online sports pool where the author wants to generate the results electronically for each user based on the user's individual picks is another example application that could benefit from this electronic generation.

  2. The authoring process is highly collaborative. Online publishing is the prototypical example. A group of geographically dispersed authors wants to contribute their pieces of content, which are then processed for multiple audience-specific outlets. The group needs highly structured units of information so all the pieces will fit together for delivery, and the formatting must be completely separated from the content. Distributed project management applications are another example.

  3. Users need the ability to search through a document archive using the metadata provided by XML tags. The search for works by William Shakespeare introduced in Chapter 1 is one example. In this case, the document author wants to provide plays online with the content marked up to indicate the role of each element. Online product catalogs are perhaps the best example of when facilitating such a finely tuned search would benefit both users and the application provider.

People are the primary consumers of Content Documents. Software applications may index and search content documents, but primarily the information within the documents themselves is meaningful to people. It follows that presentation is important. Therefore, Content Document applications must heavily involve XSL stylesheets as well as XML documents.

Development Process

Because Content Document applications often use XML to replace HTML, the development bears a certain resemblance to that of traditional Web applications. The primary differences lie in XML's cleaner separation of responsibilities between information design and presentation design. As Figure 6-1 shows, the development process for Content Document applications has the following seven steps.

Figure 6-1. Development Process for Content Document

  1. Design schemas. The primary goal of most Content Document applications is to deliver structured information to human users. Therefore, the first step in the development process is assessing what information the users need and organizing this information as a set of schemas. Because the information needs of users will vary from application to application, most Content Document applications require custom schemas. Information designers may be able to leverage existing schemas in whole or in part. However, because the value proposition of a new Content Document application is the delivery of information tailored to the needs of target users, some custom schema design is almost always necessary.

  2. Design stylesheets. The information consumer for Content Document applications is the human user. Human users process information most effectively when it is presented in a well-designed visual format. Therefore, once application developers have decided what information to present, the next step is to develop the layouts for that information. In most Content Document applications, users view documents within an XML-capable browser, using stylesheets, so layout design implies stylesheet design. Because one of the advantages of XML for Content Document applications is the customization of layouts, designers require a model of the intended audiences and their requirements. Each distinct audience segment may then need its own stylesheet for each schema related to its information needs.

  3. Implement document management. The schema design phase produces templates for the information delivered to users. The stylesheet design phase produces the layout for these templates. The next step is implementing the infrastructure necessary to marshal documents and stylesheets and then deliver them to users. This document management may be as simple as creating a filesystem directory with the appropriate permissions and configuring a Web server. It may include the use of a more sophisticated content management system to manage collaborative authoring, authorize processes for releasing content to users, and update cycles for different types of content. Completing this phase requires managers and producers first to define policies and then for administrators to configure the document management systems to reflect those policies.

  4. Author documents. With the document management infrastructure in place, the application needs the content to deliver. Most Content Document applications include static XML documents. Human authors must create these documents. As discussed in Chapter 5, they should use either ad hoc word processor-like editors or wizards that accept form-based input. Administrators have to acquire these tools and then integrate them with the document management systems. In most cases, this integration should be a simple matter of configuration, but it might also require the application of update patches or purchase of add-on products. For particularly specialized applications, developers may have to write custom authoring tools.

  5. Generate documents. Many Content Document applications include dynamic documents generated from data in external sources. For these documents, developers must establish connections to the appropriate sources and specify the rules for generating documents from the data that they contain. Hopefully, developers can use third-party tools for data integration to access the data sources. Otherwise, they may have to write custom code for this purpose. Applying the rules for generating documents requires capturing parameters from user input, and developers may implement them with a publishing tool or a scripting language. During the testing phase, developers must ensure that the external sources can handle the load imposed by the generation process for the expected number of users.

  6. Integrate front end. After providing for the creation of the content documents, developers must provide for their delivery to users by integrating the application with front-end clients. For XML-capable Web browsers, this integration may consist simply of providing a link to a starting page from a well-known Web location. For non-XML-capable Web browsers, this integration will require distributing XML plug-ins or updated versions of the browser. This step is particularly important in the testing phase because the differences in the platform and version spread of browser clients require thorough functional testing to ensure that all users have acceptable experiences.

  7. Integrate storage. In most cases, an application requires persistent storage for content documents. Developers have to integrate the document management infrastructure with this repository. XML-specific stores include automated functions for the storage and retrieval of XML documents. More generic repositories may require writing custom storage and retrieval code. Note that repository integration can occur in parallel with front-end integration. As with document generation, load testing is important to make sure the storage mechanism can handle the expected load.

For very simple applications that deliver static documents via a Web server, the Content Document development process is nearly identical to the development of a traditional Web site. Schema design is the important new element. Developing applications that deliver dynamic XML documents via a Web server is very similar to developing dynamic Web applications with HTML. Schema design and the separation between designing stylesheets and generating documents are the important new elements. However, applications that use custom document management or sophisticated repositories add steps similar to those in traditional client-server software development.

Required Staff

The similarities to traditional Web application development make the types of staff required for Content Document applications similar as well. As in HTML Web applications, there is a split between authors of static documents and designers of dynamic documents. There is an additional split between information designers and presentation designers. As one would expect, similarities emerge in project management and support personnel. The following types of staff are required.

  • Producer. Whether for an intranet, extranet, or the Internet, Content Document applications must address the requirements of their target users. Because the target user groups are usually quite large and the expectations for responsiveness are usually quite high, there must be a staff member devoted to assessing user requirements and taking appropriate action—the producer. The producer coordinates the identification of changes in user requirements with the allocation of resources necessary to implement appropriate application enhancements. During initial application development, the producer plays much the same role as a traditional project manager, determining application goals, and managing the development schedule. However, appealing to thousands of users and making daily changes require a more dynamic ongoing role similar to that of a producer in the media industry. The producer will have experience in project management, product development or design, and perhaps media production.

  • Information designer. Content Document applications deliver information to human users. The information designer, taking direction from the producer, determines what information to provide and how to organize it. This staff type participates primarily in the schema design phase. However, the information designer may also participate in the implement document management and storage integration phases. These phases require the information designer's input on the probable usage patterns of the different document types so that other types of staff may implement efficient document management infrastructure and storage access. The information designer will have experience in requirements gathering, requirements specification, and data modeling.

  • Layout designer. The layout designer develops the stylesheets for the application. These designs depend on the types of documents designed by the information designer and the information gathered about users. In some cases, the producer may arrange user focus groups in which the layout designer participates to gather this data. The layout designer will have experience in graphic design, Web page design, and perhaps user interface design.

  • Application developer. Sophisticated Content Document applications may require the coding of custom software. These applications require an application developer. The role of this staff type is to supply programming expertise for the creation of custom document management infrastructure, custom document authoring tools, and custom document generation tools. The application developer will have expertise in appropriate programming languages such as Java, Visual Basic, and Perl.

  • Document author. Document authors have domain-specific knowledge that they need to encode as XML documents. To facilitate the rapid capture of this knowledge, they should use wizard-based authoring tools, although ad hoc tools will work better if the nature of the information does not lend itself to the template approach. They will have expertise in the particular domain for which the application provides content.

  • Data integrator. For applications that generate documents from external sources, the data integrator translates the information in these sources into the appropriate XML format. In some cases, this translation may simply require using a graphical tool. In others, it may require some scripting. The data integrator will have experience in database programming or administration and perhaps in programming languages such as Java, C++, and Perl.

  • Administrator. The administrator is responsible for configuring the document management infrastructure—for instance, Web servers and content management systems—and ensuring the ongoing operation of the application. This includes the application hardware, the document management infrastructure, the authoring system, the generation system, and the persistent storage mechanism. Sophisticated applications may require more than one administrator. The administrator will have experience as a Webmaster or system administrator, perhaps as a network administrator, and perhaps as a database administrator.

When compared with the staff for traditional Web applications, the information designer is probably the most revolutionary addition to the development team. The separation of information format design from presentation design and document authoring makes explicit a role fulfilled implicitly by a number of different traditional staff types. The combination of general requirements gathering skills and specific schema design skills is likely to be the most difficult to find. In the short term, layout designers, application developers, and data integrators with XML experience may be in short supply, but their skills will be transferable enough to make learning XML relatively straightforward.


Business Documents

In this application category, XML documents contain information about fundamental business entities such as customer, order, and invoice. The crucial benefit of XML is that it allows humans and software to use these documents in the same format. The need for Business Document applications usually stems from one of three requirements.

  • Automate the workflow of document processing. Everyone agrees that moving paper documents around is an efficient means of executing business processes. However, prior to XML, automating workflow had two drawbacks: Businesspeople were tied to a particular workflow system, and the document data representations often violated business people's concepts of what a document looks like. With XML Business Documents, all workflow systems that use XML can interoperate, and businesspeople are free to view familiar documents, while the system is free to process the underlying data.

  • Synthesize familiar documents from business software systems. Typically, business systems present business information in terms of its underlying data model. Often, this representation is not particularly convenient for businesspeople, especially when a logical business document encompasses information from multiple business systems. Software engineers who build these applications do not want to hand-create each type of document. Synthesizing XML business documents using an automated tool is an efficient solution for both users and developers.

  • Automate document exchange. With the connectivity enable by the Internet, many businesses want to achieve operational efficiency by using the business document architecture discussed in Chapter 4. XML Business Documents such as Order and Invoice constitute the seed of an effective supply chain management system. Each business simply needs to accept an explicit set of Business Document definitions and then map them to data representations used by business software systems. Software engineers do not have to hand-code a translation bridge for every trading partner, and businesspeople can still view familiarly formatted documents if they wish.

The strength of Business Document applications is that people can view documents and software applications can process them, using the same underlying representation. Presentation concerns and compatibility with XML formats from other organizations are both significant, so XSLT can play a substantial role in these applications. Businesspeople definitely want to view the documents in a familiar layout, but the need for visual sophistication is less than that for Content Documents. On the other hand, software engineers need more sophisticated tools so they can efficiently map XML document definitions to the data models used by business software applications.

Development Process

In some sense, Business Document applications differ the most from traditional software development. Their general goals are to provide a means of universally defining common business concepts and provide the infrastructure for cooperatively manipulating these concepts. At the analysis level, some organizations have used information engineering or business object modeling to define common business concepts in a fashion analogous to XML schema design. But even then, they actually implement the information exchange described in these models on an ad hoc basis. XML provides a means of formalizing the implementation of information exchange services. As Figure 6-2 shows, the development process for Business Document applications has the following ten steps.

Figure 6-2. Development Process for Business Documents

  1. Adopt schemas. Because Business Document applications concern the exchange of information related to fundamental business concepts, there are often existing schemas for these concepts. In some cases, industry groups may propose a set of schemas to cover common business concepts. In others, a partner may want to agree on common schemas. In yet others, a vendor of a third-party software package may provide schemas. In all these cases, application developers have three choices of how they adopt schemas. One, they may simply adopt them as is. Two, they may work with the other parties to help define them. Three, they may choose to use parts of them, with modifications to meet their particular needs.

  2. Design schemas. In some cases, Business Document application developers may need to define fundamental business concepts that are unique to their organization or develop enhanced definitions that give their organization a competitive advantage.

    In these cases, application developers will design their own schemas.

  3. Specify document processing. Because Business Document applications typically facilitate one or more business processes, they must include rules for document processing that reflect the business process rules. For example, in supply chain applications, these rules will reflect the allowable interactions between suppliers and customers. In workforce automation applications, these rules will reflect the processes employees should take in performing their job tasks.

  4. Implement front-end processing. Business Document applications have human users. Developers must therefore implement the processing rules that apply to the documents produced and consumed by human users. These rules may include allowable actions such as "approving" or "rejecting." They may also include security restrictions. Developers must develop the code necessary to enforce such rules, or administrators must configure the off-the-shelf packages used.

  5. Design stylesheets. Human users comprise one of the information consumers for Business Document applications. Therefore, once application developers have decided what information to present and the processing rules that apply to the information, the next step is to develop the layout for that information. In some Business Document applications, users will view documents within an XML-capable browser using stylesheets, so layout design implies stylesheet design. In some cases, users may require specialized client software for certain types of document manipulation. Applying digital signatures to a business document or visualizing document data using SVG are good examples. For such applications, layout design may imply more traditional graphical user interface design.

  6. Integrate front-end. With the front-end processing rules and interface layouts in place, developers must integrate them with front-end clients. For XML-capable Web browsers, this integration may consist simply of providing a link to a starting page from a well-known Web location. For non-XML-capable Web browsers, this integration will require distributing updated versions of the browser or XML plug-ins. Sometimes, the sophistication of Business Document applications requires custom clients—perhaps for such tasks as real time collaboration. In this case, developers may have to write code that connects the client graphical user interface to the front-end processing implementation.

  7. Implement application processing. Software users comprise the other information consumer for Business Document applications. Developers must therefore implement the document processing rules that apply to the documents produced and consumed by software users. These may include required actions such as "create" or "update," as well as transaction restrictions. Developers must either write custom code necessary to enforce such rules or write scripts to drive an off-the-shelf application. Note that this step and the next two steps may occur in parallel with the previous three front-end steps. This step presents something of a challenge during the testing phase because the potential range of document data makes it difficult to achieve complete functional testing coverage.

  8. Implement transforms. Business Document applications almost always involve information exchanges with other parties. Usually there is substantial variation in the data that comes from these parties. Most Business Document applications need XSLT transforms to handle this variation. This step is most important in the design phase because that's where the decisions about the expected variations occur. The availability of XSLT generation tools usually makes their implementation straightforward once the requirements are clear.

  9. Integrate application. With the application processing rules and data transformations in place, developers must integrate them with other application components. Developers must define the interface to these components and implement the code necessary to propagate the processing rules through these interfaces. This step becomes quite important during the testing phase because quality of integration and its scalability greatly affect system acceptance.

  10. Integrate storage. In most cases, an application requires persistent storage for business documents. Developers will have to integrate the application processing infrastructure with the persistent storage mechanism. Native XML stores and data servers include automated functions for the storage and retrieval of XML documents. Many traditional DBMSs include components for such storage and retrieval, although often some custom coding to map XML documents to the DBMS's native model is required. Note that storage integration can occur in parallel with front-end and back-end steps. As in Content Document applications, this step is prominent during load testing.

A principal characteristic of Business Document application development is that developers must address the needs of both human and software users. Therefore, after an initial set of steps necessary to agree on a set of information exchange formats, the development process really has two separate paths: one for the human users and one for the software users.

This situation contrasts with traditional software development, where there are constraints from either human or software users but not both. The existence of constraints from both sources can have negative effects on the schedule in a spiral development process. Although the two branches of development may occur in parallel, once one branch has feedback that requires changes in the schema or application processing phases, the feedback affects the other branch, potentially setting off another round of feedback. Achieving the benefits of parallel development therefore requires excellent up-front design to minimize midstream changes.

Required Staff

Because the goal of Business Document applications is to align information exchange with business process flow, implementing such applications requires a wide variety of staff types. Skills have to include business partnership management, business process analysis, and development of interfaces to the different components of the enterprise information architecture. Specific types of required staff include the following.

  • Standards bearer. Because Business Document applications deal with the exchange of information about fundamental business entities, standards bodies are likely to take a strong role in defining schemas describing these entities. A standards body may be an accredited standards organization. It may also be an industry group or even an internal group within a large enterprise. In any case, the application development team needs a representative to the standards body. This standards bearer ensures that schemas describe business entities in a way that meets the needs of the team. The standards bearer needs a combination of both business and technical experience. The standards bearer will have experience in business management, business process design, and application design.

  • Business analyst. The business analyst's role is to make sure that the schemas defining business entities and the document processing rules accurately reflect the organization's business processes. The analyst collaborates with the standards bearer to define the organization's goals for the standards body and collaborates with the architect to decide which schemas to adopt and how to design internal schemas. The business analyst will have experience in the line of business operation, business process design, and perhaps data modeling.

  • Architect. Most Business Document applications span a large user population and a large number of back-end systems. Therefore, there is a need for a staff member to coordinate a large spectrum of technical design tasks. The architect works with the business analyst to specify schemas and document processing rules. He also works with application developers, user interface designers, and the different integrators to ensure that the application architecture operates within the constraints imposed by their respective components of the application. The architect will have a wide variety of technical skills, including requirements analysis, application design, and data modeling.

  • Application developer. In most cases, a Business Document application requires the development of code to implement business logic. In workforce automation applications, there is the task processing engine to implement. In applications that use business data visualization, there is data processing code to write. In applications with special client capabilities, there is client code to write. This coding requires an application developer with experience in the appropriate programming languages and tools.

  • User interface designer. For the parts of a Business Document application that interact with human users, there is a need for appropriate user interfaces. A user interface designer specializes in designing such interfaces based on how the presented information relates to the user's job task. This staff type has a number of options for the design, including the use of stylesheets, special browser plug-ins, and custom clients. Of course, these choices are constrained by the overall architecture of the application. The user interface designer will have experience in user requirements analysis, graphic design, Web page design, and user interface design.

  • Client integrator. In some cases, the user interface needs integration with the existing client infrastructure. This integration may include the deployment of updated browsers, the deployment of special plug-ins, and the deployment of custom user interface clients. For plug-ins and custom clients, there may be issues of integration with client desktop management and configuration tools. These tasks are all the purview of the client integrator, who will have skills primarily in browser configuration and desktop management.

  • Data integrator. Because a Business Document application must provide for the exchange of information with existing applications and data sources, there is the crucial problem of integrating the Business Document application with these existing systems. The data integrator translates the information format in the sources into the appropriate schema format. In some cases, this translation may simply require using a graphical tool. In others, it may require some scripting. The data integrator will have experience in database programming or administration and perhaps programming languages such as Java, C++, and Perl or the APIs of particular packaged applications.

  • Administrator. A typical Business Document application has many pieces of server infrastructure. These pieces include Web servers, application servers, data servers, DBMSs, and packaged solutions. Each requires its own administrator to install and configure the server as well as collaborate with the architect and application developers to integrate them into a single functioning system. The administrator will have experience as a Webmaster, system administrator, network administrator, or database administrator, as appropriate for the particular type of server.

Largely, all the staff types required to deploy Business Document applications already exist within enterprises and vendors. The problem is that they rarely have to coordinate their actions to the extent necessary with XML. Of course, much of this lack of coordination stems from the fact that standards bearers, business analysts, application developers, interface designers, and data integrators do not have a common basis for communication. A unique benefit of XML is that it can provide this common basis. However, project managers still have to address the problem of coordinating and facilitating communication among these separate types of staff.


Data Documents

In this application category, XML serves as the data encoding frame work for software communication. Such applications include interchangeable files, data integration, and remote interface applications. There are also opportunities to create new classes of Internet applications to solve specialized problems. Electronic software distribution (ESD), in which customers install a software program from the Internet instead of from a CD-ROM, is one example. Remote help desks, where a technical support representative remotely diagnoses problems with a computer's software configuration, is another. In fact, the Web services protocol stack itself is an excellent example of a Data Document application. So Data Document applications are particularly vital to the success of XML as a distributed computing platform.

In all these cases, two computers have to exchange a series of messages over the Internet or through a filesystem to accomplish the task. One of the barriers to implementing such protocols is agreeing on data formats. XML schemas provide a straightforward method of formally defining such format. Another implementation barrier is building the software to encode and decode data and then deliver the results to the main application for processing. By using publicly available XML engines, software developers can avoid writing much of this code themselves and, more important, are much less likely to discover errors in the encoding-decoding logic because a publicly available engine has probably undergone far more testing than a custom engine.

Software applications are the primary consumers of Data Documents. You could define a basic XSLT stylesheet for each document type so that software engineers could view data in a format convenient for debugging. Engineers get this feature essentially free.

However, it is a secondary concern when compared with the advantage they get from using off-the-shelf technology. Of course, a key requirement for achieving this benefit is access to publicly available and commercial-grade XML engines, as well as the tools for rapidly integrating them with applications written in different programming languages.

Development Process

Data Document applications are obviously very similar to traditional applications that manage input/output with the filesystem or over a network. The primary differences lie in the generalization of the information formats that XML allows. Moreover, the nature of Data Document applications usually makes them only one component of a larger application. They serve as the engine that enables information exchange; the higher-level application performs all the application-specific information processing. As Figure 6-3 shows, the development process for Protocol Document applications has the following five steps.

Figure 6-3. Development Process for Data Documents

  1. Adopt schemas. Because the primary goal of Data Document applications is to enable the exchange of information between different software applications, software developers do not usually have the luxury of designing all the schemas themselves. Most of the time, they want to exchange information either with an application that already uses particular schemas or with an application whose developers also want to have input into the schema design process. In the first case, the application developers simply adopt the schema used by the target applications. In the second case, they must work with either a formal standards body or an informal cooperative group to agree on a mutually acceptable set of schemas.

  2. Design schemas. In some cases, Data Document application developers may control all application components that exchange information. In others, their application may provide a natural focal point in a group of cooperating applications whose developers are looking for leadership in the specification of schemas. In these cases, Data Document application developers may design the necessary schemas themselves.

  3. Specify document processing. Because Data Document applications are essentially components of higher-level applications, developers must specify how the Data Document component interacts with higher-level applications. This specification may simply be an API. It may also include a set of utilities for manipulating the information contained in the protocol documents to facilitate processing by higher-level applications. For applications that use the filesystem to exchange documents, this document processing specification includes any conventions for directory and file use. For applications that use the network to exchange documents, this specification includes the messaging and network protocols as well as the allowable order of document exchange.

  4. Implement document processing. Generally, because different applications will communicate using Data Documents, there will be room for different implementations of the document processing specification. Therefore, application developers may either acquire a third-party implementation or develop one on their own. A good example of this decision is the construction of the distributed messaging component of a supply chain management package. Developers could either use an off-the-shelf SOAP implementation or write their own if they needed special features such as support for a rare transport protocol.

  5. Integrate application. Once developers have a Data Document component, they must integrate it with the higher-level application. This integration will consist of using the component's API and utilities. Some applications will use them to extract the information from protocol documents and integrate it with the data structures used by the higher-level application. Others will use them to take information from these higher-level application structures and encode them as Data Documents.

Both enterprise and vendor developers may use third-party tools and thereby skip all but the last step. These tools can't completely automate the process of mapping document data to application data, so developers may have to write some custom integration code. In most cases, developers can acquire complete protocol engines. Then they need simply integrate the higher-level application logic with the protocol engine through its APIs. Of course, enabling developers to avoid all but the last step means that some vendor has completed the first four steps. This is precisely the situation that has led to the Web services movement.

Required Staff

Because the development process for a Data Document application essentially boils down to (1) format design and (2) the use of the format within different applications, projects require staff experienced in these two areas. The types of format development staff are the interesting evolution. A standards bearer and format designer are necessary because of XML's more formal format design requirement.

  • Standards bearer. Developing Data Document components often requires cooperation with other parties that plan to use the protocol. As discussed in the Business Document section, this cooperation may take place through a formal standards body, an industry group, or informal meetings. In any case, the development team needs a representative in the cooperative development process. The standards bearer ensures that the data format schema meets the needs of the organization and that its future development plans coincide with the future development plans of the other participating organizations. The standards bearer will have a significant amount of development experience, including architectural design as well as experience cooperating with outside parties.

  • Format designer. If the Data Document application uses custom schemas, the format designer needs to design them. Whether the format schemas come from within the software development organization or from an outside organization, the format designer has to specify the document processing. This role includes designing the Data Document component APIs and specifying appropriate abstraction layers to allow replacement of messaging and network protocols as well as future extension of the component. The protocol designer will have experience in application design, API design, and wire protocol design.

  • Library developer. Each organization using a protocol needs an implementation of a corresponding Data Document component. During the document processing implementation phase, the library developer writes the code that implements the protocol, services API calls, and provides utilities. This implementation needs an appropriate level of abstraction so that different higher-level applications can use the library and application developers can swap the library out for a different implementation. The library developer will have experience implementing libraries in the necessary language such as Java, C++, or C.

  • Application developer. The point of Data Document components is that developers of higher-level applications can use them to enable their applications to exchange information. Therefore, for any given Data Document application, there may be many higher-level application developers who have to integrate the corresponding component into their applications. These developers own the corresponding development tasks for their applications and experience in the same language as the Data Document component's API.

None of the staff types required to implement Data Document applications is new. Format design has long been a recognized area of software development expertise. However, standards bearers, format designers, and library developers have to translate this expertise to XML. In most cases, the learning process should proceed quickly because self-describing data formats such as ASN.1 and s-expressions have existed for some time.


Common Issues

The process and people changes just recommended help minimize the entire set of adjustment issues associated with adopting XML. They are analogous to a general program of diet and exercise to promote physical health. In cases where an individual is at particular risk for heart disease or cancer, doctors usually prescribe specific preventive actions. With XML projects, we have enough experience to identify some of these specific risks and the appropriate precautions.

The author has encountered five common problems: development team cohesion, open source management, version skew, business and development focus, and cooperation with other parties. Most of these problems are not unique to XML, but because of either a characteristic of the technology or a simple coincidence in timing, XML can magnify the risk. Understanding the dynamic behind the prevalence of the issue will help you identify it on your own projects and devise countermeasures suited to your specific situation.

The rest of this chapter examines each of these problems in detail. First, we define the problem and explore how XML magnifies it. Then we assess the conditions that make it more likely and more severe. Finally, we discuss prevention strategies. None of these strategies is perfect, and all require some type of tradeoff. On balance, however, they should help mitigate these problems in most cases and reduce the chances of project dysfunction. Table 6-2 summarizes this information.

Development Team Cohesion

Successfully deploying applications with any new software technology requires a core set of skilled developers with a thorough understanding of its mechanics. XML makes this requirement particularly crucial. The number of developers with previous structured document experiences is rather small. Moreover, the technology continues to evolve rapidly, forming a substantial barrier to entry for new developers. So finding and retaining your set of core XML developers is a challenge.

Table 6-2. Overview of XML Specific Risks
Issue | ProblemOverview | Remediation
Development Team Cohesion | Scarcity of developers compounded by the specificity of XML skills and need for a shared conceptual model | Make pessimistic scheduling assumptions
Open Source Management | Fuzzy feature roadmaps, uncertain delivery schedules, no support, and thorny licensing issues | Assign open source guru
Version Skew | The number of different components necessary makes it difficult to establish a set of compatible versions | Maintain version dependency matrix
Business and Development Focus | With XML's breadth and availability of components, spurious use of XML can quickly grow out of control | Maintain benefit-feature-design table
Cooperation with Other Parties | Information exchange requires aligning the interests of parties with different agendas | Stage cooperation and use metrics

Two factors magnify this challenge. First, because XML has not fully matured relative to other technologies, developer experience is specific to both the types of applications built in the past and the particular tools used. A developer experienced with Content Document applications may not have the skills necessary to build Business Document applications. Someone with previous XML experience using J2EE platforms may take a substantial amount of time to become effective on a project using Microsoft .NET, even though he may also have C++ and Visual Basic experience on non-XML projects.

The fact that XML development teams tend to evolve an unusual amount of shared experience and lore further complicates personnel planning. Much of XML development revolves around establishing and interpreting a shared vocabulary. When an application involves external parties, this vocabulary includes a model of how these other parties operate. An XML development team functions effectively when each member leverages this common mental model to use the same programming idioms. A new developer cannot contribute to the project until he internalizes this model.

These factors boil down to three personnel-related risks: building the team, avoiding turnover, and catching up. The general shortage of experienced XML programmers and the specificity of this experience mean that finding developers who can jump right in can take more time than originally estimated. Developer scarcity and the need for common idiom mean that turnover can really hurt a project because there just may not be the time to find a replacement and train him to be a productive team member. In The Mythical Man Month, Frederick P. Brooks found that increasing development team size does not decrease development time by the same amount. If an XML project falls behind, this general problem combined with the XML-specific need for a shared mental model renders the addition of new developers a completely ineffective solution.

You can minimize these problems with several process- and staffing-oriented measures. On the process side, you must use pessimistic assumptions in scheduling. Assume the low end of productivity, the high end of feature creep, and the high end of turnover. You should also assign one of the senior architects or developers to maintain a dictionary of the team idioms. This dictionary won't eliminate the problem of bringing new developers up to speed, but it will reduce the amount of time required to do it. Finally, for those subsystems where dependencies make them critical to delivering on schedule, make sure that your top developers cross-review each other's work on those systems so they can step in if necessary.

On the staffing side, you want to make sure that you get developers with the right kind of experience and that you keep key capabilities within your control. Achieving the first aim means deciding on the development platform and environment at the beginning of the project and then selecting developers with appropriate experience. Achieving the second aim means resisting the temptation to use vendor professional services or outside system integrator resources for core development. Instead, use them as a cadre to train and mentor your senior developers. Otherwise, the ability to enhance and maintain the XML features of your application leaves when they leave.

Open Source Management

XML is the first major software development technology to come of age after the mainstream emergence of the open source movement. This movement stems from the Internet culture of individuals cooperating to provide software that benefits everyone without the expectation of commercial gain. By the time XML gained momentum, this movement had expanded to the point where major corporations contributed their code to open source bodies. As a result, XML software components from XML processors, to development utilities, to Web services protocol stacks are available through open source bodies.

This coincidence of timing affects development organizations in the commercial realm. In many cases, the best or even only component of a certain type is available through open source; there is no commercial alternative. Consider the case of Xerces, the open source Java XML processor. The contributions of IBM and Sun on top of those from smaller companies and skilled individuals have made it the only real choice for a full-featured Java XML processor. But unlike commercial software, open source software has fuzzy feature roadmaps, uncertain delivery schedules, no support, and thorny licensing issues.

Enterprise developers face this problem most often because they must build business-critical systems that rely on crucial, yet unsupported, functionality. Vendor developers may feel the most serious effects because the improper use of open source software can hamper their legal right to sell a product. For both types of developers, discovering only partial support for standards and cumbersome implementation idiosyncrasies can lead to frustration. They don't have the leverage to demand enhancements or even extract a commitment for enhancements by a certain time.

In the absence of support for open source components from commercial vendors of related software, such as development tools and application servers, the only viable approach is to appoint an internal open source guru. This person administrates an internal repository of all the sanctioned versions of open source components. He serves as the point person for customizing these versions with necessary bug fixes and enhancement, including creating regression tests to ensure that future versions include the changes. Furthermore, he cultivates relationships with open source developers by participating in associated mailing lists and newsgroups. When the organization needs information or action from an open source body, he leverages these relationships to reach a solution. Finally, he serves as a liaison with the legal department to vet the open source licensing agreements and negotiate any additional rights from the open source body.

To fulfill the role of open source guru effectively, a developer must first have expertise in XML and the particular programming language of the component. He must also have intimate familiarity with the project's code base so that he understands the dependencies on open source components and can evaluate potential impacts. Strong interpersonal skills and a tolerance for frustration are important for fostering productive relationships with open source developers. Unfortunately, these requirements pretty much mean assigning the lead role to a senior developer because junior ones won't have the necessary breadth. Combined with the difficulty of finding and retaining qualified developers just discussed, the need to assign a valuable resource to a support role imposes a significant indirect cost on the use of open source software.

Version Skew

XML has two great advantages: It addresses an extremely wide range of problems, and its inherent openness has nurtured an extremely wide range of components. Unfortunately, these advantages have a downside. Together, they make for an extremely large set of potential software combinations. Even with XML's focus on compatibility, the sheer number of possibilities increases the chance of conflicts.

Consider what a modest project would use in the way of standards and components: XML, Namespaces, Schema, XPath, XSLT, SOAP, WSDL, XML processor, XSLT processor, SOAP library, persistence solution, application server, Web server, schema compiler, data-oriented XSLT tool, HTML-oriented XSLT tool, and schema design tool. Each of these artifacts evolves on its own schedule. At any point in time, a version of one may be incompatible with a version of another.

The costs of version skew can be surprising. Obviously, the risk goes up with the number of artifacts involved. But it's very difficult to pare down the list beyond a certain number without sacrificing a large fraction of XML's benefits. For large projects where both the number of artifacts and the impact of the application are high, the potential damage from version skew can quickly accumulate. Furthermore, many components support only subsets of certain standards. When several components have overlapping subsets of partial support, defining the common subset presents a challenge, especially if the component providers do not offer accurate support information. These dependencies make it difficult to upgrade components individually, so enhancement costs can quickly grow.

There is no easy solution to this problem. A practical approach is to use a spreadsheet to build a version matrix. Make a list of each standard and component you plan to use, and then create a matrix by putting this list on both axes. Now each artifact intersects every other artifact in two places. You need to evaluate both intersections because dependencies are often directional. One component may be able to accept data from another but not the other way around. At each intersection, note the extent of compatibility. A vendor may merely assert compatibility with another component, or maybe it has tested compatibility internally. (Hopefully, it fully certifies compatibility.) Don't forget to assess the compatibility of standards with other standards. Using certain features of XML Schema 1.0 may make it difficult to use XPath 1.0 and XSLT 1.1.

Based on the planned architecture of the application, you should be able to create an initial matrix. As the project proceeds, you can more easily evaluate the impact of adding a new component or using a different version of an existing one. It is much easier when you know from the start that you have to upgrade two components simultaneously rather than finding out only after upgrading one. Moreover, during application maintenance, the matrix enables you to estimate the cost of particular enhancements better by determining how many components you would have to upgrade.

Business and Development Focus

With any new software technology, both businesspeople and developers become excited about it and want to try it out. With XML, this tendency can quickly grow out of control and knock projects off track. We previously discussed XML's breadth and the wide availability of open source XML components. These two characteristics may give the impression that XML is the solution to any problem and downloading a wealth of components to prove it is just a matter of a few mouse clicks.

Most good developers are naturally curious and looking to expand their problem-solving capabilities. XML's potential tempts them to use it without first evaluating the impact, and usually they underestimate the consequences. Often businesspeople fuel developers' fire because they want to promote the XML features of the software but don't understand the complexities inherent in deploying new technology. They usually overestimate the appeal of XML to the target users, who really just want a good solution in a short time. The loss of focus from undisciplined use of XML not only throws the project off schedule but can also severely damage team morale.

Software vendors are most susceptible to this problem. They usually lead the way in the use of new technologies so that it's often hard to distinguish between the necessity to innovate and the spurious use of XML. Also, the pressure to distinguish the product in the market can lead to poor decisions regarding the value of an XML-related feature. Enterprises building B2B applications with XML commonly experience this problem as well. The underlying problem is the difficulty of setting boundaries among business processes. Once an enterprise starts looking at one business process, that naturally leads them to another business process, and so on. Instead of trying to deploy an initial version with limited functionality, the team attempts to solve an enormous business problem and usually fails.

Preventing the use of XML from spiraling out of control while still maintaining enthusiasm for its use is a delicate balancing act. One useful approach is to make a business case for every new use of XML and countervailing risk assessment. Whoever proposes the feature must justify the benefit either from the perspective of life cycle cost or user satisfaction. The senior members of the team assess the impact on schedule and potential risks. It also helps to assign a devil's advocate to argue against XML. Spurious uses of XML quickly become evident when the justification is weak and the risks are high. But powerful ones also become evident when the justification is strong and the risks are low. In between, where there are both benefit and risk, you must work with the senior team members to make a decision that has buy in, perhaps not killing a feature outright but deferring it to a subsequent version.

Another approach that keeps developers focused on the application goals is maintaining a "benefit-feature-design" table to track the justification behind every use of XML. A benefit column lists the direct benefit to the user, and a feature column lists the functionality that delivers the benefit. A design column describes the software approach that enables the feature. For example, "rapidly update business processes" is a benefit, "dynamic business process execution" is the corresponding feature, and "XML business process description" is the XML design. By making sure that every benefit is substantial and there isn't a better design choice than XML, this tool creates the disciplined environment necessary for success.

Cooperation with Other Parties

Because most XML applications concern information exchange, development organizations often have to work with internal or external parties to deliver an application cooperatively. For B2B applications, you must cooperate with external parties. For EAI applications, you must cooperate with internal parties. You must agree on common XML formats, common interpretations of the problem space, and a common schedule for delivery.

Reaching these agreements can present difficulties, especially when the parties have somewhat different goals and high visibility puts the pressure on everyone to deliver. There is usually little problem when the parties have an unequal power relationship. A very large customer can usually persuade much smaller suppliers to follow its lead. However, when the parties are of equal size and stature, egotism, combined with differing agendas, can stall a project. In some cases, the failure to achieve cooperation can lead to project cancellation that results in a damaged business relationship.

To avoid this outcome, you should use a process that maximizes the incentive for cooperation. The basic approach is to break the project into relatively small stages that you can use to build trust and share in the validation that each stage brings. Having a goal of "total integration of our mutual business processes" for a B2B application sets the stage for failure. Instead, define a series of steps like "exchange purchase orders," "automated processing of purchase order to generate shipment notification," and "complete the circuit of purchase order, shipment notification, receipt acknowledgment, and invoice." This approach leaves less room for pursuing diverging goals, provides learning feedback from previous stages, and builds unity through success.

Establishing metrics constitutes an important part of the staged cooperation approach. At each stage, the team should be able to point to concrete achievements and areas for improvement in the next stage. In the early stages, these metrics will be necessarily technical. For example, after the first step in the preceding hypothetical stages, the metric might be "95% of purchase order messages were accurately processed." That would be a pretty good result and could feed into the next stage with a goal of 99%. As the project progresses, metrics naturally become more business oriented. After the last step in the hypothetical stages, the metric might be "$107M in transactions were exchanged last quarter." Team members become more committed to the project with such visible signs of success, as do management sponsors.


Chapter 7. Five XML Applications for Enterprises

    Executive Summary

    Workforce Automation

    Knowledge Management

    Trading Partner Coordination

    Application Integration

    Data Integration


Executive Summary

An enterprise application uses XML to provide an operating benefit by increasing the efficiency of business process execution or enabling new business processes. The best way to understand how to achieve such benefits in your enterprise is by examining some examples. This provides both inspiration and a context for using the technology. Up to this point, we've examined XML only in isolation. Figuring out how to leverage XML in the enterprise requires an understanding of how it operates with other components of enterprise applications.

Table 7-1 lists five examples of enterprise applications where XML can deliver significant operating benefits. The rest of this chapter discusses these applications in detail. For some of these applications, enterprise software developers may not have to write the software that manipulates XML documents. In these cases, third-party software provides these capabilities. However, there are product selection, installation, integration, administration, and extensibility issues that require the enterprise to understand how the underlying XML features work.

Table 7-1. Enterprise Applications and XML Benefits
Application Category Examples Key XML Benefit

Workforce

Automation

Content

Documents

  • Claims processing

  • Field service

 Customized structuring of information to support job tasks

Knowledge

Management

Content

Documents

  • Product planning

  • Technical support

 Extensible knowledge structuring and wrapping of existing documents

Trading Partner

Coordination

Business

Documents

  • Supply chain management

  • Service provisioning

 Negotiation of trading relationships that can then execute automatically

Application

Integration

Business

Documents

  • Inventory management

  • ERP/HR integration

 Portable business entity definitions and infrastructure for transformation

Data

Integration

Business

Documents

  • Customer management

  • Product information management

 Customized data formats and document synthesis from data sources


Workforce Automation

Business Challenge

In the increasingly complex business environment, employees often perform tasks where they could use software support. Examples include claims processing in insurance, field repair in utilities, and account management in direct sales. In these cases, there are two primary issues. First, the enterprise wants to make sure that employees follow specific policies. Second, employees need access to a variety of documentation to support their tasks. Workforce automation software can address both issues by guiding employees through the official process and electronically delivering the documentation they require.

It's certainly possible to build such solutions on top of existing application platforms and use HTML as the electronic delivery vehicle. But this approach has two drawbacks: (1) enterprises often have difficulty updating the processes encoded in the software in response to changing business conditions or feedback from employees, and (2) HTML is somewhat limited in its ability to support the delivery of a wide array of documentation to mobile workers. The browser-centric nature of HTML does not necessarily suit the devices used by these workers, and the previously discussed searching constraints make it difficult for workers to locate the documentation they need.

XML Benefit

XML can address both issues. To deliver more flexible processes, XML can serve as the format for dynamically configurable process descriptions. A general workforce automation schema enables the enterprise to encode process flow as XML documents that the system then uses to drive the guidance of worker tasks. If a new type of process arises, creating a new process description immediately enables support for that process. If employees identify the opportunity for improved efficiency of an existing process, updating the process description instantly pushes out the improved model.

As discussed in Chapter 5, XML-based CMSs have a number of advantages. They enable improved searching of the available documents, and they can accommodate a wide variety of ad hoc and structured information. Moreover, the separation of content and presentation possible with an XML content strategy better accommodates devices of different form factors. Many important job tasks involve mobile workers who primarily maintain contact with the enterprise through tablet-based PCs, portable initial assistants (PDAs), or cellular phones. Device-specific XSLT stylesheets enable applications to format content easily for the appropriate form factor using the same XML content.

Architecture

As Figure 7-1 shows, the workforce automation architecture combines elements of workflow systems and content management. When a worker executes a step in a job task, he fills out a form that notifies the task manager component of his status so, by comparing the status of the XML-formatted rules in its repository, it can determine the next step. In addition to processing the task step, it retrieves any specific supporting content from the CMS and then returns this supporting content as well as the status form for the next step in the task. If the worker needs additional supporting content, he can directly query the CMS.

Figure 7-1. Workforce Automation Architecture

The application can easily support different client devices using XSLT stylesheets. Customizing the content conceptually occurs in two steps. Based on where the worker is in the process and his personal preferences, the Web server may apply an XSLT filter to the raw XML content to generate a more focused content document. Then, based on the type of device the worker has, it applies a presentation XSLT to generate an HTML, WML, VoiceXML, or other type of document.

Key Features

This type of workforce automation application uses XML for behavior as well as content. This technique is emerging as one of the most revolutionary benefits of XML. The trading partner coordination application in the next section takes this technique to the extreme.

Development Process

Obviously, supporting worker job tasks with software requires a thorough understanding of those tasks. The analysis takes place in two phases. First, you must analyze the general information requirements faced by the worker when performing a class of tasks. You must fulfill these requirements by acquiring or commissioning the appropriate content, installing it in a CMS, and providing the necessary metadata to facilitate searching. You must also construct the appropriate filter and presentation stylesheets

Once you have satsified the general information requirements, you can begin modeling specific tasks. Usually, you will use a flowchart or other design tool to construct the model, and this tool will then generate an appropriately formatted XML process description. Because these XML documents are for all intents and purposes code, you should manage them through a standard source code respository. As processes evolve, analysts will probably spawn many different versions of each process that you want to track over time. Eventually, you can build up a library of process descriptions that will greatly decrease the time required to deploy new ones.

Schema Source

Creating process definition schemas is extremely difficult, so you probably want to rely on either the schemas from a workforce automation package or a more general workflow specification such as BPML, WSFL, or XLANG. In either case, a graphical modeling tool will make sure that you construct descriptions that conform to the schema. As for content document schemas, you will probably start with generic content schemas from the CMS or industry standards body but then customize them to support the specific information requirements of your workers.

Document Life Cycle

Obviously, the instruction for executing processes and the content to support those processes must both persist. However, their life cycles are slightly different. Humans generate both types, but machines consume process documents, whereas humans consume content documents. Their respective evolutions differ as well. Process documents evolve in accordance with changes in enterprise business policies. Content documents evolve in accordance with changes in worker information requirements. Because the process descriptions are operational data, as discussed in Chapter 5, you probably want to store them in a native XML data store or data server. The filter and presentation stylesheets act as both operational data and content. Depending on the number of stylesheets and system loads, they may have to reside in a native XML data store or data server as well.


Knowledge Management

Business Challenge

For many enterprises, knowledge represents their greatest asset. Unfortunately, this knowledge resides primarily in the brains of employees, making it difficult to control and leverage. To achieve the desired control and leverage, many enterprises turn to knowledge management systems. These systems attempt to catalog important areas of knowledge and make them available to employees throughout the enterprise. Technical support, problem diagnosis and repair, and product design are all areas where knowledge management can decrease costs or increase revenues by making the experience of key employees available to the entire organization.

XML Benefit

There are three fundamental problems in knowledge management: (1) modeling knowledge, (2) leveraging existing documents, and (3) searching the knowledge base. XML delivers advantages in each of these areas. XML schemas provide a convenient mechanism for structuring documents specifically designed to capture knowledge. For a given type of knowledge, such as home accounting software problem diagnosis, airplane engine troubleshooting, or semiconductor manufacturing process design, business analysts use a knowledge modeling tools to define a schema that models the body of knowledge. Employees then encode their knowledge of the domain by creating documents that use the appropriate schema.

The second problem in knowledge management stems from the simple fact that most enterprises already have large volumes of documents that contain knowledge. They need a means of adding to these documents supplemental information that catalogs the knowledge they contain. With XML, business analysts can create a format for catalog metadata about existing documents. These analysts or domain experts then sift through the existing document base and populate the catalog entries. A knowledge management system links the catalog and original information through foreign keys to a document in a DBMS, an HTML link to a document available through the Web, or even an XLink to an XML document library.

The third problem in knowledge management is searching the knowledge base. Because XML is highly structured, it enables far more sophisticated search techniques. Without XML, knowledge management systems are either limited to free-text searches or must include a separate analysis module for every proprietary document format. But once most of the documents apply metadata to their elements using a standard like XML, the system can use a wide variety of statistical, semantic, and artificial intelligence techniques. While most of these advanced techniques existed prior to XML, the rise of XML as a popular content format makes them far more practical.

Architecture

As Figure 7-2 shows, a knowledge management system is composed of a knowledge manager, knowledge sources, and stylesheets for presentation. A knowledge source may be a knowledge document that is a self-contained description of an area of knowledge or a combination of a catalog entry and a source document.

Figure 7-2. Knowledge Management Architecture

When an employee has an experience and wants to enter it into the knowledge base, he requests the appropriate form from the knowledge manager. As mentioned in Chapter 3, XForms is an emerging W3C standard for describing forms and is one possible solution. The knowledge manager could also use standard HTML forms and translate the submissions into an XML format. Each form includes all of the information necessary to populate the corresponding knowledge document. Using a Web browser, the employee fills out the form and submits it to the knowledge manager. The knowledge manager processes the form and creates the knowledge document.

To make existing documents available as part of the knowledge base, there are two possible techniques. Business analysts or domain experts can enter catalog information using forms the same way they would enter a direct experience. Automated agents can also scan the source documents and create catalog documents. The attractiveness of the automated option depends on how effectively the agent can generate metadata from the document content. The process may be effective only if the document contains structured information, such as an entry in a sales contact database. So preparing unstructured text documents for use by a knowledge manager is usually very labor intensive, and you should be selective in the types of text documents that receive such manual treatment.

As knowledge and catalog documents enter the system, the knowledge manager indexes them, using whatever specialized techniques offered by the particular product, and stores them in the knowledge base. Through their Web browsers, employees can then query the knowledge base for help with any problem type supported by the system. Front line technical support representatives would be able to search the combined knowledge of senior engineers. An employee views a knowledge document using the stylesheet appropriate for the combination of the document's schema and his job role. Front line technical support representatives might receive information formatted as a step-by-step procedure, while second line technical support representatives might receive additional contextual information about the underlying nature of the problem. Of course, viewing a source document from a word processor or spreadsheet linked to a catalog document requires a copy of the word processing or spreadsheet application. Employees may also register interest in types of knowledge, and the knowledge manager can then proactively inform employees through e-mail when such knowledge becomes available.

Development Process

You are most likely to evaluate, select, and purchase a knowledge management system from a vendor. Given the sophistication of knowledge search techniques, it's unlikely that you would build your own knowledge management system from scratch. However, you may customize and extend a third-party product.

In most cases, you need schemas that categorize the types of knowledge managed by the system. Designing schemas for knowledge categorization is much more difficult than for business process orchestration applications, discussed subsequently. Business processes are observable. The internal problem-solving processes of employees are not. Therefore, you need specially trained business analysts, called knowledge engineers, to assist in the schema definition process.

A companion task to designing schemas is designing the stylesheets that employees will use to view knowledge documents. Because employees from different organizations may access the same knowledge, you may need multiple stylesheets for each schema to provide customized presentations to employees using the knowledge for different job tasks. Because effectively imparting knowledge requires effectively presenting knowledge, stylesheet design is a crucial part of the development process.

Once you have the schemas, you must put knowledge into the system. Perhaps the biggest challenge is getting employees to spend time entering knowledge or creating catalog entries. Achieving this goal may require an extensive education campaign and special incentives. With the knowledge in the system, there may be some minor education necessary to teach users the best ways to use the system. However, given that they will use Web browsers to access it, the amount of education should be minimal.

Key Features

A distinguishing characteristic of knowledge management compared with traditional Web publishing systems is the fact that users, as opposed to designated document authors, input most of the information in the system. Also, for any given document, there is an asymmetry to the relationship between creator and consumer: A single person creates each document, and many people may consume it. Another important characteristic is that document consumers always find relevant documents through the knowledge manager as opposed to accessing them directly.

Schema Source

In many cases, knowledge engineers either extend existing schemas or create schemas from scratch. In certain specific applications, such as software technical support, third-party vendors may be able to create schemas that are general enough to apply to many enterprises.

Document Life Cycle

Either enterprise employees or automated agents provide the data for the knowledge manager to create documents. Enterprise employees who need assistance from the knowledge base consume these documents. In marked contrast to the other enterprise XML application types, all documents remain alive. The entire point of knowledge management is to accumulate information, so destroying documents does not make sense. Therefore, the knowledge management system must include a document repository capable of handling very large volumes of data. Usually this repository will be a CMS because the response time constraints are not tight enough to warrant a native XML store or data server.


Trading Partner Coordination

Business Challenge

One of the most costly business processes for an enterprise is establishing a new trading relationship. First, the enterprise has to locate a suitable trading partner. Then the enterprise has to negotiate the parameters of the relationship, including the time-consuming process of exploring whether the partner has a compatible business model. Finally, the enterprise must attempt to make the trading process efficient by collaborating with the partner on automation projects.

It would be nice if enterprises could reduce costs and uncertainty by automating this entire process. Unfortunately, the barrier to entry is very high. Each enterprise must first have a publishable model of its business so it can test partner business models for compatibility. Each partner must also use the same negotiation framework. Lastly, each partner must have compatible business messaging software that can execute the negotiated process. Even with all these pieces in place, the enterprise must have a way to mediate all the internal interactions with both human users and software applications necessary to support the trading relationship.

It is possible for large enterprises to custom-develop systems with most of these capabilities. Because potential partners must have the same custom software, this type of automated negotiation could take place only for extensions of existing relationships. Moreover, the complexity of the software involved makes such an undertaking costly to implement and maintain.

XML Benefit

XML can certainly address the last issue of supporting human and software actors. Standard components of the enterprise desktop such as e-mail clients and Web browsers have gained XML capabilities. Therefore, human actors can use these familiar desktop applications as their interface to the workflow system, decreasing learning and ownership costs. XML has also gained acceptance as an integration technology. Later in the chapter, we'll look at the use of XML for application integration. Therefore, XML-based trading partner coordination can easily connect with internal applications.

XML also addresses the problem of having publishable and machine interpretable trading policies. As discussed earlier in the workforce automation section of this chapter, XML documents can drive process execution. So enterprises can encode rules for negotiating agreements and their possible trading processes in XML. Then both human and software actors at other enterprises can evaluate them for compatibility.

XML-based standards make all these features cost-effective. Undertakings like ebXML seek to provide standard frameworks for process negotiation and execution. This standardization lowers costs by opening the way for third-party vendors to amortize the solution costs across enterprises and increases the chances of other enterprises having compatible software.

A side benefit of using XML both to define and to execute trading relationships is the applicability of other XML-related technologies to the resulting business data. For example, Scalable Vector Graphics (SVG) is an XML-based standard for interactive diagrams. It is perfect for business data visualization. You could use the internal and external process descriptions plus accumulated business documents to create an SVG-based system for visualizing various aspects of process flow over time. Because SVG also relies on XML, it is easy to generate such visualizations from data in XML documents.

Architecture

As Figure 7-3 shows, a trading partner coordination system has separate elements for each enterprise that participates. A conductor facilitates the communication of internal actors with other enterprises. This facilitation may include human and software actors. Human actors usually access the system through e-mail clients or Web browsers. In some cases, e-mail clients or browsers may need special plug-ins or other extensions so that they can operate on work products. Digital signatures are a good example of a feature that requires such extensions. Software actors interact with the system either through native XML interfaces or mediated through XML-based integration tools.

Figure 7-3. Trading Partner Coordination Architecture

To establish a trading relationship, the two conductors exchange information about the types of business processes they support. If their associated enterprises can cooperate in one or more areas, they initiate the negotiation process. They present their respective internal actors with the possible trading models and the parameters to negotiate. Software actors may automatically assess the costs of various proposals, but human actors make the final decision. If negotiations are successful, the two conductors confirm the relationship by exchanging the external process description they will both use and move on to the execution phase.

For each type of business document a conductor can exchange with another enterprise, it has a map of internal work products involved in producing that business document. For each work product, it has an internal process description that generates that work product. As the external process description and associated interaction with trading partners brings about the need for an internal work product, it calls up the appropriate process description and facilitates the necessary exchanges among internal human and software actors.

Key Features

The most prominent feature of trading partner coordination systems is the automated negotiation of the trading relationship itself. The coordination of the operational exchanges between enterprises is identical to that of business process orchestration, and the standardization of exchanges of applications within the enterprise is identical to that of application integration. Automating the negotiation itself adds a new dimension of increased efficiency and decreased time-to-market.

Development Process

The first step in the development process is to select, install, and configure trading partner coordination components from appropriate vendors. The complexity of the external coordination and internal integration usually makes it most cost-effective to buy most of the system. You may need multiple components, including a B2B conductor component, an XML application integration component, and a process modeling tool. In rare cases where the potential efficiency to your organization is enormous, you may decide to build some of these components from scratch.

Once you have installed the conductor, you must configure it to manage each internal and external business process. The first step is defining the work products that the conductor can assemble using internal business processes. Each work product consists of one or more schemas. The next step is defining the human and software actors necessary to populate each work product. For human actors, you may have to define XSL stylesheets to control the presentation of each work product. For software actors, you may have to integrate them with your application integration solution. The last step is defining paths among the actors and the allowable actions at each step. Software actors may automatically initiate and populate most of a purchase order once inventory falls below a certain level. But if the purchase order amount exceeds a given amount, it may route it to a human actor for approval.

After you can assemble the necessary work products, you must configure the conductor so that it can negotiate and execute trading exchanges. This process is equivalent to specifying the CPPs and the allowable rules for constructing CPAs discussed in the ebXML section of Chapter 4. The opportunity for automation absolutely relies on the use of a standard B2B framework and associated tools to support it.

Schema Source

Because business processes differ greatly from enterprise to enterprise, it might seem as if an enterprise would have to define its schemas from scratch. However, the emergence of B2B standards like BizTalk and ebXML means that there are standard schemas for most common interactions. ebXML especially strives to provide a complete framework for such interactions. An emerging area of innovation from ebXML comes in the form of automating the trading negotiation process. In the near future, this process should be standardized. While people still have to provide substantial input and final authorization, there will be schemas for automatic mediation of the process.

To define the internal processes and the parameters for allowable external processes, business analysts will use modeling tools. These modeling tools will generate process descriptions that follow a standard vocabulary. BPML, WSFL, or XLANG provide the most benefit for coordinating internal processes. ebXML's BPSS may soon offer a practical means for coordinating processes between partners.

For common business processes, the B2B standards provide standard document schemas that should be sufficient. Within your industry, there may be additional schemas that support the unique characteristics of that industry. In some case, you may want to customize these schemas or create completely new ones to account for your unique competitive advantage. You should use this option judiciously because it naturally decreases the ability of potential trading partners to accept them automatically.

Document Life Cycle

As with the process documents described for workforce automation, humans create them and machines consume them. They evolve with respect to changes in business polices. However, the internal policies governing the performance of job task and the external policies governing trading relationships are substantially different. Trading relationships naturally evolve much faster. Changing business conditions constantly spark the need for new and improved trading processes. Moreover, these processes must cross enterprise boundaries, so the evolution of trading process descriptions can often resemble collaborative authoring: Representatives from different companies work together to optimize the flow of trade constantly. As with workforce automation, these documents provide critical operational data, so they probably will require a native XML store or data server to achieve the necessary scalability.

In addition to the process descriptions, there are external and internal business documents. External exchanges occur between the conductors of two enterprises, whereas internal exchanges occur between the conductor and the internal applications of a single enterprise. External business documents represent implicit business agreements and authorizations to take action. Internal business documents represent only the internal marshalling of resources to satisfy these business agreements and perform these authorized actions. Therefore, it is important to maintain persistent versions of external documents, but internal documents may be transient. Because process documents require a native XML store or data server anyway, it probably makes sense to use the same type of mechanism to store external documents.


Application Integration

Business Challenge

Many enterprises have turned to packaged applications as a solution to rapidly automating business processes. Packaged applications typically support a finite range of business processes, so most enterprises use more than one packaged application. However, to increase efficiency further through automation, enterprises must integrate these packaged applications with each other and with custom application. Typically, achieving such integration requires extensive custom consulting or expensive tools.

XML Benefit

XML provides the foundation for simplifying this process and includes the infrastructure for inputting and outputting structured documents. If applications could agree on the structure for a set of XML documents corresponding to common business entities and transactions, they could work together. The Open Applications Group has developed just such a set of formats, and a number of leading packaged application vendors, such as Oracle, Peoplesoft, and SAP, plan to support them.

XML also offers hope when it comes to avoiding incompatibilities between integration packages. Many vendors already offer solutions for integrating applications. Usually, they translate the internal data representation of each application into a common intermediate representation. Unfortunately, different products use different intermediate representations. Therefore, if two different organizations within an enterprise use different integration packages, they often have trouble making them communicate. Many recently released integration packages use XML as the intermediate format so the chances of making them work together is higher. In many cases, it may simply be a matter of writing the appropriate XSLT transforms.

Architecture

As Figure 7-4 shows, the application integration architecture involves one or more applications communicating over a network. When one application wants to access another application, it sends it an XML-formatted request over SOAP. The target application returns an XML-formatted value or even an entire business document. For applications that cannot natively exchange XML, an XML-enable integration package can act as a proxy. It receives XML requests on behalf of these applications and translates them into their native protocols. It performs the reverse translation for the returned information and for requests from the proprietary application to the XML-enabled applications. However, unlike two XML-enabled applications, those using a proxy can never communicate directly.

Figure 7-4. Application Integration Architecture

The applications shown in Figure 7-4 do not have to be in the same enterprise. You could just as easily integrate an Inventory Management application in one enterprise with an Order Fulfillment application in another enterprise. When inventory falls below a certain level for a given part, the Inventory Management application sends an XML document over the Internet to the Order Fulfillment application at the part supplier. The definition of XML formats for business entities and transactions has the potential to simplify the development of supply chain management applications greatly. Such a system crosses the line from application integration to business process orchestration when you add layers of policymaking and rule evaluation.

Key Features

Application integration demonstrates the usefulness of the human-machine duality of XML documents. Document exchange occurs primarily between applications. However, the documents they exchange represent business entities and transactions. Usually, when two applications exchange documents, these documents are primarily interesting only to the applications themselves. Many application integration applications, however, would be meaningful to business professionals if they were formatted with an XSL stylesheet. The ability of people to audit the machine interaction easily makes enterprises more comfortable with such arrangements and introduces a degree of robustness in the face of border cases where the automated exchange fails and people must get involved.

Development Process

In certain cases, packaged application vendors provide support for XML. In many cases, even though a packaged application doesn't directly support XML, an integration solution can translate native requests to XML requests. In these cases, you must test all the applications in question to ensure that they can exchange information using the exchange protocol. In some cases, you may need to enhance internally developed applications so that they can communicate with other applications using XML. In these cases, you have two primary tasks. First, you create a protocol engine that can format a document request, establish a remote connection to the server application, send the document request to the server, receive the server's response, and extract the interchange document. Second, you create a mapping layer that will translate the data contained in the interchange document into data structures that the application can process.

Schema Source

There are two types of schemas involved in application integration: protocol schemas and payload schemas. Protocol schemas describe the formats of messages on the wire. As discussed in Chapter 4, SOAP is quickly becoming the accepted standard for this format, but there are still some products that use other protocols. Payload schemas describe the formats for business entities or business actions. Because application integration commonly occurs between operational enterprise applications, enterprises with similar operations have similar interchange requirements. Therefore, consortiums within particular industries such as financial services, semiconductor manufacturing, or telecommunications are defining payload schemas. For integrating sophisticated or unique applications, these standard schemas may be insufficient, and you must design them yourself.

Document Life Cycle

An application creates a document in response to a specific request. It then returns the document to the requesting application, which consumes it by translating the information contained in the document into internal data structures. At this point, the document no longer exists unless explicitly saved for auditing. Of course, applications may process the individual data elements in the document and save the data to their own databases.


Data Integration

Business Challenge

Unfortunately, the information constituting logically coherent business concepts such as vendor, customer, product, order, and invoice often resides in several different data sources. This information diffusion causes two problems: (1) enterprise employees find it difficult to execute job functions that require them to assemble a complete representation of such business concepts, and (2) each application must incur the performance cost of assembling the component data into complete representations of such business concepts.

XML Benefit

As with application integration, XML helps data integration solutions avoid incompatibility with each other. An obvious solution to data integration is centralizing the process of synthesizing data from different sources. Unfortunately, whatever service performs this synthesis must provide the resulting information in a format that might itself turn out to be incompatible with that used by other systems. XML addresses this issue. Because Web browsers have become a standard component of the enterprise desktop, there is a strong argument for using XML as the universal data format for human clients. Also, because many of the recent B2B and EAI solutions use XML documents, there is a strong argument for using XML as the universal data format for software clients. The widespread availability of XML-capable clients and its flexibility in structuring information make it possible for XML to become the de facto data interchange format. It also eases the use of other technologies such as SVG, as discussed in the trading partner coordination section. Making all business data available as XML naturally facilitates the use of SVG for business data visualization.

Architecture

The data servers described in Chapter 5 provide the foundation for data integration applications. As Figure 7-5 shows, the architecture for data integration includes multiple data sources, a data server, and multiple clients. The data sources could be files in filesystems, DBMSs, and applications—anything with remotely accessible information.

Figure 7-5. Data Integration Architecture

The data integration server takes data records from these sources, composes them into XML documents, and makes them available to clients. It also accepts updated documents from clients, disassembles them into their component information elements, and instructs the data sources to change the corresponding component data records. The data integration server does not have to use an XML-based mechanism to access and update the information sources. It could also use native database APIs, JDBC, or ODBC.

A client could be a Web server that then passes the documents on to end users. It could also be a B2B application that processes data from multiple sources to generate business messages. It could even be another data source that needs its own set of consolidated data, such as a data warehouse.

Key Features

What distinguishes data integration from other applications is its true middleman nature: It assembles only information; it does not use it as part of a business process. The fact that it does not necessarily use XML documents as the means of exchange with back end information sources is also an important characteristic.

Development Process

In nearly all cases, you will purchase a data server from a vendor. Chapter 5 discussed some of these vendors using a native XML store as a data server. After selecting a platform, there are two important steps in implementing a synthesis application: (1) figuring out what types of business information human and software users need and (2) locating the component data.

The first step is a needs assessment problem. You identify all the different groups that want synthesized information and explore the needs of each. Then you design a set of schemas that meet these needs. Once you know what data you require, you move on to the second step, which is a data access problem. Find out which data sources contain the required data and gain access to those sources.

With schemas in hand and access to the component data, you then map the elements and attributes specified in the schemas to data fields in information sources. The vendor should supply a tool for browsing the structure of information sources and creating the necessary mappings. However, you may have to specify a number of important parameters such as concurrency policy, transaction policy, and access control policy. Finally, you test the system to make sure that the mapping works and that the parameters specified produce the expected behavior.

Schema Source

The origin of schemas depends on why you are doing data integration. If the goal is to synthesize customized information to support different job tasks, you create a schema for each target audience segment. If you need specific formats to support custom applications, you create a schema for each type of data request these applications need to make. If you want to support B2B applications so they can exchange information with other organizations using standard formats, the schema comes from the standards body.

Document Life Cycle

The data integration server creates XML documents based on the requirements of clients. In most cases, it creates these documents dynamically for each request. The document is destroyed, as far as the data integration server is concerned, as soon as it sends the document to a client, although the client might retain one or more copies. As part of the middle tier, the data integration server may also choose to cache documents, but no part of the system ever assumes that a particular document is in the cache.


Chapter 8. Five XML Applications for Vendors

    Executive Summary

    Flexible Content Provisioning

    Information Aggregation

    Application Services

    Configuration and Logging Files

    Distributed Protocol


Executive Summary

A vendor application uses XML to provide a selling benefit by reducing the cost of delivering existing product features or enabling new product features. As with enterprise applications, the best way to understand how to achieve such benefits at your company is to consider some examples. This consideration will provide both inspiration and context for using the technology. Other than XML infrastructure products, vendors do not use XML in isolation. Figuring out appropriate uses for XML in your products requires an understanding of how it operates with other product components.

Table 8-1 lists five examples of important vendor applications where XML can deliver significant selling benefits. The rest of this chapter discusses these applications in detail. The first three apply primarily to vendors of Internet services and concern XML benefits that make new service features possible and existing service features robust. The last two apply primarily to vendors of products and concern XML benefits that make certain software features easier to develop and more interoperable.

Table 8-1. Vendor Applications and XML Benefits
Category Examples Key XML Benefit

Flexible

Content

Provisioning

Content

Documents

  • Portals

  • Online shopping

 Portable user profile definition and flexible content customization

Information

Aggregation

Business

Documents

  • PC component information

  • Schedule consolidation

 Easy format translation and document assembly

Application

Services

Protocol

Documents

  • Outsourced payroll

  • Outsourced HR

 Easier to integrate best-of-breed offerings
Configuration and Logging

Protocol

Documents

  • Component deployment

  • Interaction logs

 Compatibility of development and analysis tools

Distributed

Protocols

Protocol

Documents

  • Peer-to-peer computing

  • SOAP

 Faster development through access to standard formatting infrastructure


Flexible Content Provisioning

Business Challenge

Any content-oriented business model, whether supported by advertising fees or membership dues, relies on a dedicated viewership. Building this viewership requires catering to the needs of the target audience. Serving these needs means delivering a complete content package. For example, a Web site for online sports pools requires substantial supporting content to engage viewers such as real time game scores, team rosters, and sports-related articles. Moreover, because of the wealth of supporting content and the limited screen real estate of any client device, the Web site must customize the view of each user so he sees only the content most important to him. Our online sports pool Web site must filter content to present each user information only on the sports and teams that interest him, perhaps even customizing the presentation to use the colors of his favorite team in whichever sport is currently in season.

Traditional technologies for dynamically generating HTML have some drawbacks for content packaging and customization. The presentation-oriented nature of HTML makes it difficult to integrate third-party content directly and maintain a unique brand. The other alternative is to access third-party data sources, but the variety of possible data sources makes this approach an integration nightmare. Once a Web site manages to assemble all the necessary content, the customization problem arises. Almost always, this customization involves the use of a programming-oriented framework. The generality of these frameworks usually means that they limit the customization to a few predefined screen regions, each having a menu of canned content options.

Most important, HTML supports nonbrowser devices poorly. The rapid increase in the number of these devices with Internet connectivity makes them an important audience segment. Because of HTML's presentation focus, it is difficult to repurpose HTML for smaller-form factor devices. Programming frameworks can construct alternative presentations from data, but again the generality of these frameworks usually leads to limited display conversion options.

XML Benefit

As discussed in Chapter 5, XML dramatically improves the ability of Web sites to acquire content. They can get content through XML-based authoring tools, XML-based syndication, or XML-based data integration. Moreover, content from all these sources uses the same format, so Web sites can easily synthesize coherent packages.

Once a Web site has the content it needs, it must customize the delivery of this content for each user. This customization requires user profiles and rules for interpreting these profiles to filter the available content. XML can offer an advantage here as well. Formatting profiles as XML makes it possible to use the same infrastructure as the rest of the content. More important, it enables syndicates of related sites to exchange profile information to build a better model of user requirements easily. XML can also flexibly encode the rules for selecting a user's customized content with the technique discussed in the workforce automation section of Chapter 7. Using XML to specify behavior makes the customization process more responsive to change and makes rule specification part of the production process rather than the programming process. Putting the power to control content selection in the hands of producers rather than programmers results in a medium much more responsive to changing user needs.

Architecture

As shown in Figure 8-1, there are three main layers. The Web server layer delivers formatted content to the different types of devices. The provisioning layer selects the best content to fulfill each user request by evaluating the client device type, the preferences in the user's profile, and rules on how to match existing content to user needs. The content manager is responsible for managing the content production process by supporting all the necessary content acquisition channels.

Figure 8-1. Flexible Content Provisioning Architecture

Key Features

The key feature of flexible content provisioning is the separation of responsibility between provisioning content to users and acquiring content from providers. This separation is what delivers improved flexibility. Two separate groups can independently address both ends of the information supply chain.

Development Process

Third-party vendors obviously supply the authoring components for static content and stylesheets. They also supply some of the server components. At a minimum, third parties supply the CMS, syndication server, and data server. Certain vendors are updating their personalization and customization engines to support an XML content architecture, although leading-edge Internet services may find them inadequate and need to build their own.

Successful customization depends on effective profile definition. You must create a schema for describing the kinds of preferences users may have and the actions the engine can take based on those preferences. Preferences have three categories. The most obvious are presentation preferences such as "large fonts," "16 colors," and "bullet." Usually, these presentation preferences apply to specific view preferences such as "summary view," "prioritized view," and "drill-down view." The third category, topic preferences, determines the selected content on which these views operate. Example 8-1 shows a possible set of parameters for the online sports pool application mentioned earlier.

Example 8-1
 <Font: Courier New><Interests>
 <Sport>
  <League>NCAAMensBasketball</League>
  <Team>Stanford</Team>
 </Sport>
 <Sport>
  <League>NFLFootball</League>
  <Team>Denver</Team>
  <Team>Minnesota</Team>
 </Sport>
</Interests>

A particularly strong advantage of using XML for Web experience customization is the potential for combining preferences from different sites. The more information you have about a user, the better you can customize the user's experience. If you partner with several different service providers that use XML-based profiles, you can easily pool this information.

Whether you build your own customization engine or buy one from a third party, you still have the problem of configuring it. The first step is to identify target users and gather data on the types of information they want and the points of customization that are important. User programming specialists then encode this user model in the preference framework and create the necessary schema and stylesheet options. They connect the framework to the user interface for setting preferences. Screen designers create a set of style options for each schema and each device. They may use a two-stage stylesheet strategy where a first stylesheet creates an abstract description of the layout and a second stylesheet binds it to a particular device layout language.

Schema Source

To a great extent, the structure of the information is the Web site's competitive advantage. Therefore, you will probably want to design your own schemas for content. You may well design your own profile schema, but joining forces with a group of service providers could be more beneficial because the more information you have about users, the better you can anticipate their needs.

Document Life Cycle

The customization engine creates customized content documents from generic content documents managed by the CMS. Obviously, either the service provider or a third-party content provider preserves these generic documents. The service provider must also preserve the profiles and rules that drive content customization. Because these documents are operational data as defined in Chapter 5, sustaining high loads may require a native XML store or data server. Once the engine generates the customized content documents, users consume them via the Web server. Unless explicitly preserved, these documents are destroyed upon consumption. The whole point of customized publishing is to provide a unique experience based on the most current information available, so it is unlikely that you would save these documents.


Information Aggregation

Business Challenge

Value-added resellers have become an accepted source for products and services. They provide an important function by aggregating the components of various vendors into a complete solution for particular markets. Information aggregators are the digital equivalent of value-added resellers. They add value by combining pieces of information from a number of different sources into packages of information customized for particular markets.

In some cases, the aggregated information may facilitate the flow of physical goods and services. Consider the problem of PC configuration. Thousands of vendors sell components for PCs. Millions of customers want to order PCs with combinations of these components. How can these customers be sure that the combinations of components they order will work together? They certainly don't want to sort through all the product specifications for each component they want. Information aggregators provide the solution. To a certain extent, PC retailers act as the information aggregators and guide customers in the selection of compatible components. However, the sheer volume of component combinations and the rate at which available components change are too great even for most retailers. An independent information aggregator could address this problem by gathering information from all the component vendors and organizing it so that retailers and customers can access it to determine workable configurations of PC components.

In other cases, an aggregator may add value to information that is already valuable. Consider the providers of digital video recorders like TiVo. These providers offer a combination of hardware and software. The hardware enables consumers to record television on hard disks instead of videocassettes, improving the ease-of-use. But the software tremendously magnifies the ease-of-use advantage by giving consumers access to an interactive television schedule. Integrating the schedule with the recorder makes it possible to turn on the television, look for programs, and set the device to record them with a few clicks of the remote control. Because this schedule includes metadata about the programs, it can even predict what programs a user might want to watch and proactively record them. These features depend on having a consolidated schedule for all the different antenna, cable, and satellite channels across the country. In fact, the hardware is actually easy for consumers to build themselves, but they chose not to because they can't aggregate the schedules themselves. The scheduling information already exists, but putting it all together enables a completely new category of product.

XML Benefit

There are two barriers to efficient information aggregation: one on the input side and one on the output side. On the input side, the problem is that the information component suppliers have a wide variety of different sources, each with its own format. Extracting the desired information in a machine-readable format is an onerous task that isn't cost justified in any but the most critical areas. On the output side, the problem is ensuring that the potential information consumers can effectively use the aggregated information in the format supplied.

Not only can XML address both barriers by providing common formats, it introduces an additional value—schema design. Designing schemas that capture the essential features of information components in a particular domain helps information suppliers make their information more valuable. By designing schemas that capture the requirements of information consumers, you gain a competitive advantage. Essentially, you create value by making the market for information more efficient.

Not only do XML schemas provide a convenient mechanism for you to streamline the flow of information, the XML paradigm also provides other essential infrastructure for free. Information consumers certainly want to view the aggregated information, and XSL stylesheets provide the ideal means for that. Not only can you deliver the information to their Web browsers, you can also quickly customize the presentation for different types of information consumers. An important aspect of organizing information is indicating relationships between different pieces. XLink provides a convenient way to specify such relationships independent of the actual information. Your whole service could be creating XLinks that represented a package of documents.

Architecture

Figure 8-2 shows the architecture for a potential information aggregation application. A gatherer process accesses the XML documents from each information provider. In cases where the information provider cannot use XML, either the information provider or the aggregator must create an adaptor. The gatherer then processes the documents, performing any necessary filtering and aggregation. This processing constitutes the unique value. The gatherer then stores the resulting documents in a repository. In response to client queries, a publisher process accesses the appropriate documents in the repository, combines them with the appropriate stylesheets, and sends them to the client. Of course, the gatherer and publisher may be part of the same software executable, but there is a logical separation to their functions. A given information provider may supply more than one information aggregation application, and clients may query information from more than one information aggregator.

Figure 8-2. Information Aggregation Architecture

Key Features

The key feature of information aggregation is the flow of documents from information supplier, to information aggregator, to information consumer. At each step, the information aggregator provides value—first by extracting only the relevant information from the information supplier and then by responding to particular client queries.

Development Process

To start the development of an information aggregation application, there are three critical design tasks. First, you have to figure out what information consumers think is valuable. This process results in schemas for the document types in the repository and accompanying XSL stylesheets for each audience segment. Second, you have to figure out what information suppliers have that is relevant to what consumers want. This process results in schemas for the document types you retrieve from suppliers. Third, you have to figure out precisely how to transform the information that suppliers provide into the information consumers want.

After the design phase, you need to provide access to consumer clients and achieve access to supplier sources. A Web server is probably the easiest way to provide access to clients, although there are other possibilities, such as e-mail. A Web server is also the easiest way to achieve access to sources. Unfortunately, the reality is that most suppliers have information in existing systems and need to use a data integration server to provide the documents. You may well have to build this application yourself if suppliers are unwilling to bear the cost.

Schema Source

In most case, you want to design the schemas for documents you plan to deliver to consumers. However, industry standards could exist for certain types of information you plan to acquire from suppliers, in which case you could leverage these existing schemas. If you have to design schemas for the supplier information as well, you probably want to try to get suppliers to adopt your schemas as an industry standard.

Document Life Cycle

Information suppliers create and maintain their own documents. They may choose to maintain their documents in a repository or to generate them dynamically from back-end sources. Gatherer processes consume these documents and, from them, publisher processes create their own documents. If the publishers have to create these documents in real time, you may want to use a native XML store or data server to maintain the gathered documents to achieve fast response times. You definitely maintain these documents in a repository because they are your primary assets.Information consumers take these documents and potentially archive them at their own sites.


Application Services

Business Challenge

The flexible content provisioning application just discussed addresses the issue of offering content-oriented Web sites that deliver comprehensive packages to users. A parallel issue exists with service-oriented Web sites. Such sites offer the user application functions over the Internet. Outsourced payroll, HR, and ERP applications are all good examples. These sites must also deliver a complete package, but of functionality rather than content. Users want all the functions associated with a particular process in one place.

In most cases, it is simply not cost-effective for a single Web site operator to implement every function that users require. Suppose a provider wants to offer applications to mobile sales forces. Such a provider needs everything from enterprise-oriented functions like contact management, contract processing, and sales force compensation to travel-oriented functions like airline reservations, hotel bookings, and map delivery. Moreover, the degree of integration in an application service is a competitive advantage. There will be an arms race between competing providers to offer an ever-greater number of integrated features.

While an application service operator may be able to find providers of lower-level components, integrating them is a challenge. Most such providers don't construct their applications with third-party use in mind. They all use different conventions for the interface and different back-end infrastructure. Enabling each component provider to contribute is a brute force integration project. Only the most critical applications could justify the cost.

XML Benefit

XML and, more particularly, Web Services solve the problem of integrating multiple Internet-based applications. The reason is simple: standardization. All the third-party providers can use the same standards for describing their interfaces and responding to requests. Even more important, the application service operator can use the same standards for its internal applications. In most cases, an operator needs to own the central functionality of the offering, relying on third-party providers for supporting features. While cutting down on the proliferation of third-party interfaces, Web Services can unify both the internal and external components as well. The operator simply has to implement new functionality using Web Services and provide a Web Services proxy to existing applications. Using the same methodology for invoking functions in every component, whether external or internal, greatly simplifies the architecture, reducing both development and maintenance costs.

Architecture

As Figure 8-3 shows, the architecture for application services is relatively simple because of the standardized interfaces. User requests arrive from their browsers via a Web server. The Web server forwards the request to the Web service gateway. Each gateway may actually run on the same machine as a Web server, or each may run on its own cluster of machines to separate the load balancing needs of each. The gateway disassembles the request into subrequests to the contributing Web services, based on rules stored in its repository. When the responses come back, it assembles them for delivery using stylesheets also in its repository. All third-party Web Services use the standard protocol stack. They may have extensive integration infrastructures of their own, but this implementation remains hidden from the application operator. Hopefully, the application operator implemented all of its internal components using the Web Services protocols natively. If not, it's relatively straight forward to build proxies that translate the native protocols into the standard ones.

Figure 8-3. Application Services Architecture

Key Features

The key feature of application services is that they decompose logical user requests into detailed requests of multiple components. This type of application applies the concept of separating interface and implementation at two levels. The user doesn't realize that the responses to his requests come from an implementation including multiple components. Each component further hides its implementation behind its public Web Services interface.

Development Process

The beauty of application services is that the use of a standard protocol stack eliminates much of the development. Implementations of these protocols for use in internal components are readily available. Connecting to external components requires only one integration paradigm for all components rather than one for each component. Therefore, the only significant development task is implementing the framework for disassembling user requests into component requests and reassembling the component responses into a user response. You can accomplish the first by designing a schema that maps the user request to component requests and component responses to user responses. A framework interprets these mappings at run time for disassembly. You could create a framework for the reassembly, but you could also use XSLT.

The development of the internal components is independent of this framework development process. Certainly, if you are beginning development of these components, you should use the Web Services protocols. If you have already developed some of them, you have to write Web Services proxies. A proxy bridges the standard protocols to the proprietary protocol of a particular application. If this proprietary protocol follows typical remote procedure call design, creating this proxy should be straightforward. If it requires extensive transaction and security support, you may well have to create your own SOAP extensions to propagate this additional information.

Schema Source

The schemas for the standard protocols clearly come from standards bodies. The schemas for message payloads exchanged via SOAP come from the component developer and are documented in its WSDL definition. For internal components, you will design these schemas. You must also design the schemas for the mapping rules.

Document Life Cycle

Because this application type relies on the Web Services architecture, its document life cycle follows that of Web Services. Each service provides interface definitions that persist, so other applications can discover the interface and use the application. But the SOAP messages produced by one component and consumed by another are transitory except for logging. The rules and stylesheets, of course, persist. As with a number of the other example applications, the rules are operational data, so you want to consider the use of a native XML store or data server as their persistence mechanism. If you do choose one of these mechanisms, you probably also want to use them for storing the interface definitions and stylesheets.


Configuration and Logging Files

Business Challenge

Many software developers use configuration files, so users or administrators can specify application execution parameters. Such files are often referred to as "configuration files," "deployment descriptors," or "property sheets." For example, the file may describe the properties of a software component or the execution options for an application. A common problem in using configuration files is enabling third-party development tools to create them. For the development tool provider, a significant amount of work is often necessary to produce a properly formatted file.

Software developers use logging files to track such information as usage, application statistics, and debugging data. A common problem in using logging files is enabling users or administrators to view them and third-party analysis tools to process them. A significant amount of work is often necessary to interpret the file format properly. For example, it is often hard for administrators to reconstruct what happened in a user's Web session. If the user's shopping cart suddenly dies with thousands of dollars of merchandise selected, you might want to know why. But reconstructing the series of electronic events from the Web server log, the application server log, the DBMS log, the network logs, and the machine logs is almost impossible because they are all in vastly different formats.

XML Benefit

Using XML as the format for application input and output files makes it easier for everyone. By using off-the-shelf XML processors, software developers avoid the low-level work of file manipulation. More important, XML schemas make it possible to define file formats quickly and unambiguously. With access to the schema, software development tool and analysis tool vendors can leverage off-the-shelf processors to reduce the amount of work necessary to produce and interpret different file formats. Also, by defining an XSL stylesheet for input and output files, software developers make it easy for users and administrators to view the information in the files with their Web browsers. Sun's J2EE standard uses an XML deployment descriptor to configure the transaction and security properties of server-side Java components. A small company called TeaLeaf Technologies has an XML logging format that enables customers to address the problem of reconstructing Web sessions.

Architecture

As shown in Figure 8-4, deployment tools create XML configuration documents and store them on disk. The application then reads them from the disk and uses the information they contain to control application execution. As the application generates appropriate data, it saves XML logging files to disk. Analysis tools or Web browsers then access the recorded information. Note that the same process is possible over a network instead of through the filesystem. Administrators could centralize the control of applications by pointing them to a networked repository of configuration files. Distributed application nodes could send logging over the network to a central location to facilitate the analysis of information from all nodes.

Figure 8-4. Configuration and Logging Files Architecture

Key Features

The distinguishing feature of XML configuration and logging files is that they are directly related to application execution. XML input files control application execution. XML output files contain the results of application execution.

Development Process

For each type of file, you have to map the data types to an XML schema. For configuration files, your developers must write the code that takes the data extracted by the XML processor from input documents and puts it into the appropriate internal data structures. For logging files, your developers must write the code that takes the output data from its internal data structures and passes it to the XML processor so that it can construct output documents.

Schema Source

In some cases, there may be standard schemas. Some software component standards include schemas that define the format for the configuration file. Certain types of output files such as transaction or security logs might also have standard schemas. But because of the wide variety of possible configuration and logging files, as well as the ease of using XML for this purpose, you may often design such schemas yourself.

Document Life Cycle

Software development tools or even text editors create input documents and save them to disk. The application then consumes these input documents during application execution. Also, during execution, the application creates output documents and saves them to disk. Analysis tools consume these output documents. Both input and output documents remain on disk until explicitly destroyed. Configuration and archiving tools may also maintain repositories of these documents.


Distributed Protocol

Business Challenge

Typically, the different nodes of a distributed application communicate using an application-level protocol running over standard networking protocols. Often, the nature of the application requires the use of a particular application-level protocol. While SOAP is the standard for general XML messaging, you might need a more specific protocol. Sometimes, you would define your protocol on top of SOAP. An electronic commerce application may need to exchange information between the client and server to specify the contents of an order and authorize payment. Such a protocol would run on top of SOAP. Other times, you may need to define an alternative to SOAP. Different nodes of a distributed application server may need to exchange hardware utilization statistics to perform effective load balancing. Such a protocol probably has performance requirements different from SOAP.

XML Benefit

A large part of implementing an application-level protocol is designing message formats and implementing the engine that will take data from internal data structures and put it in the appropriate message format. XML takes care of both these problems. XML elements and attributes already provide the basic model for indicating what the information in a protocol message means. Software developers simply have to design a schema for each particular message type. XML processors take care of making sure that message instances have the appropriate format. Moreover, processors extract the information in these messages and put it into internal data structures that the application logic can manipulate. By using XML, software developers gain enormous leverage in the design and development of application-level protocols.

Developers also get the benefits of XSLT. Often, debugging distributed applications can be very difficult because it's hard to decipher the messages exchanged. When developing XML-based protocols, developers can simply specify an XSLT stylesheet for each message type, and they can view protocol messages in an easily readable format on any XML-capable Web browser.

Architecture

As Figure 8-5 shows, the architecture of XML-based distributed protocols is relatively simple. As with most application-level protocols, they probably leverage the standard TCP/IP networking protocols and the corresponding network drivers provided by operating system vendors. They also allow developers to use HTTP as the message exchange protocol. HTTP engines are widely available, and most security firewalls provide options for HTTP tunneling, making it an attractive option. This is the approach that SOAP uses. The processing of information extracted from the custom formats occurs in a library. Finally, developers write their application so that it accesses the information from the library, performs any necessary processing, places the results in a data structure representing the response, and instructs the library to encode the response message as an XML document.

Figure 8-5. Distributed Protocol Architecture

Key Features

The key characteristic of an XML-based distributed protocol is that remote messages exchanged over the network are formatted as XML documents. These messages are extremely short lived.

Development Process

Implementing an application-level protocol requires first specifying the allowable message flow. In operation, such a protocol consists of messages flowing between two nodes in the distributed application. These messages must occur in a specific order: The nodes must exchange handshake messages first, then exchange authentication messages, and so forth. After designing the message flow, you must design the message formats. For XML-based protocols, schemas specify the format of different types of messages.

With these elements in place, you must choose a lower-level protocol for message exchange. They can exchange messages over raw TCP/IP sockets or use a higher-level protocol such as HTTP or even SOAP wrappers. Then they must implement the control logic that governs the assembly, sending, receiving, and disassembly of messages.

Schema Source

For a given protocol, message types have a corresponding schema. In some cases, a protocol may be an industry standard, with the schemas part of the standard. In other cases, the protocol may be unique to a particular application, and you must specify your own schemas.

Document Life Cycle

The node sending a message in a distributed application creates the corresponding document. The node receiving the message consumes the document, at which point the document would usually be destroyed. An application could archive a complete or partial transcript of protocol messages, preserving the corresponding documents. However, the large volume of such documents would typically make such an arrangement cumbersome except perhaps during testing and debugging.


Previous Section

Glossary

Application programming interface (API)

A set of functions that an application or component exposes to external software. Programmers use an API when they create software that accesses the functionality of an application or component.

Attribute

A piece of metadata attached to an element.

See also [Attribute specification]

Attribute name
See [Attribute specification]
Attribute specification

An attribute name and corresponding attribute value in the start tag of an element.

Attribute value
See [Attribute specification]
Business object modeling

The process of representing business entities and business processes as objects, in the object-oriented programming sense, to facilitate the integration of different business application software.

CDATA

Abbreviation for character data. A string of alphanumeric characters that may not contain any white space characters.

Content model

The allowable type of content that an element may have. Possible content models include element content, data content, empty content, and mixed content.

Document element

The topmost element in an XML document's element hierarchy. All other elements within an XML document are part of a document element's content.

Document Object Model (DOM)

The W3C standard that describes a generic model that programming languages can use to interact with hierarchies of document data.

Document type definition (DTD)

A specification of the rules a group of XML documents must follow to be valid with respect to the particular DTD. Rules include the allowable types of elements, their content models, their attributes, and the allowable values of these attributes.

Element

The fundamental unit of content in an XML document. An element is composed of the element name, element content, and any specified attributes.

Element content

The content associated with an element. It appears between the start tag and end tag of the element.

Element declaration

A statement in a DTD that declares an element name and a content model for the element.

Element name

The name associated with an element. In an XML document, it appears in both the start tag and end tag of the element.

EMACS

An extensible, cross-platform text editor with many features useful for writing software code.

Entity

An XML document structuring mechanism that specifies the substitution of an entity name with a fragment of content.

Entity name
See [Entity]
Extended link

In XLink, a link that has more than one target.

Extensible Stylesheet Language (XSL)

The W3C standard for describing paginated presentation rules that apply to XML documents.

Extranet

A portion of the Internet used by an organization to foster communication with its suppliers, large customers, and strategic partners.

Formatter

A software component that takes a formatting object tree as input and produces paginated output in such formats as PDF and Postscript.

Formatting object tree

The hierarchy of format-specific elements created when an XSL processor applies a stylesheet to an XML document. This hierarchy is independent of any particular page rendering technology.

FTP

Abbreviation for File Transfer Protocol. A mechanism for uploading and downloading files between two computers connected by the Internet.

Href

The HTML syntax for specifying the target document of a hypertext link as in <A Href="http://www.foocompany.com">.

HTML

Abbreviation for Hypertext Markup Language. The language used to author Web pages.

HTTP

Abbreviation for HyperText Transfer Protocol. A mechanism for making requests and sending responses between two computers connected to the Internet. Commonly used for transferring HTML files from a Web server to a Web browser.

HTTP 200 Return

When a server's HTTP method responds to a client, an HTTP 200 Return indicates the successful execution of the method.

HTTP POST

An HTTP method where the client submits data, such as from a form page, to the server.

HTTP tunneling

The process of wrapping protocol messages in HTTP messages. Commonly used to allow the use of a protocol across a security firewall.

IDE

Abbreviation for Interactive Development Environment. A tool for developing software applications that includes visual tools such as browsers, inspectors, and probes.

IETF

Abbreviation for Internet Engineering Task Force. The standards body in charge of the basic Internet protocols such as TCP/IP and HTTP.

Information and Content Exchange (ICE)

A trade group standard for managing the syndication of content over the Internet.

Information engineering

The process of developing a comprehensive model of the information used within an organization to facilitate the integration of business application software.

Internet

The federation of interconnected public and private networks running the standard Internet Protocol.

See also [TCP/IP]

Intranet

A portion of the Internet used to connect systems within the confines of a particular organization and not accessible to outsiders.

J2EE

Abbreviation for Java 2 Enterprise Edition. A standard from Sun Microsystems that specifies a complete server development and deployment model for Java.

JDBC

The standard Java programming API for accessing databases.

Large Objects

When databases must handle large composite datatypes like text or images, they often treat them as large objects. Typically, they store metadata about the object separately from the object itself. They may also supply manipulation functions.

LDAP

Abbreviation for Lightweight Directory Access Protocol. An IETF standard describing both a lightweight version of the X.500 data model and the wire protocol for interacting with directories that use this model.

Lower Camel Case

The naming convention where variable names are constructed by connecting words, using an initial lowercase letter and an uppercase letter at the beginning of each word, as in "lowerCamelCase."

MathML

Abbreviation for Mathematics Markup Language. A W3C standard that describes an XML vocabulary for indicating the semantic and presentation properties of mathematical expressions.

Metadata

Literally, "data about data." Data used to indicate the role or meaning of other data. Database schemas and XML attributes are examples of metadata.

MIME

Abbreviation for Multipurpose Internet Mail Extensions. The IETF standard that governs how to describe the type of a document and encode it for transfer over the Internet.

Namespace

The association of names in a distributed software system with unique segments to avoid naming collisions. For example, XML uses URIs such as "http://www.foocompany.com/names/acct-REV10," and Java uses the Internet domain name hierarchy such as "com.foocompany."

Namespace name

In XML, a URI that defines the unique segment for a namespace.

Namespace prefix

In XML, the prefix that appears before element and attribute names to indicate that they belong to the corresponding namespace.

Naming collision

In distributed software systems, where two software entities developed by different organizations use the same name. For example, the accounting department and fulfillment department of a company might both use the XML element name "status" for logically different elements.

Nonrepudiation

The information security property whereby one can confirm that a particular entity performed an action. Analogous to physical signatures and implemented electronically through cryptographic signatures.

ODBC

A standard programming API, developed by Microsoft, for accessing databases.

PCDATA

Abbreviation for parsed character data. A string of alphanumeric characters that may contain white space characters

PDF

Abbreviation for Portable Document Format. A standard format, developed by Adobe Systems, for representing documents in a platform-neutral manner.

Postscript

A standard format, developed by Adobe Systems, used to represent high-resolution page layouts.

Processing instruction

A part of an XML document's prolog that enables document authors to describe the processing requirements of the document to external software applications.

Prolog

The first part of an XML document that contains processing instructions and document type declarations.

Resource Definition Framework (RDF)

A W3C standard for defining the types of resources provided by an Internet application and how to access them.

Root element
See [Document element]
SCSI

Abbreviation for Small Computer System Interface. A high-speed standard for connecting peripherals to computer systems.

Shared context

An agreement among multiple parties governing the format and interpretations of messages they plan to exchange in the future. In XML, DTDs and schema provide shared context.

Simple API for XML (SAX)

A de facto standard for event-based XML processing.

Simple link

In XLink, a link that has only one target.

Skunkworks

A colloquial term used to identify unofficial or secret projects, usually performed by a small group of highly skilled engineers. Originally refers to the division of Lockheed that worked on top-secret U.S. military airplanes.

SMIL

Abbreviation for Synchronized Multimedia Integration Language. A W3C standard that describes an XML vocabulary for choreographing multimedia presentations.

S/MIME

Abbreviation for Secure Multipurpose Internet Mail Extensions. The IETF standard that describes how to use cryptographic techniques to secure MIME documents.

SMTP

Abbreviation for Simple Mail Transfer Protocol. An Internet protocol for transferring electronic mail between mail servers.

String

A datatype declaration for strings of alphanumeric characters in many software programming languages.

Stylesheet

In XML and other structured document paradigms, a specification for which pieces of content to display, and the formatting with which to display them.

SVG

Abbreviation for Scalable Vector Graphics. A W3C standard that describes an XML vocabulary for portable two-dimensional graphics.

Tag

In a document markup language, a piece of metadata attached to an element that defines the boundaries of element content.

TCP/IP

Abbreviation for Transmission Control Protocol/Internet Protocol. A standard low-level protocol combination used for communication in many networking applications.

Transaction Boundaries

Between the transaction boundaries of a request to an application or database, either all of the requests must succeed, or none of them must.

Unified Modeling Language (UML)

A standard from the Object Management Group that describes how to represent information during object-oriented analysis and design.

Uniform resource identifier (URI)

The Internet standard for representing the type and logical location of networked resources. For example, the URI "http://www.foocompany.com/home.html" indicates the file pathname, domain name, and connection protocol for Foo Company's Web home page.

Upper Camel Case

The naming convention where variable names are constructed by connecting words, using an initial uppercase letter and an uppercase letter at the beginning of each word, as in "UpperCamelCase."

Valid

An XML document that is well-formed and conforms to the rules of a particular DTD or schema.

Validating processor

A processor that verifies whether an XML document conforms to the rules of a particular DTD or schema.

Web Distributed Authoring and Versioning (WebDAV)

The W3C specification that describes how authoring clients should interact with servers to manage content contribution, versioning, and access.

Well-formed

An XML document that follows all of the XML syntax rules.

World Wide Web Consortium (W3C)

A cooperative industry organization that defines standards relevant to Web applications.

X.500

The set of standards from the International Organization for Standardization related to directory services.

XML Linking Language (XLink)

The W3C specification that describes how to define links between XML documents.

XML Namespaces

The W3C specification that describes how to define namespaces in XML documents.

XML Path Language

The W3C specification that describes how to specify a particular element using its path from the document element.

XML Pointer Language (XPointer)

The W3C specification that describes how to specify a particular piece of content with more specificity than a particular element or attribute node.

XML Processor

An XML software program that processes text, determines its logical meaning, and creates programming data structures to represent this meaning.

XML Query

The W3C specification that describes how to query collections of XML documents based on their structure and the values of their elements and attributes.

XML Schema

The W3C specification that describes how to define the structure of XML documents and the datatypes used within documents with more specificity than DTDs.

XSL Transformations (XSLT)

The W3C specification that describes how to transform XML documents from one format to another. For example, using XSLT, developers could specify how to transform a document valid with respect to one schema into a document valid with respect to a different schema.3
    Previous Section