• | Table of Contents
Using XML with Legacy Business Applications
By Michael C. Rawlins
Publisher | : Addison Wesley
Pub Date | : August 08, 2003
ISBN | : 0-321-15494-0
Pages | : 624

"This volume offers relentlessly pragmatic solutions to help your business applications get the most out of XML, with a breezy style that makes the going easy. Mike has lived this stuff; he has a strong command of the solutions and the philosophy that underlies them.">-Eve Maler, XML Standards Architect, Sun Microsystems

Businesses running legacy applications that do not support XML can face a tough choice: Either keep their legacy applications or switch to newer, XML-enhanced applications. XML presents both challenges and opportunities for organizations as they struggle with their data.

Does this dilemma sound familiar? What if you could enable a legacy application to support XML? You can. In Using XML with Legacy Business Applications, e-commerce expert Michael C. Rawlins outlines usable techniques for solving day-to-day XML-related data exchange problems. Using an easy-to-understand cookbook approach, Rawlins shows you how to build XML support into legacy business applications using Java and C++. The techniques are illustrated by building converters for legacy formats. Converting CSV files, flat files, and X12 EDI to and from XML will never be easier!

Inside you'll find:

  • A concise tutorial for learning to read W3C XML schemas

  • An introduction to using XSLT to transform between different XML formats

  • Simple, pragmatic advice on transporting XML documents securely over the Internet

For developers working with either MSXML with Visual C++ or Java and Xerces:

  • See Chapter 3 for a step-by-step guide to enabling existing business applications to export XML documents

  • See Chapter 2 for a step-by-step guide to enabling existing business applications to import XML documents

  • See Chapter 5 for code examples and tips for validating XML documents against schemas

  • See Chapter 12 for general tips on building commerce support into an application

For end users who need a simple and robust conversion utility:

  • See Chapter 7 for converting CSV files to and from XML

  • See Chapter 8 for converting flat files to and from XML

  • See Chapter 9 for converting X12 EDI to and from XML

  • See Chapter 11 for tips on how to use these techniques together for complex format conversions


• | Table of Contents
Using XML with Legacy Business Applications
By Michael C. Rawlins
Publisher | : Addison Wesley
Pub Date | : August 08, 2003
ISBN | : 0-321-15494-0
Pages | : 624

Copyright
Advance Praise for Using XML with Legacy Business Applications
Preface
 | Acknowledgments
Chapter 1. Introduction
 | The Problem
 | What Do We Need in a Solution?
 | The Overview of a Solution
 | Architecture
 | How to Use This Book
 | What You Need to Use This Book
 | For Developers
 | How You Can Use the Utilities and Code
 | References
 | Resources
Chapter 2. Converting XML to CSV
 | Requirements
 | Running the Utility
 | Sample Input and Output
 | Design
 | Java Implementation
 | C++ Implementation
 | Comparing the Java and C++ Implementations
 | Enhancements and Alternatives
 | Resources
Chapter 3. Converting CSV to XML
 | Requirements
 | Running the Utility
 | Sample Input and Output
 | Design
 | Java Implementation
 | C++ Implementation
 | Comparing the Java and C++ Implementations
 | Enhancements and Alternatives
 | Resources
Chapter 4. Learning to Read XML Schemas
 | Rope, Anyone?
 | A Few Different Document Approaches
 | DTD Refresher
 | Foundation Concepts and Terminology
 | General Features
 | Simple Content Elements
 | Complex Content Elements
 | Understanding Namespaces
 | Structuring Schemas
 | Less Commonly Used W3C XML Schema Language Features
 | Is There Data or Not?
 | Reference
 | Resources
Chapter 5. Validating against Schemas
 | Requirements
 | Running the Utilities
 | Design
 | Java Implementation
 | C++ Implementation
 | Examples of Input Validation Failures
 | Resources
Chapter 6. Refining the Design
 | Why Refine the Design?
 | Making XML the Common Format
 | Analyzing the Legacy Non-XML Grammars
 | Describing the Legacy Non-XML Grammars
 | Representing the Legacy Non-XML Grammars in XML
 | Additional DOM Processing Considerations and Strategies
 | Multilingual Issues
 | Error Handling Strategy
 | High-Level Design
 | Detail Design
 | Java Implementation
 | C++ Implementation
 | References
 | Resources
Chapter 7. Converting CSV Files to and from XML, Revisited
 | CSV to XML: Functionality and Operation
 | XML to CSV: Functionality and Operation
 | Describing the File Formats
 | Schema Examples
 | High-Level Design Considerations
 | CSV to XML: Detail Design
 | XML to CSV: Detail Design
 | New DataCell Methods and Derived Classes
 | Java Implementation
 | C++ Implementation
 | Enhancements and Alternatives
 | Resources
Chapter 8. Converting Flat Files to and from XML
 | Flat File to XML: Functionality and Operation
 | XML to Flat File: Functionality and Operation
 | Describing the File Formats
 | Schema Examples
 | High-Level Design Considerations
 | Flat File to XML: Detail Design
 | XML to Flat File: Detail Design
 | New DataCell Methods and Derived Classes
 | Java Implementation
 | C++ Implementation
 | Enhancements and Alternatives
 | Resources
Chapter 9. Converting EDI to and from XML
 | Overview of the X12 EDI Syntax and Standards
 | X12 to XML: Functionality and Operation
 | XML to X12: Functionality and Operation
 | Describing the File Formats
 | Schema Examples
 | Supplemental Data Store for Control Numbers
 | High-Level Design Considerations
 | X12 to XML: Detail Design
 | XML to X12: Detail Design
 | New DataCell Methods and Derived Classes
 | Java Implementation
 | C++ Implementation
 | Enhancements and Alternatives
 | Reference
 | Resources
Chapter 10. Converting from One XML Format to Another with XSLT
 | Why XSLT Is Important
 | XSLT Overview
 | A General Approach to Using XSLT
 | XPath Basics
 | Structuring Stylesheets
 | A Bit of Housekeeping
 | Manipulating Content
 | Solving Typical Navigation Problems
 | Advanced Techniques for Processing Content
 | Handling Namespaces
 | Calling Non-XSLT Procedures
 | References
 | Resources
Chapter 11. Using the Conversion Techniques Together
 | Pipe and Filter Revisited
 | Sample Conversion Scenarios and Scripts
 | Building a System: Babel Blaster
 | Resources
Chapter 12. Building XML Support into a Business Application
 | What Should Be "XMLized"?
 | Devising an Architecture
 | Selecting the XML Format
 | Changing Your Code
 | What about Databases?
 | Other Approaches and APIs
 | Non-XML Issues
 | Resources
Chapter 13. Security, Transport, Packaging, and Other Issues
 | Some General Observations about Security
 | Dealing with Security
 | Transport
 | Packaging
 | Common Combinations for Security, Transport, and Packaging
 | Emerging Technologies
 | What This Means for You
 | Reference
Appendix A. GNU General Public License
 | Preamble
 | Terms and Conditions for Copying, Distribution and Modification
 | How to Apply These Terms to Your New Programs
Appendix B. Pseudocode Conventions
 | Assignment Operator
 | Mathematical Operators
 | Equality and Relational Operators
 | IF Selection Structures
 | Repetition Structure
 | Case Selection Structure
Appendix C. COM Essentials for the Non-COM Programmer
 | #import and Smart Pointers
 | Initialization and Resource Management
 | Exceptions and Errors
 | COM Classes and Data Types for C++
Bibliography
 | References
 | Resources
Credits
 | Chapter 1
 | Chapter 4
 | Chapter 5
 | Chapter 6
 | Chapter 8
 | Chapter 9
 | Chapter 12


Copyright

Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in this book, and Addison-Wesley was aware of a trademark claim, the designations have been printed with initial capital letters or in all capitals.

The author and publisher have taken care in the preparation of this book, but make no expressed or implied warranty of any kind and assume no responsibility for errors or omissions. No liability is assumed for incidental or consequential damages in connection with or arising out of the use of the information or programs contained herein.

Unless otherwise noted, the opinions presented in this book are those of the author. They should not be construed as the positions of any organization with which he may be affiliated.

The publisher offers discounts on this book when ordered in quantity for bulk purchases and special sales. For more information, please contact:

U.S. Corporate and Government Sales

(800) 382-3419

corpsales@pearsontechgroup.com

For sales outside of the U.S., please contact:

International Sales

(317) 581-3793

international@pearsontechgroup.com

Visit Addison-Wesley on the Web: www.awprofessional.com

Library of Congress Cataloging-in-Publication Data

Rawlins, Michael C.

Using XML with legacy business applications / Michael C. Rawlins.

p. cm.

ISBN 0-321-15494-0 (Paperback : alk. paper)

1. XML (Document markup language) 2. Business—Computer programs. I. Title.

QA76.76.H94R375 2003

005.7'2—dc21

2003010094

Copyright © 2004 by Pearson Education, Inc.

All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form, or by any means, electronic, mechanical, photocopying, recording, or otherwise, without the prior consent of the publisher. Printed in the United States of America. Published simultaneously in Canada.

For information on obtaining permission for use of material from this work, please submit a written request to:

Pearson Education, Inc.

Rights and Contracts Department

75 Arlington Street, Suite 300

Boston, MA 02116

Fax: (617) 848-7047

Text printed on recycled paper

1 2 3 4 5 6 7 8 9 10—CRS—0706050403

First printing, August 2003

Dedication

This book is dedicated to all of the small to medium enterprises and others who are still waiting to see the benefits of XML

and

to Dick, Don't Panic!


Advance Praise for Using XML with Legacy Business Applications

This volume offers relentlessly pragmatic solutions to help your business applications get the most out of XML, with a breezy style that makes the going easy. Mike has lived this stuff; he has a strong command of the solutions and the philosophy that underlies them.

—Eve Maler
XML Standards Architect
Sun Microsystems

Unlike other books that simply cover XML standards and specifications, this book shows you how to actually use XML to solve real-world problems. Developers trying to see through all the hype and discover the real utility of XML will finally appreciate having a book that uses industrial-strength examples to make it all crystal clear.

—Steve Vinoski
Chief Engineer, Product Innovation
IONA Technologies

The single most clear and concise discussion of XML for business messaging I have ever seen! The book is a must for anyone faced with integrating XML into business applications at the code level.

—R.T. Crowley
Senior Vice President, Research Triangle Software, Inc.
Chairman of the Board, The Electronic Commerce Institute
Author of EDI: Charting a Course to the Future

I don't know Mike Rawlins but I can tell by reading this book that he has a teacher's instinct. As any good teacher would, Mr. Rawlins uses a simple problem to reveal a complex body of material. In this case, the easily stated problem is file conversion and the rich, well-illuminated subject is XML. This book gives programmers a clear, well-marked path to XML proficiency.

—Joseph Maguire Consultant, Author, Trainer
Patterns & Practices Columnist, Smart Solutions Magazine.

Michael Rawlins handily demystifies the complexities of XML technologies while stepping us in detail through XML solutions to legacy application integration problems, designed around real-world business scenarios. He has also produced a thoroughly enjoyable book that is fun to read—outstanding.

—Daniel Appelquist
Senior Technical Architect at Vodafone
Author of XML and SQL: Developing Web Applications


Preface

Someday most business applications will be able to read and write XML files. Until that happens, you are probably going to need techniques and utilities like those presented in this book. This book is for people who use business applications that don't currently support XML and for people who develop those applications and want to build XML support into them. It is designed to provide a tool kit of techniques and utilities that can help you perform common enterprise application integration (EAI), business-to-business (B2B), or electronic data interchange (EDI) data conversion operations using XML.

Nothing in this book is rocket science. Any good programmer with experience in the relevant technologies could develop any of these techniques and utilities. The point of this book is that I've done it so that you don't have to. As is often said, good programmers develop good programs. Better programmers steal what they can and modify it. Steal this code.

If you are a user of a business application and already have or can afford to procure a capable EAI or EDI software package, you probably don't need this book. However, if you have only some simple needs that don't justify the purchase of such a package, or if for some other reason you don't want to or can't afford to spend thousands of dollars to purchase one, then this book is for you. And if you are interested in an open, nonproprietary, standards-based, and portable approach to data conversion, then this book is for you, too.


Acknowledgments

Many people helped in the production of this book. First, I want to thank my editors, Mary O'Brien and Shelley Kronzek. Without Shelley's enthusiasm and encouragement, I don't think this project ever would have gotten off the ground. Without Mary's help and persistence, I might not have finished yet. To Chrysta Meadowbrooke, thank you very much for your excellent copyediting. I also want to thank the rest of the staff at Addison-Wesley, particularly Brenda Mulligan for her help in many small ways.

My thanks to my technical reviewers—Eve Maler, Daniel K. Appelquist, Steve Vinoski, and Cameron Laird. Your candid comments helped make this a much better book. Innumerable thanks also to the Computer Science faculty at the University of Texas at Dallas, in particular to Dr. Lawrence Chung. Yes, you can teach an old dog new tricks. And thanks to many professional colleagues too numerous to mention by name, but especially those from the XML Forum of the Postsecondary Electronic Standards Council, ANSI ASC X12, the OASIS UBL Technical Committee, and the ebXML Working Group.

Finally, I want to thank my good wife, Diana, for her patience, support, and tolerance for my skewed priorities as I finished up this project.


Chapter 1. Introduction

You know what a "legacy application" is? It's one that works.

—Bill Cafiero

This chapter introduces the approach presented in the book and discusses the business and technical rationales for it. Based on these, the overall features of the approach and the major design decisions are reviewed. The chapter concludes with a description of the primary audiences and suggestions for how to use the book.


The Problem

So, what's the problem? You probably picked up this book (or are browsing this sample chapter on the Web) because the title sounded like it might have something to do with your situation. The Preface probably fleshed out that impression a bit more. By now you probably think you have the book pretty well scoped out, and you may not be that far from the truth. But to save you a bit of time, let me be very specific about the problem (or, more accurately, the problems) that this book addresses.

The primary problem is this: You have one business application that imports and exports data in XML formats, and one that doesn't. You need to make these two applications talk to each other. The XML-enabled application may be yours, or it may belong to someone with whom you (or your application's users) need to exchange data. (Imagine an important customer or government agency sending you a letter that says, "You will receive orders from us in XML format by January 15 or be assessed a $50 penalty for each paper order.") The legacy application is most probably yours. However, the shoe may be on the other foot: Your application speaks XML, but the other guy's application doesn't. He expects you to deal with it. (Imagine an important customer sending you a letter that says, "You will receive orders from us by EDI by January 15 or be assessed a $50 penalty for each paper order.")

There are really two perspectives to this primary problem, and this book addresses both. According to one perspective, the application developer should just fix the stupid application so it can speak XML. From an end user's viewpoint, this is perfectly reasonable, and I'll deal with the problem from this perspective.

From the other perspective, whoever developed the application can't or won't support XML by the time you need it (which might have been last week). There may be several reasons for this. The vendor of a commercial package may not see sufficient market demand. The vendor may have retired the product or, even worse, gone out of business. If the application was developed in-house, the original developers may be gone, retired, or dead. So, there are quite a few reasons why you might need to come up with a solution on your own. In this book I'll also deal with the problem from this perspective.

That, in about 400 words, sums up the primary problem this book addresses. However, in solving this problem from the perspective of someone who needs to come up with a solution on their own (as opposed to that of the developer who just needs to fix the stupid application), we find that the solution might apply to similar problems. Why is that? Well, if we have a method to go from a legacy flat file to XML and a method to go from XML to a legacy flat file, we have a method to go from a legacy flat file to a different legacy flat file (with XML in between). The same thing holds for other common formats such as comma-separated values (CSV) or Electronic Data Interchange (EDI). By coming up with a solution to the primary problem, we find that we have a general solution for all sorts of file format conversion problems. That is also what this book is about.

Before we leave the problem definition and start addressing the solution, there are just a couple more points we need to clarify. First, what, exactly, is a legacy application? Aside from Mr. Cafiero's pithy observation, for the purposes of this book we'll refer to a legacy application as any working application that doesn't currently provide native support for XML (that is, the application can't produce XML documents as output and consume XML documents as input).

Second, why use XML with a legacy application? That is a reasonable question, but answering it is beyond the scope of this book. In this book I assume that you have already answered that question for yourself. You're going to use XML and you have figured out why; all you want to know is how. There has been enough good general material written on the benefits of application integration and electronic commerce that I feel there's very little I can add for justification.

In regard to XML in particular, I assume that you have one application that uses XML and a legacy application that doesn't, and that you need to integrate them. To do that you need either to convert between XML and the format required by the legacy application or to build native XML support into the legacy application. You know what you need to do and why you need to do it. This book is intended to help you with how to do it.


What Do We Need in a Solution?

When we ask, "What do we need?" we're talking about requirements. There are two types of requirements: functional and nonfunctional (the latter are also known as quality requirements or system constraints). The former have to do with what the system is supposed to do. The latter have to do with how it does what it does. Both are important, and both determine the overall approach of this book.

Beyond the overall dictate of solving the problem, two distinct sets of requirements are imposed on the solution by technical end users on the one hand and by application developers on the other. I'll talk a little later about why I'm dealing with both, but for now if you don't care about the other group you can just skip the relevant paragraphs.

Functional Requirements

The technical end user who has an application that doesn't speak XML more than likely needs the solution to do one or more of the following:

  • Convert an XML-formatted file to a flat file

  • Convert a flat file to an XML-formatted file

  • Convert an XML-formatted file to a CSV file

  • Convert a CSV file to an XML-formatted file

  • Convert an EDI-formatted file to an XML-formatted file

  • Convert an XML-formatted file to an EDI-formatted file

A user may want the solution to support other formats, but CSV, flat file, and EDI should handle most cases. For example, an end user may also need to get data out of a database (relational, hierarchical, or otherwise) and put it into an XML format, or go back the other way. Sorry, but these types of problems are a bit beyond our scope. I will, however, give in Chapter 12 an overview of some approaches for doing things like this. When I present the approaches, you'll understand why problems like this exceed our scope a bit.

The developer who has an application that doesn't speak XML has some fairly simple requirements:

  • Enable the application to export data in an XML format

  • Enable the application to import data from an XML format

The word "an" is very important here. I don't say "a specific XML format"; I say "an" XML format. Why? Because once we have data in an XML format, it is fairly easy to convert it to another XML format. You want to make your life simple? Don't try to anticipate all the XML formats your users will want. Give them one, and let them convert. I'll have more to say about this later.

Those are the primary functional requirements. Both groups may have a secondary functional requirement to be able to validate the format of an XML document (which conventionally is referred to as an instance document). When validating an instance document, the format of the document is usually defined in an XML 1.0 Document Type Definition (DTD) or a schema written in the World Wide Web Consortium (W3C) XML Schema language. Both of these define things such as the Element and Attribute names used in the document and the overall structure of the document. The W3C XML Schema language allows documents to be defined in much greater detail than is possible with a DTD. These will both be discussed in Chapter 4.

Validation may need to occur either before the document is read or after it is produced. For going to and from EDI, end users may want to check that the EDI-formatted file complies with the relevant standard either before they read it or after they write it. The approach presented in this book satisfies most such requirements.

Some other functional requirements have do with enabling business applications to support the exchange of business data with other organizations electronically. Those requirements are within the scope of this book, and I'll discuss them in Chapters 12 and 13.

So, we should be pretty clear by now about what we want the solution to do. We now need to figure out how we want it to do it.

Nonfunctional Requirements: Good, Fast, and Cheap

Remember the old joke about good, fast, and cheap, that you can pick only two out of the three? Well, with any luck we might be able to get you all three. Good is pretty relative (as well as unspecific), so I won't go into a lot of detail here. The same applies for fast. So let's talk about cheap and a few other typical requirements.

If you're an end user and you're paying a relatively small price for this book instead of buying a full-featured Enterprise Application Integration (EAI) system like Mercator, we understand each other. If you find the word cheap a bit harsh, think of it this way: You can't cost-justify purchasing a full-featured package to convert a few files on an infrequent basis. You can't justify spending more on utilities than you did on the business application. Take your pick, but cheap is fine with me. If you're a developer, you want to support XML without spending the next release's entire new features budget. Return on investment is what it's all about. The bottom line is the bottom line.

So, beyond cheap, what's important? Simple usually goes hand in hand with cheap, as does easy. Beyond these biggies, there are several more that developers and information technology staff usually care about. I'll assume that you do too.

  • Maintainability: We need to be able to keep the solution running and add new functionality without too much trouble. Code that we add to the application should be easy to fix and enhance.

  • Reusability: When we develop code to solve one problem, we would like to be able to easily reuse it to solve similar problems. We don't want to keep reinventing the wheel or repeating the same snippets of code in several routines.

  • Flexibility: We shouldn't be straightjacketed into a single approach that handles just one type of situation. We should be able to modify it to handle changing circumstances.

  • Modularity: The solution should be nicely broken up into manageable pieces. This helps with reusability and maintainability.

  • Portability (or platform independence): The solution should work on a variety of platforms so we can move it to a different platform with few, if any, changes.

However, we do need to draw the line somewhere and note some things that are of lesser importance. In this book I assume that you don't care as much about the following requirements.

  • Performance: In performance we're concerned with resource usage such as CPU time required to perform conversions, memory usage, and disk space usage. This is most often a concern when a system is used for many purposes simultaneously or is otherwise somewhat constrained in a key resource such as disk space or memory. Due to the relatively low price of hardware these days, and due to the fact that the typical machine is a stand-alone PC, we're not going to be overly concerned about performance.

  • Real-time processing: If we think of performance as applying to resource usage, then real-time processing (as opposed to old-fashioned batch processing) is more relevant to the elapsed time the system takes to complete an action. We aren't concerned with a user clicking a mouse or hitting a key and waiting for an immediate response. We are concerned with batch, file-oriented interfaces.

  • Support for Web services, frameworks, and other bleeding-edge technology: We're talking legacy applications here. If you want to make your applications do WSDL, UDDI, .NET (to be discussed in Chapters 12 and 13), or whatever, that's not cheap, simple, and easy. If you're concerned about this type of stuff, you need something beyond this book. Many of the techniques discussed in this book are quite relevant to getting data into and out of the XML formats typically used by Web services, but what happens after the handoff to the Web service is beyond our scope.

Summing it all up into just a few words, in this book I present a fairly simple, pragmatic approach that can be implemented relatively quickly and at fairly low cost.


The Overview of a Solution

At the heart of this approach are a number of independent programs that can be used alone or together to build a solution appropriate for the specific problem you need to solve. The programs we will develop in this book are:

  • XML to CSV Converter: Reads and validates an XML document, and writes it out in CSV format

  • CSV to XML Converter: Reads a file in CSV format, produces an equivalent document in XML format, and validates it

  • XML to Flat File Converter: Reads and validates an XML document, and writes it out as a flat file in a user-specified structure (read from another XML file)

  • Flat File to XML Converter: Reads a flat file in a user-specified structure (read from another XML file), writes an equivalent document in XML format, and validates it

  • XML to EDI Converter: Reads and validates an XML document, and writes it out in EDI format according to a user-specified structure (read from another XML file)

  • EDI to XML Converter: Reads an EDI interchange in a user-specified structure (read from another XML file), writes an equivalent document in XML format, and validates it

In addition, we'll rely on an XSLT transformation program to transform between different XML formats. XSLT is the W3C recommendation on Extensible Stylesheet Language Transformations. The good news is that such XSLT utilities are easily (and freely) available, and we'll use an existing one rather than build one. I will present the essential techniques you need to know to code XSLT stylesheets that drive the transformations between different XML formats.

How does the solution measure up so far against our requirements? As you can easily see, it meets all the functional requirements for technical end users. Because I'll walk you through the coding techniques used in each of the programs, it should help developers meet their requirements too.

Cheap, simple, and easy? You bet. As I said before, you can download some pretty decent XSLT transformation programs for free from the Web. You can download executable versions of the programs developed in this book from the book's Web site. If you want to tweak them a bit or even use them as a basis for your own slightly different programs, you can also download the source code. The solution is very modular, since we use a number of independent programs. It is also maintainable because most of what you have to change to deal with different file organizations is done in XML files rather than in code. The XML to XML transformations are driven entirely by XSLT stylesheets, which are themselves nothing more than XML documents in the XSLT language. XSLT also offers a great deal of portability (as long as you don't use nonstandard extensions!). Other aspects of portability and platform independence are determined more by the specific implementation, so I'll discuss those later.


Architecture

According to the seminal book on software architecture, written by Shaw and Garlan [1996], software architecture "involves the description of elements from which systems are built, interactions among those elements, patterns that guide their composition, and constraints on these patterns. In general, a particular system is defined in terms of a collection of components and interactions among those components." What we're most concerned with here is what Shaw and Garlan refer to as the architectural pattern or style.

Style is the general model of how the system is composed and how the different parts interact with each other. You've probably run across object-oriented, layered, and client-server styles of software architecture. We'll use a style that should be familiar to you (especially if you do a lot of work from a UNIX shell prompt) but that you may not know by the name used here: pipe and filter.

The basic idea is that you have a number of different programs, or filters, that read input in one format, transform it, and produce output in a different format. The output of one filter is connected to the input of another via a pipe. The filters are independent of each other, with only the constraint that the output of one filter be useable as the input of the next. In our approach, the pipes are disk files or sets of files in directories, but they could just as easily be in-memory data structures, database tables, or something else. Figure 1.1 shows the basic pipe and filter style.

Figure 1.1. Basic Pipe and Filter Style

A modified pipe and filter style (Figure 1.2) allows the filters to accept input or parameters other than just the input files. This allows the filter programs to produce different types of output. In reality, most pipe and filter systems in use are of this modified style; ours is too.

Figure 1.2. Modified Pipe and Filter Style

Want to build a system that converts CSV to XML? Easy—just use the CSV to XML Converter program as the filter (Figure 1.3).

Figure 1.3. CSV to XML Application

You don't like the XML that this produces? You want to produce an XML file that conforms to your customer's invoice specification? No problem. Add the XSLT transformer to build a more general purpose application (Figure 1.4).

Figure 1.4. CSV to XML to XML Application

Now let's say you have a customer (or maybe you're a consultant and your client has a customer) who wants to send you orders using the X12 EDI standard. The version of Peachtree Accounting you're using processes only CSV files, and you don't want to buy an EDI system for just this one application. We can easily build a system to solve this problem too (Figure 1.5).

Figure 1.5. EDI to XML to CSV Application

By now it should be pretty obvious that you can take the building blocks here and make pretty much whatever pipe and filter system you might need. Once you think about it, the pipe and filter style is a very logical choice for the type of format conversion problems we're dealing with in this book. Even if you're a developer who wants to enable your product to work with XML, knowing that the XML your application reads or writes is probably going to live in this type of environment can help you make some better-informed design decisions.

Beyond our basic pipe and filter approach, it would be nice to have something that helps us build these specialized little pipe and filter applications. In Chapter 11 we'll put a lot of the pieces together and go over just how such a system might look.

Why Not Use XSLT for Everything?

There are two different opinions about using XSLT for non-XML transformations where the source or target is a format other than XML. One camp says that XSLT can and should be used for everything. The other camp says that XSLT is wonderful for transforming an XML document into a different flavor of XML document but that there are better ways to deal with non-XML formats.

So, why am I in the latter camp? It's an issue of using the right tool for the job. If you're camping and you forget an essential tool, you can open a can of beans with a hatchet or maybe even cut down a tree with your Swiss Army knife. But those aren't exactly the easiest ways to do things. It's the same way with XSLT. XSLT has a fairly concise language for identifying items in source XML documents and creating corresponding items in target XML documents. However, when dealing with other formats the language is not so elegant. For example, to parse an input fixed record length file it is necessary to use things like substring functions. Parsing an input CSV file requires repeated use of a search function to find the delimiter and then substring functions to isolate individual fields. Creating target documents in non-XML formats requires similar operations.

There are some drawbacks to using XSLT for everything. The most efficient design approach using just XSLT is to try to do everything with one XSLT stylesheet. In this approach the code that parses the input or formats the output is mixed up with the code that does the logical transformation between documents. By "logical" I mean correlating input fields to output fields, along with any manipulation or value conversions required. One example of a logical transformation is concatenating a person's first and last names into a single customer name. This approach of trying to do everything with a single stylesheet doesn't support modularity, reusability, understandability, or maintainability very well.

A less efficient approach that is somewhat more modular and reusable is to use two XSLT transformations. One transforms the non-XML format to or from XML, and the other performs the logical transformation of XML to XML. This approach is somewhat better than a single XSLT stylesheet that does everything. However, it is still hampered by a limitation in the XSLT 1.0 language that restricts it to generating just one output file per transformation. As we'll see in later chapters, several types of transformations require us to generate multiple XML documents from one non-XML input file. Some XSLT 1.0 processors include nonstandard functions that provide this functionality. There is discussion of supporting it in XSLT 1.1, but it isn't there yet as a standard. In addition, for more complex operations like dealing with EDI acknowledgments and keeping track of the documents exchanged with different trading partners, it is much more appropriate to use a programming language like Java, C, or C++ than XSLT.

An advantage of using a more conventional programming language for handling non-XML files is that we can implement a general purpose solution rather than a series of XSLT stylesheets that are each useful for only one particular file organization. In our general purpose solution fairly simple parameter files handle the differences in file organization. These files are themselves coded as simple XML documents.

In addition to these considerations, presenting the techniques for handling non-XML files with XML files is a good introduction to the type of XML programming that will be required to XML-enable legacy business applications. In other words, it's a good teaching tool.

I should mention one final caveat regarding XSLT. There are, of course, wide variations among different XSLT processors, but as a group they tend to have poorer performance than commercial EAI or EDI conversion utilities. XSLT, and therefore the architecture, is probably not well suited to production situations that need to handle large amounts of data in a short amount of time. However, earlier in the chapter I noted that in this book I assume that cost and portability are more important considerations than performance. XSLT fits the bill.

Two Implementations of the Architecture: Java and C++

In this book I present two different implementations of our architecture. Why? Basically, for reasons of platform independence and portability. However, I'm also doing it out of a pragmatic recognition that there isn't just one dominant programming language, and I want to be able to speak to both camps.

We can use an analogy to home building in order to understand the two implementations of a single software architecture. The analogy isn't exact, but there are enough high-level similarities between building houses and building software to serve our purpose.

When an architect designs a house, he or she first establishes the number and types of rooms that the client wants as well as the general characteristics. Then the architect develops a detailed floor plan with approximate room dimensions, locations of windows and fixtures, and so on. At this point the architect may not have decided yet on specific materials. The actual house could be constructed out of wood, brick, or even recycled tires; the materials will be selected during a second phase of home design. The first phase of home design corresponds to software architecture. Just as we could build two houses with the same floor plans but with different materials, we can implement a software architecture in two different languages.

So, this book presents implementations in Java and C++. The Java implementation uses the Sun Microsystems and Apache Foundation XML libraries, and the C++ implementation uses Microsoft's MSXML library.

Why Java and C++? Well, most applications these days are written in either Java or C++. Most environments in which those applications are run use standard Java or C++ libraries. "But," you say, "my legacy application is written in COBOL!" This gets to the bottom line of cheap, simple and easy. Reading (parsing), writing (serializing), and validating XML documents are not easy tasks. This is especially true in the case of validating against schemas written in the W3C XML Schema language. It takes a lot of complicated, hard-to-write, and hard-to-debug code to do such things. We don't want to write that code. We want to use libraries that someone else has written. And because we're concerned about portability and platform independence we would like to use standard libraries rather than proprietary approaches. These standard libraries are primarily object-oriented, which again makes them a good fit for Java and C++. In Chapter 12 I'll talk a bit more about some other alternatives, including some for procedural languages such as COBOL. However, our most important nonfunctional requirements point us to standard, object-oriented application programming interface (API) libraries.

Even with the standard API libraries there are implementation choices other than Java and C++. For example, Perl, Python, and Visual Basic all have fairly sophisticated XML support. Visual Basic even uses the same API library, MSXML, that we'll use for C++. The techniques discussed in the next two chapters on basic conversions involving CSV files and XML could be implemented in these languages. However, the more sophisticated conversions and utilities built later in the book are complex enough that these languages would not be appropriate. In addition, very few real applications are written in these languages. So, we'll keep our focus fairly limited and be concerned primarily with the Java and C++ implementations. However, the basic techniques will still be relevant if you want to cobble together some simple utilities using Perl or whatever. As I'll discuss shortly, the basic operations using the standard APIs are presented in pseudocode that is language independent. Even though I won't offer the specific Perl or Visual Basic syntax, you could still port the techniques fairly directly to these languages from the pseudocode design.

If you are an end user, you really don't care much whether the program is written in Java, C++, Perl, COBOL, ALGOL, or fuggly old IBM 360 Assembler. In this case, the types of libraries and utilities you can install and run in your environment dictate your choice of implementation. Java can run pretty much anywhere, so you may want to install the relevant Java libraries and filter programs. If you can't install things directly where your application is running, maybe you can hijack a PC networked to your application's box and install the conversion libraries and utilities there. In that case, the WIN32 C++ implementation might be a bit easier since there are fewer pieces to install.

Before wrapping up this section there are a few other developer-oriented issues we need to discuss. We've already decided that we're going to use a standard API library. Even with this restriction we still have a few choices. I have chosen to use the W3C's Document Object Model (the DOM). But before discussing why I'm using the DOM, let's look at what it is.

The Document Object Model

Like nearly all other basic XML technologies, the DOM is a creation of the W3C, which defines the DOM as:

a platform- and language-neutral interface that will allow programs and scripts to dynamically access and update the content, structure and style of documents. The document can be further processed and the results of that processing can be incorporated back into the presented page. [W3C 2002]

What does this mean? Basically, the DOM specifies a set of program routines, or an API. It is not itself an object; but similar to how Java does things, it specifies an interface to the object model, with that interface composed of a number of different methods. Since it is just a specification, there isn't any actual W3C-produced code. So, in general terms, why use the DOM? Again, the W3C says:

"Dynamic HTML" is a term used by some vendors to describe the combination of HTML, style sheets and scripts that allows documents to be animated. The W3C has received several submissions from members companies on the way in which the object model of HTML documents should be exposed to scripts. These submissions do not propose any new HTML tags or style sheet technology. The W3C DOM WG is working hard to make sure interoperable and scripting-language neutral solutions are agreed upon. [W3C 2002]

What does this really mean? It means they developed the DOM so that application programmers can have a standardized way to deal with documents. It is also notable that the DOM was developed first to deal with HTML documents and not XML documents. However, the XML world has embraced it wholeheartedly.

The hope was that if enough tool vendors (like Microsoft, Sun, Oracle, and so on) developed DOM implementations, application developers would have a fairly easy, standardized way to deal with documents programmatically. This has come to pass, and there are enough different implementations that porting HTML or XML code between implementations is a lot easier than it would have been if there were no DOM. The DOM also makes it fairly easy to talk about coding in a generic way; most of the XML processing logic discussed in the book's utilities is applicable to both C++ and Java environments.

The W3C has defined several different "levels" of the DOM. Level 1 was released in October 1998, Level 2 was released in November 2000, and Level 3 was still a working draft at the time of this book's preparation. Basically, each level has a few more features and a bit more functionality than the previous version. For the most part, we don't need to be too worried about the DOM level we're using. Both the C++ and Java libraries used in this book support Level 2, with a few additional features from Level 3.

Why Use the DOM?

We already agreed that we don't want to reinvent the wheel. So, from a programming perspective, the DOM fits the bill. However, it's not the only language-independent API we can use. There is one other predominate API, called SAX (Simple API for XML, not the musical instrument that John Coltrane and a certain past U.S. president played). SAX, like the DOM, specifies an interface that can be used to process XML documents in a somewhat platform-independent fashion. It doesn't matter a lot in practical terms since it's becoming customary for most XML libraries to support both, but the DOM is an "official" W3C Recommendation and SAX isn't.

How are the DOM and SAX different, and why have I chosen the DOM? The DOM handles a complete document as an object in memory, specifically a tree. SAX, on the other hand, is an event-driven API. It calls a specific method (via callbacks) for each type of XML construct it encounters as it reads in an XML instance document. The most important difference between the two APIs is that the DOM makes it fairly easy to build or manipulate a document in memory. Some SAX implementations offer ways to build documents, but there isn't a standard SAX approach. So, SAX is not as well suited in this respect. On the other hand, SAX is well suited to parsing very large XML documents that might cause performance problems (or even crashes) if we tried to read them into memory using the DOM. Since it's much simpler to use just one API if possible, I'm using the DOM in this book. I'll discuss this topic again in Chapter 12, but the choice of the DOM for this book's utilities will become more understandable as the overall design approach progresses.

I should note, though, that while the current versions of the DOM make it fairly easy to build and manipulate XML documents in memory, through Level 2 the DOM doesn't specify how XML (or HTML) documents are actually read or written. Curious but true. I would have thought those things to be fairly fundamental, but I guess the HTML heritage as well as different priorities in the W3C left those details to the implementers. The draft Level 3 requirements do finally deal with such things. The fact that actually reading or writing XML documents isn't specified in Level 2 really isn't too much of a problem since most XML libraries provide the functionality. However, because it isn't specified there are often differences in the particular methods used. We'll see this as we develop the C++ and Java code later in the book.

One final note on the DOM: While many implementations use the exact W3C method and object names in their libraries, Microsoft's MSXML library sometimes slightly modifies the names specified by the DOM. It also allows many of the DOM object properties (or attributes) to be accessed directly rather than just through get methods. This means that the C++ code in this book may have to be modified if you want to use it with a different C++ DOM library. (At least a global search and replace will be required. Other changes will probably be required as well, but those are beyond the scope of this book.)


How to Use This Book

How you use this book depends very much on what you want out of it. As I said earlier, the book is primarily focused on the requirements of two distinct sets of readers: technical end users and developers. I've attempted to structure the book so that both groups can easily use it. I'll first cover how the chapters are organized, then I'll suggest how someone from either group might want to approach a chapter.

Chapter Organization

Some chapters (like this one) are mostly general text that might be relevant to the needs of both groups. These are organized in a fashion appropriate to the topic.

All chapters contain a section of references or of resources (sometimes both). These provide information on references cited in text and suggestions for further study. I generally try to be as specific with resource URLs as I can, but in many cases I can give only general pointers. Some Web sites are reorganized so frequently that links to specific internal pages quickly become invalid. In such cases the referenced URL should have either a link to the desired resource or a search facility to help you find it.

The chapters that present utilities are organized slightly differently. Here's the general organization by sections. (Note: This is the general organization; some chapters may not have each of these sections.)

  • Requirement: A description of what the utility is supposed to do, including general specifications of the input, processing, output, parameters, and restrictions.

  • Running the Utility: A guide to running the utility from the command line, including description of the arguments and options.

  • Sample Input and Output: A review of sample input and output to help further illustrate the utility's functionality.

  • Design: The design of the utility. A high- to mid-level design overview is presented in text, with a more detailed pseudocode description that highlights DOM usage. Chapters 6 through 9, in which more complex utilities are developed, break the design section into two aspects. The first deals with high-level design or overall design considerations, while the second deals with detail design. The detail sections are not intended to be read front to back (unless you happen to need help falling asleep); they are included for reference.

  • Java Implementation: Highlights of the Java implementation of the code, with listings of important code fragments and explanation.

  • C++ Implementation: Highlights of the C++ implementation of the code, with listings of important code fragments and explanation.

  • Enhancements and Alternatives: Suggestions for how the utility might be enhanced plus discussion of different design and coding approaches.

End users will probably use the utilities chapters differently than developers will, as described in the next section.

Notes for Primary Audiences

Technical End Users

If you're a technical end user, you probably mostly care about what a utility does and how to run it. Therefore, scan the beginning of the chapter, and review the requirements. If the utility provides what you need, look at how to run it. You can skip the rest of the chapter.

Developers

If you're a developer, you probably care less about what a utility does than about how it does it. Scan the beginning of the chapter, review the requirements, and study the design. Then look at the Java or C++ implementation, depending on your language choice. You'll probably also find the enhancement suggestions interesting. You can find out how to download the source code by referring to the book's Web site, and you can get more information on related topics from the Resources section.

Other than those specific suggestions, I'll offer only this: This book is meant to be a tool kit. I don't expect anyone to read it cover to cover. Use what you need, scan what looks useful for the future, and ignore the rest.

Chapter Summaries

To aid you in exploring this tool kit, here is a chapter-by-chapter guide to what you'll find in it.

  • Chapter 1, Introduction: You're reading it.

  • Chapter 2, Converting XML to CSV: This chapter presents a very basic approach for converting an XML document to a CSV format file. I start with this topic for two reasons: (1) CSV is still the most common universal format used by desktop applications, and (2) reading an XML document is a bit easier than writing one, so we'll start with the easy stuff first. This chapter covers the basic techniques for reading and parsing XML files using the DOM, so it is a foundation chapter for developers. The example file used is a generic address book listing, since it is most amenable to the type of processing performed by this utility.

  • Chapter 3, Converting CSV to XML: This is the converse of the preceding chapter. Here we take a CSV file as input and convert it to an XML document. It is also a foundation chapter for developers since it deals with the basic techniques for creating an XML document in memory using the DOM and writing it out to disk. I again use the generic address book listing as an example.

  • Chapter 4, Learning to Read XML Schemas: This book is intended primarily for people who are going to be reading schemas, not writing them (or, at most, writing fairly simple schemas). The W3C XML Schema Recommendation is pretty obtuse, and there are some big, fat (and good) books out now that cover it in gory detail. (My Web site, http://www.rawlinsecconsulting.com, has recommendations for a few.) I cover here the basics of what you need to know, going over the essential features and a few different approaches to schema design that you're likely to encounter. This chapter is important because many of the remaining chapters use schemas.

  • Chapter 5, Validating against Schemas: In this chapter the utilities developed in Chapters 2 and 3 are modified to support validating an XML instance document against a schema on input and validating it against the schema before writing it out. Various examples of validation failures are presented with the types of error messages that the different APIs offer.

  • Chapter 6, Refining the Design: The utilities presented in Chapters 2 and 3 are fairly basic in functionality and simple in design and coding. The other utilities developed in the book have more sophisticated functionality and therefore require more complex designs and coding. This chapter lays the foundation for these utilities by discussing several issues related to processing XML documents and non-XML grammars and presenting the base classes used by the utilities developed in Chapters 7, 8, and 9.

  • Chapter 7, Converting CSV Files to and from XML, Revisited: This chapter builds on the foundation of Chapter 6 and develops CSV conversion utilities that are more capable than those developed in Chapters 2 and 3. For example, XML works best when a disk file contains a single XML business document. However, most batch imports and extracts in CSV format bundle many separate business documents into one disk file. These utilities combine many XML documents into one CSV file and split one CSV file into many XML documents. The concept of driving the conversion with a parameter file, coded as an XML document, is presented in this chapter. For the XML to CSV conversion, a purchase order is used as the sample document since small and medium-sized businesses commonly receive such documents from larger customers. The CSV to XML conversion uses an invoice as the sample document. In this fashion we deal with the two most basic documents used in the procurement cycle.

  • Chapter 8, Converting Flat Files to and from XML: Most common CSV files have a uniform format. Every row is in the same format. Flat files are not so simple. Each document usually has at least a header record, at least one type of detail record, and often a trailer or summary record. They may have many, many more types of records. They therefore require more complicated processing than CSV files. As with our CSV utilities, an XML file describes the structure of the flat file. We'll again use a purchase order as the sample document for converting from XML and an invoice as the sample document for converting from a flat file.

  • Chapter 9, Converting EDI to and from XML: This chapter presents the most complex type of conversion we'll cover, between XML and EDI formats. The grammar of EDI syntaxes is analyzed, and algorithms appropriate for processing EDI are presented. This chapter also covers other issues around EDI, such as processing Functional Acknowledgments and handling control numbers, and the preliminary functionality needed to support these requirements.

  • Chapter 10, Converting from One XML Format to Another with XSLT: This chapter presents the basics you need to know about using XSLT to transform a document from one XML format to another XML format. It covers the most commonly used features as well as a few things to watch out for. This chapter is primarily targeted for technical end users, though developers need to be aware of what XSLT can do so they can design around it.

  • Chapter 11, Using the Conversion Techniques Together: This chapter presents a few use cases and simple script examples for using the utilities together to solve some common conversion problems. It also presents the requirements and high-level design for the initial version of the Babel Blaster open source EDI/EC/EAI file conversion program. It builds on the utilities presented in the previous chapters to develop a comprehensive system for solving many types of file conversion problems.

  • Chapter 12, Building XML Support into a Business Application: The preceding chapters presented various techniques for reading and writing XML documents. However, other issues need to be addressed before deciding on an overall approach to XML-enabling an application. This chapter discusses issues such as integrating with existing code in the least disruptive manner, selecting data for import or export, deciding whether and what to validate, choosing design options other than the DOM, and processing data in other formats (such as relational databases).

  • Chapter 13, Security, Transport, Packaging, and Other Issues: Having the data in XML isn't enough. You also have to be able to get it to and from your trading partners. They may want to receive and send the data over a private Value Added Network (VAN) if they are using EDI, but more than likely they will want to use the public Internet. To do that you'll need ways to package several XML documents into one bundle (or unpack them) and handle security. This chapter addresses these other B2B considerations. I'll present a basic approach for assessing your real needs in these areas and suggest some practical strategies for supporting those needs without getting too much over your head into bleeding-edge technology.

Conventions

To help make it clear what you're looking at, I adopted some formatting conventions in this book as shown below.

Fragments of source code in Java, C++, pseudocode, and various types of XML syntax:

Source Code Program
main()
{
  // This is a C++ program
...
}

Single lines of code:

myDoc = new DocBuilder;

Fragments of non-XML files:

Doe,John,12 Lee Street,Boston,MA,01303

Command line program execution:

java MyProgram input output -a argument

The first time an important term is used it appears in bold font, for example, pipe and filter. The first time an acronym is used it appears in italics, for example, W3C, accompanied by the full name set in regular font. In some cases, important acronyms, for example, XSLT, first appear in both bold and italic.

Several of the terms used in this book have different meanings, depending on the context. For example, we can talk about an element's attributes in an XML instance document, or when discussing the DOM we can talk about the attributes (or properties) defined for any of its interfaces, including a DOM element. To help keep things straight, whenever I refer to a named DOM entity I will capitalize the term. In addition, Elements and Attributes discussed in the context of instance documents will always be capitalized.


What You Need to Use This Book

In this book I assume that you have certain prerequisite knowledge. You also need to have several software tools, source code, and some XML files in order to make full use of this book. This section tells you what you need to already know as well as what those additional resources are, where they are, and how to get them.

What You Should Already Know

All readers should already have a basic understanding of XML. Familiarity with XML 1.0 DTDs may be helpful but is not absolutely necessary. If you aren't already familiar with XML, there are several good introductions to the subject. On my Web site I give a few recommendations. If you don't mind doing a bit of somewhat technical reading, you can also review the XML Recommendation at the W3C Web site, http://www.w3.org.

End users should be comfortable with the general concept of file format conversion, that is, taking a file that exists in one format and converting it to another file in a different format. Some rudimentary programming experience will be helpful if you plan to develop XSLT stylesheets or write scripts that use the utilities developed in the book.

Application developers should be familiar with either Java or C++ and their respective basic library functions. For those interested in the C++ implementation, prior experience with Microsoft's Component Object Model (COM) is helpful but not required.

Web Site and Contact Information

The Web site for this book has nearly everything you need (or pointers to the information you want). There you will find:

  • Executable versions of the utilities in both Java jar file format and Win32 binary format

  • Full source code in C++ and Java

  • Example schemas and XSLT stylesheets

  • Sample input and output files

  • Other sample and support files

  • Updated links (current at the time of final editing)

The URL for the book's Web site is: http://awprofessional.com/titles/032115940. You can find additional examples and frequently updated references to other good books at http://www.rawlinsecconsulting.com/booksupplement.

Please feel free to contact me with any questions, comments, errata, bug reports, and so on via e-mail at mike@rawlinsecconsulting.com.

General Software

If the only XML syntax work you expect to do is to code parameter files for running the utilities in this book, probably all you need is a good text or programming editor such as emacs. For Windows users, Microsoft's XML Notepad (Beta version 1.5, May 3, 1999) is available as a free download. I would give you a URL for it, but Microsoft has an annoying tendency to move things around on its site. So, I won't go any farther than pointing you to http://www.microsoft.com and suggesting that you search for "XML Notepad". XML Notepad should ensure that you create a well-formed XML document, that is, one that complies with W3C's XML Recommendation in certain key aspects. (Note: When used with Internet Explorer version 4.01 and earlier, XML Notepad converted all characters to uppercase. Very few people are likely to have this problem now since most people run later versions of Internet Explorer. If you run into this problem, you'll need to upgrade Internet Explorer to version 5.0 or later.) The Windows Notepad, emacs, or vi will probably do just fine for very light use.

However, if you're going to do much with XSLT or schema design, spending a couple hundred dollars for a good XML tool will probably be a good investment. I list my favorites below.

  • XMLSPY by Altova GmbH: All you need is the Integrated Development Environment (IDE). It helps you design schemas in various schema syntaxes, create instance documents, validate both, and develop and test XSLT stylesheets. XMLSPY runs only on Win32 platforms (Windows NT, 2000, XP, 98, ME). A free evaluation download is available from http://www.xmlspy.com. I used version 4.3 for this book.

  • TurboXML (formerly XML Authority) by TIBCO: Similar in functionality to XMLSPY, as a Java-based application it runs on many platforms including Windows, Linux, Solaris, HP-UX, and other UNIX platforms. A free evaluation download is available from http://www.tibco.com.

Other tools with similar functionality also exist (for example, eXcelon's Stylus Studio). However, I've not tested or extensively reviewed any of them, so I don't have much to say about them. I have heard of some free, open source tools, but they were still in development when I wrote this book. If I find any good ones I'll post links on my Web site.

If you would like a different way to view an XML document (but not edit it), Internet Explorer version 5.5 is handy. It allows you to expand or contract elements with children so that you can view the complete document tree or just focus on certain branches.

This completes my recommendations for general purpose software. The developer recommendations for the Java and C++ environments are discussed next, but I must raise an important cautionary flag first.

CAUTION Don't Change Horses in the Middle of a Stream!

I feel a bit of a nag when reminding people to pay attention to the basics. But it seems that we often forget them or make exceptions when dealing with new technology. After you have downloaded the latest tools and APIs, don't update them until you've finished your project! With updates seeming to come every month or two in the XML world, there is a great temptation to apply them just as soon as they come out. However, I've had too many experiences on other projects over the years of starting to work in the morning and discovering that code that worked last night doesn't work today, even though I didn't change anything. After hours of debugging I found that new libraries had been installed and the problem wasn't my code at all. Save the updates of the APIs until your code is stable and thoroughly tested. Then download the latest, rebuild, and retest.

Java Software

The Java code in this book was developed using the libraries listed below. It may compile and run under later versions, but it hasn't been tested that way. Due to changes in support for the W3C XML Schema language, the versions listed are the minimum that I recommend.

  • Java 2 Standard Edition Version 1.3.1_03 (http://java.sun.com/j2se/1.3): To compile the code you'll need the full software development kit (SDK). If you're only going to download and run the book utilities, you'll just need the Java Runtime Environment (JRE).

  • Java XML Pack, Spring 02 (http://java.sun.com/xml/downloads): If you're only going to run the Java utilities, you don't need the full kit. However, you will need the Java jar archive files from the kit.

  • Xerces2 Java Parser 2.0.1 (http://xml.apache.org): This software is required for all Java XML development and runtime presented in this book. For the most part I use standard Java API for XML Processing (JAXP) features, but the routines to serialize a DOM document use specific Xerces2 routines. (Although the precise name of the parser is "Xerces2," I'll observe the common custom of referring to it as just "Xerces" unless I mean to refer specifically to this version of Xerces.)

  • Xalan-Java 2.3.1 (http://xml.apache.org): This software is required for Java-based XSLT transformations. Other Java-based XSLT implementations may work but have not been tested.

Refer to my Web site for specific instructions about how to install the jar files for your particular version of Java.

Note: The book's Java code was developed using version 1.2 of Sun's JAXP. The Java XML Pack has an implementation of the JAXP 1.2 specification that uses Xerces2 as the default parser. (Xerces2 conforms to the JAXP 1.1 specification.) To be sure that you get the latest bug fixes I recommend downloading the most recent version of Xerces2 from the Apache site.

The Java code was developed and tested with Borland's JBuilder Version 6 on Windows NT Workstation version 4. It was also tested on Windows 98.

C++ Software

As previously mentioned, the C++ implementation presented here is based on Microsoft's MSXML library, listed below. The code was developed and tested with MSXML version 4.0. This is the minimum version required. The code may work with later versions but has not been tested. Changes will be required to make it run with different DOM implementations.

  • MSXML 4.0 Microsoft XML Core Services (http://www.microsoft.com/downloads): This is required for both development and runtime. MSXML runs as a COM service.

  • MSXSL.EXE Command Line Transformation Utility (no version number, last updated 9/10/2001; http://msdn.microsoft.com/downloads). This is required for runtime XSLT transformations from command line.

The C++ code in this book was developed under Visual C++ 6.0 and tested on Windows NT Workstation 4.0 and Windows 98. (No, you don't need .NET!)

MXSML, Win32, and COM as Legacy Technologies?

Incredible but true! When I conceived this project, MSXML 4.0 (with full support for the final W3C XML Schema Recommendation) was still pretty new. However, before I really got going, Microsoft came out with .NET. Not only does .NET offer a different DOM API than MSXML (though reportedly it still uses MSXML under the hood), it is a completely new and different framework. XML is only a small piece.

When this happened I became a bit concerned that working with the DOM via MSXML's COM interfaces would be dealing with obsolete technology. However, on reflection I decided that it fit very well with the thrust of the book. I just didn't expect to run into legacy applications and technology in this particular area! Considering the paradigm shift that .NET imposes and the impact of migrating from Visual Studio 6.0 to Visual Studio .NET, I think there may be an audience for this book for some time to come.

Bottom line: This book gives you a way to do XML from Win32 without having to migrate to .NET.

Alternatives to MSXML?

While struggling to get the COM-related stuff working and fully debugged, I began to question my decision to use MSXML instead of some other DOM implementation that worked with C++. But, given the realities of C++ applications on Win32, I still think that MSXML is the best choice for this book. Many development shops are all Microsoft and wouldn't look at APIs from any other source. MSXML is fairly mature, installs easily, and is widely considered to offer some of the best support for W3C XML Schema. However, if you are comfortable with the licensing terms of the Apache C++ Xerces implementation, you can save yourself a lot of the COM headaches by using it instead of MSXML.


For Developers

This section describes the coding approach, conventions, and style I chose to use in this book. If you're reading this book as a technical end user, you may want to skip ahead to the next section.

General Coding Approach and Conventions

The Java and C++ code presented in this book uses object-oriented techniques. The DOM and all its various parts are object-oriented, as are the Java file operations and the C++ file operations. (I have chosen to use the C++ classes instead of the old-style C libraries.) That said, most of what we programmers really care about is procedural in nature, that is, the code that lives and works inside methods. We care most about how to call methods that manipulate DOM objects. We don't care as much about everything else.

I have not gone out of my way to make this code object-oriented. In the beginning it is fairly simple and not necessarily very heavily object-oriented. However, as the design progresses and matures through the book, I do use more object-oriented techniques when they promote reuse and extension. I'm enough of an old-school programmer to be a bit concerned about the performance implications of declaring and freeing a lot of objects dynamically. I generally tend to avoid doing so when I can. However, we're valuing reusability and extensibility over performance in this design. So, there are cases when we do create a lot of objects dynamically at runtime rather than declaring them statically at compile time. If the code lives on and it turns out to be a dog at runtime, we can investigate more efficient designs. A modular, object-oriented approach also helps us in that regard.

I have not chosen to construct elaborate object models for the non-XML entities manipulated in these programs. In addition there aren't very many class diagrams or other things as found in the Unified Modeling Language (UML). We're going to keep it simple and focus on the essentials.

That said, here are a few other notes and general rules on coding approach and style.

  • Clarity over cleverness: As any seasoned programmer knows, there are ways to write programs that are exceedingly clever. Such cleverness often yields short, efficient programs. However, in many cases these come at the expense of clarity, since it sometimes makes it harder to understand what the program is really doing. In this book where being clear means using a bit more code at the expense of being as clever as possible, I do the former rather than the latter. You can be clever in your own programs if you want, but in trying to get across some basic concepts I would rather be clear. We will follow the KISS principle: Keep it simple, stupid!

  • Error handling: Since these are basic utilities intended for light use and teaching concepts, I did not implement elaborate error handling. However, all parsing and DOM exceptions are caught and enough information is reported to enable someone reasonably experienced in XML to fix a problem.

  • DOM extensions: Some DOM implementations, notably Microsoft's MSXML in our case, offer extensions to the methods, functions, and interfaces defined in the DOM. To try to make the code platform independent and to keep the Java and C++ implementations as alike as possible, I avoid using these extensions in the code. However, where the extensions offer alternatives to the approaches I take in the code, I often comment on the alternative in the text.

  • File organization: Each class is coded in its own source file. The name of the source file generally matches the name of the class. This is, of course, standard in Java but not in C++.

  • Header files: For C++, each class has a separate header file.

  • Naming conventions: Upper camel case is used in the XML examples and for class names in the programs. Lower camel case is used for method names and variables. These choices seem to follow the prevailing conventions. With few exceptions abbreviations are avoided. Variable names in programs are prefixed with an abbreviation indicating the data type. Although in Java and C++ the line between variables and classes is sometimes hard to determine, I won't use this prefixing convention for classes.

  • Comments: The code is liberally commented. I have tried to err on the side of having too many comments rather than not enough. I hope you find them sufficient without being too tedious and obvious.

  • Formatting: Opening and closing brackets for code blocks appear on their own lines. Lines generally break around column 65. I have tried to make indentation and continuations consistent in the source files, using spaces and no tabs.

  • Pseudocode: The pseudocode presented here is based on Programming Design Language (PDL) [Caine and Gordon 1975]. This commonly used pseudocode detail design language is basically structured English with a few reserved words. I see no particular merits in it over other pseudocode languages, but I picked it to lend some consistency to the pseudocode. My usage extends PDL by including calls to specific DOM methods. Appendix B presents a summary of pseudocode conventions used in this book.

Additional C++ Considerations

A few words are in order regarding my approach to using C++. In particular, I want to point out handling of strings, exceptions, and constants.

The final ISO and ANSI C++ standard has extremely useful string classes (at last!). These are much easier to use and less prone to runtime exceptions than old C-style char arrays, char pointers, and the C string library. However, these string classes are still not universally supported "out of the box" by many major C++ compilers. They weren't supported natively by Visual C++ 6.0, which I'm using for this book. Even though some add-on open source and proprietary class libraries do support these classes, many developers use only what is standard from the compiler vendor. In addition, a lot of the legacy C++ code currently in production doesn't use these string classes. Finally, neither MSXML nor the Apache Xerces C++ API that is its best alternative use the ISO/ANSI string classes. MSXML uses COM strings (as we'll discuss in Chapter 2 and Appendix C), and the Xerces C++ API uses its own XML char class. They both support conversions to and from char arrays, not ISO/ANSI strings. So, for all these reasons I'm sticking with char arrays, char pointers, and the old C string library.

In the early days of C++ compilers, throwing exceptions for common runtime errors was discouraged because exception handling wasn't very efficient. The general view is that compilers have gotten a lot more efficient in this regard. However, a lot of the standard C++ library functions still use status codes or return values rather than throw exceptions for every little thing the way that Java does. For this reason I generally use the approach of returning status values from functions rather than throwing exceptions.

Regarding constants, some programmers believe that using #define rather than const to define constants is "evil" [Cline 2002]. I certainly prefer the Java style of defining constant class members. However, I found it extremely awkward to define constants that could be shared across classes using this approach with Visual C++ 6.0. There may be better ways to do it with compilers that provide better support for the final ISO/ANSI C++ standard. I'm sure there are people who are more clever about C++ programming than I am. At any rate, I've taken the easy way out in this book and used old C style #defines.

The general picture you should draw from this discussion is that the C++ code presented here does not necessarily represent current best practices in C++ coding. It instead represents "good" practices (I hope) as of the mid-1990s. However, I'm not really too concerned about that. If your legacy C++ applications are very old, they probably don't reflect current best practices either. Your code may look very similar to mine. On the other hand, if you are using more up-to-date compilers and following current best practices (for example, using the string class instead of char arrays), by all means keep doing what you're doing and don't regress. You should, however, still be able to follow my code and update the techniques to your current best practices as appropriate.


How You Can Use the Utilities and Code

Welcome to the wonderful world of copyrights and licenses. All the utilities and code developed in this book are open source. Their copyrights are owned by the good people at Addison-Wesley, but you can use the utilities and code under the terms of the GNU General Public License (GPL), Version 2, included as Appendix A. This is the same license under which Linus Torvalds makes the Linux operating system available. I hope you and your organization will feel comfortable using this stuff without having to get the lawyers involved.

In the final analysis the terms of the GPL take precedence, but here is what I intend.

  • You may use the executable, binary forms of the utilities in any way you wish for any purpose, whether it is public, private, commercial, not-for-profit, and so on.

  • If you use the binaries, you do so with the understanding that no warranty of any kind is offered by anyone. The only exception is the case where you have explicitly executed a contract for warranty support with someone. (Just so the lawyers and my editor at Addison-Wesley don't have anything to worry about, let me make it very clear that your purchase of this book doesn't imply such a warranty contract. If it did, the book would be a lot more expensive. However, we thank you for buying it anyway!)

  • You may freely distribute the source code and binaries in their original form, provided that they are accompanied by the GPL and that you give appropriate notice of the Addison-Wesley copyright as well as a disclaimer of warranty. As is the case with Linux, you may charge a reasonable fee for your copy or distribution costs. You may offer warranty support for a fee.

  • You may modify or use parts of the source code in your programs so long as you retain the existing copyright notices and make your modifications available under the GPL.

The synopsis above applies to everyone. Here are my intentions for more specific groups of readers and users.

  • If you are a consultant, you may make the binaries available to your clients for a nominal fee to cover actual copy costs. You may modify the source code and offer the resulting binaries to your customers, again for only a nominal copy fee, so long as you observe the GPL and make those modifications freely available to others (like me!) under the terms of the GPL. You may accept payment from your clients for performing the modifications, but you need to make sure they understand that they don't own the modified code. You may contract with your clients to offer warranty support.

  • If you are a vendor of proprietary business applications software that does not natively support XML, you may make the binaries available to your customers for a nominal fee to cover copy or distribution costs. If you want to build native XML import/export support into your business application, you may use the algorithms and DOM processing techniques in your proprietary product, but you may not use the source code in your product.

  • If you are a vendor of proprietary EAI, EDI translation and management, or other middleware software, you may use the contents of this book as an aid to your development efforts. You may use the more common and generic techniques and algorithms. You may not use either the unique design approaches or the source code.

In short, I hope that you benefit indirectly from the programs and the source code and that you feel free to share them with others. However, don't look to me for free warranty support, and don't try to make money directly from this work.

(Oh, and just to make sure I haven't forgotten anything: Everything I just said applies to the binaries and the source code but not to the book itself. It, too, is copyrighted by Addison-Wesley, but we want everyone to pay for copies. We have to get something out of this!)


References

Cafiero, Bill. 2002. Remarks while leading a panel discussion at the semimonthly meeting of the Dallas/Fort Worth Electronic Commerce Forum, July 18, in Irving, Texas.

Caine, S., and E. K. Gordon. 1975. "PDL—a Tool for Software Design." In the American Federation of Information Processing Societies' Proceedings of the 1975 National Computer Conference, Anaheim, CA, 19–22 May 1975. Montvale, NJ: AFIPS Press pp. 271–276. Available online at http://www.cfg.com/pdl81/pdlpaper.html, accessed July 11, 2002.

Cline, Marshall. 2002. "C++ FAQ Lite." Available online at http://www.atd.ucar.edu/software/c++-faq/newbie.html#faq-29.7, accessed March 27, 2003.

Shaw, Mary, and David Garlan. 1996. Software Architecture—Perspectives on an Emerging Discipline. Upper Saddle River, NJ: Prentice Hall.

W3C (World Wide Web Consortium). 2002. Document Object Model main page. Available online at http://www.w3.org/DOM/, accessed July 10, 2002.


Resources

Apache Software Foundation: http://www.apache.org

Essential XML Quick Reference, by Aaron Skonnard and Martin Gudgin (Boston, MA: Addison-Wesley, 2002). I don't directly recommend very many other books in this book, but this is one I suggest without reservation. If you deal with XML on any technical basis at all, this book needs to be in your collection.

Microsoft's XML Resources: http://msdn.microsoft.com/xml

Sun Microsystem's Java Technology and XML: http://java.sun.com/xml

World Wide Web Consortium: http://www.w3.org

The XML Cover Pages: http://xml.coverpages.org


Chapter 2. Converting XML to CSV

Bart Simpson meets Wally and the Beaver

This chapter presents a utility for converting an XML document to a single CSV file. It describes foundation programming techniques for reading and parsing XML files using the DOM.

Bart Simpson, that icon of the 1990s with attitude, to me represents a lot of what we've associated with XML as a disruptive technology. Wally and the Beaver, despite being in strict chronological terms contemporaries more of ancient IBM mainframes and punch cards than of VisiCalc and dBase, still serve well as icons of simpler times and technologies.


Requirements

For desktop applications, the most commonly supported format for importing and exporting data is the comma-separated value format. So, it makes a lot of sense to develop a utility that converts XML documents to CSV files.

The general requirement for this utility is to convert an XML document to a CSV file. Design considerations dictate that the XML document is of a specific format, that is, that it complies with a specific XML grammar. XML documents in other formats may be transformed into this format using XSLT. Here's a summary of the required functionality.

  • Input: An XML instance document in a single file. The root Element may have any name. The root Element has one or more child Elements named Row, each representing a row in the target CSV file. Each Row Element has one or more child Elements that represent the columns in the row. These are named ColumnXX, where XX is the column number, with a leading number 0. The ColumnXX Elements are simple Elements with text context. The rationale for this grammar is given in the Design section later in this chapter.

  • Processing: Each input Row Element is written to a record in the CSV output file, with the ColumnXX Elements written to the appropriate column numbers. Text content of the columns is enclosed in quotation marks. Commas are inserted as field delimiters.

  • Output: A CSV file in Row/Column organization.

One qualification needs to be made here. In the strictest terms, the XML instance document does not have to be entirely contained within a single disk file. It may contain external entities that are defined in other XML documents in other disk files. However, this is so uncommon for most of the applications for which this book is targeted that I will stay with the generalization that we process a single XML instance document in a single file.

Why the Lowest Common Denominator?

CSV-formatted files are sometimes called the lowest common denominator for data exchange. This means that most applications that are able to export or import data will at least support CSV files. But why do we say lowest common denominator? In mathematics, the least common denominator is always the number 1. Don't we really care about the greatest common denominator, or the highest or best thing that is common? English is a funny language!


Running the Utility

This section provides instructions for running the XML to CSV Converter utility from the command line.

For Java:

java XMLToCSVBasic InputFile.XML OutputFile.CSV

or

java XMLToCSVBasic -h

For C++ on Win32:

XMLToCSVBasic InputFile.XML OutputFile.CSV

or

XMLToCSVBasic -h

Options follow the parameters except for the help option, which may be specified by itself.

Parameters:

  • First: File specification of the input XML document (required). The specification may include the full or relative path name. If no path name is specified, the file is assumed to reside in the current working directory. The full file name must be specified, but there is no restriction on the extension name.

  • Second: File specification of the output CSV file (required). The specification may include the full or relative path name. If no path name is specified, the file will be created in the current working directory. An extension may be specified, but if not specified the file will be created without one. If a file with the specified name already exists, it will be overwritten.

Options:

  • -h (Help): Display a help message and exit without further processing.

Restrictions:

  • A column may have a maximum of 1,024 characters (may be modified by parameter in the program source).

  • A row may be no longer than 16,384 characters (may be modified by parameter in the program source).

  • Quotation marks may not appear in the text content of a ColumnXX Element.

  • A maximum of 99 columns per row is supported, that is, columns may be numbered from 1 to 99.

  • If duplicate ColumnXX Elements occur in a Row Element, the ColumnXX last processed is the one written out.

  • There is no absolute limit on the number of rows; the number is only practically limited by system memory.

  • The input XML document is not validated against a schema or a DTD. (Validation will be introduced in Chapter 5.)


Sample Input and Output

We use a simple address book listing for the example for this utility. Each row is an entry for a person. The columns are listed below.

  1. Last name

  2. First name

  3. Street address, first line

  4. Street address, second line

  5. City

  6. State or province

  7. Zip or postal code

  8. Country

  9. Phone number

  10. E-mail address

Here's an address book in this format, with entries.

Sample Input Document (SimpleCSV.xml)
<?xml version="1.0" encoding="UTF-8"?>
<SimpleCSV>
  <Row>
    <Column01>Jones</Column01>
    <Column02>Mary</Column02>
    <Column03>312 Renner Road</Column03>
    <Column04>Apartment C</Column04>
    <Column05>Richardson</Column05>
    <Column06>TX</Column06>
    <Column07>75080</Column07>
    <Column08>USA</Column08>
    <Column09>972-996-1051</Column09>
  </Row>
  <Row>
    <Column01>Smith</Column01>
    <Column02>Sue</Column02>
    <Column03>Highway 118</Column03>
    <Column05>Terlingua</Column05>
    <Column06>TX</Column06>
    <Column07>79852</Column07>
    <Column10>desertrat@aol.com</Column10>
  </Row>
  <Row>
    <Column01>Fred</Column01>
    <Column02>Finger</Column02>
    <Column03>PO Box 999</Column03>
    <Column05>Nome</Column05>
    <Column06>AK</Column06>
    <Column07>99762</Column07>
    <Column10>icecube@msn.com</Column10>
  </Row>
</SimpleCSV>

Note that none of the rows have data for all ten columns. Column 4 for the second line of the street address, column 8 for the country, column 9 for the phone number, and column 10 for the e-mail address are all optional.

Successful processing should produce a CSV file that looks like this:

"Jones","Mary","312 Renner Road","Apartment C","Richardson","TX","75080",
"USA","972-996-1051"
"Smith","Sue","Highway 118",,"Terlingua","TX","79852",
 ,,"desertrat@aol.com"
"Fred","Finger","PO Box 999",,"Nome","AK","99762",
 ,,"icecube@msn.com"


Design

Naming Elements

Now we get to the question of why I chose this particular XML grammar and approach to naming Elements. We're agnostic to the name of the root Element, and the reason for this should be fairly obvious. The root Element doesn't end up in the output CSV file and I'm not validating against a DTD or schema. If I were validating, the root name would matter a lot. Likewise, the choice of Row to represent a row should be fairly noncontroversial. The only thing likely to cause anyone any heartburn is the way I named the column Elements. But, to paraphrase Abraham Lincoln, you can't please all the people all the time.

The problem here is that there isn't any basis for an absolute, cut-and-dried choice; it's a matter of tradeoffs. I will attempt to show you why my particular choice best fits our basic requirements and architectural approach. To do that, we'll look at the chosen approach and some alternatives. If you choose to make different tradeoffs, you can tweak what I've started with. This discussion isn't just academic and peculiar to this particular problem. Reviewing it and understanding the tradeoffs may help you deal with similar situations.

Here are the most sensible ways I can think of for naming the column Elements.

  1. Use the same name for all the columns (e.g., Column).

  2. Use a column name with an Attribute indicating the column number (e.g., Column ColumnNumber="1").

  3. Make the column number an intrinsic part of each column's name (e.g., Column01). This is the ColumnXX naming choice mentioned earlier.

So, what's wrong with approach 1? It is very simple, and simple is good. Since XML allows an Element to be repeated, it is valid syntactically. Why do we need anything more complicated? The main reason is that if a column doesn't have a data value and all the columns share the same name, you have to include an empty column Element to indicate no value. For example:

<Column></Column>

or

<Column/>

Depending on how you create your XML, implementing this approach may require extra caution. You need to make sure you don't skip creating an Element because there isn't any text content. It's similar to putting two commas next to each other to indicate an empty field in the CSV file; it just looks a bit uglier in XML.

The secondary reason comes down to a matter of style and programming practice. Using an explicit column number makes the meaning, or the semantics, more specific. Without an explicit column number, we must build indexes in programs to identify the columns. In the XPath syntax that XSLT uses for identifying things in XML documents, we would have to use a position as an index to identify the column.

Later in this section I'll discuss another reason for not using approach 1, but for now we'll reject this approach because it requires that we use empty Elements.

So, what's wrong with approach 2? Nothing. However, I think that approach 3 ends up being simpler for most end users than approach 2. Why? I, personally, find it more intuitive to say <Column01> instead of <Column ColumnNumber="1">. Even if you don't find approach 2 more intuitive than approach 3 (after all, intuitiveness is mostly subjective), there is a significant difference in the XSLT code that end users must write to use the two approaches. It would be more appropriate to illustrate this point using a fragment of XML that we might use as input to an XSLT processor. However, instead we'll look at a fragment of XML that is the output of an XSLT processor since it takes quite a bit less explanation. The example will be very relevant for the next chapter, in which we use the same XML grammar for representing CSV files. For approach 2, the XSLT stylesheet must always include the Attribute name and value, in contrast to approach 3, in which the syntax is a bit more compact and simpler. For example, here's some sample XSLT code using approach 2.

<xsl:template match="LastName">
  <Column ColumnNumber="1">
    <xsl:value-of select="LastName" />
  </Column>
</xsl:template>

Compare that with the following sample code, which uses approach 3.

<xsl:template match="LastName">
  <Column01>
    <xsl:value-of select="LastName" />
  </Column01>
</xsl:template>

Approach 2 is sexier and cooler for programmers who can just loop through the columns using an index. However, I would rather put ugly code in the C++ or Java program to extract the column number from the Element name than force an end user, who will be writing the XSLT stylesheets, to always use the ColumnNumber Attribute. As a final note on XSLT, you can also use a "for-each" loop in XSLT. We'll talk more about this in Chapter 10.

There is one final reason why the more explicit naming style, reflected in the ColumnXX names, makes more sense in many cases than the generic Column names in approaches 1 and 2. If we are validating the input XML document against a schema, we have the opportunity to enforce some business-level validation and edits before the data is imported into the application. For example, in the sample address book for this chapter we might want to make the first line of the street address mandatory and the second line as well as the country optional. With the columns named differently we can specify in the schema that Column03 is mandatory and Column04 and Column08 are optional. There is no way to do this if all the Elements are named Column.

This example shows why, in general, I prefer using very specific Element names to convey meaning, or semantics, rather than depending on a combination of Elements and Attributes. However, I acknowledge that this preference is in some ways grounded more in philosophical predilections than technology, so I'll talk more about approach 2 when we discuss alternatives and enhancements.

Before we leave the topic of naming styles, one other question needs to be answered. Why use cryptic names like Column01 and Column09 instead of meaningful names like LastName and PhoneNumber? The reason is simple. This particular utility is designed to be a general purpose utility and not simply a utility to convert an address book. We might have a different CSV file, representing an invoice, with the invoice number in the first column and the item description in the ninth column. So, for this utility we'll use the ColumnXX style. In Chapter 7 we'll develop more sophisticated general purpose CSV conversion utilities that allow you to specify your own Element names.

Module Logic

So, having gotten that little bit of business out of the way, let's get to the heart of the matter. Here's the pseudocode for the main routine. I have chosen to create a CSVRowWriter class to handle walking through the Column Elements of a single Row Element and to write them to the output stream.

Logic for the Main Routine
Parse input arguments from command line
IF help option is specified
  display help message and exit
ENDIF
Set up DOM XML environment (dependent on implementation)
Load input XML Document (dependent on implementation)
Open output file
Initialize CSVRowWriter object
NodeList of Rows <- Call Document's getElementsByTagName for all
  Elements named Row
DO until Rows NodeList.item[index] is null
  Call CSVRowWriter write method, passing NodeList.item[index]
  Increment index
ENDDO
Close input and output files

As we can easily see, the main part of the work is done by the CSV Row Writer. But before we move to that, let's look at a few lines in the main routine in a bit more detail.

Set up DOM XML environment (dependent on implementation)

Since the DOM doesn't speak to how its environment is set up, we could anticipate that implementers would make different decisions about how to do it. They did. I'll go into the details for our Java and C++ environments.

Load input XML Document (dependent on implementation)

The semantics for the Load operation are addressed in the DOM Level 3 requirements, but in current implementations document loading also is left to the implementers. In both our Java and C++ implementations, the Load operation returns a DOM Document object (or to be more specific, a pointer to it). Again, I'll go over the details.

NodeList of Rows <- Call Document's getElementsByTagName for all
  Elements named Row

getElementsByTagName is a method of the Document interface that returns a DOM NodeList object of DOM Element objects that match the passed Element name. A NodeList is, just as the name implies, an ordered collection of DOM Nodes. In the DOM, an XML instance document can be thought of as a tree. The complete instance document tree is represented by the DOM Document object. The root or ultimate parent Element of the XML instance document is the root of the tree. Each vertex or node in the document tree, whether it is an Element, Attribute, or text content, is a DOM Node. Using this tree model, we can think of our sample instance document as the tree shown in Figure 2.1.

Figure 2.1. Example Document Tree

On first impression, after hearing a DOM Document being described as a tree, you might think that the lowest-level Elements, in this case the ColumnXX Elements, would be the leaves of the tree. As we can see from Figure 2.1, this is not the case. Even the lowest-level Element Nodes can themselves have children. And, quite relevant to this example, the text content of each of the ColumnXX Element Nodes is represented by a DOM Text Node. On the other hand, Attributes are a bit of a special case in the DOM. While they are Nodes, they are not represented as child Nodes of the Elements to which they are tied. They are represented as properties of the Element. The set of Attributes associated with an Element is returned not in a NodeList but in a DOM NamedNodeMap.

The next bit of pseudocode shows how we process the rows in the NodeList.

DO until Rows NodeList.item[index] is null
  Call CSVRowWriter write method, passing NodeList.item[index]
  Increment index
ENDDO

Row Nodes in the NodeList returned by getElementsByTagName are retrieved using the NodeList interface's item method. The item method retrieves the Node from the NodeList that corresponds to a passed index. When we reach the end of the list, item returns null.

The CSVRowWriter write method and the methods it calls are where most of the work is done. In the Java and C++ implementations, I won't discuss various ancillary functions such as constructors. The code for them is on the book's Web site. This pattern will often be followed when discussing the other programs. Our primary focus in the design is the essential processing. Here's the pseudocode for the write method.

Logic for CSVRowWriter write Method
Columns NodeList <- Get Row Element's childNodes attribute
DO until Columns NodeList.item[index] is null
  Column Name <- get Element's NodeName attribute
  Column Number <- Derive from Column Name
  IF Column Number > Highest Column
    Highest Column <- Column Number
  ENDIF
  Column Array [Column Number] <- get nodeValue of
    item[index] firstChild Node
  Increment index
ENDDO
Output Buffer <- Call formatRow, passing Column Array and
  Highest Column
Write Output Buffer

The overall strategy of the write method is to build a NodeList of the row's columns, put the text content of each column into an array entry, format the array into one continuous string, then write out the string. Note that in the DO loop we retrieve the ColumnXX Element's specific name and parse it to get the column number. Because columns can be skipped we can't rely on the index into the Column NodeList to give us the correct column number.

We have three new DOM concepts in this bit of code. The line below uses a somewhat different approach than the getElementsByTagName method we used on the Document in the main method.

Columns NodeList <- Get Row Element's childNodes attribute

The DOM Element interface offers the same getElementsByTagName that the DOM Document interface does. This call worked at the Document level because we wanted a list of all Elements named Row. However, the Row Element's children are named ColumnXX, where XX goes from 01 through 99. Because of this we need to use a different approach. The Element interface inherits the Node interface's childNodes attribute, which has a type of NodeList. This list of childNodes has all the Node's child Elements, of all names. So, we'll use it to get all the Row's Column children.

There is another difference between the two approaches. The childNodes list includes not only all the child Elements but also the Text Nodes. This is one way we can get to Text Nodes when we need them. However, there is a second way to get to the text content of an element, as I'll show shortly.

Another new DOM concept is the Node interface's NodeName attribute.

Column Name <- get Element's NodeName attribute

There are various ways to retrieve the name of a DOM Element. In this case, because we have a list of Nodes, I use the Node interface's NodeName attribute.

To get the Element's text content, we capitalize on the expected document structure. The Column Node has only one child, its Text Node. So, we can use the DOM firstChild attribute to get to it directly and get its value (the DOM nodeValue attribute), which is the text content of the Column Element.

Column Array [Column Number] <- get nodeValue of
    item[index] firstChild Node

This is a very useful approach. In many cases in this book the Elements that have text content don't have child Elements.

Note that due to this and the other specialized routines used in my chosen design, successfully running the program depends on the input XML document having a very specific structure. If the document doesn't conform to the expected structure, this version of the program can blow up with various types of exceptions. Perhaps even worse, it can produce unexpected or no output with no indication of failure. To prevent such undesirable behavior we can validate the input document against a DTD or schema before we try to walk the tree. We'll talk about such validation in the Enhancements and Alternatives section near the end of the chapter.

The remainder of the code in the CSVRowWriter write method is fairly straightforward text and file I/O processing.

Output Buffer <- Call formatRow, passing Column Array and
  Highest Column
Write Output Buffer

We concatenate the contents of each column into one string while enclosing the contents of each individual column with quotation marks and inserting delimiting commas. Then we write the string to the output file. All of this code is very language dependent and is actually not very interesting from the viewpoint of a book that focuses on XML. Aside from pointing out that you can examine the full Java or C++ source code if you wish, I won't discuss it much more.

Having laid out the logical structure of the programs, we can now look at the Java and C++ implementations. While they are structurally similar and have the differences we can expect due to the differences in the programming languages, there are also some differences related to the DOM APIs. These are due mostly to the naming conventions used in MSXML. God love Microsoft. Despite the company's stated support for standards it just has to be different!

For each of the implementations I'll reference the pseudocode, then show the specific Java or C++ code that implements that section of code. I'll focus on the differences in DOM implementation and a few other interesting bits but won't discuss all the more mundane details. Again, the full source is available on the book's Web site.


Java Implementation

The Java implementation is composed of the following source files:

  • XMLToCSVSimple.java: the main routine for the Java application

  • CSVRowWriter.java: a class with write and formatRow methods

  • SAXErrorHandler.java: a class that handles exceptions while parsing the input XML document

The formatRow method is not very relevant to the main thrust of the book, so I won't discuss it further here other than to make one comment. The formatRow method uses only a comma as a field delimiter and always encloses fields with quotes. In Chapter 7 we'll develop a more flexible approach.

main in XMLToCSVBasic.java

Let's start with this bit of pseudocode:

Set up DOM XML environment (dependent on implementation)
Load input XML Document (dependent on implementation)

In JAXP, we build a DOM Document with the DocumentBuilder class. However, we can't just declare a DocumentBuilder; we must get one from a DocumentBuilderFactory. A real-world analogy would be a metal worker who wants to build a barbecue for his backyard. He needs tools to build it, and the ultimate source of those tools is a tool and die maker. So, we declare a tool and die maker (the DocumentBuilderFactory), who then makes us a tool (the DocumentBuilder) that we can use to build our barbecue (the Document). Here's how the Java code looks.

From XMLToCSVBasic.java
//  Set up DOM XML environment
DocumentBuilderFactory Factory =
    DocumentBuilderFactory.newInstance();

//  Create the new document builder
DocumentBuilder Builder = Factory.newDocumentBuilder();

//  Set the error handler
Builder.setErrorHandler(new SAXErrorHandler());

//  Load input XML Document (dependent on implementation)
//  Use our DocumentBuilder's parse method on the
//  disk file passed on the command line.

docInput = Builder.parse(new File(sInputXMLName));

The Builder.parse call is where we start making the barbecue. Also note the Builder.setErrorHandler call. We'll talk more about that soon.

The rest of the DOM-related code is pretty straightforward. Here's the Java code with the pseudocode embedded as comments.

//  NodeList of Rows <- Call Document's
//    getElementsByTagName for all elements named Row
RowList = docInput.getElementsByTagName("Row");

//  DO until Rows NodeList.item[index] is null
//    Call CSVRowWriter write method, passing
//        NodeList.item[index]
//    Increment index
//  ENDDO
while (RowList.item(iRows) != null)
{
  RowWriter.write(RowList.item(iRows));
  iRows++;
}

write in CSVRowWriter.java

As I said earlier, this is where most of the work gets done. Again, here's the DOM-relevant snippet of code with the pseudocode as comments. The passed Row Element is referred to as nRow. All the DOM work gets done in the DO loop.

From CSVRowWriter.java—write
//  Columns NodeList <- Get Row's childNodes attribute
ColumnList = nRow.getChildNodes();

//  DO until Columns NodeList.item[index] is null
while (ColumnList.item(iRowChildren) != null)
{
  // Skip the Row's Text nodes
  if ( ColumnList.item(iRowChildren).getNodeType()
    != Node.ELEMENT_NODE)
  {
    iRowChildren++;
    continue;
  }

  // Get a shorthand name for this guy
  nColumn = ColumnList.item(iRowChildren);

  //  Column Name <- get NodeName attribute
  sColumnName = nColumn.getNodeName();

  //  Column Number <- Derive from Column Name
  iColumnNumber = (new Integer(
    sColumnName.substring(6))).intValue();

  //  IF Column Number > Highest Column
  //    Highest Column <- Column Number
  //  ENDIF
  if (iColumnNumber > iHighestColumn)
    iHighestColumn = iColumnNumber;

  //  Column Array [Column Number] <- get nodeValue of
  //    item[index] firstChild Node
  sColumnArray[iColumnNumber] =
    nColumn.getFirstChild().getNodeValue();

  //  Increment index
  iRowChildren++;
//  ENDDO
}

WARNING

Watch out for unexpected Text Nodes with whitespace!

I need to point out one thing here. At the top of the loop we have these few lines:

// Skip the Row's Text nodes
if ( ColumnList.item(iRowChildren).getNodeType()
  != Node.ELEMENT_NODE)
{
  iRowChildren++;
  continue;
}

This is because the Row's NodeList had an unexpected Node preceding each ColumnXX Element Node. We have to skip them since the only children we want to process are the ColumnXX Element Nodes. This behavior was peculiar to JAXP and Xerces. I did not observe it with MSXML.

Interestingly, I observed this behavior only when processing a file that was "pretty printed," that is, each tag started on a new line with indentation. If I processed a file with no whitespace between the tags I didn't observe it. Some debugging statements in the code verified that these were indeed Text Nodes with contents of a new line and tab characters. (Thank you, XMLSPY!)

Another interesting thing is that calling the Row Element's normalize method did not make the Text Nodes go away or consolidate them into a single Text Node. However, if you think about it, a normalize call should not affect this behavior since, technically speaking, these are not adjacent Text Nodes. They occur between the Column Nodes, not next to each other.

To eliminate these Text Nodes, the DocumentBuilderFactory class has a method called "setIgnoringElementContentWhitespace". (I love these verbose Java method names!) This method strips out the "ignorable" whitespace from the content of Elements that have only child Elements and no Text content. However, the parser only knows of such elements from a DTD or a schema, so you have to be validating the input for this method to have any effect. We haven't included validation yet in our code, so the setIgnoringElementContentWhitespace method wouldn't have helped us here. We will use this method later when we do validation in Chapter 5.

I suppose that the Xerces and MSXML developers could have long arguments about which parser behaves correctly, but it is of little interest to me. The bottom line is to know how your parser behaves and to code your programs accordingly. Files intended to be used for business application import/export or exchange with trading partners rarely have such "mixed content" where an Element can have both data and child Elements. However, beware of unexpected Text Nodes and be prepared to handle them. Don't always assume that a Node List has only Elements.

Error Handling

Both JAXP and the Xerces DOM implementation follow the basic Java architecture for exception handling. Consistent with that architecture, most of the code in the main method lies within a try block that is followed by various catch blocks. JAXP implements an interesting overall parsing architecture in that it allows you to use a different underlying XML parser than the default Xerces implementation. One way JAXP does this is by requiring such pluggable parsers to use the SAX classes related to exception handling. So, whether or not your parser is actually a SAX parser under the hood, you need a SAX exception handler if you want to handle parsing exceptions. The JAXP documentation (in Javadoc format) has this to say about declaring an exception handler for your DocumentBuilder:

If an application does not register an ErrorHandler, XML parsing errors will go unreported and bizarre behavior may result.

Given that, I tend to think it's a good idea to declare an exception handler. However, SAX parsing errors are just one kind of error, even if they tend to be the most involved to handle. Here are some other errors we want to catch:

  • SAX parsing exceptions: warnings, errors, fatal errors

  • DOM exceptions of all types: for example, attempting to reference nonexistent DOM nodes and using an out-of-range index into a NodeList

  • All other Java exceptions: most likely I/O exceptions

To catch SAX parsing exceptions we declare an error handler for our DocumentBuilder as follows:

dbBuilder.setErrorHandler(new SAXErrorHandler());

The SAXErrorHandler class will be reused in all our Java utilities that read XML documents, so it is worth looking at in a bit of detail. It has four methods, the first three of which are required by the SAX ErrorHandler interface it implements.

  • warning: Under normal circumstances this will just print exception information to the standard system error stream and continue. However, it may itself also throw another SAXException if it encounters anything really bizarre.

  • error: This gets SAX-specific exception information and rethrows the exception.

  • fatalError: This is virtually the same as the error method. It gets SAX-specific exception information and rethrows the exception.

  • getSAXExceptionInfo: This utility formats interesting SAX exception information into a string.

Here's the code:

SAXErrorHandler.java
// Standard libraries
import java.io.*;

// JAXP packages
import javax.xml.parsers.*;
import org.xml.sax.*;
import org.xml.sax.helpers.*;

public class SAXErrorHandler implements ErrorHandler
{

  //  The first three methods are required by the SAX
  //  ErrorHandler interface. They deal with warnings, errors,
  //  and fatalErrors, respectively.

  public void warning(SAXParseException spe)
    throws SAXException
  {
    System.err.println("\nWarning during parsing " +
      getSAXExceptionInfo(spe));
  }

  public void error(SAXParseException spe)
    throws SAXException
  {
    String sInfo = "\nError during parsing " +
      getSAXExceptionInfo(spe);
    throw new SAXException(sInfo);
  }

  public void fatalError(SAXParseException spe)
    throws SAXException
  {
    String sInfo = "\nFatal error during parsing " +
      getSAXExceptionInfo(spe);
    throw new SAXException(sInfo);
  }

  //  This utility method gets SAX specific information about
  //  the exception
  private String getSAXExceptionInfo(SAXParseException spe)
  {
    String sInfo = "\n" +
      "URL    = " + spe.getSystemId() + "\n" +
      "Entity = " + spe.getPublicId() + "\n" +
      "Line = " + spe.getLineNumber() + "\n" +
      "Column = " + spe.getColumnNumber() + "\n" +
      "Text = " + spe.getMessage() + "\n";
    return sInfo;
  }
}

The use of URL is not as goofy as it looks on first take. We can report not only parsing errors in the instance document we're working with but also errors in parsing any associated schema files. So, the URL may actually give us some useful information.

Our general strategy is to continue if we encounter a warning and bail out if we encounter anything more severe. When we bail out, we exit back all the way out of the try block in the main routine and execute its catch block.

Keeping with the strategy of fairly simple error handling, the catch block for the main routine is pretty simple. We want it to handle DOM exceptions and anything else we might have tripped over such as I/O errors. The DOMException class extends java.lang.RuntimeException and doesn't have any special methods associated with it. So, for DOM exceptions and everything else throwable we just print the message and a stack trace to the standard system error stream, then set an exit status to an error condition.


C++ Implementation

The C++ implementation is composed of the following files:

  • XMLToCSVBasic.cpp: the main routine for the C++ application

  • CSVRowWriter.cpp: a class with write and formatRow methods

  • CSVRowWriter.h: the header file for the CSVRowWriter class

  • handleCOMError.cpp: the function that displays information about COM errors

  • displayParseError.cpp: the function that displays information about parsing errors

  • BlasterIncludes.h: the header file for the C++ code for including all standard headers and libraries, like importing MSXML

  • BlasterPrototypes.h: the header file with prototypes for handleCOMError and handleParseError

The first thing we need to keep in mind about the C++ implementation is that MSXML is a COM component (in Microsoft's Component Object Model). That said, there is a certain amount of COM baggage and weirdness that comes along with it. If you are already working in a COM environment, you will probably be comfortable with most of the COM-related concepts in this book. If you aren't, I have put together a quick reference in Appendix C that should help you with most of what you need to know to use a COM component like MSXML. If you fall into this category it would probably be a good idea to flip back to Appendix C and read it before you proceed with this section.

main in XMLToCSVBasic.cpp

The structure of the main routine is nearly identical to the Java implementation. This subsection covers a few things that need to be pointed out.

From the Logic for main
Set up DOM XML environment (dependent on implementation)
Load input XML Document (dependent on implementation)

Due to the COM considerations and the need to check for return values rather than relying on exceptions to be thrown, this translates into a bit more code in the C++ implementation than it did in Java.

From XMLToCSVBasic.cpp
//  Set up the COM environment, since MSXML is a
//  COM Component
hResult = CoInitialize(NULL);
if (FAILED(hResult))
{
  cerr << "Failed to initialize COM environment" << endl;
  return 0;
}

//  We'll do our main processing within a try block
//  We only expect to throw COM, MSXML parse
//  and I/O exceptions
try
{
  //  Declare smart pointers for our MSXML COM Interfaces.
  //  We'll use the Document2 class for all our stuff so that
  //  we can do schema validation.
  //  We do these within the try block so that when we exit
  //  the program they will be out of scope.
  IXMLDOMDocument2Ptr spDocInput;
  IXMLDOMNodeListPtr spRowList;

  //  Create the COM DOM Document object
  hResult =
      spDocInput.CreateInstance(__uuidof(DOMDocument40));

  if FAILED(hResult)
  {
    throw cCreateInstanceError;
  }

  //  Tell it we don't want to load the document
  //  asynchronously
  spDocInput->async =  VARIANT_FALSE;

  //  Load input XML Document (dependent on implementation)
  hResult = spDocInput->load(cInputXMLName);

  // Check for errors
  if( hResult != VARIANT_TRUE)
  {
    spParseError = spDocInput->GetparseError();
    cerr << "Parsing Error" << endl;
    displayParseError(spParseError);
    throw cParseError;
  }

We first need to set up our COM environment. Next, we create spDocInput (an IXMLDOMDocument2Ptr) as a COM object. We set its async property to false since we want to wait until it is completely loaded before we proceed. Finally, we load the document using the DOM Level 3 Load semantics.

Again, the rest of the main routine is pretty straightforward. Here's the part that does the most work.

//  Initialize CSVRowWriter object
RowWriter = new CSVRowWriter(&OutputCSV);

//  NodeList of Rows <- Call Document's
//    getElementsByTagName for all elements named Row
spRowList = spDocInput->getElementsByTagName("Row");

//  DO until Rows NodeList.item[index] is null
//    Call CSVRowWriter write method, passing
//      NodeList.item[index]
//    Increment index
//  ENDDO

while (spRowList->item[iRows] != NULL)
{
  IXMLDOMElementPtr spRow = spRowList->item[iRows];
  RowWriter->write(spRow);
  iRows++;
}

Note that spRowList is an IXMLDOMNodeListPtr object.

write in CSVRowWriter.cpp

As in the Java implementation, this is where most of the work is done. The Column Array is implemented as an array of character pointers. We call "new" to get memory for each cell when we have something to put into it. We later call "delete" in the formatRow method to clear out the cell and free the memory after we have formatted it to the output buffer. The GetnodeValue method that we use on the Column Node's Text Node returns a COM VARIANT, so we use the _bstr_t COM helper class to treat it as character data. Here's the most interesting part:

From CSVRowWriter.cpp—write
//  Columns NodeList <- Get Row's childNodes attribute
spColumnList = spRow->childNodes;

//  DO until Columns NodeList.item[index] is null
while (spColumnList->item[iRowChildren] != NULL)
{
  // Get a shorthand name for this guy
  IXMLDOMNodePtr spColumn;
  spColumn = spColumnList->item[iRowChildren];

  //  Column Name <- get NodeName attribute
  strcpy(cColumnName,spColumn->nodeName);

  //  Column Number <- Derive from Column Name
  strcpy(cColumnNumber,&(cColumnName[6]));
  iColumnNumber = atoi(cColumnNumber);

  //  IF Column Number > Highest Column
  //    Highest Column <- Column Number
  //  ENDIF
  if (iColumnNumber > iHighestColumn)
  {
    iHighestColumn = iColumnNumber;
  }

  //  Get memory for this column entry, and init it
  if (cColumnArray[iColumnNumber] == NULL)
  {
    cColumnArray[iColumnNumber] = new char[MAXCOLUMNSIZE];
    memset(cColumnArray[iColumnNumber],'\0',MAXCOLUMNSIZE);
  }

  //  Column Array [Column Number] <- get nodeValue of
  //    item[index] firstChild Node
  //  GetnodeValue is returned as a COM VARIANT, so we
  //  use _bstr_t to get to the character text

  strcpy(cColumnArray[iColumnNumber],
     _bstr_t(spColumn->firstChild->GetnodeValue()));

  //  Increment index
  iRowChildren++;

} //  ENDDO

Note that in the DO loop we don't check on the Node type. Unlike the Java implementation, we encountered only ColumnXX Element Nodes in the Row's list of child Nodes (no Text Nodes).

Error Handling

As might be expected, error handling is a bit different in C++ than it is in Java. A lot of this is due to basic differences between the languages. The Java class libraries natively use exceptions extensively, for everything from I/O problems to, in our programs, SAX exceptions. C++, on the other hand, while it does provide a facility for declaring, throwing, and catching exceptions, does not by default use them very much. The standard C++ class libraries usually set error flags instead of throwing exceptions. As a consequence, I generally will not make much use of exceptions in the C++ code. The only exception to this is in the main routines, and that is due to COM. Since MSXML is a COM object, we can throw COM exceptions. However unlikely this may actually be, good programming practice requires that the main part of our code be enclosed in a try block and followed by a catch block that checks for COM exceptions. Within that block I also throw a limited set of exceptions as a graceful way to exit the try block. Consistent with the KISS principle, these are thrown as simple char exceptions. The corresponding catch block has a set of case statements that take actions appropriate to each value.

The catch blocks in the main routine call functions to display information on COM exceptions and text about any other errors. However, we display information on parse errors right after we load, rather than doing it in the catch block. We do this so that we can limit the scope of our IXMLDOM smart pointers to the try block. This basic approach will be used in most of our other main programs.

Catch Blocks in XMLToCSVBasic.cpp
//  General purpose catch block
catch (char cException)
{
  switch (cException)
  {
  case  cCreateInstanceError:
    cerr << "Failed to create Document instance" << endl;
    cerr << "HRESULT = x" << hex << hResult << endl;
  case  cParseError:
    break;
  case  cOutputFileError:
    cerr << "Failure in opening output file " <<
      cOutputCSVName << endl;
    break;
  default:
    cerr << "Unexpected char exception " << cException
      << endl;
  }
  iExitStatus = 1;
}

//  Do some simple error handling for COM exceptions
catch (_com_error &e)
{
  handleCOMError(e);
  iExitStatus = 1;
}

The routine below displays COM exception information to cerr.

handleCOMError.cpp
void handleCOMError(_com_error &e)
{
  cerr << "COM Error" << endl;
  cerr << "Code = %08l" << hex << e.Error() << endl;
  cerr << "Message = " << e.ErrorMessage() << endl;

  _bstr_t bstrSource(e.Source());
  if (bstrSource.length() != 0)
  {
    cerr << "Source = " << (LPCSTR) bstrSource << endl;
  }

  _bstr_t bstrDescription(e.Description());
  if (bstrDescription.length() != 0)
  {
    cerr << "Description = " << (LPCSTR) bstrDescription <<
         endl;
  }
}

With any luck, you'll never see this bit of code executed during routine use. However, you may run into some COM exceptions during programming development. (I did.) Make sure to catch and report whatever scraps of information COM gives you. I stole the main ideas from the MSXML code samples but made a few changes. Not all exceptions report Source or Description, so I test before trying to output them to cerr.

The following routine displays detailed information about parsing errors using properties of the IXMLDOMParseError object (that's what an IXMLDOMParseErrorPtr points to).

From handleCOMError.cpp
void handleParseError(IXMLDOMDocument2Ptr spDocInput)
{

  IXMLDOMParseErrorPtr spParseError;

  spParseError = spDocInput->GetparseError();

  cerr << "Parse Error" << endl << endl;

  cerr << "File URL = " << spParseError->url << endl;
  cerr << "Position in file = " << spParseError->filepos <<
      endl;
  cerr << "Line = " << spParseError->line << endl;
  cerr << "Position in line = " << spParseError->linepos <<
      endl;
  cerr << "Reason = " << spParseError->reason << endl;
}

Other than these differences, the overall approach to error handling in the C++ implementation is the same as that of the Java implementation. When we hit an error we try to exit as quickly and gracefully as possible, displaying as much information as might be helpful.


Comparing the Java and C++ Implementations

As noted when discussing the Java implementation, Xerces returned Text Node children of the Row Element preceding each ColumnXX Element.

Aside from the differences in error handling, we quickly see a few other differences due to the basic differences between the languages. One that stands out is how pointers are handled. In C++ pointers are explicitly declared as such and are manipulated and referenced differently than the variables and objects they point to. In Java nearly everything except a simple variable is a pointer, but the pointers are implicit. A case of this difference in the two implementations is that in C++ with MSXML we declare a pointer to a DOM object, but in Java we just declare the DOM object. We also have the expected differences in basic library functions like I/O. Most of the other differences are due to the COM baggage that MSXML carries.

One striking thing about the two implementations is how similar they are when dealing with things that the DOM Recommendation specifies and how different they are in areas where it is silent. This is a very graphic demonstration of the power of standard APIs such as DOM and SAX.


Enhancements and Alternatives

So, having built the basic functionality and seen how it works, what more can we do?

Validation of the Input Document

The most obvious enhancement we can make to this program is to validate the input document before we try to process it. As I previously pointed out, we can run into unexpected output, no output, or abnormal termination if the input document doesn't conform to the expected structure. There are two ways we can handle this.

The first way is to add additional code to the program in several places to deal with unexpected conditions. For example, in the main method we would check to see that there was at least one Row Node in the NodeList. We would want to do the same thing when checking for ColumnXX children of the Row Nodes in the CSVRowWriter write method. In addition, we would want to verify that the Row Node's children were Element Nodes and that their names started with Column before we tried to process them as columns. This is a lot of code to add, and you can easily imagine the code you would need to add in more complex applications.

Fortunately, there's an easy way out. The second alternative is to validate the document against a DTD or schema before we process it, letting the DOM API do the heavy lifting. In both the Java and C++ implementations we have to add only a line or two of code. After having given you this teaser, I'll wait until Chapter 5 to actually show you the code. There are, however, other considerations having to do with whether or not we should validate against a schema. I'll discuss these in Chapter 12.

You should now ask, "So what do I validate against?" Ah, there's the question! In Chapter 4 I'll introduce the W3C XML Schema Recommendation and in Chapter 5 the specific schema that can be used for this program and the one developed in the next chapter.

Using a ColumnNumber Attribute

When presenting the chosen document design, I discussed some alternative approaches for naming the column Elements. Let's look now at how the code would change if we used a generic Element name (Column) and an Attribute indicating the column number (that is, approach 2, instead of my chosen ColumnXX approach 3).

In the CSVRowWriter write method we replace the call to get the column's NodeName and parse the ColumnXX name to get the number with a single call to the Element Node's getAttribute method. This is shown in the pseudocode snippet below.

Column Number <- call Column Element's getAttribute to get
  value of ColumnNumber Attribute

Intuitively, this seems as if it would be a simpler approach since we would save one line of pseudocode and a few lines in Java and C++. From a programmer's perspective your intuition would be right. However, as noted earlier there are perspectives to consider other than just that of the person doing the coding. We need to consider the impact on the end users, and this approach makes their jobs harder. When given a choice, I'll always make things easier for the end user rather than the programmer.

A Recursive Algorithm

Now let's talk about a more radical alternative approach to designing this utility. We have outlined in this program the basic strategy for walking through the tree that represents a DOM Document. For each selected Node we get a list of the Node's children. Then for each of the children we get a list of children and process them however we want. Keeping with my "clarity over cleverness" orientation, I've designed specialized routines that deal specifically with the expected structure of the input XML document. However, it isn't too hard to see how this approach can be generalized into a recursive algorithm that walks the complete tree and takes actions that are appropriate to each type or name of Node encountered. Listed below is the pseudocode for a recursive approach that processes Document, Row, Column, and Text Nodes.

Logic for the Main Routine: Recursive Implementation
Parse input arguments from command line
IF help option is specified
  display help message and exit
ENDIF
Set up DOM XML environment (dependent on implementation)
Load input XML Document (dependent on implementation)
Open output file
Initialize CSVRowWriter object
Call processNode, passing Document object and null pointers for
  Column Array and Column Text
Close input and output files

The main routine is quite similar to the original main routine. We see the differences primarily in the processNode method.

Logic for the CSVRowWriter.processNode Routine: Recursive Implementation
NodeList <- Get passed Node's childNodes attributes
DO CASE of Node's nodeName
  Document:
    // Process the Document's Row children
    DO for each of the Node's children
      Call processNode, passing child Node and null pointers
        for Column Array and Column Text
    ENDDO
  Row:
    // Process the Row's ColumnXX children
    Initialize Column Array
    DO for each of the Node's children
    Call processNode, passing child Node and pointer
      to Column Array, null pointer to Column Text
    ENDDO
    Output Buffer <- Call formatRow, passing Column Array
    Write Output Buffer
  ColumnXX:
    // Process the Column's #Text children
    Column Number <- Derive from Column Name
    DO for each of the Node's children
      Call processNode, passing child Node and null pointer
        to Column Array, pointer to Column Text
    ENDDO
    Column Array[Column Number] <- Column Text
  #Text:
    // Get the text
    Column Text <- get nodeValue
  Default:
    No operation
ENDDO

If you understand how recursive programming works, this routine should be easy enough to figure out. However, we need to ask this question: Just because we can solve this problem recursively, should we? If all we wanted to do was to walk the document tree and print out Node names and values, a recursive approach might be very appropriate. However, in this case we have a different kind of problem to solve. We do different types of processing for each type of Node we encounter. The different types of processing require that we pass arguments to the routine that are used only in specific cases and not in others. Overall, we need to ask whether or not this approach contributes to our goals of simplicity, understandability, and maintainability. You may run into some kinds of XML processing problems where a recursive algorithm contributes to those goals. In this case I don't think it does.

The one advantage to this type of recursive approach is that we have a Default case to handle unexpected Nodes. However, if we validate the input XML document, this advantage is moot. Even though I don't think that a recursive algorithm has any advantages for this particular utility, I did want to point it out to you in case you might find it applicable to a problem you need to solve. In Chapters 8 and 9 I present two cases where a recursive approach does make sense.

Advanced Functionality

As useful as this XML to CSV utility is, it's probably well suited only for a limited range of applications. A different XML to CSV converter would be helpful for dealing with the following real-life situations.

  • Writing several input XML documents to a single physical output CSV file. (In the conventional way of doing things, an XML document is stored on disk as one physical file. However, most batch-oriented import facilities take in several logical documents as one physical file.)

  • Performing data type conversions such as conversion from W3C XML Schema ISO 8601 date formats to formats such as MM/DD/YY or DD/MM/YY.

  • Supporting a delimiter other than a comma.

  • Avoiding the use of quotation marks around all text fields.

These form the start of the requirements list for the more capable utility we'll build in Chapter 7.


Resources

DOM Level 2 Core Specification, November 13, 2000, by the World Wide Web Consortium. Available online at http://www.w3.org/DOM/DOMTR#dom2.

DOM Reference, Microsoft XML Core Services (MSXML) 4.0, by Microsoft Corporation. Online documentation included as part of the MSXML SDK. Available online at http://msdn.microsoft.com/xml.

Java API for XML Processing Specification 1.2, by Sun Microsystems. Online Javadoc documentation, included as part of the Java XML Pack Spring 2002. Available online at http://java.sun.com/xml.


Chapter 3. Converting CSV to XML

The Beaver finds the way to San Jose

This chapter presents a utility for converting a single CSV file to an XML instance document. It describes foundation programming techniques for creating XML files using the DOM.


Requirements

This utility provides the reverse functionality of the one we developed in the previous chapter. The file organization and rationales are the same.

The general requirement for this utility is to convert a CSV file to an XML instance document. Design considerations dictate that the XML document is of a specific format, that is, a specific XML grammar. Supporting more than one grammar for the output document will make the code a lot more complicated. Our output document may be transformed into other XML grammars using XSLT. The sample output shows both the direct output of the utility and what it might be transformed into using XSLT. The list below summarizes the required functionality.

  • Input: A CSV file in uniform Row/Column organization, that is, each row has the same column entries.

  • Processing: Each input CSV row is written to an Element named Row in the output XML document. Each column in the row is written to a child ColumnXX Element in the output XML document, where XX represents the column number. Quotation marks bracketing text and delimiting commas are discarded. Columns without content don't generate Elements.

  • Output: An XML instance document in a single file. The root Element is named SimpleCSV, with children Row and ColumnXX children of the Row.


Running the Utility

This section provides instructions for running the CSV to XML Converter utility from the command line.

For Java:

java CSVToXMLBasic InputFile.CSV OutputFile.XML

or

java CSVToXMLBasic -h

For C++ on Win32:

CSVToXMLBasic InputFile.CSV OutputFile.XML

or

CSVToXMLBasic -h

Options follow the parameters except for the help option, which may be specified by itself.

Parameters:

  • First: File specification of the input CSV file (required). The specification may include the full or relative path name. If no path name is specified, the file is assumed to reside in the current working directory. The full file name must be specified, but there is no restriction on the extension name.

  • Second: File specification of the output XML document (required). The specification may include the full or relative path name. If no path name is specified, the file will be created in the current working directory. An extension may be specified, but if not specified the file will be created without one. If a file with the specified name already exists, it will be overwritten.

Options:

  • -h (Help): Display help message and exit without further processing.

Restrictions:

  • A quotation mark (") may only be used as a delimiter to bracket the entire text context of a column. Quotation marks may not otherwise appear in the input file.

  • A column may have a maximum of 1,024 characters (may be modified by parameter in the program source).

  • A row may be no longer than 16,384 characters (may be modified by parameter in the program source).

  • A maximum of 99 columns per row is supported, that is, columns may be numbered from 1 to 99. Columns numbered higher than 99 will not appear in the output XML document.

  • There is no absolute limit on the number of rows; the number is only practically limited by system memory.

  • The output XML document is not validated against a schema or DTD. (Validation will be introduced in Chapter 5.)

We need to note one thing that isn't a restriction. XML uses special characters such as the left and right angle brackets (< and >, respectively) for key parts of its syntax. Unlike the way in which some EDI processors create documents in the X12 EDI syntax, you don't need to be concerned about whether or not any of these special characters appear in your data. The utility will handle them properly. I'll discuss this more at the end of this chapter in regard to predefined entities.


Sample Input and Output

We again use the simple address book listing for the example for this utility. Each row is an entry for a person. The columns are listed below.

  1. Last name

  2. First name

  3. Street address, first line

  4. Street address, second line

  5. City

  6. State or province

  7. Zip or postal code

  8. Country

  9. Phone number

  10. E-mail address

Here's a sample input CSV file for an address book in this format (simple.CSV).

"Jones","Mary","312 Renner Road","Apartment C","Richardson","TX","75080",
   "USA","972-996-1051"
"Smith","Sue","Highway 118",,"Terlingua","TX","79852",
  ,,"desertrat@aol.com"
"Fred","Finger","PO Box 999",,"Nome","AK","99762",
  ,,"icecube@msn.com"

Successful processing should produce an XML document that looks like the sample shown below.

Sample Output Document (SimpleCSV.xml)
<?xml version="1.0" encoding="UTF-8"?>
<SimpleCSV>
  <Row>
    <Column01>Jones</Column01>
    <Column02>Mary</Column02>
    <Column03>312 Renner Road</Column03>
    <Column04>Apartment C</Column04>
    <Column05>Richardson</Column05>
    <Column06>TX</Column06>
    <Column07>75080</Column07>
    <Column08>USA</Column08>
    <Column09>972-996-1051</Column09>
  </Row>
  <Row>
    <Column01>Smith</Column01>
    <Column02>Sue</Column02>
    <Column03> Highway 118</Column03>
    <Column05>Terlingua</Column05>
    <Column06>TX</Column06>
    <Column07>79852</Column07>
    <Column10>desertrat@aol.com</Column10>
  </Row>
  <Row>
    <Column01>Fred</Column01>
    <Column02>Finger</Column02>
    <Column03>PO Box 999</Column03>
    <Column05>Nome</Column05>
    <Column06>AK</Column06>
    <Column07>99762</Column07>
    <Column10>icecube@msn.com</Column10>
  </Row>
</SimpleCSV>

Again, as with our example in the previous chapter, none of the rows have data for all the columns since some of them (such as column 4, the second line of the street address) are optional.

Using techniques that will be discussed in Chapter 10, we can easily convert this output into the following format using XSLT.

Sample Converted Output Document
<?xml version="1.0" encoding="UTF-8"?>
<AddressBook>
  <AddressEntry>
    <LastName>Jones</LastName>
    <FirstName>Mary</FirstName>
    <Street1>312 Renner Road</Street1>
    <Street2>Apartment C</Street2>
    <City>Richardson</City>
    <State>TX</State>
    <ZipCode>75080</ZipCode>
    <County>USA</County>
    <Telephone>972-996-1051</Telephone>
  </AddressEntry>
  <AddressEntry>
    <LastName>Smith</LastName>
    <FirstName>Sue</FirstName>
    <Street1>Highway 118</Street1>
    <City>Terlingua</City>
    <State>TX</State>
    <ZipCode>79852</ZipCode>
    <E-Mail>desertrat@aol.com</E-Mail>
  </AddressEntry>
  <AddressEntry>
    <LastName>Fred</LastName>
    <FirstName>Finger</FirstName>
    <Street1>PO Box 999</Street1>
    <City>Nome</City>
    <State>AK</State>
    <ZipCode>99762</ZipCode>
    <E-Mail>icecube@msn.com</E-Mail>
  </AddressEntry>
</AddressBook>

You may ask, "Why not just create this output in the first place?" We certainly could, quite easily. However, this is intended to be a general purpose utility, not one used just to convert address books. So, the generic ColumnXX names are appropriate.


Design

Having discussed the rationale for the file organization and element naming decisions in Chapter 2, we focus here on the basics of creating and writing out an XML document using the DOM.

Main Routine

Here's the pseudocode for the main routine. In the previous chapter I created a CSVRowWriter class to handle walking through the Column Elements of a single Row Element and writing them to the output stream. This routine uses a CSVRowReader to parse an input CSV row and transform it into a Row and child ColumnXX elements.

Logic for the Main Routine
Parse input arguments from command line
IF help option is specified
  display help message and exit
ENDIF
Open input file
Set up DOM XML environment (dependent on implementation)
Create the Output XML Document (dependent on implementation)
Root Element <- Call Document's createElement method, with
  tagName of SimpleCSV
Document <-  Call Document's appendChild to append Root Element
  to Document
Initialize CSVRowReader object using the Output Document, and
  the Root Element
Read first record from file
DO until all records are read
  Increment Row Count
  Call CSVRowReader parse method, passing the record read from
    the input file
  Call CSVRowReader write method
  Read next record from input file
ENDDO
Save Output XML Document (dependent on implementation)
Close input file

The main part of the work is done by the CSVRowReader class. However, before we move on let's look a bit more at a few lines relating to the DOM.

Create the Output XML Document (dependent on implementation)

This is another case where the DOM recommendation (at least through Level 2 of the DOM) is silent, so we can expect there will be variations among the different DOM implementations. The one thing that both the JAXP and MSXML implementations have in common is that they both create a pointer to the Document Node. The Document Node, basically, has no content whatsoever. It just gives us a hook from which we can start hanging things.

Root Element <- Call Document's createElement method, with
  tagName of SimpleCSV
Document <-  Call Document's appendChild to append Root Element
  to Document

The first thing we need to do is to create the root Element of the document, calling the Document interface's createElement method. This takes a single argument of the name of the Element to be created. In our example output above, this is the SimpleCSV Element that has all of the Row Elements as children. Note for future reference that any time we want to create any type of DOM Node (except for Attributes), we must call the appropriate create method from the Document interface (the only interface that has create methods). Another important thing to note is that until we tie a new Node to something, it is just floating off by itself in space. We must use the appendChild method of the appropriate parent Node to hang it on the tree. In this case, we want the root Element to be a child of the Document Node, so we call the Document interface's appendChild method.

In the DO loop we read a record, parse it, and then write it out to the DOM tree. We'll look at these methods shortly, but the thing to note here is that we are separating the reading, parsing, and output functions. This is consistent with good programming practice: Each function has a single logical function, that is, has cohesion. It is also consistent with the design of most compilers. In the big picture, what the utilities in this book do is very similar to what compilers do.

Save Output XML Document (dependent on implementation)

Again, we'll see implementation differences. Both Xerces and MSXML implement the semantics of the DOM Level 3 save operation. In this case, we'll see that the Microsoft implementation looks in the actual code closer to the standard than Xerces.

CSVRowReader Class

The CSVRowReader parse and write methods do most of the work in this utility. In creating the CSVRowReader class, I use the following class attributes in addition to the parse and write methods:

  • Output Document

  • Root Element

  • ColumnArray

The Output Document and root Element are initialized in the class constructor, where the Column Array is also set up.

In both of our implementations the parse method deals strictly with standard Java or C++ things and has nothing to do with XML per se. However, the logic of the parse method is of interest since getting the parsing algorithm correct is very important when dealing with CSV files (as well as when dealing with EDI in Chapter 9). Let's look at the pseudocode for the parse routine, then focus on the DO loop where the actual parsing occurs.

Logic for the CSVRowReader parse Method
Inputs - InputRecord
Local - ColumnNumber, CurrentPosition, QuoteOpen, ColumnActive
ColumnNumber <- 1
CurrentPosition <- 0
ColumnActive, QuoteOpen <- false
DO until CurrentPosition = Length of InputRecord
  DO CASE of InputRecord[CurrentPosition]
    Quote:
      IF QuoteOpen is true
        QuoteOpen <- false
      ELSE
        QuoteOpen <- true
      ENDIF
      BREAK
    Comma:
      IF QuoteOpen is false
        Increment ColumnNumber
        ColumnActive <- false
        BREAK
      ENDIF
      NOTE: We fall through to Other for commas within
        Quoted strings
    Other:
      IF ColumnActive is false
        ColumnActive <- true
      ENDIF
      ColumnArray[ColumnNumber] <- ColumnArray[ColumnNumber] +
          InputRecord[CurrentPosition]
  ENDDO CASE
  Increment CurrentPosition
ENDDO

In the DO loop we are performing character-by-character parsing. I have set this up to handle column text that may or may not be delimited by quotation marks. We can't do simple string token operations looking for commas because a comma might appear within a quoted string of text. That's the main reason we use quotation marks in the first place. In "compiler speak," we are doing simple left-to-right parsing with no lookahead.

There are many ways to implement such parsers. To start out simple in this book I'm using a DO CASE strategy. The general strategy allows us to recognize quotation marks as delimiters for text and not move them to the output. A comma indicates the end of a column unless the comma is preceded by a single quotation mark, which indicates the comma is part of a text string. The first character in a column flags that the column is started. We then save that and all other characters, including the commas that have been "escaped" by preceding quotation marks, to the entry in the column array. Note that this algorithm can't deal with quotation marks that appear in column text, that is, that aren't used as delimiters. I didn't see a significant requirement to deal with that circumstance. Handling it would add more complexity to the algorithm than it was worth.

As noted, this is a fairly simple parsing algorithm appropriate for fairly simple input. EDI, on the other hand, while it has many similarities to CSV formats, has a much more complex grammar that is more amenable to more sophisticated parsing techniques. Parsing non-XML input will get a lot more interesting later in the book!

The write method does the XML DOM work.

Logic for the CSVRowReader write Method
Inputs - None
Local - ColumnNumber, Column Name, Row Element, Column Element,
    Text Node
Row Element <- Call Document's createElement method with "Row" as
    the tagName
Root Element <- Call root Element's appendChild to add Row
    Element as child
DO for all ColumnNumber from 1 to MaxColumns
  IF ColumnArray[ColumnNumber] has content
    Column Name <- "Column" and ColumnNumber
    Column Element <- Call Document's createElement method with
        ColumnName as the tagName
    Row Node <- Call Row Node's appendChild to add ColumnElement
        as child
    Text Node <- Call Document's createTextNode method with text
        from ColumnArray[ColumnNumber]
    Column Element <- Call Column Element's appendChild to add
        Text Node as child
    ENDIF
    Increment ColumnNumber
ENDDO

The first thing we do in this routine, similar to the main routine, is to create a Node from which we can hang everything else. We create the Row element and append it to the root Element of the Document Node. Note that in the main method we created the root Element and appended it to the Document Node itself. We then loop through the column array. For every column that has text we build the ColumnXX Element name, create and append the Column Element to the Row Element, then create and append a Text Node to the Column Element. The createTextNode method takes as an argument the text to be inserted into the Node.


Java Implementation

The Java implementation is composed of the following source files:

  • CSVToXMLSimple.java: the main routine for the Java application

  • CSVRowReader.java: a class with parse and write methods

Note that SAXErrorHandler (used in the Java implementation of the XML to CSV Converter utility) is missing. More on that below.

main in CSVToXMLBasic.java

Let's start with the implementation-dependent bit of pseudocode.

Set up DOM XML environment (dependent on implementation)
Create the Output XML Document (dependent on implementation)

There are at least three different ways to use JAXP and Xerces to set up the DOM and create a DOM Document. In the Xerces package org.apache.xerces.dom there is a DocumentImpl interface that allows us to create a DOM Document with a single method and no arguments.

Document docOutput = new DocumentImpl();

However, this interface is not part of the standard JAXP implementation and is listed as one of the internal APIs in the Xerces Javadocs's Implementation Documentation section. Portability and maintainability considerations lead me to avoid using anything so nonstandard and implementation-dependent.

Another alternative exists in the relatively more standard JAXP implementation. By using the DocumentBuilder class and the DocumentBuilderFactory (as we did in Chapter 2), we can get a DOMImplementation from the DocumentBuilder, then call its createDocument method. The createDocument method takes three arguments: the URI of the namespace of the document being created, the qualified name of the document, and the document type of the document. The document type can either be set to null or be set up using the DOMImplementation's createDocumentType method. The code would look something like this, assuming we left the document type null.

DocumentBuilderFactory Factory =
  DocumentBuilderFactory.newInstance();
DocumentBuilder Builder = Factory.newDocumentBuilder();
DOMImplementation DOMImpl = Builder.getDOMImplementation();
Document docOutput =
  DOMImpl.createDocument("urn:BabelBlaster.org",
  "bb:SimpleCSV");

While this is fully supported in JAXP, it is also fairly cumbersome. In addition, with the discussion of document type, this approach shows its origins of using DTDs rather than schemas. There is nothing wrong with this approach and it may be appropriate for some uses. However, we'll use a simpler method. The DocumentBuilder also has a newDocument method. Here's how we use it.

DocumentBuilderFactory Factory =
  DocumentBuilderFactory.newInstance();
DocumentBuilder Builder = Factory.newDocumentBuilder();
docOutput = Builder.newDocument();

Note that in this program we don't call the setErrorHandler method to declare an error handler class that handles SAX parsing exceptions. This is because we aren't parsing! We are creating an empty DOM Document instead of parsing an existing one. The only XML-related exceptions we throw are DOM exceptions, and SAXErrorHandler doesn't deal with those.

The code for creating the root Element and appending it to the Document Node is pretty straightforward. The code snippet below shows how these two operations are done in Java. We won't need to look at them again after this.

//  rootElement <- Call Document's createElement method,
//    with tagName of SimpleCSV
eleRoot = docOutput.createElement("SimpleCSV");

//  Document <-  Call Document's appendChild to append
//    Root to Document
docOutput.appendChild(eleRoot);

The other interesting part of the Java implementation is saving the document. Here's the code:

//  Save Output XML Document (dependent on
//    implementation)
//  Open file output stream
OutputXML = new FileOutputStream(sOutputXMLName);

//  Create an XML Serializer and set it to use
//    the file output stream
MySerializer = new XMLSerializer();
MySerializer.setOutputByteStream(OutputXML);

//  Create the Output Format for the Serializer for
//    XML, UTF-8, and indentation true
MyOutputFormat = new OutputFormat("XML","UTF-8",true);
MySerializer.setOutputFormat(MyOutputFormat);

//  Set the serializer to be a DOM serializer
MySerializer.asDOMSerializer();

//  Write out the document
MySerializer.serialize(docOutput);

In this case the Java code actually looks less like the DOM recommendation than the MSXML C++ code. Version 1.2 of JAXP doesn't have anything that directly implements the DOM Level 3 recommendation "save Document" semantics. We get this functionality from Xerces by way of the XMLSerializer in the org.apache.xml.serialize package. The documentation for the XMLSerializer is found in the Other Classes section of the Xerces API Javadocs. The XMLSerializer provides both DOM and SAX approaches to serializing a document. True to the Java way of doing things, it uses an existing java.io.OutputStream or java.io.Writer to do the actual writing. So, I created a FileOutput stream for the output file. The XMLSerializer also requires that an OutputFormat class be specified. The XMLSerializer can be set up with both the OutputStream or Writer and the OutputFormat in the constructor. However, for clarity I show them being set up with separate calls. I chose to take a simple route and used the OutputFormat constructor, which specifies creation of an XML Document, with UTF-8 encoding and indentation set to true.

The next step before actually serializing is to set the XMLSerializer to be a DOM serializer since we have to tell the XMLSerializer whether we're using DOM or SAX. Finally, the actual save semantics are implemented by calling the XMLSerializer's serialize method, passing the DOM Document as the single parameter.

DOM Level 3 had not yet been approved when this version of Xerces was developed, which was several months before I wrote this chapter. It appeared from the documentation that these classes were still undergoing some development. So, if you are writing Java code to do DOM serialization, it would be good to get the latest version of Xerces and review the functionality of the XMLSerializer and related classes.

parse in CSVRowReader.java

The parse method is straight Java without any XML involved. It is pretty much a one-for-one implementation of the pseudocode. Review the source file if you're interested.

write in CSVRowReader.java

The write method is also a lot of Java with little new in the way of XML. The only new concepts here arise in creating the Text Node and appending it to the ColumnXX Node.

From CSVRowReader.java—write
//  Text Node <- Call Document's createTextNode method
//    with text from ColumnArray[ColumnNumber]
Text texText = docOutput.createTextNode(
  sbColumnArray[iColumnNumber].toString());

//  Free the memory for this column
sbColumnArray[iColumnNumber] = null;

//  Column Element <- Call Column Element's appendChild
//    to add Text Node as child
eleColumn.appendChild(texText);

As noted above, we call the Document interface's createText method, passing the contents of the ColumnArray entry (implemented as an array of StringBuffers). Then we call the Column Element's appendChild method.


C++ Implementation

The C++ implementation is composed of the following files:

  • CSVtoXMLBasic.cpp: the main routine for the C++ application

  • CSVRowReader.cpp: the class with parse and write methods

  • CSVRowReader.h: the header file for the CSVRowReader class

  • handleCOMError.cpp: the function that displays information about COM errors

  • handleParseError.cpp: the function that displays information about parsing errors

  • BlasterIncludes.h: the header file for the C++ code for including all standard headers and libraries, like importing MSXML

  • BlasterPrototypes.h: the header file with prototypes for handleCOMError and handleParseError

Similar to the discussion of the Java implementation, we'll focus below on the aspects that are unique to C++ and MSXML.

main in CSVToXMLBasic.cpp

Setting up the COM environment is nearly the same as discussed for XMLToCSVBasic.cpp in Chapter 2. There are two differences in the setup, followed by the calls to create the root Element and append it to the Document.

From CSVRowReader main
//  Set up DOM XML environment (dependent on implementation)
//  Create the Output XML Document (dependent on
//    implementation)
//  Create the COM DOM Document
hResult = spDocOutput.CreateInstance(
    __uuidof(DOMDocument40));

if FAILED(hResult)
{
  throw cCreateInstanceError;
}

//  Create the header processing instruction
//    and append it to the Document Node
spPIHeader = spDocOutput->createProcessingInstruction("xml",
  cHeaderText);
spDocOutput->appendChild(spPIHeader);

//  Root Element <- Call Document's createElement method,
//    with tagName of SimpleCSV
spEleRoot = spDocOutput->createElement("SimpleCSV");

//  Document <-  Call Document's appendChild to append Root
//    to Document
spDocOutput->appendChild(spEleRoot);

Note that we have to call only the CreateInstance method and not also the load method. Since we don't call the load method, we also don't have to check for or deal with parsing errors. In the Java implementation, Xerces's XMLSerializer class wrote the header line (formally known as the prolog) of the XML file for us. The following processing instruction line has the XML version and encoding:

<?xml version="1.0" encoding="UTF-8"?>

MSXML didn't do this for us, so we have to explicitly create it. In the strictest sense of the XML Recommendation, the XML prolog is not a processing instruction. However, the only way we can create the prolog with MSXML is to create it as a DOM Processing Instruction Node. The createProcessingInstruction method takes two arguments: (1) a target representing the entity or application for which the instruction is intended, and (2) the text of the instruction. In this case our target is xml and the text is composed of the version and encoding Attributes of the prolog. I've created a char * constant with the instruction text. The calls to create the root Element and append it to the Document come after appending the Processing Instruction header since we want the header to be the first thing in the instance document. These calls are almost identical to the Java calls. The createElement method specifies a _bstr_t for the tagName argument, but the compiler accepts a char * and does the conversion for us.

The other thing to point out in the C++ code is the save operation.

//  Save Output XML Document (dependent on implementation)
//  Save it using Document's save method
hResult = spDocOutput->save(cOutputXMLName);

if FAILED(hResult)
{
  throw cSaveError;
}

MSXML does a very nice, concise implementation of the DOM Level 3 save semantics. A save method is provided for the Document interface, with a single argument of the output. Various types of output are supported, including disk files and another Document object. The save method specifies that the output argument is a VARIANT, but the compiler accepts a file name passed as a char *. We again use a simple homegrown char exception as a way to gracefully exit the main routine's try block.

parse in CSVRowReader.cpp

Similar to the Java implementation, this is straight C++ without any XML involved. You can find the code on the book's Web site.

write in CSVRowReader.cpp

Let's look at how we create the Text Node and append it using MSXML from C++.

//  Text Node <- Call Document's createTextNode method
//    with text from ColumnArray[ColumnNumber]
//  MSXML createTextNode specifies a BSTR for the
//    argument. Visual C++ is usually happy to convert
//    a char * but wanted an explicit cast for this
//    array element.
IXMLDOMTextPtr spTexText = spDocOutput->createTextNode(
  (char *) cColumnArray[iColumnNumber]);

//  Column Element <- Call Column Element's appendChild
//    to add Text Node as child
spEleColumn->appendChild(spTexText);

Again, this is very similar to the Java implementation. A "gotcha" here, as noted in the code comments, is that createTextNode requires that the text argument be a BSTR type. The compiler is usually very happy to handle a char * for us, but in this case it wanted a specific cast as char *. cColumnArray is an array of char * pointers. We allocate memory for them when we read a new column and release it when we've created the Text Node. Visual C++ evidently couldn't handle that level of type conversion.


Comparing the Java and C++ Implementations

In the previous chapter we talked about the differences in DOM support, error handling, and native language features. We again see differences between the two implementations, and a common theme emerges. While C++ with MSXML is a bit more complex in some areas due to setting up the COM environment and checking for COM errors, the code that actually does the DOM-related functions is much more concise. The load and save operations are single calls with no setup. Compare this with the corresponding operations in JAXP and Xerces. Creating a Document requires a DocumentBuilderFactory, a DocumentBuilder, and finally a call to the create or parse method. Saving a Document using the XML Serializer is even more involved.

To me, this is more reflective of the languages than anything else. Java tends to be fairly sparse in the code needed to do things like string manipulation and error handling, whereas it tends to be "class crazy" in its APIs. There is a wealth of classes, but finding the one that provides the functionality you need sometimes can take a bit of searching. C++, on the other hand, requires more code for error handling and string manipulation (at least when using char arrays, as discussed in Chapter 1). Where functionality is provided through an API or built-in language feature, C++ tends to be fairly concise.

Again, we see a great deal of similarity in the areas that the DOM Level 2 recommendation deals with and divergence in the areas where it is silent.


Enhancements and Alternatives

So, what else would we like to do with the CSV to XML Converter? Some of the enhancements discussed in the previous chapter also apply to this utility. Here are a few others.

Validation of the Output Document

With both implementations we are confident of creating well-formed XML. However, wouldn't it be nice in some cases to be able to validate the instance document against a schema, too? This might be particularly important if you were going to send the document directly to a consuming application, especially if that application belongs not to you but to another organization. Chapter 5 will show how to do this.

Advanced Functionality

As was the case with our XML to CSV Converter in Chapter 2, the CSV to XML Converter provides only some very basic functionality. There are several other things we might want such a converter to do.

  • Reading several input XML documents to create a single physical output CSV file. (In the conventional way of doing things, an XML document is stored on disk as one physical file. However, most batch-oriented import facilities take several logical documents in as one physical file.)

  • Using Element names that are more semantically meaningful for specific applications (e.g., LastName instead of Column01).

  • Performing data type conversions such as conversion from the W3C XML Schema ISO 8601 date formats to MM/DD/YY, DD/MM/YY, or other date formats.

  • Supporting characters other than a comma for the column delimiter.

  • Supporting characters other than a quotation mark for the text delimiter.

These form the start of the requirements list for the more capable utility we'll build in Chapter 7.

Some Observations

In reviewing the pseudocode and the implementations, it should be apparent that creating an XML instance document using the DOM is fairly simple. It requires a series of create operations on the appropriate Node type, followed by an appendChild method call to attach the Node to the appropriate parent. Other types of XML applications might require more complex processing, but a fairly serial conversion of one document format to another requires only these simple operations. There are some minor complexities involved in setting up the XML environment and dealing with the different save operations. However, even with error handling these still take less than 50 or so lines of code. The main complexity comes in keeping track of the Node to which we want to append the Node we're creating. If we take EDI as a rough indication, the majority of business documents shouldn't have trees more than about six nodes deep. With reasonably designed parsing and processing algorithms, dealing even with these types of documents shouldn't be very hard.

So, in addition to converting an XML format to a non-XML format, we've also now looked at the basic mechanics of converting from a non-XML format to an XML format. It should be clear that the DOM APIs are extremely useful and save an awful lot of coding. If you consider the previously discussed complexities of reliably parsing a simple CSV file, the idea of writing code to parse an instance document, parse one or more schemas, validate the schemas, and validate the instance document against the schemas should give you the heebie-jeebies! Use the APIs. Don't try to write your own code to read or create raw XML.

In addition to the obvious things we've explored so far, the APIs do one other thing that hasn't been obvious from our sample data. They also handle predefined entities for us. These entities represent special syntax characters declared in the basic XML 1.0 recommendation. Consider the following input row, mangled to include some of these special characters (note that I left out a quotation mark because my CSV parser can't handle it!).

"<Smith","Sue>","Highway & 118",,"Ter'lingua","TX&","79852",
,,"desertrat@aol.com"

Running it through the Java utility produces the following output.

<Row>
  <Column01>&lt;Smith</Column01>
  <Column02>Sue&gt;</Column02>
  <Column03>Highway &amp; 118</Column03>
  <Column05>Ter&apos;lingua</Column05>
  <Column06>TX&amp;</Column06>
  <Column07>79852</Column07>
  <Column10>desertrat@aol.com</Column10>
</Row>

Running it through the C++ utility produces the output below.

<Row>
  <Column01>&lt;Smith</Column01>
  <Column02>Sue&gt;</Column02>
  <Column03>Highway &amp; 118</Column03>
  <Column05>Ter'lingua</Column05>
  <Column06>TX&amp;</Column06>
  <Column07>79852</Column07>
  <Column10>desertrat@aol.com</Column10>
</Row>

Since the <, >, &, ", and ' characters all have special meaning in XML syntax, they are converted to entities that don't give parsers problems. The reverse process is performed when we read an input XML instance document and convert it to another format; the real characters are restored.

As another implementation difference, note that Xerces converts the apostrophe to an entity, while MSXML doesn't.

Some of the other complexities you might encounter in parsing XML include whitespace handling, support for different Unicode character sets (including double byte characters), entities other than the predefined entities, and embedded comments. And these situations have only to do with parsing; validation is an entirely different endeavor. I try very hard in this book not to give unqualified recommendations because it is very rare that one size fits all. However, this is one case where I'll make an exception. Don't try to write your own code to parse XML if there's any way you can avoid it. The standard APIs will save you more time and work in the long run than you can imagine.

This fairly short chapter shows how creating XML using the DOM is actually pretty easy. Having laid the foundation with these basic techniques, we can proceed to address utilities that deal with more complex legacy file formats and situations. Much of what we will be concerned with in later chapters has more to do with these legacy formats than XML. However, these chapters will be of interest to XML-oriented developers who may not have previously dealt with these types of requirements. We will also look at some of the key XML technologies that support our overall architecture. We start in Chapter 4 with the W3C XML Schema language.


Resources

DOM Level 2 Core Specification, November 13, 2000, by the World Wide Web Consortium. Available online at http://www.w3.org/DOM/DOMTR#dom2.

DOM Reference, Microsoft XML Core Services (MSXML) 4.0, by Microsoft Corporation. Online documentation included as part of the MSXML SDK. Available online at http://msdn.microsoft.com/xml.

Java API for XML Processing Specification 1.2, by Sun Microsystems. Online Javadoc documentation, included as part of the Java XML Pack Spring 2002. Available online at http://java.sun.com/xml.

Xerces2 API Javadoc, by Apache Software Foundation. Online Javadoc documentation, included as part of the Xerces2 SDK. Available online at http://xml.apache.org.


Chapter 4. Learning to Read XML Schemas

Young men preen. Old men scheme.

—Mason Cooley

This chapter addresses basic features of the W3C XML Schema language with the intention of helping you read and understand schemas written in it.


Rope, Anyone?

Back in a previous lifetime when I worked as a consultant for Digital Equipment Corporation, we all loved using the VMS operating system. However, we also joked that it was so capable and flexible that it gave us a thousand different ways to hang ourselves. The W3C XML Schema language is very similar. You can write a nearly infinite number of schemas to describe any particular instance document. Every one of these schemas can provide an accurate description of the document, be loaded into a parser, and validate the document.

XML Schemas and the Formal Study of Grammars

The situation of multiple schemas being able to validate the same instance document is very analogous to a situation in the formal study of languages and automata in computer science. In this realm of study, a language is not necessarily a spoken language like French or English. It can comprise even nonsense strings of characters so long as they exhibit certain properties. An immediate application of this field of study is building compilers that transform programs in languages like Java and C++ into executable code.

Languages can be grouped by certain properties they exhibit. One of these groups is a class that can be specified by a special type of grammar called context-free grammar, or CFG. An interesting property of these languages and CFGs is that almost an infinite number of CFGs can be developed to describe a given language. There are well-established algorithms for determining whether or not two CFGs describe the same language. I've not seen anything equivalent for XML schemas yet, but given enough time I'm sure someone will develop a tool to do it.

If there were only one way to write a schema I'm not sure I would devote a whole chapter to the topic. However, because all of this variation is possible, a certain level of instruction is called for in order to help you understand schemas. Understanding the schemas you use is important for a number of reasons. When used, schemas are the ultimate authority on the format of your XML instance documents. If you encounter parsing or validation errors you may have to refer to the schema as well as the instance document to determine the source of the error. If you are like most people who use XML, your trading partners or application vendors will tell you which schemas to use. Because there is sometimes a perception that schemas are fully definitive and self-documenting, the schema may be the only documentation you'll get regarding how your instance documents are supposed to look.

In this chapter I attempt to convey to you the basics of what you need to know to read and make sense of schemas written in W3C's XML Schema language. This chapter focuses just on understanding the suckers. In Chapter 5 we'll write code to use schemas for validation. Later in this chapter we'll talk about creating schemas, and we'll revisit the topic in Chapter 12. However, for the most part I'll avoid discussing the advantages and disadvantages of the various schema design options. The topic of best practices in schema design is fairly broad and sometimes contentious. Since I anticipate that most of you reading this will not be designing schemas, I'll limit the discussion to a more pragmatic scope.

If you want a good introduction to most of the features of the schema language, you can't do better than W3C's Primer (see the Resources section at the end of this chapter). However, this chapter takes a different tack. While the Primer tries to describe most of the important schema features, I'll focus only on features and examples you are likely to see in schemas describing business documents (as opposed to wedding invitations, Web pages, or design documents for weapons systems). Although the Primer uses a purchase order as one of its main examples, many of the schema features discussed are not commonly used in business documents (not yet, anyway).

Although I have good words to say about the Primer, which is Part 0 of the XML Schema Recommendation, I don't have such good words to say about Parts 1 and 2, Structures and Datatypes, respectively. I find parts of them nearly impenetrable, and I have a master's degree in computer science! There also seems to me to be a great deal of functionality that is unnecessarily overlapping or outright duplicative. I don't want to come across as too harsh here since I have worked on standards committees myself and know firsthand what can happen when writing a document by committee. However, even when compared with other W3C Recommendations, Parts 1 and 2 of the Schema Recommendation don't measure up very well.

There are schema languages other than the W3C XML Schema language. However, despite my mixed feelings about the W3C XML Schema Recommendation, I'm not going to talk about those other schema languages. It's not that I think that Relax NG or any of its cousins are technically deficient or harder to use than the W3C XML Schema language. Far from it. The truth is that the market just isn't very interested in them. Almost everyone looks to the W3C as the final authority on all things XML. They have spoken.

NOTE Chapter Conventions

When dealing with the topic of schemas it can sometimes be difficult to keep track of whether one is talking about an individual instance of a schema document or referring to the schema language or recommendation. To try to keep these distinct I'll use the conventions of referring to the W3C XML Schema language as "schema language" and to the W3C XML Schema Recommendation as "Schema Recommendation." I'll use "schema" to refer to an individual schema, that is, an instance document written in the W3C XML Schema language.

In addition, I frequently use the xs: namespace qualifier prefix to help make it clear that I'm referring to specific Elements from the W3C XML Schema language.


A Few Different Document Approaches

I have long maintained that you have to understand the data before you can really understand the system. A similar truth holds for schemas; the best way to understand a particular schema is to first understand an instance document that conforms to the schema. A few major variations in how instance documents look have predictable impacts on the schemas.

  • Naming conventions: You are going to see a wide variety in style, lengths, and abbreviations. You'll also see some variety in capitalization and word separators, though upper camel case (e.g., UpperCamelCase) seems to be the preferred style. Just hope that whoever wrote the schema you are using picks one style and uses it consistently. You'll probably feel like shopping for rope if they didn't.

  • Elements and Attributes: Especially for the early adopters who were somewhat constrained by the limits of DTDs and the first parsers, many people developing schemas exclusively use Elements to convey business data. However, some use a mixture of Elements and Attributes. Most who do the latter have a semantic basis for deciding which should be which, but unfortunately not all do. You will also see a style of using Attributes strictly to convey metadata (data about data) or control information, rather than for data produced or consumed by a business application. For awhile Microsoft was promoting a style that used Attributes exclusively. This allowed a few bytes to be dropped since empty Elements didn't need end tags. This style seems to have passed.

  • Specific names versus qualifier/value pairs: An example of this contrast can be found in phone numbers. The specific name approach uses a construct like the following.

    <BusinessPhoneNumber>972-555-1212</BusinessPhoneNumber>

    The qualifier/value pair approach shown below uses a generic Element name with a qualifier to convey more specific meaning. In this example the qualifier is an Attribute, but both the qualifier and value could just as easily be sibling Elements under the same parent.

    <PhoneNumber phoneNumberType="BIZ">972-555-1212</PhoneNumber>

    Again, in this chapter I'm trying not to comment too much on the merits. Just be aware that you're probably going to run into both styles.

  • Structure: This topic is a bit problematical. XML lends itself very naturally to expressing the relationships between different data items by putting them under the same parent in a hierarchy. However, in some cases this leads to duplicated or repeated data. A good example of this is a purchase order with "ship to" locations at the line item level, with several lines going to the same locations. There are proposals in some organizations to use a more relational model. In this example the "ship to" locations might be listed in a header area, each with an ID number. Then, each line item would reference the "ship to" ID rather than duplicating the complete address. I've yet to see any widely used schemas that take this type of approach, but don't be surprised if you see it.


DTD Refresher

The Schema Recommendation has been approved for two years as this book goes to press. I wouldn't be surprised if you find some applications still using DTDs instead of schemas. Let's take a quick look at DTDs, using the XML representation of our simple CSV address book file (limited to only ten columns for brevity). Here's the instance document with two rows, followed by the DTD.

SimpleCSV with DTD (SimpleCSV1DTD.xml)
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE SimpleCSV SYSTEM "SimpleCSV1.dtd">
<SimpleCSV>
  <Row>
    <Column01>Jones</Column01>
    <Column02>Mary</Column02>
    <Column03>312 Renner Road</Column03>
    <Column04>Apartment C</Column04>
    <Column05>Richardson</Column05>
    <Column06>TX</Column06>
    <Column07>75080</Column07>
    <Column08>USA</Column08>
    <Column09>972-996-1051</Column09>
  </Row>
  <Row>
    <Column01>Smith</Column01>
    <Column02>Sue</Column02>
    <Column03>Highway 118</Column03>
    <Column05>Terlingua</Column05>
    <Column06>TX</Column06>
    <Column07>79852</Column07>
    <Column10>desertrat@aol.com</Column10>
  </Row>
</SimpleCSV>

The line that contains DOCTYPE is the document type declaration. It says that the DTD associated with this document is named SimpleCSV and that it may be found at the URI specified after the SYSTEM keyword. (We'll talk more about URIs in the Understanding Namespaces section later in this chapter.)

Here's the SimpleCSV DTD.

DTD for SimpleCSV (SimpleCSV1.dtd)
<?xml version="1.0" encoding="UTF-8"?>
<!ELEMENT SimpleCSV (Row+)>
<!ELEMENT Row (Column01, Column02, Column03, Column04?, Column05,
    Column06, Column07, Column08?, Column09?, Column10?)>
<!ELEMENT Column01 (#PCDATA)>
<!ELEMENT Column02 (#PCDATA)>
<!ELEMENT Column03 (#PCDATA)>
<!ELEMENT Column04 (#PCDATA)>
<!ELEMENT Column05 (#PCDATA)>
<!ELEMENT Column06 (#PCDATA)>
<!ELEMENT Column07 (#PCDATA)>
<!ELEMENT Column08 (#PCDATA)>
<!ELEMENT Column09 (#PCDATA)>
<!ELEMENT Column10 (#PCDATA)>

The general form of an Element declaration is ELEMENT, followed by the Element name, followed by the Element content description in parentheses. The last ten lines describe the column Elements. They indicate that the columns have parsed character data (PCDATA) as content. The second line specifies that a Row Element is a sequence of Elements named Column01 through Column10. The question marks after some of the columns indicate, in the Extended Backus-Naur Form notation, that they are optional. The first line shows that the SimpleCSV document is composed of a sequence of one or more Row Elements, the plus sign (+) indicating one or more.

If our document used the ColumnNumber Attribute discussed in Chapter 2, the instance document would look like this:

<Column ColumnNumber="1">Jones</Column>

The relevant part of the DTD would look like this, assuming that use of the Attribute was required:

<!ELEMENT Column (#PCDATA)>
<!ATTLIST Column ColumnNumber CDATA #REQUIRED>

The general form of the ATTLIST declaration has the Element name followed by a series of individual Attribute declarations. Each of these declarations is a triple. The first member of the triple is the Attribute name. The second is the Attribute Type, which is usually a CDATA string or an enumeration. The third is a requirement designation (somewhat confusingly referred to in the Recommendation as the "default declaration"). The requirement designator can take on three basic forms: (1) a value of REQUIRED, meaning that the Attribute must be present; (2) a value of IMPLIED, meaning that it may be absent; and (3) a default value for the Attribute, declared in the requirement designator. In the third form, if the Attribute is not actually present in an instance document, the parser should return this default value as if the Attribute were present in the instance document with the value. The keyword FIXED may precede the default value in the declaration, indicating that the Attribute must always have the default value.

You may also encounter entities, which function like macros in programming languages. An entity is declared with a shorthand name and its contents. It can then be used in the DTD or an instance document and an XML processor will replace the Entity reference with its content. We saw an example of this in the previous chapter with the predefined entities for special syntax characters.

Those are the basics. If you need to delve any further, check my Web site for recommendations for a few good basic books on XML. The more technically inclined (or adventurous) may also want go straight to the authoritative source on DTDs, the W3C XML 1.0 Recommendation.


Foundation Concepts and Terminology

Although schemas are a tremendous advance over DTDs, they introduce some new concepts and terminology to XML that take some getting used to. It will help to review some of these before we discuss more specific features.

Elements and Types

All programming languages make a distinction between a variable, which is a named location in memory, and a variable's data type, which can put constraints on the variable's range of values or how the variable should be interpreted. In general, the closest thing that XML 1.0 DTDs have to this separation is PCDATA for parsed character data and CDATA for (unparsed) character data. The XML Schema Recommendation implements a more conventional view of variables and data types. (I should add "in general" because this is true for Elements. Attribute declarations have a bit more expressive power since they can be declared as CDATA, ID, IDREF, and so on.)

NOTE "Data Type" or "Datatype"?

The Schema Recommendation uses "datatype," while most programming languages and texts use "data type." We'll go with convention and use the latter, but keep in mind that in reference to the schema language I also mean the former.

In addition to defining a set of base types to be used for simple Elements and Attributes, the schema language also specifies a way to define complex types. These are similar to structs in C, GROUPs in COBOL, and records in other languages. This facility greatly aids the reuse of common constructs.

Both user-defined complex types and the built-in simple data types may be modified through the mechanisms of restriction and extension. These mechanisms and the most common simple data types will be discussed shortly.

Simple and Complex

One of the things I found somewhat confusing when first learning schema language is how it uses the terms "simple" and "complex." These are used in two different contexts.

In the first context, the terms "simple" and "complex" describe the type of content an Element may have, or its content model. In the most general terms, an Element may be declared in schema language with one of two content models.

  • Simple content: The Element has no child Elements.

  • Complex content: The Element may have child Elements.

These types of content are represented using the schema language Elements xs:simpleContent and xs:complexContent, respectively. In addition to simple and complex, there are various types of content models for complex content. The most common are sequence and choice. We'll talk more about these later in the chapter. Mixed content indicates both child Elements and character data are permissible. This content model is used infrequently in business document schemas intended for use directly by applications.

The other context in which the terms "simple" and "complex" are used is in referring to types.

  • Simple type: A simple type is either one of the built-in schema data types or a type derived from one of them. An Element that is of a simple type may not have Attributes.

  • Complex type: A complex type may take several forms. A complex type is used to define Elements that may have child Elements. A complex type is used to define any Element that may have Attributes. A complex type is also used to declare that an Element may have mixed content.

These are declared in schema language using the xs:simpleType and xs:complexType Elements, respectively. For both of these types, schema language supports the concept of derivation, similar to the way it is used in object-oriented analysis, using the mechanisms of extension and restriction. These mechanisms are used somewhat differently in simple and complex types.

Simple types are derived from built-in schema language data types or other user-defined simple types by restriction. Restriction of a simple type involves reducing the set of values allowed for the type.

Complex types are either declared (created by the user) or derived from other types. Complex types may be derived from simple types by extension, usually by adding one or more Attributes. Complex types may also be derived from other complex types by either restriction or extension. Restriction generally involves removing child Elements from the content model, while extension generally involves adding Elements or Attributes. Restriction and extension will be discussed in more detail in later sections.

What can be a bit confusing is that there isn't a one-to-one correspondence between simple and complex content and simple and complex types. The use of "complex" is consistent since Elements that are defined to allow complex content always have a complex type. However, there isn't any such consistency with the use of "simple." Elements that are defined to allow only simple content may be of either simple type or complex type. An Element that is defined with simple content but that allows one or more Attributes always has a complex type. However, we can define a complex type, derived from a simple type by extension, that has a simple content model and no Attributes.

The best advice I can give you here is to make sure you understand the context, that is, whether you are dealing with content models or types, and don't confuse the two.


General Features

Having laid down some basic terminology, we're ready to look at some commonly used features. We'll start with a couple of general features, then move on to simple and complex Elements.

Let's begin with a document that references a schema and the schema it references.

SimpleCSV with Schema (SimpleCSV1.xml)
<?xml version="1.0" encoding="UTF-8"?>
<SimpleCSV
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:noNamespaceSchemaLocation="SimpleCSV1.xsd">
  <Row>
    <Column01>Jones</Column01>
    <Column02>Mary</Column02>
    <Column03>312 Renner Road</Column03>
    <Column04>Apartment C</Column04>
    <Column05>Richardson</Column05>
    <Column06>TX</Column06>
    <Column07>75080</Column07>
    <Column08>USA</Column08>
    <Column09>972-996-1051</Column09>
  </Row>
  <Row>
    <Column01>Smith</Column01>
    <Column02>Sue</Column02>
    <Column03>Highway 118</Column03>
    <Column05>Terlingua</Column05>
    <Column06>TX</Column06>
    <Column07>79852</Column07>
    <Column10>desertrat@aol.com</Column10>
  </Row>
</SimpleCSV>
Schema for SimpleCSV1 (SimpleCSV1.xsd)
<?xml version="1.0" encoding="UTF-8"?>
<!-- edited with XMLSPY v4.3 U (http://www.xmlspy.com)
    by Michael C Rawlins (Rawlins EC Consulting) -->
<!--W3C Schema generated by XMLSPY v4.3 U
    (http://www.xmlspy.com)-->
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
    elementFormDefault="unqualified">
  <xs:annotation>
    <xs:documentation>This is the schema for the SimpleCSV
        format developed in Chapters 2 and 3
    </xs:documentation>
  </xs:annotation>
  <xs:element name="SimpleCSV">
    <xs:annotation>
      <xs:documentation>This is the root element for the
          document, defined anonymously as a complexType
      </xs:documentation>
    </xs:annotation>
    <xs:complexType>
      <xs:sequence>
        <xs:element name="Row" maxOccurs="unbounded">
          <xs:annotation>
            <xs:documentation>This is the Row element defined
                anonymously as a complexType
            </xs:documentation>
          </xs:annotation>
          <xs:complexType>
            <xs:sequence>
              <xs:annotation>
                <xs:documentation>In this constrained example
                    we have ten columns, all using the built-in
                    string data type and all are optional
                </xs:documentation>
              </xs:annotation>
              <xs:element name="Column01" type="xs:string"
                  minOccurs="0"/>
              <xs:element name="Column02" type="xs:string"
                  minOccurs="0"/>
              <xs:element name="Column03" type="xs:string"
                  minOccurs="0"/>
              <xs:element name="Column04" type="xs:string"
                  minOccurs="0"/>
              <xs:element name="Column05" type="xs:string"
                  minOccurs="0"/>
              <xs:element name="Column06" type="xs:string"
                  minOccurs="0"/>
              <xs:element name="Column07" type="xs:string"
                  minOccurs="0"/>
              <xs:element name="Column08" type="xs:string"
                  minOccurs="0"/>
              <xs:element name="Column09" type="xs:string"
                  minOccurs="0"/>
              <xs:element name="Column10" type="xs:string"
                  minOccurs="0"/>
            </xs:sequence>
          </xs:complexType>
        </xs:element>
      </xs:sequence>
    </xs:complexType>
  </xs:element>
</xs:schema>

We'll go into all the details shortly, but here are a couple of things to quickly note. One is that the schema is itself a well-formed XML instance document. You don't see it in the schema, but it even conforms to a schema and DTD that are part of the Schema Recommendation. The other thing to note is the indentation. As with all "ignorable whitespace" in XML documents, the indentation is strictly for readability. With a few exceptions it has no significance to XML processors. The exceptions are documentation Elements and Elements of the string data type. Both will be discussed later in the chapter.

Schema Declaration in Instance Documents

Let's first look at the schema declaration in the instance document. It takes the form of Attributes on the document's root Element.

<SimpleCSV
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:noNamespaceSchemaLocation="SimpleCSV1.xsd">

The first Attribute, xmlns, indicates the value of the xsi namespace prefix. (We'll talk more about namespaces later in this chapter.) It is needed here so that we know how to interpret the second Attribute, which is the one we're really interested in. The noNamespaceSchemaLocation (in the xsi namespace) indicates both that we don't have an explicit target namespace for this instance document and where to look for the schema. In this case, since there is no additional information, it tells us to look for SimpleCSV1.xsd in the same location as this instance document. The XML Schema Recommendation specifies that the value of this Attribute is a Uniform Resource Identifier (URI). We'll talk more about the interesting world of URIs, URNs, and URLs later in this chapter.

Comments and Documentation

Apparently, many people have bought into the notion that XML is self-documenting. Or maybe they subscribe to the old programming adage, "It was hard to write. It should be hard to read!" Whatever the case, not everyone believes in documenting his or her schemas. For the blessed folks who do, schema language offers two choices. You will see both in the schemas you use.

  • xs:documentation Element inside an xs:annotation Element: This is the preferred style. When used in this fashion, tools such as schema authors display the documentation nicely. You can see a few such examples throughout this chapter.

  • XML comments: The traditional style of creating XML comments by using the start (<!--) and end (-->) characters is allowed but not preferred. It is usually only readable when looking at the raw XML version of a schema and is not normally displayed in other views offered by schema authors. The comments in SimpleCSV1.xsd were inserted by XMLSPY. This is not a plug for XMLSPY; I just left them in so you'll know what they look like. (They will not show up in later schemas.)

Fortunately, almost no one expects anyone to read comments in instance documents. They are rarely used.

Element Declarations

Elements are declared using the schema language xs:element Element. There are two basic methods for declaring an Element. An Element may be declared with a name Attribute that specifies the name of the Element. It may also, as with the column Elements in our example, have a type Attribute that indicates the name of the Element's type. Every Element must have a type, but there are various ways in schema language to specify the type. We'll talk in this chapter about those options as we encounter their usage.

In addition to being declared via a name, an Element may be declared by a reference using the ref Attribute. In this method the reference is to another declaration where the Element has been declared with a name. This method is discussed in more detail in the subsection that deals with global Elements later in this chapter.


Simple Content Elements

In this section we'll talk about Elements that are declared with no child Elements, that is, those with simple content models. The main thing we're concerned with in regard to these Elements is imposing restrictions on the built-in schema data types, reducing the set of allowable values.

Data types for Elements with simple content are specified in Part 2: Datatypes of the XML Schema Recommendation.

Schema Built-in Data Types

Schema language provides a very large set of data types. If you work with enough different schemas for long enough, you may eventually see all the data types used. However, the set I list below accounts for most you will see in common business documents.

  • string: This data type is just what its name implies. The important thing to note is that an Element with a type of string can contain any Unicode character, including those with double-byte encoding, if so specified in the instance document prolog.

  • boolean: Logically true or false, but the full set of allowed values for this data type is true, false, 0, and 1.

  • decimal: This is a number that may have a fractional part, that is, values to the left and right of the decimal point. However, it need not have both. Float and double are similar but aren't seen as often in business document schemas. A leading sign is allowed but may be omitted if positive. Leading and trailing zeroes are optional. If the fractional part is zero, the decimal and trailing zeroes may be omitted unless they're needed to indicate precision.

  • integer: Again, this data type is just what the name implies. Specialized integer types include nonPositiveInteger, negativeInteger, long, int, and short.

  • date, time, and dateTime: These data types represent units of time as specified by ISO 8601 (the International Organization for Standardization's representations of dates and times). The most commonly used are CCYY-MM-DD, hh:mm:ss.sss, and CCYY-MM-DDThh:mm:ss, respectively. Fractional seconds may be omitted, but seconds are required. The T indicates a time zone. A Z indicates Coordinated Universal Time (UTC). The local time is indicated by using the T and following the seconds with an offset. For example, 2002-08-23T20:27:00-05:00 could indicate August 23, 2002, 3:27 PM U.S. Central Daylight Time. Be aware that an offset from UTC is not necessarily the same thing as a time zone, although a time zone could be inferred from other knowledge. For example, -05:00 could be either Central Daylight Time or Eastern Standard Time depending on the day of the year.

To reiterate, there are many other built-in data types, but the short list above encompasses those most commonly used. If you encounter a type that isn't on this list, look for the definition in the Schema Recommendation.

Don't let talk of value space, canonical representations, lexical space, lexical representations, constraining facets, and order relations put you off. Here are the meanings of the terms you need to know the most.

  • Value space: This is the logical set of values that a data type might have, independent of how it might look in a data stream. For example, the concept of positive integers represents a value space.

  • Lexical representation, lexical space: These deal with how the values look in an XML instance document. For the value space of positive integers, we have the set 1, 2, 3, 4, and so on. The value space consisting of the integer value 1 might have a lexical space of 01 and 001 in addition to 1.

  • Constraining facets: Schema designers can use these attributes on the built-in types when they want to limit the range of allowable values. We'll talk about some of the most commonly used facets.

Note that despite the rich set of types that schema language provides, you will still see people using constructs that don't take advantage of those provided. For example, instead of using the conventional boolean, I have seen a Yes No Indicator used with the values Y and N enumerated. I have also seen date types created to accommodate alphabetic month abbreviations or slash separators, such as 23-Aug-2002 or 8/23/02, respectively. Another example is a derived type for a date range to express concepts such as June 1, 2002, through July 31, 2002. Using the built-in date data type you could easily express this concept as:

<OnSaleRange>
  <BeginDate>2002-06-01</BeginDate>
  <EndDate>2002-07-31</EndDate>
</OnSaleRange>

However, to save themselves a few bytes in instance documents, some schema authors create their own derived types that allow them to say:

<OnSaleRange>2002-06-01 - 2002-07-31</OnSaleRange>

Schema language makes it very easy to do things like this. As I said earlier, there are a thousand different ways to hang yourself.

What is much more common than such practices, however, is restricting the range of values allowed for the built-in data types. There are very good business reasons for doing this. We'll talk next about some of the types of restrictions you're likely to see.

Extending and Restricting Simple Types

Simple types can be either extended or restricted. Again, we don't change the content model of a simple type since simple types always have a simple content model, that is, no child Elements may be defined. We extend the simple type into a complex type by adding one or more Attributes. We restrict a simple type by using one of its constraining facets that reduces the range of allowable values. Extension of a simple type is done by using the schema language xs:extension Element. We'll see it used in the Attribute Declarations subsection below. Restriction is done by using the schema language xs:restriction Element. We'll see examples of this next. Both extension and restriction must specify a "base," that is, the existing data type from which you are deriving your new data type.

Setting a Maximum Length

It is very common to see restrictions on the length of string Elements. To avoid truncating alphanumeric values before inserting them into a database, some programmers create schemas that make sure no one can use a string long enough to be truncated. To restrict our column lengths we can create a String1024Type type derived from string, as follows.

Setting a Maximum Length in SimpleCSV2.xsd
<xs:simpleType name="String1024Type">
  <xs:annotation>
    <xs:documentation>This user-defined type shows how we
        restrict our columns to 1024 characters
    </xs:documentation>
  </xs:annotation>
  <xs:restriction base="xs:string">
    <xs:maxLength value="1024"/>
  </xs:restriction>
</xs:simpleType>

We use the xs:simpleType Element with its name Attribute. The xs:restriction Element identifies the restriction base as the built-in string data type. We use the xs:maxLength Element as the constraining facet, with a value Attribute of 1024.

Our column Elements then are assigned this user-defined type instead of the built-in string type.

<xs:element name="Column01" type="String1024Type" minOccurs="0"/>

You can see this in context in SimpleCSV2.xsd.

I use a common convention of suffixing type names with Type. This is purely for readability. Schema language allows you to name types pretty much anything (well, anything allowable for NMTOKEN as defined in XML 1.0). A type may even have the same name as an Element that uses it. This is allowed because the names have different scope.

Setting Minimum and Maximum Values

Just as people want schemas to prevent them from truncating alphanumeric fields, they also want schemas to set upper limits on numeric fields so they don't cause truncations or overflows. Using our working example, we could restrict the zip code to a set of values as follows:

Setting Maximum and Minimum Values
<xs:simpleType name="zipCodeType">
  <xs:annotation>
    <xs:documentation>Here we define a ZIP Code as an integer
        from 1 through 99999
    </xs:documentation>
  </xs:annotation>
  <xs:restriction base="xs:integer">
    <xs:minInclusive value="1"/>
    <xs:maxInclusive value="99999"/>
  </xs:restriction>
</xs:simpleType>

This type of minimum and maximum value restriction probably makes more sense for the amount due on an invoice, but you get the idea. The next subsection shows a better way to do it.

Patterns for Identifiers

Identifiers are all around us. Social Security numbers, zip codes, and DUNS numbers (unique numbers issued by Dun & Bradstreet and used to identify business entities) are just a few. The range of values of such things is usually way too large and dynamic to practically enumerate in a schema, although business applications usually validate them in their databases. Even though we might not want to enumerate the set of allowable values, we can do a level of validation by specifying a pattern to which the identifier must conform. For example, U.S. Social Security numbers have the general pattern NNN-NN-NNNN, where N represents a digit from 0 through 9. U.S. zip codes (in the five-digit form) have a pattern of NNNNN. The common way to declare an identifier type is to restrict the string data type using the pattern facet. Here's an example for zip codes.

Specifying a Pattern in SimpleCSV2.xsd
<xs:simpleType name="zipCodeType">
  <xs:annotation>
    <xs:documentation>Here we define a ZIP Code as 5-digit
        pattern.
    </xs:documentation>
  </xs:annotation>
  <xs:restriction base="xs:string">
    <xs:pattern value="\d\d\d\d\d"/>
  </xs:restriction>
</xs:simpleType>

Each \d in the value Attribute of the xs:pattern Element indicates a decimal digit. Schema language offers many ways to specify patterns. Patterns are specified as regular expressions, which are sequences of characters that denote sets of strings. The schema language notation for regular expressions is very similar to that used in Perl. It is described in Appendix F, Part 2, of the Schema Recommendation. However, that appendix is somewhat obtuse and doesn't have any examples. If you're going to use patterns frequently and are new to regular expressions, I recommend getting a good introductory book on Perl, learning how Perl does regular expressions, then seeing how schema language differs. There's also a good resource on the Web, listed at the end of this chapter.

Code Lists

Codes aren't used in real life as much as identifiers, but they abound in business applications. There is always some debate about what constitutes a code versus what constitutes an identifier, and more confusion is created when people talk about coded identifiers! However, we can put a stake in the ground at least a bit by talking about country codes, which use short, coded values in place of longer, less constrained values that are harder to validate. For example, we use "US" instead of "United States of America" and "FR" instead of "France." Country codes are especially useful since country names may be spelled differently in different languages. Other very common code lists are for state or province codes and units of measure.

We don't yet have any good applications for codes in our working example. Suppose we wanted to add an Element for gender, with a coded value indicating male, female, or unknown. We could do this as shown below.

Specifying a Code List in SimpleCSV2.xsd
<xs:simpleType name="genderCodeType">
  <xs:annotation>
    <xs:documentation>This type specifies male, female,
        or unknown
    </xs:documentation>
  </xs:annotation>
  <xs:restriction base="xs:string">
    <xs:enumeration value="M"/>
    <xs:enumeration value="F"/>
    <xs:enumeration value="U"/>
  </xs:restriction>
</xs:simpleType>

Note that, as with some types of patterns, when we specify the enumerations we don't have to specify lengths. Note, too, that although we are talking in this section about Elements, all of these simple types we are creating by restriction can also be used for Attributes.

As I write this there are very few (if any) approved, standard schemas for standard code lists that can be reused by anyone in their own schemas just by referencing them. For example, you would think that the folks at the U.S. Post Office might post a schema with the standard codes for states, military addresses, and U.S. territories and possessions. They haven't, at least not yet. People are restating such code lists on their own. This is pretty silly, but it's our only choice for now since there isn't yet a standard way to specify a code list in a schema. (Remember, a thousand different ways to hang yourself!) However, there is hope.

The Universal Business Language Technical Committee of the Organization for the Advancement of Structured Information Systems (the UBL TC in OASIS) has drafted a proposal for a standard way to express code lists in schema language. A task group within the U.S. EDI standards group, ANSI ASC X12, reviewed it shortly before I wrote this and had a favorable assessment. At that time the UBL folks said they planned to submit the work to the appropriate group within ISO. It would be very significant if this proposal is fully fleshed out and gains acceptance because it would lay the foundation for reusable code lists.

Attribute Declarations

As I said earlier, adding Attributes to Elements and understanding what that does to simple versus complex types is one of the more awkward parts of the Schema Recommendation. However, once you get your head around this idiosyncrasy, Attribute declarations are fairly straightforward. Here's a plain-vanilla version in which we just extend our Columns from a simple type to a complex type by adding the ColumnNumber Attribute.

Specifying a ColumnNumber Attribute in SimpleCSV3.xsd
<xs:complexType name="ColumnType">
  <xs:annotation>
    <xs:documentation>Here we add the ColumnNumberAttribute,
        type integer, and optional to the ColumnType.
    </xs:documentation>
  </xs:annotation>
  <xs:simpleContent>
    <xs:extension base="xs:string">
      <xs:attribute name="ColumnNumber" type="xs:integer"
          use="optional"/>
    </xs:extension>
  </xs:simpleContent>
</xs:complexType>

The syntax for an Attribute declaration is very similar to an Element declaration. We still have the name and type Attributes, but the schema Element we use is xs:attribute. Note also that we still have simple content, as indicated by the xs:simpleContent Element, but the declaration is for a xs:complexType.

The relationships between restriction and extension and simple and complex become a bit clearer when we consider the next example. We want to add the ColumnNumber Attribute, but we also want to restrict the length of the column to 1,024 characters. We're extending the type from a simple type to a complex type but restricting the content model. We have to do this in two steps. Although it requires a few more lines in the schema, we can thank the Schema Recommendation authors for implementing the concepts with a bit of consistency. I think it would ultimately be more confusing if we could do both operations in one step. In the following snippet, we first create the String1024Type type by restriction from string, then extend it to the ColumnType complex type by adding the Attribute.

Simple Content Restriction and Extension in SimpleCSV4.xsd
<xs:simpleType name="String1024Type">
  <xs:annotation>
    <xs:documentation>This is the base for our ColumnType type,
        showing restriction, then extension.
    </xs:documentation>
  </xs:annotation>
  <xs:restriction base="xs:string">
    <xs:maxLength value="1024"/>
  </xs:restriction>
</xs:simpleType>
<xs:complexType name="ColumnType">
  <xs:annotation>
    <xs:documentation>Here we add the ColumnNumberAttribute,
        type integer, and optional to the ColumnType, but our
        base is our restricted 1024-byte string instead of the
        built-in string type.
    </xs:documentation>
  </xs:annotation>
  <xs:simpleContent>
    <xs:extension base="String1024Type">
      <xs:attribute name="ColumnNumber" type="xs:integer"
          use="optional"/>
    </xs:extension>
  </xs:simpleContent>
</xs:complexType>

This shows clearly that we create a new simple type by restriction from a built-in data type, then use it to create a new complex type by extension. Again, note that in both this and the previous example the content of our complex type is still simple.

As I noted earlier, there are many more built-in schema language data types, and there is certainly a lot more involved in creating simple content Elements than what I've presented here. However, this discussion should give you all you need to read about 90 percent of what you'll find in schemas for business data. Let's move on now to Elements with complex content.


Complex Content Elements

In this section we discuss Elements that may have child Elements, that is, those with complex content models. Although it may seem counterintuitive, in some ways these are easier to understand than the simple content Elements discussed in the previous section. One of the reasons for this is that while many features for defining simple content Elements and types are commonly used, relatively few of the features for defining complex content Elements are seen in the wild.

Complex types are specified in Part 1: Structures of the Schema Recommendation. It is, of course, the final source and arbiter of any of these concepts. However, Part 1 is so littered with constraints and validation rules that finding the basic definitions is sometimes a daunting task. Part 0: Primer is usually a better place to start.

Types of Content

In speaking of complex content, we mean an Element that may have other Elements as children. As noted earlier in the chapter, all such elements are of a complex type. Schema language provides several content models, but a single one of them, sequence, is used in nearly all cases in business document schemas. The full set of content models is listed below. There are constraints on the content models beyond what I state here, but these definitions should give you the general idea.

  • sequence: As the name indicates, sequence represents a series of Element children in a specific order. This is by far the most commonly used. If you turn back a few pages to SimpleCSV1.xsd you can see complex types defined with sequence for Row and the root document Element SimpleCSV. The sequence content model is specified by using the xs:sequence Element.

  • choice: This type allows only one of the child Elements to appear in an instance document. Very rare, though I've seen it used a few times. The choice content model is specified by using the xs:choice Element.

  • all: This allows all the child Elements to appear either once and only once or not at all, and it allows them to appear in any order. I've never seen this one used. The all content model is specified by using the xs:all Element.

  • group: This type provides a way to reuse a content model you have defined and named somewhere else. We'll talk more about it later in the chapter. I have yet to see group used either, but as organizations mature in their use of schema language I wouldn't be surprised if group begins to show up. The group content model is specified by using the xs:group Element.

  • mixed: An Element with a mixed content model may have character content in addition to child Elements. Although common in publishing, most people who design business document schemas for documents used directly by applications shun it. The mixed content model is specified by using the mixed Attribute of the xs:complexType or xs:complexContent Elements.

Another feature of schema language that can be a bit confusing is that while a complex type may have complex content, the xs:complexContent Element is not always used when defining complex types. Any of the Elements in the above list may be specified as a child of an xs:complexType Element. The xs:complex Content Element is used only when deriving a new complex type from an existing one by extension or restriction.

With a few exceptions, any of the child Elements specified in these content models may themselves have a complex content model and therefore have children. This is the mechanism whereby we describe nested Elements.

As you can see from the descriptions, some of these content models have explicit constraints on the number of times a child Element may appear. Others don't. The next subsection describes how we specify these constraints.

Mandatory, Optional, Limits

Where records (and occasionally fields or elements in an array) are allowed to repeat in a file, we usually need to set upper boundaries on how often they may appear. In addition, many applications such as EDI allow some data to be optional. We express these constraints in schema language through the minOccurs and maxOccurs Attributes. These are set on the child Elements in the content model. Again, if you turn back to SimpleCSV1.xsd, you can see a minOccurs="0" set on the ColumnXX Elements. Here it is again, in a single Element declaration from the sequence.

<xs:element name="Column01" type="xs:string" minOccurs="0"/>

To set a specific limit, we specify the appropriate integer value for the maxOccurs Attribute. An Element with no upper boundary on occurrences is specified with a maxOccurs value of "unbounded", as follows.

<xs:element name="Row" maxOccurs="unbounded">

The default for both minOccurs and maxOccurs is "1". Setting minOccurs to "0" is equivalent to saying that the Element is optional, while setting it to "1" (or greater) is equivalent to saying that it is mandatory.

There are other constraints on maxOccurs depending on the content model. For sequence (the most common) and choice, the maxOccurs value may be any integer value greater than or equal to zero, or the string "unbounded". This means that an Element may be declared so as to allow it to occur any number of times. For the all content model, maxOccurs may only be "0" or "1"; an Element in this content model may appear once at most or not at all.

Creating New Complex Types by Extension

One of the nicer features of schema language is the ability to create a new complex type by extending an existing one. The most frequent usage involves taking an existing sequence of Elements and adding a new one at the end. The example below defines RowType, followed by an extension where we add a new column for birth date. You can find the complete schema in SimpleCSV5.xsd.

Extending a Complex Type by Adding an Element in SimpleCSV5.xsd
<xs:complexType name="RowType">
  <xs:annotation>
    <xs:documentation>Here we give a named type to our Row
        Element, instead of defining it anonymously in-line.
    </xs:documentation>
  </xs:annotation>
  <xs:sequence>
   <xs:element name="Column01" type="ColumnType" minOccurs="0"/>
   <xs:element name="Column02" type="ColumnType" minOccurs="0"/>
   <xs:element name="Column03" type="ColumnType" minOccurs="0"/>
   <xs:element name="Column04" type="ColumnType" minOccurs="0"/>
   <xs:element name="Column05" type="ColumnType" minOccurs="0"/>
   <xs:element name="Column06" type="ColumnType" minOccurs="0"/>
   <xs:element name="Column07" type="ColumnType" minOccurs="0"/>
   <xs:element name="Column08" type="ColumnType" minOccurs="0"/>
   <xs:element name="Column09" type="ColumnType" minOccurs="0"/>
   <xs:element name="Column10" type="ColumnType" minOccurs="0"/>
  </xs:sequence>
</xs:complexType>
<xs:complexType name="RowWithBirthDateType">
  <xs:annotation>
    <xs:documentation>We extend RowType with a column for the
        birth date
    </xs:documentation>
  </xs:annotation>
  <xs:complexContent>
    <xs:extension base="RowType">
     <xs:sequence>
      <xs:element name="Column11" type="xs:date" minOccurs="0"/>
     </xs:sequence>
    </xs:extension>
  </xs:complexContent>
</xs:complexType>

People most commonly use this approach when creating their schemas after performing object-oriented analysis. Deriving complex types by adding Elements is very similar to extending base classes into subclasses with additional properties or methods.

Attribute Declarations

Elements with complex content can have Attributes in the same way that Elements with simple content can. If we wanted to add to our Row Element a RowNumber Attribute with the following format:

<Row RowNumber="2">

we could specify the Attribute in our RowType as shown below.

Specifying an Attribute on an Element with Complex Content in SimpleCSV6.xsd
<xs:complexType name="RowType">
  <xs:annotation>
    <xs:documentation>Here we give a named type to our Row
        Element, instead of defining it anonymously in-line.
        We also have added a RowNumber Attribute.
    </xs:documentation>
  </xs:annotation>
  <xs:sequence>
   <xs:element name="Column01" type="ColumnType" minOccurs="0"/>
   <xs:element name="Column02" type="ColumnType" minOccurs="0"/>
   <xs:element name="Column03" type="ColumnType" minOccurs="0"/>
   <xs:element name="Column04" type="ColumnType" minOccurs="0"/>
   <xs:element name="Column05" type="ColumnType" minOccurs="0"/>
   <xs:element name="Column06" type="ColumnType" minOccurs="0"/>
   <xs:element name="Column07" type="ColumnType" minOccurs="0"/>
   <xs:element name="Column08" type="ColumnType" minOccurs="0"/>
   <xs:element name="Column09" type="ColumnType" minOccurs="0"/>
   <xs:element name="Column10" type="ColumnType" minOccurs="0"/>
  </xs:sequence>
  <xs:attribute name="RowNumber" type="xs:integer"
    use="optional"/>
</xs:complexType>

You can examine the full schema for this in SimpleCSV6.xsd.


Understanding Namespaces

The Schema Recommendation relies heavily on namespaces. Although by now almost everyone involved with XML has at least heard of namespaces, a lot of confusion and misunderstanding still exist about some of the key concepts. A brief discussion is in order.

What Is a Namespace?

To begin, every XML instance document has a namespace, whether it is explicitly named or not. So, in the current environment there is no escaping them. You can't run, you can't hide.

A namespace is nothing more than a set of names. Namespaces are themselves identified by a URI, which I'll discuss next. But first let's drill in the concept with an analogy.

Consider a youth soccer league family picnic. The set of everyone attending the picnic forms our universe, or the default, unnamed namespace. Consider two people named Bob who are attending the picnic with their sons and daughters (Figure 4.1).

Figure 4.1. Two Bobs at the Picnic

Suppose that the coach calls out for Bob to help cook the hot dogs. Unless the coach is looking directly at one or the other of them, how are they to know which one should answer? Maybe they're both perfectly happy sitting in their lawn chairs, nursing brewskis, and they both ignore him. The coach gets annoyed because he also needs to get someone else started setting up some more tables. So, he calls out, "Bob Smith!" He has just used a namespace. Figure 4.2 shows the two Bobs "disambiguated" by putting them in the namespaces of the Smith and Jones surnames.

Figure 4.2. The Smith and Jones Namespaces

In a nutshell, that's all there is to namespaces. Now, there are certainly complexities in how they are used and how they are resolved, but that's the basic concept. For most things you'll have to deal with, you need to grasp only this essential concept and a few more details.

URIs, URNs, and URLs

The Schema Recommendation specifies that namespaces have a name expressed as a URI. We're all familiar with URLs from the Web, but fewer people have heard of URIs or URNs or really know what they mean. These terms are all defined in standards maintained by the Internet Engineering Task Force (IETF). IETF standards originate as Requests for Comment (RFC), so the standards that define these terms are referenced by their RFC numbers. Pointers to them are listed in the Resources section at the end of the chapter.

  • Uniform Resource Identifier (URI): RFC 2396 defines a URI as "a compact string of characters for identifying an abstract or physical resource." This is the general class from which all other types of identifiers such as URLs and URNs are derived. Any valid URL or URN is also a URI. A URI can be classified as a name, a location, or both. URIs must conform to a specific syntax specified in the RFC.

  • Uniform Resource Locator (URL): These are the familiar http://www.mumble . . . addresses we're all familiar with from the World Wide Web. URLs are specifically a URI that references a location. In addition to the familiar http protocol reference, URLs can also have protocol types such as file and ftp. The general syntax for a URL is the protocol type followed by a string identifying a resource that can be accessed via that protocol. URLs were originally specified in RFC 1738, which was later merged into RFC 2396.

  • Uniform Resource Name (URN): This is a name for a resource that is not necessarily associated with the location of that resource. URNs use the concept of namespaces. While similar at a high level to the use of namespaces in XML documents, the details are different, and URNs have an entirely different syntax. URN syntax names have a "urn" prefix, followed by a namespace ID, followed by the specific name within that name space. For example, RFC 2141, which deals with namespace syntax, could have a URN of urn:ietf:RFC:2141, as opposed to its URL of http://www.ietf.org/rfc2141.txt. URNs are defined in RFC 2141. As described in RFC 2611, the IETF intended that URNs be created and assigned through a managed process, with the namespace IDs being registered through the Internet Assigned Numbers Authority (IANA).

There are two key points to understand: (1) a namespace is a set of names, and (2) the name of that namespace is just a name. Even though a specific namespace name might have the form of a URL, it doesn't necessarily mean that there is anything at that URL location. This can be kind of confusing because the current convention is to use URL names as namespace names and post the schemas that correspond to them at those URLs. However, this is just a convention and not a requirement. The use of URNs will certainly help to resolve this confusion, but they have not yet come into fashion. My hunch is that people are shying away from them because they're not yet ready to step up to the "managed process" implications of using them. Most of the namespace names currently in use are in URL form. This includes the namespaces used by the W3C.

Namespace Qualification in Instance Documents

The thing that gives most people real heartburn about namespaces is their use in instance documents. This issue has sent some of my standards committee colleagues into apoplectic fits when the topic of namespaces comes up in discussion. Since this chapter is intended to inform you about what you are likely to encounter and is not intended to be a "best practices" guide, I'm going to avoid taking sides in this issue. I will, however, try to point out what some of the controversy and consternation is about.

The most common and simplest way to use namespaces in an instance document is to not explicitly use them at all. The instance document, in essence, has an unnamed default namespace, and all the Elements and Attributes in the document come from that namespace. Very few people have much trouble with this approach.

Another way to use namespaces is to give them a prefix. We've already seen this in the schema declaration Attributes of our instance document.

<SimpleCSV xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:noNamespaceSchemaLocation="SimpleCSV1.xsd">

The xmlns attribute assigns the xsi prefix to the namespace http://www.w3.org/2001/XMLSchema-instance. The next Attribute uses that prefix to say that the noNamespaceSchemaLocation Attribute is defined in that namespace. Also note that this is the only place in the instance document that you see a namespace or a prefix. This minimalist strategy for instance documents is the next simplest strategy (after the choice of not using named namespaces at all).

Namespace qualification in instance documents is specified by two Attributes in the schema's root xs:schema Element. elementFormDefault and attribute Form Default, respectively, say whether or not your instance document's Elements and Attributes need to have namespaces specified. There is a general preference for setting these to "unqualified," though I have seen schemas that set them to "qualified."

At the other end of the spectrum we can have instance documents in which every Element and Attribute is qualified, sometimes from different namespaces. In this form, elementFormDefault has a value of "qualified." Here's an example.

Qualified Namespace Usage in SimpleCSVNamespaces.xml
<?xml version="1.0" encoding="UTF-8"?>
<bb:SimpleCSV
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xmlns:bb="http://www.babelblaster.org/SimpleCSV"
    xmlns:bbc="http://www.babelblaster.org/common"
    xmlns="http://www.usps.org"
    xsi:schemaLocation="
      http://www.babelblaster.org/SimpleCSV SimpleCSVN.xsd
      http://www.babelblaster.org/common common.xsd
      http://www.usps.gov http://www.usps.gov/ZipCodes.xsd
      ">
  <bb:Row>
    <bbc:Column01>Jones</bbc:Column01>
    <bbc:Column02>Mary</bbc:Column02>
    <bbc:Column03>312 Renner Road</bbc:Column03>
    <bbc:Column04>Apartment C</bbc:Column04>
    <bbc:Column05>Richardson</bbc:Column05>
    <bbc:Column06>TX</bbc:Column06>
    <ZipCode xmlns="http://www.usps.org">75080</ZipCode>
    <bbc:Column08>USA</bbc:Column08>
    <bbc:Column09>972-996-1051</bbc:Column09>
  </bb:Row>
  <bb:Row>
    <bbc:Column01>Smith</bbc:Column01>
    <bbc:Column02>Sue</bbc:Column02>
    <bbc:Column03>Highway 118</bbc:Column03>
    <bbc:Column05>Terlingua</bbc:Column05>
    <bbc:Column06>TX</bbc:Column06>
    <ZipCode xmlns="http://www.usps.gov">79852</ZipCode>
    <bbc:Column10>desertrat@aol.com</bbc:Column10>
  </bb:Row>
</bb:SimpleCSV>

Note: This document won't validate since there is no U.S. Post Office namespace that defines a ZipCode Element (at least not yet, anyway).

There are tradeoffs to the two extremes. If only the default, unnamed namespace is used, the instance document is limited to a single set of names. This can make it harder in some senses to support standard vocabularies. At the other extreme, liberal use of different namespaces makes everything from DOM names (and even some method calls) in Java and C++ to XPath expressions in XSLT more cumbersome.

There are ways to use multiple namespaces in different schema documents while using just one namespace in instance documents that are valid with the schemas. We'll talk about that later in the chapter when discussing structuring schemas using several different schema files.

The W3C XML Schema-Related Namespaces

The W3C defines two specific namespaces associated with the XML Schema language.

  • http://www.w3.org/2001/XMLSchema: This namespace defines all the constructs specified by the Schema Recommendation, including all the Element and Attribute names we use in schemas. By convention this namespace is referred to by the prefix xs, though you may occasionally see xsd.

  • http://www.w3.org/2001/XMLSchema-instance: This namespace defines Attributes that can be used in XML instance documents. We have seen two so far, noNamespaceSchemaLocation and schemaLocation. By convention this namespace is referred to by the prefix xsi.

The former is used in every schema document written in the W3C XML Schema language, so it is very important. I suppose it could be used in instance documents other than schemas, but I've never seen it done. The latter is used in every instance document that declares itself to conform to a schema written in the W3C XML Schema language.


Structuring Schemas

Of the thousand ways that schema language gives you to hang yourself, the options for structuring your schemas account for at least several hundred. Here are some of the major contributors to the variety.

Global Types and Local Elements versus Global Elements

In addition to declaring your own types and creating Elements from your types, you may declare Elements and reuse them by way of a reference. An example of the syntax for doing this appears below.

<xs:element ref="Column01" minOccurs="0"/>

Column01 has been defined somewhere else in the schema, based on a type. This style is uncommon because the referenced names must have a scope that is global to the whole namespace used in the schema. As you can imagine, this requires all names to be unique. Ensuring uniqueness can be tedious in large schemas and can lead to some awkward names. It is, however, one way to ensure that names are globally unique and that things that are named the same actually are the same.

The more common practice is to define simple and complex types, which by their nature (with the exception noted in the next subsection) have names that are global in scope. These are then used to define Element names whose scope is only local to the complex type in which they are defined.

You will, however, probably see both approaches.

Named Types and Anonymous Types

Although I didn't point it out at the time, we have already seen examples of named and anonymous types. A named type has a name Attribute in the xs:simpleType or xs:complexType Element where it is defined. In contrast, an anonymous type has no name and is defined in-line, immediately following the Element where it is used. In such cases the Element has no type Attribute. SimpleCSV1.xsd, shown in the beginning of the chapter, uses only anonymous types. SimpleCSV6.xsd uses named types for the Row and ColumnXX Elements.

The scope of anonymous types is local to the Element to which they apply. As such, they are not reusable. This flies in the face of good software engineering practices for a number of obvious reasons. Anonymous types lead to a lot of duplicate, repetitive code. They are prone to inconsistency when what should be an identical Element gets defined differently in different contexts. They can be a maintenance nightmare when, for example, you need to change the zip code everywhere it is used from the standard five digits to the five plus four format. Also, anonymous types can lead to some very large schema files. For a contrast between the approaches, look at SimpleCSV7.xsd (which uses an anonymous type only for the root Element of the document, a common practice) and Simple CSV8.xsd (which uses all but one anonymous type; we had to have an extension base of the String1024Type). The former schema document is about 52 lines while the latter is about 118 lines. Using a lot of anonymously defined complex types nested within each other can also lead to some schemas that are very hard to read unless you are looking at them with a schema authoring tool that provides a graphical view.

On the other hand, if you use a type only once, it may not make much sense to give it a global name. Schemas that use a lot of named types can also make it hard to get the full picture of an Element's content. I guess complex is just complex however you structure it. There is, however, one instance in which you must use a named type even if you use it only once. You must name a type, rather than declare it anonymously, if you want to use it as a base for restriction or extension.

Modularity: The xs:include and xs:import Elements

As with source code in programs, once something gets to a certain size it starts to get a bit unwieldy in terms of understandability and management. Schema language provides two mechanisms for breaking a single logical schema into several different physical schemas. Both are Elements in the W3C XML Schema namespace.

  • xs:include: This Element directs a schema processor to read the file specified by the schemaLocation Attribute and consider it as part of the namespace of the including schema. The included schema must either have no target name space declared or have declared one that matches that of the including schema. The net effect is as if there weren't two separate schemas but only one.

  • xs:import: This Element works in a fashion similar to the xs:include Element. It also has a schemaLocation Attribute that a schema processor uses to find the schema file. However, there is a subtle difference (perhaps not so subtle when you understand the impact). You don't import a file of declarations; you import a namespace. Since you are importing a namespace, the imported schema must have either a different target namespace than the importing schema or no target namespace. The most common usage is for the two schemas to have different target namespaces. In this situation the xs:import Element has a namespace Attribute that identifies the URI of the namespace being imported, and the namespace must also be declared via the xmlns Attribute in the root xs:schema Element.

An Example of Importing Type Libraries

As I write this there are very few comprehensive libraries of common business documents in schema language. Hence, there are few examples we can draw from actual usage. However, we can review some of the approaches being considered by organizations such as ANSI ASC X12 and the OASIS UBL TC.

This example incorporates several approaches widely considered to be best practices. It is based on what Roger Costello [2003] calls the Venetian Blind Design. A similar example has been considered by groups within X12 and the UBL TC. At the time that I write this, neither group has officially adopted the features of this approach. However, it does reflect the general direction in which they are currently headed. I expect that although specific details will probably vary, the major features of this approach will become fairly common as more comprehensive libraries become available.

This example deals with a simple purchase order listing a buyer, a seller, and one or more line items. The following files comprise the example:

  • purchaseOrder.xml: a sample instance document

  • purchaseOrder.xsd: the schema for the instance document

  • procurement.xsd: a type library containing types that are specific to procurement (would probably be maintained by a specific subcommittee in a larger standards body)

  • common.xsd: a type library containing types used throughout the organization's schemas

All these files are available on the book's Web site, but I show them here to highlight specific concepts in context.

Let's begin by looking at the instance document.

purchaseOrder.xml
<?xml version="1.0" encoding="UTF-8"?>
<PO:PurchaseOrder
xmlns:PO="urn:x12:names:schemas:005010:Procurement:PurchaseOrder"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation=
  "urn:x12:names:schemas:005010:Procurement:PurchaseOrder
   purchaseOrder.xsd">
  <Buyer>
    <PartyName>
      <FirstName>Barney</FirstName>
      <LastName>Rubble</LastName>
    </PartyName>
    <Address>
      <Street>12 Rock Lane</Street>
      <City>Gravelly</City>
      <State>AR</State>
      <ZipCode>72838</ZipCode>
    </Address>
    <DUNSNumber>123456789</DUNSNumber>
  </Buyer>
  <Seller>
    <PartyName>
      <FirstName>Fred</FirstName>
      <LastName>Flintstone</LastName>
    </PartyName>
    <Address>
      <Street>452 Interstate 30</Street>
      <City>Rockwall</City>
      <State>TX</State>
      <ZipCode>75032</ZipCode>
    </Address>
    <DUNSNumber>987654321</DUNSNumber>
  </Seller>
  <OrderedItems>
    <ProductID>RH-932</ProductID>
    <ProductName>Rock Hammer</ProductName>
    <LineItemIdentifier>1</LineItemIdentifier>
    <Quantity>2</Quantity>
    <UnitOfMeasure>EA</UnitOfMeasure>
    <RequestedDeliveryDate>2003-03-04</RequestedDeliveryDate>
  </OrderedItems>
  <OrderedItems>
    <ProductID>WG-004</ProductID>
    <ProductName>Heavy Duty Work Gloves</ProductName>
    <LineItemIdentifier>2</LineItemIdentifier>
    <Quantity>4</Quantity>
    <UnitOfMeasure>PR</UnitOfMeasure>
    <RequestedDeliveryDate>2003-03-04</RequestedDeliveryDate>
  </OrderedItems>
</PO:PurchaseOrder>

Notice that the only Element with a namespace prefix is the root Element. We'll see why when we look at purchaseOrder.xsd. Note also the use of a URN for the document's namespace. Among other things, the URN specifies a version within X12's namespace.

Finally, I need to point out a syntax difference between how schema locations are specified in instance documents versus schema documents. In instance documents there is a single xsi:schemaLocation Attribute on the root Element that lists pairs of URIs and the locations where the schema may be found. We'll see when we look at the schema document that it is done a bit differently there.

In this instance document we reference only a single namespace. In regard to the location, again since none of these examples are hosted at X12, the location reference assumes that the schema is located in the same place as the instance document. The overall structure of this document is specified in purchaseOrder.xsd.

purchaseOrder.xsd
<?xml version="1.0" encoding="UTF-8"?>
<xs:schema
  targetNamespace="urn:x12:names:schemas:005010:Procurement:PurchaseOrder"
  xmlns:PO="urn:x12:names:schemas:005010:Procurement:PurchaseOrder"
  xmlns:pro="urn:x12:names:schemas:005010:Procurement:common"
  xmlns:xs="http://www.w3.org/2001/XMLSchema"
  elementFormDefault="unqualified"
  attributeFormDefault="unqualified">
  <xs:import
    namespace="urn:x12:names:schemas:005010:Procurement:common"
    schemaLocation="procurement.xsd"/>
  <xs:annotation>
  <xs:documentation>This is the generic Purchase Order schema.
      We declare the root Element with an anonymous complex type
      whose Elements are defined from library types.
  </xs:documentation>
  <xs:annotation>
  <xs:element name="PurchaseOrder">
    <xs:complexType>
      <xs:sequence>
        <xs:element name="Buyer" type="pro:BuyerType"/>
        <xs:element name="Seller" type="pro:SellerType"/>
        <xs:element name="OrderedItems" type="pro:OrderedItem"
            maxOccurs="unbounded"/>
      </xs:sequence>
    </xs:complexType>
  </xs:element>
</xs:schema>

We declare a target namespace specifically for a purchase order document, so the root Element of our instance document must explicitly be in that namespace. However, because we have set elementFormDefault to "unqualified," no Elements below the root Element in the instance document tree need to have namespace prefixes. The other main thing to note is that this is a fairly simple skeleton schema focused on showing the overall structure of the document. It makes heavy use of reusable, common types. It declares the root Element with an anonymous complex type that has a sequence. The types of each of the Elements in this sequence are defined in the Procurement type library in procurement.xsd. Because we are importing these types from the namespace defined in procurement.xsd, we qualify them with a namespace prefix. The elementForm Default setting of "unqualified" does not relieve us of this requirement since a schema processor would otherwise not know where to look for the named types.

Now, to the difference in how the schema locations are specified. Notice that in the xs:import element we import a single namespace. In this case the schema Location Attribute has a single value, which is the location of the namespace being imported by that Element. The two different usages of the schemaLocation Attribute are legal because in the instance document the schemaLocation Attribute was from the xsi namespace, while in the schema document it is from the xs namespace.

There is one final thing to note. The instance document schema is built up of components that are ultimately based on types in several libraries. However, we only see it importing the namespace of the layer immediately below it. This is because of the way libraries are layered. The layer immediately below this schema for a specific instance document is that of the type library for the responsible subcommittee.

Let's now examine procurement.xsd, the type library that purchaseOrder.xsd imports.

procurement.xsd
<?xml version="1.0" encoding="UTF-8"?>
<xs:schema
  targetNamespace="urn:x12:names:schemas:005010:Procurement:common"
  xmlns:Pro="urn:x12:names:schemas:005010:Procurement:common"
  xmlns:X12Com="urn:x12:names:schemas:005010:common"
  xmlns:xs="http://www.w3.org/2001/XMLSchema"
  elementFormDefault="unqualified"
  attributeFormDefault="unqualified">
  <xs:import namespace="urn:x12:names:schemas:005010:common"
    schemaLocation="common.xsd"/>
  <xs:annotation>
    <xs:documentation>This schema declares items unique to the
        Procurement subcommittee. This is a proposal that
        illustrates how schemas might be structured and should
        not be considered as suggesting any specific simple or
        complex types. It has the same status as task group
        working papers.
    </xs:documentation>
  </xs:annotation>
  <xs:annotation>
    <xs:documentation>This section declares extensions to common
        complex types.
    </xs:documentation>
  </xs:annotation>
  <xs:complexType name="PartyType">
    <xs:complexContent>
      <xs:extension base="X12Com:PartyType">
        <xs:sequence>
          <xs:element name="DUNSNumber" type="xs:string"/>
        </xs:sequence>
      </xs:extension>
    </xs:complexContent>
  </xs:complexType>
  <xs:annotation>
    <xs:documentation>This section declares modules common to the
        Procurement SC.
    </xs:documentation>
  </xs:annotation>
  <xs:complexType name="BuyerType">
    <xs:complexContent>
      <xs:extension base="Pro:PartyType"/>
    </xs:complexContent>
  </xs:complexType>
  <xs:complexType name="SellerType">
    <xs:complexContent>
      <xs:extension base="Pro:PartyType"/>
    </xs:complexContent>
  </xs:complexType>
  <xs:complexType name="OrderedItem">
    <xs:complexContent>
      <xs:extension base="X12Com:LineItemType">
        <xs:sequence>
          <xs:element name="RequestedDeliveryDate"
              type="xs:date"/>
        </xs:sequence>
      </xs:extension>
    </xs:complexContent>
  </xs:complexType>
</xs:schema>

This schema declares a library of all the types used in instance document schemas produced by this particular subcommittee. The target namespace is Procurement:common within the X12 namespace structure, and it imports the common X12 namespace with which we'll finish. Types imported from that namespace are qualified with the X12Com namespace prefix. In this subcommittee library we basically extend the complex types declared in the common type library, shown below.

common.xsd
<?xml version="1.0" encoding="UTF-8"?>
<xs:schema
  targetNamespace="urn:x12:names:schemas:005010:common"
  xmlns:common="urn:x12:names:schemas:005010:common"
  xmlns:xs="http://www.w3.org/2001/XMLSchema"
  elementFormDefault="unqualified"
  attributeFormDefault="unqualified">
  <xs:annotation>
    <xs:documentation>This schema contains declarations common
        to all X12 subcommittees. This is a proposal that
        illustrates how schemas might be structured and should
        not be considered as suggesting any specific simple or
        complex types. It has the same status as task group
        working papers.
    </xs:documentation>
  </xs:annotation>
  <xs:annotation>
    <xs:documentation>This section declares base lower-level
        types.
    </xs:documentation>
  </xs:annotation>
  <xs:simpleType name="NameType">
    <xs:restriction base="xs:string">
      <xs:maxLength value="50"/>
    </xs:restriction>
  </xs:simpleType>
  <xs:annotation>
    <xs:documentation>This section declares common simple types.
    </xs:documentation>
  </xs:annotation>
  <xs:simpleType name="FirstNameType">
    <xs:restriction base="common:NameType"/>
  </xs:simpleType>
  <xs:simpleType name="LastNameType">
    <xs:restriction base="common:NameType"/>
  </xs:simpleType>
  <xs:annotation>
    <xs:documentation>This section declares common complex types
        composed of simple types.
    </xs:documentation>
  </xs:annotation>
  <xs:complexType name="PersonNameType">
    <xs:sequence>
      <xs:element name="FirstName" type="common:FirstNameType"/>
      <xs:element name="LastName" type="common:LastNameType"/>
    </xs:sequence>
  </xs:complexType>
  <xs:complexType name="ResidenceAddressType">
    <xs:sequence>
      <xs:element name="Street" type="xs:string"/>
      <xs:element name="City" type="xs:string"/>
      <xs:element name="State" type="xs:string"/>
      <xs:element name="ZipCode" type="xs:string"/>
    </xs:sequence>
  </xs:complexType>
  <xs:complexType name="ProductType">
    <xs:sequence>
      <xs:element name="ProductID" type="xs:string"/>
      <xs:element name="ProductName" type="xs:string"/>
    </xs:sequence>
  </xs:complexType>
  <xs:annotation>
    <xs:documentation>This section declares common complex types
        composed of lower-level complex types.
    </xs:documentation>
  </xs:annotation>
  <xs:complexType name="PartyType">
    <xs:sequence>
      <xs:element name="PartyName"
          type="common:PersonNameType"/>
      <xs:element name="Address"
          type="common:ResidenceAddressType"/>
    </xs:sequence>
  </xs:complexType>
  <xs:annotation>
    <xs:documentation>This section declares common complex types
        derived by extension from other complex types.
    </xs:documentation>
  </xs:annotation>
  <xs:complexType name="LineItemType">
    <xs:complexContent>
      <xs:extension base="common:ProductType">
        <xs:sequence>
          <xs:element name="LineItemIdentifier"
              type="xs:integer"/>
          <xs:element name="Quantity" type="xs:decimal"/>
          <xs:element name="UnitOfMeasure" type="xs:string"/>
        </xs:sequence>
      </xs:extension>
    </xs:complexContent>
  </xs:complexType>
</xs:schema>

This main type library has types shared in common by all subcommittees. If X12 ever did anything like this it would, of course, be much larger. It would probably be split into many different files that were combined with the main schema file using the xs:import or xs:include Elements. Most of the types declared here are either simple types derived by restriction from built-in schema language data types or complex types built up from simple types. Since it is at the lowest layer, it declares only its own namespace and the standard W3C XML Schema namespace.

In summary, this example shows a layered architecture for defining type libraries. Each layer uses types only from the layer immediately below it. The example makes heavy use of named types and uses no global Element declarations. While using several different namespaces, it still allows instance documents to have minimal namespace qualification on their Elements and Attributes. It is very modular, extensible, and maintainable. Again, groups such as ANSI ASC X12 and the OASIS UBL TC will probably end up adopting an approach that looks different than the one used in this example. However, due to the advantages of this demonstrated approach I think that over time such groups will head in a similar direction.


Less Commonly Used W3C XML Schema Language Features

Up to this point I've focused on the schema language features I've most often seen used for EAI and electronic commerce applications (and therefore those you will most likely encounter). Other features are less frequently used, at least at present. However, it may help you to at least be familiar with them in case you do encounter them.

Wildcards

Schema language supports wildcards with xs:any, xs:anyAttribute, and xs:anyType. There are some good technical reasons for having wildcards in certain types of applications. But for most schemas that describe documents intended for production or consumption by business applications, and especially for those that define data exchanged between organizations, wildcards tend to be a bit problematic. They can somewhat defeat the purpose of performing schema validation against a standard schema. I am, however, aware of some standards bodies discussing wildcard use in specific extension Elements. These extension Elements would allow people to exchange nonstandard data in otherwise standard documents and would not require any changes to the standard schemas.

Default and Fixed Attributes

Ever had the experience of looking at the output of an application and not being able to figure out where certain data came from? Schema language carries over from DTDs the ability to define default Attributes for instance documents. Even if the Attribute doesn't appear in an instance document, a schema-aware XML processor will hand it over to your application when it processes the document. This feature may be of some use in scenarios where you are integrating in-house applications and have complete control over the environment. There are certainly differences of opinion, but many people don't find default values very friendly when exchanging data between organizations. We generally like to know explicitly what is being told us, with nothing implied or defaulted.

Whitespace Suppression

Space characters, tabs, carriage returns, and line feeds are considered to be whitespace. Schema language has a pattern facet that lets you specify that whitespace should be suppressed when retrieving the content of a string Element or Attribute. By default all whitespace is preserved. However, using this feature can cause leading and trailing spaces to be stripped before passing the string to an application. Just as it can be unfriendly to add data with default Attributes, if your application thinks that leading or trailing spaces are significant, it can be unfriendly to have them removed. This is not a problem for most people since they tend to trim leading and trailing spaces when creating instance documents. However, it is something to be aware of. If you have an application or trading partner that is sending you whitespace you don't want, you may be able to use this feature to your advantage. To employ it, you must modify your schema to create a new simple type by restriction, using string as the base type, and specify the whitespace facet with the value of "collapse." You then set the Elements where you want whitespace stripped to be of that type rather than string. Whitespace suppression does not affect schema validation. It affects only the values returned to you by an XML processor.

Derivation of Complex Types by Restriction

We talked earlier about deriving simple types from built-in types by restriction and about deriving complex types by extension. We didn't talk about deriving complex types by restriction. There are several reasons for this. One reason is the general mechanism by which this restriction is performed. For example, if you want to remove Elements from a sequence content model, you have to restate the sequence, listing only those Elements you want to use. This is a bit more awkward than just listing those you want to remove. In addition, most people designing hierarchies of complex types tend to follow an object-oriented approach. This generally involves defining base classes with a few common properties, then extending them into subclasses by adding properties. However, some groups are taking a "kitchen sink" approach, similar to what has always been done in EDI, and defining in the base type everything that anyone ever might want to use. They then restrict down to specific usages by removing things they don't want. However, some groups that started out this way are moving back toward a conventional derivation model. Another persuasive reason for minimizing the use of restriction to derive complex types is that, even two years after the XML Schema Recommendation was published, most APIs still don't do derivation by restriction properly. This is no wonder since the Schema Recommendation is rather obtuse in this area.

Redefinition

Simple content models can be extended by means of a union, similar to a union in C or a Redefine in COBOL. I haven't seen many applications for it. A similar-sounding but fundamentally different feature is the schema xs:redefine Element. The xs:redefine Element can be used to modify the definitions of an external schema before they are used within the redefining schema. Some people may find reasons for doing this, but the extension and restriction mechanisms work just as well in most cases.

Named Attribute Groups

Sets of Attributes that are repeatedly used together may be associated with each other by creating a named Attribute group. This serves the same function as a macro in a programming language or some uses of ENTITY in DTDs. I've not seen named Attribute groups used at all, but they could be very useful when the same Attributes are used repeatedly.

Named Model Groups

These do for Elements what named Attribute groups do for Attributes. However, I have yet to see very many grounds for using them. In most cases, things are grouped together for a semantic reason. Putting them together as children under a parent Element explicitly shows their association when used in an instance document. Named model groups let you reuse groups of Elements in schema declarations, but they have no visibility in instance documents. You can't differentiate something that is part of a named model group from a sibling that isn't. When implemented in an organized fashion, deriving complex types by extension generally tends to meet most people's needs.

Abstract Types and Substitution Groups

They best way to describe these is to use an analogy from object-oriented programming. You can define a base class, then define subclasses, all with the same named method. Then you can process objects of the subclass by referring to them as if they were all members of the base class. This type of polymorphism allows you to call a named method but have different behaviors because each of the derived classes has implemented the named method in a fashion tailored to the derived class. Abstract types are akin to the base class, and the members of a substitution group are akin to the derived classes. I've been told that this approach is used in the OASIS Security Assertion Markup Language (SAML), but I've seen it used in only one schema I've actually reviewed. The only reason I think it was used there is because, in my humble opinion, the schema designers were trying to address too many types of business processes and data in a single schema. However, they may have used it because they were generating Java classes directly from schemas (more on this in Chapter 12). Most schemas are so tightly focused in their problem domain and data that there is no need for this type of flexibility. However, as I write this X12 is considering using this feature for a very limited, specific type of constraint on a lower-level type. There may be a need to say that at least one member of a group of things is required but that there could be more than one. An example is requiring at least a name or a frequent flyer number on an airline reservation. Abstract types and substitution groups are currently in the running for expressing that type of constraint.

Keys and Uniqueness

Schema language offers four different mechanisms for relating Elements to each other. Several offer a means by which schema processors can enforce something similar to the referential integrity or unique key constraints that relational database systems offer. The four mechanisms are (1) ID with IDREF, (2) the xs:unique schema language Element, (3) xs:key and xs:keyref, and (4) XLink and XPointer. Any one of these would require at least a paragraph and an example to explain. That doesn't quite seem appropriate for a discussion about little-used features. To date, most people are content to let their applications enforce referential integrity and uniqueness and to relate different Elements in instance documents using the appropriate business data rather than schema language constructs.

Nillable Elements

Schema language offers a feature that allows you to identify an Element as being nillable, that is, capable of representing a null value. This is the same as saying that a column in a table in a relational database can be null. With this feature you can create an instance document with an Element that is empty but that has an attribute of xsi:nill with a boolean value of true. Strictly speaking, as far as the schema language syntax goes this is not the same as saying that the Element is empty or that it is absent. However, what your application chooses to do with such Elements is probably up to you. I've not seen this feature used yet. However, I will be surprised if I don't find some clever monkey using it before I retire from working with XML.

So, those are some features you probably won't see used very much any time soon. With some warning you may be able to avoid hanging yourself too badly when someone springs one on you.

The last topic of nillable Elements leads naturally to the problem in the next section.


Is There Data or Not?

It is customary to specify that certain data must be present in a document before it can successfully be imported into a business application. Many people coming from an EDI background are very accustomed to having their EDI systems verify that essential information like purchase order numbers and item IDs are present in a message. They look to schemas and schema validating parsers to help them enforce these types of constraints. However, things in this area don't quite work as many people expect. I can best show this with an example.

Let's consider a simple invoice. Our customer requires us to send them back their original purchase order number when we send them an invoice electronically. They have set their PurchaseOrderElement to be mandatory, using a min Occurs Attribute with a value of "1" in the xs:complexType sequence where it lives. It is a string field since their purchase order numbers have a mix of alphabetic and numeric characters. Which of the following PurchaseOrderNumber Elements do you think will pass their invoice schema's validation constraints, and which do you think will fail?

  1. <PurchaseOrderNumber>AZ-39AAY</PurchaseOrderNumber>

  2. <PurchaseOrderNumber> </PurchaseOrderNumber>

  3. <PurchaseOrderNumber></PurchaseOrderNumber>

  4. <PurchaseOrderNumber/>

If you understood the schema constraint as saying that a purchase order number must be present, you would think that cases 2,3, and 4 would fail. If you were from an EDI background, you would probably think that cases 3 and 4 would fail and that case 2 might not even be legal XML. However, you would be wrong. If you said that none of the cases would fail and all of them would pass, you win the bonus prize!

Especially in the EDI world, saying that a data element is present is synonymous with saying that there is data in the data element. This is not the case with XML, at least not for string Elements. The empty Elements in cases 3 and 4 satisfy the minOccurs of one constraint. Case 2 may not be so obvious. The string data type allows any Unicode character to be present in the Element, including spaces. So, a single space is valid. In EDI a single space in a data element is not valid.

There are two ways to get around this problem. The easiest is to use the token data type instead of string. Token does not allow leading or trailing spaces or any other whitespace beyond a single space between other strings of characters. However, if you think someone might do something like use two or more spaces between a first and last name, you have to fall back to the other method. Create a new simple type, derived by restriction from string, with a pattern facet of at least one character that isn't a space. Then, use that wherever you want to have at least one nonspace character. Here's an example of how this might be done.

Example of a Type Requiring at Least One Nonwhitespace Character in nonSpaceString:xsd
<xs:simpleType name="nonSpaceString">
  <xs:annotation>
    <xs:documentation>
      This is a type, derived from string by restriction, that
      requires at least one nonwhitespace character to be present.
    </xs:documentation>
  </xs:annotation>
  <xs:restriction base="xs:string">
    <xs:pattern value=".*\S.*"/>
  </xs:restriction>
</xs:simpleType>

There are many ways to write the regular expressions used in pattern facets. In this particular example I describe a string that starts and ends with zero or more characters other than the new line or carriage return. This is indicated by the .* notation. The \S notation, bracketed in the middle, indicates a single character that is not a whitespace character. (The \s notation is a "multicharacter escape" that indicates a character from the class of whitespace characters. This class, or set, is composed of the space character, tab, new line, and carriage return. Capitalizing it to \S indicates any character that is not in that class.)

I don't know whether people just aren't aware of this problem, are concerned about the processing overhead of pattern validation, or just don't care. But I have yet to see any schemas that enforce this type of constraint.

It is important to note that this problem occurs only with the string data type. The other commonly used built-in data types don't have this problem (at least not with most parser APIs!).

So, with any luck you should now be able to read and understand most schemas you will encounter. The next chapter will show programmers how to code to use schemas. In doing that we'll add validation capability to the two utilities we developed in Chapters 2 and 3.


Reference

Costello, Roger. 2003. "Global versus Local" in "XML Schemas: Best Practices." Available online at http://www.xfront.com/GlobalVersusLocal.pdf, accessed March 31, 2003.


Resources

Requests for Comments, by the Internet Engineering Task Force of the Internet Society. Base URL http://www.ietf.org/rfc/.

Uniform Resource Identifiers (URI), Generic Syntax, RFC 2396, August 1998, rfc2396.txt

Uniform Resource Locators (URL), RFC 1738, December 1994, rfc1738.txt

URN Namespace Definition Mechanisms, RFC 2611, June 1999, rfc2611.txt

URN Syntax, RFC 2141, May 1997, rfc2141.txt

XML Schema Recommendation, by World Wide Web Consortium. Base URL http://www.w3.org/TR/.

Part 0: Primer, xmlschema-0.html

Part 1: Structures, xmlschema-1.html

Part 2: Datatypes, xmlschema-2.html

XML Schemas: Best Practice Homepage, at Roger Costello's xFront site. Base URL http://www.xfront.org/BestPracticesHomepage.html. This very good site addresses several issues having to do with the W3C XML Schema language. Particularly in regard to this chapter, I recommend the piece on regular expressions as used in the W3C XML Schema language.


Chapter 5. Validating against Schemas

Like conscience, too, love needs no authentication or validation by any authority outside itself.

—A. J. Toynbee

This chapter describes how to validate instance documents against schemas from Java and C++. The two utilities from Chapters 2 and 3 are enhanced to include schema validation as an option.


Requirements

For better or for worse, people often want more from XML instance documents than they do from conscience or love: They frequently want validation. Do you blame them?

The most immediate problem is that we want to be able to validate input instance documents against a schema so that the application that processes them doesn't abnormally terminate or produce incorrect output. Another concern is that whoever is using our XML output would like to know it is valid before they try to process it. This is essentially the same problem, just from a different perspective. But it's someone else who will feel the pain if documents aren't valid, not us. In this short chapter we'll modify the XML to CSV and CSV to XML Converter utilities so that validation is an option.

A related issue is whether or not to validate in the first place. When is it sufficient, or preferable, to just ensure that a document is well formed as opposed to valid in regard to a schema or DTD? This is a rather complex issue because it gets into broader considerations of application design. I'll discuss it in Chapter 12.


Running the Utilities

Refer to Chapters 2 and 3 for the full instructions for running the XML to CSV and CSV to XML Converters from the command line. This section describes the additional parameters and restrictions.

Options:

For both CSVToXMLBasic and XMLToCSVBasic, we add a validate option.

  • -v (Validate): Validate the input instance document before processing, or validate the created document before writing it to disk. Input documents are validated against the schema specified in the root Element noNamespaceSchemaLocation attribute. The default schema for output documents is CSVFile.xsd.

For CSVToXMLBasic, we add two qualifiers.

  • -sSchemaFileName: Override the default schema file name of CSVFile.xsd. SchemaFileName represents the local name of the schema file without the directory specification.

  • -o (Omit): Omit the schema declaration in the output document.

Options o and v are incompatible. If option o is specified, option s has no effect.

Restrictions:

The following restrictions are imposed when using schema files other than CSVFile.xsd.

  • For validation of output instance documents, the schemas must be located in the directory pointed to by the BBSCHEMAS environment variable.

  • Schemas must be in W3C XML Schema language and must be valid according to that language. DTDs are not supported.

  • The schema must have elementFormDefault set to "unqualified."

  • Namespace prefixes may not be used on any Elements in instance documents.

  • Element names may not be changed.

  • The Element content of the root CSVFile and Row Elements may not change other than to reduce the number of ColumnXX Elements in the sequence.

  • In the sequence for the Row Element, the value of the minOccurs Attribute of the ColumnXX Elements may be either "0" or "1". A max Occurs Attribute value of "0" may cause validation errors, while a value greater than "1" may have unpredictable results.

  • The types of the ColumnXX Elements may be changed as long as the types used have simple content models.


Design

The specific method calls for validating instance documents against schemas comprise an area beyond the scope of the current W3C Recommendations. So, we won't find many design issues in common between our Java and C++ implementations. However, there are a few.

The first issue is to decide where to put the schema files. We have a few options.

  • Host them on the Internet at Addison-Wesley's Web site: Probably okay, but that site isn't really set up for this kind of thing (nor is it meant to be). In addition, there are some problems with putting things on the Internet. You have to have a network connection up and running, which might not always be the case for someone using a dial-up service. Files on the Internet require a different base URL than local files, so we would have to accommodate both styles. Some APIs handle these details transparently for us, while others may not.

  • Host them on my site or someplace such as SourceForge: Okay, and probably easier to manage, but with this choice the files are still on the Internet.

  • Host them locally on your computer: All around a better idea. You don't have to have a network connection up and running, you can easily place your own schemas there, and it makes coding a bit easier.

Even with locally hosted schema files, you can still read and validate an input instance document against a schema hosted somewhere else on the Internet. I'm just not going out of my way to code things so that you can create an instance document that way. It might work with the current versions of JAXP and MSXML, but it might not. Again, you can transform your document with XSLT if you need to use a schema on the Internet.

That decided, in order to accommodate as many runtime environments as possible we'll use a local environment variable to specify the base URL of our schemas: BBSCHEMAS (for Babel Blaster schemas). For most shells used on UNIX systems this is done by assigning a value to the variable and optionally exporting it, for example:

BBSCHEMAS=/babelblaster/schemas; export BBSCHEMAS

For WIN32 systems, you can use a set statement from the command line or set it through the appropriate control panel function. For example, on my Windows NT system, from System in the Control Panel, on the Environment tab, I set a User Variable for BBSCHEMAS with the following file URL:

file:///E:/Book/Schemas/

The BBSCHEMAS Environment Variable and Java

OK, put me on the side of those who think that deprecating System.getenv was a pretty goofy idea. Yeah, I know, some operating system platforms don't have anything that corresponds to an environment variable. However, removing the functionality makes life a bit more difficult for those of us who work on platforms that do support it.

The bottom line is that we still need a parameter specifying the schema file path that is configurable by the user at runtime. So, for the Java environment we'll use System.getProperty and specify the -D option when we invoke the Java virtual machine. For example, when I run the modified utilities in this chapter I run:

java -DBBSCHEMAS=file:///E:/book/schemas/ classname, etc.

Or, in a script on WIN32 I run:

java -DBBSCHEMAS=%BBSCHEMAS% classname, etc.

Figure out what works best for you, and thank you very much, Sun.

The other design aspect shared by both implementations is the need to write the schema file reference to the output instance document. As we saw in the previous chapter, to do this we add two Attributes to the root Element of the document. This is done in the main routine of CSVToXMLBasic, immediately after creating the root Element and appending it to the Document Node. The first of the two Attributes declares the XMLSchema-instance namespace. To set this we call the root Element's setAttribute method, passing the Attribute name of xmlns:xsi and the URI value of the xsi namespace. For the second Attribute we set the value of an Attribute that lives in that namespace, noNamespace SchemaLocation. Because the Attribute is from a different namespace we need to use the DOM Element's setAttributeNS method, which takes three arguments. The first is the namespace URI of the Attribute, the full URI value of the xsi namespace. The second is the qualified name of the Attribute, that is, the prefix and the local name. The third is the Attribute value. Here's the pseudocode.

Logic to Add Schema-related Attributes
Root Element <- Call Document's createElement method, with
    tagName of SimpleCSV
Document <-  Call Document's appendChild to append Root to
    Document
Root Element <- Call Element's setAttribute method, with
    arguments of xmlns:xsi and
    http://www.w3.org/2001/XMLSchema-instance
Root Element <- Call Element's setAttributeNS method, with
    arguments of http://www.w3.org/2001/XMLSchema-instance,
    xsi:noNamespaceSchemaLocation, and
    the URL of our schema file

CAUTION MSXML Doesn't Support setAttributeNS

This is one area where MSXML lags behind Xerces in standards support. It's kind of a "gotcha" if you don't watch out for it. See the discussion below in the C++ Implementation section.

The other thing we need to do that is common to both implementations is to handle the new command line options. This is pretty plain-vanilla Java and C++, so I won't talk about it further here.

As a final word, since we're starting to build a more capable system and getting to more complex options, I'm adopting a different naming scheme for the root Element. For this chapter and these two more capable utilities I'm going to use CSVFile instead of SimpleCSV.


Java Implementation

In validation we again encounter one of the areas where it takes us many more lines of code to do something with Java and JAXP than it does with C++ and MSXML. This is partially because the JAXP and Xerces implementations give us many more configuration options for our parser. The other reason is that the current versions of the JAXP and Xerces implementations don't give us a handy way to validate a DOM Document in memory before we write it disk. We have to do a bit of Java magic and create our own method.

Let's start with input validation because we have to build on these techniques when doing output validation.

Input Validation in XMLToCSVBasic.java

JAXP performs validation in the DocumentBuilder's parse method. There is no special "validate" method as such. We tell JAXP that we want validation by specifying the type of DocumentBuilder the DocumentBuilderFactory should make. This is kind of like knowing that you want to build a barbecue out of titanium instead of stainless steel and calling the toolmaker to have him make tools to work with the former rather than the latter.

So, what options do we want to set on the DocumentBuilderFactory? Aside from the obvious one of telling it we want to validate, there are a few others. Here's the relevant code added to XMLToCSVBasic.java.

Validation Code in XMLToCSVBasic.java
//  Set up DOM XML environment
DocumentBuilderFactory Factory =
  DocumentBuilderFactory.newInstance();

//  Set the factory to create a Document Builder that
//    is:
//  Namespace aware - necessary for schema validation
Factory.setNamespaceAware(true);
//  Ignores whitespace on Element only nodes
Factory.setIgnoringElementContentWhitespace(true);
//  Ignores comments
Factory.setIgnoringComments(true);
//  Set the schema language - these attributes are
//  specific to Xerces2
Factory.setAttribute(JAXPConstants.JAXP_SCHEMA_LANGUAGE,
          JAXPConstants.W3C_XML_SCHEMA);
 //  Validating, if requested
if (boValidate)
{
  Factory.setValidating(true);
}

//  Create the new document builder
DocumentBuilder Builder = Factory.newDocumentBuilder();

We set the options on the Factory after creating a new instance. We first set the Factory to be aware of namespaces. To validate an instance document against a schema the DocumentBuilder must at least be able to handle the XMLSchema-instance namespace where the noNamespaceSchemaLocation Attribute lives. The next two properties don't directly affect validation, but we're interested in them anyway. As we mentioned in Chapter 2, Xerces returns "ignorable whitespace" as Text Nodes unless you specifically tell it not to. We're telling it not to. We're also telling it to ignore comments since we don't care about them. The setAttribute method on the DocumentBuilderFactory is the next method that directly deals with validation. JAXP provides this mechanism for the purpose of setting parameters that govern the behavior of the underlying parser. In this case we're telling Xerces to use the W3C XML Schema language as its schema language when doing validation. These constants are defined in the org.apache.xerces.jaxp.JAXP Constants interface, and I have chosen to use them from that source. However, literal values are also accepted. JAXP_SCHEMA_LANGUAGE has a value of:

http://java.sun.com/xml/jaxp/properties/schemaLanguage

W3C_XML_SCHEMA has a value of:

http://www.w3.org/2001/XMLSchema

If we don't set this attribute of the Factory, we get validation against a DTD.

The last option we set configures the Factory to build a validating parser via the setValidating method. Note that if we configured a validating parser and an instance document does not specify a schema through either the no Namespace Schema Location or the schemaLocation Attributes, the parser throws an exception.

So, even though there is a bit of configuration that we have to do, input validation in Java with JAXP and Xerces is not very involved. Validation errors are handled in the same fashion as general parsing errors from instance documents that aren't well formed. They throw SAX exceptions that we catch and report using the SAXExceptionHandler.

Output Validation in CSVToXMLBasic.java

Input validation was fairly simple once we performed the setup. On the other hand, validating output is not so easy with the current versions of JAXP and Xerces. This may have changed by the time you read this, so it will be worthwhile to check the JAXP and Xerces Javadocs to see what is available. However, right now we have only a few options.

We could do DOM Level 3 validation. The Xerces dom3 packages have some preliminary validation methods based on the DOM Level 3 drafts. However, the current Level 3 validation draft doesn't quite offer us exactly what we're looking for, which is a single method that validates a complete DOM Document. In addition, the Xerces dom3 packages come with warnings about instability. We don't want to use anything unstable, do we?

We could use the internal Xerces implementation classes and try to do validation the same way that Xerces does. There are such overwhelming disadvantages to this approach that I won't even discuss the benefits. Even if we could comfortably rely on these classes and methods staying stable between major releases, we are still tying ourselves more than I would like to one particular API implementation. The other obvious disadvantage is that we would need to figure out how to use these classes. Certainly doable, but not necessarily something I want to tackle within the scope of this book.

So, where does that leave us? The only mechanism that JAXP offers for validation is to validate while parsing. It may not be very elegant or efficient, but that is exactly what we'll do. We'll serialize our DOM Document to an in-memory I/O stream, then parse it from that stream to check for validation failures. Serializing the document an additional time (even though it is in memory rather than to disk) and parsing it is not exactly the most efficient approach. But, given our priorities and the tools we have to work with, it is the most appropriate approach. Remember, keeping things simple is more important to us than performance.

To implement this approach we create a DOMValidator class. The source code is in DOMValidator.java. This class has three private properties set up in the constructor.

  • ValOutput: a ByteArrayOutputStream to which we serialize our input DOM Document

  • ValSerializer: an XMLSerializer (such as was used in Chapter 3), configured to use ValOutput

  • Builder: a DocumentBuilder configured to validate against W3C schemas

The validation work is done in the validate method.

The validate Method from DOMValidator.java
public void validate(Document docInput)
    throws SAXException, java.io.IOException
  {
    //  Serialize the document to the byte array output stream
    ValSerializer.serialize(docInput);
    ValOutput.flush();

    //  Set up the input byte stream for reading it back
    ByteArrayInputStream ValInput =
      new ByteArrayInputStream(ValOutput.toByteArray());

    //  Validate the document while parsing it from the input
    //  byte stream
    Builder.parse(ValInput);
  }

The code is really pretty simple. We serialize the DOM Document to the byte array output stream. Then we convert that to a byte array that passed as input via the constructor to a byte array input stream. We finally parse the serialized document by reading it from the byte array input stream. Because we are dealing with input and output streams we can throw an IOException, and because we are parsing we can throw a SAXException.

We call this method from our main method as follows.

if (boValidate)
{
  Validator = new DOMValidator();
  Validator.validate(docOutput);
}

If JAXP and Xerces eventually come up with something that provides the same functionality, we can just replace these calls and throw away DOMValidator. Until then, though kind of a kludge, this approach does the job well enough.


C++ Implementation

MSXML makes the validation task a bit easier in some ways and a bit harder in others. Validation is easier once you've figured out how MSXML does things, but figuring out how it does things can be kind of hard.

One of the benefits of MSXML is that it has a separate validate method in addition to the load method. This is very handy, but it is also a bit confusing because the default behavior of the load method is to validate while parsing (a feature of MSXML that is not very well documented). This behavior can be changed, as we'll see with input validation. That aside, separating loading and validation is handy because when XML is our output, once we have built the DOM Document in memory we can call the validate method to validate the DOM Document against its schema before we save it.

There is, however, a bit of setup involved in validation and there are some "gotchas" lurking out there to bite unwary passersby. Let's look at input validation first because it is simpler.

Input Validation in XMLToCSVBasic.cpp

As I noted, the default behavior of the load method is to validate while parsing. So, if we haven't specified validation on the command line, we want to disable validation while loading. We do that by setting the validateOnParse property of the DOM Document to false.

spDocInput->validateOnParse = VARIANT_FALSE;

But what should we do if we want to validate? Do we perform validation while parsing via the load method, or do we just load without validation and then call validate? The quick answer is "both." We want to validate while parsing and call the validate method. Why both calls? The main reason is that the validate method does not report as much error information in the ParseError object as does the load method. So, we want to use load, with the default behavior of validating while parsing, as our primary validation tool. However, the load method by itself is not sufficient. In my testing I found that load was unable to report an error in cases where either there was no schema specified or there was a problem with the schema. As a result, for the most reliable validation we want to call both methods. Since we don't need to do anything to change the default behavior of load, we only need to add the call to validate. Here's the code to add to XMLToCSVBasic.cpp.

Validation Code in XMLToCSVBasic.cpp
//  Validate the input document
if (boValidate)
{
  spParseError = spDocInput->validate();
  if( spParseError->errorCode != S_OK)
  {
    cerr << "Validation Error" << endl;
    displayParseError(spParseError);
    throw cValidationError;
  }
}

The only real trick here is that while the load operation returns just an HRESULT (you need to explicitly call the Document's getParseError), the validate method returns an IXMLDOMParseError. We pass that to the displayParseError routine and then throw an exception to exit the try block. Validation errors typically show only reason text and error codes, not any other useful information.

Output Validation in CSVToXMLBasic.cpp

There were a few idiosyncrasies with validating XML as input, but validating output can be downright tricky. I expected output validation to be a piece of cake, but it took me a couple of hours to figure out what was going on and to make it work correctly. Going through the exercise may help you understand a bit more about how MSXML does things.

The first time I coded the routine I just added a block of code to CSVToXMLBasic.cpp that was almost identical to the input validation snippet shown above in XMLToCSVBasic.cpp. It didn't work. I got a validation failure message indicating that "the root element had no associated DTD/schema."

MSXML stores schemas internally and makes them available through the IXMLDOMSchemaCollection/XMLSchemaCache object. I was already familiar with the "Validating an XML Document against an XML Schema Using C++" example in the MSXML online documentation. The example goes through a somewhat convoluted process of creating a schema collection, associating it with the instance document to be validated, creating and loading the schema document, and finally adding it to the schema collection before calling the instance document's validate method. I guessed that MSXML couldn't identify the schema merely from the root Element's noNamespaceSchemaLocation Attribute. So, I added similar code to my main routine. The document still didn't validate, but I was getting closer to the source of the problem. This time the validation failure message indicated that the noNamespaceSchemaLocation Attribute "is not defined in the DTD/Schema." So, it didn't recognize that Attribute as being part of my instance document's default target namespace. However, it didn't recognize the Attribute as being from the xsi namespace either!

This led me to review how I had added that Attribute. It occurred to me that it might be significant that MSXML does not offer the DOM Level 2 setAttribute NS method on the Element interface. This is one of the rare cases in which MSXML doesn't support the standards as well as Xerces. Since it wasn't offered, I had fallen back to adding the noNamespaceSchemaLocation Attribute using the setAttribute method. However, on further investigation the MSXML documentation advised me that a namespace qualified Attribute could not be added using that method. Instead the Attribute must be added using the Document interface's createNode method. So, I modified the code to add the Attribute, as you see in the next snippet.

Adding the noNamespaceSchemaLocation Attribute
//  Next set the schema location.
//  MSXML requires that namespace qualified Attributes
//  be created as Nodes, then set.
variant_t varType((short)NODE_ATTRIBUTE);
spSchemaLocationAttribute = spDocOutput->createNode(
  varType,"xsi:noNamespaceSchemaLocation",
  "http://www.w3.org/2001/XMLSchema-instance");
spEleRoot->setAttributeNode(spSchemaLocationAttribute);
// We can finally set it now.
spEleRoot->setAttribute("xsi:noNamespaceSchemaLocation",
  cSchemaFileFullPath);

This finally worked: The instance document validated before I called the save method. However, it made me wonder whether or not all the schema cache stuff was really necessary. After all, we didn't need it on input validation. So, I commented out the code that created the schema collection, read the schema document, and so on, then just cut straight to the validate method. It worked. It seems that since MSXML didn't recognize the noNamespaceSchemaLocation Attribute as being in the xsi namespace, it didn't use the value of that Attribute to load the schema document. Once MSXML understood the Attribute properly, MSXML used it. I hacked out the schema cache code, and what you see in CSVToXMLBasic.cpp is almost identical to what you see in XMLToCSVBasic.cpp.

In some cases you may actually need to load and use an MSXML schema cache. Be aware that they exist, and if you have trouble with validation, try using one. However, the code works fine without them as long as you're doing pretty basic stuff such as I show in these utilities. I'm generally not a fan of writing code I don't have to write, so you won't see me using schema caches.


Examples of Input Validation Failures

JAXP with Xerces and MSXML report validation errors just a bit differently, but they take essentially the same approach. Let's look at a few examples from validating XML instance documents as input. Unless otherwise noted, all instance documents use the CSVFile.xsd schema. I have omitted the Java stack trace.

Here's what we see when there are no Row Elements.

JAXP with Xerces
[View full width]
org.xml.sax.SAXException:
Error during parsing
Text = cvc-complex-type.2.4.b: The content of element 'CSVFile' is not complete. It must
  match '("":Row){1-UNBOUNDED} '.
URL = file:e:/book/chapters/ch05/csvfilenorows.xml
Entity = null
Line = 3
Column = 11
MSXML
Parsing Error
Reason :
Element cannot be empty according to the DTD/Schema.

Source Text :
</CSVFile>
File URL :
file:///e:/book/chapters/ch05/csvfilenorows.xml
Position in file = 175
Line           = 3
Position in line = 3

Here's what we see when we have an unexpected element, Column999, as a child of the Row Element.

JAXP with Xerces
org.xml.sax.SAXException:
Error during parsing
Text = cvc-complex-type.2.4.a: Invalid content starting with element 'Column999'.
The content must match (((((((((((((((((((((((((((((((((
((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((
("":Column01){0-1},("":Column02){0-1}),("":Column03){0-1}),("":Column04){0-1}),
("":Column05){0-1}),("":Column06){0-1}),("":Column07){0-1}),("":Column08){0-1}),
("":Column09){0-1}),("":Column10){0-1}),("":Column11){0-1}),("":Column12){0-1}),
("":Column13){0-1}),("":Column14){0-1}),("":Column15){0-1}),("":Column16){0-1}),
("":Column17){0-1}),("":Column18){0-1}),("":Column19){0-1}),("":Column20){0-1}),
("":Column21){0-1}),("":Column22){0-1}),("":Column23){0-1}),("":Column24){0-1}),
("":Column25){0-1}),("":Column26){0-1}),("":Column27){0-1}),("":Column28){0-1}),
("":Column29){0-1}),("":Column30){0-1}),("":Column31){0-1}),("":Column32){0-1}),
("":Column33){0-1}),("":Column34){0-1}),("":Column35){0-1}),("":Column36){0-1}),
("":Column37){0-1}),("":Column38){0-1}),("":Column39){0-1}),("":Column40){0-1}),
("":Column41){0-1}),("":Column42){0-1}),("":Column43){0-1}),("":Column44){0-1}),
("":Column45){0-1}),("":Column46){0-1}),("":Column47){0-1}),("":Column48){0-1}),
("":Column49){0-1}),("":Column50){0-1}),("":Column51){0-1}),("":Column52){0-1}),
("":Column53){0-1}),("":Column54){0-1}),("":Column55){0-1}),("":Column56){0-1}),
("":Column57){0-1}),("":Column58){0-1}),("":Column59){0-1}),("":Column60){0-1}),
("":Column61){0-1}),("":Column62){0-1}),("":Column63){0-1}),("":Column64){0-1}),
("":Column65){0-1}),("":Column66){0-1}),("":Column67){0-1}),("":Column68){0-1}),
("":Column69){0-1}),("":Column70){0-1}),("":Column71){0-1}),("":Column72){0-1}),
("":Column73){0-1}),("":Column74){0-1}),("":Column75){0-1}),("":Column76){0-1}),
("":Column77){0-1}),("":Column78){0-1}),("":Column79){0-1}),("":Column80){0-1}),
("":Column81){0-1}),("":Column82){0-1}),("":Column83){0-1}),("":Column84){0-1}),
("":Column85){0-1}),("":Column86){0-1}),("":Column87){0-1}),("":Column88){0-1}),
("":Column89){0-1}),("":Column90){0-1}),("":Column91){0-1}),("":Column92){0-1}),
("":Column93){0-1}),("":Column94){0-1}),("":Column95){0-1}),("":Column96){0-1}),
("":Column97){0-1}),("":Column98){0-1}),("":Column99){0-1})'.
URL = file:e:/book/chapters/ch05/csvfilecolumn999.xml
Entity = null
Line = 12
Column = 16
MSXML
Parsing Error

Reason :
Element content is invalid according to the DTD/Schema.
Expecting: Column09, Column10, Column11, Column12, Column13, Column14, Column15,
Column16, Column17, Column18, Column19, Column20, Column21, Column22, Column23,
Column24, Column25, Column26, Column27, Column28, Column29, Column30, Column31,
Column32, Column33, Column34, Column35, Column36, Column37, Column38, Column39,
Column40, Column41, Column42, Column43, Column44, Column45, Column46, Column47,
Column48, Column49, Column50, Column51, Column52, Column53, Column54, Column55,
Column56, Column57, Column58, Column59, Column60, Column61, Column62, Column63,
Column64, Column65, Column66, Column67, Column68, Column69, Column70, Column71,
Column72, Column73, Column74, Column75, Column76, Column77, Column78, Column79,
Column80, Column81, Column82, Column83, Column84, Column85, Column86, Column87,
Column88, Column89, Column90, Column91, Column92, Column93, Column94, Column95,
Column96, Column97, Column98, Column99.

Source Text :
  <Column999>972-996-1051</Column999>
File URL :
file:///e:/book/chapters/ch05/csvfilecolumn999.xml
Position in file = 468
Line           = 12
Position in line = 16

Here's what we see when Column11 is a date type, and the date is not in the correct format.

JAXP with Xerces
org.xml.sax.SAXException:
Error during parsing
Text = cvc-type.3.1.3: The value '15-May-2002' of element 'Column11' is not valid.
URL = file:e:/book/chapters/ch05/csvfiledate.xml
Entity = null
Line = 13
Column = 37
MSXML
Parsing Error

Reason :
The element: 'Column11'  has an invalid value according to its data type.

Source Text :
  <Column11>15-May-2002</Column11>
File URL :
file:///e:/book/chapters/ch05/csvfiledate.xml
Position in file = 523
Line           = 13
Position in line = 28

Here's what we see when we specify validation on the command line, but the instance document doesn't have a reference to a schema.

JAXP with Xerces
org.xml.sax.SAXException:
Error during parsing
Text = cvc-elt.1: Cannot find the declaration of element 'SimpleCSV'.
URL = file:E:/Book/JavaCode/CH05/XMLToCSV/SimpleCSV.XML
Entity = null
Line = 2
Column = 12
MSXML
Validation Error

Reason :
Validate failed because the root element had no associated DTD/schema.

Position in file = 0
Line           = 0
Position in line = 0

Finally, here's what we see when one of our xsd:element declarations in the schema is invalid.

JAXP with Xerces
[View full width]
org.xml.sax.SAXException:
Error during parsing
Text = Error:  sequence content must be zero or more of element, group, choice, sequence
  or any.  Saw "{1} ".
URL = file:///E:/Book/Schemas/CSVFileBad.xsd
Entity = null
Line = 34
Column = 72
MSXML
Validation Error

Reason :
Validate failed because the root element had no associated DTD/schema.

Position in file = 0
Line           = 0
Position in line = 0

Note that in this final example JAXP reported the URL of the schema file and not the instance document. With MSXML we got a validation error reported instead of a parsing error. My guess is that MSXML hit the parsing error on the schema file a bit farther down the stack. However, since we rely on MSXML to do a lot of the work in the background for us and we didn't explicitly set up a schema cache and load the schema file, MSXML probably lost the specific information on the schema file as MSXML unwound the stack.

Remember our discussion about MSXML validating during parsing? In the first three examples above we saw that the failure started with our program text "Parsing Error", while the last two started with "Validation Error". This indicates that, even though all the errors in these examples were technically validation errors rather than parsing errors, they were actually encountered in the call to the load method and handled as parsing errors rather than being encountered in the call to the validate method. Note that we also saw the position in the file, the line number, and the position in the line. If we had loaded without validation and then just called the validate method, we would have gotten a validation error and not received this level of detail. Note also that the errors in the last two examples were reported as validation errors rather than as parsing errors. MSXML didn't find these schema problems while loading!

These examples again reveal an interesting contrast between JAXP with Xerces and MSXML. We have coded both implementations of our utility to do the same things. They validate only if we tell them to. They both report errors if we requested validation and the instance document doesn't reference a valid schema. However, because of the differences in the APIs' behaviors we have to take somewhat different approaches. JAXP won't validate unless we tell it to, and if it does validate it will throw an exception if the instance document doesn't declare a schema. MSXML validates on parsing by default, but it doesn't report a parsing error if no schema reference was found. We have to make an additional call to the validate method to determine that. There's still a lot of inconsistency in the XML world. I hope that DOM Level 3 clears up some of this, but I'm not holding my breath.

Validation is indeed a useful tool. We'll see that judicious use of schema validation can save us from writing some validation code in the utilities we'll be developing in later chapters. Through the discussions in the previous chapter and continued discussions throughout the rest of the book, you should begin to get some ideas about how you can use validation to your benefit. You may also begin to see where it might get in your way!


Resources

DOM Reference, Microsoft XML Core Services (MSXML) 4.0, by Microsoft Corporation. Online documentation included as part of the MSXML SDK. Available online at http://msdn.microsoft.com/xml.

Java API for XML Processing Specification 1.2, by Sun Microsystems. Online Javadoc documentation, included as part of the Java XML Pack Spring 2002. Available online at http://java.sun.com/xml.

Xerces2 API Javadoc, by Apache Software Foundation. Online Javadoc documentation, included as part of the Xerces2 SDK. Available online at http://xml.apache.org.


Chapter 6. Refining the Design

It affords me no satisfaction to commence to spring an arch before I have got a solid foundation.

—Henry David Thoreau

This chapter lays the design foundation for the more advanced utilities developed in the next three chapters. High-level design issues are discussed and base classes are created. It is primarily oriented toward developers, though users of the utilities may find useful background on how to describe the grammars of the legacy formats and the file description documents that encode those grammars.


Why Refine the Design?

So, what place does this chapter have in the book and in the overall scheme of things? Why am I bothering to refine the design? Well, I'm trying to achieve several overall goals with this book. One of the goals is to provide people with some working, simple utilities to perform several different types of common data format conversion tasks. It will save us time and effort if we lay out a design with some specific parameters and focus strongly on reusability.

Another goal is to design these utilities so that you can easily modify them if you need to add or change the functionality. So, we also want to focus on extensibility and maintainability. The utilities we developed up to this point were primarily focused on simple functionality and demonstrating the use of basic DOM and API features. We need to shift to a broader focus to meet the other needs.

Finally, another important goal of the book is to assist developers in building XML support into their applications. While the utilities may not be directly related to that task, several of the design issues discussed here are. The utilities present a good case study for examining several of the techniques for and issues around creating and processing XML.


Making XML the Common Format

In the first chapter we set central roles in our overall architecture for XML as the "common denominator" format and logical conversions via XSLT (as opposed to the syntax conversions that the conversion utilities perform). However, it's time we expanded on those concepts and got a bit more specific. Not only do we want XML to be the common format but also wherever possible we want the data content to be compliant with native schema language data types. This means, for example, that a date in our common XML format is expressed, compliant with ISO 8601 and the schema date data type, as 2002-09-06 instead of 06-Sep-2002 or 9/6/02. A dollar amount that might be expressed in a COBOL extract file or in X12 EDI as a number with two implied decimal places is represented with an explicit decimal in XML, compliant with the built-in decimal data type.

We may not always be able to automatically use the XML data types as our common format. For example, if your application has a field for a local time and another for a time zone, there's no easy way to automatically convert those to the ISO 8601 format of the Coordinated Universal Time (GMT or Zulu) followed by an offset. However, we can save a lot of code in XSLT stylesheets if we convert data types where it is appropriate.

Why do we care about data representation at this level of detail? The main reason is that these built-in schema language data types are the data representations that will most likely be used natively by business applications using XML, whether you or your trading partners are employing those applications. Importantly, as we see standards emerge for common business documents such as purchase orders and invoices, these standards are predominantly using these built-in types instead of creating their own. If our utilities can put the data into these representations for us, then we save having to continually write the same code into our XSLT stylesheets.

The term "canonical XML" has often been given to XML used in this fashion. However, there are other definitions as well. See the sidebar for more details. Given the confusion about just what canonical XML is and the way some people seem to be misusing the term, I'm not going to use it except when I talk about it in the W3C sense (which will be rarely in this book).

What Is Canonical XML?

You know what XML is by now. But what does it mean for XML to be canonical? You have to crack a big dictionary to find an appropriate alternate definition of canonical. We're not looking for something having to do with church canon law, although there is an authoritative flavor about the term. The last alternate definition in my old Webster's [1971] defines canonical as "relating to various of the simplest and most significant forms or schemata to which general equations, statements, or expressions may be reduced without loss of generality." It also compares this to "normal form," which we should be familiar with from relational database design. My dictionary goes on to define "normal form" as, from logic, "a canonical or standard fundamental form of a statement to which others can be reduced." So, using this generic definition we get the idea that canonical XML might involve being in a common, general format to which all other data formats can be converted. Since we're using XML as the common format to and from which all data is converted, our use fits with this general definition. Many people seem to be using the term this way.

However, the W3C also has a Recommendation on Canonical XML [W3C 2000] that assigns a much more specialized meaning to the term. (There are other related Recommendations such as one on Exclusive XML Canonicalization, which deals with aspects of digital signatures in XML documents. But the one I'm discussing here is the first and primary Recommendation.) This Recommendation deals largely with determining whether or not two XML documents are logically equivalent by defining a canonical form to which both can be converted. For example, two instance documents may be in compliance with XML 1.0 and convey the same data, but they may be slightly different. For example, one might use CDATA to convey an ampersand and the other might use a predefined entity for the same purpose. In addition, for the same Element, the Attributes in one document might be in a different order than the Attributes of the same Element in the other document. "Canonical" as defined in this Recommendation tends to agree more strictly with the definition of "canonical" that is used in the study of logic.

I don't know about you, but in most matters XML I'll defer to the W3C.


Analyzing the Legacy Non-XML Grammars

As I noted when developing the CSV to XML utility, as complicated as XML-based grammars can be, the grammars of other formats aren't necessarily dumb and simple. We can get into trouble and ask for processing problems if we don't analyze grammars correctly and process them with the appropriate algorithms. While I'm not going to go heavily into language analysis and parsing algorithms, I am going to borrow a few techniques from compiler construction and the formal study of languages. If you've studied computer science you'll probably already be familiar with most of these concepts. However, if you haven't or if it has been a few years, this discussion will help build a foundation.

Let's start by reviewing and clarifying some basic terminology. The main terms we're concerned about are "language" and "grammar." We'll use these terms as they are conventionally used in the formal study of languages in computer science. In this sense language is defined as "a set of strings involving symbols from some alphabet" [Martin 1991]. This very broad definition allows us to consider not only English as a language but also a set of seemingly random strings of ones and zeroes as a language. A grammar is a set of rules that describes a language. For example, various rules describe proper English grammar, such as "A complete sentence must have a noun and a verb." The languages we are concerned with have similar grammar rules. If a string of symbols conforms to the rules of the grammar, it is said to be "in the language."

With these definitions in hand, let's make sure that we understand the scope of the problem. We'll start by using XML itself as an example. XML is by itself a fairly simple language. Its grammar is defined in the XML 1.0 Recommendation. This standard specifies Unicode as the language's alphabet, and it covers such things as how to use special characters (such as angle brackets and ampersands) to specify tags and entities. XML's main use comes from its extensibility, that is, the ability to use it to define other markup languages. In essence, a new markup language is not defined by, for example, ANSI ASC X12, the Open Applications Group, RosettaNet, or Commerce One, but by every individual document specification produced by such groups. Each DTD or schema defines a specific language. In the strictest terms, OASIS's Universal Business Language is not a language, but its Order document is. XML 1.0 defines the general characteristics of these languages, and as such it defines a class of languages. The XML 1.0 DTD or the W3C Schema language is used to describe each of these specific languages in very precise terms.

We face a similar situation in developing utilities that handle specific non-XML formats. Just as languages based on XML 1.0 correspond to a class of grammars, each of the formats we want to support corresponds to a separate class of grammars. For example, the X12 EDI syntax, that is, its basic grammar, is defined in the X12.6 standard. However, each X12 Transaction Set standard, such as the 850 Purchase Order, through its grammar defines a specific language based on the X12.6 grammar. Our challenge in developing generalized utilities is that we don't want solely to support specific grammars such as a UN/EDIFACT ORDER; instead we want to support classes of grammars such as UN/EDIFACT (based on ISO 9735).

In supporting classes of grammars we need to consider the overall characteristics of the grammars. Where the class of grammar is defined by a standard, as is the case with ANSI X12 and UN/EDIFACT EDI, most of the work is done for us already. However, where the class of grammar is not defined formally, as is the case with CSV or flat files, we'll have to abstract and generalize the essential features in order to come up with a definition of the grammar. In addition, even the EDI grammars will need some "tweaking" to be easily processed in our XML world.

All of the grammars with which we are concerned fall into a special class of grammars called "context-free grammars." Various notations can be used to describe such grammars, but the one most commonly used is Backus-Naur form (BNF). There are various flavors of BNF. X12.6 defines its own and uses it to describe the X12.6 grammar. The W3C also has something called Extended Backus-Naur Form (EBNF). EBNF is defined in the XML 1.0 Recommendation. Since we're working in an XML world we'll use EBNF as the base for our notation. I say "base" because in describing classes of grammars rather than specific grammars even EBNF has some limitations.

In BNF an individual grammar rule is known as a production. A set of productions makes up the complete grammar. An individual production is similar to a mathematical equation, with a single symbol on the left-hand side and an expression on the right. An expression is composed of one or more symbols combined in various ways, such as concatenation or choice. Symbols are further classified as being terminal (composed of one or more characters in the alphabet that aren't further reduced into expressions) or nonterminal (defined only in terms of the expressions to which they are equated).

As an example, let's consider a language composed of all strings of three zeroes or ones, in any combination of zeroes and ones, in any order. We can define this language using the following grammar expressed in EBNF notation.

The Language of Three Zeroes or Ones
Sentence ::= zero_or_one zero_or_one zero_or_one
zero_or_one ::= "0" | "1"

Sentence is the starting nonterminal symbol in the grammar. zero_or_one is the other nonterminal. The first production shows a concatenation of three of these symbols. "0" and "1" are terminal symbols in the grammar, shown in quotes as literals from the alphabet. The second production shows a choice of one or the other of these two terminals. The strings 000, 010, and 101 are in the language.

Getting a bit more specific, most of the grammars we are dealing with are not only context-free grammars but also members of a special class of grammars called "LR(0) grammars." This is a term from compiler construction that describes a set of languages that can be parsed from left to right with zero characters of lookahead. It can also be derived from terminal symbols to nonterminal symbols in a bottom-up fashion starting at the right side of a syntax tree. It is beyond the scope of this book to really explain what all of that means. The important thing to understand from this concept is that for most of these file formats we can fully identify a character from an input file, that is, understand its meaning and place in the grammar, simply by considering what we have already read from the file. We don't need to read ahead beyond the character to understand what we should do with it.

Understanding the types of grammars we are using is also very important because it gives us guidance about parsing algorithms that might be appropriate for processing the grammar. We can look to the study of grammars and parsing algorithms in compiler construction for some ideas about how to best process these non-XML files as input. (I should note that some subsets of our legacy grammars also fall into an even narrower class of grammars called "regular expressions." I won't delve too much further into this area, though, because BNF with its orientation toward context-free grammars has become the de facto standard syntax for expressing language grammars. However, the fact that some are regular expressions is significant because this points us to yet other parsing algorithms, as we'll see in Chapters 7 and 9 on CSV and EDI, respectively.)

We'll start our grammar analysis in the next chapter by analyzing the grammar of CSV files. This will be fairly simple since we're restricting the types of CSV files we're supporting to those in which every row has the same format, that is, the type of data in each column is the same for every row, similar to a spreadsheet with column headings. In this case we're concerned primarily with the grammar of a row. In dealing with flat files we'll pay more attention to the grammar of the file structure since we need to understand the organization and sequence of the various types of records in the file. Finally, in dealing with EDI we'll be concerned with both the grammar of the file (or EDI document) and the grammar of a record (or EDI segment) in the file.

A final point, directly related to my previous comments about tweaking EDI grammars, is how we're going to use our grammars. Our utilities are primarily concerned with converting legacy non-XML formats to and from XML. As such, whether or not the legacy files strictly comply with their full grammars is not a consideration for us. We only need to understand and specify enough of the grammar to enable us to convert the format to and from XML. For example, in developing the utilities we're not going to be concerned with whether or not all of the required segments and elements are present in an X12 EDI transaction set. We just want to be able to understand enough of its structure to accurately convert it to and from XML. Doing full grammar checking would add a huge amount of complexity to our code, and that would be in conflict with several of our requirements.

So, you ask, if we're not going to do full compliance checking in the utilities, how can we be sure that the non-XML formats comply with our business constraints? How can we be sure we're sending or receiving valid EDI? Saying we're not going to build validation of the non-XML formats in the utilities doesn't mean we're not going to support full (or almost full) grammar and content validation. We'll do it with schemas. We'll talk about how in just a minute, but let's finish talking about the non-XML grammars.


Describing the Legacy Non-XML Grammars

We've decided that we need to approach grammar analysis of our non-XML formats in a more rigorous fashion and that we're going to use EBNF as one of our primary tools. However, the machine still needs to know about the grammars. How are we going to inform the machine about the grammars? We could use EBNF or something similar such as Prolog, but that would involve building the appropriate interpreter into our utilities. Scratch that idea! So what other choices do we have? We'll need to devise a representation for the grammar, and my intuition tells me that the representation may in part be dependent on the format we use to inform the machine. So, what choices do we have about format?

We could encode the grammar somehow in a database. This is a reasonable idea in many ways. Databases can be secure, reliable ways to store data. Databases also give us a way to enforce some constraints on the data. However, adding database support to our utilities comes with quite a bit of baggage. To ensure portability with various database engines, we would probably want to use Open DataBase Connectivity (ODBC) or Java Database Connectivity (JDBC). Database support gives us another technology to deal with and adds complexity. We would also have to give the end user a mechanism to load the grammar into the database, that is, a user interface. Again, this adds complexity. Finally, using a database to encode the grammar imposes a runtime requirement on the user to have an appropriate database engine installed. The disadvantages outweigh the advantages.

We could encode the grammar in flat files or CSV files. Although not quite as secure or reliable as databases, these formats would certainly be adequate. For reading these formats we could use techniques already developed for reading flat and CSV files. Flat files can be created with a text editor, and CSV files can be exported from a spreadsheet. So, we have at least a basic user interface. Also, the users won't need a database engine at runtime. Despite these advantages there are a few disadvantages. Creating flat files with a text editor is very vulnerable to user data entry error. We can put code into the utilities to detect many of these types of errors, but again this adds complexity to the code. Also, some types of errors may be very hard to detect. Getting a field name or position off by one character in the grammar file can lead to errors that may not be detected by the utilities and may only be detected by looking for incorrect output. That is probably not acceptable. Finally, although grammar descriptions could be created with a text editor or a spreadsheet, these are not exactly the friendliest methods from a user perspective. So, flat files or CSV files aren't a very good choice.

That leaves us with XML, which is a pretty good choice since we're working in an XML world. Right? We have two choices in XML representation: (1) we could create our own XML-based languages for describing our grammars, or (2) we could use W3C XML Schema language. Let's look at the latter option first.

I find the idea of using schema language for describing non-XML grammars very enticing. The user would only have to develop a single description to cover both the XML and non-XML formats of a business document. Tools such as XMLSPY and TurboXML could be used to develop these schemas. In the utility that converts XML to the non-XML format we could use default Attributes to provide the information necessary for conversion. For example, we can get the information about file organization and record sequence, that is, the file's grammar, from sequence definitions in complex types. In addition, a schema could define DataFormat, Offset, and Length Attributes with default values on a Field Element representing a field in a flat file record. The utility could then access this information from the DOM tree and create the output accordingly. There might be additional information we could not "shoehorn" into the standard schema language features, but there is a mechanism for specifying this information. Most schema language Elements allow users to add their own Attributes provided they are from a namespace other than the schema language namespace.

However, there are some distinct disadvantages to this schema-based approach. To understand the grammar of non-XML input files we would have to interpret the schema. Schema documents can be very complex, and we would really like to avoid writing all the code required to interpret them. Tools can help us understand and use schemas, but such tools aren't without their drawbacks. MSXML offers something called the Schema Object Model (SOM). It provides a way to parse and give programs access to schemas in much the same way that the DOM enables processing of instance documents. In the Java world, the open source Castor framework developed by exolab.org also offers an object model with very similar functionality [exolab 2002]. However, the SOM is not a standard. Unlike the DOM APIs offered by JAXP and MSXML, Castor's and MSXML's SOM APIs are not logically equivalent implementations of the same model. There are some significant differences in the interfaces, methods, and properties offered by these two implementations. We would not be able to develop one approach that would work for both of our implementation environments as we did with the DOM.

Finally, for practical considerations we would have to set some constraints on how the schemas are written. The main reason for this is that schema language, as we noted in Chapter 4, is incredibly flexible. There are a near infinite number of schemas that could be written to specify the same instance document. Handling all the different approaches to schema design would make our schema processing code very complex. It would be simpler to code and maintain if we restricted schemas to a certain style. For example, we might want to have all types declared anonymously, in-line, rather then being declared as named types, perhaps in an imported schema. We would then need to make sure that these constraints were observed. One approach for doing this would be to write a schema for schemas that enforced these constraints, then validate our schemas against this schema at runtime. It would basically specify what the W3C's DTD or schema for schemas do but would be a much more restrictive model. However, this approach might be just as complex as writing the required validation code in the utility. In addition, writing schemas to support description of non-XML legacy grammars may make it harder in some ways to write schemas for validation, that is, enforcement of business constraints.

Finally, the schema-based approach would require all end users to become reasonably proficient in writing schema language. I feel comfortable encouraging people to become proficient in reading schemas, but I don't necessarily feel so comfortable when it comes to writing schemas. Also, end users would have to write schemas not only in schema language but also in a specific style, with customized extensions required by our conversion utilities.

So, as enticing as I find the approach of using schema language, I must regretfully give up on it, at least for this phase of the project. That leaves us with using XML instance documents.

Our choice is to develop XML-based languages to describe the grammars of each of our non-XML formats. This approach has many advantages. We can make the languages fairly simple. We can write schemas to validate that the instance documents conform to the languages and adequately describe the grammars. We can use our DOM techniques to read the grammar describing instance documents. These instance documents can be created using tools as simple as Microsoft's XML Notepad or as sophisticated as XMLSPY. They could even be created with a text editor, although I don't recommend it. The main disadvantage of this approach is that if end users want to do full validation against business constraints, they also have to develop a schema for the XML representation of the grammar. That means they would have to develop two metadata documents instead of one. However, they are free to develop that schema in whatever fashion they desire, and they don't need to develop it if they don't care about validating business constraints.

So, we'll develop XML-based languages to describe our non-XML grammars. We'll call the documents that specify these languages our "file description documents" since we'll be specifying a few details beyond just the grammars. The approach will become clearer when we look at the first case, CSV files, in the next chapter. Another advantage of this approach will become clear when we look at the grammar of flat files and EDI messages. If we construct our grammars properly, we can use the DOM as our internal data structure for storing the grammar rather than having to create a different mechanism. This extremely powerful feature will become more evident when we look at the pseudocode. It saves us from having to develop our own data structure for describing the grammar.


Representing the Legacy Non-XML Grammars in XML

We'll take a detailed look in the next three chapters at each of the languages we'll use to describe the grammars of our legacy file formats. However, let's look at the overall approach since it will help clarify some of the classes and methods discussed later in this chapter. There are four major areas we want to address here.

  • The XML instance documents that correspond to our legacy file formats

  • The file description documents that describe the grammar and other characteristics of our legacy file formats

  • The schemas for our file description documents

  • The schemas that are optionally used to validate our instance documents

Instance Document Design

As discussed in Chapter 4, the place to start is how we want the instance document to look. The bullet items below outline the major decisions involved. In the next subsection we'll make the same decisions in regard to the file description documents that describe our legacy non-XML formats.

  • Elements and Attributes: I use Elements only for application data, mainly because the general purpose utilities we're discussing are designed to handle a wide variety of logical documents. The legacy formats in and of themselves offer little basis for deciding what should be an Attribute instead of an Element. For example, when dealing with a generic CSV row all the columns have the same significance. Building in the capability to designate some data fields as Attributes of others that are Elements adds complexity both for the end user and for the developer. The rationale is discussed in the section on DOM processing considerations (later in this chapter) in a bit more detail.

  • Naming conventions: The utilities are designed to enable users to assign Element names of their choice. I recommend simple, common business terms set in upper camel case. You are free to use whatever approach makes sense to you. You will probably only see and use these names when setting up the conversion utilities and in the XSLT stylesheets that convert to and from your chosen XML format.

  • Specific names versus qualifier/value pairs: This is somewhat of a moot point since our legacy formats already reflect the decision. Your XML document is going to look like your flat file or EDI document.

  • Structure: The structure will match the logical hierarchy of the legacy format, not necessarily its physical hierarchy. For example, our CSV format has only one record type, so the hierarchy is flat. Flat files and EDI formats can have groups nested within groups, yielding hierarchies that are sometimes several levels deep.

  • Namespaces: We're not using them; we're going to use the unnamed default target namespace. Named namespaces, versus the default unnamed namespace, are useful in many circumstances. For example, they can disambiguate Element and Attribute names that would otherwise be duplicates by associating them with different namespaces. They can also make it more explicit that certain names come from other domains. However, in the case of our legacy file converters, the XML instance documents that we read and write have relevance only to our utilities. They most often will serve as intermediate formats for XSLT transformations and won't be used directly by any other applications. Named namespaces, particularly if we require Elements to have namespace prefixes, add no clarity and add complexity to our code. We're following the principle of KISS, remember?

File Description Document Design

General Considerations

As I alluded to previously, in addition to specifying the grammars of our legacy non-XML formats, the file description documents also describe physical characteristics of the input files. For example, we specify record terminators and certain characteristics of the output XML documents such as schema URLs. Again, let's look first at the general choices we need to make about the design of these documents. In the previous section we made decisions about how our instance documents should look. Here we consider how the file description documents should look. We allowed a bit more flexibility in the instance document design than we will here largely due to the variation in legacy file formats. Also, we don't want to impose restrictions where they aren't absolutely required. However, in the case of the file description documents, our processing code is relatively more tightly linked with their document design style. We'll impose a bit more order to make our coding easier.

  • Naming conventions: I have chosen relatively intuitive, descriptive names for the Elements and Attributes, with no formal analysis behind them. I have also chosen to use upper camel case.

  • Specific names versus qualifier/value pairs: This is not much of a consideration for the file description documents. To the extent that it applies, I use specific names.

  • Structure: The structure follows a logical breakdown of the file and grammar characteristics. Where we have a flat grammar, as in CSV files, the XML representation is pretty flat with only three levels. More complex grammars such as EDI can have several levels.

  • Namespaces: Again, for most of the considerations noted in the previous section, we're not using them.

The other major choice that needs to be made about the general format of the file description documents is how we use Elements and Attributes. This one deserves a bit more discussion than just a bullet point. Reviewing my thought processes and what I ended up doing may help you with similar decisions in the future.

My initial inclination was to use only Elements, matching the appearance and processing approach used for the instance documents that correspond to our legacy formats. However, when dealing with the grammars, particularly when considering the grammars of flat files and EDI documents, I wanted to use some Attributes. I felt the grammars would be easier to work with if the nonterminal symbols, such as groups, records, and fields, were represented as Elements. There were a few properties of groups and records that I also needed to represent, but I didn't want to depict those as Elements on a peer level with the Elements representing the grammar symbols. For example, I didn't want a flat file's record identifier tag to be a sibling Element to those representing the fields in the record's grammar. That led to creating a few selected Attributes for such things. Then, as I began to design the code, it became apparent that certain field characteristics, such as column number and data type, might be more easily accessed as Attributes. This was leading me to an inconsistent design in which some parts of the document used only Elements and other parts, in particular the grammar, used only Attributes. In the end I decided to go completely the other way and just use Elements for structure and Attributes to depict all values. Basically, I decided to adopt the approach used in W3C Schema language. I felt consistency was important within the document, that it would make processing the document easier. More importantly, I felt it would be easier for end users of the utilities if I followed a consistent approach for all parts of the document.

Major Sections and Elements

The grammars of our legacy formats are described using the following basic items. The exact Element names may vary depending on the legacy grammar being described.

  • Grammar Element: In the file description document, this is the root Element of the subtree that describes the grammar. Depending on the legacy file grammar, it may have RecordDescription or GroupDescription Elements as children.

  • GroupDescription Element: This Element describes a set of records (and perhaps other groups). Its first child Element is always a RecordDescription. It may have one or more other RecordDescription or GroupDescription Elements as children. The GroupDescription Element is not used in our CSV file grammar since we support only one record type and therefore have no groups.

  • RecordDescription Element: This Element describes the structure and overall characteristics of a record. Its child Elements are FieldDescription Elements.

  • FieldDescription Element: This Element has a number of Attributes that describe characteristics of the field in the legacy format. It always specifies a name, a data type, and a number. Depending on the legacy format it may also specify information such as length, minimum or maximum length, offset, and fill character.

EDI grammars add another layer, but we'll talk about that in Chapter 9.

Schemas for File Description Documents

General Schema Design Considerations

I develop schemas for each of the file description documents and validate the documents against the schemas. Using schema validation enforces some predictability about the documents and makes writing the code to process them a bit easier.

As discussed in Chapter 4, several issues must be considered when designing schemas. We'll discuss schema design for application files in the next subsection and in more detail in Chapter 12, but here is a summary of some of the major design decisions regarding the schemas for the file description documents.

  • Naming conventions: We've discussed most of them already, but I do want to note that I adhere to a convention of appending "Type" to named types.

  • Structuring: One schema specifies the file description document when the legacy format is the source of the conversion; another is used when the legacy format is the target. Due to the fact that the two schemas have very similar data, I declare most of the named types in the common schema for that legacy format. The common schema is then included in the source and target schemas. In addition, a project-wide (or Babel Blaster–common) schema specifies types used in all the conversions. Figure 6.1 shows the organization.

    Figure 6.1. Structuring of File Description Document Schemas

  • Global Elements versus named types and local Elements: I use named types and local Elements since, due to my programming background, this approach seems to come more naturally. It is very similar to declaring classes, extending them into subclasses, and creating objects of a class.

  • Named versus anonymous types: If a type is reused, it is named. This holds true for all the types in the include schema. The main exception to this overall approach is in the source and target schemas. The root Element declaration in each has an in-line anonymous type declaration in which its child Elements are declared as a sequence.

  • Namespaces: As discussed in Chapter 4, we could develop an approach that uses named namespaces in schemas but still avoids using them in instance documents. However, our overall schema architecture is simple enough that segmenting it into two or more namespaces won't add much clarity or other value. To keep things simple, except for the things we use from the schema language namespace, we're not going to use named namespaces in our schemas. We will stay with the approach of using the unnamed default target namespace.

Common Schema for File Description Documents

The common schema, BBCommonFileDescription.xsd, defines several types that are reused in the other schemas. In doing so, it uses a few schema features that deserve comment.

Since all the data values are conveyed as Attributes of empty Elements, some special things needed to be done. The Primer of the XML Schema Recommendation gives one example of such an empty Element. It defines an anonymous complexType with complexContent but defines no child Elements. It defines only Attributes. This approach works well, except that it derives the complexType by restriction from the anyType data type. This means that in an instance document you may add other Attributes. (It seems that the schema for the W3C XML Schema language takes this approach.) However, I wanted validation to be a bit tighter than that, so I created an EmptyType complexType with no child Elements or Attributes and used it as the restriction base for all types used for Elements.

Setting up the DelimiterType for Attributes declaring delimiter values was interesting since there were several choices. The basic requirement for most delimiters, such as column delimiters in a CSV file, was that the user has the ability to declare them as literal characters or as hexadecimal values. The latter is important for EDI files and for future support of EBCDIC-encoded data. My first approach was to declare separate Elements for the character and hex values, but this seemed a bit cumbersome for both the user and for my coding.

A second approach suggested that I might just declare the DelimiterType as token with length of one and use the syntax (&#x Hexadecimal number ;) for entering characters to enter the hex values. There are two problems with this approach. The first is that the hex value represented not the actual hexadecimal value in the data stream but the hexadecimal value of the character's "code point" in the ISO/IEC 10646 standard. The concept here is that a number of different characters in different character sets, each having different code points, might all be encoded in a data stream with the same bit pattern. I suspect that most of the potential users of the utilities might be much more familiar with actual encoded hex values than with code points, so this approach didn't seem appropriate. The second problem is that, although I didn't fully investigate this, it appeared to me from the code point ranges listed that it might not be possible to use this syntax to specify control characters encoded with hex values below x20. These control characters are sometimes used as delimiters in EDI syntaxes.

This led me to the third approach, which I adopted: to declare the DelimiterType as a union of a token with a length of one and a hexBinary with a length of one. The latter restriction hung me up briefly since the length is not exactly intuitive. The length facet on hexBinary applies to the length of the binary octet that the hexBinary number represents, not the string length of the hexBinary number itself. So, to get a single byte, such as the line feed character, we use a length of one rather than the length of its hexadecimal representation (x0A). It's worth noting, too, that to use delimiters that have special meaning in XML the corresponding predefined entities must be used. For example, use &quot; instead of a literal quotation mark. Defining the RecordTerminatorType was a similar process, except it is a union of a one-byte hexBinary and a token with enumerations of W and U, representing Windows-style and UNIX-style physical record terminators.

NOTE Adding New Data Types

When modifying the utilities to add a new data type, add the enumeration for the type's coded value to the BBDataType type in this schema.

Schemas for Source and Target Documents

How We'll Use Schemas

Schemas for the source and target XML representations of our legacy formats play an important but optional role in this approach. We decided that we'll use our own XML-based languages for describing the grammars of our non-XML formats. In addition, we decided that our utilities, in and of themselves, are not going to perform full validation against business constraints when converting but will perform only the validation necessary for converting our non-XML formats to and from XML. So, what role do schemas play in the architecture?

Schemas play an important role by providing the primary means by which business constraints are enforced. We will validate our XML formats against schemas that define the XML representations of our non-XML formats. Rather than writing a lot of our own code to validate business constraints, we're going to take advantage of the schema validation offered by the XML APIs.

For example, we won't check the raw X12 850 Purchase Order for compliance with the X12 standard (or implementation guideline). We will instead support schema validation of the XML instance document produced from the X12 850 by our EDI to XML utility. Our common strategy in all of these conversion utilities will be to create a one-for-one correspondence (or, in stricter terms, one-to-one and "on to," or isomorphic, correspondence) between the XML and the non-XML formats. So, if the data satisfies the appropriate business constraints in the XML representation, as defined in a schema, changing the syntax to a non-XML representation should make no difference in satisfying the constraints. The same is true in reverse. Using XML as the format in which our data content is validated further cements XML's central place in our architecture.

Creating the Schemas

Although I recommend that users at least be able to read schemas, I want to avoid any requirement that they be fluent in schema language. The design approach I present here keeps that consideration paramount. KISS comes first. I propose the users start with what is basically a simple approach to designing schemas. I recommend an initial approach of using a tool such as XMLSPY or TurboXML, feeding it an instance document, and having it generate a schema. For most users this will beat hand-coding a schema from scratch every time. Users must review the generated schema and will almost certainly need to clean up and correct a few things such as the assigned data types, minOccurs, and maxOccurs. (This is discussed in more detail in Chapter 7.) However, this approach is much easier than trying to create a schema completely by hand. If such a tool isn't available, users can take the examples from the next three chapters and hack them up in a text editor or XML Notepad to suit their needs. However, I highly recommend a full-featured XML tool.

If a user is fluent with schema language, he or she could do something as ambitious as developing type libraries for the segments and data elements in the full UN/EDIFACT standard. I'll discuss some considerations and approaches in more detail in Chapter 12. But if he or she just wants to load a document into a tool like XMLSPY and click on Create Schema, that's okay too. The important thing to note is that users have considerable freedom in developing schemas that meet their needs.


Additional DOM Processing Considerations and Strategies

We talked in Chapters 2 and 3 about some basic DOM programming techniques. We're starting to deal with more complex documents, so it is appropriate to discuss some more advanced topics and potholes on the road to creating robust DOM programs that are easy to maintain.

A primary consideration is whether we want to retrieve data from a DOM document by name or whether we want to traverse the document tree in some particular order and process things as we encounter them. To access data by name we can use a few basic methods. The ones listed below are sufficient for the points I want to make, although other methods are also available.

  • getElementsByTagName: retrieves an Element by its name, returning a NodeList; offered by the Document and Element interfaces

  • getAttribute: retrieves an Attribute value by name, returning the text value of the Attribute; offered by the Element interface

  • getNamedItem: retrieves a Node by name from a NamedNodeMap; offered by the NamedNodeMap interface

On the other hand, to traverse a document in a particular order, several attributes (and get methods) offered by the Node interface allow us to walk the Document tree in nearly any fashion we might want.

  • parentNode: the Node's parent Element, as a Node

  • childNodes: a NodeList of all child nodes

  • firstChild: the first child Node of the Node; same as childNodes.item[0]

  • lastChild: the last child Node of the Node; same as childNodes.item[childNodes.length - 1]

  • nextSibling: the next sibling of the selected Node; corresponds to parentNode.childNodes[index of this Node + 1]

  • previousSibling: the next sibling of the selected Node; corresponds to parentNode.childNodes[index of this Node - 1]

In addition, several methods return a NodeList, and we can select individual Nodes from the list using its item method.

The hitch here is that we have to be a bit careful when mixing these two approaches because when we have retrieved an Element using one of these Node attributes, even though we might know it is an Element, according to the DOM it is a Node, which is the Element interface's base class. To get access to the Element interface's methods and attributes we have to cast the retrieved Node as an Element or copy it to an Element (the exact technique depends on the programming language). For example, in Visual C++ using MSXML we can do the following:

IXMLDOMDocument2Ptr spMyDoc;
IXMLDOMElementPtr spMyElement;
IXMLDOMNodeListPtr spNodeList;
...
spNodeList = spMyDoc->getElementsByTagName("TheName");
spMyElement = spNodeList->item(0);
MyValue = spMyElement->getAttribute("MyAttribute");

In Java using Xerces we could do the equivalent operations:

Document MyDocument;
Element MyElement;
NodeList NodeList;

NodeList = Document.getElementsByTagName("TheName");
MyElement = (Element) NodeList.item[0];
MyValue = MyElement.getAttribute("MyAttribute");

There are two things to bear in mind here. The first is that casting a base class object to a derived class object or otherwise creating a derived class object from a base class object can be risky programming practice. If the base class object isn't really a member of the target derived class, you can throw runtime exceptions. You want to be absolutely sure about what you're doing. One way to avoid exceptions is to check the Node type before doing a cast or setting a derived class item equal to a base class item.

Casting Nodes back to Elements lets you use the getAttribute and getElementsByTagName methods of the Element interface, which can be very convenient. Aside from the runtime exceptions, you need to be aware of the types of the various interface attributes and the types returned by various methods. Compilers are usually very helpful (or annoying, depending on your perspective) in this regard, so you're probably not going to miss doing a cast when one is required.

However, if you want to completely avoid the potential problems, there are other approaches for retrieving items by their names. I don't use them very much in this book because they require a lot more code. However, here are the outlines for two basic options, depending on whether you want to get a child Element or an Attribute.

  • Element: Walk the list of the Element's child Nodes using either a child Nodes NodeList or the firstChild, followed by calls to nextSibling. Then check the nodeName to see if you have the Element you are seeking.

  • Attribute: Use the Node's "attribute" attribute, which is a Named Node Map of the Element's Attributes. You can then use the getNamedItem method in the NamedNodeMap interface, passing the Attribute name. This returns the Attribute Node, and you can then call that Node's getNodeValue method to return the text of the Attribute.

You can easily see that getting an Element requires a lot more code and probably CPU cycles than the getElementsByTagName method. Getting an Attribute doesn't necessarily visibly invoke a lot more processing in your code (though we can't be absolutely sure what the DOM API is doing for us), but it certainly is a lot more verbose than the Element's getAttribute interface.

The bottom line here is that I recommend using whatever navigation or selection method seems appropriate to what you want to accomplish. If it involves casting a Node back to an Element, just be careful about it.

There is one other thing to be aware of when traversing part of a document tree using the Node attributes. Comments, Processing Instructions, and whitespace Text Nodes can all be in a childNodes NodeList or a nextSibling (unless you have created the NodeList with the getElementsByTagName method). If you want to process only Elements you will need to check for the Node type. You'll find several places in the Java and C++ code where I skip over non-Element Nodes.

Let's look now at how I apply these considerations to the instance documents processed and created by our utilities and at how we define and process the file description documents.

As I said above, these are general purpose utilities that don't inherently know anything about the XML documents that correspond to our legacy formats. They know about the legacy formats only in abstract terms. For example, the CSV to XML utility doesn't know anything about an invoice file in CSV format or the individual XML format invoices it produces. It knows only about the abstract, general characteristics of CSV files. At runtime it gets a lot of information about the target XML format from the CSV Source file description document. But since it needs to be general purpose and process other types of business documents, we need to code for the common characteristics. So, as I said earlier, we just use Elements to represent the application data. This means that except for Elements that convey structure information, such as record and group description Elements, we treat all Elements with equal semantic significance (or insignificance). In practical terms this means that we don't have any reason to retrieve or process Elements by name when converting from XML to a legacy format. We instead just walk the document tree and process Elements according to their characteristics as described in the file description documents.

We have a somewhat different situation when processing our file description documents. When initializing the converters we want to retrieve specific information about the legacy formats. For example, we need to know the record terminators. If we want to include a schema reference in the target XML document, we need to retrieve the schema URL from the file description document. Finally, we need to retrieve the Grammar Element. In these cases we retrieve Elements and Attributes by name. However, when processing the grammar we take a somewhat different approach. In almost all cases we walk through the document's subtree containing the grammar representation on a Node-by-Node basis. This is because the way we navigate the grammar must match the way we process the source and target instance documents and files. We process legacy formats sequentially, which corresponds to walking the grammar subtree sequentially (a preorder traversal) except when we backtrack and repeat a group or record. The same logic applies to our source instance documents. We process them sequentially (as a preorder traversal of the DOM document tree), so we do the same thing with the grammar.

I want to make one final, general point about DOM programming before moving on. Several of the DOM methods we call can return a null pointer if the requested object doesn't exist. If we then try to call a method of the returned object and the reference is null, we throw an exception. For example, if a document doesn't have a RecordTerminator Element, the getElementsByTagName method would return a null pointer instead of a NodeList. We would throw an exception if we tried to access item[0] of the NodeList. In many scenarios safe programming would dictate that we check for such null pointers before making such method calls. However, because our file description document is validated against a schema, by the time we get to bits of code like this we know how the DOM Document in memory is going to look. So, we can dispense with most of our checks for null pointers. This is another argument in favor of performing schema validation if you can.

To summarize, I think the main thing to get from this discussion and previous points is that there is no one-size-fits-all solution. There is no approach that will be best for all types of documents and processing situations. In addition, even for the same circumstances, different people are going to have different opinions about the "best" solution because they value differently the underlying requirements such as efficiency or maintainability. I hope that walking through some of these decisions I've made and the tradeoffs I've considered will help you deal with similar problems.


Multilingual Issues

Most developers targeting their products for markets other than just the English-speaking world (or the German-speaking world, if you prefer) are well aware of the several issues surrounding multilingual support. We need to address two of these in relation to the utilities we're developing.

The first issue is the ability of the utilities to manipulate documents in several different character sets. XML (through its support for Unicode), Java, and Win32 all support several different character sets including multibyte sets. I have chosen Unicode's UTF-8 as the character encoding specified in the XML prolog of the instance documents these utilities create. Due to the design of UTF-8 most requirements should be accommodated. However, if you need a different encoding, I will assume you are aware of the issues involved. Here is some relevant information about the utilities as currently implemented.

  • All record lengths, field lengths, and offsets are specified in terms of bytes, not characters.

  • Delimiters such as line feed record terminators, CSV column separators, quotation marks, and EDI segment terminators are handled as single-byte characters.

  • Conversions that involve examining and manipulating data treat the data as single-byte characters. Examples include converting dates to and from ISO 8601 formats and numeric conversions.

  • String fields are considered strictly as a series of bytes with no attention paid to character encoding.

The second issue is in the area of text messages displayed by the utilities. The text messages created within the utilities are all in English. However, they are mostly coded as string constants in a common class or header file. This should make it fairly easy to create versions that display messages in other languages. A more ambitious enhancement would be to build tables with the messages in the appropriate languages and let the end user configure the desired language. For this version I'm not that ambitious.

I have not extensively researched whether or not JAXP with Xerces or MSXML have the ability to generate error message text in the computer's local language. There is no mention of multilingual messages in the documentation for either API, so I'm going to assume it isn't there. If you need such functionality, you should probably research whether or not the API you selected does, whether it supports your specific language requirements, and the specific mechanisms you need to employ to take advantage of the functionality.


Error Handling Strategy

I said in Chapter 1 that we were going to take a fairly simple approach to error handling. I'm not going to change that direction, but we need to be a bit more specific about what we're going to do.

There is a bit of debate in the XML world about just how strict validation should be. The debate centers on questions like whether or not a somewhat lax model of validation should be supported for certain applications or types of problems, rather than a very strict model. For example, some people think that a string Element that exceeds its maximum length facet should just cause the APIs to issue a truncation warning instead of throwing an error and terminating parsing or validation. All the parsers I know of support only a valid or invalid state; there isn't any in-between. If parsers were to do something in-between, it would be nice if the Schema Recommendation defined those in-between levels. It really doesn't as currently written. This discussion is, of course, somewhat tangential to our main concerns, but it gives a flavor of the issues involved.

My guiding principle in error handling is to ensure that users get what they expect and only what they expect, or get nothing at all. Some may consider this a rather severe and unforgiving strategy, but the cost of manually correcting and reprocessing is probably less in the long term than inadvertently introducing bad data into an application. So, when in doubt I'll terminate the processing.

What this means for the conversions is that if the data I'm manipulating doesn't match what the algorithm expects, the utilities will terminate. For example, if I'm converting a date in MM/DD/YYYY format to the ISO date format of YYYY-MM-DD but the users specified a two-digit year (02 instead of 2002), the conversion attempt will terminate. It will also terminate if a string is truncated on conversion unless we explicitly say we're going to allow truncation on that field. I'll be more specific about termination conditions for each of our supported data types, but this gives you the idea.

In Java we'll follow the usual path and create and throw our own exceptions. In C++ we'll follow the conventions established in Chapter 1 and use return codes if we can force an orderly exit back to the main procedure. If this turns out to be awkward for a specific circumstance, we'll just call an error routine and exit. For both implementations we'll have standard error messages with text coded as constants in separate interfaces or header files.

A final word. To keep the code simple in many cases we'll assume that the XML data is what we expect it to be. We're not going to go to much trouble to validate it and check for errors. That's what schema validation is for. For example, when converting from an Element with a schema language date data type, we're going to assume that we're working with data that fits the pattern of YYYY-MM-DD. If we get data that is too short we may throw a runtime exception. However, when handling our non-XML legacy formats we'll put in enough logic to gracefully exit when we attempt to process bad data rather than throwing nasty exceptions.


High-Level Design

Having dealt with these general issues we can now lay out the high-level design of our utilities. All the utilities for which the legacy format is the source format share a common structure, as do all for which the legacy format is the target format. Both sets are built on several common classes developed in this chapter. Derived classes specific to the legacy format are built in each of the next three chapters.

Source Converter Processing

Figure 6.2 shows a simplified high-level collaboration diagram of the classes and processing of the source converter. Exact details vary for each of the converters, but this diagram and description should provide a good high-level understanding of the design.

Figure 6.2. Source Converter Collaboration Diagram

The main routine processes command line arguments and then calls the processFile method of the SourceConverter class appropriate to the legacy format. The processFile method creates disk directories for the output XML documents as determined by the processing options. It also creates DOM Document objects and serializes them to disk. To process the input file, for each record in the input file, it calls the readRecord method of the RecordReader class appropriate to the legacy format. The processFile method then calls the RecordReader's parseRecord method to extract the individual fields from the input record. A DataCell object is created for each field in the input record, with the specific derived class type determined by the data type of the field. The toXML method of each DataCell object is then invoked to convert the contents of the field from the legacy data type to a corresponding schema language data type. Finally, the processFile method calls the RecordReader's writeRecord method to write the DataCell object contents to DOM Elements representing the legacy record and its fields.

Target Converter Processing

Figure 6.3 shows a simplified high-level collaboration diagram of the classes and processing of the target converter. Again, the exact details vary for each of the converters.

Figure 6.3. Target Converter Collaboration Diagram

The main routine is very similar to that of the source converter in that it processes the command line arguments. However, instead of just passing along the name of one legacy source file to the TargetConverter class, it reads a directory to retrieve the file names of the XML documents to process. It also opens and closes the output legacy format file. For each XML document file in the input directory it calls the TargetConverter's processDocument method. This method parses the input XML document file and loads it into a DOM Document object tree. Then, for each Element that represents a record in the legacy format, it calls the RecordWriter's parseRecord method. This method creates a DataCell object for each Element in the DOM tree that corresponds to a field in the legacy format record. As with the source converter, the derived class of the DataCell object is determined by the legacy data type of the target field. Finally, the RecordWriter's writeRecord method is called to write the legacy format record. In its processing it calls the fromXML method of each DataCell object to convert the data representation from the schema language data type to the legacy format data type.

Summary of Classes

The utilities are built using three families of classes.

  1. Converters: These handle the high-level tasks associated with a conversion. The methods in these classes are responsible for the main processing flow of converting to or from a legacy format.

  2. Record handlers: These classes handle the details of reading, parsing, or writing a record in a legacy format.

  3. Data cells: These classes handle the details of converting field contents between a legacy data type and a schema language data type.

Figures 6.4, 6.5, and 6.6, respectively, show the inheritance diagrams for these three families of classes. In this chapter we develop the base classes of each family. In Chapters 7, 8, and 9 we develop the derived classes appropriate for each legacy format.

Figure 6.4. Converter Class Inheritance

Figure 6.5. RecordHandler Class Inheritance

Figure 6.6. DataCell Class Inheritance

    [ Team LiB ]


Detail Design

We're now ready to look at some more nuts-and-bolts logic and code. In this section we'll review the models for our main routines and examine some base classes from which we'll derive the classes used in implementation.

Main Routine Structures

All of our main routines will be similar, but they will fall into two distinct types. One type will convert non-XML formats to XML, and the other will convert XML to non-XML formats. Aside from this, we'll code the main processing logic as callable functions and the main routines that handle command line processing as basic shells. This strategy gives us the option to not only create stand-alone utilities for now but also to build later an integrated system that calls all the functions from one main routine.

Main Routine for Legacy Format to XML Conversion
Arguments:
  Input File Name
  Output Directory Name
  File Description Document Name

Options:
  Validate Output
  Help

Verify options and display help
Create new SourceConverter, passing validation option,
    Output Directory Name, and File Description Document Name
Call SourceConverter's processFile method, passing
    Input File Name
Display completion message
Main Routine for XML to Legacy Format Conversion
Arguments:
  Input Directory Name
  Output File Name
  File Description Document Name
Options:
  Validate Input Documents
  Help

Verify options and display help
Open Output File
Create new TargetConverter, passing Output Stream and
    File Description Document Name
Set up implementation specific DOM environment
DO for all Input Documents in Input Directory
  New DOM Document <- Load, parse, and validate input document
  Call TargetConverter's processDocument method, passing
      DOM Document
ENDDO
Close Output Stream
Display completion message with number of documents processed

This approach gives us the flexibility in an enhanced utility to call the Target Converter's processDocument method on a DOM Document in memory, perhaps after it has been created from another DOM Document via an XSLT transformation.

The main difference between the utilities for converting to and from XML is in the structure of the main shell routine. For converting to XML most of the work occurs in handling the command line arguments. In converting from XML we need to open the input directory, then loop through the list of files in the directory, loading and parsing each and passing the resulting DOM document to the processDocument method.

To implement the converters we'll develop a converter base class, with derived source and target converter classes. Each legacy format will derive a source and target class.

NOTE Chapter Conventions

When discussing the classes in this section, the arguments and return values listed for the various functions are logical only and are intended to convey the basic functionality. They will vary for the Java and C++ implementations. For example, if a method returns a string, the Java version may actually return a string whereas the C++ version may take a char * pointer as an argument, passing the output area by reference, and write the results to that area. In addition, if some of the methods return no value, the C++ implementation may return a status value. The Java version may throw an exception.

Also, I list here only the primary methods. Various set and get methods may be omitted.

Converter Base Class

Overview

The Converter class is the base class from which the source and target converters are derived. The bullets below summarize the primary attributes and methods.

Attributes:

  • DOM Document File Description Document

  • DOM Element Grammar Element

  • String Root Element Name

Methods:

  • Constructor

  • loadFileDescriptionDocument

Methods
Constructor

The base class Converter constructor performs only some very basic setup.

Logic for the Converter Constructor Method
Arguments:
  None

Set up implementation dependent environment for loading and
    parsing the File Description Document
loadFileDescriptionDocument

This operation is coded as a separate function since the EDI converter may need to load more than one file description document as it processes an EDI interchange.

Logic for the Converter loadFileDescriptionDocument Method
Arguments:
  String File Description Document name

Returns:
  Status value or throws exception

DOM Document File Description Document <- Load, parse, and
    validate File Description Document from passed file name
Grammar Element <- call File Description Document's
    getElementsByTagName for "Grammar"
Root Element Name <- call Grammar Element's getAttribute
    on "ElementName"

SourceConverter Base Class (Extends Converter)

Overview

The SourceConverter, derived from the Converter class, is the base class for the CSV, flat file, and EDI source converter classes developed in the next three chapters. The lists below summarize the major attributes and methods.

Attributes:

  • DOM Document Output Document

  • RecordReader Reader

  • String Base Output Directory

  • String Schema Location URL

  • Boolean Validation Option

Methods:

  • Constructor

  • processFile (virtual)

  • processGroup

  • saveDocument

The processGroup method is listed here for completeness but will be developed in Chapter 8.

Methods
Constructor

The operations common to all our legacy format source converters include setting up the base output directory name and implementation-dependent DOM setup for output document validation.

Logic for the SourceConverter Constructor Method
Arguments:
  Boolean Validation Option
  String Base Output Directory

Call base class constructor
Validation Option <- Passed validation option
Base Output Directory <- From passed base output directory,
    Appending directory separator character if required
Perform any implementation specific setup for validating
    output documents before writing them
saveDocument

This operation is coded as a separate function since it may be called within a while loop that reads records from the input file and after exiting the while loop to save the last document.

Logic for the SourceConverter saveDocument Method
Arguments:
  DOM Document Output Document
  String Output File Name
  Boolean Validate

Returns:
  Status value or throws exception

IF Validate is true
  Validate output DOM Document
ENDIF
Save DOM Output Document to passed File Name, dependent on
    implementation

An enhanced version of this utility might give the option of looking up and calling an in-memory XSLT transformation on the DOM Document before it is saved to disk. We might even hand off the DOM Document in memory to a message handling service for e-commerce applications. We won't get that advanced in this book (KISS over efficiency), but the options certainly exist.

Derived Class Method
processFile

This method is declared as a virtual method in the base class and implemented separately in each of the derived classes for our legacy formats. The general processing model is to read a single file with multiple logical business documents, create a separate XML instance document from each logical document in the input file, and write a separate file for each of the XML instance documents. In an e-commerce type of situation where some of the documents are intended for different external organizations, we'll write the documents for each organization to a separate directory. The listing below shows the general logic.

Logic for the SourceConverter processFile Method
Arguments:
  String Input File Name

Initialize Sequence Number
Open Input File
Call RecordReader's setInputStream method, passing the input
    Stream
Record Length <- Call RecordReader's readRecord method
DO while Record Length => 0
  Record Grammar <- get Grammar for Record Type
  Call RecordReader's parseRecord method, passing Record
      Grammar
  Call RecordReader's toXMLType method
  IF Break on Partner ID
    Output Directory <- Base Output Directory + Partner ID
    IF New Partner
      Store new Partner in Partner Array
      Make Output Directory for Partner
    ENDIF
  ENDIF
  IF Break on Document
    IF Output DOM Document != null
      Call saveDocument, passing Output DOM Document and
        Output File Path
      Set Output DOM Document to null
    ENDIF
    Create new Output DOM Document
    Create Root Element and Append to Output DOM Document
    IF Schema Location URL is not null
      Create noNamespaceSchemaLocation Attribute and
          append to Root Element
    ENDIF
    Increment Sequence Number
    Pad Sequence Number to 3 digits with leading zeroes
    Output File Path <- Output Directory + Sequence Number
        + ".xml"
    Call RecordReader's setOutputDocument method, passing
        Output DOM Document
  ENDIF
  Call RecordReader's writeRecord method, passing Parent Element
      and Grammar Element for Record
  Record Length <- Call RecordReader's readRecord method
ENDDO
IF Output DOM Document != null
  Call saveDocument, passing Output DOM Document and
    Output File Path
ENDIF
Close input stream
Display completion message with the number of documents
processed

TargetConverter Base Class (Extends Converter)

Overview

The TargetConverter, derived from the Converter class, is the base class for the CSV, flat file, and EDI target converter classes developed in the next three chapters. The lists below summarize the major attributes and methods.

Attributes:

  • RecordWriter Object

Methods:

  • Constructor

  • processDocument (virtual)

  • processGroup

The processGroup method is listed here for completeness but will be developed in Chapter 8.

Method
Constructor

Similar to the SourceConverter's constructor, the TargetConverter's constructor is very basic.

Logic for the TargetConverter Constructor Method
Arguments:
  String File Description Document Name

Call base class constructor
Call loadFileDescriptionDocument from passed document

You will note that in this constructor, unlike the SourceConverter constructor, we load the file description document. This is because of differences in how we will process EDI files in Chapter 9. An EDI interchange that we read may have several different documents in it and may need several different file description documents. So, we don't want to load the file description document in the base class constructor. However, when we write an EDI interchange we'll just accept one document type as input. This is the same model we follow for CSV and flat files. We'll load the file description document in the base class target converter.

Derived Class Method
processDocument

As with the processFile method in the SourceConverter, this method is declared as a virtual method in the base class and implemented separately in each of the derived classes for our legacy formats. The general processing model is to read from multiple XML documents in multiple files and create a single output file in the legacy format. We'll also make our life somewhat easier by reading the files from a single directory rather than multiple directories. The listing below shows the general logic.

Logic for the TargetConverter processDocument Method
Arguments:
  DOM Document Input Document
Root Element <- Call Input Document's getDocumentElement method
Root Element Name <- Call Root Element's getTagName
IF (Root Element Name != Root Element Name from Grammar Element)
  Return Error
ENDIF
Child Element <- Get Root Element's firstChild, skipping over
    non-Element Nodes
DO for all Record Elements, starting with first child of Root
  Record Grammar Element <- Get Record Grammar Element from
      Document Grammar Element
  Call RecordWriter's parseRecord method, passing Record Element
      and Record Grammar Element
  Call RecordWriter's writeRecord method
ENDDO

That's the basic setup. Pretty simple, huh? Simple is good. Simple is robust, maintainable, and extensible.

RecordHandler Base Class

Overview

This class provides basic support for handling records in non-XML-formatted files. It is the ultimate base class from which all of our various reader and writer classes are derived. It has a few basic data structures and utility methods.

Attributes:

  • Array of Pointers DataCell Array

  • Integer Highest Cell

  • DOM Document File Description Document

  • Byte or Character Array Record Buffer

  • Integer Record Buffer Length

  • Byte Record Terminator1

  • Byte Record Terminator2

Terminator1 is used for all record formats except fixed length flat files. It is used to store the primary record terminator character for variable length files and the segment terminator for EDI. Terminator2 is used to store the secondary record terminator character for variable length files. It usually has a value only if the physical record terminators are a carriage return and line feed pair.

Methods:

The methods defined in this class are very basic utility methods.

  • Constructor

  • createDataCell

  • getElementText

  • getFieldValue

  • setDelimiter

  • setTerminator

Methods
Constructor

The constructor function basically just sets up the DataCell Array and initializes the other class Attributes.

Logic for the RecordHandler Constructor Method
Arguments:
  DOM File Description Document

Create DataCell Array
Create Record Buffer
Set Record Buffer Length to zero
Highest Cell <- -1
Save Passed File Description Document
createDataCell

This method creates the DataCell derived class corresponding to the passed data type, increments Highest Cell, and loads the new cell into the next entry in the CellArray.

Logic for the RecordHandler createDataCell Method
Arguments:
  Integer Field Number
  DOM Element Field Grammar

Returns:
  Pointer to DataCell Object
Cell DataType <- call Field Grammar's getAttribute on DataType
For each supported data type do IF block as follows
IF (Cell DataType = Constant Code for the type)
  Create new DataCell derived class for the type
ENDIF
IF new DataCell is null
  Return error
ENDIF
Increment HighestCell;
CellArray[HighestCell] <- New Cell
return new DataCell

A brief word is in order regarding how we fill and use the DataCell Array. There are basically two approaches to how we could use the Array: (1) we could use an indexed approach in which the field number is used as a direct index into the DataCell Array, or (2) we could just fill the array from the bottom up.

In the indexed approach, we can use the field number (which we determine from parsing or from the grammar) as an index into the DataCell Array, loading the nth Array entry with the contents of the nth field. This approach has a certain elegance and makes some of the algorithms a bit simpler than the other approach. In the most efficient implementation we could use an indexed approach not only for accessing the DataCell Array but also for accessing the Grammar Elements that describe the fields in the legacy record format.

The other approach is to just fill the Array from the bottom up, starting at index zero, without trying to maintain any correspondence between the Array index and the field number. This approach means that when we need the field number we have to retrieve it from the DataCell object in the Array, which means a bit more code than the other approach.

Due to the fact the EDI has subfields (that is, subelements), with a single DataCell Array we can't use the indexed approach for all our formats. This requires a one-for-one, static correspondence between field (and subfield) numbers and Array entries. To implement it we would have to set up a secondary Array for each EDI data element that has subfields, with the associated pointers, and so on. This starts to get more complicated and clever than it is worth. So, since in pragmatic terms we are limited to the bottom up, sequential loading approach for EDI I'm going to use it for all the legacy formats. One consistent approach is easier to code and maintain in the long run than two or more.

NOTE Adding New Data Types

If you decide to add a new data type to the code, in addition to creating the derived DataCell class you need to edit this routine. Follow the comments in the Java or C++ code to add an IF test and create the DataCell derived class of your new type.

getElementText

This simple utility method gets the Text Node associated with the source Element, returning the Text Node's value. If an Element has only one Text Node as its child, and we can be sure there is content in the Node, we can probably do this with one line of Java or C++ code. However, we can't always be sure of this since some Elements, such as those with the schema language built-in data type of string, may have Comment Nodes as children. I'm providing a utility method with enough code in it to avoid the more common exceptions, such as an empty Element.

Logic for the RecordHandler getElementText Method
Arguments:
  DOM Element Source Element

Returns:
  String Text Content

Child Node <- get firstChild of Source Element
DO while Child Node != null and Child Node Type != Text Node
  Child Node <- Child Node's nextSibling
ENDO
IF Child Node == null
  Return null
ENDIF
Return Child Node's getNodeValue
getFieldValue

This method searches the DataCell Array for the cell corresponding to the passed field number and returns the contents of the field when found. It throws an exception or returns an error status if the requested field is not found in the Array.

Logic for the RecordHandler getFieldValue Method
Arguments:
  Integer Field Number

Returns:
  String Field Contents

DO for all Cells in DataCell Array from 0 through Highest Cell
  IF Field Number = CellArray's getFieldNumber
    Return CellArray's getField
  ENDIF
ENDDO
Return error
setDelimiter

Although this routine involves only a few lines of code, the functions are performed frequently.

Logic for the RecordHandler setDelimiter Method
Arguments:
  String Delimiter

Returns:
  Byte Delimiter

IF length of Delimiter = 1
  return Delimiter as byte
ENDIF
return Delimiter converted to byte from hex string
setTerminator

This method sets the Record Terminators used when reading and writing variable length records. It accepts a Record Terminator string as a U to set UNIX-style line feed (x0A) record termination, W for Windows-style carriage return and line feed pair (x0D x0A), another literal character, or a two-character hexadecimal value converted to a byte. We don't need to consider any other cases in the code because schema validation ensures we'll get only these values.

Logic for the RecordHandler setTerminator Method
Arguments:
  String Terminator

Returns:
  Nothing

IF length of Terminator = 1
  DO CASE of Terminator
    'U':
      Record Terminator1 = line feed
      BREAK:
    'W':
      Record Terminator1 = carriage return
      Record Terminator2 = line feed
      BREAK:
    other:
      Record Terminator1 = Terminator
  ENDDO
ELSE
  Record Terminator1 = Terminator converted from hex
ENDIF

RecordReader Base Class (Extends RecordHandler)

Overview

This class provides basic support for reading records from non-XML-formatted files and converting them to XML. Many of the methods are used in the derived classes, though some are overridden. We will add methods as we build the various utilities.

Attributes:

  • Input Stream

  • DOM Document Output Document

Methods:

  • Constructor

  • getRecordType (virtual)

  • parseRecord (virtual)

  • readRecord (virtual)

  • readRecordVariableLength

  • setInputStream

  • setOutputDocument

  • toXMLType

  • writeRecord

Methods
Constructor

The RecordReader has two very basic constructor functions. The first takes a single argument of file description document name and doesn't do anything but call the base RecordHandler constructor. The second version also saves the Input Stream, which is the second argument to the constructor.

readRecordVariableLength

This method reads a physical record, as a variable length record, from the input file. A bit of discussion on the approach is in order. There are a few options for reading variable length records from input files.

  • Native Java or C++ readLine method: This approach is suitable for many purposes and is very simple to implement. However, it won't work for files that have a terminator other than a line feed or carriage return and line feed pair, and I have been bitten by it a few times. In cases where a Windows file has line feeds embedded in the record in addition to the carriage return and line feed terminators, the records may be split when they shouldn't be. The other two approaches don't have this problem because we look for the terminators ourselves.

  • Block read with position index: This approach involves reading a fixed length buffer from the input file and keeping an index of the current position in the buffer. When a new record is requested, we search forward in the buffer from the current position until we encounter the terminator(s), then copy that substring to the returned input record. We then set the current position index to the next position beyond the returned record. This method is very efficient, but buffer and index management can get a bit complicated when logical records span physical blocks, which is usually the case.

  • Single character read: This approach involves reading one character at a time from the input stream, building the returned input record until we encounter the terminator(s). This is not quite as simple as readLine, but it is simpler than the block read. The main disadvantage is that the single character read is not as efficient as the other methods. However, we can ease the pain quite a bit by doing buffered reads. The Java API gives us a way to do this nicely. For now, the C++ implementation uses the default buffering provided by Visual C++. This can be modified if performance requires it by using the filebuf class and the ifstream setbuf method. Buffering still doesn't help us any with the stack and call overhead of a method call to get each byte from the input file. However, since we're going for KISS over performance, this is the route we'll take.

The code for the readRecordVariableLength method, though a bit tedious, is fairly straightforward. The pseudocode below gives the general idea. The Java and C++ implementations vary a bit in the details due to the language differences.

Logic for the RecordReader readRecordVariableLength Method
Arguments:
  None

Returns:
  Integer Record Length - Returns 0 at end of file

Clear Record Buffer
Record Buffer Length <- 0
IF Record Terminator2 = 0
  DO until Record Terminator1 is read from file
    Input Byte <- get next byte from Input Stream (language
        dependent)
    Append Byte to Record Buffer
    Increment Record Buffer Length
  ENDDO
ELSE
  DO until last two characters in Record Buffer are
      Record Terminator1 and Record Terminator2
    Input Byte <- get next byte from Input Stream (language
        dependent)
    Append Byte to Record Buffer
    Increment Record Buffer Length
  ENDDO
  Clear last two bytes in Record Buffer
  Subtract 2 from Record Buffer Length
ENDIF
Return Record Buffer Length, or -1 if end of file
setInputStream

This method sets the class's Input Stream attribute to the passed input stream. We add this utility method to enable future enhancements such as processing all the files in an input directory rather than just a single input file.

setOutputDocument

This method sets the class's Output Document attribute to the passed value.

toXMLType

This method loops through the active entries in the DataCell Array and calls the toXML method of each.

Logic for the RecordReader toXMLType Method
Arguments:
  None

Returns:
  Status or throws exception

DO for all Cells in DataCell Array from 0 through Highest Cell
  Call DataCell Array entry's toXML method
ENDDO
writeRecord

This method writes a parent record Element and child Field Elements with contents from the DataCell Array to the Output Document. We're including it in the base RecordReader class because both the CSVRecordReader and FlatRecord Reader use it.

Logic for the RecordReader writeRecord Method
Arguments:
  DOM Element Output Document Parent Element
  DOM Element Record Grammar Element

Returns:
  Status or throws exception

Element Name <- Call Grammar Element's getAttribute for
    "ElementName"
Record Element <- call Output Document's createElement
Parent Element <- call Parent's appendChild to append Record
  Element
DO for all DataCells in array up through Highest Cell
  Call toElement on DataCell to create Element, load the text,
      and attach it to the parent
  Clear Cell Array Entry
ENDDO
Highest Cell <- -1
Derived Class Methods

All the derived classes implement the following methods, which are declared as virtual in the base class. These are the areas in which the processing differs the most among the various derived classes.

getRecordType

This method returns the record tag from the input record.

Logic for the RecordReader getRecordType Method
Arguments:
  None

Returns:
  String Record Tag
Get Record Tag field location information from File Description
    Document if required
Parse record to locate field with the record tag
Return field value
parseRecord

This method examines the input non-XML record and stores the field contents into DataCell objects of the appropriate derived class into the DataCell Array. Here is the general processing flow. The exact logic for each of the derived classes will, of course, vary since the record formats themselves are quite different.

Logic for the RecordReader parseRecord Method
Arguments:
  DOM Element Record Grammar

Returns:
  Status or throws exception

DO until end of input record
  Increment Field Number
  Get characteristics of next field by getting next Field
      Grammar child of Record Grammar Element
  Create DataCell derived class corresponding to field data type
  Copy field contents from Input Record to DataCell buffer
  Set Highest Cell to Field Number
ENDDO
readRecord

This is a convenience method that provides a general interface to each derived class's routines for reading physical records. It is called from the base SourceConverter Class's processGroup method.

RecordWriter Base Class (Extends RecordHandler)

Overview

This class provides basic support for writing records from XML documents to non-XML-formatted files. As with the RecordReader, many of the methods are used in the derived classes, though some are overridden. Again, we will add methods as we build the various utilities.

Attributes:

  • Output Stream

Methods:

  • Constructor

  • parseRecord

  • writeRecord (virtual)

Methods
Constructor

Like the constructor for the RecordReader, the RecordWriter's constructor function doesn't do very much, either. The basic version of the constructor takes a single argument of the file description document, then passes it to the base class RecordHandler constructor. The other version takes a second argument of the Output Stream and saves it after calling the base class constructor.

parseRecord

This method retrieves the child field Element Nodes from a Record Element parent, storing the text contents in DataCell objects of the appropriate derived class in the DataCell Array. This base class method is used in both the CSVRecordWriter and FlatRecordWriter classes. A similar but somewhat more involved method is used in the EDIRecordWriter class.

Logic for the RecordWriter parseRecord Method
Arguments:
  DOM Element Record
  DOM Element Record Grammar

Returns:
  Void

Field Grammar <- Get first Grammar child Element from Record
    Grammar Element, skipping over non-Element nodes
Grammar Field Name <- call Field Grammar's getAttribute
    on "ElementName"
DO for all Field Elements that are children of Record Element
  ElementText <- call getElement Text on Field Element
  IF Element Text is empty
    Proceed to next field
  ENDIF
  Field Name <- Field Element's tagName attribute
  DO UNTIL Field Name == Grammar Field Name or Grammar Element
      is null
    Get next sibling Field Grammar Element, skipping over
        non-Element Nodes
    Grammar Field Name <- Field Element's tagName attribute
  ENDDO
  IF Grammar Element is null
    Return error
  ENDIF
  Field Number <- Get call Field Grammar's getAttribute
      on "Number"
  NewCell <- call createDataCell, passing Field Number
      and Grammar
  NewCell <- call NewCell's putField passing Element Text
ENDDO
Derived Class Method

Again, the following method is declared as virtual in the base class; specific methods are defined in the derived classes. Each has a method that follows the general logic described here.

writeRecord

This method writes the contents of the DataCell Array to the output legacy non-XML record.

Logic for the RecordWriter writeRecord Method
Arguments:
  None

Returns:
  Void

DO for all DataCells in Array
  Call fromXML to convert field from XML data type
  Call prepareOutput to handle justification, filling, etc.
  Copy cell contents to Record Buffer
  Clear Array entry for cell
ENDDO
Append appropriate terminators to Record Buffer
Call language's write routines to do physical write of
    Record Buffer

DataCell Base Class

Overview

This class is the base for all the classes we use to represent the various data types. With a few minor variations, for each specific data type that we support in a non-XML format we provide a derived class that handles converting that type to and from the corresponding Schema data type.

Attributes:

  • Byte or Character Array Cell Buffer

  • DOM Element Field Grammar

  • Integer Buffer Length

  • Integer Field Number

  • Integer Subfield Number

The Subfield Number attribute is used only for EDI, but we include it in the base class to make processing and class derivation a bit easier. This is a pragmatic approach, not a purist one!

Methods:

The most important methods are listed here. Other various minor methods set and get single class attributes or Attributes of the Grammar Element.

  • Constructor

  • fromXML (overridden in derived classes)

  • getField

  • prepareOutput (overridden in derived classes)

  • putByte

  • putField

  • toElement

  • toXML (overridden in derived classes)

  • trim (C++ only)

Methods
Constructor

The constructor method is very basic. It takes two arguments: the Field Number and the Field Grammar Element. It stores these passed arguments in the appropriate attributes of the class and initializes the other class attributes.

getField

This method has no arguments and returns the contents of the Cell Buffer and the value of the Buffer Length.

putByte

This method has a single argument of a byte of data. It appends the passed byte to the Cell Buffer.

putField

This method takes two arguments: a byte or char array of the contents of a legacy format field and the length of the field passed as an integer. It stores the passed field contents in the Cell Buffer.

toElement

This method creates a new Element using the passed name, adds text from the Field Contents, and attaches the Element to the passed parent Element. If the Cell Buffer contents are empty a new Element is not created.

Logic for the DataCell toElement Method
Arguments:
  DOM Element Parent
  DOM Document Output Document

Returns:
  Void

IF Buffer Length = 0
  Return
ENDIF
Element Name <- Get "ElementName" Attribute from Grammar
    Element
New Element <- call Output Document's createElement method
Parent <- append New Element to Parent
Text Node <- call Output Document's createTextNode method,
    Passing Cell Buffer contents
New Element <- append Text Node to New Element
trim

This is a C++ function only since Java provides a trim method for its String class. It trims leading and trailing whitespace (all characters with an integer value less than or equal to the space character) from the Cell Buffer.

Derived Class Methods

As with the RecordHandler classes, these methods are overridden in the derived classes. Base versions with minimal functionality are provided here. None of these methods takes any arguments, and all either return a completion status or throw an exception.

fromXML

This method converts the buffer contents from the schema language data type format to the non-XML format. The base class version returns with no action and is appropriate for data types that require no conversion.

prepareOutput

After format conversion with fromXML, this routine performs additional formatting tasks that are required before the data can be written to the output. Specific operations are dependent on the data type. Typical tasks include space filling to the minimum field length, adding or removing leading zeroes, and truncating.

toXML

This method converts the buffer contents from the non-XML format to the format required by the corresponding schema language data type. This base class version returns with no action. It is appropriate for data types such as alphanumeric text or decimal numbers that require no conversion.


Java Implementation

In both our Java and C++ implementations we are going to start building a common library. This is more evident in Java, where building the Java package can be seen in the source code. All that happens in C++ is that we start building a common object library. We'll use net.babelblaster as the prefix for our packages.

The Java implementation is composed of the following source files, which correspond to the base classes described previously. We also add the DOM Validator class we developed in Chapter 5, our standard SAXErrorHandler, a Babel Blaster exception class, and a class for text constants.

  • DataCell.java: in package net.babelblaster.cell

  • Converter.java: in package net.babelblaster.converter

  • SourceConverter.java: in package net.babelblaster.converter

  • TargetConverter.java: in package net.babelblaster.converter

  • RecordHandler.java: in package net.babelblaster.record

  • RecordReader.java: in package net.babelblaster.record

  • RecordWriter.java: in package net.babelblaster.record

  • DOMValidator.java: in package net.babelblaster.utility

  • SAXErrorHandler.java: in package net.babelblaster.utility

  • BlasterException.java: in package net.babelblaster.utility

  • TextConstants.java: in package net.babelblaster.utility

As I said earlier, we're not going to look at the Java or C++ code in detail in the remainder of the book unless there's something really significant to discuss. I do want to note a few things though.

All our data is handled as bytes and byte arrays rather than strings or string buffers. For example, the Cell Buffer in the DataCell class is implemented as a byte array. In addition, all the I/O operations deal with bytes. It might be simpler in some cases to handle these as strings, but this strategy gives us future flexibility for supporting data encodings such as EBCDIC that might not be processed like string data. Some of the derived classes may convert data to strings or string buffers as part of the conversion to or from XML. We'll do this only after we already have the data in a string-compatible format, and the native storage format will remain bytes. One effect of this approach is that as a convenience all our DataCell putField and getField operations have byte array and string versions (the latter is a toString method). In addition to the default constructor we also have versions that allow the field contents to be initialized, passing them either as a byte array or string.

The readRecordVariableLength method in RecordReader is implemented with a BufferedInputStream. The only really awkward part about the code is the fact that the single character read method returns an integer instead of a byte. We have to use the Integer class to convert to a byte before we append the byte to the input record.

Per the previous comments, we're creating a TextConstants class where we'll declare the text commonly used in our programs. Some of the main routines will have unique text corresponding to the particular program, but wherever practical we'll declare the text here.


C++ Implementation

The following C++ source and header files implement the base classes described in the chapter. As with the Java implementation, we'll also include our standard error handling routines as well as some standard header files.

  • Converter.cpp

  • Converter.h

  • SourceConverter.cpp

  • SourceConverter.h

  • TargetConverter.cpp

  • TargetConverter.h

  • DataCell.cpp

  • DataCell.h

  • RecordHandler.cpp

  • RecordHandler.h

  • RecordReader.cpp

  • RecordReader.h

  • RecordWriter.cpp

  • RecordWriter.h

  • handleCOMError.cpp

  • handleParseError.cpp

  • BlasterIncludes.h

  • BlasterPrototypes.h

  • TextConstants.h

A few words are in order about string handling beyond what I noted in Chapter 1. Since C++ handles bytes for I/O as character arrays, we won't have some of Java's complexities in converting to and from bytes and strings. However, C++ character arrays certainly have their own problems and peculiarities. In particular, the string functions don't always handle non-string data very well (for example, when it contains null characters). When there's a future possibility of dealing with non-string data, such as numeric data coded in one of the EBCDIC or native IBM data types, we'll use memcpy rather than string copy operations. We'll also do binary reads and writes of our data files rather than assume that the data is all character string data. Another difference between Java String and C++ character arrays is that in C++ we (conventionally) must allocate space for the array at compile time, hard-coding an array size. To avoid buffer overflows several of the string-based data types in the file description documents have maxLength facets imposed on them. When dealing with XML as the conversion source, I also check the sizes of strings that are returned from the DOM API methods before copying them to the C++ character arrays.

We'll store text constants in the TextConstants.h header file, serving the same function in the C++ environment as the TextConstants class in Java.


References

exolab. 2002. "Castor XML Schema Support." Available online at http://castor.exolab.org/xmlschema.html, accessed September 7, 2002.

Martin, John C. 1991. Introduction to Languages and the Theory of Computation, 2nd ed. Boston, MA: WCB/McGraw-Hill.

Merriam-Webster, Inc. 1971. Webster's Third New International Dictionary. Chicago: Encyclopaedia Britannica, Inc.

W3C (World Wide Web Consortium). 2000. Canonical XML Version 1.0. Available online at http://www.w3.org/TR/xml-c14n, accessed May 17, 2003.


Resources

Compilers—Principles, Techniques, and Tools, by Alfred V. Aho, Ravi Sethi, and Jeffrey D. Ullman (Reading, MA: Addison-Wesley, 1988). This is the classic textbook on compiler construction. It is known as the "dragon book," from the cover illustration of a knight doing battle with a dragon.

Unicode information can be found at http://www.unicode.org.

XML Schema Recommendation, by World Wide Web Consortium. Base URL http://www.w3.org/TR/.

Part 0: Primer, xmlschema-0.html

Part 1: Structures, xmlschema-1.html

Part 2: Datatypes, xmlschema-2.html


Chapter 7. Converting CSV Files to and from XML, Revisited

From Mayfield[*] to San Jose and back

[*] Mayfield, USA, was the home of the Cleaver family on the "Leave It to Beaver" show.

This chapter presents an evolution of the CSV utilities developed in Chapters 2 and 3. It builds on the generalized approach presented in Chapter 6 and adds functionality such as handling multiple logical documents in one physical CSV file. This chapter also presents the first instance of a more formal approach to grammar analysis.

This and the next two chapters present the evolution of our techniques for converting to and from XML and legacy file formats. Each chapter presents two utilities: one to convert the legacy format to XML and the other to convert it from XML. The chapters retain the overall organization presented earlier, compressed a bit to handle both types of conversion. Having presented the general DOM programming approaches in Chapters 2 and 3, we can now focus on overall functional and design aspects. In doing so we find that the two types of conversions (CSV files to and from XML) have many things in common. Hence, it makes sense to discuss them in the same chapter.

In all three chapters we will start with general user information, followed by design, and finally the implementation in Java and C++. Sections deal with converting to XML, converting from XML, and general considerations that apply to both situations.


CSV to XML: Functionality and Operation

Requirements

The general requirement for this utility is to convert a CSV file containing one or more logical documents to one or more XML instance documents, each representing a single logical document. This is a significant improvement over the utility presented in Chapter 3, which was restricted to handling a single logical document. In addition, many of the restrictions on both the CSV file format and the grammar of the resulting XML document are removed. Here's a summary of the required functionality.

  • Inputs: A CSV file in uniform Row/Column organization, that is, each row has the same column entries. Records are of variable length, and columns are separated by a specified terminator. Column contents may be delimited by a text delimiter character in the first and last characters of the column. The input CSV file may consist of more than one logical document. The second input is an XML file description document (as discussed in Chapter 6) describing the CSV file and the grammar of the XML document to be produced.

  • Processing: Each input CSV row is written to an Element whose name is derived from the file description document. Each column in the row is written to a child Element, with the name also derived from the file description document. Column content is converted to Schema data types as specified in the file description document. Empty columns do not create Elements in the output document. Processing breaks occur on a new logical document and optionally on a new trading partner.

  • Outputs: One or more XML instance documents, each in a single file. The root Element name is derived from the grammar in the file description document. The file name is formed by appending a three-digit sequence number to the root Element name and adding the extension .xml. If break on trading partner has been specified, the documents for each trading partner are placed in a separate subdirectory. The subdirectories are named according to the trading partner IDs in the Partner Break column.

It may be fairly obvious why we want to create a different XML document for each logical document in the input CSV file. However, it might be helpful to review why we might want to segregate the XML documents for different trading partners into different directories. Despite the proliferation of "standards" (remember the old line, "I love standards, there are so many to choose from"), it is still customary for different trading partners to require equivalent business data to be processed using different formats. Even when they choose, for example, the same version of a UBL Order, they may use it differently. Given that, we might want to apply different XSLT transformations for the documents we send to different trading partners. Even if the documents are exactly the same format (which is unlikely), we might still want to segregate instance documents by destination since each trading partner has a unique network destination and perhaps uses different file transfer protocols, security measures, and so on. Separate directories facilitate this segregation.

Running the Utility

This section provides instructions for running the revised CSV to XML Converter utility from the command line.

For Java:

java CSVToXML InputFile.CSV OutputDirectory FileDescription.XML

or

java CSVToXML -h

For C++ on Win32:

CSVToXML InputFile.CSV OutputDirectory FileDescription.XML

or

CSVToXML -h

Options follow the parameters except for the help option, which may be specified by itself.

Parameters:

  • First: File specification of the input CSV file (required). The specification may include the full or relative path name. If no path name is specified, the file is assumed to reside in the current working directory. The full file name must be specified, but there is no restriction on the extension name.

  • Second: Path specification of the output directory (required). The directory must exist. Either a relative or an absolute path name may be specified. The trailing directory separator character is optional. If no break on trading partner is specified, all the created XML files are placed in this directory. If break on trading partner has been specified, then a subdirectory for each trading partner is created beneath this directory.

  • Third: File specification of the File Description XML instance document (required). If no path name is specified, the file is assumed to reside in the current working directory. The full file name must be specified, but there is no restriction on the extension name.

Options:

  • -v (Validate): Validate the created XML documents before writing them to disk. The documents are validated against the schema specified in the file description document.

  • -h (Help): Display a help message and exit without further processing.

Restrictions:

Unless otherwise noted, all numeric limits may be modified by changing parameters in the program source and appropriate type definitions in the file description document schemas.

  • Column delimiter: Any character, not just a comma, may be specified as the column delimiter. However, the selected column delimiter may not appear in a column's contents unless the column's contents are delimited by the specified text delimiter.

  • Text delimiter: Any character, not just a quotation mark, may be specified as the text delimiter. When used the text delimiter must be the first and last characters in the column. In all other cases it is treated as column content. If a character other than the column delimiter appears after the second text delimiter in a column, a parsing error occurs.

  • A column may have a maximum of 1,023 characters.

  • A row may be no longer than 16,383 characters.

  • A maximum of 99 columns per row is supported.

  • There is no absolute limit on the number of rows; the number is only practically limited by system memory.

  • Each column must be assigned a unique Element name.

  • Column Grammar Elements must be specified in ascending order by column number.

  • Element names are limited to 127 characters.

  • Path lengths for complete file specifications are limited to 127 characters.

  • Schema location URIs are limited to 127 characters.

  • A maximum of 999 output XML documents from an input CSV file is supported.

  • A maximum of 100 different trading partner destinations in an input CSV file is supported.

Sample Input and Output: Invoice

For our sample CSV file in this chapter we use a document commonly exported in electronic commerce situations from small businesses to larger customers that mandate that documents be exchanged electronically. We use a simple invoice. In this example, each row exported from a desktop bookkeeping and order management system has the same format. Each row represents one item to be invoiced, and we want all the rows for a specific invoice number to be grouped together into one invoice document. In addition, we want the invoices destined for different trading partners to be separated from each other, based on the customer ID column.

Our sample business is Big Daddy's Gourmet Cocoa, a specialty manufacturer of instant hot chocolate beverages that sells primarily to grocery chains. All the columns in our example invoice (Table 7.1) are required except for the second line of the Ship to Street address (column 8) and the Ship to Country (column 12), which is assumed to be the same as Big Daddy's country (in this case, the United States).

Shown below is the sample input invoice that corresponds to the CSV format in Table 7.1. (Note: Line breaks and indentation have been inserted for readability.)

Sample Input CSV File (Invoices.csv)
BQ003,2002041,11/12/2002,AZ999345,12/12/2002,
 "Yazoo Grocers - NE Distribution Center",
 "12 Industrial Parkway, NW","",Portland,ME,04101,,
 HCVAN,12,2.59,"Instant Hot Cocoa Mix - Vanilla flavor",31.08
BQ003,2002041,11/12/2002,AZ999345,12/12/2002,
 "Yazoo Grocers - NE Distribution Center",
 "12 Industrial Parkway, NW","",Portland,ME,04101,,
 HCMIN,24,2.53,"Instant Hot Cocoa Mix - Mint flavor",60.72
BQ003,2002042,11/12/2002,AW999346,12/12/2002,
 "Yazoo Grocers - SE Distribution Center",
 "Dock 37","3975 Hwy 75",Atoka,OK,74525,,
 HCVAN,36,2.59,"Instant Hot Cocoa Mix - Vanilla flavor",93.24
BQ003,2002042,11/12/2002,AW999346,12/12/2002,
 "Yazoo Grocers - SE Distribution Center",
 "Dock 37","3975 Hwy 75",Atoka,OK,74525,,
 HCMIN,72,2.53,"Instant Hot Cocoa Mix - Mint flavor",182.16
AY001,2002043,11/12/2002,2002-0967,12/12/2002,
 "Corner Drug and Sundries",
 "14 Main Street","",Wichita,KS,67201,,
 HCVAN,24,2.59,"Instant Hot Cocoa Mix - Vanilla flavor",62.16
BR095,2002044,11/12/2002,4397-0498,12/12/2002,
 "Big Box Discounters - Store # 97",
 "37 MegaMall","",Azusa,CA,91702,,
 HCMIN,120,2.53,"Instant Hot Cocoa Mix - Mint flavor",303.60
BR095,2002044,11/12/2002,4397-0498,12/12/2002,
 "Big Box Discounters - Store # 97",
 "37 MegaMall","",Azusa,CA,91702,,
 HCVAN,360,2.59,"Instant Hot Cocoa Mix - Vanilla flavor",932.40
BR095,2002044,11/12/2002,4397-0498,12/12/2002,
 "Big Box Discounters - Store # 97",
 "37 MegaMall","",Azusa,CA,91702,,
 HCDUC,240,2.59,"Instant Hot Cocoa Mix - Dutch Chocolate flavor",621.60
BR095,2002045,11/12/2002,4345-0498,12/12/2002,
 "Big Box Discounters - Store # 45",
 "45 Highway 76","",Branson,MO,65615,,
 HCMIN,72,2.53,"Instant Hot Cocoa Mix - Mint flavor",182.16
BR095,2002045,11/12/2002,4345-0498,12/12/2002,
 "Big Box Discounters - Store # 45",
 "45 Highway 76","",Branson,MO,65615,,
 HCDUC,96,2.59,"Instant Hot Cocoa Mix - Dutch Chocolate flavor",248.64
DQ349,2002046,11/12/2002,987-43671,12/12/2002,
 "Maple Leaf Grocers - DC #1",
 "987 Yorkland Blvd","",Willowdale,ON,M2J 4Y8,CAN,
 HCMOC,360,2.69,"Instant Hot Cocoa Mix - Mocha flavor",968.40

Table 7.1. Logical Layout for the Invoice

Column Number

Column Name

Data Type

Description

1

Customer Number

Alphanumeric

Identifier we have assigned to the customer in our system

2

Invoice Number

Alphanumeric

System-assigned invoice number

3

Invoice Date

Date

Date of invoice, formatted MM/DD/YYYY

4

PO Number

Alphanumeric

Customer purchase order number

5

Due Date

Date

Date that the invoice amount is due for payment

6

Ship to Name

Alphanumeric

Name of the receiving location for the shipped order

7

Ship to Street 1

Alphanumeric

First address line of the receiving location

8

Ship to Street 2

Alphanumeric

Second address line of the receiving location

9

Ship to City

Alphanumeric

City of the receiving location

10

Ship to State or Province

Alphanumeric

State or province of the receiving location

11

Ship to Postal Code

Alphanumeric

Postal code of the receiving location

12

Ship to Country

Alphanumeric

Country of the receiving location

13

Item ID

Alphanumeric

Our identifier for the ordered item

14

Item Invoiced Quantity

Decimal number

Number of units shipped

15

Item Unit Price

Decimal number

Unit price in U.S. dollars

16

Item Description

Alphanumeric

Description of the ordered item

17

Extended Amount Due

Decimal number

Total amount due for the invoiced item, in U.S. dollars (unit price multiplied by the quantity invoiced)

Successful processing should produce XML documents that look like the following. (I show only the first three.)

Sample Output Documents
<?xml version="1.0" encoding="UTF-8"?>
<Invoice xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:noNamespaceSchemaLocation="CSVInvoice.xsd">
  <InvoiceLine>
    <CustomerNumber>BQ003</CustomerNumber>
    <InvoiceNumber>2002041</InvoiceNumber>
    <InvoiceDate>2002-11-12</InvoiceDate>
    <PONumber>AZ999345</PONumber>
    <DueDate>2002-12-12</DueDate>
    <ShipToName>Yazoo Grocers - NE Distribution Center
        </ShipToName>
    <ShipToStreet1>12 Industrial Parkway, NW</ShipToStreet1>
    <ShipToCity>Portland</ShipToCity>
    <ShipToStateOrProvince>ME</ShipToStateOrProvince>
    <ShipToPostalCode>04101</ShipToPostalCode>
    <ItemID>HCVAN</ItemID>
    <ItemQuantity>12</ItemQuantity>
    <UnitPrice>2.59</UnitPrice>
    <ItemDescription>Instant Hot Cocoa Mix - Vanilla flavor
        </ItemDescription>
    <ExtendedPrice>31.08</ExtendedPrice>
  </InvoiceLine>
  <InvoiceLine>
    <CustomerNumber>BQ003</CustomerNumber>
    <InvoiceNumber>2002041</InvoiceNumber>
    <InvoiceDate>2002-11-12</InvoiceDate>
    <PONumber>AZ999345</PONumber>
    <DueDate>2002-12-12</DueDate>
    <ShipToName>Yazoo Grocers - NE Distribution Center
        </ShipToName>
    <ShipToStreet1>12 Industrial Parkway, NW</ShipToStreet1>
    <ShipToCity>Portland</ShipToCity>
    <ShipToStateOrProvince>ME</ShipToStateOrProvince>
    <ShipToPostalCode>04101</ShipToPostalCode>
    <ItemID>HCMIN</ItemID>
    <ItemQuantity>24</ItemQuantity>
    <UnitPrice>2.53</UnitPrice>
    <ItemDescription>Instant Hot Cocoa Mix - Mint flavor
        </ItemDescription>
    <ExtendedPrice>60.72</ExtendedPrice>
  </InvoiceLine>
</Invoice>

<?xml version="1.0" encoding="UTF-8"?>
<Invoice xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:noNamespaceSchemaLocation="CSVInvoice.xsd">
  <InvoiceLine>
    <CustomerNumber>BQ003</CustomerNumber>
    <InvoiceNumber>2002042</InvoiceNumber>
    <InvoiceDate>2002-11-12</InvoiceDate>
    <PONumber>AW999346</PONumber>
    <DueDate>2002-12-12</DueDate>
    <ShipToName>Yazoo Grocers - SE Distribution Center
        </ShipToName>
    <ShipToStreet1>Dock 37</ShipToStreet1>
    <ShipToStreet2>3975 Hwy 75</ShipToStreet2>
    <ShipToCity>Atoka</ShipToCity>
    <ShipToStateOrProvince>OK</ShipToStateOrProvince>
    <ShipToPostalCode>74525</ShipToPostalCode>
    <ItemID>HCVAN</ItemID>
    <ItemQuantity>36</ItemQuantity>
    <UnitPrice>2.59</UnitPrice>
    <ItemDescription>Instant Hot Cocoa Mix - Vanilla flavor
        </ItemDescription>
    <ExtendedPrice>93.24</ExtendedPrice>
  </InvoiceLine>
  <InvoiceLine>
    <CustomerNumber>BQ003</CustomerNumber>
    <InvoiceNumber>2002042</InvoiceNumber>
    <InvoiceDate>2002-11-12</InvoiceDate>
    <PONumber>AW999346</PONumber>
    <DueDate>2002-12-12</DueDate>
    <ShipToName>Yazoo Grocers - SE Distribution Center
        </ShipToName>
    <ShipToStreet1>Dock 37</ShipToStreet1>
    <ShipToStreet2>3975 Hwy 75</ShipToStreet2>
    <ShipToCity>Atoka</ShipToCity>
    <ShipToStateOrProvince>OK</ShipToStateOrProvince>
    <ShipToPostalCode>74525</ShipToPostalCode>
    <ItemID>HCMIN</ItemID>
    <ItemQuantity>72</ItemQuantity>
    <UnitPrice>2.53</UnitPrice>
    <ItemDescription>Instant Hot Cocoa Mix - Mint flavor
        </ItemDescription>
    <ExtendedPrice>182.16</ExtendedPrice>
  </InvoiceLine>
</Invoice>

<?xml version="1.0" encoding="UTF-8"?>
<Invoice xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:noNamespaceSchemaLocation="CSVInvoice.xsd">
  <InvoiceLine>
    <CustomerNumber>AY001</CustomerNumber>
    <InvoiceNumber>2002043</InvoiceNumber>
    <InvoiceDate>2002-11-12</InvoiceDate>
    <PONumber>2002-0967</PONumber>
    <DueDate>2002-12-12</DueDate>
    <ShipToName>Corner Drug and Sundries</ShipToName>
    <ShipToStreet1>14 Main Street</ShipToStreet1>
    <ShipToCity>Wichita</ShipToCity>
    <ShipToStateOrProvince>KS</ShipToStateOrProvince>
    <ShipToPostalCode>67201</ShipToPostalCode>
    <ItemID>HCVAN</ItemID>
    <ItemQuantity>24</ItemQuantity>
    <UnitPrice>2.59</UnitPrice>
    <ItemDescription>Instant Hot Cocoa Mix - Vanilla flavor
        </ItemDescription>
    <ExtendedPrice>62.16</ExtendedPrice>
  </InvoiceLine>
</Invoice>


XML to CSV: Functionality and Operation

Requirements

The general requirement for this utility is to convert one or more XML instance documents, each representing a single logical document, into a single CSV file. As with the CSV to XML utility, this is a significant improvement over the utility presented in Chapter 2. Again, many of the restrictions on both the CSV file format and the grammar of the resulting XML document are removed. Here's a summary of the required functionality.

  • Inputs: A directory containing one or more XML instance documents, each in a single file. All have the same grammar. The second input is an XML file description document (as discussed in Chapter 6) describing the XML documents and the CSV file to be produced.

  • Processing: Each input Element corresponding to a row (as described in the file description document) is written to a record in the CSV output file. Each of its child Elements corresponding to columns are written to the appropriate column numbers. Column delimiter characters are inserted, and data types are converted according to the column descriptions in the file description document.

  • Output: A CSV file in row and column organization.

Running the Utility

This section provides instructions for running the revised XML to CSV Converter utility from the command line.

For Java:

java XMLToCSV InputDirectory OutputFile.csv FileDescription.XML

or

java XMLToCSV -h

For C++ on Win32:

XMLToCSV InputDirectory OutputFile.csv FileDescription.XML

or

XMLToCSV -h

Options follow the parameters except for the help option, which may be specified by itself.

Parameters:

  • First: Path specification of the input directory (required). Either a relative or an absolute path name may be specified. The trailing directory separator character is optional.

  • Second: File specification of the output CSV file (required). The specification may include the full or relative path name. If no path name is specified, the file will be created in the current working directory. An extension may be specified, but if not specified the file will be created without one. If a file with the specified name already exists, it will be overwritten.

  • Third: File specification of the File Description XML instance document (required). If no path name is specified, the file is assumed to reside in the current working directory. The full file name must be specified, but there is no restriction on the extension name.

Options:

  • -v (Validate): Validate the XML documents before processing. The documents are validated against the schema referenced in the document root Element.

  • -h (Help): Display a help message and exit without further processing.

Restrictions:

The restrictions are the same as those specified for the CSV to XML utility, plus the following.

  • Field Elements in the input XML document must be ordered sequentially according to their column positions in the output CSV row.

Sample Input and Output: Purchase Order

Following through with our simple procurement example, we'll use a purchase order as the sample document. (See Table 7.2 for the layout. Note that the Description column in this table includes a required or optional designation. This is something we'll be concerned about validating when importing into the application but not when we export from it.) We'll have several XML documents representing purchase orders from different customers, each ordering the gourmet hot chocolate mixes. We want to convert the XML documents to CSV format, in a single file, so that we can import them into our desktop bookkeeping and order management system.

Here are three purchase orders that follow this logical organization.

Sample PurchaseOrders01.xml
<?xml version="1.0" encoding="UTF-8"?>
<PurchaseOrder>
  <POLine>
    <CustomerNumber>BQ003</CustomerNumber>
    <PONumber>AZ999345</PONumber>
    <PODate>2002-11-12</PODate>
    <RequestedDeliveryDate>2002-11-15</RequestedDeliveryDate>
    <ShipToName>
      Yazoo Grocers - NE Distribution Center
    </ShipToName>
    <ShipToStreet1>12 Industrial Parkway, NW</ShipToStreet1>
    <ShipToCity>Portland</ShipToCity>
    <ShipToState>ME</ShipToState>
    <ShipToPostalCode>04101</ShipToPostalCode>
    <ItemID>HCVAN</ItemID>
    <OrderedQty>12</OrderedQty>
    <UnitPrice>2.59</UnitPrice>
    <ItemDescription>
      Instant Hot Cocoa Mix - Vanilla flavor
    </ItemDescription>
  </POLine>
  <POLine>
    <CustomerNumber>BQ003</CustomerNumber>
    <PONumber>AZ999345</PONumber>
    <PODate>2002-11-12</PODate>
    <RequestedDeliveryDate>2002-11-15</RequestedDeliveryDate>
    <ShipToName>
      Yazoo Grocers - NE Distribution Center
    </ShipToName>
    <ShipToStreet1>12 Industrial Parkway, NW</ShipToStreet1>
    <ShipToCity>Portland</ShipToCity>
    <ShipToState>ME</ShipToState>
    <ShipToPostalCode>04101</ShipToPostalCode>
    <ItemID>HCMIN</ItemID>
    <OrderedQty>24</OrderedQty>
    <UnitPrice>2.53</UnitPrice>
    <ItemDescription>
      Instant Hot Cocoa Mix - Mint flavor
    </ItemDescription>
  </POLine>
</PurchaseOrder>

Table 7.2. Logical Layout for the Purchase Order

Column Number

Column Name

Data Type

Description

1

Customer Number

Alphanumeric

Required; identifier assigned to the customer in our system

2

PO Number

Alphanumeric

Required; customer purchase order number

3

PO Date

Date

Required; date that the purchase order was issued

4

Requested Delivery Date

Date

Optional; date on which delivery of the ordered items is requested

5

Ship to Name

Alphanumeric (delimited)

Required; name of the receiving location for shipped order

6

Ship to Street 1

Alphanumeric (delimited)

Required; first address line of the receiving location

7

Ship to Street 2

Alphanumeric (delimited)

Optional; second address line of the receiving location

8

Ship to City

Alphanumeric

Required; city of the receiving location

9

Ship to State or Province

Alphanumeric

Required; state or province of the receiving location

10

Ship to Postal Code

Alphanumeric

Required; postal code of the receiving location

11

Ship to Country

Alphanumeric

Optional; country of the receiving location

12

Item ID

Alphanumeric

Required; identifier for the ordered item

13

Item Quantity

Decimal number

Required; number of units ordered

14

Item Unit Price

Decimal number

Optional (since price may be determined by contract); unit price in U.S. dollars

15

Item Description

Alphanumeric (delimited)

Optional; description of the ordered item

Sample PurchaseOrders02.xml
<?xml version="1.0" encoding="UTF-8"?>
<PurchaseOrder>
  <POLine>
    <CustomerNumber>BQ003</CustomerNumber>
    <PONumber>AW999346</PONumber>
    <PODate>2002-11-12</PODate>
    <RequestedDeliveryDate>2002-11-15</RequestedDeliveryDate>
    <ShipToName>
      Yazoo Grocers - SE Distribution Center
    </ShipToName>
    <ShipToStreet1>Dock 37</ShipToStreet1>
    <ShipToStreet2>3975 Hwy 75</ShipToStreet2>
    <ShipToCity>Atoka</ShipToCity>
    <ShipToState>OK</ShipToState>
    <ShipToPostalCode>74525</ShipToPostalCode>
    <ItemID>HCVAN</ItemID>
    <OrderedQty>36</OrderedQty>
    <UnitPrice>2.59</UnitPrice>
    <ItemDescription>
      Instant Hot Cocoa Mix - Vanilla flavor
    </ItemDescription>
  </POLine>
  <POLine>
    <CustomerNumber>BQ003</CustomerNumber>
    <PONumber>AW999346</PONumber>
    <PODate>2002-11-12</PODate>
    <RequestedDeliveryDate>2002-11-15</RequestedDeliveryDate>
    <ShipToName>
      Yazoo Grocers - SE Distribution Center
    </ShipToName>
    <ShipToStreet1>Dock 37</ShipToStreet1>
    <ShipToStreet2>3975 Hwy 75</ShipToStreet2>
    <ShipToCity>Atoka</ShipToCity>
    <ShipToState>OK</ShipToState>
    <ShipToPostalCode>74525</ShipToPostalCode>
    <ItemID>HCMIN</ItemID>
    <OrderedQty>72</OrderedQty>
    <UnitPrice>2.53</UnitPrice>
    <ItemDescription>
      Instant Hot Cocoa Mix - Mint flavor
    </ItemDescription>
  </POLine>
</PurchaseOrder>
Sample PurchaseOrders03.xml
<?xml version="1.0" encoding="UTF-8"?>
<PurchaseOrder>
  <POLine>
    <CustomerNumber>AY001</CustomerNumber>
    <PONumber>2002-0967</PONumber>
    <PODate>2002-11-12</PODate>
    <RequestedDeliveryDate>2002-11-14</RequestedDeliveryDate>
    <ShipToName>Corner Drug and Sundries</ShipToName>
    <ShipToStreet1>14 Main Street</ShipToStreet1>
    <ShipToCity>Wichita</ShipToCity>
    <ShipToState>KS</ShipToState>
    <ShipToPostalCode>67201</ShipToPostalCode>
    <ItemID>HCVAN</ItemID>
    <OrderedQty>24</OrderedQty>
    <UnitPrice>2.59</UnitPrice>
    <ItemDescription>
      Instant Hot Cocoa Mix - Vanilla flavor
    </ItemDescription>
  </POLine>
</PurchaseOrder>

Successful processing of these three documents should produce a CSV file that looks like the one shown below. (Note: Line breaks and indentation have been inserted for readability.)

Sample Output CSV File
BQ003,AZ999345,11/12/2002,11/15/2002,
 "Yazoo Grocers - NE Distribution Center",
 "12 Industrial Parkway, NW",,"Portland",ME,04101,,
 HCVAN,12,2.59,"Instant Hot Cocoa Mix - Vanilla flavor"
BQ003,AZ999345,11/12/2002,11/15/2002,
 "Yazoo Grocers - NE Distribution Center",
 "12 Industrial Parkway, NW",,"Portland",ME,04101,,
 HCMIN,24,2.53,"Instant Hot Cocoa Mix - Mint flavor"
BQ003,AW999346,11/12/2002,11/15/2002,
 "Yazoo Grocers - SE Distribution Center",
 "Dock 37","3975 Hwy 75","Atoka",OK,74525,,
 HCVAN,36,2.59,"Instant Hot Cocoa Mix - Vanilla flavor"
BQ003,AW999346,11/12/2002,11/15/2002,
 "Yazoo Grocers - SE Distribution Center",
 "Dock 37","3975 Hwy 75","Atoka",OK,74525,,
 HCMIN,72,2.53,"Instant Hot Cocoa Mix - Mint flavor"
AY001,2002-0967,11/12/2002,11/14/2002,
 "Corner Drug and Sundries",
 "14 Main Street",,"Wichita",KS,67201,,
 HCVAN,24,2.59,"Instant Hot Cocoa Mix - Vanilla flavor"


Describing the File Formats

As discussed in Chapter 6, we're going to use our own XML-based language to describe the formats of our non-XML files. Although the different formats will have many characteristics in common, we'll have a different language for each. In addition, because some characteristics, such as those dealing with XML output, are not relevant for both to and from XML conversions, we'll have a separate language (and schema) for each direction. In this section I describe the organization, Elements, Attributes, and usage of the file description document. In the design section we'll review the schemas for the file description documents as well as the sample invoice and purchase order.

The CSV file description documents have three major sections, each represented by an Element that is a direct child of the root Element.

  • PhysicalCharacteristics: the CSV file characteristics

  • XMLOutputCharacteristics: XML output characteristics, required only when converting to XML

  • Grammar: CSV file grammar

The language that describes conversion from XML is a subset of the one that describes conversion to XML. Given this, the file description documents and the schemas that specify them are very similar. We'll review them in the upcoming section on high-level design.

The following three subsections describe the major sections of the file description documents. Each major section is handled by an Element that is a child of the file description document's root Element, CSVFileDescription.

CSV Physical Characteristics

The CSV file's physical characteristics are described in the PhysicalCharacteristics Element. For CSV formats we need to specify the record terminator and the column and text delimiters.

Table 7.3 shows the child Elements of the PhysicalCharacteristics Element. All are required.

XML Output Characteristics

Characteristics governing the output XML documents are described in the XMLOutputCharacteristics Element. This Element is used only when converting from CSV files to XML.

Table 7.4 shows the child Elements of the XMLOutputCharacteristics Element.

Table 7.3. Child Elements of the PhysicalCharacteristics Element

Child Element

Attribute

Schema Language Data Type

Description

Allowable Values

RecordTerminator

value

union of U, W, and hexBinary

Specifies the physical record terminator as a UNIX-style line feed character, a Windows-style carriage return and line feed pair, or a hexadecimal value

U, W, or a two-character hexadecimal number from 00 through FF representing a single byte

ColumnDelimiter

value

union of a single token character and hexBinary

Column delimiter expressed as a literal character or a hexadecimal value

A single nonwhitespace character or a two-character hexadecimal number from 00 through FF representing a single byte

TextDelimiter

value

union of a single token character and hexBinary

Text delimiter expressed as a literal character or a hexadecimal value

A single nonwhitespace character or a two-character hexadecimal number from 00 through FF representing a single byte

CSV File Grammar

The data types of the columns and other characteristics of the CSV file (that is, the grammar) are described in the Grammar Element. We need to notice an important point, a new aspect of functionality that we are introducing beyond that provided in Chapter 3's basic utility. Because we are describing the grammar of the CSV file, we can also take the opportunity to specify the names of the Elements that we use in the corresponding XML representation. We are no longer restricted to the "Row" and "ColumnXX" Element names that we used in Chapters 2 and 3. While this does add minimal complexity to the processing, it allows us to assign semantically meaningful names to the Elements in the XML document.

Table 7.5 shows the Grammar Element and its children. All are required unless noted. Indentation in the Element column shows hierarchichal parent/child relationships. The Allowable Child Elements column lists the specific details of the hierarchy.

Table 7.4. Child Elements of the XMLOutputCharacteristics Element

Child Element

Attribute

Schema Data Type

Description

Allowable Values, Restrictions, or Comments

SchemaLocationURL

value

anyURI

URL of the schema file for the out-put document. Will be written as the value of the root Element's noNamespaceSchemaLocation Attribute.

Optional. If not specified the noNamespaceSchemaLocation Attribute will not be written. An error will occur if output validation is requested and this Element is not present.

DocumentBreakColumn

value

NonNegativeInteger

Number of the column that dictates the start of a new document when its content changes (for example, an invoice number in the second column).

Required. If a value of zero is specified, the entire CSV file will be written to a single XML document in the output directory.

PartnerBreakColumn

value

NonNegativeInteger

Number of the column that dictates a different trading partner when its content changes (for example, a customer number in the first column).

Required. If a value of zero is specified, all output documents will be created in the output directory instead of creating a separate subdirectory for each trading partner. If DocumentBreakColumn is zero, this Element is ignored.

Table 7.5. CSV Grammar Characteristics in the Grammar Element

Element

Allowable Child Element

Attribute

Schema Language Data Type

Description

Allowable Values, Restrictions, or Comments

Grammar

RowDescription

Describes the grammar of both the CSV file and the corresponding XML representation

The Grammar Element has only a single RowDescription child Element due to the restriction that all the rows in the CSV file have the same organization.

ElementName

NMTOKEN

Specifies the name of the document's root Element

When creating XML documents, the specified name is assigned to the document's root Element. When creating a CSV file, the input XML document's root Element must match this name.

RowDescription

ColumnDescription

Describes a row in a CSV file and the corresponding XML representation

At least one Column-Description child Element is required.

ElementName

NMTOKEN

Specifies the name of the Element representing a row

Maximum length reflects restriction on the length of Element names.

ColumnDescription

None

Describes a column in a CSV file and the corresponding XML representation

One ColumnDescription Element is required for each column in a CSV file row. All columns must be specified.

ElementName

NMTOKEN

Specifies the name of the Element representing a column

Maximum length reflects restriction on the length of Element names.

FieldNumber

positiveInteger

Specifies the number of a column, starting at one

Maximum value reflects restriction on the number of columns per row.

DataType

token

Specifies the data type of a column in the CSV file

The supported data types developed in this chapter are shown in Table 7.6. The Grammar data type code values are used.

DelimitText

boolean

Indicates (for XML to CSV conversion) whether or not to use the Text-Delimiter when writing the column

Optional

Table 7.6 shows the CSV file data types developed in this chapter. Other types will be added in later chapters. The Enhancements and Alternatives section at the end of this chapter describes how to add new data types. Table 7.6 shows the type of the data as it appears in the CSV file, the corresponding schema language data type as used in the XML representation, and comments about the data type.

Table 7.6. CSV File Data Types

CSV Data Type

Grammar Data Type Code

Schema Language Data Type

Comments

Alphanumeric

AN

string

When converting from CSV to XML, leading and trailing spaces, tabs, line feeds, and carriage returns are trimmed. (Actually, any character with an integer value less than or equal to a space character is trimmed.) All other white space within the string is preserved.

Real number

R

decimal

When converting from CSV to XML, leading zeroes and whitespace are trimmed. Leading minus signs are preserved, and leading plus signs are removed. The code currently supports only the period character (.) as the decimal point.

Date in MM/ DD/YYYY format

DMMsDDsYYYY

date

Month and day may be either one or two digits each.

Example File Description Documents

Here are the file description documents for the invoice and purchase order examples.

Sample InvoiceCSVSourceDescription.xml
<?xml version="1.0" encoding="UTF-8"?>
<CSVSourceFileDescription
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:noNamespaceSchemaLocation="CSVSourceFileDescription.xsd">
<PhysicalCharacteristics>
    <RecordTerminator value="W"/>
    <ColumnDelimiter value=","/>
    <TextDelimiter value="&quot;"/>
  </PhysicalCharacteristics>
  <XMLOutputCharacteristics>
    <DocumentBreakColumn value="2"/>
    <PartnerBreakColumn value="1"/>
    <SchemaLocationURL value="CSVInvoice.xsd"/>
  </XMLOutputCharacteristics>
  <Grammar ElementName="Invoice">
    <RowDescription ElementName="InvoiceLine">
      <ColumnDescription FieldNumber="1"
          ElementName="CustomerNumber" DataType="AN"/>
      <ColumnDescription FieldNumber="2"
          ElementName="InvoiceNumber" DataType="AN"/>
      <ColumnDescription FieldNumber="3"
          ElementName="InvoiceDate" DataType="DMMsDDsYYYY"/>
      <ColumnDescription FieldNumber="4"
          ElementName="PONumber" DataType="AN"/>
      <ColumnDescription FieldNumber="5"
          ElementName="DueDate" DataType="DMMsDDsYYYY"/>
      <ColumnDescription FieldNumber="6"
          ElementName="ShipToName" DataType="AN"/>
      <ColumnDescription FieldNumber="7"
          ElementName="ShipToStreet1" DataType="AN"/>
      <ColumnDescription FieldNumber="8"
          ElementName="ShipToStreet2" DataType="AN"/>
      <ColumnDescription FieldNumber="9"
          ElementName="ShipToCity" DataType="AN"/>
      <ColumnDescription FieldNumber="10"
          ElementName="ShipToStateOrProvince" DataType="AN"/>
      <ColumnDescription FieldNumber="11"
          ElementName="ShipToPostalCode" DataType="AN"/>
      <ColumnDescription FieldNumber="12"
          ElementName="ShipToCountry" DataType="AN"/>
      <ColumnDescription FieldNumber="13"
          ElementName="ItemID" DataType="AN"/>
      <ColumnDescription FieldNumber="14"
          ElementName="ItemQuantity" DataType="R"/>
      <ColumnDescription FieldNumber="15"
          ElementName="UnitPrice" DataType="R"/>
      <ColumnDescription FieldNumber="16"
          ElementName="ItemDescription" DataType="AN"/>
      <ColumnDescription FieldNumber="17"
          ElementName="ExtendedPrice" DataType="R"/>
    </RowDescription>
  </Grammar>
</CSVSourceFileDescription>
Sample PurchaseOrderCSVTargetDescription.xml
<?xml version="1.0" encoding="UTF-8"?>
<CSVTargetFileDescription
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:noNamespaceSchemaLocation="CSVTargetFileDescription.xsd">
  <PhysicalCharacteristics>
    <RecordTerminator value="U"/>
    <ColumnDelimiter value=","/>
    <TextDelimiter value="&quot;"/>
  </PhysicalCharacteristics>
  <Grammar ElementName="PurchaseOrder">
    <RowDescription ElementName="POLine">
      <ColumnDescription FieldNumber="1"
          ElementName="CustomerNumber" DataType="AN"/>
      <ColumnDescription FieldNumber="2"
          ElementName="PONumber" DataType="AN"/>
      <ColumnDescription FieldNumber="3"
          ElementName="PODate" DataType="DMMsDDsYYYY"/>
      <ColumnDescription FieldNumber="4"
          ElementName="RequestedDeliveryDate"
          DataType="DMMsDDsYYYY"/>
      <ColumnDescription FieldNumber="5"
          ElementName="ShipToName" DataType="AN"
          DelimitText="true"/>
      <ColumnDescription FieldNumber="6"
          ElementName="ShipToStreet1" DataType="AN"
          DelimitText="true"/>
      <ColumnDescription FieldNumber="7"
          ElementName="ShipToStreet2" DataType="AN"
          DelimitText="true"/>
      <ColumnDescription FieldNumber="8"
          ElementName="ShipToCity" DataType="AN"
          DelimitText="true"/>
      <ColumnDescription FieldNumber="9"
          ElementName="ShipToStateOrProvince" DataType="AN"/>
      <ColumnDescription FieldNumber="10"
          ElementName="ShipToPostalCode" DataType="AN"/>
      <ColumnDescription FieldNumber="11"
          ElementName="ShipToCountry" DataType="AN"/>
      <ColumnDescription FieldNumber="12"
          ElementName="ItemID" DataType="AN"/>
      <ColumnDescription FieldNumber="13"
          ElementName="OrderedQty" DataType="R"/>
      <ColumnDescription FieldNumber="14"
          ElementName="UnitPrice" DataType="R"/>
      <ColumnDescription FieldNumber="15"
          ElementName="ItemDescription" DataType="AN"
          DelimitText="true"/>
    </RowDescription>
  </Grammar>
</CSVTargetFileDescription>

Note that in these documents and in the associated schemas, the URLs for the schema locations are all relative. They specify only the file name and not the full path location. So, a processor would expect these to all reside in the same path. I'll follow this convention in this and the next two chapters.


Schema Examples

As we discussed in Chapter 6, schemas are optional in our approach. We can successfully run the utilities without them. However, if we want to be sure that certain business constraints are enforced, such as making specific fields mandatory, schemas are required. Schema validation is the mechanism we have chosen to enforce those constraints. In Chapter 6 I talked about an approach for designing these schemas. The basic recommendation is to take an instance document of the desired type, such as an invoice or purchase order, load it into a tool like XMLSPY, and have the tool create the schema. Here we'll review some schemas for our sample documents, the invoice and purchase order.

In this book I'm trying, for the most part, to avoid discussing specific products and how to use them. I'm not going to go through detailed instructions of how to use XMLSPY or TurboXML to generate a schema. Both products have good online help; use it. However, in this section I do show you the schemas for the invoice and purchase order as I modified them after using XMLSPY to generate them. Then I briefly discuss the kinds of corrections and cleanup I had to perform. First, here are the corrected schemas.

Invoice Schema (CSVInvoice.xsd)
<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
    elementFormDefault="unqualified">
  <xs:element name="Invoice">
    <xs:complexType>
      <xs:sequence>
        <xs:element name="InvoiceLine" maxOccurs="unbounded">
          <xs:complexType>
            <xs:sequence>
              <xs:element name="CustomerNumber" type="xs:token"/>
              <xs:element name="InvoiceNumber" type="xs:token"/>
              <xs:element name="InvoiceDate" type="xs:date"/>
              <xs:element name="PONumber" type="xs:token"/>
              <xs:element name="DueDate" type="xs:date"/>
              <xs:element name="ShipToName" type="xs:token"/>
              <xs:element name="ShipToStreet1" type="xs:token"/>
              <xs:element name="ShipToStreet2" type="xs:token"
                  minOccurs="0"/>
              <xs:element name="ShipToCity" type="xs:token"/>
              <xs:element name="ShipToStateOrProvince"
                  type="xs:token"/>
              <xs:element name="ShipToPostalCode"
                  type="xs:token"/>
              <xs:element name="ShipToCountry" type="xs:token"
                  minOccurs="0"/>
              <xs:element name="ItemID" type="xs:token"/>
              <xs:element name="ItemQuantity"
                  type="xs:positiveInteger"/>
              <xs:element name="UnitPrice" type="xs:decimal"/>
              <xs:element name="ItemDescription"
                  type="xs:token"/>
              <xs:element name="ExtendedPrice"
                  type="xs:decimal"/>
            </xs:sequence>
          </xs:complexType>
        </xs:element>
      </xs:sequence>
    </xs:complexType>
  </xs:element>
</xs:schema>
Purchase Order Schema (CSVPurchaseOrder.xsd)
<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
    elementFormDefault="unqualified">
  <xs:element name="PurchaseOrder">
    <xs:complexType>
      <xs:sequence>
        <xs:element name="POLine" maxOccurs="unbounded">
          <xs:complexType>
            <xs:sequence>
              <xs:element name="CustomerNumber" type="xs:token"/>
              <xs:element name="PONumber" type="xs:token"/>
              <xs:element name="PODate" type="xs:date"/>
              <xs:element name="RequestedDeliveryDate"
                  type="xs:date"/>
              <xs:element name="ShipToName" type="xs:token"/>
              <xs:element name="ShipToStreet1" type="xs:token"/>
              <xs:element name="ShipToStreet2" type="xs:token"
                  minOccurs="0"/>
              <xs:element name="ShipToCity" type="xs:token"/>
              <xs:element name="ShipToStateOrProvince"
                  type="xs:token"/>
              <xs:element name="ShipToPostalCode"
                  type="xs:token"/>
              <xs:element name="ShipToCountry" type="xs:token"
                  minOccurs="0"/>
              <xs:element name="ItemID">
                <xs:simpleType>
                  <xs:restriction base="xs:token">
                    <xs:enumeration value="HCDUC"/>
                    <xs:enumeration value="HCMIN"/>
                    <xs:enumeration value="HCMOC"/>
                    <xs:enumeration value="HCVAN"/>
                  </xs:restriction>
                </xs:simpleType>
              </xs:element>
              <xs:element name="OrderedQty"
                  type="xs:positiveInteger"/>
              <xs:element name="UnitPrice" type="xs:decimal"
                  minOccurs="0"/>
              <xs:element name="ItemDescription" type="xs:token"
                  minOccurs="0"/>
            </xs:sequence>
          </xs:complexType>
        </xs:element>
      </xs:sequence>
    </xs:complexType>
  </xs:element>
</xs:schema>

The exact schema generated and the corrections you must make depend on the tool you use and the particular options you select. Here are some of the changes I had to make to the invoice schema that XMLSPY 4.3 generated for me using my default options.

  • A value of "qualified" was assigned to the root Element's elementForm Default Attribute. I changed it to "unqualified" since we don't want to enforce namespace prefixes.

  • InvoiceNumber was enumerated and assigned a type of int. I changed the type to token and removed the enumeration.

  • A data type of string was assigned to ShipToName, ShipToStreet1, ShipTo Street2, ShipToCity, ShipToStateOrProvince, ShipToPostalCode, ShipToCountry, and ItemDescription. I wanted the schema validation to enforce a business constraint that there must be at least one nonspace character in these fields. The schema language string data type is insufficient for this purpose. At least two alternatives will enforce this constraint. I discussed one at the end of Chapter 6 (restricting the string data type with a pattern facet). For this example I used a somewhat easier (and lazier) approach of just using the token data type. We can justify this for our fictional scenario by saying we know that our data for these columns never contains two consecutive spaces. If the data in your real-life situation doesn't conform to this contrivance, you'll need to use the pattern facet.

  • Enumerations were generated for CustomerNumber, PONumber, ShipToCity, and ShipToStateOrProvince, restricting the string data type that XMLSPY had assigned them. I removed the enumerations.

  • ItemID was assigned a data type of string, further restricted by the enumerated IDs. I changed the data type to token and removed the enumeration for ItemID. Since the source of the data is our invoicing system, we can assume we're using valid item IDs and we don't need our schema to validate them. We have a somewhat different situation for the purchase order schema. We can't have the same confidence that our customers will send us the correct IDs for the items they order. Let's assume that we've decided for our application architecture that we would like the schemas to enforce validation of item IDs. (Several considerations for schema validation are discussed in Chapter 12.) In the standards world there has been an ongoing debate about what constitutes a "code" that might generally be schema validated and what constitutes a "unique identifier" that might not generally be schema validated. From a strict semantic perspective, our item IDs (especially when expressed as UPC codes, as we'll see in Chapters 9 and 10) are unique identifiers rather than codes. However, we'll bypass these theoretical arguments with a more pragmatic consideration. Big Daddy's Gourmet Cocoa sells only about a dozen different items, and it is very easy and practical to enumerate their IDs in a schema. So, we will.

  • XMLSPY also created enumerations for ItemQuantity, UnitPrice, and ExtendedPrice, which I removed. ItemQuantity was assigned a data type of short, which I changed to positiveInteger.

Other than these corrections, XMLSPY did a pretty good job. It set up the root and row Elements, Invoice and InvoiceLine, as anonymous types. This does not particularly foster reusability, but that's not really an issue for this application. It correctly determined that ShipToStreet2 and ShipToCountry are optional, assigning a minOccurs value of "0" to each, and that all the other columns are mandatory. It gave us the default minOccurs value of "1" on InvoiceLine and a maxOccurs value of "unbounded."

So, the tools can do most, but not all, of the schema creation work for you. As simple as these schemas are, you might be able to code them by hand. However, I would not recommend that approach for more complex cases, particularly for those we'll see in the next two chapters.


High-Level Design Considerations

Our approach to grammar analysis is now going to be a bit more formal than that presented in Chapters 2 and 3. I'll start the design discussion in this and the next two chapters by examining the grammar of the legacy file format. I'll present a logical analysis, followed by the schema for the file description document. In the previous section I presented the basic content of that document, but here I'll discuss its relation to the grammar. The grammar analysis will also lay a foundation for the parsing algorithms presented later in this chapter.

Grammar Analysis and Description

The grammar of our legacy file formats can be broken down into two separate grammars: (1) the grammar of records and groups of records within the file, and (2) the grammar of the fields within a record. In the case of CSV files, we have imposed the restriction that each row has the same format. The CSV file grammar can be expressed rather simply with the following BNF production.

CSV File Grammar
CSVFile ::= row+

The plus sign (+) indicates one or more. So, this simply says that a CSV file contains one or more rows. For CSV files, the row grammar is the interesting part. Remember, here we don't want to describe the grammar of a specific CSV file; instead we want to abstract the essential features of CSV files and develop a grammar that describes the whole class of CSV files. So, here's the CSV row grammar. Note that we follow through with the complete grammar by using the row nonterminal symbol from the file grammar.

CSV Row Grammar
row ::= column (column_delimiter column?)* | (column_delimiter
column?)+
column ::= column_characters_A+ |
           text_delimiter column_characters_B+ text_delimiter
column_characters_A ::= All allowed characters except
                        column_delimiter
column_characters_B ::= All allowed characters except
                        text_delimiter

Again, the plus sign (+) indicates one or more occurrences and the vertical bar or pipe (|) indicates an exclusive OR choice. The asterisk (*) indicates zero or more occurrences, the question mark (?) indicates optionality, and the parentheses are used to establish groupings, the same way they are used in mathematical equations. I've taken a bit of a shortcut in the last two productions by falling back to text rather than terminal and nonterminal symbols, but I think it is clearer than trying to enumerate the full set of allowed characters.

Bear in mind that any number of BNF productions can describe this grammar. Let's not get confused about whether or not this is the most elegant way to express the grammar; let's just focus on the one I present.

So, what does this grammar tell us? The first production tells us that a row can have:

  • A single column with nothing after it

  • A single column followed by any combination of empty and filled columns

  • An empty first column followed by any combination of empty and filled columns

This production tells us that all the following rows are legal, assuming we use a comma as the column delimiter.

Mary,had,a,little,lamb
Mary,had
Mary,,a,,lamb
Mary,,a,,,
,had,a,little,lamb
,had,a,,
,,,

Note that the grammar allows a row to end with empty columns or to be completely empty. Many applications won't produce such rows. However, since we're aiming to accommodate the widest possible class of CSV files I saw no reason to impose this restriction. This approach also makes the parsing algorithm a bit easier. But even though we'll be able to parse rows ending with empty columns, we're not going to create such rows. As discussed in Chapter 9, the grammar of EDI records (segments) doesn't allow empty fields at the end.

The last three productions basically tell us that a column either may have any character other than the selected column delimiter or, if it is delimited by the text delimiter in the first and last positions, may include the column delimiter. Again, this allows a much wider range of variations than are usually permitted by any particular application. Many delimit only alphanumeric columns that contain the column delimiter (usually a comma), but some delimit all columns regardless of type. Our approach allows us to accommodate nearly all cases. As we'll shortly see, this also keeps the parsing algorithm from getting too complex since we don't need to be concerned with the data type of the column as we are parsing. Note, however, than when we convert from XML to CSV, we use the DelimitText Element of the column's grammar to determine whether or not we should delimit the column content with the text delimiter character.

File Description Document Schemas

In designing the schemas for the invoice and purchase order I was fairly safe in letting a tool do most of the work. However, for the file description documents I start from the ground up. Much of the rationale was discussed in Chapter 6, so I'll point out here only things that I didn't mention there. We'll also discuss other issues related to schema design in Chapter 12.

There are four schemas presented in this subsection.

  1. CSVSourceFileDescription.xsd: This is the schema for file description documents that describe conversions in which the source format is a CSV file.

  2. CSVTargetFileDescription.xsd: This schema is for file description documents that describe conversions in which the target format is a CSV file.

  3. CSVCommonFileDescription.xsd: This type library schema is used by the two previous schemas.

  4. BBCommonFileDescription.xsd: This schema specifies types used in all the Babel Blaster conversions. It primarily defines the type for the supported Babel Blaster data types (corresponding to the DataCell derived classes) and the enumeration of the codes for that type.

So, here are the schemas, with a few more comments interspersed where appropriate. If you recall the lessons of Chapter 4 and review the comments in Chapter 6 about the union data type, you should be able to read these schemas fairly well.

CSV Source File Description Schema (CSVSourceFileDescription.xsd)
<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
    elementFormDefault="unqualified"
    attributeFormDefault="unqualified">
  <xs:include schemaLocation="CSVCommonFileDescription.xsd"/>
  <xs:element name="CSVSourceFileDescription">
    <xs:annotation>
      <xs:documentation>
        This schema specifies the format of File Description
        Documents when converting from CSV files as source to XML
        documents as targets
      </xs:documentation>
    </xs:annotation>
    <xs:complexType mixed="false">
      <xs:sequence>
        <xs:element name="PhysicalCharacteristics"
            type="CSVPhysicalCharacteristicsType"/>
        <xs:element name="XMLOutputCharacteristics"
            type="CSVXMLOutputCharacteristicsType"/>
        <xs:element name="Grammar"
            type="CSVGrammarType"/>
      </xs:sequence>
    </xs:complexType>
  </xs:element>
</xs:schema>
CSV Target File Description Schema (CSVTargetFileDescription.xsd)
<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
    elementFormDefault="unqualified"
    attributeFormDefault="unqualified">
  <xs:include schemaLocation="CSVCommonFileDescription.xsd"/>
  <xs:element name="CSVTargetFileDescription">
    <xs:annotation>
      <xs:documentation>
        This schema specifies the format of File Description
        Documents when converting from XML documents as source to
        CSV files as targets
      </xs:documentation>
    </xs:annotation>
    <xs:complexType mixed="false">
      <xs:sequence>
        <xs:element name="PhysicalCharacteristics"
            type="CSVPhysicalCharacteristicsType"/>
        <xs:element name="Grammar"
            type="CSVGrammarType"/>
      </xs:sequence>
    </xs:complexType>
  </xs:element>
</xs:schema>

You'll notice that these two schemas are nearly identical. Aside from the difference in the root document Element name, the source file schema has a required XMLOutputCharacteristics Element that does not appear in the target file schema.

CSV File Description Common Schema (CSVCommonFileDescription.xsd)
<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
    elementFormDefault="unqualified"
    attributeFormDefault="unqualified">
  <xs:include schemaLocation="BBCommonFileDescription.xsd"/>
  <xs:complexType name="CSVPhysicalCharacteristicsType"
      mixed="false">
    <xs:annotation>
      <xs:documentation>
          Describes the CSV physical record organization
      </xs:documentation>
    </xs:annotation>
    <xs:sequence>
      <xs:element name="RecordTerminator">
        <xs:complexType>
          <xs:complexContent>
            <xs:extension base="EmptyType">
              <xs:attribute name="value"
                  type="RecordTerminatorValueType"
                  use="required"/>
            </xs:extension>
          </xs:complexContent>
        </xs:complexType>
      </xs:element>
      <xs:element name="ColumnDelimiter" type="DelimiterType"/>
      <xs:element name="TextDelimiter" type="DelimiterType"/>
    </xs:sequence>
  </xs:complexType>
  <xs:complexType name="CSVXMLOutputCharacteristicsType"
      mixed="false">
    <xs:annotation>
      <xs:documentation>
        Describes characteristics of the output XML document
      </xs:documentation>
    </xs:annotation>
    <xs:sequence>
      <xs:element name="DocumentBreakColumn">
        <xs:complexType mixed="false">
          <xs:complexContent mixed="false">
            <xs:extension base="EmptyType">
              <xs:attribute name="value" type="BreakColumnType"
                  use="required"/>
            </xs:extension>
          </xs:complexContent>
        </xs:complexType>
      </xs:element>
      <xs:element name="PartnerBreakColumn">
        <xs:complexType mixed="false">
          <xs:complexContent mixed="false">
            <xs:extension base="EmptyType">
              <xs:attribute name="value" type="BreakColumnType"
                  use="required"/>
            </xs:extension>
          </xs:complexContent>
        </xs:complexType>
      </xs:element>
      <xs:element name="SchemaLocationURL" minOccurs="0">
        <xs:complexType>
          <xs:complexContent>
            <xs:extension base="EmptyType">
              <xs:attribute name="value" type="anyURI127"
                  use="required"/>
            </xs:extension>
          </xs:complexContent>
        </xs:complexType>
      </xs:element>
    </xs:sequence>
  </xs:complexType>
  <xs:complexType name="CSVGrammarType">
    <xs:annotation>
      <xs:documentation>
        Describes the grammar of the CSV file
      </xs:documentation>
    </xs:annotation>
    <xs:sequence>
      <xs:element name="RowDescription">
        <xs:annotation>
          <xs:documentation>
            Describes a row in the CSV file. Currently, all rows
            must have the same format so we only allow a single
            one of these Elements.
          </xs:documentation>
        </xs:annotation>
        <xs:complexType>
          <xs:sequence>
            <xs:element name="ColumnDescription" maxOccurs="100">
              <xs:annotation>
                <xs:documentation>
                  Describes a column in the row. The current
                  design limits us to one hundred columns per
                  row.
                </xs:documentation>
              </xs:annotation>
              <xs:complexType>
                <xs:complexContent>
                  <xs:extension base="FieldGrammarType">
                    <xs:attribute name="DelimitText"
                        type="xs:boolean" use="optional"
                        default="false"/>
                  </xs:extension>
                </xs:complexContent>
              </xs:complexType>
            </xs:element>
          </xs:sequence>
          <xs:attribute name="ElementName" type="NMToken127"
              use="required"/>
        </xs:complexType>
      </xs:element>
    </xs:sequence>
    <xs:attribute name="ElementName" type="NMToken127"
        use="required"/>
  </xs:complexType>
  <xs:simpleType name="BreakColumnType">
    <xs:annotation>
      <xs:documentation>
        Enforces restrictions on column number for partner and
        document break
      </xs:documentation>
    </xs:annotation>
    <xs:restriction base="xs:nonNegativeInteger">
      <xs:maxExclusive value="100"/>
    </xs:restriction>
  </xs:simpleType>
</xs:schema>
Babel Blaster File Description Common Schema (BBCommonFileDescription.xsd)
<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
    elementFormDefault="unqualified"
    attributeFormDefault="unqualified">
  <xs:annotation>
    <xs:documentation>
      This schema specifies types common to all Babel Blaster
      conversion utilities.
    </xs:documentation>
  </xs:annotation>
  <xs:annotation>
    <xs:documentation>
      These complex types define reused types with Attributes
      only and no Element children.
    </xs:documentation>
  </xs:annotation>
  <xs:complexType name="EmptyType">
    <xs:annotation>
      <xs:documentation>
        Base Type for empty types
      </xs:documentation>
    </xs:annotation>
  </xs:complexType>
  <xs:complexType name="DelimiterType">
    <xs:annotation>
      <xs:documentation>
        Base type for defining delimiters
      </xs:documentation>
    </xs:annotation>
    <xs:complexContent>
      <xs:extension base="EmptyType">
        <xs:attribute name="value" type="DelimiterValueType"
            use="required"/>
      </xs:extension>
    </xs:complexContent>
  </xs:complexType>
  <xs:complexType name="FieldGrammarType" mixed="false">
    <xs:annotation>
      <xs:documentation>
        Base type for defining field grammars
      </xs:documentation>
    </xs:annotation>
    <xs:complexContent mixed="false">
      <xs:extension base="EmptyType">
        <xs:attribute name="FieldNumber" type="FieldNumberType"
            use="required"/>
        <xs:attribute name="ElementName" type="NMToken127"
            use="required"/>
        <xs:attribute name="DataType" type="BBDataType"
            use="required"/>
      </xs:extension>
    </xs:complexContent>
  </xs:complexType>
  <xs:annotation>
    <xs:documentation>
      These simple types define unions of other simple types.
    </xs:documentation>
  </xs:annotation>
  <xs:simpleType name="DelimiterValueType">
    <xs:annotation>
      <xs:documentation>
        Type for column and text delimiters. A union of a
        single character (as a token of length one) or a two-
        byte hex value.
      </xs:documentation>
    </xs:annotation>
    <xs:union memberTypes="Token1 HexBinary1"/>
  </xs:simpleType>
  <xs:simpleType name="RecordTerminatorValueType">
    <xs:annotation>
      <xs:documentation>
        Type for value attribute of RecordTerminator - union of
        U,W, and 2 byte Hex.
      </xs:documentation>
    </xs:annotation>
    <xs:union memberTypes="OSTerminatorType HexBinary1"/>
  </xs:simpleType>
  <xs:annotation>
    <xs:documentation>
      These simple types specify restrictions on built-in schema
      data types.
    </xs:documentation>
  </xs:annotation>
  <xs:simpleType name="OSTerminatorType">
    <xs:annotation>
      <xs:documentation>
        Enumerations for OS terminator values
      </xs:documentation>
    </xs:annotation>
    <xs:restriction base="xs:token">
      <xs:enumeration value="U"/>
      <xs:enumeration value="W"/>
    </xs:restriction>
  </xs:simpleType>
  <xs:simpleType name="HexBinary1">
    <xs:annotation>
      <xs:documentation>
        Type for a single-byte hex number
      </xs:documentation>
    </xs:annotation>
    <xs:restriction base="xs:hexBinary">
      <xs:length value="1"/>
    </xs:restriction>
  </xs:simpleType>
  <xs:simpleType name="Token1">
    <xs:annotation>
      <xs:documentation>
        Token with length of 1
      </xs:documentation>
    </xs:annotation>
    <xs:restriction base="xs:token">
      <xs:length value="1"/>
    </xs:restriction>
  </xs:simpleType>
  <xs:simpleType name="BBDataType">
    <xs:annotation>
      <xs:documentation>
        These are the supported native Babel Blaster data types
        for CSV files. Add an enumeration element when adding a
        new data type.
      </xs:documentation>
    </xs:annotation>
    <xs:restriction base="xs:token">
      <xs:enumeration value="AN"/>
      <xs:enumeration value="R"/>
      <xs:enumeration value="DMMsDDsYYYY"/>
    </xs:restriction>
  </xs:simpleType>
  <xs:simpleType name="NMToken127">
    <xs:annotation>
      <xs:documentation>
        Data type for Element names. Restricted to 127 characters
        since C++ char arrays are 128.
      </xs:documentation>
    </xs:annotation>
    <xs:restriction base="xs:NMTOKEN">
      <xs:maxLength value="127"/>
    </xs:restriction>
  </xs:simpleType>
  <xs:simpleType name="FieldNumberType">
    <xs:annotation>
      <xs:documentation>
        Enforces restriction on maximum number of fields
      </xs:documentation>
    </xs:annotation>
    <xs:restriction base="xs:positiveInteger">
      <xs:maxExclusive value="100"/>
    </xs:restriction>
  </xs:simpleType>
  <xs:simpleType name="anyURI127">
    <xs:annotation>
      <xs:documentation>
        Enforces restriction on maximum length of URI
      </xs:documentation>
    </xs:annotation>
    <xs:restriction base="xs:anyURI">
      <xs:maxLength value="127"/>
    </xs:restriction>
  </xs:simpleType>
</xs:schema>

The schema representation of the grammar is actually very simple. The schema is somewhat of an abstraction of the grammar in that we can ignore physical characteristics such as column and text delimiters. It basically reduces us to saying that the grammar, as expressed in the Grammar Element, is composed of a single row description in the RowDescription Element. A row is composed of one or more columns as described in the ColumnDescription Element. The schema, in effect, reduces the description of the grammar to the following productions.

CSV File Grammar in Schema
CSVFileGrammar ::= rowGrammar
rowGrammar ::= columnGrammar+

Again, we'll see the more complex version used when we discuss the parsing algorithm.


CSV to XML: Detail Design

We're finally ready to start talking about code. In this section I'll go over the design for converting a CSV file to one or more XML documents. We'll also look at the grammar in terms of developing the parsing algorithm.

Main Program

As discussed in Chapter 6, in order to develop reusable modules that can be linked together into programs and not just cobbled together in scripts, we put the main processing logic in a routine called from a shell main program.

Logic for the Shell Main Routine for CSV to XML
Arguments:
  Input CSV File Name
  Output Directory Name
  File Description Document Name

Options:
  Validate output
  Help

Validate and process command line arguments
IF help option specified
  Display help message
  Exit
ENDIF
Create new CSVSourceConverter object, passing:
    Validation option
    Output Directory Name
    File Description Document Name
Call CSVSourceConverter processFile method, passing
    the Input File Name
Display completion message

On first glance this type of structure may seem like a bit of overkill. It would be if we were going to process only a single input file. However, it facilitates future enhancements such as accepting a directory as input and converting all the files in the directory.

CSVSourceConverter Class (Extends SourceConverter)

Overview

The CSVSourceConverter, then, is the main driver for the actual conversion. Here are its attributes and methods. It inherits all the attributes and methods of its base classes, the SourceConverter and Converter classes (see Chapter 6).

Attributes:

  • CSVRecordReader Object

  • Array of Strings Partner Array

  • String Schema Location URL

  • Integer Partner Break Column

  • String Saved Partner ID

  • Integer Document Break Column

  • String Saved Document ID

Methods:

  • Constructor

  • processFile

  • testDocumentBreak

  • testPartnerBreak

Methods
Constructor

The constructor method for our CSVSourceConverter object sets up that object, but most importantly it also sets up the CSVRecordReader object. Here's the logic for it.

Logic for the CSVSourceConverter Constructor Method
Arguments:
  Boolean Validation Option
  String Output Directory Name
  String File Description Document Name

Call base class constructor
Initialize Partner Array
Call loadFileDescriptionDocument using passed File
    Description Document Name
Schema Location URL <- Call File Description Document's
    getElementsByTagName on "SchemaLocationURL", then
    getAttribute on "value"
IF Schema Location URL is null and Validation is true
  Throw Exception
ENDIF
Document Break Column <- Get Document Break Column value from File
    Description Document
Partner Break Column <- Get Partner Break Column value from File
    Description Document
Initialize Saved Partner ID and Saved Document ID
Create CSVRecordReader object, passing:
    File Description Document
processFile

After the initialization performed in the constructor method, the main processing is performed by the CSVSourceConverter's processFile method. This method converts one input CSV file into one or more output XML documents based on the input parameters.

Logic for the CSVSourceConverter processFile Method
Arguments:
  String Name of input file

Returns:
  Status or throws exception

Output Directory Path <- Base Directory + directory separator
Initialize Output Document to null
Initialize Sequence Number
Row Grammar <- Get RowDescription from Grammar Element via
    getElementsByTagName, and item[0] from NodeList
Open input file
Call CSVRecordReader's setInputStream method
Record Length _<- Call CSVRecordReader's
    readRecordVariableLength method
DO while Record Length => 0
  Call CSVRecordReader's parseRecord method to parse columns into
      DataCell Array, passing Row Grammar Element
  Call CSVRecordReader's toXMLType method to convert column contents
      in DataCell Array to their schema language datatype
      representation
  Partner Break <- Call testPartnerBreak
  IF Partner Break = true
    Output Directory Path <- Base Directory +
        Partner ID + directory separator
    Lookup Partner in Partner Array
    IF Partner is not in Array
      Create output directory from Output Directory Path
      Partner Array <- Add Partner ID
    ENDIF
  ENDIF
  Document Break <- Call testDocumentBreak
  IF Document Break = true or Output Document is null
    IF Output Document exists
      Call saveDocument
    ENDIF
    Create new Output Document
    Create Root Element, using Root Element Name from Grammar,
        and append to Output Document
    IF Schema Location URL is not NULL
      Create noNamespaceSchemaLocation Attribute and append
          to Root Element
    ENDIF
    Increment Sequence Number, and pad with leading zeroes
        to three digits
    IF Document Break Column != 0
      Output File Path <- Output Directory Path + Root Element
          Name + Sequence Number + ".xml"
    ELSE
      Output File Path <- Output Directory Path + Root Element
          Name + ".xml"
    ENDIF
    Call CSVRecordReader's setOutputDocument method for new
        Output Document
  ENDIF
  Call CSVRecordReader's writeRecord method to write row and column
      Elements from CSVRecordReader's DataCall Array, passing
      Root Element of Output Document and Row Grammar Element
  Record Length <- Call CSVRecordReader's readRecord method
ENDDO
IF Output Document is not null
  Call base class saveDocument method
ENDIF
Close input file
Display completion message with number of documents processed

You'll note that we check for a new partner or document after converting to XML formats via the toXML method. We could check before converting to XML, but performing the conversion first ensures that we are working with data that is not only in string format but also normalized by removing leading and trailing whitespace. The testDocumentBreak and testPartnerBreak methods do the checks for us. Note that if no break is specified for new documents, we unconditionally return false for break on new partner.

testDocumentBreak

This method tests the value of the document break column to determine whether the current row starts a new document.

Logic for the CSVSourceConverter testDocumentBreak Method
Arguments:
  None

Returns:
  Boolean - true if new partner and false if not

IF Document Break Column is zero
  return false
ENDIF
DocumentID <- Call CSVRecordReader's getFieldValue method
    passing the Document Break Column number
IF DocumentID = Saved Document ID
  return false
ENDIF
Saved Document ID <- Document ID
Return true
testPartnerBreak

This method tests the value of the partner break column to determine whether the trading partner in the current row is different from that of the preceding document.

Logic for the CSVSourceConverter testPartnerBreak Method
Arguments:
  None

Returns:
  Boolean - true if new partner and false if not

IF Partner Break Column is zero or if Document Beak Column
    is zero
  return false
ENDIF
Partner ID <- Call CSVRecordReader's getFieldValue method
    passing the Partner Break Column number
IF Partner ID = Saved Partner ID
  return false
ENDIF
Saved Partner ID <- Partner ID
Return true

CSVRecordReader Class (Extends RecordReader)

Overview

We can see that most of the detailed work is performed by the CSVRecordReader class. It inherits several attributes and methods from its RecordReader and RecordHandler base classes (see Chapter 6). Here is a summary of its extensions to those classes. We'll review each of the methods.

Attributes:

  • DOM Nodelist Column Grammars

  • Character or Byte Column Delimiter

  • Character or Byte Text Delimiter

  • Integer Column Number

  • Integer Grammar Index

  • Integer Parsing State

Methods:

  • Constructor

  • parseRecord

  • saveCharacter

Note: In the Java and C++ implementations we also enumerate class-wide constants for the parsing states used by the parseRecord method.

Methods
Constructor

Here is the logic for the CSVRecordReader constructor method.

Logic for the CSVRecordReader Constructor Method
Arguments:
  DOM Document File Description Document

Call RecordReader base class constructor, passing File
    Description Document
Record Terminator <- Get "RecordTerminator" Element's value
    Attribute from File Description Document
Call setTerminator to set the Record Terminator1 and
    Record Terminator2
Column Delimiter <- Get "ColumnDelimiter" Element's value
    Attribute from File Description Document
Text Delimiter <- Get "TextDelimiter" Element's value
    Attribute from File Description Document
parseRecord

The CSVRecordReader's parseRecord method is where we finally take a more rigorous approach to the grammar of a CSV row. We can borrow several techniques and approaches from compiler construction to develop a good parsing algorithm. Most of those approaches are overkill (and indeed, some programmers may think even this discussion is overkill!), but taking advantage of some of the simpler techniques can go a long way toward keeping us out of trouble. We'll also be using them in the parseRecord method we use for EDI formats, which involves a more complex grammar.

The starting point, of course, is the grammar of a CSV row. We reviewed it in BNF earlier in the section. We now need to consider more carefully the characteristics of the grammar of a row. I show it again below so that you don't have to flip back several pages.

CSV Row Grammar
row ::= column (column_delimiter column?)* |
        (column_delimiter column?)+
column ::= column_characters_A+ |
           text_delimiter column_characters_B+ text_delimiter
column_characters_A ::= All allowed characters except
                        column_delimiter
column_characters_B ::= All allowed characters except
                        text_delimiter

If we examine the grammar closely, we can see that we can completely determine the meaning of a character, that is, its place in the grammar, simply by considering the characters that precede it. We don't need to do lookahead parsing, that is, examining one or more characters that follow the current character. Our CSV row conforms to the definition of a class of grammars called "regular expressions." This fact makes life a lot easier for us than it might be with more complex grammars.

One thing it means is that we can process the grammar with a fairly simple tool known as a finite state automaton. This is an abstract machine that consists of a number of states and specifies the input that causes the machine to move from one state to another. Such machines are easy to depict with state transition diagrams. Once diagrammed it is straightforward to develop a parsing algorithm. Figure 7.1 shows the state transition diagram for parsing a CSV row.

Figure 7.1. State Transition Diagram for Parsing a CSV Row

In this diagram the circles show the various states and the arrows show the characters in the row that cause the movement between states. Generally, the states correspond to the nonterminal symbols in the CSV row grammar described above, and the transitions correspond to the terminal symbols in the grammar, that is, the characters in the row. However, we have added the transitional states of New Column, Start Delimited Column, and Finish Delimited Column. These all correspond to the delimiter characters. Those familiar with finite state automatons will note that this is not a fully specified state machine in two regards. We don't have a final accepting state; the machine simply terminates at the end of the CSV row. Also, we don't show transitions to a so-called "dump" state where we terminate due to unexpected input. For example, if we have just scanned the closing text delimiter of a delimited column and entered the Finish Delimited Column state, anything other than the column delimiter is invalid input and will move us into the dump state.

Now, to turn this grammar and the state machine into not only a parsing approach but also a processing algorithm, we only need to add the actions that are performed during each state. This is simple since we're going to perform only two actions. First we save the input character to the DataCell object by calling the CSVRecordReader's saveCharacter method upon entering (or reentering) the Regular Column and Delimited Column states. Then we increment the column number each time we reenter the New Column state. Note that the algorithm includes "other" cases for unexpected input that correspond to moving to the dump state.

Logic for the CSVRecordReader parseRecord Method
Arguments:
  None

Returns:
  Error status or throw exception

Column Number <- 1
Column Grammars NodeList <- call Row Grammar Element's
    getElementsByTagName on "ColumnDescription"
GrammarsIndex <- -1
Parsing State <- New Column
DO until end of input record or Parsing State is Error
  Input Character <- Next character from input record
  DO CASE of Parsing State
    New Column:
      DO CASE of Input Character
        Column Delimiter:
          Increment Column Number
          BREAK
        Text Delimiter:
          Parsing State <- Start Delimited Column
          BREAK
        other:
          Call saveCharacter
          Parsing State <- Regular Column
          BREAK
      ENDDO
      BREAK
    Regular Column:
      DO CASE of Input Character
        Column Delimiter:
          Parsing State <- New Column
          Increment Column Number
          BREAK
        other:
          Call saveCharacter
          BREAK
        ENDDO
      BREAK
    Start Delimited Column:
      DO CASE of Input Character
        Text Delimiter:
          Parsing State <- Finish Delimited Column
          BREAK
        other:
          Call saveCharacter
          Parsing State <- Delimited Column
          BREAK
      ENDDO
      BREAK
    Delimited Column:
      DO CASE of Input Character
        Text Delimiter:
          Parsing State <- Finish Delimited Column
          BREAK
        other:
          Call saveCharacter
          BREAK
      ENDDO
      BREAK
    Finish Delimited Column:
      DO CASE of Input Character
        Column Delimiter:
          Parsing State <- New Column
          Increment Column Number
          BREAK
        other:
          Parsing State <- Error
          BREAK
      ENDDO
      BREAK
  ENDDO
ENDDO
IF Parsing State = Error
  Return error
ENDIF
Return success
saveCharacter

The saveCharacter method is fairly straightforward. We create a new DataCell object if we're not currently processing one. We then save the input character to the DataCell's buffer.

Logic for the CSVRecordReader saveCharacter Method
Arguments:
  Character Input Character

Returns:
  Integer GrammarIndex

IF Parsing State is not Regular Column or Delimited Column
  Increment Grammar Index
  Grammar Column Number <- Call Column Grammar NodeList
      item(Grammar Index) getAttribute on "FieldNumber", and
      convert to integer
  DO while Grammar Column Number < Column Number
    Increment Grammar Index
    IF Column Grammar NodeList item(Grammar Index) is null
      return error
    ENDIF
    Grammar Column Number <- Call Column Grammar NodeList
        item(Grammar Index) getAttribute on "FieldNumber",
        and convert to integer
  ENDDO
  IF (Grammar Column Number > Column Number)
    return error
  ENDIF
  call RecordHandler's createDataCell method
ENDIF
Call DataCell Array[Highest Cell] putByte method to append
    Input Character to DataCell buffer

Except for the new DataCell derived classes we'll develop in a later section in this chapter, this wraps up the design of our utility to convert from CSV files to XML documents. We'll next go over the design of the XML to CSV converter. Believe it or not, it is quite a bit simpler.


XML to CSV: Detail Design

Main Program

Again, we put the main processing logic in a routine that is called from a shell main program. The shell main program functions are described next.

Logic for the Shell Main Routine for XML to CSV
Arguments:
  Input Directory Name
  Output CSV File Name
  File Description Document Name

Options:
  Validate input
  Help

Validate and process command line arguments
IF help option specified
  Display help message
  Exit
ENDIF
Open output file
Create new CSVTargetConverter object, passing:
    Output Stream
    File Description Document Name
Set up implementation dependent DOM environment for loading,
    parsing, and validating input documents
Open input directory
Get first file from input directory
DO for all files in input directory
  Input Document <- Load input file, validating it if requested
  Call CSVTargetConverter processDocument method, passing
      the Input Document
  Increment number of documents processed
ENDDO
Close output file
Display completion message with number of documents processed

CSVTargetConverter Class (Extends TargetConverter)

Overview

The CSVTargetConverter is the main driver for the conversion. It is derived from the TargetConverter base class (see Chapter 6). It takes an input XML Document as a DOM Document and writes it to the CSV file output stream. This structuring facilitates using the converter in future situations in which the input might come from an XSLT transformation in memory, producing a DOM Document as the result, rather than just a disk file. Here are its attributes and methods.

Attributes:

  • None

Methods:

  • Constructor

  • processDocument

Methods
Constructor

Similar to the source converter, the constructor method for the CSVTarget Converter object sets up that object and the CSVRecordWriter object.

Logic for the CSVTargetConverter Constructor Method
Arguments:
  Output Stream CSV Output file
  File Description Document Name

Call base class constructor, passing File
    Description Document Name
Create CSVRecordWriter object, passing:
    File Description DOM Document and Output Stream
processDocument

The bulk of the processing is performed in the CSVTargetConverter's processDocument method. This method converts one input XML Document and writes it to the CSV output stream based on the input parameters. The logic is much simpler than the CSVSourceConverter's processFile method since we don't have to test for document or partner breaks or deal with saving DOM documents.

Logic for the CSVTargetConverter processDocument Method
Arguments:
  DOM Document Input Document

Returns:
  Error status or throws exception

Row Grammar <- Get "RowDescription" Element from Grammar Element
Grammar Row Element Name <- get Row Grammar Element's
    "ElementName" Attribute
Root Element <- Get Document's documentElement attribute
Root Element Name <- Get Root Element's tagName attribute
IF Root Element Name != Grammar Root Element Name
  Return error
ENDIF
DO for child Elements of Root Element
  Row Element Name <- Get Row Element's tagName attribute
  IF (Row Element Name != Grammar Row Element Name)
    Return error
  ENDIF
  Call CSVRecordWriter's parseRecord method to load DataCell
      Array from Elements for Row and Columns, passing
      Record Element and Row Grammar
  Call CSVRecordWriter's writeRecord to write output record
      from DataCell Array
ENDDO
Return success

CSVRecordWriter Class (Extends RecordWriter)

Overview

The CSVRecordWriter class is derived from the RecordWriter base class (see Chapter 6). We have only a few extensions to that class. We use the parseRecord method inherited from the RecordWriter base class.

Attributes:

  • Character or Byte Column Delimiter

  • Character or Byte Text Delimiter

Methods:

  • Constructor

  • writeRecord

Methods
Constructor

Here is the logic for the CSVRecordWriter constructor method.

Logic for the CSVRecordWriter Constructor Method
Arguments:
  DOM Document File Description Document
  Output Stream

Call RecordWriter base class constructor, passing File
    Description Document and Output Stream
Record Terminator <- Get "RecordTerminator" Element's "value"
    Attribute from File Description Document
Call setTerminator to set the Record Terminators
Column Delimiter <- Get "ColumnDelimiter" Element's "value"
    Attribute from File Description Document
Text Delimiter <- Get "TextDelimiter" Element's "value"
    Attribute from File Description Document
writeRecord

The writeRecord method is unique to each RecordWriter derived class. This method in the CSVRecordWriter class takes the content of the DataCell Array, formats it to the CSV data types, builds a string for the output record, and finally writes it to the output stream.

Logic for the CSVRecordWriter writeRecord Method
Arguments:
  None

Returns:
  Error status or throws exception

Initialize Output Record Buffer
Output Column Number <- 1
DO for Index from 0 through Highest Cell
  Cell Column Number = Get Cell's Field Number
  DO while Output Column Number < Cell Column Number
    Append Column Delimiter to Output Record Buffer
    Increment Output Column Number
  ENDDO
  Call fromXML to convert Cell contents from XML datatype
  Delimit Text Flag <- call Cell's delimitText method
  Cell Contents <- Call getField to retrieve cell buffer
      contents
  Clear Array entry
  IF Delimit Text Flag is true
    Append Text Delimiter character to Output Record Buffer
    Append Cell Contents to Output Record Buffer
    Append Text Delimiter character to Output Record Buffer
  ELSE
    Append Cell Contents to Output Record Buffer
  ENDIF
ENDDO
Highest Cell <- -1
Append base RecordHandler's Record Terminators to Output Record
    Buffer
Call language's write routines to do physical write of
    Output Record Buffer
Return success


New DataCell Methods and Derived Classes

For the data types supported for CSV files we add two new methods to the base class and create three new derived classes. Depending on the conversion requirements, the new derived classes may have their own implementation of the fromXML or toXML method, or they may use the implementation in the base class.

New DataCell Methods

delimitText

This method checks to see whether a DelimitText Element in the cell's Grammar Element is set to true. It is primarily used by the XML to CSV utility to determine whether or not a column should have the text delimiter added to it before writing it to the output CSV file.

Logic for the DataCell delimitText Method
Arguments:
  None

Returns:
  Boolean - true if DelimitText Attribute is present and has a
      value of true, false otherwise

Delimit Text <- call Grammar Element's getAttribute
    on "DelimitText"
IF Delimit Text String is null or if Delimit Text String
    is false
  return false
ENDIF
Return true
trimLeadingZeroes

This method trims leading zeroes from numeric data types. It also trims leading and trailing whitespace. It is designed to properly trim leading zeroes and spaces from the following numeric representations while retaining the sign characters:

  • Sign character in first position followed by leading spaces or zeroes

  • Leading spaces followed by leading sign character

  • Leading spaces or zeroes with no sign character

Note that this algorithm is somewhat permissive in that it will convert and pass data that isn't numeric or that has more than one sign character. Again, we depend on schema validation to detect most of those kinds of problems. We keep our code simple here by doing the minimum required.

Logic for the DataCell trimLeadingZeroes Method
Arguments:
  None

Returns:
  Error status or throws exception

Initialize Sign Character
IF first character is + or - sign character
  Sign Character <- first Character
ENDIF
Call trim (C++ base class method or native Java) to remove
    leading and trailing whitespace from Cell Buffer
IF Cell Buffer is empty after trimming
  return
ENDIF
Position <- 0
IF first character is + or - sign character
  Sign Character <- first Character
  Position <- 1;
ENDIF
DO while Position < Buffer Length and
    Cell Buffer[Position] = "0"
  Position++
ENDDO
IF Position = Buffer Length because it contains only zeroes
  Decrement Position so that we have at least one zero
ENDIF
IF Sign Character is present
  Cell Buffer <- Sign Character + Cell Buffer substring starting
      at Position
ELSE
  Cell Buffer <- Cell Buffer substring starting at Position
ENDIF
Return success

DataCellAN Class

This class handles conversion to and from an alphanumeric data type and the schema language string data type. It uses the base class fromXML method but implements its own version of the toXML method.

Logic for the DataCellAN toXML Method
Arguments:
  None

Returns:
  Error status or throws exception

Call trim (C++ utility method or native Java) to remove leading
    and trailing whitespace from Cell Buffer
Return success

DataCellReal Class

This class handles conversion to and from a real number (or decimal) data type and the schema language decimal data type. The main thing to note about this class is that the fromXML method trims spaces. If the source XML document is validated, the source Element with a schema language data type of decimal should never have spaces in it. However, we can't depend on validation being performed, and other methods that we'll develop later depend on there being no spaces in the Cell Buffer. So, we remove them with the fromXML method.

Logic for the DataCellReal fromXML Method
Arguments:
  None

Returns:
  Error status or throws exception

Trim Spaces
Remove leading plus sign if present
Return success
Logic for the DataCellReal toXML Method
Arguments:
  None

Returns:
  Error status or throws exception

Call trimLeadingZeroes base class method to remove leading
    zeroes and leading and trailing spaces
Return success

DataCellDateMMsDDsYYYY Class

This class handles conversion to and from the date data type in MM/DD/YYYY format to the schema language date data type in ISO 8601 date format, that is, YYYY-MM-DD. In the MM/DD/YYYY format both the MM and the DD may be either one or two digits in length.

Note: Various library functions are available in Java and C++ that might make the implementation a bit more efficient than the one presented here. However, in the interest of keeping things simple for myself, as I'm implementing in both languages, I've chosen basic algorithms that work equally well in either language. All positions are expressed as offsets from the first character at position zero.

Also, remember that we're only putting enough validation into these routines to avoid nasty runtime exceptions. Schema validation is our primary method for ensuring that we have good dates.

Logic for the DataCellDateMMsDDsYYYY fromXML Method
Arguments:
  None

Returns:
  Error status or throws exception

IF Buffer Length != 10
  Return error
ENDIF
Month <- Cell Buffer characters at offsets 5 and 6
Day <- Cell Buffer characters at offsets 8 and 9
Year <- Cell Buffer characters at offsets 0 through 3
Cell Buffer <- Month + forward slash + Day + forward slash
    + year
Return success
Logic for the DataCellDateMMsDDsYYYY toXML Method
Arguments:
  None

Returns:
  Error status or throws exception

Month State = 0
Day State = 1
Year State = 2
State <- Month
Cell Buffer < trim leading and trailing whitespace from
    Cell Buffer
IF Buffer Length > 10
  Return error
ENDIF
TempChar <- First character in Cell Buffer
DO until end of Cell Buffer
  IF TempChar = Slash
    Increment State
  ELSE
    DO CASE of State
      Month:
        Append TempChar to Month
        BREAK
      Day:
        Append TempChar to Day
        BREAK
      Year:
        Append TempChar to Year
        BREAK
      other:
        Return Error
    ENDDO
  ENDIF
  TempChar <- Next character in Cell Buffer
ENDDO
IF length of Month is 1
  Month <- "0" + Month
ENDIF
If Length of Day is 1
  Day <- "0" + Day
ENDIF
IF (Length of Month != 2) OR (Length of Day != 2) OR
    (Length of Year != 4)
  Return error
ENDIF
Cell Buffer = Year + dash + Month + dash + Day
Return success


Java Implementation

The Java implementation of the design is composed of the source files listed below.

  • CSVToXML.java: the main routine for the Java application with CSV as the source

  • CSVSourceConverter.java (in package net.babelblaster.convert): the class that drives conversion from CSV to XML

  • CSVRecordReader.java (in package net.babelblaster.record): the class that handles reading CSV records and converting them to XML

  • XMLToCSV.java: the main routine for the Java application with CSV as the target

  • CSVTargetConverter.java (in package net.babelblaster.convert): the class that drives conversion from XML to CSV

  • CSVRecordWriter.java (in package net.babelblaster.record): the class that handles reading XML and building and writing CSV records

  • DataCellAN.java (in package net.babelblaster.cell): the class that handles alphanumeric fields in legacy formats

  • DataCellReal.java (in package net.babelblaster.cell): the class that handles real (or decimal) number fields in legacy formats

  • DataCellDateMMsDDsYYYY.java (in package net.babelblaster.cell): the class that handles date fields in MM/DD/YYYY representation in legacy formats

Since we are focusing on the logical design, have already covered the basic DOM operations with Java, and discussed earlier some advanced DOM programming considerations, I won't have very many comments here or in the C++ section.

The Java implementation is portable, so we can't make assumptions about the directory path separator character. To get it we use the System class getProperty method on the file.separator property, as follows:

sDirectorySeparator = System.getProperty("file.separator");

To create an output directory in the CSVSourceConverter's processFile method, we use the mkdir method of the File class.

File FilePath = new File(sOutputDirectory);
FilePath.mkdir();

To read an input directory in the XMLToCSV main program, we again use the File class, passing the input directory name in the constructor. We then call the list method, returning an array of strings with the file names. These are the local names within the directory, so we must prefix them with the directory path before opening them.

InputDirectory = new File(sInputDirectoryName);
FilesList = InputDirectory.list();

if (FilesList != null)
{
//  Get first file from input directory
//  DO for all files in input directory
for (i=0; i < FilesList.length; i++)
{
  //  Build input file name
  sInputFileName = sInputDirectoryName + FilesList[i];

  //  DOM Document <- Load input file, validating it if
  //    requested
  docInput = Builder.parse(new File(sInputFileName));


C++ Implementation

The C++ implementation is composed of the source files listed below.

  • CSVToXML.cpp: the main routine for the C++ application with CSV as the source

  • CSVSourceConverter.cpp: the class that drives conversion from CSV to XML

  • CSVSourceConverter.h: the header file for the CSVSourceConverter class

  • CSVRecordReader.cpp: the class that handles reading CSV records and converting to XML

  • CSVRecordReader.h: the header file for the CSVRecordReader class

  • XMLToCSV.cpp: the main routine for the C++ application with CSV as the target

  • CSVTargetConverter.cpp: the class that drives conversion from XML to CSV

  • CSVTargetConverter.h: the header file for the CSVTargetConverter class

  • CSVRecordWriter.cpp: the class that handles reading XML and building and writing CSV records

  • CSVRecordWriter.h: the header file for the CSVRecordWriter class

  • DataCellAN.cpp: the class that handles alphanumeric fields in legacy formats

  • DataCellAN.h: the header file for the DataCellAN class

  • DataCellReal.cpp: the class that handles real (or decimal) number fields in legacy formats

  • DataCellReal.h: the header file for the DataCellReal class

  • DataCellDateMMsDDsYYYY.cpp: the class that handles date fields in MM/DD/YYYY representation in legacy formats

  • DataCellDateMMsDDsYYYY.h: the header file for the DataCellDateMMsDDsYYYY class

Directory operations are the only new C++ and WIN32 coding concepts introduced in these programs. To create an output directory in the CSVSourceConverter's processFile method, we use the _mkdir function.

iReturn = _mkdir(cOutputDirectory);

To read an input directory in the XMLToCSV main program, we use the FindFirstFile library routine, followed by successive calls to FindNextFile. FindFirstFile returns a File HANDLE and a WIN32_FIND_DATA structure from which we can retrieve the file name. FindNextFile returns us the next HANDLE and its WIN32_FIND_DATA structure. Here are the relevant lines from the code.

HANDLE hFind;
WIN32_FIND_DATA FindFileData;
...
//  For FindFirstFile, append wildcards
if ( strcmp( &(cInputDirectoryFindPath[i-3]),"*.*") != 0)
{
  strcat(cInputDirectoryFindPath,"*.*");
}

//  Open input directory
//  Get first file from input directory
hFind = FindFirstFile(cInputDirectoryFindPath,
    &FindFileData);
if (hFind == INVALID_HANDLE_VALUE)
{
  throw cInputDirectoryOpenError;
}

//  DO for all files in input directory
do
{
  //  Skip directory file paths
  if (FindFileData.cFileName[0] == '.')
  {
    continue;
  }

  //  Build input file name
  memset(cInputFileName,0,MAX_PATH_LENGTH);
  strcpy(cInputFileName,cInputDirectoryName);
  strcat(cInputFileName,FindFileData.cFileName);
  ... Load, parse, and process the document
} while (FindNextFile(hFind, &FindFileData) == TRUE);
//  ENDDO

//  Close the input directory
FindClose(hFind);


Enhancements and Alternatives

These utilities present significant enhancements to the ones we developed earlier in the book. We can use our own choice of Element names, handle multiple logical documents, and have data types automatically converted for us. However, there are still a few things we could improve.

Additional Data Types

Although I add more types in later chapters, there's still a good possibility that you might like the utilities to support additional data types. For example, you might want the utilities to handle dates in MM/DD/YY or DD/MM/YYYY formats. The architecture and implementation are both designed to allow you to add new derived DataCell classes without much effort. In addition to coding the class, you need only change the RecordHandler's createDataCell method and the BBCommonFileDescription.xsd schema.

Variety of Record Types

What if your CSV file has rows with different formats instead of all being the same? For example, what if you have a row for a purchase order header followed by rows for line items, instead of one row per line item with the header information repeated in each row? This leads us quite nicely into the next chapter, which deals with this type of file organization, only with fixed length records. We'll discuss that enhancement a bit at the end of Chapter 8.

Efficiency and Performance

As I have said repeatedly, while efficiency and performance are important, in this book I'm putting more emphasis on simplicity, clarity, maintainability, and reusability. However, if we need to wring some efficiency out of the code, there are a few prime targets we can examine first.

The perceptive will notice that when processing source CSV files we scan the input record twice. We scan it once to find the record terminator(s) as we read the input row, then a second time as we parse the columns out of the row. We could modify the parsing algorithm in the CSVRecordReader's parseRecord method so that the input came directly from our file read operation rather than as a record buffer we have already read from disk. However, in doing so we would lose the ability to reuse the RecordReader's readRecordVariableLength method and we would add a bit of complexity to the parseRecord method.

Another area of inefficiency is in repeated copying of Cell Buffer contents. This is probably most notable in the Java implementation, where we store the data as a byte array but frequently convert to and from String for formatting and using the DOM API methods. However, we also do conversions in C++ because the MSXML DOM API methods deal with string data as VARIANTs or BSTRs. For the Java implementation we could maintain both a byte array and a String or StringBuffer for the Cell Buffer contents. We could read data into the byte array from disk and convert to String as part of the toXML method. We would do the reverse operation when processing XML as the source. This would require changes only to the DataCell class and its derived classes.

For C++ and MSXML, perhaps the most direct way to eliminate many of the conversions is to deal natively with the string data as VARIANTs and BSTRs rather than converting to and from char arrays. I have not taken that approach since one of my goals has been to try to keep the code at least somewhat portable. However, if you want to try to optimize the code (or your code) for a Microsoft environment, this is certainly one thing to investigate.

However, all of these efficiency tweaks are probably insignificant compared with the DOM overhead, and I'm not sure the performance gains would be worth the effort. The DOM API is an essential part of the architecture and the implementations, and there isn't much we can do about DOM performance without radically changing the design. I think in most cases it would probably be more cost effective to buy faster hardware than to redesign the utilities. The economics of using a DOM API with a royalty-free license, as opposed to trying to develop a more efficient approach that doesn't use one, are just too compelling.


Resources

DOM Level 2 Core Specification, November 13, 2000, by the World Wide Web Consortium. Available online at http://www.w3.org/DOM/DOMTR#dom2,

DOM Reference, Microsoft XML Core Services (MSXML) 4.0, by Microsoft Corporation. Online documentation included as part of the MSXML SDK. Available online at http://msdn.microsoft.com/xml.

Java API for XML Processing Specification 1.2, by Sun Microsystems. Online Javadoc documentation, included as part of the Java XML Pack Spring 2002. Available online at http://java.sun.com/xml.

Xerces2 API Javadoc, by Apache Software Foundation. Online Javadoc documentation, included as part of the Xerces2 SDK. Available online at http://xml.apache.org.


Chapter 8. Converting Flat Files to and from XML

Ohio is the flat place between Hoboken and Malibu.

—John Fleischman

This chapter presents utilities for converting flat files to and from XML documents. A generalized grammar for describing flat file logical organization is discussed and used in the design.


Flat File to XML: Functionality and Operation

Requirements

This utility converts a flat file containing one or more logical documents to one or more XML instance documents, each representing a single logical document. Here's a summary of the required functionality.

  • Inputs: A flat file in fixed or variable length format, with each field having a fixed length. A specific field in each record contains a record identifier. The input flat file may consist of more than one logical document. The second input is a file description document (as discussed in Chapter 6) that describes the flat file and the grammar of the XML document to be produced.

  • Processing: An Element is created in the output document for each logical group of records in the flat file, and each input flat file record is written to an Element that is a child of that group Element. Each field in the record is written to a child Element of the record Element. The organization of records into groups, of fields into records, and their Element names are derived from the file description document. Field content is converted to schema language data types as specified in the file description document. Empty fields, that is, those with zero length after whitespace has been trimmed, do not create Elements in the output document. Processing breaks optionally occur on a change in a trading partner field.

  • Output: One or more XML instance documents, each in a single file. The root Element name is derived from the grammar in the file description document. The file name is formed by appending a three-digit sequence number to the root Element name and adding the extension .xml. If break on trading partner has been specified, the documents for each trading partner are placed in a separate subdirectory. The subdirectories are named according to the trading partner IDs in the partner break field.

Running the Utility

This section provides instructions for running the flat file to XML conversion utility from the command line.

For Java:

java FlatToXML InputFile.dat OutputDirectory FileDescription.XML

or

java FlatToXML -h

For C++ on Win32:

FlatToXML InputFile.dat OutputDirectory FileDescription.XML

or

FlatToXML -h

Options follow the parameters except for the help option, which may be specified by itself.

Parameters:

  • First: File specification of the input flat file (required). The specification may include the full or relative path name. If no path name is specified, the file is assumed to reside in the current working directory. The full file name must be specified, but there is no restriction on the extension name.

  • Second: Path specification of the output directory (required). The directory must exist. Either a relative or an absolute path name may be specified. The trailing directory separator character is optional. If no break on trading partner is specified, all the created XML files are placed in this directory. If break on trading partner has been specified, then a subdirectory for each trading partner is created beneath this directory.

  • Third: File specification of the file description document (required). If no path name is specified, the file is assumed to reside in the current working directory. The full file name must be specified, but there is no restriction on the extension name.

Options:

  • -v (Validate): Validate the created XML documents before writing them to disk. The documents are validated against the schema specified in the file description document.

  • -h (Help): Display a help message and exit without further processing.

Restrictions:

Unless otherwise noted, all numeric limits may be modified by changing parameters in the program source and appropriate type definitions in the file description document schemas.

  • A field may have a maximum of 1,023 bytes.

  • A record may be no longer than 16,383 bytes.

  • A maximum of 100 fields per record is supported.

  • There is no absolute limit on the number of records; the number is only practically limited by system memory.

  • Each field must be assigned a unique Element name.

  • Element names are limited to 127 characters.

  • It is recommended, though not required, that field grammar Elements be specified in their record description Element in ascending order by offset.

  • Path lengths for complete file specifications are limited to 127 characters.

  • Schema location URIs are limited to 127 characters.

  • A maximum of 999 output XML documents from an input flat file is supported.

  • A maximum of 100 different trading partner destinations in an input flat file is supported.

  • The field indicating a break on trading partner must be in the beginning record of a logical document.

  • Trading partner IDs must be valid directory names for the operating system where the utility is run.

Sample Input and Output: Invoice

As in Chapter 7, we're going to use invoices from Big Daddy's Gourmet Cocoa for our example. However, Big Daddy has now upgraded to a more capable order management and bookkeeping system. The new system supports a more comprehensive flat, hierarchical file structure than the CSV formats supported by the previous system.

The simple invoice example is composed of two levels of record groups. The group at the top level is the invoice itself, consisting of a header record, ship to address, one or more line item groups, and a summary record. The second group, the line item group, contains a line item record and an item description record.

This particular file has variable length records. Although Big Daddy's system uses variable length records, we could just as easily specify a fixed length record. Table 8.1 shows the logical layout of the invoice file.

Figure 8.1 shows the sample input invoice flat file.

Table 8.1. Logical Layout for the Invoice

Group

Record

Record Tag

Field Number

Field Name

Offset

Length

Data Type

Description

Invoice

Header

HDR

1

Record Tag

0

3

Alphanumeric

Record identifier

2

Customer Number

3

20

Alphanumeric

Identifier we have assigned to the customer in our system

3

Invoice Number

23

20

Alphanumeric

System-assigned invoice number

4

Invoice Date

43

8

Date

Date of invoice, formatted YYYYMMDD

5

PO Number

51

20

Alphanumeric

Customer purchase order number

6

Due Date

71

8

Date

Date that invoice amount is due for payment, formatted as YYYYMMDD

Invoice

Ship To

SHP

1

Record Tag

0

3

Alphanumeric

Record identifier

2

Ship to Name

3

40

Alphanumeric

Name of the receiving location for the shipped order

3

Ship to Street 1

43

30

Alphanumeric

First address line of the receiving location

4

Ship to Street 2

73

30

Alphanumeric

Second address line of the receiving location

5

Ship to City

103

20

Alphanumeric

City of the receiving location

6

Ship to State or Province

123

3

Alphanumeric

State or province of the receiving location

7

Ship to Postal Code

126

10

Alphanumeric

Postal code of the receiving location

8

Ship to Country

136

3

Alphanumeric

Country of the receiving location

Invoice/Line Item

Line

LIN

1

Record Tag

0

3

Alphanumeric

Record identifier

2

Item ID

3

20

Alphanumeric

Our identifier for the ordered item

3

Item Invoiced Quantity

23

10

Decimal number, space filled

The number of units to be invoiced

4

Item Unit Price

33

10

Implied decimal, two places, zero filled

Unit price in U.S. dollars

5

Extended Amount Due

43

10

Implied decimal, two places, zero filled

Total amount due for the invoiced item in U.S. dollars (unit price multiplied by the quantity invoiced)

Invoice/Line Item

Item Description

DSC

1

Record Tag

0

3

Alphanumeric

Record identifier

2

Item Description

3

80

Alphanumeric

Description of the ordered item

Invoice

Summary

SUM

1

Record Tag

0

3

Alphanumeric

Record identifier

2

Total Amount

3

10

Implied decimal, two places, zero filled

Total amount due on invoice in U.S. dollars

3

Number of Lines

13

10

Integer, space filled

Total number of invoice lines

Figure 8.1 Sample Input Flat File (Invoices.dat)
[View full width]
         10        20        30        40        50        60        70        80
  90        100       110       120       130
         |         |         |         |         |         |         |         |
  |         |         |         |         |         |
HDRBQ003               2002041             20021112AZ999345            20021212
SHPYazoo Grocers - NE Distribution Center  12 Industrial Parkway NW
               Portland            ME 04101
LINHCVAN                       120000000259000003108
DSCInstant Hot Cocoa Mix - Vanilla flavor
LINHCMIN                       240000000253000006072
DSCInstant Hot Cocoa Mix - Mint flavor
SUM0000009180         2
HDRBQ003               2002042             20021112AW999346            20021212
SHPYazoo Grocers - SE Distribution Center  Dock 37                       3975 Hwy 75
               Atoka               OK 74525
LINHCVAN                       360000000259000009324
DSCInstant Hot Cocoa Mix - Vanilla flavor
LINHCMIN                       720000000253000018216
DSCInstant Hot Cocoa Mix - Mint flavor
SUM0000027540         2
HDRAY001               2002043             200211122002-0967           20021212
SHPCorner Drug and Sundries                14 Main Street
               Wichita             KS 67201
LINHCVAN                       240000000259000006216
DSCInstant Hot Cocoa Mix - Vanilla flavor
SUM0000006216         1
HDRBR095               2002044             200211124397-0498           20021212
SHPBig Box Discounters - Store # 97        37 MegaMall
               Azusa               CA 91702
LINHCMIN                      1200000000253000030360
DSCInstant Hot Cocoa Mix - Mint flavor
LINHCVAN                      3600000000259000093240
DSCInstant Hot Cocoa Mix - Vanilla flavor
LINHCDUC                      2400000000259000062160
DSCInstant Hot Cocoa Mix - Dutch Chocolate flavor
SUM0000185760         3
HDRBR095               2002045             200211124345-0498           20021212
SHPBig Box Discounters - Store # 45        45 Highway 76
               Branson             MO 65615
LINHCMIN                       720000000253000018216
DSCInstant Hot Cocoa Mix - Mint flavor
LINHCDUC                       960000000259000024864
DSCInstant Hot Cocoa Mix - Dutch Chocolate flavor
SUM0000043080         2
HDRDQ349               2002046             20021112987-43671           20021212
SHPMaple Leaf Grocers - DC #1              987 Yorkland Blvd
               Willowdale          ON M2J 4Y8   CAN
LINHCMOC                     36000000000269000096840
DSCInstant Hot Cocoa Mix - Mocha flavor
SUM0000096840         1

Listed below are the first three XML documents produced by the utility from this input file.

FlatInvoice001.xml
<?xml version="1.0" encoding="UTF-8"?>
<FlatInvoice
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:noNamespaceSchemaLocation="FlatInvoice.xsd">
  <Header>
    <RecordID>HDR</RecordID>
    <CustomerNumber>BQ003</CustomerNumber>
    <InvoiceNumber>2002041</InvoiceNumber>
    <InvoiceDate>2002-11-12</InvoiceDate>
    <PONumber>AZ999345</PONumber>
    <DueDate>2002-12-12</DueDate>
  </Header>
  <ShipTo>
    <RecordID>SHP</RecordID>
    <ShipToName>
      Yazoo Grocers - NE Distribution Center
    </ShipToName>
    <ShipToStreet1>12 Industrial Parkway NW</ShipToStreet1>
    <ShipToCity>Portland</ShipToCity>
    <ShipToStateOrProvince>ME</ShipToStateOrProvince>
    <ShipToPostalCode>04101</ShipToPostalCode>
  </ShipTo>
  <LineItemGroup>
    <LineItem>
      <RecordID>LIN</RecordID>
      <ItemID>HCVAN</ItemID>
      <ItemQuantity>12</ItemQuantity>
      <UnitPrice>2.59</UnitPrice>
      <ExtendedPrice>31.08</ExtendedPrice>
    </LineItem>
    <ItemDescription>
      <RecordID>DSC</RecordID>
      <Description>
        Instant Hot Cocoa Mix - Vanilla flavor
      </Description>
    </ItemDescription>
  </LineItemGroup>
  <LineItemGroup>
    <LineItem>
      <RecordID>LIN</RecordID>
      <ItemID>HCMIN</ItemID>
      <ItemQuantity>24</ItemQuantity>
      <UnitPrice>2.53</UnitPrice>
      <ExtendedPrice>60.72</ExtendedPrice>
    </LineItem>
    <ItemDescription>
      <RecordID>DSC</RecordID>
      <Description>
        Instant Hot Cocoa Mix - Mint flavor
      </Description>
    </ItemDescription>
  </LineItemGroup>
  <Summary>
    <RecordID>SUM</RecordID>
    <TotalAmount>91.80</TotalAmount>
    <NumberOfLines>2</NumberOfLines>
  </Summary>
</FlatInvoice>
FlatInvoice002.xml
<?xml version="1.0" encoding="UTF-8"?>
<FlatInvoice
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:noNamespaceSchemaLocation="FlatInvoice.xsd">
  <Header>
    <RecordID>HDR</RecordID>
    <CustomerNumber>BQ003</CustomerNumber>
    <InvoiceNumber>2002042</InvoiceNumber>
    <InvoiceDate>2002-11-12</InvoiceDate>
    <PONumber>AW999346</PONumber>
    <DueDate>2002-12-12</DueDate>
  </Header>
  <ShipTo>
    <RecordID>SHP</RecordID>
    <ShipToName>
      Yazoo Grocers - SE Distribution Center
    </ShipToName>
    <ShipToStreet1>Dock 37</ShipToStreet1>
    <ShipToStreet2>3975 Hwy 75</ShipToStreet2>
    <ShipToCity>Atoka</ShipToCity>
    <ShipToStateOrProvince>OK</ShipToStateOrProvince>
    <ShipToPostalCode>74525</ShipToPostalCode>
  </ShipTo>
  <LineItemGroup>
    <LineItem>
      <RecordID>LIN</RecordID>
      <ItemID>HCVAN</ItemID>
      <ItemQuantity>36</ItemQuantity>
      <UnitPrice>2.59</UnitPrice>
      <ExtendedPrice>93.24</ExtendedPrice>
    </LineItem>
    <ItemDescription>
      <RecordID>DSC</RecordID>
      <Description>
        Instant Hot Cocoa Mix - Vanilla flavor
      </Description>
    </ItemDescription>
  </LineItemGroup>
  <LineItemGroup>
    <LineItem>
      <RecordID>LIN</RecordID>
      <ItemID>HCMIN</ItemID>
      <ItemQuantity>72</ItemQuantity>
      <UnitPrice>2.53</UnitPrice>
      <ExtendedPrice>182.16</ExtendedPrice>
    </LineItem>
    <ItemDescription>
      <RecordID>DSC</RecordID>
      <Description>
        Instant Hot Cocoa Mix - Mint flavor
      </Description>
    </ItemDescription>
  </LineItemGroup>
  <Summary>
    <RecordID>SUM</RecordID>
    <TotalAmount>275.40</TotalAmount>
    <NumberOfLines>2</NumberOfLines>
  </Summary>
</FlatInvoice>
FlatInvoice003.xml
<?xml version="1.0" encoding="UTF-8"?>
<FlatInvoice
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:noNamespaceSchemaLocation="FlatInvoice.xsd">
  <Header>
    <RecordID>HDR</RecordID>
    <CustomerNumber>AY001</CustomerNumber>
    <InvoiceNumber>2002043</InvoiceNumber>
    <InvoiceDate>2002-11-12</InvoiceDate>
    <PONumber>2002-0967</PONumber>
    <DueDate>2002-12-12</DueDate>
  </Header>
  <ShipTo>
    <RecordID>SHP</RecordID>
    <ShipToName>Corner Drug and Sundries</ShipToName>
    <ShipToStreet1>14 Main Street</ShipToStreet1>
    <ShipToCity>Wichita</ShipToCity>
    <ShipToStateOrProvince>KS</ShipToStateOrProvince>
    <ShipToPostalCode>67201</ShipToPostalCode>
  </ShipTo>
  <LineItemGroup>
    <LineItem>
      <RecordID>LIN</RecordID>
      <ItemID>HCVAN</ItemID>
      <ItemQuantity>24</ItemQuantity>
      <UnitPrice>2.59</UnitPrice>
      <ExtendedPrice>62.16</ExtendedPrice>
    </LineItem>
    <ItemDescription>
      <RecordID>DSC</RecordID>
      <Description>
        Instant Hot Cocoa Mix - Vanilla flavor
      </Description>
    </ItemDescription>
  </LineItemGroup>
  <Summary>
    <RecordID>SUM</RecordID>
    <TotalAmount>62.16</TotalAmount>
    <NumberOfLines>1</NumberOfLines>
  </Summary>
</FlatInvoice>


XML to Flat File: Functionality and Operation

Requirements

The general requirement for this utility is to convert one or more XML instance documents, each representing a single logical document, into a single flat file. Here's a summary of the required functionality.

  • Inputs: A directory containing one or more XML instance documents, each in a single file. All have the same grammar. The second input is a file description document (as discussed in Chapter 6) that describes the XML documents and the flat file to be produced.

  • Processing: Each input Element corresponding to a record (as described in the file description document) is written to a record in the flat output file. Each of its child Elements corresponding to fields are written to the appropriate fields in the record. Data types are converted according to the field descriptions in the file description document.

  • Output: A flat file with fixed length fields and variable or fixed length records.

Running the Utility

This section provides instructions for running the XML to flat file conversion utility from the command line.

For Java:

java XMLToFlat InputDirectory OutputFile.dat FileDescription.XML

or

java XMLToFlat -h

For C++ on Win32:

XMLToFlat InputDirectory OutputFile.dat FileDescription.XML

or

XMLToFlat -h

Options follow the parameters except for the help option, which may be specified by itself.

Parameters:

  • First: Path specification of the input directory (required). Either a relative or an absolute path name may be specified. The trailing directory separator character is optional.

  • Second: File specification of the output flat file (required). The specification may include the full or relative path name. If no path name is specified, the file will be created in the current working directory. An extension may be specified, but if not specified the file will be created without one. If a file with the specified name already exists, it will be overwritten.

  • Third: File specification of the file description document (required). If no path name is specified, the file is assumed to reside in the current working directory. The full file name must be specified, but there is no restriction on the extension name.

Options:

  • -v (Validate): Validate the XML documents before processing. The documents are validated against the schema referenced in the document root Element.

  • -h (Help): Display a help message and exit without further processing.

Restrictions:

The restrictions are the same as those specified for the flat file to XML utility, plus the following.

  • As when flat files are the source, it is recommended (though not required) that field grammar Elements be specified in their record description Element in ascending order by offset. However, the sequence of fields in the input XML document must match the sequence defined in the record grammar.

  • The record identifier field Element need not be present in the input document. It will be converted if present, but the contents will be overwritten by the record identifier specified in the grammar.

  • Fields without contents in the input XML document are initialized to the fill character specified in the field's grammar.

  • If field definitions overlap one or more bytes in the record, the byte(s) are written from the last field specified in the grammar.

  • When writing fixed length records, the output record is initialized to ASCII spaces. If a different fill character is desired, it must be specified in a field grammar. If a byte in the record is not defined within a field, it is written with an ASCII space.

  • When writing variable length records, the output record is initialized to null characters and is not written beyond the end of the last field defined for the record. To use a fill character other than a null character, it must be specified in a field grammar. If a byte between fields that have content is not defined within a field grammar, it is written with a null character.

Sample Input and Output: Purchase Order

We'll use the cocoa purchase order for converting from XML to a flat file format. The particular orders we will produce for Big Daddy's system are similar to the invoices in that they contain two logical groups of records. The top-level group is the overall purchase order, containing a header record, a ship to record, and one or more line item groups. The line item group consists of a line item record and an item description record. Table 8.2 shows the logical layout of the purchase order file.

The three purchase order instance documents shown below follow the logical organization specified in Table 8.2.

Sample FlatPurchaseOrders01.xml
<?xml version="1.0" encoding="UTF-8"?>
<PurchaseOrder
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:noNamespaceSchemaLocation="FlatPurchaseOrder.xsd">
  <Header>
    <CustomerNumber>BQ003</CustomerNumber>
    <PONumber>AZ999345</PONumber>
    <PODate>2002-11-01</PODate>
    <RequestedDeliveryDate>2002-11-15</RequestedDeliveryDate>
  </Header>
  <ShipTo>
    <ShipToName>
      Yazoo Grocers - NE Distribution Center
    </ShipToName>
    <ShipToStreet1>12 Industrial Parkway, NW</ShipToStreet1>
    <ShipToCity>Portland</ShipToCity>
    <ShipToStateOrProvince>ME</ShipToStateOrProvince>
    <ShipToPostalCode>04101</ShipToPostalCode>
  </ShipTo>
  <LineItem>
    <Item>
      <ItemID>HCVAN</ItemID>
      <OrderedQty>12</OrderedQty>
      <UnitPrice>2.59</UnitPrice>
    </Item>
    <ItemDescription>
      <Description>
        Instant Hot Cocoa Mix - Vanilla flavor
      </Description>
    </ItemDescription>
  </LineItem>
  <LineItem>
    <Item>
      <ItemID>HCMIN</ItemID>
      <OrderedQty>24</OrderedQty>
      <UnitPrice>2.53</UnitPrice>
    </Item>
    <ItemDescription>
      <Description>
        Instant Hot Cocoa Mix - Mint flavor
      </Description>
    </ItemDescription>
  </LineItem>
</PurchaseOrder>

Table 8.2. Logical Layout for the Purchase Order

Group

Record

Record Tag

Field Number

Field Name

Offset

Length

Data Type

Description

PO

Header

HDR

1

Record Tag

0

3

Alphanumeric

Required; record identifier

2

Customer Number

3

20

Alphanumeric

Required; identifier assigned to the customer in our system

3

PO Number

23

20

Alphanumeric

Required; customer purchase order number

4

PO Date

43

8

Date

Required; date that the purchase order was issued

5

Requested Delivery Date

51

8

Date

Optional; date on which delivery of the ordered items is requested

PO

Ship To

SHP

1

Record Tag

0

3

Alphanumeric

Required; record identifier

2

Ship to Name

3

40

Alphanumeric

Required; name of the receiving location for the shipped order

3

Ship to Street 1

43

30

Alphanumeric

Required; first address line of the receiving location

4

Ship to Street 2

73

30

Alphanumeric

Optional; second address line of the receiving location

5

Ship to City

103

20

Alphanumeric

Required; city of the receiving location

6

Ship to State or Province

123

3

Alphanumeric

Required; state or province of the receiving location

7

Ship to Postal Code

126

10

Alphanumeric

Required; postal code of the receiving location

8

Ship to Country

136

3

Alphanumeric

Optional; country of the receiving location

PO/Line Item

Line Item

LIN

1

Record Tag

0

3

Alphanumeric

Required; record identifier

2

Item ID

3

20

Alphanumeric

Required; our identifier for the ordered item

3

Item Ordered Quantity

23

10

Decimal number, space filled

Required; the number of units ordered

4

Item Unit Price

23

10

Implied decimal, two places, zero filled

Optional; unit price in U.S. dollars

5

Extended Amount

43

10

Implied decimal, two places, zero filled

Optional; total amount due for the ordered item in U.S. dollars (unit price multiplied by the quantity ordered)

PO/Line Item

Item Description

DSC

1

Record Tag

0

3

Alphanumeric

Required; record identifier (record itself is optional)

2

Description

3

80

Alphanumeric

Required; description of the ordered item

Sample FlatPurchaseOrders02.xml
<?xml version="1.0" encoding="UTF-8"?>
<PurchaseOrder
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:noNamespaceSchemaLocation="FlatPurchaseOrder.xsd">
  <Header>
    <CustomerNumber>BQ003</CustomerNumber>
    <PONumber>AW999346</PONumber>
    <PODate>2002-11-01</PODate>
    <RequestedDeliveryDate>2002-11-15</RequestedDeliveryDate>
  </Header>
  <ShipTo>
    <ShipToName>
      Yazoo Grocers - SE Distribution Center
    </ShipToName>
    <ShipToStreet1>Dock 37</ShipToStreet1>
    <ShipToStreet2>3975 Hwy 75</ShipToStreet2>
    <ShipToCity>Atoka</ShipToCity>
    <ShipToStateOrProvince>OK</ShipToStateOrProvince>
    <ShipToPostalCode>74525</ShipToPostalCode>
  </ShipTo>
  <LineItem>
    <Item>
      <ItemID>HCVAN</ItemID>
      <OrderedQty>36</OrderedQty>
      <UnitPrice>2.59</UnitPrice>
    </Item>
    <ItemDescription>
      <Description>
        Instant Hot Cocoa Mix - Vanilla flavor
      </Description>
    </ItemDescription>
  </LineItem>
  <LineItem>
    <Item>
      <ItemID>HCMIN</ItemID>
      <OrderedQty>72</OrderedQty>
      <UnitPrice>2.53</UnitPrice>
    </Item>
    <ItemDescription>
      <Description>
        Instant Hot Cocoa Mix - Mint flavor
      </Description>
    </ItemDescription>
  </LineItem>
</PurchaseOrder>
Sample FlatPurchaseOrders03.xml
<?xml version="1.0" encoding="UTF-8"?>
<PurchaseOrder
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:noNamespaceSchemaLocation="FlatPurchaseOrder.xsd">
  <Header>
    <CustomerNumber>AY001</CustomerNumber>
    <PONumber>2002-0967</PONumber>
    <PODate>2002-11-09</PODate>
    <RequestedDeliveryDate>2002-11-14</RequestedDeliveryDate>
  </Header>
  <ShipTo>
    <ShipToName>Corner Drug and Sundries</ShipToName>
    <ShipToStreet1>14 Main Street</ShipToStreet1>
    <ShipToCity>Wichita</ShipToCity>
    <ShipToStateOrProvince>KS</ShipToStateOrProvince>
    <ShipToPostalCode>67201</ShipToPostalCode>
  </ShipTo>
  <LineItem>
    <Item>
      <ItemID>HCVAN</ItemID>
      <OrderedQty>24</OrderedQty>
      <UnitPrice>2.59</UnitPrice>
    </Item>
    <ItemDescription>
      <Description>
        Instant Hot Cocoa Mix - Vanilla flavor
      </Description>
    </ItemDescription>
  </LineItem>
</PurchaseOrder>

Figure 8.2 shows the resulting flat file.


Describing the File Formats

Describing the CSV file format in Chapter 7 was fairly simple due to the restrictions we placed on it. The most significant of these was that every row has the same logical format. We're going to allow more variation in our flat file formats. Applications that use flat files to import or export data typically support several different logical record formats and group these records into repeating units. We'll need to specify more information about the grammar of our flat files than we did with CSV files. In addition to data types and other characteristics of fields, we'll need to specify the details of all the record types as well as how the records are grouped together.

As with the CSV format, the flat file format's file description document has three major sections, each represented by an Element that is an immediate child of the root Element.

  • PhysicalCharacteristics: the flat file characteristics

    Figure 8.2 Sample Output Flat File (PurchaseOrders.Dat)
    [View full width]
             10        20        30        40        50        60        70        80
  90        100       110       120       130
         |         |         |         |         |         |         |         |
  |         |         |         |         |         |

HDRBQ003               AZ999345            2002110120021115
SHPYazoo Grocers - NE Distribution Center  12 Industrial Parkway, NW
               Portland            ME 04101
LINHCVAN                       1200000002590000000000
DSCInstant Hot Cocoa Mix - Vanilla flavor
LINHCMIN                       2400000002530000000000
DSCInstant Hot Cocoa Mix - Mint flavor
HDRBQ003               AW999346            2002110120021115
SHPYazoo Grocers - SE Distribution Center  Dock 37                       3975 Hwy 75
               Atoka               OK 74525
LINHCVAN                       3600000002590000000000
DSCInstant Hot Cocoa Mix - Vanilla flavor
LINHCMIN                       7200000002530000000000
DSCInstant Hot Cocoa Mix - Mint flavor
HDRAY001               2002-0967           2002110920021114
SHPCorner Drug and Sundries                14 Main Street
               Wichita             KS 67201
LINHCVAN                       2400000002590000000000
DSCInstant Hot Cocoa Mix - Vanilla flavor

  • XMLOutputCharacteristics: XML output characteristics, required only when converting to XML

  • Grammar: flat file grammar

Flat File Physical Characteristics

The PhysicalCharacteristics Element describes the file's physical characteristics. This Element is required for both the source and target conversion utilities.

Table 8.3 shows the child Elements of the PhysicalCharacteristics Element. All are required unless otherwise noted.

XML Output Characteristics

Characteristics governing the output XML documents are described in the XMLOutputCharacteristics Element. This Element is used only when converting from flat files to XML.

Table 8.4 shows the child Elements of the XMLOutputCharacteristics Element. All are required unless otherwise noted.

Flat File Grammar

The grammar of a flat file is described in the Grammar Element. Although the XML representation of groups of records in flat files may be fairly intuitive, a few diagrams might help make it clearer.

Figure 8.3 shows a typical stream of records in a flat file, using our cocoa invoice as an example. For brevity only the record tags appear in the figure.

Figure 8.3. Record Stream in the Invoice File

If we look only at the records we can't for certain deduce much about the logical structure of a document. We would probably suspect that the HDR record started a new document and that perhaps the LIN and DSC records were a repeating group. However, we don't know for certain just by looking at the document; we must verify our suspicions by consulting the file specification or the application designer. For our purposes, we use Table 8.1 as our specification. This allows us to interpret the stream as shown in Figure 8.4.

Figure 8.4. Record Stream in the Invoice File, with Groups Added

Figure 8.4, in essence, shows what is known as a syntax tree. Figure 8.5 converts the brackets into nodes in the tree. I show siblings at the same level in the diagram to make relationships more obvious.

Figure 8.5. Syntax Tree for the Invoice File

The logical structure in Figure 8.5 now finally starts to look like something we might see in XML. All we have to do to make the transformation complete is to change the text from record identifiers and descriptions to XML Element names (Figure 8.6).

Figure 8.6. Invoice Document in XML

Table 8.3. Child Elements of the PhysicalCharacteristics Element

Child Element

Child Element

Attribute

Schema Data Type

Description

Allowable Values, Restrictions, or Comments

RecordFormat

Specifies the physical format of the record

Only one of Fixed or Variable is allowed.

Fixed

Length

positiveInteger

Specifies the physical record length

Maximum value reflects restriction on record length as noted in restrictions list in text.

Variable

RecordTerminator

union of U, W, and hexBinary

Designates a UNIX-style line feed, Windows-style carriage return and line feed pair, or a hexadecimal value

U, W, or a two-character hexadecimal number from 00 through FF representinga single byte.

TagInfo

Specifies the location of the record identifier within the record

The tag contents will be interpreted as an alpha-numeric string, with leading and trailing white-space removed. Must be the same offset and length for every record type.

Offset

nonNegativeInteger

Specifies the offset from zero in bytes for the first position of the tag

Maximum value reflects restriction on record length as noted in restrictions list in text.

Length

positiveInteger

Specifies the length of the tag in bytes

Maximum value reflects restriction on field length as noted in restrictions list in text.

Table 8.4. Child Elements of the XMLOutputCharacteristics Element

Child Element

Attribute

Schema Data Type

Description Description

Allowable Values, Restrictions, or Comments

SchemaLocationURL

value

anyURI

URL of the schema file for the output document. Will be written as the value of the root Element's noNamespaceSchemaLocation Attribute.

Optional. If not specified the noNamespaceSchemaLocation Attribute will not be written. An error will occur if output validation is requested and this Element is not present.

PartnerBreak

Information about a field that dictates a different trading partner when its content changes (for example, a customer number in the first field of the invoice).

Optional. Field contents are interpreted as an alphanumeric string and must be valid as a directory name for the operating system. If not specified, all output documents will be created in the output directory instead of creating a separate subdirectory for each trading partner.

Offset

nonNegativeInteger

Offset from zero in bytes for the first position of the field.

Maximum value reflects restriction on record length as noted in the restrictions list in the text.

Length

positiveInteger

Length of the field in bytes.

Maximum value reflects restriction on field length as noted in the restrictions list in the text.

Now the transformation is complete. However, one other diagram may be helpful in fully understanding the file description documents and how the utilities use them. The logical structure of the grammar of our invoice file exactly matches the structure of the XML representation of the invoice document (Figure 8.7). The Element names in the file description document are shown in boldface type, while the invoice Elements they specify are shown in italics. Note that we define each Element in the invoice document only once and don't repeat the GroupDescription for each occurrence of the LineItemGroup Element.

Figure 8.7. Grammar Description of the Invoice Document

For a more detailed discussion of the analysis of flat file grammars, refer to the High-Level Design Considerations section. Table 8.5 shows the details of the Grammar Element and its child Nodes. All are required unless noted. The indentation in the Element column shows the approximate hierarchical relationships. The Allowable Child Elements column lists the specific details of the hierarchy.

Table 8.6 shows the data types supported for the flat file format. To those we developed for the CSV file format in Chapter 7 we add a new numeric and a new date data type.

For all types, a runtime error occurs if Truncatable is false and the length of the XML Element contents exceeds the field length.

I should make a note here about truncating versus rounding fractional digits. In these utilities I always truncate and never round. I've had enough bad experiences with floating point arithmetic that I'm taking the easy way out and just truncating. If you need to round fractional digits, you can use an XSLT transformation or whatever means you use to put the data into the proper XML source format. Or, if you want to modify the source code, you can take an approach similar to the one I discuss in the Enhancements and Alternatives section at the end of the chapter.

Table 8.5. Flat File Grammar Characteristics in the Grammar Element

Element

Allowable Child Elements

Attribute

Schema Language Data Type

Description

Allowable Values, Restrictions, or Comments

Grammar

RecordDescription, GroupDescription

Describes the grammar of both the flat file and the corre-sponding XML representation.

The first child Element of the Grammar Element must be a RecordDescription Element. It may be followed by any combination of RecordDescription or GroupDescription Elements.

ElementName

NMTOKEN

Specifies the name of the document's root Element.

When creating XML documents, the specified name is assigned to the document's root Element. When creating a flat file, the input XML document's root Element must match this name. Maximum length reflects restriction on length of Element names.

TagValue

token

The value of the Header record's record identifier field.

Maximum length reflects restriction on field length. Do not include trailing spaces if the tag length is less than the length specified in the TagInfo Element.

GroupDescription

RecordDescription, GroupDescription

Describes the grammar of a group of records.

Any combination of RecordDescription or GroupDescription Elements can follow the first RecordDescription Element.

ElementName

NMTOKEN

Specifies the name of the Element representing the group.

Maximum length reflects restriction on length of Element names.

TagValue

token

The value of the record identifier field described for the first record in the group.

Maximum value reflects restriction on field length. Do not include trailing spaces if the tag length is less than the length specified in the TagInfo Element.

RecordDescription

FieldDescription

Describes the grammar of an individual record and the corresponding XML, a RecordDescription is required representation

A RecordDescription Element is required for each unique record type in the file.If a record type may appear at different for each position.

ElementName

NMTOKEN

Specifies the name of the Element representing a row.

Maximum length reflects restriction on length of Element names.

TagValue

token

The value of the record identifier field described by the TagInfo Element above.

Maximum length reflects restriction on field length. Do not include trailing spaces if the tag length is less than the length specified in the TagInfo Element.

FieldDescription

None

Describes the characteristics of a field in the flat file and the corresponding XML representation.

One FieldDescription Element is required for each field in the flat file record. If a range of characters within the record is not covered by a field description, they will be ignored for flat file source conversions and space filled for flat file target conversions.

ElementName

NMTOKEN

Specifies the name of the Element representing the field.

Maximum length reflects restriction on length of Element names.

FieldNumber

positiveInteger

Specifies the number of the field, starting at one.

Maximum value reflects restriction on the number of fields per record.

DataType

token

Specifies the data type of the field in the flat file.

The supported data types developed in this chapter are shown in Table 8.6. The Grammar data type code values are used.

Offset

nonNegative-Integer

Specifies the offset from zero in bytes for the first position of the field.

Maximum value reflects restriction on record length.

Length

positiveInteger

Specifies the length of the field in bytes.

Maximum value reflects restriction on field length.

Truncatable

boolean

Indicates whether or not truncation is permitted. See comments regarding truncation in Table 8.6.

Optional, defaults to false.

FillCharacter

union of single character string and hex-Binary

When converting to flat files as the target, the field will be padded with this character if the source XML Element content is missing or shorter than the field length.

Optional, defaults to an ASCII space character. A single literal character or a two-character hexadecimal number from 00 through FF representing a single byte may be specified.

Table 8.6. Flat File Data Types

Flat File Data Type

Grammar Data Type Code

Schema Data Type

Actions with Flat File as Source

Actions with Flat File as Target

Actions with Flat File as Target if Truncatable Is True

Alphanumeric

AN

string

Leading and trailing white-space (any character with an integer value less than or equal to a space character) is trimmed. All other white-space within the string is preserved.

If the source is shorter than the field length, the field is left-justified and filled to the right with the fill character.

The string is right-truncated to the field length.

Real number

R

decimal

Leading zeroes and leading plus signs are removed. All whitespace is trimmed.

The number is right-justified within the field. Leading characters are set according to the fill character. If the fill character is a zero, the minus sign if present is placed in the left-most position. For all other fill characters the minus sign immediately precedes the most significant digit.

Fractional digits to the right of the decimal point are truncated until the Element contents are equal to the field length. An error occurs if digits to the left of the decimal exceed the field length.

Implied decimal number

Nx, where x represents the number of implied decimal places

decimal

Leading zeroes and leading plus signs are removed. All whitespace is trimmed.

The number is right-justified within the field. If the number source decimal number exceeds x, the number is right-truncated to x fractional digits. Zeroes are added as fractional digits if the source number has fewer than x fractional digits. Leading characters are set according to the fill character. If the fill character is a zero, the sign character is placed in the left-most position. For all other fill characters the sign character immediately precedes the most significant digit.

Ignored, not truncatable.

Date in YYYYMMDD format

DYYYYMMDD

date

N/A

The date is left-justified within the field and filled with the specified fill character if the field is longer than 8 characters.

Ignored, not truncatable.

Date in MM/DD/YYYY format

DMMsDDsYYYY

date

Month and day may be either one or two digits each.

The date is left-justified within the field and filled with the specified fill character if the field is longer than 10 characters.

Ignored, not truncatable.

Example File Description Documents

Here are the file description documents for the flat file invoice and purchase order examples.

Sample InvoiceFlatSourceDescription.xml
<?xml version="1.0" encoding="UTF-8"?>
<FlatSourceFileDescription
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:noNamespaceSchemaLocation="FlatSourceFileDescription.xsd">
  <PhysicalCharacteristics>
    <RecordFormat>
      <Variable RecordTerminator="W"/>
    </RecordFormat>
    <TagInfo Offset="0" Length="3"/>
  </PhysicalCharacteristics>
  <XMLOutputCharacteristics>
    <SchemaLocationURL value="FlatInvoice.xsd"/>
    <PartnerBreak Offset="3" Length="20"/>
  </XMLOutputCharacteristics>
  <Grammar ElementName="FlatInvoice" TagValue="HDR">
    <RecordDescription ElementName="Header" TagValue="HDR">
      <FieldDescription FieldNumber="1"
          ElementName="RecordID" DataType="AN"
          Offset="0" Length="3"/>
      <FieldDescription FieldNumber="2"
          ElementName="CustomerNumber" DataType="AN"
          Offset="3" Length="20"/>
      <FieldDescription FieldNumber="3"
          ElementName="InvoiceNumber" DataType="AN"
          Offset="23" Length="20"/>
      <FieldDescription FieldNumber="4"
          ElementName="InvoiceDate" DataType="DYYYYMMDD"
          Offset="43" Length="8"/>
      <FieldDescription FieldNumber="5"
          ElementName="PONumber" DataType="AN"
          Offset="51" Length="20"/>
      <FieldDescription FieldNumber="6"
          ElementName="DueDate" DataType="DYYYYMMDD"
          Offset="71" Length="8"/>
    </RecordDescription>
       <RecordDescription ElementName="ShipTo" TagValue="SHP">
         <FieldDescription FieldNumber="1"
             ElementName="RecordID" DataType="AN"
             Offset="0" Length="3"/>
         <FieldDescription FieldNumber="2"
             ElementName="ShipToName" DataType="AN"
             Offset="3" Length="40"/>
         <FieldDescription FieldNumber="3"
             ElementName="ShipToStreet1" DataType="AN"
             Offset="43" Length="30"/>
         <FieldDescription FieldNumber="4"
             ElementName="ShipToStreet2" DataType="AN"
             Offset="73" Length="30"/>
         <FieldDescription FieldNumber="5"
             ElementName="ShipToCity" DataType="AN"
             Offset="103" Length="20"/>
         <FieldDescription FieldNumber="6"
            ElementName="ShipToStateOrProvince"
            DataType="AN" Offset="123" Length="3"/>
         <FieldDescription FieldNumber="7"
             ElementName="ShipToPostalCode" DataType="AN"
             Offset="126" Length="10"/>
         <FieldDescription FieldNumber="8"
             ElementName="ShipToCountry"
             DataType="AN" Offset="136" Length="3"/>
       </RecordDescription>
       <GroupDescription ElementName="LineItemGroup"
           TagValue="LIN">
         <RecordDescription ElementName="LineItem" TagValue="LIN">
           <FieldDescription FieldNumber="1"
               ElementName="RecordID" DataType="AN"
               Offset="0" Length="3"/>
           <FieldDescription FieldNumber="2"
               ElementName="ItemID" DataType="AN"
               Offset="3" Length="20"/>
           <FieldDescription FieldNumber="3"
               ElementName="ItemQuantity" DataType="R"
               Offset="23" Length="10"/>
           <FieldDescription FieldNumber="4"
               ElementName="UnitPrice" DataType="N2"
               Offset="33" Length="10"/>
           <FieldDescription FieldNumber="5"
               ElementName="ExtendedPrice" DataType="N2"
               Offset="43" Length="10"/>
         </RecordDescription>
         <RecordDescription ElementName="ItemDescription"
             TagValue="DSC">
           <FieldDescription FieldNumber="1"
               ElementName="RecordID" DataType="AN"
               Offset="0" Length="3"/>
           <FieldDescription FieldNumber="2"
               ElementName="Description" DataType="AN"
               Offset="3" Length="80"/>
         </RecordDescription>
       </GroupDescription>
       <RecordDescription ElementName="Summary" TagValue="SUM">
         <FieldDescription FieldNumber="1"
             ElementName="RecordID" DataType="AN"
             Offset="0" Length="3"/>
         <FieldDescription FieldNumber="2"
             ElementName="TotalAmount" DataType="N2"
             Offset="3" Length="10"/>
         <FieldDescription FieldNumber="3"
             ElementName="NumberOfLines" DataType="N0"
             Offset="13" Length="10"/>
       </RecordDescription>
     </Grammar>
   </FlatSourceFileDescription>
Sample PurchaseOrderFlatTargetDescription.xml
<?xml version="1.0" encoding="UTF-8"?>
<FlatTargetFileDescription
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:noNamespaceSchemaLocation=
        "FlatTargetFileDescription.xsd">
  <PhysicalCharacteristics>
    <RecordFormat>
      <Variable RecordTerminator="W"/>
    </RecordFormat>
    <TagInfo Offset="0" Length="3"/>
  </PhysicalCharacteristics>
  <Grammar ElementName="PurchaseOrder" TagValue="HDR">
    <RecordDescription TagValue="HDR" ElementName="POHeader">
      <FieldDescription FieldNumber="1"
          ElementName="RecordID" DataType="AN"
          Offset="0" Length="3"/>
      <FieldDescription FieldNumber="2"
          ElementName="CustomerNumber" DataType="AN"
          Offset="3" Length="23"/>
      <FieldDescription FieldNumber="3"
          ElementName="PONumber" DataType="AN"
          Offset="23" Length="20"/>
      <FieldDescription FieldNumber="4"
          ElementName="PODate" DataType="DYYYYMMDD"
          Offset="43" Length="8"/>
      <FieldDescription FieldNumber="5"
          ElementName="RequestedDeliveryDate" DataType="DYYYYMMDD"
          Offset="51" Length="8"/>
    </RecordDescription>
    <RecordDescription TagValue="SHP" ElementName="ShipTo">
      <FieldDescription FieldNumber="1"
          ElementName="RecordID" DataType="AN"
          Offset="0" Length="3"/>
      <FieldDescription FieldNumber="2"
          ElementName="ShipToName" DataType="AN"
          Offset="3" Length="40" Truncatable="true"/>
      <FieldDescription FieldNumber="3"
          ElementName="ShipToStreet1" DataType="AN"
          Offset="43" Length="30" Truncatable="true"/>
      <FieldDescription FieldNumber="4"
          ElementName="ShipToStreet2" DataType="AN"
          Offset="73" Length="30" Truncatable="true"/>
      <FieldDescription FieldNumber="5"
          ElementName="ShipToCity" DataType="AN"
          Offset="103" Length="20"/>
      <FieldDescription FieldNumber="6"
          ElementName="ShipToStateOrProvince" DataType="AN"
          Offset="123" Length="3"/>
      <FieldDescription FieldNumber="7"
          ElementName="ShipToPostalCode" DataType="AN"
          Offset="126" Length="10"/>
      <FieldDescription FieldNumber="8"
          ElementName="ShipToCountry" DataType="AN"
          Offset="136" Length="3"/>
    </RecordDescription>
    <GroupDescription ElementName="LineItem" TagValue="LIN">
      <RecordDescription TagValue="LIN" ElementName="Item">
        <FieldDescription FieldNumber="1"
            ElementName="RecordID" DataType="AN"
            Offset="0" Length="3"/>
        <FieldDescription FieldNumber="2"
            ElementName="ItemID" DataType="AN"
            Offset="3" Length="20"/>
        <FieldDescription FieldNumber="3"
            ElementName="OrderedQty" DataType="R"
            Offset="23" Length="10" FillCharacter=" "/>
        <FieldDescription FieldNumber="4"
            ElementName="UnitPrice" DataType="N2"
            Offset="33" Length="10" FillCharacter="0"/>
        <FieldDescription FieldNumber="5"
            ElementName="ExtendedAmount" DataType="N2"
            Offset="43" Length="10" FillCharacter="0"/>
      </RecordDescription>
      <RecordDescription TagValue="DSC"
          ElementName="ItemDescription">
        <FieldDescription FieldNumber="1"
            ElementName="RecordID" DataType="AN"
            Offset="0" Length="3"/>
        <FieldDescription FieldNumber="2"
            ElementName="Description" DataType="AN"
            Offset="3" Length="80" Truncatable="true"/>
      </RecordDescription>
    </GroupDescription>
  </Grammar>
</FlatTargetFileDescription>


Schema Examples

The sample schemas for the flat file examples are very similar to the CSV examples. The primary differences should be evident from examining the primary differences between the instance documents. In the CSV file representations all data is in a sequence of identically formatted Elements, each representing a single row in the CSV file. In the flat file representations the data is broken up into separate record formats, and the line item detail and description records are grouped together.

Invoice Schema (FlatInvoice.xsd)
<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
    elementFormDefault="unqualified">
  <xs:element name="FlatInvoice">
    <xs:complexType>
      <xs:sequence>
        <xs:element name="Header">
          <xs:complexType>
            <xs:sequence>
              <xs:element name="RecordID" type="xs:token"/>
              <xs:element name="CustomerNumber" type="xs:token"/>
              <xs:element name="InvoiceNumber" type="xs:token"/>
              <xs:element name="InvoiceDate" type="xs:date"/>
              <xs:element name="PONumber" type="xs:token"/>
              <xs:element name="DueDate" type="xs:date"/>
            </xs:sequence>
          </xs:complexType>
        </xs:element>
        <xs:element name="ShipTo">
          <xs:complexType>
            <xs:sequence>
              <xs:element name="RecordID" type="xs:token"/>
              <xs:element name="ShipToName" type="xs:token"/>
              <xs:element name="ShipToStreet1" type="xs:token"/>
              <xs:element name="ShipToStreet2" type="xs:token"
                  minOccurs="0"/>
              <xs:element name="ShipToCity" type="xs:token"/>
              <xs:element name="ShipToStateOrProvince"
                  type="xs:token"/>
              <xs:element name="ShipToPostalCode"
                  type="xs:token"/>
              <xs:element name="ShipToCountry" type="xs:token"
                  minOccurs="0"/>
            </xs:sequence>
          </xs:complexType>
        </xs:element>
        <xs:element name="LineItemGroup" maxOccurs="unbounded">
          <xs:complexType>
            <xs:sequence>
              <xs:element name="LineItem">
                <xs:complexType>
                  <xs:sequence>
                    <xs:element name="RecordID" type="xs:token"/>
                    <xs:element name="ItemID" type="xs:token"/>
                    <xs:element name="ItemQuantity"
                        type="xs:decimal"/>
                    <xs:element name="UnitPrice"
                        type="xs:decimal"/>
                    <xs:element name="ExtendedPrice"
                        type="xs:decimal"/>
                  </xs:sequence>
                </xs:complexType>
              </xs:element>
              <xs:element name="ItemDescription">
                <xs:complexType>
                  <xs:sequence>
                    <xs:element name="RecordID" type="xs:token"/>
                    <xs:element name="Description"
                        type="xs:token"/>
                  </xs:sequence>
                </xs:complexType>
              </xs:element>
            </xs:sequence>
          </xs:complexType>
        </xs:element>
        <xs:element name="Summary">
          <xs:complexType>
            <xs:sequence>
              <xs:element name="RecordID" type="xs:token"
                  minOccurs="0"/>
              <xs:element name="TotalAmount" type="xs:decimal"/>
              <xs:element name="NumberOfLines"
                  type="xs:nonNegativeInteger" minOccurs="0"/>
            </xs:sequence>
          </xs:complexType>
        </xs:element>
      </xs:sequence>
    </xs:complexType>
  </xs:element>
</xs:schema>
Purchase Order Schema (FlatPurchaseOrder.xsd)
<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
    elementFormDefault="unqualified">
  <xs:element name="PurchaseOrder">
    <xs:complexType>
      <xs:sequence>
        <xs:element name="Header">
          <xs:complexType>
            <xs:sequence>
              <xs:element name="CustomerNumber" type="xs:token"/>
              <xs:element name="PONumber" type="xs:token"/>
              <xs:element name="PODate" type="xs:date"/>
              <xs:element name="RequestedDeliveryDate"
                  type="xs:date"/>
            </xs:sequence>
          </xs:complexType>
        </xs:element>
        <xs:element name="ShipTo">
          <xs:complexType>
            <xs:sequence>
              <xs:element name="ShipToName" type="xs:string"/>
              <xs:element name="ShipToStreet1" type="xs:string"/>
              <xs:element name="ShipToStreet2" type="xs:string"
                  minOccurs="0"/>
              <xs:element name="ShipToCity" type="xs:string"/>
              <xs:element name="ShipToStateOrProvince"
                  type="xs:token"/>
              <xs:element name="ShipToPostalCode"
                  type="xs:string"/>
              <xs:element name="ShipToCountry" type="xs:token"
                  minOccurs="0"/>
            </xs:sequence>
          </xs:complexType>
        </xs:element>
        <xs:element name="LineItem" maxOccurs="unbounded">
          <xs:complexType>
            <xs:sequence>
              <xs:element name="Item">
                <xs:complexType>
                  <xs:sequence>
                    <xs:element name="ItemID">
                      <xs:simpleType>
                        <xs:restriction base="xs:token">
                          <xs:enumeration value="HCMIN"/>
                          <xs:enumeration value="HCVAN"/>
                          <xs:enumeration value="HCMOC"/>
                          <xs:enumeration value="HCDUC"/>
                        </xs:restriction>
                      </xs:simpleType>
                    </xs:element>
                    <xs:element name="OrderedQty"
                        type="xs:decimal"/>
                    <xs:element name="UnitPrice"
                        type="xs:decimal"/>
                    <xs:element name="ExtendedAmount"
                        type="xs:decimal" minOccurs="0"/>
                  </xs:sequence>
                </xs:complexType>
              </xs:element>
              <xs:element name="ItemDescription" minOccurs="0">
                <xs:complexType>
                  <xs:sequence>
                    <xs:element name="Description"
                        type="xs:string"/>
                  </xs:sequence>
                </xs:complexType>
              </xs:element>
            </xs:sequence>
          </xs:complexType>
        </xs:element>
      </xs:sequence>
    </xs:complexType>
  </xs:element>
</xs:schema>


High-Level Design Considerations

Grammar Analysis and Description

While our discussion of CSV grammars focused strictly on the grammar of a row, our consideration of flat file grammars will focus on the logical structure of the file. Flat file record formats are usually not complex (at least in the way we will deal with them).

Before we look at the BNF, let's look at a few examples. Our sample invoice and purchase order files have the following record organization (except that the purchase order doesn't have a summary record). To focus on the structure, I use just the record IDs, and indentation shows the groupings.

HDR
SHP
 LIN
 DSC
SUM

The following example uses somewhat arbitrary record tags.

A010
B020
 C030
 D040
 E050
  F060
  G070
 H080
I090
J100

The thing that flat file formats usually have in common, although there are a few exceptions, is that each record in the hierarchy has a unique identifier. Individual record formats are usually not used in more than one location in the logical hierarchy of the file. With this characteristic it would be quite easy to just use the record identifier as an index into the record grammar. However, doing this would impose for all cases the restriction that each record format and location in the hierarchy be unique. This might make the utility unusable for those few odd designs that do repeat the same record at different places in the hierarchy. In addition, if possible, we would like to generalize our approach to the problem so that it might apply to other formats using groups, such as EDI. (EDI does not observe this unique record ID restriction.)

If we abstract the essential features of flat file organization, we see that the following BNF adequately describes the abstract essence of all our cases.

Flat File Grammar
FlatFile ::= document+
document ::= group
group ::= record (record | group)*

What this says, basically, is that a flat file contains one or more logical documents. Each logical document is a group of records. A group has a start record and contains zero or more records or other groups. The group production is a fairly simple recursive definition, but it is the key to describing flat file hierarchies. It has a great deal of descriptive power. All the examples we have covered so far can be described by this simple grammar. As we'll see in Chapter 9, most EDI formats also can be described by this grammar. The most interesting result of this analysis is that, as a recursive definition, it suggests that we might use a recursive algorithm for processing. We'll see that in later sections on detail design.

We complete the BNF by describing the grammar of a flat file record.

Flat File Record Grammar
record ::= (field* recordID field+) | (field+ recordID field*)
recordID ::= field, with an enumerated set of unique values and
    the same offset and length for all records in the file
field ::= a string of bytes with a specified offset and length

These three productions basically say that a record contains two or more fields, a field is described by its offset and length, and one of the fields must contain an identifier for the record. The record identifier field must have the same offset and length in all the file's record types. Again, for the final productions strict BNF fails us and we resort to natural language. But I think you get the idea.

Of course, the grammar of any particular flat file type is more complicated than these abstractions. However, in order to build general purpose converters we must find an abstract grammar that is general enough to handle most, if not all, of our cases. I think you'll find that this grammar is sufficiently general.

File Description Document Schemas

As with CSV formats there are four schemas involved in describing flat files.

  1. FlatSourceFileDescription.xsd: This schema is for file description documents that describe conversions in which the source format is a flat file.

  2. FlatTargetFileDescription.xsd: This schema is for file description documents that describe conversions in which the target format is a flat file.

  3. FlatCommonFileDescription.xsd: This type library schema is used by the two previous schemas.

  4. BBCommonFileDescription.xsd: This schema specifies types used in all the Babel Blaster conversions. We are adding a few things to the similar schema introduced in Chapter 7.

To prevent further cluttering of this chapter with XML code listings, I'll just discuss the schemas in general terms and list only some particularly interesting snippets. Again, the schemas are available on the Web.

The schemas for the source and target file description documents are again basic shells. The common type library for flat files is a bit more interesting. It uses a few schema language features that I noted aren't normally used in schemas for common business documents. Two of these are shown in the xs:complexType Element that defines the FlatGrammarType type.

FlatGrammarType complexType in FlatCommonFileDescription.xsd
  <xs:complexType name="FlatGrammarType">
    <xs:annotation>
      <xs:documentation>
        Describes the grammar of the flat file
      </xs:documentation>
    </xs:annotation>
    <xs:sequence>
      <xs:element name="RecordDescription"
          type="FlatRecordType"/>
      <xs:choice maxOccurs="unbounded">
        <xs:element name="RecordDescription"
            type="FlatRecordType"/>
        <xs:element name="GroupDescription"
            type="FlatGroupType"/>
      </xs:choice>
    </xs:sequence>
    <xs:attributeGroup ref="ElementAndTagType"/>
 </xs:complexType>

The first schema language feature to note is the xs:choice Element, representing a choice content model. We use it because it maps very well to the BNF production for our group grammar:

group ::= record (record | group)*

Again, this production says that a group consists of a record followed by zero or more records or groups. The "record or group" concept is logically a choice. In BNF we use the vertical bar symbol (|) to represent it, and it maps directly to the choice content model in schema language. While the choice content model is used very seldom in business document schemas, its use here is entirely appropriate.

The second feature to note is the use of the xs:attributeGroup Element. We use the ElementName and TagValue Attributes on three different Elements in the schema, so it makes sense just to code them as a reusable Attribute group. The definition is shown below.

ElementAndTagType attributeGroup in FlatCommonFileDescription.xsd
  <xs:attributeGroup name="ElementAndTagType">
    <xs:annotation>
      <xs:documentation>
        ElementName and TagValue
      </xs:documentation>
    </xs:annotation>
    <xs:attribute name="ElementName" type="xs:NMTOKEN"
        use="required"/>
    <xs:attribute name="TagValue" type="TagValueType"
        use="required"/>
  </xs:attributeGroup>


Flat File to XML: Detail Design

Chapter 7, in dealing with CSV files, gave a good introduction to how the architecture is implemented. There are only a few significant differences between processing flat files and processing CSV files. So, while I'll present the pseudocode that shows the logic of the Java and C++ implementations for flat file conversion, the discussion will focus on the aspects that are different from CSV processing.

Main Program

As discussed in Chapter 6, in order to develop reusable modules that can be linked together into programs and not just cobbled together in scripts, we put the main processing logic in a routine that is called from a shell main program. The shell main program functions, described next, are essentially identical to the CSV main routine.

Logic for the Shell Main Routine for Flat to XML
Arguments:
  Input File Name
  Output Directory
  File Description XML Document

Options:
  Validate output
  Help

Validate and process command line arguments
IF help option specified
  Display help message
  Exit
ENDIF
Create new FlatSourceConverter object, passing:
    Validation option
    Output Directory
    File Description XML Document
Call FlatSourceConverter processFile method, passing
    the input file name
Display completion message

FlatSourceConverter Class (Extends SourceConverter)

Overview

The FlatSourceConverter is the main driver for the actual conversion. It inherits all the attributes and methods of its base classes, the SourceConverter and Converter classes (see Chapter 6).

Attributes:

  • FlatRecordReader Object

  • Array of Strings Partner Array

  • Integer Partner Break Field Offset

  • Integer Partner Break Field Length

  • String Saved Partner ID

Methods:

  • Constructor

  • processFile

  • processGroup (added to SourceConverter base class)

  • testPartnerBreak

Methods
Constructor

The constructor method for our FlatSourceConverter object sets up that object as well as the FlatRecordReader object.

Logic for the FlatSourceConverter Constructor Method
Arguments:
  Boolean Validation option
  String Output Directory
  String File Description Document Name

Call base class constructor
Initialize Partner Array
Call loadFileDescriptionDocument from passed File Description
    Document Name
Schema Location URL <- Call File Description Document's
    getElementsByTagName on "SchemaLocationURL", then
    getAttribute on "value"
IF Schema Location URL is null and Validation is true
  Throw Exception
ENDIF
Partner Break Field Offset <- 0
Partner Break Field Length <- 0
NodeList Temp <- call File Description Document's
    getElementsByTagName for "PartnerBreak"
IF (Temp length = 1)
  Partner Break Element <- Temp NodeList item(0)
  Partner Break Offset <- call Partner Break Element's
getAttribute
      for "Offset", and convert to integer
  Partner Break Length <- call Partner Break Element's
      getAttribute for "Length", and convert to integer
ENDIF
Initialize Saved Partner ID
Create FlatRecordReader object, passing:
    File Description Document
processFile

The main processing is driven by the FlatSourceConverter's processFile method. This method converts one input flat file into one or more output XML documents based on the input parameters.

Logic for the FlatSourceConverter processFile Method
Arguments:
  String Input File Name

Returns:
  Status or throws exception

Output Directory Path <- Base Directory + directory separator
Initialize Output Document to null
Initialize Sequence Number
Header Record Tag <- Get Grammar Element's "TagValue" Attribute
Open input file
Call FlatRecordReader's setInputStream method
Record Length <- Call FlatRecordReader's readRecord method
Record Tag <- Call FlatRecordReader's getRecordType method
IF (Record Tag != Header Record Tag)
  Return error or throw exception
ENDIF
DO while Record Length => 0
  Partner Break <- Call testPartnerBreak
  IF Partner Break = true
    Output Directory Path <- Base Directory +
        Partner ID + directory separator
    Lookup Partner in Partner Array
    IF Partner is not in Array
      Create output directory from Output Directory Path
      Partner Array <- Add Partner ID
    ENDIF
  ENDIF
  Create new Output Document
  Increment Sequence Number, and pad with leading zeroes
      to three digits
  Output File Path <- Output directory path + Root Element
      Name + Sequence Number + ".xml"
  Call FlatRecordReader's setOutputDocument method for new
      Output Document
  Record Length <- Call processGroup to process the document,
      passing the Root Element and the Grammar Element
  IF (Record Length > 0)
    Header Tag <- Call FlatRecordReader's getRecordType method
    IF (Header Tag != Grammar Header Record Tag)
      Return error or throw exception
    ENDIF
  ENDIF
  Call saveDocument
ENDDO
Close input file
Display completion message with number of documents processed

There are several similarities between this processFile method and the CSVSourceConverter's processFile method. We do very similar processing for partner lookup, directory and file management, and saving documents. However, instead of processing records individually, we process a document as a whole using the processGroup method.

processGroup (Base Class SourceConverter Method)

We noted in the discussion of flat file grammars that the recursive definition for the group production lends itself to processing by a recursive algorithm. I also noted near the end of Chapter 2 that while we can often process XML using recursive algorithms, doing so doesn't always offer any advantages over nonrecursive approaches. However, for our purposes in this utility a recursive approach, implemented with the processGroup method, is quite appropriate and very powerful. It processes the first record in a group, then all the other records. If one of the records starts another logical group, the processGroup method calls itself.

Since both flat files and EDI files have the same logical group structures (at least when processing their XML representations in our architecture) we can use the same method for processing them. So, although we introduce the processGroup method in this chapter, we're actually adding it to the SourceConverter base class.

Logic for the FlatSourceConverter processGroup Method
Arguments:
  DOM Node Parent
  DOM Element Group Grammar

Returns
  RecordLength of last input record read, or EOF

Group Element Name <- Group Grammar getAttributeValue
    For "ElementName"
Group Element <- createElement using Group Element Name
Parent Element <- appendNode Group Element
Grammar Element Name <- Record Grammar getNodeName
IF Grammar Element Name = "Grammar"
       and Schema Location URL is not NULL
       Create namespace Attribute for SchemaInstance and append
           to Group Element
       Create noNamespaceSchemaLocation Attribute and append
           to Group Element
ENDIF
Child Node <- Get Grammar Element's firstChild
DO while Child Node is not an Element Node
  Child Node <- nextSibling
ENDDO
Record Grammar Element <- Child Node
Record Element Name <- Record Grammar Element's
    getAttributeValue For "ElementName"
//  Process the first record in the group
Call RecordReader's parseRecord
Call RecordReader's toXML
Call RecordReader's writeRecord
// This advance in the grammar makes sure that we don't repeat
//  the starting record of the group
Record Grammar Element <- Get next Record Element from
    Group Grammar
Grammar Record Tag <- Record Grammar getAttribute for
    "TagValue"
Record Length <- call RecordReader's readRecord
DO until end of file
  Record Tag <- call RecordReader's getRecordTag
  DO until Grammar Record Tag = Record Tag
    Record Grammar Element <- Get next Record Element from
        Group Grammar
    IF Record Grammar Element is NULL
      return Record Length //  This record is not part of the group
    ENDIF
  ENDDO
  Grammar Element Name <- Record Grammar getNodeName
  IF Grammar Element Name = "GroupDescription"
    Do recursive call of processGroup
  ELSE
    Record Element Name <- Record Grammar getAttributeValue for
        "ElementName"
    Call RecordReader's parseRecord
    Call RecordReader's toXML
    Call RecordReader's writeRecord
    Record Length <- call RecordReader's readRecord
  ENDIF
ENDDO
Return Record Length

The logic is mostly straightforward, but it may help to review the recursive and termination cases. The first time processGroup is executed it is called from processFile after we have read the header record for a logical document. We pass processGroup the Document as the parent Node to which to append all the record Elements we create in processGroup. We also pass the complete Grammar Element from the file description document as the grammar for the group. After we have processed the header record, we advance to the next Element in the grammar. We then read the next record from the file. We advance Elements in the grammar until we match the record identifier of the record that we read from the file. If it is a normal record (that is, it doesn't start another group), we process it. However, if the matching grammar Element indicates that it starts a group (indicated by an Element name of "GroupDescription" instead of "RecordDescription"), we execute the recursive call. In this circumstance we pass the Document's root Element as the parent Element and the group's grammar Element (the GroupDescription Element) as the grammar. We thus start a new instance of processGroup and proceed as before.

We reach the termination case when we have read a record that is not part of the grammar of the current group. This case is recognized when we advance through all the grammar Elements that are children of the current group grammar and don't find a TagValue Attribute that matches the record's identifier. If we're processing a lower-level group in the logical document hierarchy (and a higher execution point on the stack), we exit processGroup and return to the previous iteration of processGroup. If the record is part of the group being processed by that iteration, we resume processing. However, if it isn't part of that group, we again exit. This holds true if the record we have read starts another iteration of the same type of record group. This continues until we finally exit back to processFile. If the current record is a header record, we save the current Document to disk and begin a new iteration of the while loop. However, if the record is not a header record, we have encountered a record that is either not defined for this type of document or is not where we have said it would be in the grammar. In that case we force an abnormal termination.

testPartnerBreak

This testPartnerBreak method serves the same purpose as the one in the CSVSourceConverter class, but the processing is a bit different. In addition to a few other minor differences, we trim trailing whitespace through the call to the Flat RecordReader's getFieldValue method.

Logic for the FlatSourceConverter testPartnerBreak Method
Arguments:
  None

Returns:
  Boolean - true if new partner and false if not

IF Partner Break Length is zero
  return false
ENDIF
PartnerID <- Call FlatRecordReader's getFieldValue method
    passing the Partner Break Offset and Partner Break Length
IF PartnerID = Saved Partner ID
  return false
ENDIF
Saved Partner ID <- Partner ID
Return true

FlatRecordReader Class (Extends RecordReader)

Overview

The FlatRecordReader introduces only a few new attributes but has some new and overridden methods. It inherits several attributes and methods from its RecordReader and RecordHandler base classes (see Chapter 6).

Attributes:

  • Integer Fixed Record Length

  • Integer Record ID Field Offset

  • Integer Record ID Field Length

Methods:

  • Constructor

  • getFieldValue

  • getRecordType

  • parseRecord

  • readRecord

  • readRecordFixedLength

Methods
Constructor

In the FlatRecordReader constructor method, we primarily retrieve values from the file description document that the reader needs for processing. The logic should look familiar by now.

Logic for the FlatRecordReader Constructor Method
Arguments:
  DOM Document File Description Document

Call RecordReader base class constructor, passing File
    Description Document
Record Format Element <- Get RecordFormat Element from File
    Description Document
Child Element <- Get first childNode from Record Format Element,
    advancing over non-Element Nodes
Fixed Record Length <- 0
IF Child Element NodeName = "Fixed"
  Fixed Record Length <- Call Child Element's getAttribute for
      "Length"
ELSE
  Record Terminator <- Call Child Element's getAttribute for
      "RecordTerminator"
  Call setTerminator to set the Record Terminators
ENDIF
Tag Info Element <- Get "RecordTagInfo" Element from File
    Description Document
Record ID Field Offset <- Call Tag Info's getAttribute for
    "Offset"
Record ID Field Length <- Call Tag Info's getAttribute for
    "Length"
getFieldValue

This method gets the complete value of a record's field with a single call. It is declared and implemented only in the FlatRecordReader class and isn't used in any of the other classes derived from the RecordReader class. The reason for this is that the other legacy formats (CSV and EDI) parse input records on a character-by-character basis. They save a field's characters to a DataCell object as they parse the record. In contrast, with the flat file format we parse records on a field-by-field basis, extracting fields from the record buffer based on offsets and lengths. This method is called from the FlatRecordReader's parseRecord method to perform that extraction. In addition, this method is designed to be used for extracting the record identifier, and until we have it we don't know how to parse the full record and load the DataCell Array.

The method is also called from the FlatRecordReader's testPartnerBreak method and the FlatRecordReader's getRecordType method. We call it for these special cases and not as a general purpose method for extracting field values from the input record buffer since in Java and C++ that operation takes only a couple lines of code. In addition to extracting the value, getFieldValue trims trailing whitespace and returns an error if the field is empty.

Logic for the FlatRecordReader getFieldValue Method
Arguments:
  Integer Field Offset
  Integer Field Length

Returns:
  Field value; throws exception or returns status

IF Field Offset > Record Buffer Length
  Return error
ENDIF
IF Field Offset + Field Length > Record Buffer Length
  Field Length <- Record Buffer Length - Field Offset
ENDIF
Field Value <- Extract from Record Buffer according to
    Passed Field Offset and adjusted Field Length
Field Value <- Trim trailing whitespace (<= ASCII space
    character)
IF length of Field Value = 0
  Return error
ENDIF
Return Field Value
getRecordType

This method extracts the value of the Record ID field from the input record buffer, trimming trailing whitespace. In the current implementation the field value is returned in its raw form, interpreted as alphanumeric string data.

Logic for the FlatRecordReader getRecordType Method
Arguments:
  None

Returns:
  Record ID tag value; throws exception or returns status

Record Tag Value <- call getFieldValue, passing Record ID Field
    Offset and Record ID Field Value
Return Record Tag Value or status
parseRecord

This method is not as interesting as the one in the CSVRecordReader because the grammar is not as complex. Essentially, we walk the FieldDescription Elements of the RecordDescription Element, extract the field contents according to their defined offsets and lengths, and create new DataCell objects.

Logic for the FlatRecordReader parseRecord method
Arguments:
  DOM Element Record Grammar

Returns:
  Error status or throw exception

Field Grammar NodeList <- call Record Grammar's
    getElementsByTagName for "FieldDescription"
DO until end of Field Grammar Node List
  Field Grammar Element <- Next item in Field Grammar NodeList
  Field Number <- call Field Grammar Element's getAttribute on
      "FieldNumber"
  Field Offset <- call Field Grammar Element's getAttribute on
      "Offset"
  Field Length <- call Field Grammar Element's getAttribute on
      "Length"
  IF Field Offset + 1 > Record Buffer Length
    Return success //  End of record
  ENDIF
  IF Field Offset + Field Length > Record Buffer Length
    Field Length <- Record Buffer Length - Field Offset
  ENDIF
  Field Length <- Record Buffer Length - Field Offset
  Field Value <- Extract from Record Buffer according to
      Field Offset and adjusted Field Length
  New Cell <- call RecordHandler's createDataCell method, passing
      Field Number and Field Grammar Element
  Call New Cell's putFieldValue method, passing Field Value
ENDDO
readRecord

This convenience method provides a way to protect calling routines from the variations in how we physically process records for flat files. It performs either a fixed or variable length read, depending on the physical characteristics gleaned from the file description document.

Logic for the FlatRecordReader readRecord Method
Arguments:
  None

Returns:
  Record Length or EOF

IF Fixed Record Length > 0
  Return call to readRecordFixedLength
ENDIF
Return call to base RecordReader's readRecordVariableLength
readRecordFixedLength

This method calls the language-specific routines for reading a fixed length record from a flat file.

Logic for the FlatRecordReader readRecordFixedLength Method
Arguments:
  DOM Element Record Grammar

Returns:
  Error status or throw exception

Clear Record Buffer
Record Buffer <- Call language-specific routine to read a fixed
    number of bytes from the input record stream
IF not EOF
  Return Fixed Record Length
ENDIF
Return EOF


XML to Flat File: Detail Design

Main Program

The shell main program functions are very similar to those of the XML to CSV converter.

Logic for the Shell Main Routine for XML to Flat
Arguments:
  Input Directory Name
  Output Flat File Name
  File Description Document Name

Options:
  Validate input
  Help

Validate and process command line arguments
IF help option specified
  Display help message
  Exit
ENDIF
Open output file
Create new FlatTargetConverter object, passing:
    Output Stream
    File Description Document Name
Set up implementation dependent DOM environment for loading,
    parsing, and validating input documents
Open input directory
Get first file from input directory
DO for all files in input directory
  Input Document <- Load input file, validating it if requested
  Call FlatTargetConverter processDocument method, passing
      the Input Document
  Increment number of documents processed
ENDDO
Close output file
Display completion message with number of documents processed

FlatTargetConverter Class (Extends TargetConverter)

Overview

The FlatTargetConverter again is very similar to the CSVTargetConverter. However, like the FlatSourceConverter, we use a recursive algorithm to process the logical record groups in input documents. We use the base TargetConverter's processGroup method, described in this subsection.

Attributes:

  • None

Methods:

  • Constructor

  • processDocument

  • processGroup (base class method)

Methods
Constructor

The constructor method for our FlatTargetConverter object sets up that object and the FlatRecordWriter object.

Logic for the FlatTargetConverter Constructor Method
Arguments:
  Output Stream Flat Output file
  File Description Document Name

Call base class constructor, passing File
    Description Document Name
Create FlatRecordWriter object, passing:
    File Description DOM Document and Output Stream
processDocument

The bulk of the processing is performed in the FlatTargetConverter's processDocument and processGroup methods. This method converts one input XML Document and writes it to the flat file output stream based on the input parameters. Most of the actual work is done in the processGroup method.

Logic for the FlatTargetConverter processDocument Method
Arguments:
  DOM Document Input Document

Returns:
  Error status or throws exception

Root Element <- Get Document's documentElement attribute
Root Element Name <- Get Root Element's tagName attribute
IF Root Element Name != Grammar Root Element Name
  Return error
ENDIF
Call processGroup, passing Root Element and Grammar Element
Return success
processGroup (Base Class TargetConverter Method)

As with the processGroup method of the SourceConverter we use a recursive algorithm, but the details are a little bit different for the termination case. This method is used for both flat file and EDI conversions.

Logic for the FlatTargetConverter processGroup Method
Arguments:
  DOM Element Parent Element
  DOM Element Group Grammar

Returns:
  Error status or throws exception

Record Grammar Node <- Get firstChild from Grammar Element
    skipping over non-Element Nodes
Record Grammar Element <- Record Grammar Node

//  Process the Group's starting Record Element
Record Element <- Get first childNode from Parent Element,
    skipping over non-Element Nodes

Call RecordWriter's parseRecord, passing Record Element and
    Record Grammar Element
Call RecordWriter derived class's writeRecord, passing Record
    Grammar Element

// This advance makes sure that we don't repeat starting record
//  in group
Record Grammar Element <- Get next Record Element from
    Group Grammar, skipping over non-Element Nodes
Grammar Tag = call Record Grammar Element's getAttribute for
    "ElementName"

Record Element <- Get Record Element's nextSibling, skipping over
    non-Element Nodes

DO until all child Elements of Parent have been processed
  Record Tag <- call Record Element's getNodeName
  DO until Grammar Tag = Record Tag
    Record Grammar Element <- Get next Record Element from
        Group Grammar, skipping over non-Element Nodes
    IF Record Grammar Element is NULL
      Return error  //  This record is not part of the group
    ENDIF
  ENDDO
  Grammar Element Name <- Call Record Grammar Element's
      getNodeName
  IF Grammar Element Name = "GroupDescription"
    Call processGroup, passing Record Element and Record Grammar
        Element
  ELSE
    Call RecordWriter's parseRecord, passing Record Element and
        Record Grammar Element
    Call RecordWriter derived class's writeRecord, passing
        Record Grammar Element
  ENDIF
  Record Element <- Get Record Element's nextSibling, skipping
      over non-Element Nodes
ENDDO

This method is called the first time from processDocument, which passes it the root Element of the Document and the complete Grammar Element. It processes the Element that represents the header record of the logical document, then advances to process its sibling records and groups. Record Elements are processed in the same fashion as the header record Element. Again, recall that in our XML representation of flat files a group of records is explicitly represented by an Element, with the group members (records or other groups) as child Elements. When we encounter an Element that represents a group (that is, when the Element's grammar Element has the name "GroupDescription"), we make a recursive call.

The termination cases are a bit different than those in the FlatSource Converter. In the XML representation the groups are explicitly represented, so we don't have a normal case of encountering an Element that isn't in the current group grammar. If we do hit one, this is an error; we return an error or throw an exception. For normal processing the method continues execution until all the Elements in the source document's group have been processed, then we return back to the caller.

FlatRecordWriter Class (Extends RecordWriter)

Overview

As is the case with the CSVRecordWriter class, the FlatRecordWriter is derived from the RecordWriter base class (see Chapter 6). The most important method developed in this class is the writeRecord method. We use the parseRecord method inherited from the RecordWriter base class.

Attributes:

  • Integer Fixed Record Length

  • Integer Record ID Field Offset

  • Integer Record ID Field Length

Methods:

  • Constructor

  • writeRecord

Methods
Constructor

The logic for the FlatRecordWriter constructor method is essentially the same as that for the FlatRecordReader.

Logic for the FlatRecordWriter Constructor Method
Arguments:
  DOM Document File Description Document
  Output Stream

Call RecordWriter base class constructor, passing File
    Description Document and Output Stream
Record Format Element <- Get "RecordFormat" Element from File
    Description Document
Child Element <- Get first childNode from Record Format Element,
    advancing over non-Element Nodes
Fixed Record Length <- 0
IF Child Element NodeName = Fixed
  Fixed Record Length <- Call Child Element's getAttribute for
      "Length"
ELSE
  Record Terminator <- Call Child Element's getAttribute for
      "RecordTerminator"
  Call setTerminator to set the Record Terminators
ENDIF
Tag Info Element <- Get "RecordTagInfo" Element from File
    Description Document
Record ID Field Offset <- Call Tag Info's getAttribute for
    "Offset"
Record ID Field Length <- Call Tag Info's getAttribute for
    "Length"
writeRecord

The writeRecord method handles the aspects of record formatting unique to each legacy format. We should again note two restrictions mentioned at the beginning of the chapter. The record identifier field will be converted if it is present in the input XML document, but the value will be overridden by the value specified in the grammar. If a fixed record length is specified the record is initialized to ASCII spaces. Variable length records are initialized with the null character.

The approach for writing a record to flat files is somewhat different than the approach we used for writing to CSV files. For CSV files we walked through the DataCell Array and built the output buffer from its contents. Column number was key, and if a column was missing, a column delimiter was inserted. Due to the requirement to fill missing fields with a default fill character, we must take a different approach and drive the conversion from the record grammar instead of the contents of the DataCell Array. We will cycle though the FieldDescription Elements of the RecordDescription, find matching fields in the DataCell Array, and build the output buffer accordingly. If a field in the grammar is not present in the DataCell Array built from the input, we fill it using the fill character specified in the grammar.

Logic for the FlatRecordWriter writeRecord Method
Arguments:
  DOM Element Record Grammar

Returns:
  Error status or throws exception

IF Fixed Record Length > 0
  Initialize Output Record Buffer to null characters
ELSE
  Initialize Output Record Buffer to ASCII spaces
ENDIF
Record Length <- 0
Cell Array Index <- 0
Field Grammar NodeList <- Call Record Grammar's
    GetElementsByTagName for "FieldDescription"
DO for all items in Field Grammar NodeList
  Field Grammar Element <- next item from Field Grammar NodeList
  Grammar Field Number <- call Field Grammar Element's
      getAttribute for "FieldNumber"
  Offset <- Call Grammar Element's getAttribute for "Offset"
  Length <- Call Grammar Element's getAttribute for "Length"
  Field Written <- false
  IF (Index <= Highest Cell)
    Field Number <- call CellArray[Index] getFieldNumber
    IF (Field Number = Grammar Field Number)
      Call CellArray[Index] fromXML
      Call CellArray[Index] prepareOutput
      Field Contents <- Call CellArray[Index] getField
      Output Buffer <- Insert Field Contents at Offset for Length
      Clear CellArray[Index]
      Increment Cell Array Index
      Field Written <- true
    ENDIF
  ENDIF
  IF ( Field Written = false)
    Fill Character <- Call Grammar Element's getAttribute for
        "FillCharacter"
    Output Buffer <- Insert Fill Character at Offset for Length
  ENDIF
  Working Length <- Offset + Length
  IF (Working Length > Record Length)
    Record Length = Working Length
  ENDIF
ENDDO
Highest Cell <- -1
Record ID Value <- Call Record Grammar Element's getAttribute
    for "TagValue"
Output Buffer <- Insert Record ID Value at Record ID Field
    Offset for Record ID Field Length
IF Fixed Record Length = 0
  Append base RecordHandler's Record Terminators to Output
      Buffer
  Increment Record Buffer Length
ENDIF
Call language's write routines to do physical write of
    Output Buffer for Record Length
Return success


New DataCell Methods and Derived Classes

For this chapter, in addition to adding a few new derived classes to support our new data types, we also need to add methods to existing classes to handle the requirements of formatting fields for flat file records. We allow the text contents of the field Elements in our input XML documents to be of any length (subject to whatever restrictions you define in your schemas, should you choose to use schemas). However, the fields in our flat file records have fixed lengths. So, if the content from the XML document is shorter than the field, we must fill and justify it appropriately. Conversely, if the content from the XML document is longer than the field, we must truncate it if allowed or force an error if not.

We'll first discuss the new methods we're adding to the base DataCell class to support these requirements. We'll next look at the prepareOutput method we're adding to the existing derived classes. We'll wrap up with the new derived classes added for this chapter.

New DataCell Methods

Three new methods are added to the DataCell base class to support filling and justification. The primary method is fillField, called from the prepareOutput method in the derived classes. It retrieves the appropriate Length and FillCharacter Attributes from the field grammar Element, then calls the justifyFieldLeft or justify FieldRight methods.

Note that in addition to retrieving a Length Attribute from the Grammar Element, fillField also attempts to retrieve a MinLength Attribute. Although we don't need this feature for flat files, we'll need it later for EDI formats, so I include it now. We don't need to be concerned in the code about having both values defined. The file description document schemas ensure that we'll have the one appropriate for the particular legacy format. We supply a default fill character (an ASCII space) if the grammar Element doesn't have a FillCharacter Attribute. This also fulfills the requirements of EDI and similar formats.

Logic for the DataCell fillField Method
Arguments:
  Character Justify - L or R for left or right justification
  Integer Fixed Length - For fixed length, if specified

Returns:
  Status or throws exception

Length <- Fixed Length
IF (Length = 0)
  Minimum Length <- Call Grammar Element's getAttribute for
      "MinLength"
  Length <- Minimum Length, converted from string
ENDIF
IF (Buffer Length >= Length)
  return
ENDIF
Fill Character <- Call Grammar Element's getAttribute for
    "FillCharacter"
IF Fill Character is empty
  Fill Character = ASCII space
ENDIF
IF ( Justify = R )
  Call justifyFieldRight, passing Length and Fill Character
ELSE
  Call justifyFieldLeft, passing Length and Fill Character
ENDIF
Return success
Logic for the DataCell justifyFieldLeft Method
Arguments:
  Integer Length
  Character Fill Character

Returns:
  Status or throws exception

Dependent on language features, fill Cell Buffer right portion
    for Length with passed Fill Character
Return success
Logic for the DataCell justifyFieldRight Method
Arguments:
  Integer Length
  Character Fill Character

Returns:
  Status or throws exception

Dependent on language features, shift Cell Buffer contents to the
    right-most positions within passed Length, and fill left
    with passed Fill Character
Return success

New Methods in DataCell Derived Classes

We defined three derived classes to support CSV formats in Chapter 7. To use those classes now for flat files we must add a prepareOutput method. It is very similar in most classes, but it varies depending on justification and whether or not truncation is allowed. Note again that for flat file formats we need to use only the Length Attribute of the grammar Element, but for EDI support in the next chapter we retrieve the MaxLength Attribute if Length is not present.

DataCellAN prepareOutput Method

For the alphanumeric data type supported by the DataCellAN class, we left-justify and truncate if specified by the Truncatable Attribute of the Grammar Element.

Logic for the DataCellAN prepareOutput Method
Arguments:
  None

Returns:
  Status or throws exception

Length <- Call Grammar Element's getAttribute for "Length"
Call fillField, passing 'L' for left justification, and Length
IF ( Length = 0 )
  Maximum Length <- Call Grammar Element's getAttribute for
      "MaxLength"
  Length <- Maximum Length, converted from string
ENDIF
IF (Buffer Length <= Length)
  Return success
ENDIF
Truncate <- Call Grammar Element's getAttribute for
    "Truncatable"
IF ( Truncate = false )
  Return error
ENDIF
Depending on language features, truncate Cell Buffer contents to
    specified Length
Buffer Length <- Length
Return success
DataCellReal prepareOutput Method

For the real (or decimal) data type supported by the DataCellReal class, we right-justify. If a sign character is present in the input XML Element, its position in the output is dependent on the fill character. If the fill character is a zero, the sign character is placed in the left-most position. For all other fill characters the sign character immediately precedes the most significant digit. If the field is truncatable and it exceeds the maximum length specified, fractional digits to the right of the decimal point (and the decimal point if required) are truncated. If this is done and the digits to the left of the decimal still exceed the maximum length, an error occurs.

Logic for the DataCellReal prepareOutput Method
Arguments:
  None

Returns:
  Status or throws exception

Length <- Call Grammar Element's getAttribute for "Length"
Sign Character <- First character in Cell Buffer
Call fillField, passing 'R' for right justification, and Length
IF (Sign Character = "+" or "-") and (First character in Cell
    Buffer = "0")
  Dependent on language features, find Sign Character in Cell
      Buffer and swap with first character in Cell Buffer
ENDIF
IF ( Length = 0 )
  Maximum Length <- Call Grammar Element's getAttribute for
      "MaxLength"
  Length <- Maximum Length, converted from string
ENDIF
IF (Buffer Length <= Length)
  Return success
ENDIF
Truncate <- Call Grammar Element's getAttribute for
    "Truncatable"
IF ( Truncate = false )
  Return error
ENDIF
Depending on language features, truncate Cell Buffer contents to
    specified Length stopping when Length is achieved or decimal
    point is truncated
IF (Buffer Length > Length)
  Return failure
ENDIF
Buffer Length <- Length
Return success
DataCellDateMMsDDsYYYY prepareOutput Method

The DataCellDateMMsDDsYYYY class supports the date data type with dates in an MM/DD/YYYY format. For this type we left-justify and fill with spaces. This class is not truncatable, and we don't check the maximum length since the fromXML method ensures that we have a length of ten characters.

Logic for the DataCellDateMMsDDsYYYY prepareOutput Method
Arguments:
  None

Returns:
  Status or throws exception

Length <- Call Grammar Element's getAttribute for "Length"
Call fillField, passing 'L' for left justification, and Length
Return success

DataCellN Class

This class handles conversion to and from a number with an implied decimal point and the schema language decimal data type. The data type code for this class is Nx, where x represents the number of implied decimal places.

For the fromXML method, if the number of fractional digits in the source decimal number exceeds the number of implied decimal places, it is right-truncated to the number of implied places. Zeroes are added as fractional digits if the source number has fewer fractional digits than the specified number of implied places.

Logic for the DataCellN fromXML Method
Arguments:
  None

Returns:
  Error status or throws exception

IF first Character in Cell Buffer = '+'
  Cell Buffer <- Shift Cell Buffer left one character to remove
      plus sign
  Decrement Buffer Length
ENDIF
Data Type <- Call Grammar Element's getAttribute for "DataType"
Implied Places <- Extract portion of Data Type following 'N'
Difference Places <- Implied Places
Find decimal point in Number Depending on language features
IF Decimal Point present
  Fractional Digits <- Digits to the right of the decimal point
  Cell Buffer <- Shift fractional digits to the left to overlay
      the decimal point, and clear the last byte
  Decrement Buffer Length
  Difference Places <- Implied Places - Fractional Digits
  IF (Difference Places < 0)
    Cell Buffer <- Truncate right-most Difference Places digits
    Buffer Length <- Buffer Length - Difference Places
  ENDIF
ENDIF
IF (Difference Places > 0)
  Cell Buffer <- Append Difference Places number of zeroes
ENDIF
Return success

For the toXML method we trim leading zeroes and whitespace, then insert the explicit decimal point.

Logic for the DataCellN toXML Method
Arguments:
  None

Returns:
  Error status or throws exception

Call trimLeadingZeros to remove leading zeroes and leading and
    trailing whitespace from Cell Buffer
Data Type <- Call Grammar Element's getAttribute for "DataType"
Implied Places <- Extract portion of Data Type following
'N'
IF Implied Places > 0 OR Buffer Length is zero
  Return
ENDIF
IF Buffer Length after trimming is less than Implied Places
      plus one
   Pad to Implied Places with zeroes, plus a zero to the left
       of the decimal
ENDIF
Cell Buffer <- Substring of Cell Buffer from 0 through
    (Length - Implied Places) + "." + leftmost Implied Places
    characters of Cell Buffer
ELSE

Return success

The prepareOutput method is a bit simpler than for the real number data type since we have already handled fractional digits and we don't truncate. As with the real data type, the number is right-justified within the field. Again, leading characters are set according to the fill character. If the fill character is a zero, the sign character is placed in the left-most position. For all other fill characters the sign character immediately precedes the most significant digit. We don't truncate. Since we don't check length in the fromXML method (as is the case with the date data types), we check length here and return an error if the length exceeds the defined fixed or maximum length.

Logic for the DataCellN prepareOutput Method
Arguments:
  None

Returns:
  Status or throws exception

Length <- Call Grammar Element's getAttribute for "Length"
Sign Character <- First character in Cell Buffer
Call fillField, passing 'R' for right justification, and Length
IF (Sign Character = "+" or "-") and (First character in Cell
    Buffer = "0")
  Dependent on language features, find Sign Character in Cell
      Buffer and swap with first character in Cell Buffer
ENDIF
IF ( Length = 0 )
  Maximum Length <- Call Grammar Element's getAttribute for
      "MaxLength"
  Length <- Maximum Length, converted from string
ENDIF
IF (Buffer Length > Length)
  Return failure
ENDIF
Return success

DataCellDateYYYYMMDD Class

This class handles conversion to and from the date data type in the common YYYYMMDD format to the schema language date data type in ISO 8601 date format, that is, YYYY-MM-DD. In keeping with our strategy of using schema validation as our primary data validation mechanism, I include error checking sufficient to prevent only most runtime exceptions.

Logic for the DataCellDateYYYYMMDD fromXML Method
Arguments:
  None

Returns:
  Error status or throws exception

IF Buffer Length != 10
  Return error
ENDIF
Month <- Cell Buffer characters at offsets 5 and 6
Day <- Cell Buffer characters at offsets 8 and 9
Year <- Cell Buffer characters at offsets 0 through 3
Cell Buffer <- Year + Month + Day
Return success
Logic for the DataCellDateYYYYMMDD toXML Method
Arguments:
  None

Returns:
  Error status or throws exception

Cell Buffer < trim leading and trailing whitespace from
    Cell Buffer
IF Buffer Length != 8
  Return error
ENDIF
Year <- Cell Buffer characters at offsets 0 through 3
Month <- Cell Buffer characters at offsets 4 and 5
Day <- Cell Buffer characters at offsets 6 and 7
Cell Buffer <- Year + "-" + Month + "-" + Day
Return success

The prepareOutput method performs left-justification and fills as necessary with the specified fill character. This date data type may not be truncated, so there is no code that deals with truncation. In addition, since the fromXML method ensures that we have the correct length we're not concerned with exceeding the specified length.

Logic for the DataCellDateYYYYMMDD prepareOutput Method
Arguments:
  None

Returns:
  Status or throws exception

Length <- Call Grammar Element's getAttribute for "Length"
Call fillField, passing 'L' for left justification, and Length
Return success


Java Implementation

The Java implementation of the design is composed of the source files listed below. We're not breaking any new ground in any of this code, so I won't add anything to the detailed design discussion presented in the previous sections.

  • FlatToXML.java: the main routine for the Java application with flat files as source

  • FlatSourceConverter.java (in package net.babelblaster.convert): the class that drives conversion from flat files to XML

  • FlatRecordReader.java (in package net.babelblaster.record): the class that handles reading flat file records and converting to XML

  • XMLToFlat.java: the main routine for the Java application with flat files as target

  • FlatTargetConverter.java (in package net.babelblaster.convert): the class that drives conversion from XML to flat files

  • FlatRecordWriter.java (in package net.babelblaster.record): the class that handles reading XML and building and writing flat file records

  • DataCellN.java (in package net.babelblaster.cell): the class that handles legacy format number fields with implied decimal places

  • DataCellDateYYYYMMDD.java (in package net.babelblaster.cell): the class that handles date fields in YYYYMMDD representation in legacy formats


C++ Implementation

The C++ implementation is composed of the source files listed below. As is the case with the Java implementation, there's no new ground and not a lot to elaborate.

  • FlatToXML.cpp: the main routine for the C++ application with flat files as source

  • FlatSourceConverter.cpp: the class that drives conversion from flat files to XML

  • FlatSourceConverter.h: the header file for the FlatSourceConverter class

  • FlatRecordReader.cpp: the class that handles reading flat file records and converting to XML

  • FlatRecordReader.h: the header file for the FlatRecordReader class

  • XMLToFlat.cpp: the main routine for the C++ application with flat files as target

  • FlatTargetConverter.cpp: the class that drives conversion from XML to flat files

  • FlatTargetConverter.h: the header file for the FlatTargetConverter class

  • FlatRecordWriter.cpp: the class that handles reading XML and building and writing flat file records

  • FlatRecordWriter.h: the header file for the FlatRecordWriter class

  • DataCellN.cpp: the class that handles legacy format number fields with implied decimal places

  • DataCellN.h: the header file for the DataCellN class

  • DataCellDateYYYYMMDD.cpp: the class that handles date fields in YYYYMMDD representation in legacy formats

  • DataCellDateYYYYMMDD.h: the header file for the DataCellDateYYYYMMDD class


Enhancements and Alternatives

Perhaps I flatter myself, but I tend to think the utilities developed in this chapter are fairly comprehensive. However, there are always a few enhancements we can make.

Additional Data Types

The data types that come to my mind first are real legacy types associated with IBM mainframes. For example, if we had alphanumeric fields in EBCDIC rather than ASCII encoding, and would want to do EBCDIC to ASCII conversion. If we did this we would probably also need to support the associated numeric data types such as zoned decimal and packed decimal. There might be a few wrinkles we would have to consider in regard to specifying and retrieving record tags. However, for the most part we again would need to code only the DataCell derived classes, change the RecordHandler's createDataCell method, and add the enumerations for the data types to the BBCommonFileDescription.xsd schema.

CSV Record Formats

We discussed this a bit at the end of the Chapter 7. If you are following the architecture by now, it should be fairly obvious that the FlatSourceConverter and FlatTargetConverter could easily use the CSVRecordWriter and CSVRecord Reader instead of the flat file versions we implemented in this chapter. We would need to add or modify a few methods here and there in the CSV record handlers to properly handle record identifiers, but that is about all.

Rounding versus Truncation

I said earlier in the chapter that for numeric fields that allowed truncation of excess fractional digits we would truncate rather than round. However, I realize that there may be business cases for rounding, and doing the rounding in XSLT may not necessarily be the most elegant solution. To add rounding we would need to modify the prepareOutput methods of the DataCellReal and DataCellN classes. To avoid the nastiness of floating point arithmetic I would probably split the source number into an integer and fractional digits. Then I would treat the fractional digits as an integer, rounding up at the appropriate decimal position, truncating, then concatenating the remaining fractional digits to the integer portion on the left of the decimal.

Group Fields

Many legacy applications have what is similar to a GROUP field in COBOL, that is, a structure of two or more fields grouped together that may repeat. To support these with the current implementation each member field of each group must be specifically defined in the file description documents. Doing this loses any notion of a group. Supporting this type of functionality is fairly low on my priority list, but it could be accomplished using a strategy similar to how we'll handle composite data structures in EDI in Chapter 9.

Redefined Fields

COBOL specifically supports a REDEFINES clause, similar to a union in C. I'm not sure of the advantages of supporting this feature, but I'm also sure that someone might think of some. For example, supporting REDEFINES might be attractive to people who want to develop file description documents directly from COBOL copybooks. I've not tested it but I see no reason why the FlatToXML converter couldn't support redefined fields right now. The XMLToFlat converter might also support the feature, but if it does it would, in essence, write out and overlay the redefined bytes once for each redefinition. I see no advantage in that!


Chapter 9. Converting EDI to and from XML

EDI has been described as complex low technology.

—Paul Kimberley

This chapter presents basic utilities for converting Electronic Data Interchange (EDI) files in the U.S. X12 standard to and from XML documents. Support for other standards, such as UN/EDIFACT, and more comprehensive functionality are discussed.

The quote above from Kimberley [1991] is one of my favorite statements about EDI. Even in 1991 EDI could only be considered low-tech. Think about it: delimited file formats designed to save every last byte, summary records with record counts, unencrypted security and authorization information, slow dial-up connections, and asynchronous or bisynchronous communications and file transfer protocols such as XMODEM and 2780/3780. Then think about the complexity of making it all work together, and add to that the complexity of integrating it with business applications! Even with a growing movement toward sending EDI over the public Internet, the statement still rings true.

In contrast, although XML's first promise was for a simple way to encode data in files, with all the secondary and tertiary standards and technologies that have been built on top of it I can only characterize XML as complex high technology. So, where is the happy medium? In this chapter I'll try to cut through some of the complexity of both EDI and XML and present a workable approach to low-tech and high-tech peaceful coexistence.

I'm going to assume that you know a little about EDI already. However, in case you don't or in case you need a refresher regarding certain aspects, I'll start the chapter with a brief overview of some relevant features. I'm going to cover only the basic functionality required to convert EDI in the ANSI X12 format to and from XML. There are two important reasons for this, discussed briefly below.

One reason is that some of the functionality that lends complexity to EDI (such as sequence number generation, auditing, and acknowledgment processing) goes beyond the scope of the file format conversion that is the focus of this book. I will address the minimum of such additional functionality that is required in most implementations, but I won't go beyond that. It is more appropriate to deal with the more complex functionality in the Babel Blaster project that we'll discuss in Chapter 11.

The other reason has to do with limiting the EDI syntax supported by the utilities built in this chapter to the X12 EDI standard. Even within the X12 standard, there are subtle but important variations between the different versions of the standard that have been released over the years. I am going to focus primarily on version 004010 since it is most commonly implemented now due to Y2K upgrades. Beyond X12, there are certainly other important EDI standards, not the least of which is UN/EDIFACT. (The full name, rarely used, is United Nations Electronic Data Interchange for Administration, Commerce and Transport.) However, the ISO 9735 syntax (in two versions) used by UN/EDIFACT is more complex in many ways than the X12 syntax. Again, it makes sense to start with the basics and then proceed to the advanced topics. While many of the key processing routines developed in this chapter encompass functionality beyond what is required for version 004010 of the X12 standard, the executable utilities themselves will not. As with the more complex functionality, I'll discuss support for other EDI standards and other versions of X12 in Chapter 11 with the Babel Blaster project.


Overview of the X12 EDI Syntax and Standards

We need to get some terminology straight first. I'm never quite sure whether to start at the bottom or the top when trying to explain the X12 EDI standards. Since this book is about XML and XML is about documents, let's start at the middle with X12's notion of a document. I'll note where the UN/EDIFACT standard uses different terminology.

  • Transaction set: This is X12's notion of a document. In the strictest terms a transaction set is the actual instance document in a data stream. However, even in the standards the term is sometimes ambiguously used to describe the definition or designation of a document type in the standard rather than an instance of the type. Each transaction set type is oriented toward exchanging a specific type of business information. X12 defines standards for over 300 different types of transaction sets. A transaction set may be specified so as to convey just one logical business document, as is the case with most purchase orders. However, it may be specified to allow transmission of several logical business documents within one transaction set instance. An example is the Health Care Claim, which can convey several claims for several patients and several providers. (A real mess in my opinion, but I wasn't around when it was being defined and I'm not sure I could have been persuasive enough to change the outcome anyway.) Transaction set types are assigned three-digit numeric designations, such as 850 for the Purchase Order. A transaction set may be divided into header, detail, and summary areas. In a data stream a transaction set is enveloped within a transaction set header (the ST segment) and a transaction set trailer (the SE segment). In the UN/EDIFACT standard a document is a message, and it is enveloped by different segments.

  • Segment: A segment is analogous to a record in a flat file or, more accurately, a row in a CSV file. Segments are variable length. A two- or three-character segment identifier (often referred to as a tag) appears at the beginning of each segment and is commonly used as the name of the segment. The fields (or data elements as described below) within a segment are delimited by a data element separator. A segment ends with a segment terminator. In UN/EDIFACT the segment tag is analogous to X12's segment identifier. Special segments associated with envelopes, such as the interchange header and trailer segments, are called control segments in X12 and service segments in UN/EDIFACT.

  • Segment group: Transaction sets may combine a sequence of several segments into a repeating group. In some places in the X12 standards and in common usage the term loop is also used to refer to these groups. Groups may contain other groups. There are a few different types of groups; the most common one is an unbounded loop. This is a group of segments identified by the starting segment of the group, which must appear if the group is used. A common example is an N1 loop that starts with the N1 Name segment. It is used to convey name, address, and other details such as contact information.

  • Data element: A data element is similar to a field in a flat file record, although it is not quite so simple since there are three basic flavors of data elements. A simple data element is an atomic data item used on its own within a segment. A composite data structure (sometimes incorrectly referred to as a composite data element) is similar to a group field in flat file records. It is a reusable group of several simple data elements. An individual data element within a composite data structure is called a component data element. The component data elements within a composite data structure are delimited by a component data element separator. In UN/EDIFACT the term composite data element is correct and is analogous to X12's composite data structure.

  • Functional group: For transmission one or more transaction sets are bundled together into a functional group. A functional group has a designated type associated with the business purpose of the enclosed transaction sets. Generally, each transaction set has a unique functional group type, so there will usually be only a single type of transaction set within any given functional group. However, there are a few exceptions. Some transaction sets may be used in several different types of functional groups, and a very few transaction sets are assigned the same functional group identifier. A functional group is enveloped by a functional group header (the GS segment) and a functional group trailer (the GE segment). The GS segment conveys several pieces of control information such as application sender and receiver identifiers, date and time stamps, and the X12 version and release of the enclosed transaction sets. Functional groups are mandatory in X12 but optional in UN/EDIFACT (which of course has different header and trailer segments).

  • Interchange: For transmission one or more functional groups are bundled together into an interchange, which can be thought of as an envelope. An interchange starts with an interchange control header (the ISA segment) and ends with an interchange control trailer (the IEA segment). The interchange is the smallest unit of data transmitted between trading partners. Most EDI traffic, even now, is conveyed by private, third-party value-added networks (VANs) that function as store-and-forward EDI mailbox services. The ISA segment has sender and receiver identifiers used to route interchanges through these systems. In addition to other control information, the ISA segment also specifies the delimiters used in the interchange. UN/EDIFACT, as well as having a different set of interchange header and trailer segments, also has default delimiters. The default delimiters may be overridden with a set specified in the optional UNA segment that precedes an interchange.

Those are the basic concepts and terms. There are, of course, quite a few more details, and I'll deal with some of them as the chapter progresses. I'll also present some examples. However, as I mentioned, I'm assuming that you know a little bit about EDI already and I'm not going to provide an extensive tutorial on the X12 syntax. If you need one please refer to my Web site (www.rawlinsecconsulting.com) or any one of the several good books on the subject (several are reviewed on my Web site). You can also consult the X12 standards (available for purchase from the ANSI Accredited Standards Committee X12 at www.x12.org). The standards most relevant to this book are listed below.

  • X12.5, Interchange Control Structures: Deals with interchange structure and interchange control segments such as the ISA and IEA.

  • X12.6, Application Control Structure: Defines the rest of the basic syntax features of X12 such as transaction sets, segments, data elements, and functional groups.

  • X12.1, Transaction Set; X12.22, Segment Directory; and X12.3, Data Element Dictionary Standards: X12.5 and X12.6 define the syntax, such as the characteristics of a segment. The standards named in this bullet define the actual business content, such as the specific data elements that make up the N1 Name segment.

These standards are published as a full set, with one major version and two minor releases per year. However, some individual standards are available. The versions are designated by a three-digit version number and a three-digit release number within the version. The major version has a release designation of 000. The preceding zeroes in the version are sometimes dropped in informal use. For example, people may refer to 4010 instead of 004010.

In addition to X12's transaction set standards, most major EDI users publish so-called "implementation guides" for their trading partners. X12 defined the transaction set standards so they could be used by a very broad range of users. This is commonly (and somewhat derisively) known as the "kitchen sink approach" since they include "everything but the kitchen sink." For this reason, for most transaction sets quite a few segments and data elements are used only by specialized communities of users. Only a small set of items is used by all users. The implementation guides specify the subset of segments and data elements of the transaction set standards that the issuer expects to be used for particular data exchange situations.

Now that we have this basic terminology in hand we can discuss the utilities and what they do.


X12 to XML: Functionality and Operation

Requirements

This utility converts an X12 interchange containing one or more functional groups, each containing one or more transaction sets, into XML instance documents. An instance document is created for each transaction set. Here's a summary of the required functionality.

  • Inputs: A valid X12 interchange. The second input is a directory path where file description documents (as discussed in Chapter 6) relevant to the interchange may be found.

  • Processing: An appropriate file description document is loaded for each transaction set in the interchange. An Element is created in the output document for each data element, each composite data structure, each segment, and each segment group. The organization of a transaction set is derived from a file description document. Data element content is converted from X12 data types to schema language data types as specified in the file description document. Empty or missing data elements do not create Elements in the output document. For each functional group processed, an XML representation of a 997 Functional Acknowledgment is created.

  • Output: One or more XML instance documents, each in a single file. The root Element name is derived from the grammar in the file description document. The file name is formed by appending the transaction set control number to the root Element name and adding the extension .xml. The documents for each functional group are placed in a separate subdirectory. The subdirectories are named according to the sender IDs and group types in the GS segment. The 997 Functional Acknowledgments are created in the passed output directory and named as discussed below.

You may recall my earlier comment that I would focus primarily on the basic functionality required in most trading situations. Although Functional Acknowledgments are capable of reporting syntax errors, they are most often used only as return receipts and are required in most trading situations. Functional Acknowledgments are clearly beyond the scope of file format conversion, but for this utility to meet the minimum requirements for most trading situations we must be able to generate them. Note, however, that we will generate them only to indicate acceptance (that is, receipt) of a group and not to report errors.

Running the Utility

This section provides instructions for running the X12 to XML utility from the command line.

For Java:

java X12ToXML InputFile.dat OutputDirectory

FileDescriptionDirectory

or

java X12ToXML -h

For C++ on Win32:

X12ToXML InputFile.dat OutputDirectory FileDescriptionDirectory

or

X12ToXML -h

Options follow the parameters except for the help option, which may be specified by itself.

Parameters:

  • First: File specification of the input interchange file (required). The specification may include the full or relative path name. If no path name is specified, the file is assumed to reside in the current working directory. The full file name must be specified, but there is no restriction on the extension name.

  • Second: Path specification of the output directory (required). The directory must exist. Either a relative or an absolute path name may be specified. The trailing directory separator character is optional. A subdirectory for each functional group is created beneath this directory. The subdirectories are named by concatenating the Functional Group Identifier Code in GS01 to the Application Sender's Code in GS02. Output files within these directories are named by appending the Transaction Set Control Number in ST02 to the Root name as specified in the file description document's Grammar Element.

  • Third: Directory path for the file description documents (required). The trailing directory separator character is optional. During processing the utility uses identifying information in the control segments to determine the file description document required to convert the transaction set currently being processed (see the discussion on naming requirements below). These file description documents must reside in this directory path.

Options:

  • -v (Validate): Validate the created XML documents before writing them to disk. The documents are validated against the schema specified in the file description document.

  • -h (Help): Display a help message and exit without further processing.

Naming requirement for file description documents:

File description documents used with this utility must conform to a specific naming convention for their local file names. Their names must be formed by concatenating the following values:

  • GS Application Sender's Code in GS02, followed by a hyphen

  • GS Functional Identifier Code in GS01, followed by a hyphen

  • ST Transaction Set Identifier Code in ST01, followed by a hyphen

  • GS Version/Release in GS08

  • A file type extension of .xml

(Note: Some values of GS Version/Release, such as those sometimes used for TDCC or UCS versions, may not yield valid file names for the operating system.)

As an example of the naming convention, suppose we receive an interchange from Big Box Discount Stores that contains 850 Purchase Orders in version 004010 of X12. Suppose they use BIGBOX for the sender ID in the GS segment. The X12ToXML utility will look in the passed directory for a file description document named:

BIGBOX-PO-850-004010.xml

A version 1.0 Babel Blaster requirement is to remove this naming restriction, but it's not a bad convention to follow anyway.

Naming convention for Functional Acknowledgments:

An XML representation of an X12 997 Functional Acknowledgment is created for each functional group the utility processes. The files for these XML documents are placed in an FA_Out subdirectory of the passed directory. Individual acknowledgments are named by concatenating the following values:

  • GS Application Sender's Code in GS02, followed by a hyphen

  • GS Functional Identifier Code in GS01

  • A file type extension of .xml

For example, the Functional Acknowledgment for Big Box Discount Stores purchase orders on a UNIX system would be named:

FA_Out/BIGBOXBUYER-PO.xml

Restrictions:

Unless otherwise noted, all numeric limits may be modified by changing parameters in the program source and appropriate type definitions in the file description document schemas.

  • An X12 data element may contain a maximum of 1,023 bytes.

  • A segment may be no longer than 16,383 bytes.

  • A maximum of 100 X12 data elements per segment is supported. This includes repeated elements and component data elements within composite data structures.

  • There is no absolute limit on the number of segments; the number is only practically limited by system memory.

  • Each X12 data element must be assigned a unique Element name.

  • XML Element names are limited to 127 characters.

  • Data element grammar Elements must be specified in their segment description Element in ascending order by their position within the segment or composite data structure.

  • Path lengths for complete file specifications are limited to 127 characters.

  • Schema location URIs are limited to 127 characters.

  • A maximum of 999 output XML documents from each functional group is supported.

  • The TA1 Interchange Acknowledgment control segment is not currently supported and will cause a parsing error if encountered. (It is very seldom used. Supporting it is among the Babel Blaster version 1.0 requirements.)

  • GS sender IDs must be valid directory names for the operating system where the utility is run.

  • In the current implementation there can be only one instance in an interchange of a functional group type per GS sender ID. This is due to the naming convention used for output directories.

  • Functional Acknowledgments are created only to indicate acceptance of a functional group. Neither transaction set acceptance nor error reporting are supported.

Sample Input and Output: 850 Purchase Order

Big Daddy's Gourmet Cocoa has hit the big time! The company is now selling to major retail grocery and discount chains, and they all want to send their orders as X12 850 Purchase Orders. Of course, they all have slightly different implementation guides, but Table 9.1 shows an abbreviated summary of a fairly representative guide. Note: The Segment Group column is empty if the segment is not part of a segment group. The BEG and a few other header level segments fall into this category.

You'll note a few things about this sample implementation. Only the N1 segment is transmitted; there isn't a full N1 loop. It is becoming increasingly common to use identifiers such as DUNS or DUNS+4 numbers to reference a location or party already in the system rather than transmitting full address information. The PO1 segment has only a single product code, though sometimes more than one is sent (I assume as a convenience and crosscheck for the vendor). Overall, this is a fairly compact implementation of the 850 Purchase Order.

The sample interchange below shows one functional group with two transaction sets.

Sample Input Interchange (PurchaseOrders.X12)
ISA*00*    *00*     *ZZ*BIGBOX     *12*
  9727839573   *030123*0600*U*00401*000000357*0*P*>~
GS*PO*BIGBOX*9727839573*20030123*0600*114*X*004010~
ST*850*0001~
BEG*00*SA*4497-0561**20030123~
DTM*001*20030206~
N1*ST*BIG BOX - STORE #97*92*001234567S097~
PO1*1*10*CA*30.36**UP*35790000122~
PID*F****Instant Hot Cocoa Mix - Mint flavor~
PO1*2*30*CA*31.08**UP*35790000724~
PID*F****Instant Hot Cocoa Mix - Vanilla flavor~
CTT*2~
SE*10*0001~
ST*850*0002~
BEG*00*SA*4445-0323**20030123~
DTM*001*20030206~
N1*ST*BIG BOX - STORE #45*92*001234567S045~
PO1*1*20*CA*30.36**UP*35790000122~
PID*F****Instant Hot Cocoa Mix - Mint flavor~
PO1*2*40*CA*31.08**UP*35790000641~
PID*F****Instant Hot Cocoa Mix - Dutch Chocolate flavor~
CTT*2~
SE*10*0002~
GE*2*114~
IEA*1*000000357~

Table 9.1. Sample X12 850 Purchase Order Implementation Guide

Segment Group

Segment

Tag

Data Element Position

Data Element or Composite Data Structure Number andName

X12 Data Type

Minimum Length

Maximum Maximum Length

Description and Comments

Beginning Segment for Purchase Order

BEG

01

353 Transaction Set Purpose Code

ID

2

2

00 for original purchase order

02

92 Purchase Order Type Code

ID

2

2

SA for stand-alone order

03

324 Purchase order number

AN

1

22

Buyer-assigned purchase Order Number

05

373 Date

DT

8

8

Purchase order date

Date/Time Reference

DTM

01

374 Date/Time Qualifier

ID

3

3

002 for requested delivery date or 010 for requested ship date

02

373 Date

DT

8

8

Header N1

Name

N1

01

98 Entity Identifier Code

ID

2

3

One N1 loop is sent with an ST in this element

02

93 Name

AN

1

60

The name of the receiving location

03

66 Identification Code Qualifier

ID

1

2

92 for Assigned by Buyer or Buyer's Agent

04

67 Identification Code

AN

2

80

The buyer-assigned DUNS+4 number (the buyer's DUNS number followed by a four-digit receiving location code)

PO1

Baseline Item Data

PO1

01

350 Assigned Identification

AN

1

20

The line item number

02

330 Quantity Ordered

R

1

15

03

355 Unit or Basis for Measurement Code

ID

2

2

Usually EA for each or CA for case

04

212 Unit Price

R

1

17

06

235 Product/ Service ID Qualifier

ID

2

2

Usually UP for UPC Consumer Package Code (1-5-5-1)

07

234 Product/ Service ID

AN

1

48

The UPC code for the item

PO1

Product/ Item Description

PID

01

349 Item Description Type

ID

1

1

F for free-form item description

05

352 Description

AN

1

80

The item description

Transaction Totals

CTT

01

354 Number of Line Items

N0

1

6

The total number of line items in the transaction set

The line breaks (and indentation on the continuation of the ISA segment) exist here only for readability. They would not appear in a compliant data stream.

Listed below are the two XML documents produced by the utility from this interchange. This and the XMLToX12 utility allow the user to specify the choice of names for the Elements that represent the various X12 components. I use a convention based on segment IDs and data element position. You could use more descriptive English names if you prefer.

Sample Output (X12PurchaseOrder0001.xml)
<?xml version="1.0" encoding="UTF-8"?>
<X12PurchaseOrder
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:noNamespaceSchemaLocation="X12PurchaseOrder.xsd">
  <BEG>
    <BEG01>00</BEG01>
    <BEG02>SA</BEG02>
    <BEG03>4497-0561</BEG03>
    <BEG05>2003-01-23</BEG05>
  </BEG>
  <DTM>
    <DTM01>001</DTM01>
    <DTM02>2003-02-06</DTM02>
  </DTM>
  <N1Header>
    <N1>
      <N101>ST</N101>
      <N102>BIG BOX - STORE #97</N102>
      <N103>92</N103>
      <N104>001234567S097</N104>
    </N1>
  </N1Header>
  <PO1Group>
    <PO1>
      <PO101>1</PO101>
      <PO102>10</PO102>
      <PO103>CA</PO103>
      <PO104>30.36</PO104>
      <PO106>UP</PO106>
      <PO107>35790000122</PO107>
    </PO1>
    <PID>
      <PID01>F</PID01>
      <PID05>Instant Hot Cocoa Mix - Mint flavor</PID05>
    </PID>
  </PO1Group>
  <PO1Group>
    <PO1>
      <PO101>2</PO101>
      <PO102>30</PO102>
      <PO103>CA</PO103>
      <PO104>31.08</PO104>
      <PO106>UP</PO106>
      <PO107>35790000724</PO107>
    </PO1>
    <PID>
      <PID01>F</PID01>
      <PID05>Instant Hot Cocoa Mix - Vanilla flavor</PID05>
    </PID>
  </PO1Group>
  <CTT>
    <CTT01>2</CTT01>
  </CTT>
</X12PurchaseOrder>
Sample Output (X12PurchaseOrder0002.xml)
<?xml version="1.0" encoding="UTF-8"?>
<X12PurchaseOrder
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:noNamespaceSchemaLocation="X12PurchaseOrder.xsd">
  <BEG>
    <BEG01>00</BEG01>
    <BEG02>SA</BEG02>
    <BEG03>4445-0323</BEG03>
    <BEG05>2003-01-23</BEG05>
  </BEG>
  <DTM>
    <DTM01>001</DTM01>
    <DTM02>2003-02-06</DTM02>
  </DTM>
  <N1Header>
    <N1>
      <N101>ST</N101>
      <N102>BIG BOX - STORE #45</N102>
      <N103>92</N103>
      <N104>001234567S045</N104>
    </N1>
  </N1Header>
  <PO1Group>
    <PO1>
      <PO101>1</PO101>
      <PO102>20</PO102>
      <PO103>CA</PO103>
      <PO104>30.36</PO104>
      <PO106>UP</PO106>
      <PO107>35790000122</PO107>
    </PO1>
    <PID>
      <PID01>F</PID01>
      <PID05>Instant Hot Cocoa Mix - Mint flavor</PID05>
    </PID>
  </PO1Group>
  <PO1Group>
    <PO1>
      <PO101>2</PO101>
      <PO102>40</PO102>
      <PO103>CA</PO103>
      <PO104>31.08</PO104>
      <PO106>UP</PO106>
      <PO107>35790000641</PO107>
    </PO1>
    <PID>
      <PID01>F</PID01>
      <PID05>
        Instant Hot Cocoa Mix - Dutch Chocolate flavor
      </PID05>
    </PID>
  </PO1Group>
  <CTT>
    <CTT01>2</CTT01>
  </CTT>
</X12PurchaseOrder>


XML to X12: Functionality and Operation

Requirements

The general requirement for this utility is to convert one or more XML instance documents, each representing a single logical document, into an X12 transaction set. The transaction sets are bundled into one functional group and interchange. Here's a summary of the required functionality.

  • Inputs: There are two user-specified inputs. The first is a directory containing one or more XML instance documents, each in a single file. All have the same grammar. The second input is a file description document (as discussed in Chapter 6) that describes the XML documents and the X12 interchange to be produced. The utility also uses a document for maintaining EDI control numbers, as described in a later section of this chapter.

  • Processing: Each input Element corresponding to a segment (as described in the file description document) is written to a segment in a transaction set. Each of its child Elements corresponding to data elements are written to the appropriate positions in the segment. Schema language data types are converted to X12 data types according to the data element descriptions in the file description document. XML Elements corresponding to segment groups and composite data structures are not directly used. Envelope segments are created based on values from the file description document, and incremented sequence numbers from the control number document.

  • Output: A valid X12 interchange.

Again, in keeping with the basic functionality required in most trading situations, we must go a bit beyond simple file format conversion. Many major EDI users audit the control numbers used in interchanges and functional groups and reject interchanges with duplicate control numbers. Some also check with a trading partner if there is a missing number in a sequence of control numbers. So, we're including minimal support for sequence number generation.

Running the Utility

This section provides instructions for running the XML to X12 utility from the command line.

For Java:

java -DBBHOME=BBHomeDirectory XMLToX12 InputDirectory

OutputFile.X12 FileDescription.XML

or

java XMLToX12 -h

Note: The -D option for the Java virtual machine defines the BBHOME system property. Replace BBHomeDirectory with the directory path on your system for the Babel Blaster Home directory. The utility uses this path to retrieve the EDIControl.xml file.

For C++ on Win32:

XMLToX12 InputDirectory OutputFile.X12 FileDescription.XML

or

XMLToX12 -h

Note: The BBHOME environment variable must be set to the directory path on your system for the Babel Blaster Home directory. The utility uses this path to retrieve the EDIControl.xml file.

As we did in Chapter 5, we're using a system property in Java and an environment variable in C++. Refer back to that chapter for a more detailed discussion of the rationale and how to set environment variables.

Options follow the parameters except for the help option, which may be specified by itself.

Parameters:

  • First: Path specification of the input directory (required). Either a relative or an absolute path name may be specified. The trailing directory separator character is optional.

  • Second: File specification of the output interchange file (required). The specification may include the full or relative path name. If no path name is specified, the interchange file will be created in the current working directory. An extension may be specified, but if not specified the file will be created without one. If a file with the specified name already exists, it will be overwritten.

  • Third: File specification of the file description document (required). If no path name is specified, the file is assumed to reside in the current working directory. The full file name must be specified, but there is no restriction on the extension name.

Options:

  • -v (Validate): Validate the XML documents before processing. The documents are validated against the schema referenced in the document root Element.

  • -h (Help): Display a help message and exit without further processing.

Since a specific file description document file name is passed to the utility, there are no restrictions on how the file is named.

Note: This utility requires that at least the skeleton EDIControl.xml file be present in a directory pointed to by the BBHOME system property in the Java implementation or by the operating system environment variable in the C++ implementation.

Restrictions:

The restrictions are the same as those specified for the X12 to XML utility, plus the following.

  • The segment identifier Element need not be present in the input document. It will be converted if present, but the contents will be overwritten by the segment identifier specified in the grammar.

  • All the XML documents processed must be converted to the same X12 transaction set type. One functional group with all these transaction sets is produced, and only one functional group is written to each interchange.

Sample Input and Output: 810 Invoice

Big Daddy's big customers who are sending orders by X12 EDI also want the invoices to be sent to them by X12 EDI. Big Daddy doesn't want to pay the $25 to $100 penalty for each invoice those customers have to process manually. So, Big Daddy needs to create X12 810 Invoices from his system. Table 9.2 shows a summary of a typical 810 implementation. Note: The Segment Group column is empty if the segment is not part of a segment group.

Table 9.2. Sample X12 810 Invoice Implementation Guide

Segment Group

Segment

Tag

Data Element Position

Data Element or Composite Data Structure Number and Name

X12 Data Type

Minimum Length

Maximum Length

Description and Comments

Beginning Segment for Invoice

BIG

01

373 Date

DT

8

8

The invoice date

02

76 Invoice Number

AN

1

22

Seller-assigned invoice number

04

324 Purchase Order Number

AN

1

22

Relevant buyer-assigned purchase order number

Header N1

Name

N1

01

98 Entity Identifier Code

ID

2

3

Two N1 loops must be sent: one with the Ship To location contains an ST in this element, the other with the Remit To party contains an RE in this element

02

93 Name

AN

1

60

For ST, the name of the receiving location; for RE, the vendor name

03

66 Identification Code Qualifier

ID

1

2

For ST, 92 for Assigned by Buyer or Buyer's Agent; for RE, 9 for DUNS number

04

67 Identification Code

AN

2

80

For ST, the buyer-assigned DUNS+4 number (the buyer's DUNS number followed by a four-digit receiving location code); for RE, the DUNS number

Terms of Sale/Deferred Terms of Sale

ITD

01

336 Terms Type Code

ID

2

2

01 for Basic

03

338 Terms Discount Percent

R

1

6

04

370 Terms Discount Due Date

DT

8

8

Date payment is due for discount to apply

05

351 Terms Discount Days Due

N0

1

3

Number of days in the discount period

06

446 Terms Net Due Date

DT

8

8

Date on which the net amount is due

07

386 Terms Net Days

N0

1

3

Number of days until the net amount is due

IT1

Baseline Item Data (Invoice)

IT1

01

350 Assigned Identification

AN

1

20

Sometimes optional, but usually must match the line item number on the corresponding purchase order

02

358 Quantity Invoiced

R

1

10

03

355 Unit or Basis for Measurement Code

ID

2

2

Must match the unit of measurement of the corresponding purchase order; usually EA for each or CA for case

04

212 Unit Price

R

1

17

06

235 Product/Service ID Qualifier

ID

2

2

Usually UP for UPC Consumer Package Code (1-5-5-1)

07

234 Product/Service ID

AN

1

48

The UPC code for the item

IT1

Product/Item Description

PID

01

349 Item Description Type

ID

1

1

F for free-form item description

05

352 Description

AN

1

80

The item description

Total Monetary Value Summary

TDS

01

610 Amount

N2

1

15

Total amount of invoice before terms discount

Transaction Totals

CTT

01

354 Number of Line Items

N0

1

6

The total number of line items in the transaction set

As with the sample implementation guide for our 850 Purchase Order, only the N1 segment is transmitted; there isn't a full N1 loop. Like the PO1 segment, the IT1 segment has only a single product code. Though you would think that major buyers could get by with just one product identifier, I have seen some implementation guides that require three. In addition to a standard UPC code they require their own internal identifier and their vendor's catalog identifier. Sellers don't have much choice but to comply if they want to keep the business.

Here are two invoice instance documents that correspond to the 810 Invoice. As noted earlier I use a naming convention based on segment ID and data element position. You may choose whatever Element names you like.

Sample Input (X12Invoice01.xml)
<?xml version="1.0" encoding="UTF-8"?>
<X12Invoice
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:noNamespaceSchemaLocation="X12Invoice.xsd">
  <BIG>
    <BIG01>2003-01-27</BIG01>
    <BIG02>2003030</BIG02>
    <BIG04>4497-0561</BIG04>
  </BIG>
  <N1Header>
    <N1>
      <N101>ST</N101>
      <N102>BIG BOX - STORE #97</N102>
      <N103>92</N103>
      <N104>001234567S097</N104>
    </N1>
  </N1Header>
  <N1Header>
    <N1>
      <N101>RE</N101>
      <N102>BIG DADDY'S GOURMET COCOA</N102>
      <N103>9</N103>
      <N104>987651234</N104>
    </N1>
  </N1Header>
  <ITD>
    <ITD01>01</ITD01>
    <ITD03>5</ITD03>
    <ITD04>2003-02-27</ITD04>
    <ITD05>30</ITD05>
    <ITD06>2003-02-27</ITD06>
    <ITD07>31</ITD07>
  </ITD>
  <IT1Group>
    <IT1>
      <IT101>1</IT101>
      <IT102>10</IT102>
      <IT103>CA</IT103>
      <IT104>30.36</IT104>
      <IT106>UP</IT106>
      <IT107>35790000122</IT107>
    </IT1>
    <PID>
      <PID01>F</PID01>
      <PID05>Instant Hot Cocoa Mix - Mint flavor</PID05>
    </PID>
  </IT1Group>
  <IT1Group>
    <IT1>
      <IT101>2</IT101>
      <IT102>30</IT102>
      <IT103>CA</IT103>
      <IT104>31.08</IT104>
      <IT106>UP</IT106>
      <IT107>35790000724</IT107>
    </IT1>
    <PID>
      <PID01>F</PID01>
      <PID05>Instant Hot Cocoa Mix - Vanilla flavor</PID05>
    </PID>
  </IT1Group>
  <TDS>
    <TDS01>1236.00</TDS01>
  </TDS>
  <CTT>
    <CTT01>2</CTT01>
  </CTT>
</X12Invoice>
Sample Input (X12Invoice02.xml)
<?xml version="1.0" encoding="UTF-8"?>
<X12Invoice
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:noNamespaceSchemaLocation="X12Invoice.xsd">
  <BIG>
    <BIG01>2003-01-27</BIG01>
    <BIG02>2003030</BIG02>
    <BIG04>4445-0323</BIG04>
  </BIG>
  <N1Header>
    <N1>
      <N101>ST</N101>
      <N102>BIG BOX - STORE #45</N102>
      <N103>92</N103>
      <N104>001234567S045</N104>
    </N1>
  </N1Header>
  <N1Header>
    <N1>
      <N101>RE</N101>
      <N102>BIG DADDY'S GOURMET COCOA</N102>
      <N103>9</N103>
      <N104>987651234</N104>
    </N1>
  </N1Header>
  <ITD>
    <ITD01>01</ITD01>
    <ITD03>5</ITD03>
    <ITD04>2003-02-27</ITD04>
    <ITD05>30</ITD05>
    <ITD06>2003-02-27</ITD06>
    <ITD07>31</ITD07>
  </ITD>
  <IT1Group>
    <IT1>
      <IT101>1</IT101>
      <IT102>20</IT102>
      <IT103>CA</IT103>
      <IT104>30.36</IT104>
      <IT106>UP</IT106>
      <IT107>35790000122</IT107>
    </IT1>
    <PID>
      <PID01>F</PID01>
      <PID05>Instant Hot Cocoa Mix - Mint flavor</PID05>
    </PID>
  </IT1Group>
  <IT1Group>
    <IT1>
      <IT101>2</IT101>
      <IT102>40</IT102>
      <IT103>CA</IT103>
      <IT104>31.08</IT104>
      <IT106>UP</IT106>
      <IT107>35790000641</IT107>
    </IT1>
    <PID>
      <PID01>F</PID01>
      <PID05>
        Instant Hot Cocoa Mix - Dutch Chocolate flavor
      </PID05>
    </PID>
  </IT1Group>
  <TDS>
    <TDS01>1850.40</TDS01>
  </TDS>
  <CTT>
    <CTT01>2</CTT01>
  </CTT>
</X12Invoice>

Here is the resulting interchange.

Sample Output (Invoices.X12)
ISA*00*     *00*     *12*9727839573  *ZZ*
  BIGBOX      *030125*1058*U*00401*000000006*0*P*~
GS*IN*9727839573*BIGBOX*X*20030125*1058*6*004010~
ST*810*0001~
BIG*20030127*2003030**4497-0561~
N1*ST*BIG BOX - STORE #97*92*001234567S097~
N1*RE*BIG DADDY'S GOURMET COCOA*9*987651234~
ITD*01**5*20030227*30*20030227*31~
IT1*1*10*CA*30.36**UP*35790000122~
PID*F****Instant Hot Cocoa Mix - Mint flavor~
IT1*2*30*CA*31.08**UP*35790000724~
PID*F****Instant Hot Cocoa Mix - Vanilla flavor~
TDS*123600~
CTT*2~
SE*12*0001~
ST*810*0002~
BIG*20030127*2003030**2003-01-23~
N1*ST*BIG BOX - STORE #45*92*001234567S045~
N1*RE*BIG DADDY'S GOURMET COCOA*9*987651234~
ITD*01**5*20030227*30*20030227*31~
IT1*1*20*CA*30.36**UP*35790000122~
PID*F****Instant Hot Cocoa Mix - Mint flavor~
IT1*2*40*CA*31.08**UP*35790000641~
PID*F****Instant Hot Cocoa Mix - Dutch Chocolate flavor~
TDS*185040~
CTT*2~
SE*12*0002~
GE*2*6~
IEA*1*000000006~

Again, the line breaks (and indentation on the continuation of the ISA segment) would not appear in actual usage and are included only to improve readability.


Describing the File Formats

With EDI formats we reach the peak of complexity in describing file formats. In addition to describing the grammar of segments and transaction sets, we also need information about various delimiters and values for things like sender and receiver IDs.

It should come as no surprise that our X12 file description document has three major sections, each represented by an Element that is an immediate child of the root Element.

  • PhysicalCharacteristics: the X12 interchange characteristics, required only when converting to X12

  • XMLOutputCharacteristics: the XML output characteristics, required only when converting to XML

  • Grammar: the transaction set grammar

X12 File Physical Characteristics

An interchange's physical characteristics are described in the PhysicalCharacteristics Element. This Element is required for only the target conversion utility. The relevant information is extracted directly from the input interchange for the source conversion utility.

Table 9.3 shows the child Elements of the PhysicalCharacteristics Element. All are required unless otherwise noted.

Note that a few things are missing from the child Elements and Attributes defined for the PhysicalCharacteristics Element.

  • Interchange control standards identifier: Prior to version 4020 of X12 this was the ID value sent in ISA11. The only value ever approved was a U, so I hard-code it in the utility.

  • Interchange and group control numbers: These come from a separate source as will be discussed in a later section.

  • Interchange and group date and time stamps: These are generated by the XMLToX12 utility at runtime.

Although the Repetition Separator and repeating data elements were not introduced until version 4020, I include the delimiter here for future compatibility.

XML Output Characteristics

Characteristics governing the output XML documents are described in the XMLOutputCharacteristics Element. This Element is used only when converting from X12 interchanges to XML.

Table 9.4 shows the child Element of the XMLOutputCharacteristics Element. This time we have only a single child Element; as with the other file formats the schema reference is optional.

Table 9.3. Child Elements of the PhysicalCharacteristics Element

Child Element

Child Element

Attribute

Schema Data Type

Description

Allowable Values, Restrictions, or Comments

Delimiters

SegmentTerminator

value

union of a single token character and hexBinary

The segment terminator character expressed as a literal character or a hexadecimal value

A single nonwhitespace character or a two-character hexadecimal number from 00 through FF representing a single byte

ElementSeparator

value

union of a single token character and hexBinary

The element separator character expressed as a literal character or a hexadecimal value

A single nonwhitespace character or a two-character hexadecimal number from 00 through FF representing a single byte

ComponentSeparator

value

union of a single token character and hexBinary

The component data element separator character expressed as a literal character or a hexadecimal value

A single nonwhitespace character or a two-character hexadecimal number from 00 through FF representing a single byte

RepetitionSeparator

value

union of a single token character and hexBinary

The repetition separator character expressed as a literal character or a hexadecimal value

Optional, not used for X12 versions prior to 4020; a single nonwhitespace character or a two-character hexadecimal number from 00 through FF representing a single byte

ISAInformation

AuthorizationQualifier

value

string

Populates ISA01

May be spaces

AuthorizationInfo

value

string

Populates ISA02

May be spaces

SecurityQualifier

value

string

Populates ISA03

May be spaces

SecurityInfo

value

string

Populates ISA04

May be spaces

SenderIDQualifier

value

token

Populates ISA05

SenderID

value

token

Populates ISA06

Trailing spaces not permitted; will be inserted by the utility if required

ReceiverIDQualifier

value

token

Populates ISA07

ReceiverID

value

token

Populates ISA08

Trailing spaces not permitted; will be inserted by the utility if required

VersionNumber

value

token

Populates ISA12

Will normally be 00401 for version/release 004010 of X12

TestIndicator

value

token

Populates ISA15

P for production or T for Test

GSInformation

FunctionalIDCode

value

token

Populates GS01

ApplicationSenders-Code

value

token

Populates GS02

ApplicationReceivers-Code

value

token

Populates GS03

ResponsibleAgency-Code

value

token

Populates GS07

X for X12 or T for TDCC

VersionRelease

value

token

Populates GS08

The X12 version and release and/or an industry identifier code

STInformation

TransactionSetIDCode

value

token

Populates ST01

Table 9.4. Child Element of the XMLOutputCharacteristics Element

Child Element

Attribute

Schema Data Type

Description

Allowable Values, Restrictions, or Comments

SchemaLocationURL

value

anyURI

URL of the schema file for the output document; will be written as the value of the root Element's noNamespace-SchemaLocation Attribute

The Element is optional. If not specified the noNamespace-Schema Location Attribute will not be written. An error will occur if output validation is requested and this Element is not present.

Transaction Set Grammar

The grammar of X12 transaction sets is the most complex grammar we'll deal with in this book. In addition to having a complex grammar of segments, composite data structures, and data elements, we also have defined segment groups.

It is important to note again here that, unlike some approaches to converting X12 transaction sets to XML, I'm not doing an exact isomorphic representation of a physical X12 transaction set. I'm doing a logical representation of an X12 transaction set. Specifically, segment groups aren't visible as such in a transaction set, and while we can certainly identify composite data structures by their delimiters, they aren't necessarily highly visible as logical units. I show both structures explicitly as XML Elements. A segment group has its member segments (and groups, if any) shown as children, and a composite data structure is depicted as an XML Element with its component data elements as child XML Elements. This is the same approach depicted in the figures in Chapter 8 except that we add another layer for the composite data structures.

If you're familiar with X12 EDI, you may think there are a few things missing from the grammar. If we were concerned with strict X12 compliance, you would be correct. However, we aren't. Our primary concern is transforming data in an X12 format into and out of an XML representation. As such, we don't need to concern ourselves with constraints such as mandatory and optional designations, code values, maximum occurrences, or even whether or not a data element is allowed to repeat. We will rely on schema validation of the XML representation to enforce these types of constraints.

As in previous chapters, the High-Level Design Considerations section contains a more detailed analysis and discussion of X12 grammar. Table 9.5 shows the details of the Grammar Element and its child Nodes. All are required unless noted. A few things should be noted about the table. The Grammar Element reflects an implementation guide, or a usage, of an X12 transaction set standard and not that full standard. If a construct is defined in any level of the X12 transaction set standard but not used in the particular implementation, the corresponding Grammar Element is not required. For example, if an SAC loop for special allowances and charges is defined for the transaction set standard but not used in the implementation, the corresponding GroupDescription Element is not required. If the fourth data element of the BEG beginning segment for the 850 Purchase Order is not used in an implementation, the SimpleElementDescription for BEG04 is not required in the SegmentDescription Element for the BEG. The indentation in the Element column shows the approximate hierarchical relationships. The Allowable Child Elements column lists the specific details of the hierarchy.

NOTE Do Not Include ST and SE Segments

Although ST Transaction Set Header and SE Transaction Set Trailer segments are part of the published standards for transaction sets, they must not be included in the Grammar Element. The utilities consider them control segments and derive their definitions elsewhere.

Table 9.6 shows the X12 data types defined in version 004010 of the X12.6 standard. For the X12 utilities we support only those data types defined in the X12 standards and don't support all the data types we developed in previous chapters. I do, however, show the correspondence to the other Babel Blaster data types. The correspondence to schema language data types is somewhat approximate in many cases.

Again, I include Truncatable as an optional Attribute for all target X12 simple data elements. This is not part of the X12 standard but is a feature of the XMLToX12 utility. For all types, a runtime error occurs if Truncatable is false and the length of the XML Element contents exceeds the field length. If, on the other hand, you want to allow truncation, you can include the Attribute in the SimpleDataElement Element and set the value to true.

Example File Description Documents

The file description documents for the X12 850 Purchase Order and 810 Invoice examples are available from the book's Web site as:

  • PurchaseOrderX12SourceDescription.xml

  • InvoiceX12TargetDescription.xml

Table 9.5. Transaction Set Grammar Characteristics in the Grammar Element

Element

Allowable Child Elements

Attribute

Schema Language Data Type

Description

Allowable Values, Restrictions, or Comments

Grammar

SegmentDescription, GroupDescription

Describes the grammar of both the transaction set and the corresponding XML representation.

The Grammar Element may have any combination of Segment-Description or GroupDescription Elements as children.

ElementName

NMTOKEN

Specifies the name of the document's root Element.

When creating XML documents, the specified name is assigned to the document's root Element. When creating an X12 interchange, the input XML document's root Element must match this name. Maximum length reflects restriction on the length of Element names.

TagValue

token

Specifies the segment identifier of the transaction set's beginning segment.

Note: This is the first segment expected in the implementation following the ST segment.

GroupDescription

SegmentDescription, GroupDescription

Describes a segment group.

One GroupDescription Element is required for each position at which the segment group might appear in the implementation. The first child Element must be a Segment-Description Element. It may be followed by any combination of SegmentDescription or Group Description Elements.

ElementName

NMTOKEN

Specifies the name of the Element representing the group.

Maximum length reflects restriction on the length of Element names.

TagValue

token

Specifies the segment identifier of the group's beginning segment.

SegmentDescription

CompositeStructure-Description, SimpleElement-Description

Describes the grammar of an individual X12 segment and the corresponding XML representation.

One Segment Description Element is required for each position at which the segment might appear in the implementation. One or more Simple ElementDescription or CompositeStructure Description Elements are required to specify the data elements used within the segment.

ElementName

NMTOKEN

Specifies the name of the Element representing a segment.

Maximum length reflects restriction on the length of Element names.

TagValue

token

Specifies the value of the segment identifier as specified in the X12 standard.

CompositeStructure-Description

SimpleElement-Description

Describes the grammar of an X12 composite data structure and the corresponding XML representation.

One CompositeStructure- Description Element is required for each position in a segment where the composite data structure might appear. One or more Simple-Element Description Elements are required to specify the X12 component data elements used within this composite.

ElementName

NMTOKEN

Specifies the name of the XML Element representing the data element.

Maximum length reflects restriction on the length of Element names.

FieldNumber

positive-Integer

Specifies the data element position within the segment.

Maximum value reflects restriction on the maximum number of data elements per segment.

SimpleElement-Description

None

Describes the characteristics of an X12 simple data element and the corresponding XML representation.

One SimpleElement Description Element is required for each position in a segment or composite data structure where the simple data element might appear.

ElementName

NMTOKEN

Specifies the name of the XML Element representing the data element.

Maximum length reflects restriction on the length of Element names.

FieldNumber

positive-Integer

Specifies the data element position within the segment.

Maximum value reflects restriction on the maximum number of data elements per segment.

SubField-Number

positive-Integer

Specifies the component data element position within the composite data structure.

Optional; this Attribute is required only when the data element is part of a composite data structure. It has a default value of zero if missing. Maximum value reflects restriction on the maximum number of component data elements per composite data structure.

DataType

token

Specifies the X12 data type of the data element.

See Table 9.6.

MinLength

nonNegative-Integer

Specifies the minimum length for the data element, as specified in the standard or implementation guide.

MaxLength

positive-Integer

Specifies the maximum length for the data element, as specified in the standard or implementation guide.

Truncatable

boolean

Indicates whether or not truncation is permitted. See comments regarding truncation in Table 9.6.

Optional, defaults to false.

Table 9.6. X12 Data Types

X12 Data Type

Grammar Data Type Code

Schema Language Data Type

Actions with X12 as Source

Actions with X12 as Target

Action with X12 as Target if Truncatable Is True

Comments and Restrictions

Numeric, Nn

X12-Nx, where x represents the number of implied decimal places (corresponds to Nx)

Decimal

Leading zeroes are removed. All whitespace is trimmed.

The number is rightjustified within the field. If the number of fractional digits in the source decimal number exceeds x, the number is right-truncated to x fractional digits. Zeroes are added as fractional digits if the source number has fewer than x fractional digits. Leading zeroes are added if the source is shorter than the target. If a minus sign character is present in the source, it appears in the left-most position in the target. The plus sign is not converted.

Ignored, not truncatable.

Decimal Number, R

X12-R (corresponds to R)

Float or double

Leading zeroes are removed. All whitespace is trimmed.

The number is right-justified within the field. Leading zeroes are added if the source is shorter than the target. If a minus sign character is present in the source, it appears in the left-most position in the target. The plus sign is not converted.

Fractional digits to the right of the decimal point are truncated until the Element contents are equal to the field length. An error occurs if digits to the left of the decimal exceed the field length.

The X12 decimal number data type supports base 10 exponential notation using an uppercase E followed bythe exponent. The schema language float and double data types also allow a lowercase e, which we convert to an uppercase E. The schema language data type allows the special values—0, INF, -INF, and NaN for minus zero, positive infinity, negative infinity, and "not a number," respectively. These are not converted and force an error if present since there are no equivalent X12 values.

Identifier, ID

X12-ID

token, constrained with enumeration facets

Leading and trailing whitespace (any character with an integer value less than or equal to a space character) is trimmed. All other whitespace within the string is preserved.

If the source is shorter than the minimum length, the data is left-justified and filled to the right with spaces. Leading and trailing whitespace is trimmed. If trimming results in a single space character, the data element becomes empty (with exceptions for ISA data elements).

Not truncatable.

Identifier data elements may either have their values enumerated in the standard or reference an external code list. While it is generally the rule that neither leading nor trailing spaces are present, trailing spaces are allowed if required to satisfy minimum length requirements. ID data elements with only spaces are allowed in the ISA segment.

String, AN

X12-AN (corresponds to AN)

string

Leading and trailing whitespace (any character with an integer value less than or equal to a space character) is trimmed. All other whitespace within the string is preserved.

If the source is shorter than the minimum length, the data is left-justified and filled to the right with spaces. Leading and trailing whitespace is trimmed. If trimming results in a single space character, the data element becomes empty.

The string is right-truncated to the field length.

X12.6 says that leading spaces are considered significant. However, in more than ten years of working with EDI I've never seen them used. We're going to trim them if we find any.There is another significant difference between the data types. In the X12 there must be at least one nonspace character, while a single space character in XML is schema valid.

Date, DT

X12-DT

date

For six-digit dates the prefix 20 is added for hundred years.

The thousand and hundred years are omitted for six-digit dates.

Ignored, not truncatable.

The X12 committee sort of punted when it came to defining dates. Both YYYYMMDD (or CCYYMMDD if you pre -fer) and YYMMDD are supported in the X12.6 definition. However, most date data elements, such as 373 Date, use YYYYMMDD since they have both minimum and maximum lengths of 8.

Time, TM

X12-TM

time

Seconds, if absent, are added. Fractional seconds, if specified, are preserved.

N/A

Fractional seconds and seconds are truncated to satisfy maximum length requirements.

Binary, B

Not supported.

The concepts and appearance are very similar to what we saw in previous chapters, so I'm not inserting the file description documents for the X12 utilities into the book. However, there are a few very important points that need to be made. These particular file description documents reflect usage as defined in an implementation and not the full standard. But, again in keeping with our strategy of using schema validation as the primary mechanism for enforcing business rules, you won't see any mandatory, optional, maximum count, or code list information in these definitions. If you wish to enforce these types of constraints for a particular implementation, you can develop a schema for the XML representation of the transaction set.

This aspect of the file description documents leads to another observation about EDI conversions. The file description documents describe transaction set grammar only to a level sufficient for an XML representation. While these two examples describe only the segments and data elements that appear in these particular implementations, there would be no harm if they described items that aren't used. These unused groups, segments, and data elements in the grammar would simply be ignored during processing. For this reason it would be possible to develop a file description document directly from the standard for a transaction set that covers all possible implementations of the transaction set. So, while we would probably want schemas that validated particular implementations, we could conceivably have only one file description document for each version and release of a transaction set.


Schema Examples

As you might expect, there are several approaches for defining schemas for the XML representations of our X12 transaction sets. There is no single approach that is necessarily the "right" approach for everyone. For the XML representations of our flat files and CSV files I took the very easy route of letting a tool such as XMLPY generate the schemas for me. A more comprehensive approach might be to develop type library schemas, as discussed in Chapter 4, for the X12 Data Element Dictionary and Segment Directory. We could then develop instance document schemas for transaction sets that used data element and segment type definitions from the type libraries. An ambitious hacker (with enough cash to purchase the "table data" version of the standards from X12) could even develop a utility that created all these schemas in an automated fashion. However, the main drawback to such an approach (aside from the coding and expense) would be that the resulting schemas would validate against only the full standard and not against a particular implementation. Most users are concerned with validating against their particular use of a transaction set standard, that is, their implementation, instead of the full standard. Another side effect is that, given the effort and expense of developing these type libraries, there might be only one (or at most a few) versions developed and shared among several users. This would lock users into the XML Element names defined in the type libraries and restrict the flexibility built into the utilities.

I started writing this particular section with the idea that I might develop type library schemas just to present another approach. However, after considering the issue and writing the previous paragraph I have persuaded myself (and I hope you as well) that again the simple approach is the best. The sample X12PurchaseOrder and X12Invoice schemas listed below were mostly autogenerated by XMLSPY. Again, I modified them a bit to correct values such as minimum and maximum occurrences, schema language data type, and enumerations.

Since this chapter is fairly long already, I won't include the schemas directly. You may retrieve them from the book's Web site as:

  • X12PurchaseOrder.xsd: for the XML representation of our X12 850 Purchase Order

  • X12Invoice.xsd: for the XML representation of our X12 810 Invoice


Supplemental Data Store for Control Numbers

In reviewing the utilities developed in this chapter you can start to get the feeling of how a comprehensive file conversion application might look. Such systems usually have internal data stores for things such as configuration information and auditing. In this chapter we have such a data store for interchange and group control numbers. It has been very common for application developers to implement such data stores as relational databases. Having made the choice to store information about file descriptions as XML documents, we are already on the path to using XML documents as our data stores. For reasoning very similar to what we used in Chapter 6 when considering how to store our file descriptions, we'll now formalize this decision to use XML documents for all our data stores.

The data store described here is a preliminary implementation and will probably evolve as the Babel Blaster project develops. For that reason, and because it isn't normally directly accessed or manipulated by users, I give only a high-level description and don't discuss the schema.

Control numbers used in generating interchange and functional group headers are maintained in EDIControl.xml. In moving toward Babel Blaster version 1.0, this file is located in a BBHOME directory defined by a system property in the Java implementation and an environment variable in the C++ implementation. For this version of the utility, we're only concerned with control information related to outbound documents, that is, those where X12 is the target format. This information is captured in the X12OutboundControl Element. Table 9.7 shows the organization of this Element. Indentation in the Element column shows hierarchical relationships.

The XMLToX12 utility creates the appropriate entries in the EDI control document if they don't already exist. If they do exist, the LastInterchangeNumber Element and the appropriate LastGroupNumber Element are updated. The X12 standards are not very specific about control number management, so there are several different approaches. Some EDI management systems allow the users to maintain several different interchange sender IDs and to associate more than one group sender ID with an interchange sender ID. The small to medium business users for whom this book's utilities are targeted probably don't need such comprehensive functionality, so I track only control numbers generated by the receiver ID. Interchange control numbers are maintained by the trading partner's interchange receiver ID and incremented by one for each interchange generated. A separate group control number is maintained by the trading partner for each combination of group receiver ID and functional group type. The group control number also is incremented by one for each group generated.

Table 9.7. Organization of the X12OutboundControl Element

Element

Attribute

Schema Language Data Type

Comments

TradingPartner

Tracks the control information for a trading partner. One TradingPartner Element is required for each unique interchange receiver ID.

ReceiverID

token

The interchange receiver identifier in ISA08.

LastInterchangeNumber

positiveInteger

Used to populate the interchange control number in ISA13.

FunctionalGroup

Tracks the control information for each functional group type sent to the trading partner.

ReceiverID

token

The application receiver's code in GS03.

FunctionalIDCode

token

The functional identifier code in GS01.

LastGroupNumber

positiveInteger

Used to populate the group control number in GS06.


High-Level Design Considerations

Grammar Analysis and Description

I have mentioned that EDI grammar is the most complicated grammar we'll deal with in this book because we must be concerned with both the overall file structure and the grammar of an individual record (or EDI segment). In this section I will finally prove it to you.

It takes some fairly complicated processing to handle EDI grammar. I prefer to tackle this task only once by developing a generalized approach that can be used for most classes of EDI grammars. For that reason we'll consider grammar features that aren't necessarily part of version 004010 of X12 but may be features of later versions or other syntaxes such as UN/EDIFACT's ISO 9735 syntax. We're not going to concern ourselves with interchange and group structure in this analysis since they are fairly fixed for each syntax and sometimes require special processing. However, where applicable we'll apply the segment grammar analysis to our processing of control segments.

As usual, we'll start with the overall structure of a logical document. In X12 this is a transaction set and in UN/EDIFACT a message. In looking at the overall structure, as is the case with segment grammar, we are concerned not with the definition of a document standard, per se, but with the grammar of an instance of the document as it appears in a data stream.

EDI Document Grammar
document ::= (segment | segment_group)*
group ::= segment (segment | segment_group)*

Note that the first production differs from its counterpart in the flat file analysis. In the flat file analysis we defined a document as a group. Header and trailer control segments are generally considered part of the published document standards, and if we included them, the group production would be applicable. However, we have omitted them, and we have a different production for this document because the first segment in a document following the header control segment is not necessarily mandatory. In X12 transaction sets it is customary for a mandatory segment such as the BEG or BIG in our examples to be defined immediately following the ST segment. However, this is not part of X12.6 and the X12 Design Guidelines specifically state that transaction sets are not required to have a unique beginning segment. I've not thoroughly reviewed ISO 9735, the UN/EDIFACT design rules and guidelines, or the message definitions. However, knowing that the X12 syntax has this caveat is sufficient grounds for expressing the first production as I have.

If I had included the header control segment in the grammar then, like the flat file grammar, the first production could equate a document with a group. There are tradeoffs with both approaches. Had header and trailer control segments been included the processDocument and processFile methods could be developed very simply. Like those in the flat file classes they could primarily just call processGroup. However, while making things easier for the developer this approach makes things harder in a few ways for the user. The user would have to include the header and trailer control segments in the Grammar Element of all file description documents. When converting from XML to X12 the user would have to provide the XML representations of them also. This would require code in every component that produces such XML documents, such as XSLT style sheets. When given a choice between making things harder for users and making things harder for developers, I generally favor users. Hence, we use the first production as written above.

The definition of a segment group in most EDI syntaxes is fairly specific, and the generalized grammar in the second production should cover all of them. It is worth noting that X12.6 specifically states that the first segment of a segment group shall not appear in any other position within the group. The production accommodates this restriction. On the other hand, UN/EDIFACT is not so restrictive. There is at least one message, BAPLIE, in which the starting segment of a segment group may appear later within the group. Parsing ambiguity is prevented by making mandatory at least one segment that precedes the second usage and one segment that succeeds it. Our grammar is general enough to also accommodate this model.

We complete the BNF by describing the grammar of an EDI segment. These productions accommodate both repeating data elements and the release character used in UN/EDIFACT's ISO 9735 syntax. Note that the terminology in the productions doesn't follow X12 exactly in all cases since we're trying to describe an abstract grammar that also applies to other EDI syntaxes. However, the concepts are the same.

EDI Segment Grammar
segment ::= segment_identifier
            (element_separator data_element?)*
            element_separator data_element segment_terminator
segment_identifier ::= char char char?
data_element ::= simple_element | composite_structure |
                 repeating_element
composite_structure ::=
    (component_element? component_separator)* component_element
component_element ::= simple_element
repeating_element ::=
    ((simple_element? repetition_separator)* simple_element ) |
    ((composite_structure? repetition_separator)*
        composite_structure)
simple_element ::=  ((release_char special_char) | char )+
delimiters ::= segment_terminator | element_separator |
               component_separator | repetition_separator |
               release_char
segment_terminator ::= special_char
element_separator ::= special_char
component_separator ::= special_char
repetition_separator ::= special_char
release_char ::= special_char
special_char ::= a character from the approved character set that
    doesn't appear in the data stream (except for the
    release_char) and that is not used for any of the other
    delimiters
char ::= a character from the approved character set, excluding
    delimiters unless preceded by the release_char

Though this is not nearly as complicated as even the basic XML 1.0 grammar, it is still a somewhat complicated grammar, if only for the number of productions. Let's see what it actually says.

  • The segment production says that a segment beings with a segment identifier and ends with an element separator, a data element, and a segment terminator. In between the segment tag and the final element separator there may be zero or more data elements or empty data elements, each preceded by an element separator. This production specifically says that trailing element separators are not allowed.

  • The segment identifier production says that it is a two- or three-character string.

  • The data element production amplifies how the data element is used in the segment production, saying that it can be a simple data element, a composite data structure, or a repeating data element.

  • The composite structure production says that a composite data structure must end with a simple data element. The final simple data element may be preceded by zero or more simple data elements or empty data elements, each followed by a component data element separator.

  • The repeating element production says that it may be composed of either simple data elements or composite data structures, but not both. It must end with a data element that may be preceded by zero or more data elements or empty data elements, each followed by the repetition separator.

  • The remaining productions clarify simple data elements and delimiters.

The grammar of control segments is a subset of this grammar. Repeating data elements are not allowed. X12 control segments do not contain composite data structures, and the release character is not supported in X12. I have also heard it asserted that the release character is not allowed in UN/EDIFACT service segments. I've been unable to confirm this assertion in either version 2 or 4 of ISO 9735. However, even if the release character is permitted in a UN/EDIFACT service segment, I've never seen one actually used. I think it highly unlikely that people would use a character needed for releasing in any of the identifiers used in these segments. For these reasons we can often use somewhat simpler approaches (such as string token routines) for processing control segments.

So, if you weren't convinced before that EDI grammars can be fairly complicated, I hope you are by now! I frequently encounter people who think that XSLT is an appropriate tool for parsing raw EDI. I still on occasion field questions from people who want to write their own programs to process raw EDI rather than buying a translator or using something like the utilities developed in this chapter. A lot of pain, suffering, and avoidable expense would be saved if such people reviewed this or a similar analysis before contemplating such dubious endeavors.

This analysis prepares the ground for the parsing algorithm we'll develop for the EDIRecordReader's parseRecord method. It will also help make clearer some of the processing performed in the EDIRecordWriter's writeRecord method.

File Description Document Schemas

As with our other formats, there are four schemas involved in describing X12 interchanges and their xml representation.

  1. X12SourceFileDescription.xsd: This schema is for file description documents that describe conversions in which the source is an X12 interchange.

  2. X12TargetFileDescription.xsd: This schema is for file description documents that describe conversions in which the target format is an X12 interchange.

  3. X12CommonFileDescription.xsd: This type library schema is used by the two previous schemas.

  4. BBCommonFileDescription.xsd: Again, this is our common type library.

The schemas introduced in this chapter are very similar to those developed in Chapters 7 and 8. The main difference in our X12 schemas is that instead of using the FieldGrammarType from our common schema, we define an ElementGrammarType. The main reason we do this is that our X12 utilities don't support all the other data types we have defined. We create a new X12DataType simple type for the DataType Attribute instead of using the BBDataType.

Note that the SegmentType complex type, unlike the flat file's FlatRecordType and the CSV's RowDescription Element, allows two different types of child Elements in its sequence. We can have a mix of SimpleElementDescription Elements and CompositeStructureDescription Elements. We indicate this with a choice content model within the sequence.


X12 to XML: Detail Design

If you have reviewed Chapters 7 and 8, you should have a fairly good understanding of the architecture and basic coding approach used for these utilities. You will find that the utilities developed in this chapter have many similarities to those developed in the two previous chapters. However, X12 interchanges are very different in some key aspects from CSV files and flat files, so we will see some interesting processing differences. As with Chapter 8 I will focus on the unique aspects of these utilities and refer you back to Chapter 7 if you wish to better understand the architecture and coding approach.

Main Program

As with the other utilities our main program is a basic shell that calls methods from the SourceConverter base class.

Logic for the Shell Main Routine for X12 to XML
Arguments:
  Input File Name
  Output Directory Name
  Directory Path Name for File Description Documents

Options:
  Validate output
  Help

Validate and process command line arguments
IF help option specified
    Display help message
    Exit
ENDIF
Create new X12SourceConverter object, passing:
    Validation option
    Output Directory
    Directory Path for File Description Documents
Call X12SourceConverter processFile method, passing
    the input file name
Display completion message

The main difference between this and the CSV and flat file main programs is that instead of dealing with a specific file description document we deal with a directory path where we can find them. This is due to the fact that an X12 interchange may contain several types of transaction sets, each of which requires a different file description document. We use information from the control segments in the X12 interchange to identify the type of conversion we need to perform. Then we load the appropriate file description document from the source directory based on the naming convention specified at the beginning of this chapter.

X12SourceConverter Class (Extends SourceConverter)
Overview

The X12SourceConverter is the main driver for the actual conversion. It inherits all the attributes and methods of its base classes, the SourceConverter and Converter classes.

Attributes:

  • X12RecordReader Object

  • String FDD Path (file description document directory path)

  • String Saved FDD Name

  • String Output Directory Path

  • String FA Output Directory Path

  • String Group SenderID

  • String GroupID

  • String X12Version

  • String Tag

  • String Included Transaction Sets

  • Integer Received Transaction Sets

Methods:

  • Constructor

  • processFile

  • processDocument

  • writeFunctionalAck

Methods
Constructor

The constructor method for our X12SourceConverter object sets up that object as well as the X12RecordReader object.

Logic for the X12SourceConverter Constructor Method
Arguments:
  Boolean Validation option
  String Output Directory Name
  String File Description Document Directory Path

Call base class constructor
FDD Path <- From passed path name
Create FA_Out directory
Create X12RecordReader object
processFile

The main processing is driven by the X12SourceConverter's processFile method. Note that there are several differences between this method and the one in the FlatSourceConverter. This method processes the control segments for the interchange and functional groups, then calls processDocument to process each transaction set.

Logic for the X12SourceConverter processFile Method
Arguments:
  String Input File Name

Returns:
  Status or throws exception

Open input file
Call X12RecordReader's setInputStream method
Call X12RecordReader's parseISA method
Call X12RecordReader's logISA method
Record Length <- Call RecordReader's readRecordVariableLength
IF Record Length = 0
  Return Error
ENDIF
Tag <- Call X12RecordReader's getRecordType
DO while Tag = GS to process all functional groups
  Call X12RecordReader's parseGS method
  Call X12RecordReader's logGS method
  Group Sender ID <- Call X12RecordReader's
      getControlSegmentElement for GS02
  Group ID <- Call X12RecordReader's
      getControlSegmentElement for GS01
  X12Version <- Call X12RecordReader's
      getControlSegmentElement for GS08
  Output Directory Path <- Base Directory +
      Group Sender ID + Group ID + directory separator
  Create Output Directory
  Record Length <- Call RecordReader's readRecordVariableLength
  IF Record Length = 0
    Return Error
  ENDIF
  Tag <- Call X12RecordReader's getRecordType
  Received Transaction Sets <- 0
  DO while (Tag = ST) to process all transaction sets
    Increment Received Transaction Sets
    Status <- call processDocument
    IF Status != SUCCESS
      Return error
    ENDIF
  ENDDO
  IF Tag != GE
    Return Error
  ENDIF
  Call X12RecordReader's parseGE method
  Call X12RecordReader's logGE method
  Included Transaction Sets <- call X12RecordReader's
      getControlSegmentElement for GE01
  IF GSID != "FA"
    Call writeFunctionalAck
  ENDIF
  Record Length <- Call RecordReader's readRecordVariableLength
  IF Record Length != 0
    Tag <- Call X12RecordReader's getRecordType
  ELSE
    Return error
  ENDIF
ENDDO
IF (Tag != IEA)
  Return Error
ENDIF
Call X12RecordReader's parseIEA method
Call X12RecordReader's logIEA method
Close input file
Return success
processDocument

The processDocument method reads and processes a single X12 transaction set starting with the ST header and ending with the SE trailer. This method loads the appropriate file description document if one is not already loaded, and based on it the method creates and saves a single output XML document representing the transaction set.

Logic for the X12SourceConverter processDocument Method
Arguments:
  None

Returns:
  Status or throws exception

Call X12RecordReader's parseST method
Call X12RecordReader's logST method
Transaction SetID <- From ST01
FDD Name <- FDD Path + Group Sender ID + '-' + Group ID + '-' +
    Transaction SetID + "-" + Version/Release + ".xml"
IF (FDD Name != Saved FDD Name)
  Call loadFileDescriptionDocument, passing FDD Name
  Call EDIRecordReader's setFileDescriptionDocument
  Saved FDD Name <- FDD Name
  Schema Location URL <- from File Description Document
  Throw exception if no schema and validation requested
ENDIF
Create new Output DOM Document
Create Root Element, using Root Element Name from Grammar,
    and append to Output DOM Document
IF Schema Location URL is not NULL
  Create namespace Attribute for SchemaInstance and append
      to Group Element
  Create noNamespaceSchemaLocation Attribute and append
      to Root Element
ENDIF
Call RecordReader's setOutputDocument method for new
    document
Transaction Set Control Num <- ST02
Output File Path <- Output Directory Path + Root Element
    Name + Transaction Set Control Num + ".xml"
Record Length <- Call RecordReader's readRecordVariableLength
IF Record Length = 0
  Return error
ENDIF
Tag <- Call X12RecordReader's getRecordType
Child Node <- Get Grammar Element's firstChild
DO while Child Node is not an Element Node
  Child Node <- ChildNode's nextSibling
ENDDO
Segment Grammar Element <- Child Node
Grammar Tag <- Call Segment Grammar getAttribute for "TagValue"
DO until (Tag = SE) at end of transaction set
  DO until Grammar Tag = Tag
    Child Node <- ChildNode's nextSibling
    IF Segment Grammar Element is NULL
      return error  //  This Segment is not part of the document
    ENDIF
    IF Child Node nodeType != ELEMENT
      CONTINUE
    ENDIF
    Segment Grammar Element <- Child Node
    Grammar Tag <- Call Segment Grammar getAttribute for
        "TagValue"
  ENDDO
  Grammar Element Name <- Segment Grammar getNodeName
  IF Grammar Element Name = "GroupDescription"
    Record Length = Call processGroup, passing Root Element and
        Segment Grammar Element
  ELSE
    Call X12RecordReader's parseRecord, passing
        Segment Grammar Element
    Call X12RecordReader's toXML
    Call X12RecordReader's writeRecord, passing Root Element and
        Segment Grammar Element
    Record Length <- Call RecordReader's
        readRecordVariableLength
  ENDIF
  IF Record Length = 0
    Return error
  ENDIF
  Tag <- Call X12RecordReader's getRecordType
ENDDO
Call X12RecordReader's parseSE method
Call X12RecordReader's logSE method
Call saveDocument
Record Length <- Call RecordReader's readRecordVariableLength
IF Record Length = 0
  Return error
ENDIF
Tag <- Call X12RecordReader's getRecordType
Return Success

The logic and operations are very similar to those in the SourceConverter's processGroup method developed in Chapter 8. You'll note that the logic for the main DO loop here is almost an exact match for the logic in the main DO loop of that method. In addition to the functional differences noted at the beginning of this section, the main differences are due to the structure of X12 interchanges as defined in the standards. A transaction set must start with an ST segment and end with an SE segment. We enter the method after having just read an ST segment, and the last segment we process is an SE segment. We exit the method after reading the next segment. We also know from the standards that the trailing SE segment must be followed by either another ST header segment or the GE group trailer segment. If we reach the end of the file at any point in this routine, we are processing an incomplete interchange and return an error.

writeFunctionalAck

This method writes an XML representation of the 997 Functional Acknowledgment. Since we use the 997 only as a return receipt, to indicate acceptance of the group and not to report errors, we write data only sufficient for writing the AK1 and AK9 segments.

Logic for the X12SourceConverter writeFunctionalAck Method
Arguments:
  None

Returns:
  Status or throws exception

Output File Name <- FA Output Directory Path + GS Sender ID + "-" +
    Functional Group ID Code + ".xml"
Create output DOM FA Document
Create Root Element "TS997" and append to FA document
Create AK1 Element and append to Root
Create AK101 Element and append to AK1
Create Text Node with Functional Group ID Code
    and append to AK101
Create AK102 Element and append to AK1
Create Text Node with GS Control Number and append to AK102
Create AK9 Element and append to Root
Create AK901 Element and append to AK9
Create Text Node with "A" and append to AK901
Create AK902 Element and append to AK9
// AK902 is count of included transaction sets from GE01
Create Text Node with Included Transaction Sets
   and append to AK902
// AK903 is count of transaction sets actually received in group
Create AK903 Element and append to AK9
Create Text Node with Received Transaction Sets
   and append to AK903
// AK904 is count of transaction sets accepted - We're going to
//   say we accepted all of them that we received
Create AK904 Element and append to AK9
Create Text Node with Received Transaction Sets
   and append to AK904
Save FA Document

EDIRecordReader Class (Extends RecordReader)

Overview

The EDIRecordReader is the generalized base class for several derived classes that deal with specific EDI syntaxes. In this chapter we develop an X12RecordReader derived class. However, as you'll see, the attributes and methods we develop here are applicable to other syntaxes, so we move them to a common base class.

Attributes:

  • Character Element Separator

  • Character Component Separator

  • Character Repetition Separator

  • Character Release Character

Methods:

  • Constructor

  • getRecordType

  • parseRecord

  • setFileDescriptionDocument

  • writeRecord

Methods
Constructor

The EDIRecordReader constructor method does very little processing. It calls the base RecordReader constructor method, passing along the null value for the file description document, and initializes the class member attributes.

getRecordType

This method extracts the segment identifier from the input record buffer.

Logic for the EDIRecordReader getRecordType Method
Arguments:
  None

Returns:
  Record ID tag value; throws exception or returns status

Record Tag Value <- Get first characters of input record up to
    the Element Separator
Return Record Tag Value or status
parseRecord

This is perhaps the most interesting method developed in this chapter. To develop it we draw on the BNF grammar analysis presented at the start of the High-Level Design Considerations section. We'll start with some general observations that lead us to the characteristics and outlines of a processing algorithm. The state transition diagram will follow that. We'll finish off with the full algorithm.

As I said in the grammar analysis, the grammar of EDI segments is fairly complex. If for no other reason this is due to the number of productions. It would take a fairly complex state machine and associated algorithm to directly implement the grammar I outlined. Fortunately, we don't have to take that approach. We can simplify things somewhat by observing that we mainly want to extract data from simple data elements. From strictly that perspective we don't care very much whether an element is a stand-alone simple data element, a component data element within a composite data structure, an instance of a repeating simple element, or a component data element within an instance of a repeating composite. We do, of course, in the big picture care about these distinctions, but not when it comes to parsing the input segment. We care about them when we actually save the data. The good news is that we have a way to make these distinctions when we save the data. We have the segment grammar, and if we walk the grammar as we parse the segment, we can make the necessary distinctions.

So, here is the general outline of a processing algorithm. We maintain pointers to the current DataCell and the current grammar Element. We also maintain a pointer to the grammar for the current composite data structure.

  • We start processing by parsing past the segment identifier to the first element separator.

  • If we encounter a delimiter such as an element separator, a component data element separator, or a repetition separator, we point the current grammar Element to the appropriate element within the segment or composite grammar. We also clear the pointer to the current DataCell.

  • If we encounter a normal character in an input stream, we save it. If the current DataCell pointer is null, we first create a new DataCell, passing the current grammar Element.

  • If we encounter the release character, we treat the next character as a normal character and save it.

  • If we encounter a delimiter that should not immediately follow a different delimiter, we move to an error state. Examples of this type of error would be the cases of a repetition separator or a data element separator immediately following a component data element separator. These cases indicate an invalid segment because trailing component data element separators within a composite data structure are not allowed.

  • The last character in the segment immediately preceding the segment terminator should be a normal character and not a delimiter. We can check for this condition by testing whether or not the current DataCell pointer is null. If it is null, the last character processed was a delimiter and we exit the routine with an error. If it isn't null, the last character processed was a normal character and we exit the parsing routine normally.

With this outline we can construct the state transition diagram shown in Figure 9.1. Four states correspond to each of our four delimiters, one state corresponds to a data element, and one is an error state. To make the state machine more closely reflect the sense of the algorithm, I named the states for the actions taken rather than for the symbol from the segment grammar.

  • Next Element, entered after parsing the data element separator: If the element position within the segment is greater than the Field Number of the current Grammar Element, advance the current Grammar Element until the Field Number is equal to or greater than the Element Position. If greater, set the current Grammar Element to null. Otherwise, if the current Grammar Element describes the grammar of a composite data structure, point the current composite grammar pointer to it and set the current Grammar Element to point to its first child Element. Otherwise, clear the current composite grammar pointer. Clear the DataCell pointer.

  • Next Component, entered after parsing the component data element separator: Point the current Grammar Element to the next child Element of the current composite Grammar Element. Clear the DataCell pointer.

  • Next Repeat, entered after parsing the repetition separator: If the composite grammar pointer is null, we don't change the current Grammar Element. If it is not null, we reset the current Grammar Element to point to the first child of the current composite Grammar Element. Clear the DataCell pointer.

  • Escape, entered after parsing the Release character: Set the current state to Save Character so that the next character in the segment is saved.

  • Save Character, entered after parsing a normal character or an escaped delimiter: If the DataCell pointer is null, create a new DataCell. Save the current character to the DataCell.

  • Error, entered after encountering a parsing error in one of the other states: Terminate processing and exit the routine.

Figure 9.1. State Transition Diagram for Parsing an EDI Segment

In Figure 9.1 the double circle on the Save Character state indicates an accepting state and on the Error state indicates a termination state.

From the perspective of more formal analysis, this outline makes a lot of sense. If we were to successively replace each of the nonterminal symbols in the productions of the segment grammar with terminal symbols (with a bit of leeway for the char and special char productions), we could eventually reduce the grammar to a single production. It would not be very readable or useful, but it would be a single production. Most importantly, it would be a single production that could be converted to a regular expression. This is significant because it proves that we can process a segment grammar with a finite state automaton and the type of state transition diagram shown in Figure 9.1. I leave to you the exercise of actually performing the reduction and creating the regular expression. The diagram is convincing enough for me.

Finally, we combine the outline of the algorithm and the information from the state transition diagram, yielding the following processing algorithm.

When we enter the method there's a segment loaded in the Record Buffer and we have identified the segment type. The method has also been passed the Grammar Element for the segment. When reviewing the algorithm, bear in mind that if a data element isn't used in a segment or composite data structure, we don't require that it be defined in the grammar. So, we advance through the grammar as we advance through the appropriate delimiters, but we match data element position within segment against the Grammar Element's FieldNumber Attribute and component data element position within composite data structure against the Grammar Element's SubFieldNumber Attribute.

The Save Character state is the only accepting state or successful completion state. If we exit the main loop in any other state, the segment is invalid because it ends with a delimiter. In such cases we move to the Error state.

Logic for the EDIRecordReader parseRecord Method
Arguments:
  DOM Element Segment Grammar

Returns:
  Error status or throw exception

Segment Child <- get Segment Grammar's firstChild
DO until Segment Child nodeType is Element
 Segment Child <- Segment Child nextSibling
ENDDO
Current Grammar <- Segment Child
Grammar Position <- Current Grammar getAttribute for
    Field Number
Composite Grammar <- null
Element Position <- 0
Component Position <- 0
Parsing State <- Next Element
Advance Pos to point to the Record Buffer's first
    Element Separator
DO while Pos < Buffer Length and
    Parsing State != Parsing Error
  CurChar = Record Buffer[Pos]
  //  Advance to the next state
  DO CASE of Parsing State
    Next Element:
      IF CurChar = Element Separator
        Parsing State = Next Element
      ELSE
        IF CurChar = Component Separator
          Parsing State = Next Component
        ELSE
          IF CurChar = Repetition Separator
            Parsing State = Next Repeat
          ELSE
            IF CurChar = Release Character
              Parsing State = Escape
            ELSE
              Parsing State = Save Character
            ENDIF
          ENDIF
        ENDIF
      ENDIF
      BREAK
    Next Component:
      IF CurChar = Element Separator
        Parsing State = Parsing Error
      ELSE
        IF CurChar = Component Separator
          Parsing State = Next Component
        ELSE
          IF CurChar = Repetition Separator
            Parsing State = Parsing Error
          ELSE
            IF CurChar = Release Character
              Parsing State = Escape
            ELSE
              Parsing State = Save Character
            ENDIF
          ENDIF
        ENDIF
      ENDIF
      BREAK
    Next Repeat:
      IF CurChar = Element Separator
        Parsing State = Parsing Error
      ELSE
        IF CurChar = Component Separator
          Parsing State = Next Component
        ELSE
          IF CurChar = Repetition Separator
            Parsing State = Next Repeat
          ELSE
            IF CurChar = Release Character
              Parsing State = Escape
            ELSE
              Parsing State = Save Character
            ENDIF
          ENDIF
        ENDIF
      ENDIF
      BREAK
    Escape:
      Parsing State = Save Character
      BREAK
   Save Character:
    IF CurChar = Element Separator
        Parsing State = Next Element
      ELSE
        IF CurChar = Component Separator
          Parsing State = Next Component
        ELSE
          IF CurChar = Repetition Separator
            Parsing State = Next Repeat
          ELSE
            IF CurChar = Release Character
              Parsing State = Escape
            ELSE
              Parsing State = Save Character
            ENDIF
          ENDIF
        ENDIF
      ENDIF
      BREAK
  ENDDO
  //  Take action appropriate to state
  DO CASE of Parsing State
    Next Element:
      CurrentCell = null
      Increment ElementPosition
      // Get the next the Grammar Element for the next data
      //   element in the segment
      DO while Element Position > Grammar Position
        Segment Child <- Segment Child nextSibling
        IF Segment Child = null
          Return error for no Grammar Element
        ENDIF
        IF Segment Child nodeType != Element
          CONTINUE
        ENDIF
        Grammar Position <- Current Grammar getAttribute for
            "FieldNumber"
      ENDDO
      //  If we find a match then do the setup, otherwise set
      //    the current Grammar Element to null
      IF Element Position = Grammar Position
        Node Name <- Segment Child nodeName
        IF Node Name = "CompositeStructureDescription"
          Composite Grammar <- Segment Child
          Composite Child <- Composite Grammar firstChild
          DO until Composite Child nodeType is Element
            Composite Child <- Composite Child nextSibling
          ENDDO
          Current Grammar <- Composite Child
          Component Position <- 1
        ELSE
          Composite Grammar <- null
          Composite Child <- null
          Component Position <- 0
          Current Grammar <- Segment Child
        ENDIF
      ELSE
        Current Grammar <- null
      ENDIF
      BREAK
    Next Component:
      CurrentCell = null
      Increment Component Position
      Grammar SubPosition <- Current Grammar getAttribute for
          "SubFieldNumber"
      // Get the next Grammar Element for the next data
      //   element in the Composite
      DO while Component Position > Grammar SubPosition
        Composite Child <- CompositeChild nextSibling
        IF Composite Child = null
          Return error for no Grammar Element
        ENDIF
        IF Composite Child nodeType != Element
          CONTINUE
        ENDIF
        Grammar SubPosition <- Current Grammar getAttribute for
            "SubFieldNumber"
      ENDDO
      //  If we find a match then do the setup, otherwise set
      //    the current Grammar Element to null
      IF Component Position = Grammar SubPosition
        Current Grammar <- Composite Child
      ELSE
        Current Grammar <- null
      ENDIF
      BREAK
    Next Repeat:
      // If we're processing a composite, reset the grammar and
      //   index back to the start of the composite. Otherwise
      //   do nothing and stay with the current grammar.
      Current Cell = null
      IF Composite Grammar != null
        Composite Child <- Composite Grammar firstChild
        DO until Composite Child nodeType is Element
          Composite Child <- Composite Child nextSibling
        ENDDO
        Current Grammar <- Composite Child
        Component Position <- 1
      ENDIF
      BREAK
    Release:
      //  Take no action
      BREAK
    Save Character:
      IF Current Cell = null
        IF Current Grammar = null
          Return error for no Grammar Element
        ENDIF
        Current Cell = call createDataCell, passing Element
            Position and Current Grammar
        IF ComponentPosition != 0
          Call Current Cell's setSubFieldNumber, passing
            Component Position
        ENDIF
      ENDIF
      Call Current Cell putByte, passing CurChar
      BREAK
    Parsing Error:
      //  Take no action
      BREAK
  ENDDO
  Increment Pos
ENDDO
//  Check if we are finishing with a delimiter
IF Parsing State != Save Character
  Parsing State = Parsing Error
ENDIF
IF Parsing State = Parsing Error
  Display parsing error message with Element Position,
    Component Position, Pos, and segment contents
  Return error
ENDIF
Return success

One restriction needs to be noted regarding the algorithm. While it parses input correctly, the actions taken in the Next Repeat state may not meet all requirements. The X12 Design Rules imply that the position of a particular instance of a repeating data element may have semantic significance. I've not investigated the UN/EDIFACT rules, but I wouldn't be surprised if they said something similar. But regardless of what the standards say, there may certainly be implementations where position within a repeating data element is significant. For example, there might be a usage in which the first occurrence of an element would be loaded into the first column of a spreadsheet row while the fifth occurrence (or fourth repetition) would be loaded into the fifth column. In this parsing algorithm positional information is lost if there are empty data elements within a repeat since we don't write empty XML Elements from them. If the second repetition were missing in our example, repeats three and later would be shifted one column to the left. I'm not sure how significant this deficiency is, but investigating and perhaps providing better repeat support is among the version 1.0 requirements for the Babel Blaster project. Again, repeating data elements were not supported at all in X12 004010, which is the initial focus of these utilities.

setFileDescriptionDocument

The constructors in the CSV and flat file record readers set the file description document pointer for us. However, when processing EDI we need to read from the interchange before the SourceConverter loads the file description document. In addition, we may use several such documents while processing an interchange. This method gives us a way to set and change the file description document.

Logic for the EDIRecordReader setFileDescriptionDocument Method
Arguments:
  DOM Document File Description Document

Returns:
  Nothing

Base RecordHandler File Description Document <- From passed
   DOM Document File Description Document
Return

Note: I contemplated moving this method to the base RecordHandler class but didn't because I couldn't think of any class other than the EDIRecordReader that needed it.

writeRecord

This method is very similar to the one in the base RecordWriter class. It creates an XML Element that represents a segment (or record) and writes the contents of the DataCell Array to child Elements of the record Element. We could use that base class method if it weren't for the fact that for EDI formats we also write Elements that correspond to composite data structures. These have child Elements that correspond to the component data elements within a composite.

So, we are faced with the classic conundrum of object-oriented analysis and design: What do we put in the generalized base class and what do we put in the specialized derived class? This is a particularly thorny problem when it comes to methods that are very similar in the base and derived classes. My gut feeling tells me not to unnecessarily risk breaking something that already works. So, rather than including logic in the base class method to handle the special case of composite data structures, I'm going to develop a specialized method for this derived class. If it appears similar enough and doesn't break any of the functionality in the base class method, for Babel Blaster version 1.0 I may promote it to the base class.

The main difference between this and the base RecordWriter's writeRecord method is that we get a NodeList of the segment Grammar Element's composite data structure grammars. If a DataCell Array entry has a subField number, we find the grammar for the appropriate composite and make the new Element a child of the composite.

Logic for the EDIRecordReader writeRecord Method
Arguments:
  DOM Element Output Document Parent Element
  DOM Element Record Grammar Element

Returns:
  Status or throws exception

Element Name <- Call Grammar Element's getAttribute for
    "ElementName"
Record Element <- call Output Document's createElement, passing
    Element Name
Parent Element <- call Parent's appendChild to append
    Record Element
Current Parent <- RecordElement
Composite Parent <- null
Composite Position <- 0
Composite Grammars <- call Record Grammar's getElementsByTagName
    for "CompositeStructureDescription"
DO for all DataCells in array up through Highest Cell
  // Test if this is a component within a composite
  Component Position <- call DataCell's getSubFieldNumber
  IF Component Position != 0
    //  See if we are starting a new composite
    Cell Field Number <- Call DataCell's getFieldNumber
    IF Cell Field Number != Composite Position
      // Find the grammar and create the Element
      DO for all Elements in Composite Grammars NodeList until
          Cell Field Number = Composite Position
        Composite Position <- Call Composite Grammars item's
           getAttribute for "FieldNumber"
      ENDDO
      Composite Name <- Call Composite Grammars current item
          Element's getAttribute for "ElementName"
      Composite Parent <- call Output Document's createElement,
          passing Composite Name
      Record Element <- call Record Element appendChild to append
          Composite Parent
      Current Parent <- Composite Parent
    ENDIF
  ELSE
    // simple element
    Current Parent <- Record Element
    Composite Parent <- null
    Composite Position <- 0
  ENDIF
  Call toElement on Cell Array entry, passing Current Parent
  Clear Cell Array Entry
ENDDO
Highest Cell <- -1

I should note that this algorithm does not properly support all aspects of repeating data elements. It will work fine if the repeated data elements are simple data elements. However, it will not work correctly if the repeating unit is a composite data structure. All the child component data elements for all repetitions would be written under one composite parent Element in the output XML document. This is yet another area where current support is limited to version 004010 of X12 and where we have another requirement for Babel Blaster version 1.0. My impression is that we might fix this by adding a RepeatPosition member attribute to the DataCell class. We would set it in the parseRecord method and in this method check for a break in RepeatPosition to see if we need to write a new composite parent Element. We could also use RepeatPosition to write a RepeatPosition Attribute to the output XML Element for a repeating Element.

X12RecordReader Class (Extends EDIRecordReader)

Overview

The EDIRecordReader is the generalized base class for several derived classes dealing with specific EDI syntaxes. The X12RecordReader has attributes and methods that deal specifically with X12 control segments. The current implementation is hard-coded to support version 004010 of X12. One of the Babel Blaster 1.0 requirements is to enable support for different versions.

Attributes:

  • String array for data elements from the ISA Interchange Control Header segment

  • String array for data elements from the GS Functional Group Header segment

  • String array for data elements from the ST Transaction Set Header segment

  • String array for data elements from the GE Functional Group Trailer segment

Methods:

  • Constructor

  • getControlSegmentElement

  • logGE

  • logGS

  • logIEA

  • logISA

  • logSE

  • logST

  • parseGE

  • parseGS

  • parseIEA

  • parseISA

  • parseSE

  • parseST

Methods

Most of the methods do very little, so with a few exceptions I'll discuss them only in general terms. The constructor method calls the EDIRecordReader constructor, initializes the class member attributes, and exits. The parse methods simply use the language-specific string token routines to parse the Record Buffer and load the class member attributes corresponding to the data elements in the segment. The log methods in this initial implementation display segment information from the appropriate class member attributes. A requirement for later Babel Blaster versions is to enhance these log routines to write output to a data store. We may not use or call all the methods in this version, so some of them exist for future requirements.

Note when examining the code that the text strings in the log methods are hard-coded and not set by parameters. The X12 standards are most commonly used by English speakers. Expressing all of the message text as parameters adds coding complexity. Since these methods will probably be completely rewritten in the next version I didn't think it appropriate.

getControlSegmentElement

Since there are several values that other classes need to retrieve from the header and trailer control segments, we develop a generalized method to retrieve them.

Logic for the X12RecordReader getControlSegmentElement Method
Arguments:
  String Segment ID
  Integer Element position within segment

Returns:
  Status or throws exception

IF Segment ID = "ISA"
  Return ISA[Element position]
ENDIF
IF Segment ID = "GS"
  Return GS[Element position]
ENDIF
IF Segment ID = "GE"
  Return GE[Element position]
ENDIF
IF Segment ID = "ST"
  Return ST[Element position]
ENDIF
IF Segment ID = "SE"
  Return SE[Element position]
ENDIF
Return failure
parseISA

The parseISA method is perhaps the most important method in the X12Record Reader class. It parses the Interchange Control Header segment and loads the base class delimiters based on the parsing results.

Logic for the X12RecordReader parseISA Method
Arguments:
  None

Returns:
  Status or throws exception

Record Buffer <- call language-specific routines to read 106
    bytes from input file stream
Buffer Length <- 106
IF character at Record Buffer index 93 = '-'
  Get one more byte from input file and append to Record Buffer
  Buffer Length <- 107
ENDIF
IF Record Buffer doesn't start with "ISA"
  Return Error for not an ISA segment
ENDIF
Element Separator <- Record Buffer at index 3
Component Separator <- Record Buffer at index
    Buffer Length - 2
Record Terminator1 <- Record Buffer at index
    Buffer Length - 1
Using language-specific string token routines,
    load class attributes ISA01 through ISA16

Once again this method is hard-coded to process version 004010 of X12 since it doesn't look for the Repetition Separator in ISA11.

One other little bit of code deserves comment. It is a common belief that the ISA segment has a fixed length of 106 bytes. Most commercially available EDI translators rely on this assumption. Strictly speaking, the ISA segment doesn't have a fixed length; all the data elements within it do since their minimum length is the same as their maximum length. However, the N0 data type of the ISA13 Interchange Control Number allows a leading minus sign, and X12.6 says that leading minus signs are not counted against maximum length requirements. So, in nearly all circumstances an actual data stream has a control number of 9 bytes. However, I do know of at least one usage (to me, an odd one) where a negative control number can appear and ISA13 is 10 bytes long, yielding a 107-byte ISA segment. It takes only a couple lines of code to support this little quirk, so I thought I would throw it in.


XML to X12: Detail Design

Main Program

The shell main program functions are very similar to those of the other XML to legacy format utilities. We do, however, add calls (1) before processing all documents to write the X12 interchange and group header segments, and (2) after finishing to write the trailer segments.

Logic for the Shell Main Routine for XML to X12
Arguments:
  Input Directory Name
  Output X12 Interchange Name
  File Description XML Document Name

Options:
  Validate input
  Help

Validate and process command line arguments
IF help option specified
  Display help message
  Exit
ENDIF
Open output file
Create new X12TargetConverter object, passing:
    Output Stream
    File Description Document Name
Set up implementation dependent DOM environment for loading,
    parsing, and validating input documents
Open input directory
Get first file from input directory
Call X12TargetConverter writeHeaders method
DO for all files in input directory
  Input Document <- Load input file, validating it if requested
  Call X12TargetConverter processDocument method, passing
      the Input Document
  Increment number of documents processed
ENDDO
Call X12TargetConverter writeTrailers method
Close output file
Display completion message with number of documents processed

X12TargetConverter Class (Extends TargetConverter)

Overview

The X12TargetConverter is again very similar to the CSV and flat file target converters, but again we add some complexities due to the grammar differences and the requirement to create the X12 control segments.

Attributes:

  • None

Methods:

  • Constructor

  • processDocument

  • writeHeaders

  • writeTrailers

Methods
Constructor

The constructor method sets up the X12TargetConverter object and the X12 RecordWriter object.

Logic for the X12TargetConverter Constructor Method
Arguments:
  Output Stream X12 Interchange Output file
  String File Description Document Name

Call base class constructor, passing File
    Description Document Name
Create X12RecordWriter object, passing:
    File Description DOM Document and Output Stream
processDocument

As was the case with the X12SourceConverter's processFile method, we aren't able to simply call the base TargetConverter class's processGroup method to perform the bulk of the required processing. Because of the way we have defined the grammar of X12 transaction sets, the first segment after the transaction set header isn't always required and we can't treat the document as just another group. We must include some of the same types of operations that we have in the main loop of the processGroup method, then call that method when we actually do need to process a group. We can use the processGroup method from the Target Converter base class.

Logic for the X12TargetConverter processDocument Method
Arguments:
  DOM Document Input Document

Returns:
  Error status or throws exception

Root Element <- Get Document's documentElement attribute
Root Element Name <- Get Root Element's tagName attribute
IF Root Element Name != Grammar Root Element Name
  Return error
ENDIF
Call X12RecordWriter's writeST method to write Transaction Set
    Header segment
Segment Grammar Node <- Get firstChild from Grammar Element
    skipping over non-Element Nodes
Segment Grammar Element <- Segment Grammar Node
Grammar Tag <- Call Segment Grammar's getAttribute for
    "ElementName"
Segment Element <- Get first childNode from Root Element,
    skipping over non-Element Nodes
DO until all child Elements of the Root have been processed
  Segment Name <- call Segment Element's getNodeName
  DO until Grammar Name = Segment Name
     Segment Grammar Element <- Get next Segment Element from
        Grammar, skipping over non-Element Nodes
    IF Segment Grammar Element is NULL
      Return error  //  This record is not part of the document
    ENDIF
    Grammar Name <- Call Segment Grammar's getAttribute for
        "ElementName"
  ENDDO
  Grammar Element Name <- Call Segment Grammar Element's
      getNodeName
  IF Grammar Element Name = "GroupDescription"
    Call processGroup, passing Record Element and Record Grammar
        Element
  ELSE
    Call X12RecordWriter parseRecord, passing Record Element and
        Record Grammar Element
    Call X12RecordWriter writeRecord, passing Record
        Grammar Element
  ENDIF
  Record Element <- Get Record Element's nextSibling, skipping
      over non-Element Nodes
ENDDO
Call X12RecordWriter's writeSE method to write Transaction Set
    Trailer Segment
Return success
writeHeaders

This method calls the X12RecordWriter's methods to write the interchange and group header control segments. This and the writeTrailers methods are included here because the main program knows nothing of the X12RecordWriter class. Its methods to write header and trailer segments can be invoked only by a call to one of the X12TargetConverter's methods.

Logic for the X12TargetConverter writeHeaders Method
Arguments:
  None

Returns:
  Status or throws exception

Call X12RecordWriter writeISA
Call X12RecordWriter writeGS
Return success
writeTrailers

This method calls the X12RecordWriter's methods to write the interchange and group trailer control segments.

Logic for the X12TargetConverter writeTrailers Method
Arguments:
  Integer Number of Included Transaction Sets

Returns:
  Status or throws exception

Call X12RecordWriter writeGE
Call X12RecordWriter writeIEA
Return success

EDIRecordWriter Class (Extends RecordWriter)

Overview

Our analysis showed us that there were sufficient commonalities among the possible RecordReader classes for EDI to XML conversions (one derived class for each specific EDI syntax) to create a base EDIRecordReader class and put the common functionality into it. We have the same situation with the RecordWriter derived classes used to process EDI formats as the target of our conversions. So, in this class we develop some common functions for converting from XML to generic EDI segments, maintaining control information that will be used by the derived classes. Again, we build in preliminary support for the repetition separator and the release character, even though they aren't implemented in version 004010 of X12.

Attributes:

  • DOM Document EDI Control

  • Character Element Separator

  • Character Component Separator

  • Character Repetition Separator

  • Character Release Character

  • Integer Segment Count

Methods:

  • Constructor

  • parseRecord

  • writeRecord

  • writeEDIControl

The writeEDIControl method writes the updated EDI Control document to the EDIControl.xml file in BBHOME. It is fairly simple and very dependent on the DOM API implementation, so I won't discuss it further below.

Methods
Constructor

When we read EDI we parse the input to determine the delimiters. However, when we write it we must derive the delimiters from a data store. The primary purpose of this method is to get the delimiters from the file description document. In addition, the constructor method loads the EDIControl.xml file for sequence number generation.

Logic for the EDIRecordWriter Constructor Method
Arguments:
  DOM Document File Description Document
  Output Stream

Call RecordWriter base class constructor, passing File
    Description Document and Output Stream
Delimiters Element <- Call File Description Document's
    getElementsByTagName for "Delimiters"

Segment Terminator Element <- call Delimiters
    getElementsByTagName
        for "SegmentTerminator"
Delimiter Value = call Segment Terminator's getAttribute for
    "value"
Record Terminator1 <- Call setDelimiter, passing Delimiter Value

Element Separator Element <- call Delimiters
    getElementsByTagName
        for "SegmentTerminator"
Delimiter Value = call Element Separator Element's getAttribute
    for "value"
Element Separator <- Call setDelimiter, passing Delimiter Value

Component Separator Element <- call Delimiters
    getElementsByTagName
        for "ComponentSeparator"
Delimiter Value = call Component Separator Element's
    getAttribute
        for "value"
Component Separator <- Call setDelimiter, passing Delimiter Value

Repetition Separator Element <- call Delimiters getElementsByTagName
    for "RepetitionSeparator"
IF Repetition Separator Element is not null
  Delimiter Value = call Repetition Separator Element's getAttribute
      for "value"
  Repetition Separator <- Call setDelimiter, passing Delimiter Value
ENDIF

Release Character Element <- call Delimiters getElementsByTagName
    for "ReleaseCharacter"
IF Release Character Element is not null
  Delimiter Value = call Release Character Element's getAttribute
      for "value"
 Release Character <- Call setDelimiter, passing Delimiter Value
ENDIF

BBHomePath <- from system property or environment variable for
    BBHome
EDIControlPath <- BBHomePath + "EDIControl.xml"
Call implementation dependent methods to load EDIControl Document
parseRecord

This method is very similar to the parseRecord method of the RecordReader base class, although again we add some functionality that is solely needed for processing the EDI representations of composite data structures. Rather than confuse the base class method I have created a more specialized method for the derived class. For Babel Blaster 1.0 this might be moved to the base class if design review and testing verify that it will work fine for other legacy formats.

Logic for the EDIRecordWriter parseRecord Method
Arguments:
  DOM Element Record
  DOM Element Record Grammar

Returns:
  Void

Field Grammar <- Get first Grammar child Element from Record
    Grammar Element, skipping over non-Element nodes
Grammar Field Name <- call Field Grammar's getAttribute
    on "ElementName"
DO for all Field Elements that are children of Record Element
  // Test for a composite data structure
  Composite <- False
  childNode <- get Field Element's firstChild
  DO until childNode is null or Composite is true
    IF childNode nodeType = ELEMENT
      Composite <- true
    ELSE
      childNode <- childNode's nextSibling
    ENDIF
  ENDDO
  IF Composite is false
    Element Text <- call getElement Text on Field Element
    IF Element Text is empty
      Proceed to next field
    ENDIF
  ENDIF
  Field Name <- Field Element's tagName attribute
  DO UNTIL Field Name == Grammar Field Name or Grammar Element
        is null
    Get next sibling Field Grammar Element, skipping over
        non-Element Nodes
    Grammar Field Name <- Field Element's tagName attribute
  ENDDO
  IF Grammar Element is null
    Return error
  ENDIF
  IF Composite is true
    Call base class parseRecord, passing Field Element and
        Grammar Element
  ELSE
    Field Number <- Call Field Grammar's getAttribute
        on "FieldNumber"
    New Cell <- call createDataCell, passing Field Number
        and Grammar
    New Cell <- call New Cell's putField, passing Element Text
  ENDIF
ENDDO

Note that we don't need a special method to parse the XML representation of a composite data element. We can use the base class's parseRecord method because the structure of a composite data element's XML representation and grammar are the same as records in our other legacy formats.

writeRecord

Since we can identify a grammar that applies to the segments of most EDI syntaxes, we can develop this common method to write such segments. The overall logic flow is very similar to that of the CSVRecordWriter since both deal with delimited record formats. As with that method we walk the DataCell Array. For this method we must keep track of a few more pieces of information so that we write the correct delimiters. The Element Position counter keeps track of data element position within the output segment. The Component Position counter keeps track of component data element position within a composite data structure. The Previous Element Position indicates whether or not we have a repeating data element.

Logic for the EDIRecordWriter writeRecord Method
Arguments:
  DOM Element Segment Grammar

Returns:
  Error status or throws exception

Initialize Output Record Buffer to null
Segment Tag <- Call Segment Grammar's getAttribute for "TagValue"
Output Buffer <- Segment Tag
Set Buffer Length to Tag length
Element Position <- 0
Previous Element Position <- 0
Component Position <- 1
DO for Index from 0 through Highest Cell
  // Test for new element
  Cell Position = Get Cell's Field Number
  // Reset to start new composite if needed
  IF Element Position != Cell Position
    Component Position <- 1
  ENDIF
  // Write preceding data element separators
  DO while Element Position < Cell Position
    Append Element Separator to Record Buffer
    Increment Element Position
  ENDDO
  // Write preceding repetition separator
  // NOTE - An additional test is needed here to write repeating
  //     composite data structures correctly
  IF Element Position = Previous Element Position
    Append Repetition Separator to Record Buffer
  ENDIF
  Previous Element Position = Element Position
  // Write preceding component data element separators
  Cell SubPosition = Get Cell's SubField Number
  DO while Component Position < Cell SubPosition
    Append Component Separator to Record Buffer
    Increment Component Position
  ENDDO
  Call fromXML to convert Cell contents from XML data type
  Call prepareOutput to format Cell contents for EDI output
  Cell Contents <- Call getField to retrieve Cell Buffer
      contents
  Clear Array entry
  Append Cell Contents to Record Buffer
ENDDO
Highest Cell <- -1
Append base RecordHandler's Record Terminator1 to Record Buffer
Call language's write routines to do physical write of
    Record Buffer
Increment Segment Count
Return success

Again, this version of the method doesn't fully support composite data structures when used as repeating data elements. A Babel Blaster 1.0 requirement is to widen support beyond the demands of X12 version 004010.

X12RecordWriter Class (Extends EDIRecordWriter)

Overview

The X12RecordWriter is derived from the EDIRecordWriter. It handles the specialized requirements of writing X12 control segments.

Attributes:

  • DOM Element TP Element for trading partner information

  • String Interchange Control Number

  • String Group Control Number

  • Integer Transaction Set Count

  • String Date

  • String Time

Methods:

  • Constructor

  • writeGE

  • writeGS

  • writeIEA

  • writeISA

  • writeSE

  • writeST

Methods

As with the case of the X12RecordReader class, only a few of these methods are very significant. The constructor method doesn't do anything except call the base class constructor. The methods that write transaction set and group trailer segments, writeGE and writeSE, are simple methods that write the number of included components, followed by the control number that was written in the header segment. The methods that write the header segments and the interchange trailer are worthy of a bit more discussion. I'll discuss them below, starting with the outer control segments of the interchange, moving inward.

writeISA

Based on trading partner information in the file description document, this method looks up the last used interchange control number from the EDIControl.xml data store. The method increments the control number, updates the value in EDIControl, and calls language and system dependent functions to get the current date and time for the ISA and GS segments. The method then populates the rest of the ISA segment from other Elements in the file description document and writes the ISA segment.

Logic for the X12RecordWriter writeISA Method
Arguments:
  None

Returns:
  Status or throws exception

ISA Info <- Call File Description Document's
    GetElementsByTagName for "ISAInformation"
ISA ReceiverID <- Call ISAInfo's getElementsByTagName for
    "ReceiverID", then getAttribute for "value"
X12 Control <- Call EDI Control document's getElementsByTagName
    for "X12OutboundControl"
TP NodeList <- Call X12 Control's getElementsByTagName for
    "TradingPartner"
DO for all entries in TP NodeList or until
    Control Receiver ID = ISA Receiver ID
  Control Receiver ID <- call TP NodeList item's getAttribute
    for "ReceiverID"
ENDDO
IF no match found
  TP Element <- call EDI Control's createElement method using
      Element name of "TradingPartner"
  X12 Control <- call X12 Control's appendChild to attach
      TP Element
  Interchange Number <- 1
  Call TP Element's setAttribute for "ReceiverID", passing
      ISA Receiver ID as value to set
ELSE
  Interchange Number <- Call TP Element's getAttribute for
      "LastInterchangeNumber"
  Increment Interchange Number
ENDIF
Call TP Element's setAttribute for "LastInterchangeNumber",
    passing Interchange Number as value to set
Date <- Get date from system, and reformat to YYMMDD
Time <- Get time from system, and reformat to HHMM
Get remaining values from ISA Info Element and pad to required
    Lengths
Initialize Record Buffer
Build ISA Segment in Record Buffer with "ISA" segment tag, values
    from ISA Info Element, Date, Time, and Delimiters from base
    class members, appending Record Terminator1 as segment
    terminator
Write Record Buffer using language-specific methods
Return success

We could, perhaps, use a more sophisticated method for looking up the trading partner in EDIControl than the algorithm in this method. However, I don't see significant performance gains in using something like XQuery for the number of entries likely to exist in this document. Also, we would be complicating the code by adding yet another type of technology.

writeGS

This method retrieves the last group control number used from the EDIControl file and populates the remainder of the segment from values from the GSInformation Element in the file description document. It also initializes the transaction set count.

Logic for the X12RecordWriter writeGS Method
Arguments:
  None

Returns:
  Status or throws exception

Transaction Set Count <- 0
GSInfo <- Call File Description Document's getElementsByTagName
    for "GSInformation"
GSReceiverID <- Call GSInfo's getElementsByTagName for
    "ApplicationReceiversCode", then getAttribute for "value"
GSType <- Call GSInfo's getElementsByTagName for
    "FunctionalIDCode", then getAttribute for "value"
GroupNodeList <- Call TP Element's getElementsByTagName for
    "FunctionalGroup"
DO for all entries in GroupNodeList or until
    ControlReceiverID = GSReceiverID and
    ControlType = GSType
  ControlReceiverID <- call GroupNodeList item's getAttribute
      for "ReceiverID"
  ControlType <- call GroupNodeList item's getAttribute
      for "FunctionalIDCode"
ENDDO
IF no match found
  GroupElement <- call EDI Control's createElement method using
      Element name of "FunctionalGroup"
  TP Element <- call TP Element's appendChild to attach
      GroupElement
  Group Number <- 1
  Call GroupElement's setAttribute for "ReceiverID", passing
      GSReceiverID as value to set
  Call GroupElement's setAttribute for "FunctionalIDCode",
      passing GSType as value to set
ELSE
  Group Number <- Call Group Element's getAttribute for
      "LastGroupNumber"
  Increment Group Number
ENDIF
Call Group Element's setAttribute for "LastGroupNumber",
    passing Group Number as value to set
Get remaining values from GSInfo Element
Initialize Record Buffer
Build GS Segment in Record Buffer with "GS" segment tag, values
    from GSInfo Element, Date, Time, and Delimiters from base
    class members, appending RecordTerminator1 as segment
    terminator
Write Record Buffer using language-specific methods
Return success
writeST

This method is quite a bit simpler that the writeISA and writeGS methods since we start a new sequence of transaction set control numbers with each new functional group we write. We need to retrieve only one value from the file description document.

Logic for the X12RecordWriter writeST Method
Arguments:
  None

Returns:
  Status or throws exception

Increment Transaction Set Count
Pad Transaction Set Count to four digits with leading zeroes
   if required
STInfo <- Call File Description Document's getElementsByTagName
    for "STInformation"
STType <- Call STInfo's getAttribute for "TransactionSetIDCode"
Initialize Record Buffer
Build ST Segment in Record Buffer with "ST" segment tag, STType,
    and Transaction Set Count, using Element Separator from
    base class members and appending RecordTerminator1 as segment
    terminator
Write Record Buffer using language-specific methods
Segment Count <- 1
Return success
writeIEA

Writing the trailer segment is pretty simple since we write only two values. However, for control number integrity we don't save the EDIControl document until after the interchange trailer is successfully written.

Logic for the X12RecordWriter writeIEA Method
Arguments:
  None

Returns:
  Status or throws exception

Initialize Record Buffer
Build IEA Segment in Record Buffer with "IEA" segment tag,
    "1" included functional group, and Interchange Control
    Number, using Element Separator from base class members
    and appending Record Terminator1 as segment terminator
Write Record Buffer using language-specific methods
Save EDI Control Document to EDI Control File Path
Return success


New DataCell Methods and Derived Classes

We can use most of the DataCell classes and methods we developed in the previous chapters. We need only a few new ones in this chapter to satisfy some of the X12 data type formatting requirements. We'll handle the X12 data types as follows.

  • Numeric: We create a DataCellX12N class from the existing DataCellN class. The DataCellX12N class uses the base class's fromXML and toXML methods but has its own prepareOutput method since a minus sign, if present, is not included when considering the length, and we use only a zero for the fill character.

  • Decimal Number: We create a DataCellX12R class from the existing DataCellReal class. Like the DataCallX12N class, this new class uses the base class's toXML method. However, some different processing is required to handle exponential notation in the fromXML and prepareOutput methods. In addition, the length does not include the leading minus sign, the decimal point, or the E exponent indicator, and we use only a zero for the fill character.

  • Identifier and String: These are handled by the existing DataCellAN class. We modify the prepareOutput method a bit to use an ASCII space as the fill character for these two data types. We also do not truncate an Identifier type.

  • Date: We create a new DataCellX12DT class derived from the DataCellDateYYYYMMDD class. The new class's methods call the base class methods. In addition, the new methods append or remove the two most significant digits of the year if the maximum length is six digits.

  • Time: This data type is handled by a new DataCellX12TM class.

DataCellX12N Class (Extends DataCellN)

As noted above, this class is derived from the DataCellN class developed in Chapter 8. It uses base class methods for everything but the prepareOutput method, which handles the formatting requirements of the X12 Numeric data type.

Logic for the DataCellX12N prepareOutput Method
Arguments:
  None

Returns:
  Status or throws exception

NegativeNumber <- false
IF first character in Cell Buffer is "-"
  Cell Buffer <- Shift Cell Buffer left one character to remove
      Minus sign
  Negative Number <- true
  Decrement Buffer Length
ENDIF
Minimum Length <- Call Grammar Element's getAttribute for
    "MinLength", converting to integer
IF (Buffer Length < Minimum Length)
  Call justifyFieldRight, passing Minimum Length and zero
      as Fill Character
ENDIF
Maximum Length <- Call Grammar Element's getAttribute for
    "MaxLength", converting to integer
IF (Buffer Length > Maximum Length)
  Return failure
ENDIF
IF Negative Number = true
  Cell Buffer <- Shift Cell Buffer right one character to insert
      Minus sign
  Increment Buffer Length
ENDIF
Return success

DataCellX12R Class (Extends DataCellReal)

As noted above, this class is derived from the DataCellReal class developed in Chapter 8. It uses that class's toXML method but provides its own fromXML and prepareOutput methods.

Logic for the DataCellX12R fromXML Method
Arguments:
  None

Returns:
  Error status or throws exception

Trim Spaces
IF Cell Buffer = -0, INF, -INF, or NaN
  Return error for unsupported constant in X12 Decimal date type
ENDIF
Convert to uppercase since XML schema language data types allow
    a lower case 'e' to indicate the exponent
IF first Character in Cell Buffer = "+"
  Cell Buffer <- Shift Cell Buffer left one character to remove
      plus sign
  Decrement Buffer Length
ENDIF
Return success
Logic for the DataCellX12R prepareOutput Method
Arguments:
  None

Returns:
  Status or throws exception

NegativeNumber <- false
ExponentialNotation <- false
IF first character in Cell Buffer is "-"
  Cell Buffer <- Shift Cell Buffer left one character to remove
      Minus sign
  Negative Number <- true
  Decrement Buffer Length
ENDIF
Minimum Length <- Call Grammar Element's getAttribute for
    "MinLength", converting to integer
Maximum Length <- Call Grammar Element's getAttribute for
    "MaxLength", converting to integer
IF Cell Buffer contains "E"
  Increment Minimum Length
  Increment Maximum Length
  Exponential Notation <- true
ENDIF
IF Cell Buffer contains a decimal point
  Increment Minimum Length
  Increment Maximum Length
ENDIF
IF (Buffer Length < Minimum Length)
  Call justifyFieldRight, passing Minimum Length and zero
      as Fill Character
ENDIF
IF (Buffer Length > Maximum Length)
  Truncate <- Call Grammar Element's getAttribute for
      "Truncatable"
  IF Truncate = false OR Exponential Notation = true
    Return truncation error
  ENDIF
  Depending on language features, truncate Cell Buffer contents to
      specified Length stopping when Length is achieved or
      decimal point is truncated
  IF (Buffer Length > Maximum Length)
    Return failure
  ENDIF
  Buffer Length <- Maximum Length
ENDIF
IF Negative Number = true
  Cell Buffer <- Shift Cell Buffer right one character to insert
      Minus sign
  Increment Buffer Length
ENDIF
Return success

DataCellX12DT Class (Extends DataCellDateYYYYMMDD)

This class relies heavily on the base class methods. The only thing we really need to deal with is X12's support of a YYMMDD date format. We know we're using that format if the maximum length of the data element is six digits.

Logic for the DataCellX12DT fromXML Method
Arguments:
  None

Returns:
  Error status or throws exception

Call base class fromXML method
Maximum Length <- Call Grammar Element's getAttribute for
    "MaxLength"
IF Maximum Length = "6"
  Shift Cell Buffer two characters to the left
  Buffer Length <- 6
ENDIF
Return success
Logic for the DataCellX12DT toXML Method
Arguments:
  None

Returns:
  Error status or throws exception

Cell Buffer < trim leading and trailing whitespace from
    Cell Buffer
IF Buffer Length = 6
  Cell Buffer <- "20" + Cell Buffer
  Buffer Length <- 8
ENDIF
Return call to base class toXML method

In converting from YYMMDD to YYYYMMDD by adding a prefix of 20, we're making a somewhat arbitrary assumption that any six-digit dates are from the 21st century. I think this is fairly reasonable since, by now, if anyone cares about using 20th-century dates in an X12 transaction set, they most likely are using YYYYMMDD so as not to leave anything to chance.

DataCellX12TM Class

This class handles conversion to and from the X12 Time data type to the schema language time data type based on ISO 8601. The differences in representation are shown earlier in the chapter in Table 9.6.

Logic for the DataCellX12TM fromXML Method
Arguments:
  None

Returns:
  Error status or throws exception

IF Buffer Length < 8
  Return error
ENDIF
Hours <- Cell Buffer characters at offsets 0 and 1
Minutes <- Cell Buffer characters at offsets 3 and 4
Seconds <- Cell Buffer characters at offsets 6 and 7
Fractional Seconds <- Cell Buffer characters 9 and beyond,
    if present
Cell Buffer <- Hours + Minutes + Seconds + Fractional Seconds
Buffer Length <- String length of Cell Buffer
Return success
Logic for the DataCellX12TM toXML Method
Arguments:
  None

Returns:
  Error status or throws exception

Cell Buffer < trim leading and trailing whitespace from
    Cell Buffer
IF Buffer Length < 4
  Return error
ENDIF
Hours <- Cell Buffer characters at offsets 0 and 1
Minutes <- Cell Buffer characters at offsets 2 and 3
Seconds <- Cell Buffer characters at offsets 4 and 5, if present,
    otherwise "00"
Fractional Seconds <- Cell Buffer characters 6 and beyond
    if present
Cell Buffer <- Hours + ":" + Minutes + ":" + Seconds + "."
    + Fractional Seconds
Buffer Length <- String length of Cell Buffer
Return success

The prepareOutput method truncates the data if required. We will somewhat arbitrarily assume that if the X12 target data element is not large enough to handle seconds or fractions of a second, users will not mind if we discard them.

Logic for the DataCellX12TM prepareOutput Method
Arguments:
  None

Returns:
  Status or throws exception

Maximum Length <- Call Grammar Element's getAttribute for
    "MaxLength"
IF Buffer Length > Maximum Length
  Truncate Cell Buffer from left to Maximum Length
  Buffer Length <- Maximum Length
Return success


Java Implementation

The Java implementation of the design is composed of the source files listed below.

  • X12ToXML.java: the main routine for the Java application with an X12 interchange as source

  • X12SourceConverter.java (in package net.babelblaster.convert): the class that drives conversion from X12 to XML

  • EDIRecordReader.java (in package net.babelblaster.record): the base class that handles reading EDI segments and converting to XML

  • X12RecordReader.java (in package net.babelblaster.record): the class that handles reading X12 control segments

  • XMLToX12.java: the main routine for the Java application with an X12 interchange as target

  • X12TargetConverter.java (in package net.babelblaster.convert): the class that drives conversion from XML to an X12 interchange

  • EDIRecordWriter.java (in package net.babelblaster.record): the class that handles reading XML and building and writing EDI segments

  • X12RecordWriter.java (in package net.babelblaster.record): the class that handles building and writing X12 control segments

  • DataCellX12N.java (in package net.babelblaster.cell): the class that handles the X12 Number data type

  • DataCellX12R.java (in package net.babelblaster.cell): the class that handles the X12 Decimal data type

  • DataCellX12DT.java (in package net.babelblaster.cell): the class that handles the X12 Date data type

  • DataCellX12TM.java (in package net.babelblaster.cell): the class that handles the X12 Time data type

It is mildly interesting that both the XMLToX12 and X12ToXML converters directly read or manipulate three different XML documents in the same program. This is trivial since in the Document Object Model these are all three different objects. The APIs keep track for us of what belongs to whom.

We use only a few new features in this code that we haven't seen before. To get the Date and Time data elements for the ISA interchange and GS group header segments we use the java.util.Date and java.text.SimpleDateFormat classes. This excerpt from the writeISA method shows how we do this.

//  Date <- Get date from system, and reformat to YYMMDD
//  Time <- Get time from system, and reformat to HHMM
Date CurrentTime = new Date();
SimpleDateFormat dtFormatter =
    new SimpleDateFormat("yyyyMMddhhmm");

String sDateTime = dtFormatter.format(CurrentTime);
sDate6 = sDateTime.substring(2,8);
sDate8 = sDateTime.substring(0,8);
sTime = sDateTime.substring(8);

To parse the control segments in the X12RecordReader we use the string tokenizer in java.util.StringTokenizer. The code that precedes this excerpt creates a String representation of the data element separator. As a class member attribute it has a Java data type of byte. This code from the parseISA method loads the ISA array of String with the data elements from the ISA segment.

//  Using language-specific string token routines,
//  load class attributes ISA01 through ISA16
StringTokenizer st =
    new StringTokenizer(ISASegment, sElementSeparator);
for (i=0; i < 17; i++)
{
  ISA[i] = st.nextToken();
  if (!st.hasMoreTokens() && ( i != 16 ))
  {
    throw new BlasterException(
      TextConstants.sExceptionInsuffientElements +
      "Expected: 16  Found: " + i + " in ISA \n" +
      ISASegment);
  }
}


C++ Implementation

The C++ implementation is composed of the source files listed below.

  • X12ToXML.cpp: the main routine for the C++ application with an X12 interchange as source

  • X12SourceConverter.cpp: the class that drives conversion from X12 to XML

  • X12SourceConverter.h: the header file for the X12SourceConverter class

  • EDIRecordReader.cpp: the class that handles reading EDI segments and converting to XML

  • EDIRecordReader.h: the header file for the EDIRecordReader class

  • X12RecordReader.cpp: the class that handles reading X12 control segments

  • X12RecordReader.h: the header file for the X12RecordReader class

  • XMLToX12.cpp: the main routine for the C++ application with an X12 interchange as the target

  • X12TargetConverter.cpp: the class that drives conversion from XML to an X12 interchange

  • X12TargetConverter.h: the header file for the X12TargetConverter class

  • EDIRecordWriter.cpp: the class that handles reading XML and building and writing EDI segments

  • EDIRecordWriter.h: the header file for the EDIRecordWriter class

  • X12RecordWriter.cpp: the class that handles building and writing X12 control segments

  • X12RecordWriter.h: the header file for the X12RecordWriter class

  • DataCellX12N.cpp: the class that handles the X12 Number data type

  • DataCellX12N.h: the header file for the DataCellX12N class

  • DataCellX12R.cpp: the class that handles the X12 Decimal data type

  • DataCellX12R.h: the header file for the DataCellX12R class

  • DataCellX12DT.cpp: the class that handles the X12 Date data type

  • DataCellX12DT.h: the header file for the DataCellX12DT class

  • DataCellX12TM.cpp: the class that handles the X12 Time data type

  • DataCellX12TM.h: the header file for the DataCellX12TM class

The writeISA method uses the time and localtime functions from time.h to get the date and time for the interchange and group headers.

// Set time zone from TZ environment variable. If TZ is not set,
// the operating system is queried to obtain the default value
// for the variable.
//
tzset();

// Get System Time
time( &ltime );
today = localtime( &ltime );

sprintf(cYear,"%d",(today->tm_year + 1900));
sprintf(cMonth,"%02d",(today->tm_mon + 1));
sprintf(cDay,"%02d",(today->tm_mday ));
sprintf(cHour,"%02d",(today->tm_hour ));
sprintf(cMinute,"%02d",(today->tm_min ));

The X12RecordReader's methods that parse control segments use the familiar C strtok function.


Enhancements and Alternatives

I have tried to be fairly thorough in supporting the minimum syntax requirements for version 004010 of X12. As I've noted elsewhere in the chapter, we can certainly enhance these utilities by supporting other versions of X12 and other EDI standards such as UN/EDIFACT. In addition to these important enhancements, there are a few other major features we should consider for Babel Blaster requirements.

  • Auditing: For our various logging functions, log entries to a data store rather than displaying the output to the current system output.

  • Acknowledgment tracking: This is related to auditing support. We generate Functional Acknowledgments for trading partners but don't tie those we have received from them to data we sent.

  • Handling multiple interchanges: The X12ToXML utility depends on having only one valid interchange in the input file. However, it is common when connecting to value-added networks to receive one transmission that may contain several interchanges.

  • Binary data: The X12 and UN/EDIFACT standards provide ways to convey binary data (usually encoded in hexadecimal or a similar format) in a data element. The most common uses are for including images or foreign data content in an EDI document. For minimal support we need to be able to extract binary data from an EDI document that we received and insert it into one that we wish to send. We may or may not want to put binary data directly into the XML representation of our EDI documents. If our source document is in an EDI format, the decision of whether or not to remove the binary data depends on how most business applications would handle the data, that is, as part of the document or as a separate file. For EDI documents as the target, we could retrieve the binary data from a separate file when we create the EDI by using a file pointer rather than explicitly including the data in the source XML document. If binary data is included in the XML document, the schema language hexBinary and base64Binary data types offer us two choices for suitably encoded binary data.

We might also want to remove some of the minor restrictions noted for each of the utilities. For example, we might want the X12ToXML utility to correctly process an interchange that has two functional groups of the same type.


Reference

Kimberley, Paul. 1991. Electronic Data Interchange. New York: McGraw Hill.


Resources

DOM Level 2 Core Specification, November 13, 2000, by the World Wide Web Consortium. Available online at www.w3.org/DOM/DOMTR#dom2.

DOM Reference, Microsoft XML Core Services (MSXML) 4.0, by Microsoft Corporation. Online documentation included as part of the MSXML SDK. Available online at http://msdn.microsoft.com/xml.

Java API for XML Processing Specification 1.2, by Sun Microsystems. Online Javadoc documentation, included as part of the Java XML Pack Spring 2002. Available online at http://java.sun.com/xml.

X12.5 Interchange Control Structures, Release 004010, December 1997, by the Accredited Standards Committee X12 of the American National Standards Institute. Available for purchase from the Data Interchange Standards Association at www.x12.org.

X12.6 Application Control Structure, Release 004010, December 1997, by the Accredited Standards Committee X12 of the American National Standards Institute. Available for purchase from the Data Interchange Standards Association at www.x12.org.

Xerces2 API Javadoc, by Apache Software Foundation. Online Javadoc documentation, included as part of the Xerces2 SDK. Available online at http://xml.apache.org.


Chapter 10. Converting from One XML Format to Another with XSLT

Go to, let us go down, and there confound their language, that they may not understand one another's speech.

—Genesis 11:7

This chapter presents a basic introduction to XSLT and XPath. It is primarily intended to provide end users with a practical tutorial on the XSLT and XPath features most applicable to electronic commerce and EAI applications. It may also give application developers a realistic understanding of XSLT and the role it can play in application architectures.

In this chapter we change our focus from the relatively detailed, developer-oriented concerns of the previous four chapters to more user-oriented issues. While the chapter retains a technical focus, it is at a lighter, introductory level.


Why XSLT Is Important

Let me make my opinion perfectly clear. I've been working on standardizing XML-based languages for electronic commerce for five years. After enduring countless conference calls, endless committee meetings, a few major initiatives that have yet to yield any fruit, and a few death marches, I've come to the conclusion that there isn't going to be a single standard XML-based language for common business documents. (Note that I'm using the singular "language" in the broad sense, as opposed to the more precise but technically focused use of the "language" of a specific type of document.)

Why am I so pessimistic? There are several reasons, but let me mention just a few of the most important. People categorize and name things in myriad ways. Even when they may basically agree with each other, there are inherent difficulties in communicating complex, abstract concepts. Achieving consensus is often daunting, and difficulty increases proportionally to the size of the group that needs to achieve consensus. As if those factors weren't sufficient, there are already several different XML business languages in use. Once people have started using something it is hard for them to change. Given these factors, I think it unlikely that there will be a single XML-based business language for even the English-speaking world. Beyond that, consider that businesses in France, Germany, Russia, or Japan might want to receive their orders and invoices using XML documents with Elements and Attributes using names from their own spoken languages. Finally, even in the unlikely event that such a single standard language could be developed, it is quite another matter for every application developer to provide native support for it. I would truly be pleased if history proves that I'm overly pessimistic, but the passage from Genesis seems particularly appropriate. It is an interesting observation that the UN/EDIFACT EDI standard, with its relative lack of names in any particular language, has proven to be useful to people using a wide variety of spoken languages. There is a somewhat ironic parallel between the all-powerful W3C sending forth XML and the God of Genesis confusing people's speech.

While we must acknowledge this bleak situation, all is not lost. It may not have given us salvation, but the W3C has at least given us a palliative. It may even be good enough for a wide range of situations. This is interesting because, if we can take the W3C's statements at face value, it is not necessarily what it originally set out to do. The XSL Transformations Recommendation, Version 1.0, contains the following little caveat right at the beginning of the document.

XSLT is designed for use as part of XSL, which is a stylesheet language for XML. In addition to XSLT, XSL includes an XML vocabulary for specifying formatting. XSL specifies the styling of an XML document by using XSLT to describe how the document is transformed into another XML document that uses the formatting vocabulary.

XSLT is also designed to be used independently of XSL. However, XSLT is not intended as a completely general-purpose XML transformation language. Rather it is designed primarily for the kinds of transformations that are needed when XSLT is used as part of XSL. [W3C 1999]

This is a situation in which the familiar law of unintended consequences comes into play. It is often the case that when a clever tool or technology is developed, people put it to uses that the originators never intended or anticipated. While this statement says that the W3C didn't intend to develop a completely general purpose XML transformation language, I wonder if the Consortium quite anticipated or understood the babble of overlapping dialects that would emerge when it created XML. I'm also not convinced that the members quite understood some of the ways in which people will actually use XML documents. There is a legitimate need for a general purpose XML transformation language, and I don't know of a better candidate than XSLT.

I intend in this chapter not only to show how XSLT can help deal with the babble but also to prove that it is a very appropriate tool for many environments. XSLT processors can perform a wide variety of logical document conversion tasks. They can even do things that many of the mappers packaged with EDI management systems can't do. XSLT processors may not at present be the most efficient means of converting documents, but when combined with conversion utilities such as those developed in this book, they can be a key part of a low-cost, portable, and platform-independent electronic commerce or EAI solution. And with the emergence of XSLT compilers and hardware accelerators, performance limitations may likely become less of an issue in the future.


XSLT Overview

XSLT is another markup language based on XML. While I've witnessed quite a few doofus speakers and writers referring to XML itself as a programming language, when it comes to XSLT I think I might agree with them. (I'm sure a few computer science purists might think that even XSLT doesn't qualify as a programming language, but I'm not that dogmatic.) Like the DOM, XSLT deals with XML documents as trees. In very broad terms, XSLT specifies how an input document, referred to as a source tree, is transformed into an output document, referred to as a result tree, based on the contents of an XSLT document called a stylesheet. An XSLT processor is a software component that performs this transformation. The XSLT language uses XPath (XML Path Language), another language defined by a W3C recommendation. Among other things, XPath is used to identify specific parts of the source tree for processing.

XSLT and XPath are both very broad, capable languages. I can't cover all of their depth in just one chapter. However, I don't have to. XSLT and XPath can be used to perform XML transformations for a broad range of applications including not just application integration and electronic commerce but also publishing, word processing, and Web page design. XSLT and XPath were designed primarily to be used with XSL for formatting. People write whole books on each of these two technologies. But the fact of the matter is that most commonly performed business document transformations can be accomplished using a small subset of XSLT and XPath features. So, the basics of what you need to know can be covered in one chapter.

If you want to know more, which you probably will if you work with XSLT very much, there are several places you can go for help. Unlike the Schema Recommendation, I find the XSLT and XPath Recommendations to be well written and easy to understand (for the most part, once you get past the terminology). I highly recommend them. One of the reasons for their readability is that the authors put many of the conformance requirements in separate sections rather than interspersing them with other user-oriented content. On my Web site I also recommend several very good books that deal with more detailed features of XSLT and XPath and offer examples beyond what the Recommendations provide.

A Simple Example: Hello World

I think the best way to teach a programming language is to start with a very simple, contrived example. So, here is an XSLT version of Hello World.

Stylesheet (HelloWorld.xsl)
<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="1.0"
    xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
  <xsl:template match="/">
    <HelloWorld>Howdy!</HelloWorld>
  </xsl:template>
</xsl:stylesheet>

Here is our source document.

Source (Goodbye.xml)
<?xml version="1.0" encoding="UTF-8"?>
<Goodbye>
  you all
</Goodbye>

We can run the source document through an XSLT processor using our HelloWorld.xsl stylesheet and produce the following result.

Result (HelloWorld.xml)
<?xml version="1.0" encoding="UTF-8"?>
<HelloWorld>
  Howdy!
</HelloWorld>

In fact, we can process any XML document using this stylesheet and produce exactly this same result. Let's look at the major parts of this stylesheet and understand why.

The first thing to note is that the stylesheet is itself a well-formed XML document. The root xsl:stylesheet Element declares the XSLT namespace and uses xsl: as a namespace prefix. One of the restrictions on using XSLT is that all the XSLT Elements and Attributes in a stylesheet must be qualified with a namespace prefix that resolves to the XSLT namespace. You need to know that xsl:transform can be used as a synonym for xsl:stylesheet. That the Recommendation allows either indicates to me some serious philosophical differences in the work group that wrote it. So serious, in fact, that rather than achieving consensus on one name over the other they took the very unusual step of punting and allowing either. However, it takes only a cursory reading of the Recommendation to perceive a preference for stylesheet over transform, so that's what I use.

In our example there is only one Element in the body of the stylesheet document, the xsl:template Element. To aid in clarity, I am going to use in this chapter's text the convention of using the xsl: prefix on XSLT Element names. However, there are times when the generic "template" by itself rather than "xsl:template" reads better, so I will use it too.

The xsl:template Element defines what is called a template rule. It is very aptly named because the contents of the template Element function as a template (or pattern or cookie cutter if you prefer) for a portion of the result tree. The analogy of using a template in manufacturing to stamp out several copies of a part is not quite so evident in this example because we just produce one fragment of the result tree. In our example the content of the template Element is the entire body of the result tree document. This cookie cutter aspect will become more obvious as we look at other examples.

The rule aspect of the template Element relates to the conditions under which the template is invoked or applied. These are determined by various Attributes of xsl:template or by the contents of another xsl:template. In our example the rule is defined by the presence of the match Attribute. The value of the match Attribute is a pattern, based on a subset of XPath, that identifies a particular set of nodes in the source tree. Such a set is called a node-set. An XSLT processor basically functions by starting with the root of the source tree and doing a preorder traversal of the tree (top down, left to right). When it finds a node in a node-set that matches the pattern of a template's match Attribute, the contents of the xsl:template are written to the result tree. Any XSLT Elements in the body of the template Element are processed before writing the contents to the result tree. Since there are various means by which these child Elements may invoke other templates, this entire process can be recursive. In our example the value of the match Attribute is "/", a single forward slash. This is a special abbreviation for the source tree's root.

So, now that we know what the pieces are we can understand how this stylesheet works. The XSLT processor starts at the root of the source tree and sees if there is a template rule that matches it. Our xsl:template has a match Attribute value indicating the source tree root, so the processor examines the contents. There are no Elements from the XSLT namespace, so it writes the entire contents to the result tree. There are no other templates, so it terminates. Now, since all we are matching on is the source tree root, it should now be fairly obvious why we can use any XML document as a source and produce our Hello World document as the result.

Those are the bare-bones basics. Let's look at another example before we start digging into more details and concepts.

Another Simple Example: Changing Tag Names

Hello World has value only as an instructional tool. However, this next example comes very close to something you might actually use in a real-life situation. Suppose that your desktop bookkeeping system and your customer contact organizer are both able to import and export XML documents. Although it is unlikely, let us further suppose for this example that both systems use exactly the same information. However, they use different Element names to tag that information. Suppose that a prospect you've been courting quite awhile finally places an order, and you need to move their contact and billing information into the bookkeeping system. You can extract the information from your contact management system, but you need to change the Element names before you can import the document into your bookkeeping system. Here is a stylesheet that does it.

Stylesheet (ProspectToCustomer.xsl)
<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="1.0"
xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
  <xsl:template match="Prospect">
    <Customer>
      <FirstName>
        <xsl:value-of select="FName"/>
      </FirstName>
      <MiddleName>
        <xsl:value-of select="MI"/>
      </MiddleName>
      <LastName>
        <xsl:value-of select="LName"/>
      </LastName>
      <Organization>
        <xsl:value-of select="Company"/>
      </Organization>
      <StreetLine1>
        <xsl:value-of select="Street1"/>
      </StreetLine1>
      <StreetLine2>
        <xsl:value-of select="Street2"/>
      </StreetLine2>
      <CityOrMunicipalUnit>
        <xsl:value-of select="City"/>
      </CityOrMunicipalUnit>
      <StateOrProvince>
        <xsl:value-of select="State"/>
      </StateOrProvince>
      <PostalCode>
        <xsl:value-of select="Zip"/>
      </PostalCode>
    </Customer>
  </xsl:template>
</xsl:stylesheet>

Here are the source document and the result produced by applying the stylesheet.

Source (Prospect.xml)
<?xml version="1.0" encoding="UTF-8"?>
<Prospect>
  <FName>Wilma</FName>
  <MI>J.</MI>
  <LName>Peterson</LName>
  <Company>Consolidated Consolidators</Company>
  <Street1>Suite 35</Street1>
  <Street2>14 Friendly Lane</Street2>
  <City>Amarillo</City>
  <State>TX</State>
  <Zip>79101</Zip>
</Prospect>
Result (Customer.xml)
<?xml version="1.0" encoding="UTF-8"?>
<Customer>
  <FirstName>Wilma</FirstName>
  <MiddleName>J.</MiddleName>
  <LastName>Peterson</LastName>
  <Organization>Consolidated Consolidators</Organization>
  <StreetLine1>Suite 35</StreetLine1>
  <StreetLine2>14 Friendly Lane</StreetLine2>
  <CityOrMunicipalUnit>Amarillo</CityOrMunicipalUnit>
  <StateOrProvince>TX</StateOrProvince>
  <PostalCode>79101</PostalCode>
</Customer>

You'll notice one new XSLT Element and another new concept in this stylesheet. The new Element is the xsl:value-of Element. There are various ways to use it, but here we insert a Text Node that contains the string value of the Element named in the select Attribute. You'll also notice that we used the value of "Prospect" in the match Attribute of the xsl:template rather than the "/" value for the document root. The reason for this is that Prospect as the document Element is not the same thing as the document root. We often refer to the "document Element" as the "root Element," but neither of these is the same thing as the document root. The document root is the ultimate parent of everything in the document and has just one child Element that is the root or document Element. Aside from the document root Element, the direct children of the document root include the XML prolog and any processing instructions or comments that precede the document root Element. So, having set our so-called context node to the root Element Prospect, the select Attributes of the xsl:value-of Elements identify the specific children of that Element by name using an XPath expression. We'll talk more about XPath in a later section in this chapter.

These two stylesheets offer examples of one particular approach to using XSLT. There are others. In the next section I briefly review some of these other approaches and describe the one we'll be using in this chapter.


A General Approach to Using XSLT

As I said earlier in the beginning of this chapter, both XSLT and XPath are very rich languages. XSLT, in particular, is so rich that there are several different approaches and paradigms for using it. Not all of them are appropriate for the types of transformations we're interested in. However, I need to mention them so that when you run across them in further XSLT study, you'll know what they are.

  • Literal Result Elements: Both examples presented so far use this the style. It involves including the actual markup, text, and so on (that is, the "literal result"), in the body of the appropriate xsl:template so that it gets copied into the result tree. Text from an Element's Text Nodes and text from Attribute values is inserted in the result tree by xsl:value-of and, in some cases, xsl:apply-templates (which I'll discuss shortly). As we saw from the second example, an xsl:template's content can be a mix of literal results and XSLT Elements.

  • "Literal Result Element as Stylesheet": This is the terminology used by the XSLT Recommendation. The authors of another Addison-Wesley title [Skonnard and Gudgin 2002] refer to it as an "exemplar-based" transformation (they're some of the same folks who like to say "transform" instead of "stylesheet"). In this model an XML document that isn't an XSLT stylesheet has Elements from the XSLT namespace embedded within it. When an XSLT processor processes a source document and uses this type of document as a stylesheet, it turns the XSLT Elements into the appropriate text and replaces them with the text. This design is roughly equivalent to having xsl:stylesheet as the root Element and a single xsl:template that matches on the source document root. There are several restrictions to using this approach, the most severe of which is that the whole document is a template and you can declare no other templates. This imposes several restrictions on how we process complex business documents, and except in a very few cases it is most likely not an appropriate model for us.

  • Creating Elements and Attributes with the xsl:element and xsl:attribute Elements: There is an alternative to inserting literal text and markup like <HelloWorld>Howdy!</HelloWorld> in the content of an xsl:template. In the content of the xsl:template you can put xsl:element or xsl:attribute (with the appropriate Attributes and content) to create an Element or Attribute, respectively, in the result tree.

Of these three approaches I have a strong preference for the first, using literal result Elements. I do need to note that some caution must be taken in using this approach. Creating Elements and Attributes only with xsl:element and xsl:attribute should result in a well-formed XML document. However, if you get the literal markup wrong in a literal result Element, you can create an output XML document that is not well formed. However, the literal result approach, being less verbose and tedious, is much easier than the other options outlined above.

In addition to these general approaches, XSLT also supports two distinctly different programming paradigms.

  • Rule-based or logic (also called "declarative"): This paradigm is oriented around a set of rules that don't necessarily have to be in any particular order. Logical rules are given a symbolic representation, and programs are merely sets of these rules. Logic programming focuses on specifying the desired results and leaves it to the machine to figure out how to produce them. Prolog is perhaps the best known example of a logic programming language.

  • Imperative (also called "procedural"): This is perhaps the most commonly used and familiar programming paradigm. It focuses on algorithms and data structures. In contrast to logic programming, the imperative programming paradigm focuses primarily on telling the machine specifically how to produce a result. Most of our traditional programming languages are imperative, and even the object-oriented languages are largely imperative. Familiar examples include C, COBOL, and FORTRAN. Java and C++, even though object-oriented, are still based on the imperative model.

There is no escaping the fact that XSLT is predominately rule-based. For proof, we need look no further than the name "template rule" and the xsl:template content, which specifies a portion of the result tree rather than necessarily how to produce the result. However, XSLT does have some procedural features. It supports conditional processing with xsl:if and xsl:choose (a DOCASE). It also supports iterative processing with xsl:for-each. In addition, some of the ways in which templates can be invoked are more procedural than rule-based. We'll see examples of many of these later in the chapter.

My preference is to not stray very far from the basic nature of XSLT and to use the rule-based model wherever it will do the job. In some specific cases a procedural approach is appropriate, but I tend to stick with the general rule-based model. That is what I will present in this chapter.

XSLT uses XPath extensively, and we are now at the point where we need to discuss some XPath basics before we can get any further with XSLT.


XPath Basics

XPath is a language for identifying nodes in an XML document tree. XPath, unlike most of the other work that the W3C has done on XML, is not itself an XML-based language. However, its compact syntax makes it easy to use for things such as Attribute values within languages that are XML based, such as XSLT. XSLT primarily uses XPath in specifying values for the match and select Attributes of various Elements.

One of the fundamental concepts of XPath is the expression. An expression generally evaluates to a specific node or group of nodes (the node-set mentioned earlier) in a document tree. However, it can evaluate to other things as we'll see later in this chapter.

The form of XPath expression most commonly used in XSLT is known as a location path. A simplified syntax is shown in the following BNF productions, based on Section 2 of the XPath Recommendation.

Basic Syntax for a Location Path Expression
LocationPath ::= RelativePath | AbsolutePath
AbsolutePath ::= '/' RelativePath?
RelativePath ::= Step | RelativePath '/' Step

As you can see, the main difference between an absolute path and a relative path is that the absolute path starts at the document root with the '/' forward slash character. A relative path is always stated in relation to a context node. In XSLT, the current node usually evaluates to or is based on the context node.

There are full and abbreviated forms of path names. I'll mention the full names here and there, but for the most part I prefer the abbreviated names since they are usually more readable.

Location paths in their simplest form very much resemble and are analogous to UNIX directory paths. Take the following little XML document as an example:

Location Path Example 1
<?xml version="1.0" encoding="UTF-8"?>
<LocationPathExample>
  <Grandfather Name="Bob">
    <Father Name="Junior">
      <Son Name="Trey"/>
    </Father>
  </Grandfather>
</LocationPathExample>

There are several points to note about this example.

  • The /LocationPathExample expression points to the root document Element LocationPathExample. This is like having a directory named Location Path Example directly under the root of a UNIX file system.

  • The /LocationPathExample/Grandfather expression points to Bob.

  • The /LocationPathExample/Grandfather/Father expression points to Bob's son, Junior.

  • The /LocationPathExample/Grandfather/Father/Son expression points to Junior's son, Trey (Robert III in some circles, Bubba in others).

These are all akin to typing cd from a UNIX shell command line to set your present working directory to a specific directory. Relative paths work in a similar fashion.

If the context node is Grandfather, we use the relative path Father to point to Junior. This is like typing:

cd Father

instead of:

cd /LocationPathExample/Grandfather/Father

In a similar fashion, if the context node is Father, we can use a relative path of Son rather than /LocationPathExample/Grandfather/Father/Son.

The real key to understanding location paths is to understand the location step used in the third production. A location step has the following simplified form, based on the syntax from Section 2 of the XPath Recommendation.

Basic Syntax for a Location Step
Step ::= AxisSpecifier? NodeTest Predicate*
AxisSpecifier ::= AxisName '::' | '@'?
AxisName ::= A set of twelve names such as "child", "ancestor",
   "attribute"
NodeTest ::= NameTest | NodeType '(' ')'
NameTest ::= '*' | NCName ':' '*" | Qname
NodeType ::= 'comment' | 'text' | 'processing-instruction' |
    'node'
Predicate ::= '[' Expr ']'
Expr ::= a logical expression, a variable reference, a literal,
    a number, a function call, or another expression

Let's look at the three major components of a step and then see what happens when they are combined.

An axis specifier basically tells you which direction to go in the tree relative to where you are now, that is, the context node. It is similar to an X-Y-Z coordinate system in geometry but applied to a two-dimensional tree. Giving an XSLT processor an axis specifier in a location step is kind of like an air traffic controller telling a pilot on landing approach to make a 90-degree turn and descend at a 10-degree glide slope. An axis specifier can have a long name like descendent-or-self or an abbreviation like //. The axis specifier that we'll by far use the most is child. It is so commonly used that it is the default axis. The axis specifier can in fact be omitted, and that is why, unlike the EBNF in the XPath Recommendation, I show the axis specifier as optional in the production above. Even though child is the most frequently used axis, be confident that there are specifiers that let you move in a single step from the context node to any adjacent node in the tree. Once you get there, you can go to any of its adjacent nodes, and so on.

A node test lets us test for a type of node (not very common in our applications but certainly possible) or a node with a particular name (very common). The name can be a wildcard, a namespace followed by a wildcard, a name with a namespace prefix, or an unqualified local name. In our applications we'll mostly use just a simple, unqualified Element or Attribute name. You can certainly include a namespace prefix, but if people design their documents in a user-friendly way you won't need namespace prefixes on Elements or Attributes in instance documents. So, you won't need them in location path expressions that find things in those instance documents. According to the BNF, the node test is really the only part of a location step that is required. However, later in this chapter we will see some special cases where it isn't used.

Expressions, used as the predicate of a location step, are perhaps the most complex part of XPath. They include such things as positional references along the current axis, various types of Boolean tests, and a built-in function library. The good news is that we don't have to use predicates very much in our types of applications. I will introduce you to a few of the most useful. The clever and creative will no doubt find uses for several others.

Working together in a location step, the axis sets a direction and the node test tells the processor what to look for in that direction. If the processor might find more than one node and instead return a node-set, we can use a predicate to filter nodes out of the node-set and narrow it down to just the node or nodes we're interested in. When we string location steps together in a relative or absolute path the results are cumulative. A single step evaluates to one node-set; it is used as a set of context nodes for the next step that yields another set of nodes, and so on.

Let's look at a few examples to help solidify these concepts. We'll add another Father, Billy, to the location path example we just discussed.

Location Path Example 2
<?xml version="1.0" encoding="UTF-8"?>
<LocationPathExample>
  <Grandfather Name="Bob">
    <Father Name="Junior">
      <Son Name="Trey"/>
    </Father>
    <Father Name="Billy">
      <Son Name="Willy"/>
      <Son Name="Billy Joe"/>
      <Son Name="Billy Bob"/>
    </Father>
  </Grandfather>
</LocationPathExample>

The simplest type of location step has only the implied child axis specifier and a node test. The absolute location path expression /LocationPathExample has one step following the initial slash (/). This path specifies the child Element named LocationPathExample of the document root.

For a bit more complex example, let's use an axis specifier other than child. We'll assume that our current context node is the Son Element with a Name Attribute of Willy. Using the axis specifier of parent with the abbreviated form "..", we create a two-step location path of "../Son". The first step of ".." has only the axis specifier, taking us to the parent Element Father with the name Attribute of Billy. The next step specifies the Element children named Son along the implied child axis. This yields a node-set of the brothers Willy, Billy Joe, and Billy Bob.

To show how a predicate is used to filter nodes, let's assume that we didn't necessarily know which of the brothers was our current context node but that we wanted to point to Billy Bob. We add a node test to specify a Name Attribute with a value of "Billy Bob". So, adding this to second location step of the previous example, we have a relative location path of "../Son[@Name='Billy Bob']".

So, those are the basics of XPath. There are, of course, exceptions to the generalities presented here. Explaining them is one of the reasons those other books are so long. The XPath Recommendation is also very specific about explaining exceptions, so use it when you can't quite figure out why something isn't working as you thought it would.

Now that we understand the basics of XPath, let's look at how XSLT uses it. XSLT allows use of the full XPath expression syntax in the select Attribute of several XSLT Elements. A restricted form of expression known as a pattern is used in the match Attribute of xsl:template. A pattern is the same as a location path expression except that in the location steps the only axes that are allowed are child, attribute, and the // abbreviation for descendent-or-self. Two or more of these restricted location paths may be logically ORed together with the pipe symbol (|) to form a more complex pattern. In effect, the set of patterns that might be used in a particular document is a subset of the set of location path expressions that might be used.

You can't use XPath in a match or select Attribute without somehow applying a template. The various ways to organize and call templates can have an impact on how we construct our XPath expressions. We'll look at some of these next.


Structuring Stylesheets

We've looked at a couple of examples and some basic concepts. I'll now wrap up this portion of the chapter with a few more general issues before looking at some specific techniques you will find useful. One of these general issues relates to the way we structure our stylesheets.

A stylesheet can have more than one xsl:template. In fact, this is probably the norm. In the rule-based programming paradigm, templates can be applied from other templates in a fashion similar to expanding the nonterminal symbols in BNF. In a procedural paradigm, we can break out certain types of processing into a template and call it in much the same way we might call a function in C. We can even pass parameters to an xsl:template, similar to passing arguments to a C function, and get results returned.

Let's look at two simple examples that demonstrate both techniques. We return to our example of converting a prospect to a customer, but we modify it so that we have more than one prospect or customer in a document. The source document from the contact management system appears below.

Source (Prospects.xml)
<?xml version="1.0" encoding="UTF-8"?>
<Prospects>
  <Prospect>
    <FName>Wilma</FName>
    <MI>J.</MI>
    <LName>Peterson</LName>
    <Company>Consolidated Consolidators</Company>
    <Street1>Suite 35</Street1>
    <Street2>14 Friendly Lane</Street2>
    <City>Amarillo</City>
    <State>TX</State>
    <Zip>79101</Zip>
  </Prospect>
  <Prospect>
    <FName>Fred</FName>
    <MI>Q.</MI>
    <LName>Williamson</LName>
    <Company>Independent Grocers of Oklahoma</Company>
    <Street1>PO Box 98</Street1>
    <City>Bartlesville</City>
    <State>OK</State>
    <Zip>74003</Zip>
  </Prospect>
  <Prospect>
    <FName>John</FName>
    <MI>C.</MI>
    <LName>Hernandez</LName>
    <Company>Super Service Grocers</Company>
    <Street1>Suite 900</Street1>
    <Street2>35 Lakeshore Drive</Street2>
    <City>Chicago</City>
    <State>IL</State>
    <Zip>60601</Zip>
  </Prospect>
</Prospects>

We want the result document to look as follows for our order management system.

Result (Customers.xml)
<?xml version="1.0" encoding="UTF-8"?>
<Customers>
  <Customer>
    <FirstName>Wilma</FirstName>
    <MiddleName>J.</MiddleName>
    <LastName>Peterson</LastName>
    <Organization>Consolidated Consolidators</Organization>
    <StreetLine1>Suite 35</StreetLine1>
    <StreetLine2>14 Friendly Lane</StreetLine2>
    <CityOrMunicipalUnit>Amarillo</CityOrMunicipalUnit>
    <StateOrProvince>TX</StateOrProvince>
    <PostalCode>79101</PostalCode>
  </Customer>
  <Customer>
    <FirstName>Fred</FirstName>
    <MiddleName>Q.</MiddleName>
    <LastName>Williamson</LastName>
    <Organization>Independent Grocers of Oklahoma</Organization>
    <StreetLine1>PO Box 98</StreetLine1>
    <StreetLine2/>
    <CityOrMunicipalUnit>Bartlesville</CityOrMunicipalUnit>
    <StateOrProvince>OK</StateOrProvince>
    <PostalCode>74003</PostalCode>
  </Customer>
  <Customer>
    <FirstName>John</FirstName>
    <MiddleName>C.</MiddleName>
    <LastName>Hernandez</LastName>
    <Organization>Super Service Grocers</Organization>
    <StreetLine1>Suite 900</StreetLine1>
    <StreetLine2>35 Lakeshore Drive</StreetLine2>
    <CityOrMunicipalUnit>Chicago</CityOrMunicipalUnit>
    <StateOrProvince>IL</StateOrProvince>
    <PostalCode>60601</PostalCode>
  </Customer>
</Customers>

We can construct a stylesheet that uses two templates: one for the root Element that sets up the way the overall document should look, and one for each customer. This example uses xsl:apply-templates.

Stylesheet (ProspectsToCustomersApply.xsl)
<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="1.0"
    xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
  <xsl:template match="/Prospects">
      <Customers>
        <xsl:apply-templates select="Prospect"/>
      </Customers>
  </xsl:template>
  <xsl:template match="Prospect">
    <Customer>
      <FirstName>
        <xsl:value-of select="FName"/>
      </FirstName>
      <MiddleName>
        <xsl:value-of select="MI"/>
      </MiddleName>
      <LastName>
        <xsl:value-of select="LName"/>
      </LastName>
      <Organization>
        <xsl:value-of select="Company"/>
      </Organization>
      <StreetLine1>
        <xsl:value-of select="Street1"/>
      </StreetLine1>
      <StreetLine2>
        <xsl:value-of select="Street2"/>
      </StreetLine2>
      <CityOrMunicipalUnit>
        <xsl:value-of select="City"/>
      </CityOrMunicipalUnit>
      <StateOrProvince>
        <xsl:value-of select="State"/>
      </StateOrProvince>
      <PostalCode>
        <xsl:value-of select="Zip"/>
      </PostalCode>
    </Customer>
  </xsl:template>
</xsl:stylesheet>

The xsl:apply-templates Element in the root Element's xsl:template tells the processor to apply the templates it finds for the children of the current node. We use the select Attribute in this example to specifically process only the Prospect child Elements of the root Prospects Element. However, since there aren't any other children, we could have left off the select Attribute. When the processor starts executing the xsl:apply-templates it sees if there are any other templates in the stylesheet whose patterns match that of the current node, that is, the current Prospect child of the Prospects root Element. The processor finds a match on the second template and applies that template for each of the children.

We can use this type of structure to break up a complex result document into a shell for the main body and fragments for each child Element. This strategy can greatly aid readability and maintainability, just like breaking up a big program into a main program and a number of smaller subroutines.

Note that this example follows the rule-based paradigm. We focus very much on how we want the result to look and don't tell the processor how to produce the result. The xsl:call-template example below looks very similar and produces the same result, but it follows a different paradigm.

Stylesheet (ProspectsToCustomersCall.xsl)
<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="1.0"
    xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
  <xsl:template match="/Prospects">
      <Customers>
        <xsl:for-each select="Prospect">
          <xsl:call-template name="OutputCustomer"/>
        </xsl:for-each>
      </Customers>
  </xsl:template>
  <xsl:template name="OutputCustomer">
    <Customer>
      <FirstName>
        <xsl:value-of select="FName"/>
      </FirstName>
      <MiddleName>
        <xsl:value-of select="MI"/>
      </MiddleName>
      <LastName>
        <xsl:value-of select="LName"/>
      </LastName>
      <Organization>
        <xsl:value-of select="Company"/>
      </Organization>
      <StreetLine1>
        <xsl:value-of select="Street1"/>
      </StreetLine1>
      <StreetLine2>
        <xsl:value-of select="Street2"/>
      </StreetLine2>
      <CityOrMunicipalUnit>
        <xsl:value-of select="City"/>
      </CityOrMunicipalUnit>
      <StateOrProvince>
        <xsl:value-of select="State"/>
      </StateOrProvince>
      <PostalCode>
        <xsl:value-of select="Zip"/>
      </PostalCode>
    </Customer>
  </xsl:template>
</xsl:stylesheet>

There are a few key differences between this and the previous stylesheet. In the content of the first xsl:template we have an xsl:for-each Element that iteratively selects the Prospect child Elements of the root Element. For each of these it performs an xsl:call-template of the xsl:template named OutputCustomer. The second xsl:template is identical to the one in the first stylesheet except that the name Attribute replaces the match Attribute. In this example we are explicitly giving the processor instructions about the algorithm it should use to produce the result tree.

While I generally tend to favor the xsl:apply-templates rule-based paradigm over the xsl:for-each and xsl:call-template procedural paradigm, there are times when an xsl:call-template can make a lot of sense in structuring a complex result tree. Take the case where we have a very complicated document like a health care claim. Such documents are usually broken down into major sections. Suppose that the structure of our source document doesn't match very well the structure of the result document. We could lay out the overall look of the result document and build the result tree using the following template.

<xsl:template match="/Claim">
  <HealthCareClaim>
    <xsl:call-template name="OutputProvider"/>
    <xsl:call-template name="OutputInsured"/>
    <xsl:call-template name="OutputPatient"/>
    <xsl:call-template name="OutputServicesProvided"/>
  </HealthCareClaim>
</xsl:template>

Even with the rule-based approach using xsl:apply-templates, there are times when you need to be cautious. One of these is when you don't have a template that exactly matches a node in the source tree. In cases like this the processor uses default templates to output the string values of the relevant Elements and its children if any are present. Using xsl:apply-templates on an Element with no child Elements or Attributes usually produces the same results as an xsl:value-of. However, if the Element has children you probably won't get the results you want. Business applications generally cough when data from a bunch of different fields are glommed together rather than being segregated into nice, discrete units.

Another case where I advise caution is when two or more templates match an expression for a source node. There are well-defined rules for deciding which of the templates should be used, but the best thing to do is to avoid coding stylesheets in ways that allow this to happen. Care also needs to be taken when the same Element or Attribute names are used in different parts of a source document. If your templates match only on the node name, templates may get applied to source elements that you didn't want processed. This would produce output in the result document in places where you didn't expect it. So, be careful about how you construct the expressions used for match and select Attributes, and be sure to test them with the widest variety of inputs allowed for the source document.

In addition to breaking up a stylesheet into several xsl:templates, we can also break a logical stylesheet into several different files. The xsl:include and xsl:import Elements bring xsl:templates from other stylesheets into the current one. Similar to the way in which schemas are included in other schemas, included xsl:templates are indistinguishable from those defined in the including stylesheet. There are some subtle differences between imported templates and native templates, particularly in the area of deciding which of two or more templates should be applied when there's a match. I've not seen very many applications for xsl:import yet, so I don't consider those subtleties as appropriate topics for a tutorial. However, be aware that they exist, and be sure to research them before you use xsl:import.

Another point needs to be made here. In these examples I often refer to the first, second, or third xsl:template. By convention I put the xsl:template that matches the root or the root Element first. However, except for the rare (and we hope avoidable) cases where more than one template matches a source tree node, the order of xsl:templates doesn't matter. The xsl:template that matches the root Element could be anywhere in the stylesheet and the result tree would look the same.

We have two more bits of housekeeping to cover before looking at some techniques.


A Bit of Housekeeping

The xsl:output Element

XLST is oriented toward transforming a source tree into a result tree. What, exactly, a processor does with the result tree is not within the scope of the XSLT Recommendation. However, XSLT does give the stylesheet designer a way to request that an XSLT processor creates output with certain characteristics. Such requests are made by using the xsl:output Element. It is optional, and although the Recommendation says that processors should use the information in this Element to produce the output, the Recommendation doesn't require that they do. However, the most commonly used processors do pay attention to the xsl:output Element.

The xsl:output Element has several optional Attributes, but we'll use only a few. The method Attribute allows users to request XML, HTML, text, or their own defined type of output. We'll use "xml" as the value. We'll also use the encoding Attribute to specify an encoding of UTF-8. Some processors produce UTF-16 as the default, but for ASCII-oriented systems (and people) we would generally prefer UTF-8. For readability in pasting example result documents into this book, I also used the indent Attribute with a value of "yes". You might not want to use it in production applications since it generates unneeded whitespace.

Running Transformations from a Command Line

You've probably asked by now, "OK, this is all well and good, but how do I actually run one of these transformations?" IDEs like XMLSPY or TurboXML provide menu options for performing XSLT transformations. However, some IDEs may require you to include in your source document a stylesheet processing instruction, such as the following.

<?xml-stylesheet type="text/xsl" href="ProspectToCustomer.xsl"?>

While this may be fine for an IDE, you wouldn't want to rely on this approach in most production situations. No matter how good a supplier you are, Ford and GM are unlikely to put your own stylesheet reference in the purchase orders they send you.

Unless you build XSLT transformations into your business application (which I'll discuss briefly in Chapter 12), you'll probably invoke them from a command line utility in a script or via some other utility. There are two such command line XSLT processors available for use with the APIs used in this book. The Java version is Xalan. We use the Process application defined in the xslt class as shown below.

java org.apache.xalan.xslt.Process -IN Prospects.xml

-XSL ProspectsToCustomersApply.xsl -OUT Output.xml

The jar files for JAXP, Xerces, and Xalan need to be installed in the directory appropriate to your Java runtime environment. The command line version comes with the API libraries, so there's no special installation. There is also a C++ version of Xalan.

In the MSXML world, Microsoft offers a free download of a command line utility called msxsl. It is run like any other program from a command line prompt. Here's how we would run it for the same scenario.

msxsl Prospects.xml ProspectsToCustomersApply.xsl -o Output.xml

The msxsl utility uses the MSXML API library, so you'll have to have it installed. It seems to me that Microsoft has a tendency to move things around on its Web site and make links to internal pages somewhat difficult. So, to obtain msxsl I recommend that you go to the main Microsoft site and do a search for msxsl.

There are, of course, other such utilities (for example, xsltproc). Xalan and msxsl are the two I used for this book, and they'll do for this tutorial.

Okay, we've laid down some basic concepts, discussed some basic issues, and done some housekeeping. Let's look now at a few of the common types of transformations you'll need to perform. The first set, presented in the next section, has to do with manipulating content of only a few Elements and/or Attributes. The section following that deals with the types of problems frequently encountered when the structures of the source and result trees are different or when we need to move things to different places in the trees.


Manipulating Content

Adding and Removing Content

Some books talk about adding and removing content as if it were a significant topic. I think this orientation comes from dealing with result documents that look very similar to source documents, except for adding or subtracting a few Elements or Attributes. In our world this is almost never the case. Even if we are fortunate enough to have similar structures for the source and results trees, they will rarely if ever use the same Element and Attribute names. So, the concept of adding and removing content is somewhat foreign.

However, we do need to deal with these concepts at a very gross level. To add content to a result tree that wasn't in the source tree, we simply insert the appropriate XSLT Elements or literal result markup in the content of the appropriate xsl:template. This is exactly what we did with the first example. There are various ways to ensure that content in the source tree doesn't appear in the result tree. In the most common case, the only things that show up in the result tree are the Elements and Attributes for which we specifically ask, usually by using xsl:value-of. If we don't ask for something, by default it doesn't show up.

Splitting Data Content

A somewhat common problem is splitting data from a single Element or Attribute in the source tree into two or more locations in the result tree. Let's again go back to our scenario of converting prospects to customers, but this time we'll assume that the contact management system has only a single field for the full name. (This is again somewhat of a contrived example, but I can't think of a better one right now.)

We want to turn this:

Source (CombinedContent.xml)
<?xml version="1.0" encoding="UTF-8"?>
<CombinedContent>
  <FullName>Fred Public</FullName>
</CombinedContent>

into this:

Result (SplitContent.xml)
<?xml version="1.0" encoding="UTF-8"?>
<SplitContent>
  <FirstName>Fred</FirstName>
  <LastName>Public</LastName>
</SplitContent>

Here's a template that does it.

Stylesheet (SplitContent.xsl)
<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="1.0"
    xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
  <xsl:output method="xml" version="1.0" encoding="UTF-8"
       indent="yes"/>
  <xsl:template match="/CombinedContent">
    <SplitContent>
      <FirstName>
        <xsl:value-of select="substring-before(FullName,' ')"/>
      </FirstName>
      <LastName>
        <xsl:value-of select="substring-after(FullName,' ')"/>
      </LastName>
    </SplitContent>
  </xsl:template>
</xsl:stylesheet>

The two xsl:value-of Elements in this example use a function for the expression value of the select Attribute. The substring-before and substring-after functions together work very much the same way that the string token functions do in Java or C/C++. We call substring-before to initialize, then succeeding calls to substring-after return succeeding tokens.

NOTE Real Stylesheets Are More Complicated!

Most of the transformations discussed in this and other sections would normally be coded as templates in a larger, more complex stylesheet. However, I handle the source fragments as complete documents and the stylesheets as complete, stand-alone stylesheets so that you may repeat and experiment with the transformations. You can find all the source documents and stylesheets on the book's Web site.

Combining Data Content

Okay, let's go back the other way. Suppose we need to combine the contents of two different Elements into one. Here are two examples.

First and Last Names to Full Name

This stylesheet does the exact reverse operation of the one in the previous example, combining first and last names into a full name string.

Stylesheet (CombineContent.xsl)
<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="1.0"
    xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
  <xsl:output method="xml" version="1.0" encoding="UTF-8"
      indent="yes"/>
  <xsl:template match="/SplitContent">
    <CombinedContent>
      <FullName>
        <xsl:value-of select="FirstName"/>
        <xsl:text> </xsl:text>
        <xsl:value-of select="LastName"/>
      </FullName>
    </CombinedContent>
  </xsl:template>
</xsl:stylesheet>

The use of the FullName start and end tags and the xsl:value-of Element should be fairly intuitive by now. The new thing added in this stylesheet is the xsl:text Element. Under normal circumstances whitespace is stripped from the content of XSLT Elements. There are ways to override this behavior. However, to specifically insert one or more space characters into an Element or Attribute content in the result tree we use xsl:text. Anything within xsl:text, including whitespace, is treated as literal text and written to the result tree.

Converting Time and Zone to ISO 8601

Here's a more involved and more real-world example that uses the same techniques shown above. Many legacy applications maintain the local time of an event. If they maintain the time zone at all, it is probably in a separate field. The ISO 8601 time formats represent local time with an offset from coordinated universal time. For example, 3:15 PM in the U.S. Eastern Standard time zone would be 15:15:00-05:00. For our example, suppose we're processing shipping or customs documents that indicate the time of arrival of an ocean vessel. Our application stores the time of this event using a 24-hour clock and a three-character code for the local time zone. Our ship arrived in New York Harbor at 3:15 in the afternoon.

We want to convert this:

Source (TimeAndZoneToISO8601.xml)
<?xml version="1.0" encoding="UTF-8"?>
<ConvertTime>
  <ArrivalTime>
    <HourOfDay>15:15</HourOfDay>
    <TimeZone>EST</TimeZone>
  </ArrivalTime>
</ConvertTime>

to this:

Result (TimeAndZoneInISO8601.xml)
<?xml version="1.0" encoding="UTF-8"?>
<ConvertedTime>
  <ArrivalTime>15:15:00-05:00</ArrivalTime>
</ConvertedTime>

Here's the stylesheet.

Stylesheet (TimeAndZoneToISO8601.xsl)
<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="1.0"
    xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
  <xsl:output method="xml" version="1.0" encoding="UTF-8"
      indent="yes"/>
  <xsl:template match="/ConvertTime/ArrivalTime">
  <ConvertedTime>
    <ArrivalTime>
      <xsl:value-of select="HourOfDay"/>
      <xsl:text>:00</xsl:text>
      <xsl:if test="TimeZone = 'EST'">-05:00</xsl:if>
      <xsl:if test="TimeZone = 'PST'">-08:00</xsl:if>
    </ArrivalTime>
  </ConvertedTime>
  </xsl:template>
</xsl:stylesheet>

We use xsl:text to output the seconds string ":00" to the result tree instead of just using the literal value, though we really don't have to. I've used it here since it seems a bit more readable to me than just inserting the literal value. The feature added in this stylesheet is the xsl:if Element. We perform conditional processing to add an eight-hour offset for West Coast ports using Pacific Standard Time and a five-hour offset for our East Coast ports. The test Attribute on xsl:if is an XPath expression that evaluates to a boolean data type. We test the string content of the TimeZone Element against a literal value. String values such as 'EST' in our example must be enclosed in single quotes since the whole Attribute value is in double quotes.

This is an example of using the procedural programming paradigm in a stylesheet that otherwise follows a rule-based paradigm. Note that the conditional support is fairly meager; there is no if/then/else structure. More complicated conditional processing such as the DOCASE are handled using the xsl:choose, xsl:when, and xsl:otherwise Elements.

Changing an Attribute to an Element

Due to the different ways in which people think about data, you are very likely to see the same item of data treated as an Attribute in one XML document and as an Element in someone else's idea of the same business document. In those types of situations you'll need to convert data between Elements and Attributes.

Here is a fairly common example. Most purchase orders list both a quantity and a unit of measure for a product item. Suppose that our order management system has the capability to import a purchase order in XML. Suppose that the designers decided to depict unit of measure and quantity as separate Elements in an overall line item Element. Suppose further that someone sends us a purchase order coded in compliance with some standard, wherein the standards developers considered that unit of measure was an attribute of quantity ordered rather than a stand-alone bit of data. We will need to convert this:

Source (AttributeToElement.xml)
<?xml version="1.0" encoding="UTF-8"?>
<LineItem>
  <QuantityOrdered UOM="CA">100</QuantityOrdered>
</LineItem>

into this:

Result (ElementToAttribute.xml)
<?xml version="1.0" encoding="UTF-8"?>
<LineItem>
  <UOM>CA</UOM>
  <QtyOrdered>100</QtyOrdered>
</LineItem>

The conversion is very easy, as shown below.

Stylesheet (AttributeToElement.xsl)
<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="1.0"
    xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
  <xsl:output method="xml" version="1.0" encoding="UTF-8"
      indent="yes"/>
  <xsl:template match="/LineItem">
    <LineItem>
      <UOM>
        <xsl:value-of select="QuantityOrdered/@UOM"/>
      </UOM>
      <QtyOrdered>
        <xsl:value-of select="QuantityOrdered"/>
      </QtyOrdered>
    </LineItem>
  </xsl:template>
</xsl:stylesheet>

The at sign (@) preceding UOM in the location step tells the processor that we want the UOM Attribute of the QuantityOrdered Element.

Changing an Element to an Attribute

Now it's time to generate an invoice, and the billing module of our order management system follows the same conventions as the order entry module. Our customer uses the invoice created by the same standards body that developed the schema used for the purchase order. So, we need to change unit of measure from an Element to an Attribute. We use the same source and result documents as in the previous example, but we reverse their roles. Again, the transformation is pretty simple.

Stylesheet (ElementToAttribute.xsl)
<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="1.0"
    xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
  <xsl:output method="xml" version="1.0" encoding="UTF-8"
    indent="yes"/>
  <xsl:template match="LineItem">
    <LineItem>
     <QuantityOrdered UOM="{UOM}">
       <xsl:value-of select="QtyOrdered"/>
     </QuantityOrdered>
   </LineItem>
  </xsl:template>
</xsl:stylesheet>

Notice the curly braces ({ } ) on UOM within the quotation marks that delimit the UOM Attribute value. This special syntax tells the processor to evaluate the expression within the braces and replace the braces and expression with the resulting string. This is a very easy way to insert source tree or other values into an Attribute value in a result tree; in fact, it is the only easy way to do it. I must confess that in my initial forays into XSLT I did not discover this particular nifty little feature. I inserted values into Attributes using a convoluted mix of predefined entities and xsl:text with disabled output escaping. This is not a trick you want to try at home.

These few techniques should give you what you need for the most common types of transformations of one or a few data values. We now move to problems in dealing with larger groups of data.


Solving Typical Navigation Problems

Transformations are fairly straightforward when your source tree and result tree have pretty much the same overall structure. However, things can sometimes get pretty messy when they don't. I have on several occasions encountered application extract files from which EDI systems could not produce the desired output and vice versa. The only way around these types of problems is to write pre- or postprocessing programs to restructure the extract or import file.

Even though I've earned a few extra consulting fees by coding such auxiliary programs, I can't tell you how thrilled I am to report that I have yet to see or imagine an XSLT transformation that needed this type of preprocessing. XSLT with XPath expressions is extremely flexible and powerful. I'll go over just a couple of the common problems that are simply impossible to deal with using some conventional EDI systems.

Mapping a Flat Structure to a Hierarchy

I have written at least one preprocessing program for the very real problem of mapping a flat structure to a hierarchy. Some low-end bookkeeping systems export invoices one line item at a time, with each line item containing all the information about the invoice. Each line in the file has the same format. This is the basic organization of the files handled by our CSVToXML converter in Chapter 7. So, how do we turn something like this into a more conventional invoice with a header section and a detail section for each line item? It's really not very difficult.

Here's our source file. It's an example invoice from Chapter 7 with two line items.

Source (FlatToHierarchy.xml)
<?xml version="1.0" encoding="UTF-8"?>
<Invoice>
  <InvoiceLine>
    <CustomerNumber>BQ003</CustomerNumber>
    <InvoiceNumber>2002041</InvoiceNumber>
    <InvoiceDate>2002-11-12</InvoiceDate>
    <PONumber>AZ999345</PONumber>
    <DueDate>2002-12-02</DueDate>
    <ShipToName>
      Yazoo Grocers - NE Distribution Center
    </ShipToName>
    <ShipToStreet1>12 Industrial Parkway, NW</ShipToStreet1>
    <ShipToCity>Portland</ShipToCity>
    <ShipToStateOrProvince>ME</ShipToStateOrProvince>
    <ShipToPostalCode>04101</ShipToPostalCode>
    <ItemID>HCVAN</ItemID>
    <ItemQuantity>12</ItemQuantity>
    <UnitPrice>2.59</UnitPrice>
    <ItemDescription>
      Instant Hot Cocoa Mix - Vanilla flavor
    </ItemDescription>
    <ExtendedPrice>31.08</ExtendedPrice>
  </InvoiceLine>
  <InvoiceLine>
    <CustomerNumber>BQ003</CustomerNumber>
    <InvoiceNumber>2002041</InvoiceNumber>
    <InvoiceDate>2002-11-12</InvoiceDate>
    <PONumber>AZ999345</PONumber>
    <DueDate>2002-12-02</DueDate>
    <ShipToName>
      Yazoo Grocers - NE Distribution Center
    </ShipToName>
    <ShipToStreet1>12 Industrial Parkway, NW</ShipToStreet1>
    <ShipToCity>Portland</ShipToCity>
    <ShipToStateOrProvince>ME</ShipToStateOrProvince>
    <ShipToPostalCode>04101</ShipToPostalCode>
    <ItemID>HCMIN</ItemID>
    <ItemQuantity>24</ItemQuantity>
    <UnitPrice>2.53</UnitPrice>
    <ItemDescription>
      Instant Hot Cocoa Mix - Mint flavor
    </ItemDescription>
    <ExtendedPrice>60.72</ExtendedPrice>
  </InvoiceLine>
</Invoice>

We want the result document to look like the following file.

Result (HierarchyToFlat.xml)
<?xml version="1.0" encoding="UTF-8"?>
<Invoice>
  <Header>
    <CustomerNumber>BQ003</CustomerNumber>
    <InvoiceNumber>2002041</InvoiceNumber>
    <InvoiceDate>2002-11-12</InvoiceDate>
    <PONumber>AZ999345</PONumber>
    <DueDate>2002-12-02</DueDate>
  </Header>
  <ShipTo>
    <ShipToName>Yazoo Grocers - NE Distribution Center</ShipToName>
    <ShipToStreet1>12 Industrial Parkway, NW</ShipToStreet1>
    <ShipToStreet2/>
    <ShipToCity>Portland</ShipToCity>
    <ShipToStateOrProvince>ME</ShipToStateOrProvince>
    <ShipToPostalCode>04101</ShipToPostalCode>
    <ShipToCountry/>
  </ShipTo>
  <LineItemGroup>
    <LineItem>
      <ItemID>HCVAN</ItemID>
      <ItemQuantity>12</ItemQuantity>
      <UnitPrice>2.59</UnitPrice>
      <ExtendedPrice>31.08</ExtendedPrice>
    </LineItem>
    <ItemDescription>
      <Description>Instant Hot Cocoa Mix - Vanilla flavor</Description>
    </ItemDescription>
  </LineItemGroup>
  <LineItemGroup>
    <LineItem>
      <ItemID>HCMIN</ItemID>
      <ItemQuantity>24</ItemQuantity>
      <UnitPrice>2.53</UnitPrice>
      <ExtendedPrice>60.72</ExtendedPrice>
    </LineItem>
    <ItemDescription>
      <Description>Instant Hot Cocoa Mix - Mint flavor</Description>
    </ItemDescription>
  </LineItemGroup>
</Invoice>

Here's the stylesheet.

Stylesheet (FlatToHierarchy.xsl)
<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="1.0"
    xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
  <xsl:output method="xml" version="1.0" encoding="UTF-8"
      indent="yes"/>
  <xsl:template match="/Invoice">
    <Invoice>
      <xsl:call-template name="OutputHeader"/>
      <xsl:call-template name="OutputShipTo"/>
      <xsl:apply-templates select="InvoiceLine"/>
    </Invoice>
  </xsl:template>
  <xsl:template name="OutputHeader">
    <Header>
      <CustomerNumber>
        <xsl:value-of select="InvoiceLine[1]/CustomerNumber"/>
      </CustomerNumber>
      <InvoiceNumber>
        <xsl:value-of select="InvoiceLine[1]/InvoiceNumber"/>
      </InvoiceNumber>
      <InvoiceDate>
        <xsl:value-of select="InvoiceLine[1]/InvoiceDate"/>
      </InvoiceDate>
      <PONumber>
        <xsl:value-of select="InvoiceLine[1]/PONumber"/>
      </PONumber>
      <DueDate>
        <xsl:value-of select="InvoiceLine[1]/DueDate"/>
      </DueDate>
    </Header>
  </xsl:template>
  <xsl:template name="OutputShipTo">
  <ShipTo>
    <ShipToName>
        <xsl:value-of select="InvoiceLine[1]/ShipToName"/>
    </ShipToName>
    <ShipToStreet1>
        <xsl:value-of select="InvoiceLine[1]/ShipToStreet1"/>
    </ShipToStreet1>
    <ShipToStreet2>
        <xsl:value-of select="InvoiceLine[1]/ShipToStreet2"/>
    </ShipToStreet2>
    <ShipToCity>
        <xsl:value-of select="InvoiceLine[1]/ShipToCity"/>
    </ShipToCity>
    <ShipToStateOrProvince>
        <xsl:value-of
            select="InvoiceLine[1]/ShipToStateOrProvince"/>
    </ShipToStateOrProvince>
    <ShipToPostalCode>
        <xsl:value-of select="InvoiceLine[1]/ShipToPostalCode"/>
    </ShipToPostalCode>
    <ShipToCountry>
        <xsl:value-of select="InvoiceLine[1]/ShipToCountry"/>
    </ShipToCountry>
  </ShipTo>
  </xsl:template>
  <xsl:template match="InvoiceLine">
  <LineItemGroup>
    <LineItem>
      <ItemID>
        <xsl:value-of select="ItemID"/>
      </ItemID>
      <ItemQuantity>
        <xsl:value-of select="ItemQuantity"/>
      </ItemQuantity>
      <UnitPrice>
        <xsl:value-of select="UnitPrice"/>
      </UnitPrice>
      <ExtendedPrice>
        <xsl:value-of select="ExtendedPrice"/>
      </ExtendedPrice>
    </LineItem>
    <ItemDescription>
      <Description>
        <xsl:value-of select="ItemDescription"/>
      </Description>
    </ItemDescription>
  </LineItemGroup>
  </xsl:template>
</xsl:stylesheet>

There are only a few key points to understanding how this stylesheet works. You'll notice that the first xsl:template sets up the overall look of the result tree, calls two xsl:templates by name, and then invokes the final one by xsl:apply-templates. The two named xsl:templates extract values from the first InvoiceLine Element. The select expressions in the OutputHeader xsl:template use "[1]". This is an XPath predicate of the location step expression that says use the first occurrence of this node on the current axis. Since the default axis is child, and that axis moves in document order, we extract the values from the first InvoiceLine child of the current context node of Invoice. Document order is defined by the W3C as being the same order you would have if you read an XML input document serially, character by character, from a file or input stream. This may seem intuitive or overkill for some people, but specifications always have to make the obvious explicit.

The OutputShipTo xsl:template works in exactly the same way as OutputHeader. This approach in the first few xsl:template Elements lets us create and populate our two header level Elements from the contents of the first line item. The final xsl:template and the xsl:apply-templates Element that invokes it revert back to the rule-based model we are familiar with from our other examples.

I should note that because in this example we use most of the same Element names in the result tree that we do in the source, there are some other XSLT features that we could use to make this stylesheet even more compact. For simplicity's sake I'm just using features that we've covered so far.

So, that was much easier that it is in some EDI mappers I've worked with! Let's see how to go back the other way.

Mapping a Hierarchy to a Flat Structure

Although the transformation for mapping a hierarchy to a flat structure is a mirror image of what we just did, the XSLT is not. Here is how the stylesheet looks, using the same source and result files from our example above but with their roles reversed.

Stylesheet (HierarchyToFlat.xsl)
<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="1.0"
    xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
  <xsl:output method="xml" version="1.0" encoding="UTF-8"
      indent="yes"/>
  <xsl:template match="/Invoice">
    <Invoice>
      <xsl:apply-templates select="LineItemGroup"/>
    </Invoice>
  </xsl:template>
  <xsl:template match="LineItemGroup">
  <InvoiceLine>
    <CustomerNumber>
      <xsl:value-of select="/Invoice/Header/CustomerNumber"/>
    </CustomerNumber>
    <InvoiceNumber>
      <xsl:value-of select="/Invoice/Header/InvoiceNumber"/>
    </InvoiceNumber>
    <InvoiceDate>
      <xsl:value-of select="/Invoice/Header/InvoiceDate"/>
    </InvoiceDate>
    <PONumber>
      <xsl:value-of select="/Invoice/Header/PONumber"/>
    </PONumber>
    <DueDate>
      <xsl:value-of select="/Invoice/Header/DueDate"/>
    </DueDate>
    <ShipToName>
      <xsl:value-of select="/Invoice/ShipTo/ShipToName"/>
    </ShipToName>
    <ShipToStreet1>
      <xsl:value-of select="/Invoice/ShipTo/ShipToStreet1"/>
    </ShipToStreet1>
    <ShipToStreet2>
      <xsl:value-of select="/Invoice/ShipTo/ShipToStreet2"/>
    </ShipToStreet2>
    <ShipToCity>
      <xsl:value-of select="/Invoice/ShipTo/ShipToCity"/>
    </ShipToCity>
    <ShipToStateOrProvince>
      <xsl:value-of
          select="/Invoice/ShipTo/ShipToStateOrProvince"/>
    </ShipToStateOrProvince>
    <ShipToPostalCode>
      <xsl:value-of select="/Invoice/ShipTo/ShipToPostalCode"/>
    </ShipToPostalCode>
    <ShipToCountry>
      <xsl:value-of select="/Invoice/ShipTo/ShipToCountry"/>
    </ShipToCountry>
    <ItemID>
      <xsl:value-of select="LineItem/ItemID"/>
    </ItemID>
    <ItemQuantity>
      <xsl:value-of select="LineItem/ItemQuantity"/>
    </ItemQuantity>
    <UnitPrice>
      <xsl:value-of select="LineItem/UnitPrice"/>
    </UnitPrice>
    <ItemDescription>
      <xsl:value-of select="ItemDescription/Description"/>
    </ItemDescription>
    <ExtendedPrice>
      <xsl:value-of select="LineItem/ExtendedPrice"/>
    </ExtendedPrice>
  </InvoiceLine>
</xsl:template>
</xsl:stylesheet>

We take the same rule-based approach in outlining the overall structure of the result tree in the first xsl:template. There isn't much to it this time. Aside from defining the document's root Element, we perform an xsl:apply-templates on the LineItemGroup. The second xsl:template, matching that select, is where the real work happens. It is very similar to what we have seen so far except in the use of absolute location paths. To repeatedly insert the contents of our Header and ShipTo Elements into each line item, we use xsl:value-of with select expressions that point to the specific relevant Elements in the source tree. For the Elements relating to the line item details and descriptions, we just use the relative path name style we have used previously.

Tips for Dealing with Other Navigation Problems

The two problems we just looked at are perhaps among the ones that most vex users of certain EDI management systems. However, these problems are by no means the only ones. Keeping the following tips in mind may make life a bit easier for you when you're dealing with such problems.

  • Start with the result! Use a rule-based approach and code your first xsl:template to reflect the overall structure of your result document. Use xsl:call-template by name if you need to set up some branches in the result tree that bear no relation to anything in the source tree. Use iterative processing with xsl:apply-templates when you do have a correspondence between source and result Elements.

  • Within an xsl:template you can use an absolute location path expression to select values from anywhere in the source tree. As with some of our examples, you can even select specific occurrences of Elements or even the Attributes of these Elements.

  • Also within a template you can use relative path location expressions to select values from up, down, sideways, up then down, or any combination of directions to get to other data related to the result node you're creating.

  • If you have several source nodes with the same subtree path expression but need to process only certain ones, you can be selective. As we saw with one of our examples, you can code select expressions to test for specific values in source tree Elements or Attributes.

The thing to remember is that, unlike the advice the droll Yankee gave to the lost tourist, "You can get there from here!"


Advanced Techniques for Processing Content

In this section I present a few other techniques for your tool kit. The first has to do with overcoming a side effect of one style of XSLT coding. The others have to do with providing some functionality that is often essential to electronic commerce or EAI applications.

Omitting Empty Elements and Attributes

If you look back to HierarchyToFlat.xml, the desired result file for one of the examples above, you'll notice that ShipToStreet2 and ShipToCountry are empty Elements. This may be okay for some applications and schemas, but it may not be for others. These were created because in our xsl:template content we had as a literal result the start and end tags for these Elements, and the select for the xsl:value-of didn't find a match in the source tree. We can tell if we might run into a situation like this by either knowing the business data or reviewing the schema for the source document. If there's a minOccurs of zero on an Element in a sequence, then you should code to avoid creating empty Elements if you can't handle them in your result document.

There is a fairly simple technique for making sure that we don't create such empty Elements. Let's take a simple case of converting a buyer's name and address from one format to another. There may or may not be a BuyerStreet2 Element. We want the result tree to contain a Street2 Element if there is a BuyerStreet2 Element in the source but not to create one if there isn't. The source tree fragment appears below.

Source (MissingElement.xml)
<?xml version="1.0" encoding="UTF-8"?>
<MissingElement>
  <BuyerName>My Name</BuyerName>
  <BuyerStreet1>My Street</BuyerStreet1>
  <BuyerCity>My City</BuyerCity>
  <BuyerState>TX</BuyerState>
  <BuyerZip>99999</BuyerZip>
</MissingElement>

We want the result to look like the following file, with no empty Street2 Element.

Result (NoEmpties.xml)
<?xml version="1.0" encoding="UTF-8"?>
<NoEmpties>
  <Name>My Name</Name>
  <Street1>My Street</Street1>
  <City>My City</City>
  <State>TX</State>
  <Zip>99999</Zip>
</NoEmpties>

Here's the stylesheet.

Stylesheet (NoEmpties.xsl)
<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="1.0"
    xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
  <xsl:output method="xml" version="1.0" encoding="UTF-8"
      indent="yes"/>
  <xsl:template match="/MissingElement">
    <NoEmpties>
      <Name>
        <xsl:value-of select="BuyerName"/>
      </Name>
      <Street1>
        <xsl:value-of select="BuyerStreet1"/>
      </Street1>
      <xsl:if test="BuyerStreet2">
        <Street2>
          <xsl:value-of select="BuyerStreet2"/>
        </Street2>
      </xsl:if>
      <City>
        <xsl:value-of select="BuyerCity"/>
      </City>
      <State>
        <xsl:value-of select="BuyerState"/>
      </State>
      <Zip>
        <xsl:value-of select="BuyerZip"/>
      </Zip>
    </NoEmpties>
  </xsl:template>
</xsl:stylesheet>

XSLT is so flexible that there are other ways to handle this, but this method is pretty simple and straightforward. All we need to do is to put the literals for the Street2 start and end tags and the associated xsl:value-of into an xsl:if. The expression in the select simply has the name of the source Element. The expression evaluates to true if the Element is present and false if it isn't.

Converting Coded Values

In electronic commerce and EAI applications we frequently need to convert one coded value to another. A common example is when customers order goods using standard UPC codes or their own catalog numbers, and orders that you import into your order management system must use your internal item identifiers. Let's again use Big Daddy's Gourmet Cocoa as an example. The company needs to convert from the UPC numbers its customers are sending to the item identifiers used by the order management system. We'll just use a fragment of the line item as the source tree to demonstrate the conversion. Our basic problem is that we want to convert this:

Source (ItemUPCs.xml)
<?xml version="1.0" encoding="UTF-8"?>
<OrderedItems>
  <LineItem>
    <ItemUPC>35790000724</ItemUPC>
    <Qty>12</Qty>
    <UnitPrice>2.59</UnitPrice>
  </LineItem>
  <LineItem>
    <ItemUPC>35790000122</ItemUPC>
    <Qty>24</Qty>
    <UnitPrice>2.59</UnitPrice>
  </LineItem>
</OrderedItems>

to this:

Result (ItemIDs.xml)
<?xml version="1.0" encoding="UTF-8"?>
<LineItems>
  <Item>
    <ItemID>HCVAN</ItemID>
    <OrderedQty>12</OrderedQty>
    <UnitPrice>2.59</UnitPrice>
  </Item>
  <Item>
    <ItemID>HCMIN</ItemID>
    <OrderedQty>24</OrderedQty>
    <UnitPrice>2.59</UnitPrice>
  </Item>
</LineItems>

I hate to overuse the word "straightforward," but again that describes the transformation (at least, once you grasp the basic concepts). There are two XML documents involved. One, of course, is the stylesheet. The other is a catalog document that serves as a cross-reference table. Let's look at the catalog document first.

Lookup Table (ItemCatalog.xml)
<?xml version="1.0" encoding="UTF-8"?>
<!--This is an item catalog document for converting from UPC
    codes to our own item IDs-->
<Catalog>
  <Item>
    <ID>HCMIN</ID>
    <UPC>35790000122</UPC>
    <Description>
      Instant Hot Cocoa Mix - Mint flavor
    </Description>
  </Item>
  <Item>
    <ID>HCVAN</ID>
    <UPC>35790000724</UPC>
    <Description>
      Instant Hot Cocoa Mix - Vanilla flavor
    </Description>
  </Item>
  <Item>
    <ID>HCMOC</ID>
    <UPC>35790000999</UPC>
    <Description>
      Instant Hot Cocoa Mix - Mocha flavor
    </Description>
  </Item>
  <Item>
    <ID>HCDUC</ID>
    <UPC>35790000641</UPC>
    <Description>
      Instant Hot Cocoa Mix - Dutch Chocolate flavor
    </Description>
  </Item>
</Catalog>

(Okay, those of you who work in the grocery industry, just bite your tongues. I know those values aren't constructed as valid UPC numbers, but gimme a break!)

Here's the stylesheet.

Stylesheet (ItemUPCToIDLookup.xsl)
<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="1.0"
    xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
  <xsl:output method="xml" version="1.0" encoding="UTF-8"
      indent="yes"/>
  <!--Declare the catalog document and read it in -->
  <xsl:variable name="vCat"
      select="document('ItemsCatalog.xml')"/>
  <!--Template for root Element of result tree -->
  <xsl:template match="/OrderedItems">
    <LineItems>
      <xsl:apply-templates select="LineItem"/>
    </LineItems>
  </xsl:template>
  <!--This is the template for a line item in our result
      tree -->
  <xsl:template match="LineItem">
    <!--Grab the UPC Number so that it is available within the
        context of the select from the catalog -->
    <xsl:variable name="vUPC" select="ItemUPC"/>
    <Item>
      <ItemID>
        <!-- Select the Item from the catalog-->
        <xsl:value-of
            select="$vCat/Catalog/Item/ID[../UPC = $vUPC]"/>
      </ItemID>
      <OrderedQty>
        <xsl:value-of select="Qty"/>
      </OrderedQty>
      <UnitPrice>
        <xsl:value-of select="UnitPrice"/>
      </UnitPrice>
    </Item>
  </xsl:template>
</xsl:stylesheet>

As I said, the whole thing is pretty simple, but you have to get your head around a few concepts first. We've already seen things similar to most of what's in this stylesheet. Only a few things are actually new.

The first new concept is loading the catalog document. In its select expression the stylesheet declares a variable named vCat using the XSLT document function. A variable in XSLT functions in some ways like a variable in conventional programming languages. However, I think the choice of this term was unfortunate because some differences in behavior and usage can lead to confusion. Once a value has been assigned to an XSLT variable it can't be changed. In addition, a variable can represent more than just a single atomic value. In our case it represents the contents of an entire document. So, you might think of it really as more of a constant or macro than a variable. The document function is one of several such functions defined by XSLT that augment the built-in XPath functions. Using it in an expression allows us to use documents beyond just our source document. There are various ways to call the document function, but in this example our first (and only) argument is the file specification (or URI) of the document to include. We could just use the document function in every select expression where we needed to retrieve a value from the catalog. However, each time the XSLT processor would read in the requested document. Reading it into a variable at the top of the stylesheet means that we read it in only once.

So, now we have the catalog in memory and can do something with it in the xsl:template for the LineItem Element. We save the value of the ItemUPC Element into another variable. It may look like we're continually assigning a new value to the vUPC variable, but actually we aren't. It has scope only within the template and gets created fresh each time the template is invoked. We save the UPC number from the source tree so that we can use it for matching against the UPC number in the catalog document. The reason we need to save it will become evident shortly.

We pull the value from the catalog into the ItemID Element in the result tree using our customary xsl:value-of. However, the expression in the select Attribute looks a bit different than what we've seen so far. Instead of having the familiar absolute or relative path reference to something in the source tree, it is a location path to something in the catalog document. We prefix vCat with a dollar sign ($) to indicate a variable reference. This sets the root of the catalog document's tree as the first step in our location path. The rest of the steps, Catalog/Item/ID, bring us to the Element we want. Because there are four Elements in our catalog that match that path, we have to filter down to the one we want with this predicate:

[../UPC = $vUPC]

This also requires a bit of explanation. We previously grabbed the UPC number from the source tree into the vUPC variable. We needed to do that because our location path expression in the select Attribute points us to an entirely different context node. It would be fairly convoluted to try to get back to the source tree's UPC number within this select expression, so we take the easy route and first save the value to the vUPC variable. The other part of the test in the predicate uses an abbreviated syntax for relative path expressions. The first part of the whole select expression set the context node to the ItemID Element in the catalog, but we want to test against the value of its sibling Element, UPC. We do this with a two-step relative location path in the predicate. In the first step we go back up to the parent with the abbreviated syntax (..), and in the second step we use the default child axis to go to the parent's UPC child Element. This notation is again similar to a relative UNIX directory path.

This is the easy, dumb, and simple way to perform lookups. If you have more than a few items in your lookup table, XSLT provides a feature to declare keys and retrieve items using keys. Depending on the XSLT processor implementation and the size of your lookup table, using keys can be quite a bit more efficient than the method I describe here. However, showing you how it works is more involved than I want to get in this tutorial. Refer to my Web site for several other good XSLT books if you want to learn more about it.

Creating Powerful Lookup Tables for Item Numbers

The lookup approach discussed in this section becomes more and more useful as you understand all that it can do. We can use the table in the example not only to look up our own item ID from a UPC number but also to do the reverse lookup. We merely have to change the source of our lookup variable and the Element names referenced in the select Attribute's location step and predicate. But it doesn't have to stop there. All of the various permutations on UPC numbers (2-5-5, 1-5-5, 1-1-5-5, and so on) can be included in the same table and cross-referenced not only against an internal item number but also against each other. You could even include relevant supplemental information such as units per case, weights, dimensions, and other measurements often needed for shipment notification and invoices. Going one step beyond this, you could add a stylesheet reference to this document that would display selected information in HTML as a Web page. What an aid that might be toward ensuring that your customers order by using valid item numbers! In addition to performing lookups for standard numbers like UPC or EAN (European Article Numbering) codes, you could also include Elements for customer-specific item numbers.

As I write this I'm working with an EDI client who has just such a mess of item numbers and supplemental information to manage. My client's EDI system has a pretty good table lookup system. However, if it did all that a simple document like the example in this sidebar could do, my work with this client would be a lot easier.

Handling Calculations

We can use arithmetic support in XPath expressions to insert calculated values into the result tree based on values in the source tree. Let's take a hypothetical example in which our order management system is able to export shipment notices as XML documents. These documents include Elements for quantity ordered, quantity shipped, and quantity previously shipped. However, we have a big customer who insists that we send information about backordered quantity, too. Here's the source document.

Source (AddAndSubtract.xml)
<?xml version="1.0" encoding="UTF-8"?>
<AddAndSubtract>
  <QtyOrdered>1000</QtyOrdered>
  <QtyShipped>500</QtyShipped>
  <QtyPreviouslyShipped>0</QtyPreviouslyShipped>
</AddAndSubtract>

Here is the desired result.

Result (AddedAndSubtracted.xml)
<?xml version="1.0" encoding="UTF-8"?>
<AddedAndSubtracted>
  <QtyOrdered>1000</QtyOrdered>
  <QtyShipped>500</QtyShipped>
  <QtyPreviouslyShipped>0</QtyPreviouslyShipped>
  <QtyBackordered>500</QtyBackordered>
</AddedAndSubtracted>

And here's the stylesheet.

Stylesheet (AddAndSubtract.xsl)
<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="1.0"
    xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
  <xsl:output method="xml" version="1.0" encoding="UTF-8"
      indent="yes"/>
  <xsl:template match="AddAndSubtract">
    <AddedAndSubtracted>
      <xsl:copy-of select="QtyOrdered"/>
      <xsl:copy-of select="QtyShipped"/>
      <xsl:copy-of select="QtyPreviouslyShipped"/>
      <QtyBackordered>
        <xsl:value-of
     select="QtyOrdered - (QtyShipped + QtyPreviouslyShipped)"/>
      </QtyBackordered>
    </AddedAndSubtracted>
  </xsl:template>
</xsl:stylesheet>

Yes, folks, we can do math in an XPath expression! We're also using a new feature in this stylesheet. We use the xsl:copy-of Element to copy Elements from our source tree into the result tree when they have the same names.

Simple addition and subtraction are usually fairly easy and reliable. However, some authorities advise you to be cautious about multiplication and division. Some XSLT processors implement these simple math functions with floating point operations, which means that 1 + 1 can sometimes equal 1.9999999 rather than 2. However, I ran the following example with both msxsl and Xalan and got the same (correct) results. All I can say is to consult the documentation of your XSLT processor (or API library) and perform some due diligence testing. If things look okay, do math with confidence.

Source (Multiply.xml)
<?xml version="1.0" encoding="UTF-8"?>
<Multiply>
  <QtyOrdered>48</QtyOrdered>
  <UnitPrice>2.59</UnitPrice>
</Multiply>
Stylesheet (Multiply.xsl)
<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="1.0"
      xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
  <xsl:output method="xml" version="1.0" encoding="UTF-8"
      indent="yes"/>
  <xsl:template match="/Multiply">
    <Multiplied>
      <xsl:copy-of select="QtyOrdered"/>
      <xsl:copy-of select="UnitPrice"/>
      <ExtendedPrice>
        <xsl:value-of select="QtyOrdered * UnitPrice"/>
      </ExtendedPrice>
    </Multiplied>
  </xsl:template>
</xsl:stylesheet>
Result (Multiplied.xml)
<?xml version="1.0" encoding="UTF-8"?>
<Multiplied>
  <QtyOrdered>48</QtyOrdered>
  <UnitPrice>2.59</UnitPrice>
  <ExtendedPrice>124.32</ExtendedPrice>
</Multiplied>


Handling Namespaces

Mostly what we've talked about in this chapter is how to perform some common transformation tasks. This section still deals with a situation you are very likely to encounter, but it deals strictly with XSLT care and feeding. We're talking about namespaces. Namespaces can be tricky any time you work with them, and XSLT is no exception. Fortunately, we can do a few things to keep ourselves out of too much trouble.

There are several key points to remember about namespaces. Your stylesheet, first and foremost, is still an XML document. To be well formed it must obey all the usual rules concerning namespace handling. The upshot to this is that namespace declarations need to be made at the appropriate scope within the document. This is usually in the root xsl:stylesheet Element, but it doesn't have to be. If you don't declare your namespaces in the root Element, just make sure you put them where their scope is appropriate.

Another thing to remember is that if your source document uses namespace prefixes on some of your Elements, you had better include that prefix in your match expressions. (There are some other ways to reference namespaces in XPath expressions, but the simplest way is just to use the prefix used in the source document.)

The most common namespace problems that most of us will encounter have to do with converting between documents that use named namespaces and those that don't. Let's take a simple example of converting from our own invoice document, which doesn't use named namespaces, to a standard invoice that references a named namespace. The standard invoice also has a schema reference. Here's our source document.

Source (NamespacesAdd.xml)
<?xml version="1.0" encoding="UTF-8"?>
<MyInvoice>
  <InvoiceNum>QA-Z33WX</InvoiceNum>
  <DueDate>2003-02-15</DueDate>
</MyInvoice>

We need to create the following result document.

Result (NamespacesAdded.xml)
<?xml version="1.0" encoding="UTF-8"?>
<std:Invoice
    xsi:schemaLocation="urn:org.standardsrus.names
    http://www.standardsrus.schemas/V0756/invoice.xsd"
    xmlns:std="urn:org.standardsrus.names"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <InvoiceNumber>QA-Z33WX</InvoiceNumber>
  <InvoiceDueDate>2003-02-15</InvoiceDueDate>
</std:Invoice>

Here is the stylesheet for the transformation.

Stylesheet (NamespacesAdd.xsl)
<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="1.0"
    xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
    xmlns:std="urn:org.standardsrus.names"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" >
  <xsl:output method="xml" version="1.0" encoding="UTF-8" indent="yes"/>
  <xsl:template match="MyInvoice">
    <std:Invoice xsi:schemaLocation="urn:org.standardsrus.names
        http://www.standardsrus.schemas/V0756/invoice.xsd">
      <InvoiceNumber>
        <xsl:value-of select="InvoiceNum"/>
      </InvoiceNumber>
      <InvoiceDueDate>
        <xsl:value-of select="DueDate"/>
      </InvoiceDueDate>
    </std:Invoice>
  </xsl:template>
</xsl:stylesheet>

Going this way isn't too bad. The main things to note are that we declare all our namespaces in the stylesheet's root Element and that in the literal result root Element we add the xsi:schemaLocation Attribute. When the source is processed with the stylesheet, all the namespace declarations in the stylesheet root (except for the XSLT namespace) are copied into the root Element of the result document.

Going back the other way, from a source that uses named namespaces to a result that doesn't, can be tricky. Let's begin with the NamespacesAdded.xml document as the source and process it using the following stylesheet.

Stylesheet (NamespacesNotRemoved.xsl)
<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="1.0"
    xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
    xmlns:std="urn:org.standardsrus.names">
  <xsl:output method="xml" version="1.0" encoding="UTF-8"
      indent="yes"/>
  <xsl:template match="std:Invoice">
    <MyInvoice>
      <InvoiceNum>
        <xsl:value-of select="InvoiceNumber"/>
      </InvoiceNum>
      <DueDate>
        <xsl:value-of select="InvoiceDueDate"/>
      </DueDate>
    </MyInvoice>
  </xsl:template>
</xsl:stylesheet>

Note that we have to include the std: namespace prefix on the root Element in the match Attribute. By using the stylesheet above we get the following undesired result document.

Result (NamespacesNotRemoved.xml)
<?xml version="1.0" encoding="UTF-8"?>
<MyInvoice xmlns:std="urn:org.standardsrus.names">
  <InvoiceNum>QA-Z33WX</InvoiceNum>
  <DueDate>2003-02-15</DueDate>
</MyInvoice>

So, how do we get rid of the namespace reference on the root Element of our result document? We use a special Attribute on the stylesheet root Element, exclude-result-prefixes. For this Attribute's value we put a whitespace-separated list of the prefixes of all the namespace declarations that we don't want copied into the result document's root Element.

Stylesheet (NamespacesRemoved.xsl)
<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="1.0"
    xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
    xmlns:std="urn:org.standardsrus.names"
    exclude-result-prefixes="std">
  <xsl:output method="xml" version="1.0" encoding="UTF-8"
    indent="yes"/>
  <xsl:template match="std:Invoice">
    <MyInvoice>
      <InvoiceNum>
        <xsl:value-of select="InvoiceNumber"/>
      </InvoiceNum>
      <DueDate>
        <xsl:value-of select="InvoiceDueDate"/>
      </DueDate>
    </MyInvoice>
  </xsl:template>
</xsl:stylesheet>
Result (NamespacesRemoved.xml)
<?xml version="1.0" encoding="UTF-8"?>
<MyInvoice>
  <InvoiceNum>QA-Z33WX</InvoiceNum>
  <DueDate>2003-02-15</DueDate>
</MyInvoice>

Everything is the same in the stylesheet except for the addition of this one Attribute.

I'm sure that if you work with XSLT long enough you will get bitten by other types of namespace problems. Keep the basics in mind, have your references ready, and I'm sure you'll persevere.


Calling Non-XSLT Procedures

This last section of the chapter deals with a feature that many EDI translators have: giving the user an option to call their own routines. These are known by various names such as user exits, escape routines, and user functions. In XSLT this feature is referred to as an extension function. The exact method for supporting extension functions isn't standard among all XSLT processors, so you can expect that there are variations. In this section I'll present a simple example of one way it can be done with Xalan, and I'll briefly discuss some approaches for MSXML.

Even when just considering Xalan there are several approaches and variations for supporting extension functions. I present here a fairly simple, basic approach that involves calling a static method in a class that isn't part of a package.

As a sample scenario, let's imagine that we have placed an order and that we want to determine the status of the order. XSLT doesn't know anything about orders or their status, so we'll implement this functionality with an extension function. Our simplified source document looks like this.

Source (Extension.xml)
<?xml version="1.0" encoding="UTF-8"?>
<ExtensionSource>
  <Cust>Mike</Cust>
  <Bevg>Espresso</Bevg>
</ExtensionSource>

And here's the result we want to produce.

Result (ExtensionResult.xml)
<?xml version="1.0" encoding="UTF-8"?>
<ExtensionResult>
  <Customer>Mike</Customer>
  <Beverage>Espresso</Beverage>
  <OrderStatus>Your espresso is ready, sir.</OrderStatus>
</ExtensionResult>

(May I please have a croissant to go with that?)

The Java method we want to invoke from our stylesheet is getOrderStatus. It resides in a MyExtensions class, as shown here.

MyExtensions Class (MyExtensions.java)
public class MyExtensions {
  public static String getOrderStatus()
  {
    return "Your espresso is ready, sir.";
  }
}

The function returns a Java String object that gets mapped to an XSLT string.

The approach involves three basic steps. We first have to tell Xalan about the extension function. This involves making it part of a specific namespace that Xalan associates with Java. In the stylesheet's root Element we declare the namespace http://xml.apache.org/xslt/java. We assign it a namespace prefix of java, although any syntactically valid prefix would work. Since we've declared this namespace in the stylesheet we'll have to take special action to prevent its declaration from appearing in the result tree. We do this by using the exclude-result-prefixes Attribute discussed earlier in the chapter.

Secondly, we call the function in the appropriate place in the stylesheet. We use it in an XPath expression in the same way that we call a built-in XPath function. In our example we prefix the function name getOrderStatus with the java namespace prefix and the MyExtensions class name. The XPath expression is the value of the select Attribute of an xsl:value-of Element. Through these mechanisms the string returned by the function becomes the content of the Order Status Element.

Finally, when we actually run Xalan, we must be sure that the compiled Java class file (or jar archive if we have it in one) is in our class path.

Here's the stylesheet.

Stylesheet (Extension.xsl)
<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="1.0"
    xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
    xmlns:java="http://xml.apache.org/xslt/java"
    exclude-result-prefixes="java">
  <xsl:output method="xml" version="1.0" encoding="UTF-8"
      indent="yes"/>
  <xsl:template match="ExtensionSource">
    <ExtensionResult>
      <Customer>
       <xsl:value-of select="Cust"/>
      </Customer>
      <Beverage>
       <xsl:value-of select="Bevg"/>
      </Beverage>
      <OrderStatus>
       <xsl:value-of
            select="java:MyExtensions.getOrderStatus()"/>
      </OrderStatus>
    </ExtensionResult>
  </xsl:template>
</xsl:stylesheet>

The XSLT processor in MSXML 4.0 doesn't support extension functions or elements per se. Support for extensions is provided through embedded script implementations using msxsl:script and external objects using addObject. Consult the MSXML reference for details. If you're using a processor other than Xalan or MSXML's XSLT processor (which msxsl calls), consult its documentation if you want to add your own extensions.


References

Skonnard, Aaron, and Martin Gudgin. 2002. Essential XML Quick Reference. Boston, MA: Addison-Wesley.

W3C (World Wide Web Consortium). 1999. XSL Transformations, Version 1.0, W3C Recommendation, November 16. Available online at www.w3.org/TR/1999/REC-xslt-19991116, accessed January 26, 2003.


Resources

DOM Reference, Microsoft XML Core Services (MSXML) 4.0, by Microsoft Corporation. Online documentation included as part of the MSXML SDK. Available online at http://msdn.microsoft.com/xml.

XML Path Language (XPath), Version 1.0, W3C Consortium Recommendation. Available online at www.w3.org/TR/1999/REC-xpath-19991116.

XSL-List listserv, an open forum on XSL hosted by Mulberry Technologies. Subscription instructions available online at www.mulberrytech.com/xsl/xsl-list/. This list is a good place to find solutions for specific XSLT problems, but the message traffic can be a bit overwhelming and it covers a lot more than just XSLT. See the URL listed above for further information and instructions for subscribing.

Suggestions for books on XSLT are available from my Web site at www.rawlinsecconsulting.com.

More examples: Nearly all the examples in this chapter dealt with very small snippets of source documents and small stylesheets. To see some more comprehensive examples of stylesheets that transform typical documents such as purchase orders and invoices, visit my Web site.


Chapter 11. Using the Conversion Techniques Together

Solving the puzzle

This chapter presents end user suggestions for using the techniques and utilities together to solve common conversion problems. I also present the preliminaries of an overall system that might be built based on the utilities.


Pipe and Filter Revisited

Each of the utilities and techniques developed (or discussed, as in the case of XSLT) in the preceding chapters performed a single conversion or transformation. This may be fine for some types of problems, but most of us dealing with legacy systems will probably need more than one transformation. Let's revisit our modified pipe and filter model from Chapter 1. The figure is repeated here as Figure 11.1.

Figure 11.1. Modified Pipe and Filter Style

Each of the things we've built or discussed so far is a filter. All of our legacy format to XML conversion utilities started with initial inputs of legacy files, used parameters from file description documents (or command line options for the basic CSV to XML converter), and produced one or more XML documents as output. The reverse was true for our XML to legacy format converters. XSLT follows this same model by taking a source tree as input, getting parameters from the stylesheet, and producing a result tree as the output. We now need to string these pieces together into systems that solve common conversion problems. In this implementation our conversion utilities and an XSLT processor are the filters, and the file system is the pipe. We'll talk later in the chapter about using a more efficient pipe.

Before we go further, though, some of you might be thinking, "Hey, this is just slow, inefficient, old-fashioned batch processing!" You're exactly right. The degenerate case of the pipe and filter style is indeed batch. But to that I say, "So what?" We're talking about legacy systems here, and most of the import and export operations we're dealing with happen in batch mode.

Sure, a lot of applications really benefit from more sophisticated real-time processing and more tightly integrated interfaces. However, it doesn't make sense to spend a lot of money sending out invoices in real time if you're giving your customers thirty days to pay. This is true not only of our legacy to and from XML conversions and XSLT transformations but also of several other technical solutions to common business problems. We must always consider the overall context in which a system will be used and the overriding consideration of a reasonable return on investment. From this perspective, sophisticated real-time interfaces are often not justifiable and batch is often good enough.

We're coding these little systems in batch mode and we're going to be very happy about it. We'll talk about some more efficient options later in the chapter.


Sample Conversion Scenarios and Scripts

Let's look at a few real-life use cases in which we can build a functioning system from the components discussed above.

Purchase Order: UBL to XML to CSV

There are a lot of desktop bookkeeping systems in use that still use CSV files for importing and exporting data. I know of one, I'll call it BlueberryBush, that uses CSV formats very similar to the examples presented in Chapter 7. A few major buyers are starting to use XML formats for their purchase orders. Several such formats are emerging, but very few have much cachet as standards.

We talked in Chapter 4 about the business document schemas being developed by OASIS's UBL technical committee. Let's suppose that a big grocery chain adopts the UBL Order format. The grocer gives its suppliers an option to log into a Web site and download the orders in the UBL Order format. Let's say that Big Daddy's Gourmet Cocoa is running BlueberryBush and receives these UBL orders. Figure 11.2 shows the processing flow. Note that in this figure and others later in the chapter I use a convention of referring to the XML representation of a legacy format by prefixing it with the legacy format name. For example, the XML representation of a CSV format, used as input to the XMLToCSV utility, is named "CSV XML."

Figure 11.2. UBL Order to XML to CSV Order

We have several options for what to do on our receiving system based on its capabilities. Let's assume, since BlueberryBush is a Win32 application, that Big Daddy's PC systems support consultant, Sheila, knows her way around pretty well. She sets up and installs MSXML, the CSV conversion utilities, and a Perl interpreter. She tells Big Daddy to download the orders into a specific directory, then run this little job from a desktop icon. Here's the DOS bat file that will do the conversion.

bat File to Convert UBL to CSV (UBLOrderToCSV.bat)
rem UBLOrderToCSV.bat
rem Transform UBL Orders in Received to Transformed
perl UBLOrderToCSV.pl
rem Convert CSV version of XML to CSV Orders
XMLToCSV C:\Work\Transformed C:\Work\Invoices.csv
    C:\Blaster\XMLToCSVInvoice.xml -v

To run through all the UBL orders and turn them into the XML format that the XMLToCSV utility expects, she codes a small Perl script.

Perl Script to Transform UBL Orders with XSLT (UBLOrderToCSV.pl)
#!/usr/bin/perl
opendir(INDIR, 'C:\\Received')
  or die "Problem opening current directory";
$file = readdir(INDIR);
while ($file)
{
  if ($file =~ m/.xml$/)
  {
    print "Transforming ", $file, "\n";
    $comm = "msxsl C:\\Received\\" . $file .
        " C:\\Blaster\\UBLToCSV.xsl -o C:\\Transformed\\" .
    $file;
    system $comm;
  }
  $file = readdir(INDIR);
}
closedir INDIR;

The Perl script opens the C:\Received directory, and for each file it finds that has an xml extension it builds a command line for msxsl. The command line takes the input XML document in C:\Received and transforms it with the UBLToCSV.xsl stylesheet into a file of the same name in the C:\Transformed directory. The script then invokes the command line with the system function. We can certainly get more diligent about error recovery and housekeeping in these two scripts, but the basic operations would remain the same.

Invoice: Flat File to XML to EDI

Now let's expand on the conversions in Chapter 8. BlueberryBush has been bought by MegaSoftware and retired. Big Daddy applies his special BlueberryBush user discount to upgrade to MASH2004. This program uses the flat files we discussed in Chapter 8. Big Daddy has started to sell to some major grocery chains, which want him to send their invoices using X12 EDI, version 004010. However, he still doesn't need to do a lot of EDI, and he doesn't want to spend thousands of dollars for a capable desktop EDI system. He also doesn't want to spend his time on a WebEDI service entering invoices manually. Sheila sets up the FlatToXML and XMLToX12 utilities and a stylesheet to perform the conversions. Figure 11.3 shows the process.

Figure 11.3. Flat File Invoice to XML to X12 Invoice

The bat file and Perl scripts are very similar to the CSV conversion.

bat File to Convert Flat File to X12 (InvoiceToX12.bat)
rem InvoiceToX12.bat
rem Convert flat invoice to XML
FlatToXML C:\Work\Invoices C:\Work\XMLInvoices
    C:\Blaster\FlatToXMLInvoice.xml -v
rem Transform the XML representations from flat to X12
perl InvoiceToX12.pl
rem Convert XML to X12 invoices
XMLToX12 C:\Work\X12XMLInvoices C:\Out\Invoices.x12
    C:\Blaster\XMLToX12Invoice.xml -v
Perl Script to Transform from Flat File XML to X12 XML (InvoiceToX12.pl)
#!/usr/bin/perl
opendir(INDIR, '.')
  or die "Problem opening current directory";
$file = readdir(INDIR);
while ($file)
{
  if ($file =~ m/pl/)
  {
    print "Transforming ", $file, "\n";
    $comm = "msxsl C:\\Work\\XMLInvoices\\" . $file .
        " C:\\Blaster\FlatToX12Invoice.xsl
        -o C:\\Work\\X12XMLInvoices\\" . $file;
    system $comm;
  }
  $file = readdir(INDIR);
}
closedir INDIR;

This little system performs pretty much the same function as a good desktop EDI system at a small fraction of the cost.

Campaign Contribution Reporting: CSV to XML to Flat File

Let's now look at a scenario that's a bit different than the first two above. I'm on the board of a nonprofit group based in Seattle known as the Electronic Reporting Design Institute (www.erdi.org), which seeks to facilitate government-mandated reporting of contributions to political campaigns. That data is currently collected and reported to state and federal governments in a variety of formats including paper, CSV files exported from spreadsheets, and various flat file formats. The data is collated and stored by various means, but for this discussion let's assume that several states import the data into their systems by flat files (at least when they can get it electronically). The ERDI is working on a standard XML document to report campaign contributions. But, even if and when this schema gets created, it will probably take awhile for the campaign organizations and political parties to acquire software that would create these documents in a native format. How do we solve this problem?

Here's one solution using the utilities from this book. It involves two pieces. Figure 11.4 shows the first piece.

Figure 11.4. Campaign Organization Reporting

First, reporting organizations are given a standard CSV format to use when exporting data from a spreadsheet or other application. They are also given the CSVToXML utility with an appropriate file description document, schema for the resulting file, and an XSLT stylesheet to transform the result into the standard ERDI format. (I'm aware that new versions of certain spreadsheet products have the ability to create XML. But do you think these campaign organizations are running them?) The users enter the data into their spreadsheet or other application and then export it to a CSV file. They run the XSLT transformation on the CSV file, then send the document to the state. Figure 11.5 shows what the state does with it after that.

Figure 11.5. State Processing

When the state receives the document, the users have an XSLT stylesheet to transform the ERDI format into the XML representation of the existing flat file format, run the XMLToFlat utility on the result, and load the flat file into the reporting system. Figure 11.5 assumes that the state system is on a UNIX box running Xalan and the Java version of the utilities.

Figure 11.6 shows the whole system, representing both the reporting organization and the state.

Figure 11.6. Complete Campaign Finance Reporting Solution

All of this could be accomplished with very simple Windows bat files and UNIX scripts very similar to what we've seen so far. This would be a very simple, low-cost, and readily available solution for everyone involved.


Building a System: Babel Blaster

All the utilities in the book were designed so they could be used stand-alone as filters either by themselves or in combination with other filters. However, they were also designed so they could be built into a more comprehensive system. To develop that system the code from the examples has gone into an open source project being conducted under the GNU General Public License at http://babelblaster.sourceforge.net. As with any open source project run by volunteers, the requirements, what gets done, and when (or if) are all subject to who gets involved and how much time they can contribute. I'll outline here my proposals for the requirements for versions 1.0 and 1.1 and sketch out a few ideas about the overall architecture. If you want to find out the current status or help with the project, send me an e-mail (mike@rawlinsecconsulting.com).

Why "Babel Blaster"?

I am a great fan of the Hitchhiker's Guide to the Galaxy series written by the late Douglas Adams. In that series people were able to understand each of the various galactic languages by inserting a small yellow Babel fish into one of their ears. As this project got started I thought, what a great name for a file format converter! Unfortunately, a few other people also thought it was a good name and have used it for purposes too similar to what I had in mind. I thought about it a bit more and remembered my older brother's great delight in Zaphod Beeblebrox and his famous drink, the Pan Galactic Gargle Blaster. And I thought, that's it: Babel Blaster!

It conveys the idea of blasting away the babble of different document formats. And it has just the right kind of attitude.

Version 1.0 Requirements

I talked in Chapter 1 about functional and nonfunctional requirements. Let's approach Babel Blaster the same way.

Functional Requirements

We discussed enhancements in several chapters. Here is some of the functionality I propose for version 1.0.

  • Support a series of transformations rather than a single transformation step.

  • Support CSV files with multiple record formats and record groups.

  • Add support for other data types, primarily different date formats.

  • Support processing a directory of legacy format files rather than just a single file.

  • Fully support ANSI X12 EDI syntax features up through version 004060.

  • Fully support UN/EDIFACT EDI in both ISO 9735 versions (2 and 4).

  • Track Functional Acknowledgments received for transmitted data and provide other auditing and tracking capabilities.

  • Enable processing of a transmission file that contains several EDI interchanges.

  • Provide a relatively easy way to automatically create file description documents (or the version 1.0 equivalent) specifying the grammar of EDI messages.

Nonfunctional Requirements

Again, nonfunctional requirements of the system relate not to what the system does but to how it does it. These aspects deal with attributes of the system or its use such as cost, performance, and ease of use. For version 1.0 we'll keep the requirements I laid out in Chapter 1 such as maintainability and portability and add a few other important ones.

  • Ensure backward compatibility: Here we're mainly concerned with the file description documents and other components that end users develop to use the stand-alone utilities developed in the book. The system must provide either direct backward compatibility or easy migration utilities.

  • Migrate the C++ version to open DOM API: I used MSXML for the book because I wanted to address DOM programming for people trying to use it with C++. However, MSXML poses certain complexities due to COM. I also believe that if the project is moving to an open source model, it shouldn't rely on APIs that aren't.

  • Improve ease of use: I don't find the utilities particularly hard to use, but I'm sure that improvements could be made. This is almost always a persistent requirement when upgrading a product, and the list seems incomplete without it.

A nongoal for this version is performance improvements.

This seems to me a fairly comprehensive list. Getting all of these features into version 1.0 will depend on having enough volunteers involved. If you like to code this kind of thing, talk to me!

Architectural Overview

Even though I tried very hard to build these utilities using reusable modules, as we start looking at a larger architecture a few things may have to change. As any experienced designer knows, the shape of individual components can change as the overall system in which they are used changes. I'm sure that the lower-level classes will be okay, but the converter classes may need some adjustments. It is prudent to defer major architecture changes until after requirements have been finalized, so I'm not going to try to sketch out even a block diagram at this point. However, we can think about some general characteristics of the architecture.

To facilitate ease of use we can provide users with a single program to invoke. We can further simplify use by letting them specify the name of the input and a single identifier for the type of conversion to perform. This identifier would point to a set of data in a data store that would contain the file description document specification (or equivalent) and other relevant details. With EDI formats as the input, all that might be necessary is a generic identifier for the EDI syntax since EDI interchanges have a lot of identifying information in them.

The main program would retrieve the details from the data store about the first transformation step and execute the appropriate filter. Then, based on other details from the data store regarding succeeding transformation steps, it would invoke the appropriate filters and write the final output. Output location could be determined either by information retrieved from the data store or from a command line argument. Allowing a user-specified system-level command for either the initial or final step would provide a way to invoke file transmission utilities.

The overall architectural style remains pipe and filter, but the individual filters retain an object-oriented design. This means that the main program creates converter objects of the appropriate classes as it needs them.

Trading Partner/Application Information

In Chapter 9 we identified a need for a data store for EDI control information. In the current version of the XMLToX12 utility the data store was used only for generating sequential control numbers for interchanges and functional groups. This XML document could be expanded to include audit entries for interchanges and groups that are transmitted. It could also be used for tracking Functional Acknowledgments received. Because it would be an XML document, creating formatted reports, displaying it as a Web page, or developing simple queries against it would be fairly easy using standard tools.

In addition, we can observe that while the grammars of our various file formats are similar, there are some fundamental differences in processing CSV and flat files versus EDI. For EDI we can conceivably have one generic grammar that could cover all uses of an X12 transaction set or UN/EDIFACT message. However, for populating control segments in outbound interchanges a different set of information is required for each trading partner. It makes sense to set up a data store for EDI trading partner information and to break out the EDI grammars into separate documents. In addition, for ease of use we could set up entries in this data store, keyed off of short identifiers, that could be used to retrieve the file description documents for other legacy formats.

Linking Pipes and Filters

While we're still going to use pipes and filters for the general architectural style, we have at least two different choices for the types of pipe we use. The current utilities use the file system as a pipe. For simplicity, version 1.0 could continue to use the file system and simply fork child processes to execute stand-alone filters as necessary. However, all our conversion utilities are coded to deal with DOM documents. If we're passing them off to something else that can use a DOM document as input, there's no reason to write it out to the disk and read it back in again. This is no problem for the converters and presents only a minor problem for XSLT transformations. It is likely we'll want to set up direct calls to Xalan (Java or C++) from the main program rather than fork a child process.

Version 1.1 Requirements

Functional Requirements

I don't have many ideas at this point, but a couple come to mind.

  • Native data exchange capability: Performing file format conversion is essential to many electronic commerce scenarios. However, it is worthless unless you can also move the data around. Building in a native capability for transmitting and receiving data would save users from having to do their own integration with a suitable utility. The top candidates for file transfer and packaging protocols might be FTP, HTTP, SMTP, S/MIME, ebXML messaging, EDIINT AS1/AS2, or SOAP (more about these in Chapter 13). The primary network protocol that should be supported is IP (for the public Internet). However, there may also be a need in some situations to support other network protocols using dial-up access. Integrating Babel Blaster with one or more existing open source utilities that provide these capabilities could save considerable development effort. However, links to proprietary systems could be provided, or we could certainly investigate building the appropriate functionality from scratch.

  • Conversion of legacy IBM data formats: I put this one off until version 1.1 because I'm not sure of the demand and because of some of the complexity. Converting between EBCDIC and ASCII isn't too bad, but supporting all the numeric formats can be fairly involved.

Nonfunctional Requirements

We have avoided much emphasis on performance because we wanted to focus on maintainability, understandability, and functionality. However, I think if Babel Blaster gets to version 1.1 we might want to look at performance and ease of use.

  • Performance: In several areas the coding and storage techniques might be made more efficient by a few very localized changes. I'm most concerned with byte to string conversions in Java and with all the dynamic creation and destruction of DataCell objects in both implementations. While the design depends on this behavior, we might make it more efficient by statically allocating buffers for the DataCell contents and pointing to them rather than asking the system to reallocate them with each new DataCell object.

  • Ease of use: I'm not quite sure what we can do here yet, but I'm sure there will be something. The book utilities and version 1.0 rely on command line programs and XML IDEs like XMLSPY and TurboXML for everything else. Perhaps a nice graphical user interface for the front end?

As I mentioned earlier, if you'd like to volunteer your ideas and skills for Babel Blaster, feel free to contact me.


Resources

Babel Blaster, an open source project at SourceForge. Available online at http://babelblaster.sourceforge.net.

Open source Perl interpreters for various operating systems. Available online from Active State at www.activeware.com.


Chapter 12. Building XML Support into a Business Application

When we try to pick out anything, by itself, we find it hitched to everything else in the universe.

—John Muir

This chapter discusses general issues involved in integrating XML with business applications at the code level.


What Should Be "XMLized"?

Even though John Muir lived before the age of computers and thought in terms of the natural world, I don't think he would be too surprised to learn that his observation holds true for the world of business applications. Very few of our business applications are used completely stand-alone; they almost always share data and have interactions with other applications. The data sharing may be crude and the interactions made by humans, but the "hitching" is still there. So, the easy answer to identifying what we should turn into XML is this: Anything that can be consumed by another application or produced by another application for yours to consume is a candidate for representation in an XML format. This is consistent with the basic approach in this book of using XML as an interchange format. In addition to this generalization, you may want to consider XML as the native data storage format for certain types of application information.

Beyond these rather general guidelines, the road ahead has gotten a bit confused over the last year or so due to the introduction of so-called "Web services." I say "so-called" because the term has been loosely applied to many fundamentally different technologies. It is sometimes hard to tell whether someone discussing a Web service is talking about a real architecture or just engaging in a bit of marketing hype by describing old products with the latest buzz words. This situation wasn't helped by the fact that it took the W3C's Web Services Architecture Working Group a long time to achieve an apparent consensus on a concise definition of a Web service that had any real descriptive power (and at the time I proofread this chapter they were still discussing it!). Be that as it may, some real and potentially revolutionary concepts and technologies form the core of what is commonly understood as the Web services architecture. The general consensus is that the Web services architecture involves building distributed applications that run over the Internet (or at least over IP, meaning private subnets instead of just the public Internet). Some of the key technologies commonly included in this model are XML, SOAP (which started out as Simple Object Access Protocol but now is just SOAP), Web Services Description Language (WSDL), and Universal Description, Discovery, and Integration (UDDI).

I say that the road has become confused because the level at which we integrate applications has become a bit confused. There is a fundamental difference between importing or exporting batches of documents and "exposing" key application functionality as a Web service on the Internet. So, I feel a need to clarify exactly what we're going to cover in this chapter and in Chapter 13. This chapter will basically take the position of traditional application integration, that is, importing and exporting batches of documents (or other data). We'll begin and end at the XML document. Where it came from and where it goes to (at least at the technical level) are issues we'll discuss in Chapter 13.

This doesn't mean that this chapter (or the rest of the book) doesn't apply to your Web services strategy if you choose to develop one. If the Web services approach is about anything, it is about effectively using XML. So is this book. A Web service might provide the XML document to your application or you might hand off the document to a Web service. We're dealing with the XML documents that supply the business content component of a Web service, but any other consideration of Web services architecture is beyond our scope.


Devising an Architecture

Several issues must be addressed when considering how to build XML support into your application. We've talked a bit in earlier chapters about choosing an appropriate API and processing paradigm. We'll talk a bit more about it later in this chapter, too. Beyond the considerations we've already mentioned throughout the book, there are two others we need to review.

Add-on or Major Redesign?

This is as much an assessment for scoping your project as it is a choice to make. You need to determine whether you are going to add incremental functionality and modules or whether you are going to do a major application redesign for XML. Depending on several factors, XML might be supported as just one more import/export format. Here are a few criteria to bear in mind when making the assessment.

  • What language is your application written in, and what is your development environment? We touched on this briefly in Chapter 1 and will discuss it more later, but here's a basic assessment. If your application is written in Java, C++, or Visual Basic, or if your development and runtime environment supports linking with modules written in those languages, you will have a relatively easy time integrating at the code level. If your application is written in a strictly procedural language like C, COBOL, or FORTRAN and you can't link with C++, you do have some code-level integration options, but they aren't necessarily going to be very cheap or pretty. You may want to rely on external utilities (such as those developed in this book) for XML support.

  • From a code-level perspective, how well does the current application support importing and exporting data? If the code is modular and well structured, supporting XML as an additional format may not require much redesign. However, if it's a real mess of spaghetti, you may want to redesign the code before trying to build in XML support.

  • From a functional perspective, what is the support for traditional, document-oriented electronic commerce? Many applications with good support for EDI integration offer users features for managing information needed for trading relationships and by trading partners. (We'll discuss some specifics later.) It is a near certainty that many of the same features would be needed for electronic commerce using XML rather than EDI formats. If you offer these features now, an XML add-on may not require much work. If you don't, you are looking at a major project.

This brief list serves to introduce the topic. More issues and details will arise as we look at integration in greater detail.

If, How, and Where to Validate

This is a fundamental choice, and you'll benefit by making it as early as possible. We'll assume in this discussion that you're going to use a standard DOM or SAX API that offers validation against W3C schemas. A major choice you need to make is whether or not you are going to have the API validate documents against a schema. This is primarily of concern for documents that your application will consume, but it may also relate to documents that your application will produce. A related decision is the choice of schema, which we'll talk about shortly; you can pick a standard (if you can find an appropriate one) or you can create your own. If you do create your own, you have flexibility about how strictly you write it.

Schema validation has its tradeoffs just like any other design decision. Here are a few.

  • Schema validation may incur performance penalties. We may be concerned about CPU and memory usage on a loaded system, delays in elapsed time, or both. Both can be considerations if you are validating against large schemas, especially if they are spread over several different files. Time delays due to network latency can be a factor if the schemas are standard schemas that must be retrieved from some organization's server on the Internet, though various schema caching mechanisms may ease this. The overall performance penalty may or may not be very great, but I know of at least a few organizations that don't use validation because of performance concerns. A compromise alternative to never validating might be to validate during testing but then to not validate after moving to production if schema validation errors have been eliminated from the normally anticipated production data.

  • The choice to forego validating can make writing code harder. Particularly if you are using the DOM, being able to make reliable assumptions about the structure of a document you are reading can make writing the code much easier. If you aren't able to assume that an Element will be present, you must test for null pointer returns from methods such as get ElementsByTagName or nextSibling before trying to retrieve an Attribute or child Elements from the Element. Users don't like null pointer exceptions. Also, in general terms the code can be trickier to write if you can't be sure that the structure of the document is what you expect. If you get a document that is well formed but has subtrees in the wrong places, what is your application going to do?

  • Schema validation is either on or off; there is no in-between. Some EDI management systems offer the ability to turn on or off the validation of coded values against a code list while still validating for required segments and data elements. It would be nice if the XML APIs offered strict or lax validation and various flavors of validation. However, the XML Schema Recommendation doesn't define levels of validation so the APIs don't either. Most of them stop cold at the first invalid component, no matter how seemingly trivial the noncompliance might be.

  • Schema validation removes the ability to correct errors from within the application (generally). Let me give you an example: Some systems that import sales orders received by EDI hold them in a review state, allowing a clerk to review them and correct for errors or missing data before accepting them into the system. If your application has a similar feature and you validate in the schema for an invalid item number, the data may never reach such a review state. (One way around this limitation would be to load the document without validation and somehow mark it as invalid.)

  • Schema validation removes many validation burdens from application code. This is perhaps one of the greatest benefits to validating against schemas. You let the API validate that your business rules are satisfied, and you don't have to write code for your application to do validation.

  • There are limits to schema validation. Schema validation is very good for determining the presence or absence of data and for ensuring the data type and contents of individual items of data. However, by itself it won't allow you to enforce constraints based on conditional relationships between different data items. Specifically, it won't flag one Element or Attribute as being invalid based on the contents of a different Element or Attribute. Some types of processing are rife with these types of requirements. Let's take an example in health care. A health care claim might have a coded field for the type of medical procedure performed. Some types of procedures might require information in other Elements that isn't relevant to other procedures. A tooth extraction would require that the tooth be identified, whereas fitting a bite guard would not. Some readers may be thinking that this is stupid; problems like this could easily be avoided by modeling the data correctly. You're probably right, but the fact of the matter is that data is not always modeled to prevent the need for this type of validation. It is a very real concern for some applications.

Beyond these considerations, if you do validate you must determine where your schemas are going to be stored. This decision in itself may be relevant to whether or not you validate. For example, if your application is a desktop bookkeeping system that will process standard purchase orders received from customers, you may not want to enforce validation if a significant number of your users rely on dial-up Internet connections. You can't validate if you can't get to the schema. If your bookkeeping system requires that imported orders be validated and if you don't validate, you don't process the order. If you validate against a remote schema, how reliable is the server that hosts the schema? If you provide your own schemas, like any other application component you must decide on a directory location. You must also take steps to ensure that schemas won't be modified or inadvertently deleted by users.

I have one final word on schema validation. Just because a document is schema valid doesn't mean in and of itself that the document is what you expect. The default behavior of most APIs is to validate against the schema referenced in the document's root Element. That schema could be the one you coded for, or it could be something completely unrelated. You could be processing a valid shipment notice when you were expecting a valid invoice. So, to properly use schema validation in your architecture, you'll need to do things beyond just validating against a schema. The first step is to check the root Element name. We routinely did this in several of the book's routines. The other thing is to retrieve and check the document namespace and the schema location Attributes from the document's root Element. There is no way you can be absolutely sure of the document you're processing unless you can match the schema location URI to a known schema. If the document uses a named namespace, verifying that it matches what you expect provides another level of assurance. (Add that to your list of reasons for using namespaces.) I haven't verified namespace names in the book's utilities, but it is something you should consider doing in your applications.


Selecting the XML Format

Use a Standard or Roll Your Own?

There are always benefits to not reinventing the wheel, and this rule holds true when considering schemas for common business documents. However, though a sad prospect to consider, the wheel you're looking for may not have been invented yet. This is one area where I will offer some somewhat unqualified guidance. If you can find a schema that fits your purposes and has acceptance in your community as a standard, then by all means use it. If you can't find one that fits or if you find more than one and there is no consensus in your trading community or user base about which one is "standard," then you have at least a few different ways you can go. If one or more schemas might work but aren't universally accepted, there are a few criteria you could use to select among them. One criterion might be the number and quality of other programs, stylesheets, sample documents, and documentation available for the schemas. Another might be whether or not you can use the schemas on a royalty-free basis. If you do find an existing schema that is attractive from a number of these perspectives yet doesn't fully accommodate your data, it may still be a good fit for you if you can easily customize or extend it. However, if all else fails, don't have any hesitancy about developing your own schema.

Choosing or creating an XML format is an important decision, but your particular choice is probably not going to be extremely critical. If people don't like the formats you have selected, they can always perform transformations (particularly if you make it easy for them, as we'll discuss later in the chapter). Go back and review the first part of Chapter 10 on XSLT if you like; transformations are something we are going to live with for several years to come, if not forever. The rest of this section deals with general issues regarding designing your formats.

General Document Design Decisions

In Chapter 4 we discussed various issues regarding the design of XML documents that are independent of their schema representation. Here are a few of them again, from the perspective of designing your own documents.

  • Naming conventions: The most pragmatic advice I can give you is to use names that will be familiar to users of your application. If a field is called Purchase Order Number on the screen or a printed report, don't give it an Element name of CustomerOrderIdentifier. Issues such as use of upper and lower case, abbreviations, and word separators are matters of style. Choose one you like, then stick to it once you've chosen.

  • Elements and Attributes: This one again is kind of a religious issue. If you have defined criteria for deciding whether or not an item of data should be an Element or Attribute, and if you can apply those criteria on a consistent basis, then by all means use a mix of Elements and Attributes if you wish. If you can't be consistent, your users will thank you if you stick with Elements only.

  • Structure: Should you go for a relational, nested, or some other type of structure? XML lends itself most to a nested, hierarchical structure for organizing and grouping data. Most people follow this style, but there are some exceptions. Again, your users will thank you if you make a firm design decision and apply it consistently.

  • Reuse of existing logical formats: If you currently provide a facility to import a purchase order in a flat file, using an XML document of the same logical structure may allow you to use similar processing logic when importing an XML purchase order.

Again, this list is not exhaustive. These are some of the major decisions you'll have to make, but I'm sure there will be others. The next issue has more to do with the data that you include in a document than a particular set of design choices.

Providing Identifying Information

This is probably more of a concern for electronic commerce applications than for application integration, but it may be an issue there too depending on the situation. In the EDI world, most EDI management systems rely on one or a few specific fields in an outbound document to look up the EDI-related details about the trading partner. This is usually a customer or vendor number and is used as a key to the trading partner setup in the EDI system.

Although the world of e-commerce using XML is still evolving, things may be slightly different in that world than they have been in the EDI world. There's a tendency among utilities that move XML around to just consider documents as payload and not look inside them. This is unlike EDI management systems that must examine the application data in order to transform it. The strategy you will have to follow will depend on the particular methods or systems used for data transport. The bottom line is that you may have requirements for providing identifying information within a document or by some external means such as specific file names, locations, or key values that are passed in method calls.

Schema Design

Schema design is a very broad and complex topic. As much as I would like to offer you my knowledge and opinions about it, I'm afraid it would qualify for another complete chapter, if not another book. I have to set an appropriate scope somewhere for a book that has already turned out longer than planned, so I'm going to point you elsewhere for details about schema design. There are several good resources listed at the end of the chapter. Beyond that I'll offer only a few general observations.

Despite what some authorities may tell you, no single technique is right for all circumstances. If you're going to design only one or a few schemas, probably any approach that provides the required validation will be adequate. This includes letting an IDE like XMLSPY or TurboXML generate a schema from a representative instance document as I discussed in Chapter 6. On the other end of the spectrum, if you are designing several documents for a fairly large system, you would be well served to take a more disciplined approach. I favor creating type library schemas containing simple and complex types that are reused in other schemas, similar to the approach discussed in Chapter 4 that was considered by X12 and OASIS. However, there are certainly other techniques.


Changing Your Code

At some point when building in XML support at the code level, you have to consider the actual code. That's what we'll do in this section.

Input and Import

If you have existing validation routines for import and data entry, reusing them for XML import will save you a lot of work. As I said earlier, there are limits to schema validation. If your application needs to validate order item numbers by looking them up in a database, you'll need to call this code when you import XML data. A well-designed system might use the same modules for validating an imported purchase order that it does for one entered by a human operator. If your system isn't coded this way, building in XML support might be a good opportunity for centralizing and cleaning up your validation logic.

The specific changes you need to make in your code depend on how you import data from the user's perspective. Do you provide a batch import from a command line, or is it driven by a user interface where the user selects files to import? If by batch it may make sense to build a completely separate program to import XML. If it's by a user interface, the determining design factor will probably be the degree to which the logic for reading the source, validating it, and storing it is segregated and modular. As you could see from the code examples, the logic for loading an XML document and retrieving data from it can be fairly involved. If your code for reading, validating, and storing is already mixed together in one module, adding XML calls is only going to make the code more complicated and less maintainable. You might be better served to code a completely separate module for XML import. Or you can again take advantage of the opportunity to redesign your code and clean it up a bit.

If the functions are already nicely segregated and the code is modular, you can probably code a specific XML import routine to load the data into the same data structures used when reading from other sources. You can then pass these data structures off to existing validation and storage routines and be done with the job.

Printed Output and Export

Not every application offers the ability to import data by means other than a user keying it. However, nearly every application has the ability to create printed documents, even if it doesn't provide features for exporting data files. Certain steps always have to be performed. Certain aspects aren't that much different for printing, exporting flat files, or exporting XML documents.

The key thing to bear in mind when coding your XML export as opposed to a flat file export is that we generally export several logical documents to one physical flat file. For XML we must export each logical document to a separate physical file. A well-formed XML document can have only a single top-level Element, the document root Element. Writing two documents to the same file violates this constraint.

We can still model our XML export on an existing extract or print routine. The example below shows typical approaches at a high level. In this simplified example of an invoice export, we'll assume that we're extracting data from two relational database tables, one for the overall invoice information and the other for item details. We're going to write the invoices to disk files, though they might be handed off to communications modules as DOM Document objects. Here's the logic for a typical flat file extract routine.

Flat File Extract Routine
Arguments:
  Selection criteria, e.g., invoice numbers
  Output file name

Open output file
Build and open cursor for selecting invoice headers
Fetch first invoice header row
DO for each invoice header
  Format data for export
  Write record to output file
  Open secondary cursor for line item details
  Fetch first line item detail row
  DO For each line item detail row
    Format data for export
    Write record to output file
  ENDDO
  Close secondary cursor
ENDDO
Close main cursor
Close output file
Return

The XML export routine follows the same general logic for extracting the data from the database but differs in some key areas. We'll again assume a DOM API. Other models for processing XML will differ in a few details, but the overall flow will be identical.

XML Export Routine
Arguments:
  Selection criteria, e.g., invoice numbers
  Output directory specification

Set up API for creating XML document
Build and open cursor for selecting invoice headers
Fetch first invoice header row
DO for each invoice header
  Create new XML document
  Create root Element and attach to document
  Build output filename
  Create invoice header element and attach to root
  DO for each column to be added to document
    Create Element and attach to invoice header
    Create Text node and attach to column Element
  ENDDO
  Open secondary cursor for line item details
  Fetch first line item detail row
  DO for each line item detail row
    Create line item Element and attach to root Element
    DO for each column in line item row to be added to document
      Create Element and attach to line item Element
      Create Text node and attach to column Element
    ENDDO
  Close secondary cursor
  ENDDO
  Save DOM document to file
ENDDO
Close main cursor
Return

There is an important difference, aside from the obvious details of formatting and writing a flat file record as opposed to creating Elements. Rather than opening the output file at the beginning of the routine and closing it at the end, for each iteration of the DO loop that processes an invoice header row we create and save a new XML document.

Of course, every application will be somewhat different, but this example illustrates some of the major issues to consider.

Adding XSLT

You will be doing your users a great favor if you accept the fact that they won't be dealing with just one XML format for each type of document. Assume that transformation is going to be the norm, and make it easy for users. One way you can do this from a functional perspective is to build in support for XSLT. This could be done by giving users the ability to associate a stylesheet with an XML export or import. It will probably be most useful if the association is done on a very specific basis. For example, for an order management and billing system, make the association at the customer level and for each specific type of document. In this fashion the appropriate XSLT transformation could be performed for each document from that customer before you import it and on each document for that customer after you export it.

Unlike the DOM, there is no universal, language-independent standard API for XSLT processors. However, both Xalan and MSXML have support for invoking transformations on in-memory documents. Using this strategy you could load, parse, and validate a document; transform it if necessary into your expected format; validate again if desired; and then process it. For export you could create the output document, validate it if desired, transform it into the expected format, validate again if desired, and then serialize it to disk or hand it off as a DOM object to an API that will transmit it to the destination.

For programmers working in the Java world, Xalan provides code-level XSLT support primarily through some classes first defined in JAXP 1.1. The relevant Java package is javax.xml.transform. Within that package the most important classes are the Transformer class with its transform method and the TransformerFactory that creates Transformer instances. The model is somewhat similar to the DocumentBuilderFactory that builds DocumentBuilder instances. If using the DOM API, a DOMSource object is created from a DOM Document (or other Node) and passed to the transform method. This returns a DOMResult object from which a DOM Document object can be retrieved.

The model in MSXML is a bit simpler. XSLT support is built into the DOM (non standard, to be sure, but certainly convenient). The method we're probably most interested in is the transformNodeToObject method. For our primary interest in transforming complete documents, it takes a DOM Node representing a stylesheet as an argument and returns a COM Variant output object, of which one of the variations can be a DOM Document. The online documentation provided with MSXML has a pretty good C++ example for how to call this method.

Another way you can make transformations easier for your users is to provide some sample XSLT stylesheets for converting to and from your selected formats and various other formats. These other source and target formats might include things such as the OASIS UBL standards, the Open Applications Group's OAGIS standards, standards from UN/CEFACT or X12 (when they become available), and various vendor formats.[*] For any particular usage scenario your users would probably have to modify the sample stylesheets a bit, but giving them something to start with would be a big help.

[*] OASIS, Organization for the Advancement of Structured Information Systems; UBL, Universal Business Language; OAGIS, Open Applications Group Integration Specification; UN/CEFACT, United Nations Centre for Trade Facilitation and Electronic Business.

Making XML the Native Output Format

If you've taken the trouble to go this far and add an XML export function, one thing you could consider is just using XML as the output format. Provide XSL stylesheets with the formatting objects appropriate for various types of output and you have succeeded in the often sought after goal of separating presentation from content. Once you have created the XML document your users can do with it what they like. And if they don't like your default printed formats, with a reasonable stylesheet graphical user interface they can create their own. This has to be much easier than building custom formatting routines into your application. Stylesheets for printing could be associated with specific documents and customers just as stylesheets for exporting are.

Making XML a Native Storage Format

Even with some "XML databases" popping up here and there, I doubt if XML will ever replace the relational databases used for most business applications. The relational model with SQL and ODBC is just too well known, proven by time, and well entrenched. If it isn't broken, why fix it? However, applications use several types of data (for example, configuration files) that aren't necessarily stored in relational databases even if the bulk of application data is. The utilities in this book use XML for such a purpose. Using XML in these situations can often be easier than writing code in native Java or C++. This is especially true if you want to make sure that the configuration information is valid before loading it.


What about Databases?

I feel obliged to say something about using XML with databases (probably for no reason other than one of my editors suggested it to me!). However, in a book that I've tried to make general and not specific to any product it's awfully hard to say very much about database support for XML. The latest versions of most major relational database management systems (RDBMSs) have some degree of XML support. Some of it is fairly sophisticated. The hitch is that there isn't a standard way to support XML in an RDBMS. Given that state of affairs, I offer two pieces of advice. The first is that you should review the features of your particular product and see if it offers the functionality you need. The second is to be aware that if you use your product's XML features, you may be locking yourself into a specific vendor.

Beyond those two main bits of advice there are other issues to consider. You may be running an old version of the database management system that doesn't provide all this nifty XML support. If your application was developed with this old version, there may very well be issues in migrating the database to the new version without changing, or at least rebuilding, the application. Beyond that, there are always licensing issues. Despite the fact that software vendors would like us to believe that everyone buys support contracts, a lot of companies don't. It sometimes costs a good deal of money to upgrade to the next version of a database management system even if you do have a support contract. This can be an issue particularly if you are an applications vendor. Changes in the underlying database engine can affect a very wide group of users.


Other Approaches and APIs

I've used the DOM for all the XML processing in the Java and C++ code for this book. If you've followed me up to this point I think you can probably understand why that was an appropriate choice. However, it may not be the most appropriate choice for all circumstances, given your own particular situation and requirements. I said in Chapter 1 that there were other approaches. It's time to talk about them a bit more.

Simple API for XML

SAX isn't a "real" standard in that it hasn't been blessed by any standards body, but the SAX implementations probably follow SAX no more or less than people's DOM implementations conform to the W3C Recommendation. Again, SAX is based on an event-driven model that uses callbacks for each type of XML construct it encounters in an input stream. Unlike the DOM, SAX just reads a data stream in serially and triggers events. It doesn't build a tree (or anything, for that matter) in memory. Thus it can be quite appropriate if you have to process very large XML documents that would consume considerable memory if processed using the DOM.

Several API libraries for Java and C++ support SAX, so it is widely available. However, a major downside with SAX, as I said in Chapter 1, is that there isn't a standard way to create documents using SAX. The Apache Xerces distribution has SAX-based classes for creating XML output. So does MSXML. However, they're different.

Still, the choice of SAX over DOM is subject to several nonfunctional considerations, not the least of which is the programming paradigm. Are you more comfortable with event-based programming, or do you prefer working with trees? Take your pick.

Generated Class Bindings

An increasing number of products and freely available tools will generate Java or C++ classes for you if given a schema for input. Here are a few of those products.

  • The Castor project, described as "an open source data binding framework for Java" and "a mapping framework between Java objects, XML documents, SQL & OQL databases and LDAP directories." For details, visit www.exolab.org.

  • The Java Architecture for XML Binding (JAXB), which "provides an API and tools that automate the mapping between XML documents and Java objects." JAXB is part of the Java Web Services Developer Pack. More information and free downloads are available at http://java.sun.com/xml/jaxb.

  • XMLSPY, Version 5, Enterprise Edition, a proprietary offering in this space. According to the vendor, this product "includes a built-in Code generator which can generate program code bindings of XML Schema components in Java, C++, or Microsoft C#." You can find more information at www.xmlspy.com.

The promise of tools like this is that if you give them a schema, they'll generate all the code necessary to let you access an XML document just like you would any other C++ or Java object. There is no complicated DOM, SAX, or other lower-level XML-specific code to write. This solution may be superior to DOM programming for many situations and is probably worth your consideration. However, despite all its benefits we need to keep in mind some of the potential drawbacks.

  • Cost is one drawback. For tools like Castor and JAXB, cost isn't much of a consideration (though using open source software itself might be an issue for some organizations). For a tool like the Enterprise Edition of XMLSPY, cost could be a consideration.

  • There isn't a standard for generating Java or C++ classes from schemas, but if you think about it a standard probably isn't applicable.

  • You don't want to modify the generated code. If the schema is changed you'll have to feed it to the tool again and generate new code, wiping out your changes.

  • Tools may favor some styles of schema design over others. I've not yet worked with any of these tools. However, intuitively and from discussions with colleagues who have, it seems that the specific code that gets generated may be dependent on the particular approach to schema design. Different schema features may yield different depictions in code. The tools may steer you toward certain schema design styles in order to optimize the generated code. The schema styles may or may not be desirable when considered from perspectives other than code generation. For example, a schema design style that yields very nice Java classes may not necessarily be the most understandable from data analysis or reusability perspectives. In addition, when you are working with schemas that others have created, the generated code may not be the most desirable from a programming language perspective. However, you may have no way to modify what gets generated.

  • Most of the tools I'm aware of make it very easy to use XML documents as input, but they may or may not do much for you when you need to serialize a document to disk. Pay careful attention to the features.

  • Determine which APIs the tool uses in the generated code. If it uses a standard API like Xerces or MSXML in the generated code you're probably pretty safe. If instead it calls proprietary APIs, do thorough testing. It may be too much of a black box.

  • If you need to perform schema validation, check how well the tool complies with the W3C XML Schema Recommendation.

  • The generated code may or may not be very efficient.

  • In the worst case, the generated code may not be bug free with all inputs.

I've been around long enough to remember some early code generation products and to remember that they never caught on despite the promised benefits. Do a thorough assessment. The tools may make processing small, simple documents very easy. However, for larger, more complex documents with many optional Elements and Attributes, most of your program logic may deal more with content than with the particular APIs. A code generator may or may not save you significant effort over DOM programming.

I'm sure that 40 years ago similar concerns were raised by old assembly language programmers warning about the drawbacks of third-generation languages like COBOL and FORTRAN. So, call me an old (or new) fuddy-duddy if you like. The best advice I can offer is to do a thorough evaluation, including testing with a wide variety of inputs, before you commit to a particular tool. Despite the potential drawbacks, I do need to say that these tools get one thing right. They start with the data model.

Options for Procedural Languages

It's not that you don't have any options with strictly procedural languages like C, COBOL, and FORTRAN. It's just that your options are somewhat limited and nonstandard. I'll discuss three basic options here.

The first option involves linking routines in an XML supporting object-oriented language, probably C++, with your application. Digital Equipment (later Compaq and now part of Hewlett Packard) as early as 1978 supported a calling and linking standard that allowed modules written in any language on the VMS operating system to call modules in any other language. Even today not all operating systems and development environments offer this support or offer it as transparently. However, many do. If you happen to be fortunate enough to have this option, it's probably the easiest route to adding XML support to your application. Develop all your XML handling modules in C++, design so that you're sure you can pass the data back and forth, and you have the job done.

The second option is specialized API libraries, software packages, or compilers that provide XML APIs directly to these procedural languages. Several open source and proprietary alternatives are available. Here are a few examples.

  • LT XML for C, open source: www.ltg.ed.ac.uk/software/xml/

  • Libxml in C, XML C library for the Gnome project, open source: http://xmlsoft.org

  • expat, James Clark's XML Parser Toolkit in C, open source: www.jclark.com/xml/expat.html, with version 2.0 work being done at http://expat.sourceforge.net

  • XML4cobol for COBOL, proprietary: www.xml4cobol.com

  • Fujitsu NetCOBOL compiler, proprietary: www.netcobol.com

APIs like these can certainly provide native XML support to an existing application. However, there are several issues to consider. As I said earlier, they generally don't provide native support for standard APIs like SAX and the DOM. Another issue is whether or not they support schema validation. Some older tools may not even support XML namespaces. Find out! Other issues to consider are the same as for any other development tool. The cost of the tool, its quality, its support, and the stability of the vendor (or the breadth and depth of the open source community) are usually of the highest importance.

The third option to consider is reassessing whether or not you really need native XML support. If your application is coded entirely in C, COBOL, or FORTRAN, you probably still do a lot of processing in batch mode. Unless you have particular requirements for real-time behavior (perhaps something similar to a CICS[*] transaction processing monitor), consider auxiliary stand-alone conversion utilities like those developed in this book.

[*] Customer Information Control System (CICS) is IBM's venerable mainframe-based transaction processing control system.


Non-XML Issues

Nearly everything considered in this chapter related specifically to XML. However, you may be considering building XML support into your application because of increasing demands for integration with other applications or the need to exchange common business documents electronically with trading partners. Requirements like these come with a few issues that have nothing to do with XML specifically but that you may want to consider.

Looking Up and Converting Coded Values

In Chapter 10 we talked about looking up UPC numbers used in purchase orders from customers and converting them to the item numbers used in our own systems. This is just one example of a common problem. Chapter 9 hinted at another potential problem in the discussion of the X12 purchase order example. A common practice in retail and grocery orders is to send an identifier, such as a DUNS number with a four-digit location suffix, as a ship to location rather than sending the full name and physical address. It is common for EDI management systems to allow users to define their own lookup and cross-reference tables for such purposes. However, the situation might be easier for users to deal with if lookup and cross-reference features were included in their business applications. I have seen such features in high-end enterprise resource planning (ERP) systems but have not seen them frequently in lower-end systems.

User-Defined Fields

In addition to lookup and cross-reference tables, there may be data that, while not cross-referenced or converted, is required by trading partners. One example is a customer-assigned vendor number that might be required in all invoices and shipment notices sent to that customer. The vendor number could certainly be hard-coded into the appropriate XSLT stylesheets. However, it would be much more user friendly if there were a user-defined field where it could be maintained in the application's customer information subsystem. Similar information might need to be stored for each ship to location and each line item of a purchase order. For example, a UPC number might be the primary item identifier used to cross-reference against the item numbers used internally. But there may be one or more secondary identifiers that need to be captured and sent back to customers on invoices and purchase orders. I know it sounds like overkill, but it is common practice.

There are examples beyond just the traditional procurement cycle. State and federal agencies may assign organizations identifiers for certain types of regulatory compliance. If compliance data needs to be reported electronically, having fields for the identifiers in your own company record would be much easier than hard-coding them in your XSLT stylesheets.

Business Process Support

This is another area that goes beyond the scope of this book, but you need to be aware of this issue. The book has dealt with data and technical approaches for processing data at a fairly detailed level. Business process support has to do with the overall context in which the data is exchanged. This is most often a factor when dealing with business processes that cross organizational boundaries but may be a factor with internal systems that manage work flow.

In the cross-organizational sector, several groups such as RosettaNet, the Uniform Code Council in cooperation with the European Article Numbering Association (UCC/EAN), and UN/CEFACT have or are beginning to develop models that describe common business processes such as procurement, fulfillment, and payment. These models describe a series of actions that might together constitute a process. They define not only the data that might be exchanged in each step but also the conditions that must be satisfied to move from one step to the next. For example, a purchase order might be exchanged in one step of a procurement process. The process might then specify the conditions under which the order is accepted or rejected. The next step might specify that a purchase order acknowledgment be returned with specific information regarding acceptance or rejection. The model might specify different succeeding steps depending on the order status. These models can be very comprehensive, with some even considering what constitutes a legally binding contract.

It is difficult at this time to know whether or not these efforts will have an impact on requirements for business applications, and if so how great the impact will be. In some scenarios the impact would be considerable. In the case of ebXML (Electronic Business XML, www.ebxml.org), full support for the framework requires that applications be configurable using business process information read from a special type of XML document. Applications may also need to maintain certain information about the state of business processes that they don't currently track.

My own opinion is that many of these initiatives are premature. They are too ambitious not only in terms of the ability of application vendors to absorb the requisite technology but also in terms of the ability of organizations to adopt the processes. If these initiatives gain significant following, my best assessment is that it will still be several years before they do have significant impact on most business applications. It would, however, be prudent to at least monitor them.


Resources

Code-Level XSLT Integration

Detailed documentation about invoking XSLT transformations from application code is available from the following resources.

javax.xml.transform package documentation in the Java API for XML Processing Specification 1.2, by Sun Microsystems. Online Javadoc documentation, included as part of the Java XML Pack Spring 2002. Available online at http://java.sun.com/xml.

transformNodeToObject method documentation in Microsoft XML Core Services (MSXML) 4.0, by Microsoft Corporation. Online documentation included as part of the MSXML SDK. Available online at http://msdn.microsoft.com/xml.

Databases and XML

Using XML with relational databases is discussed in depth in the following resource.

XML and SQL—Developing Web Applications, by Daniel K. Appelquist (Boston, MA: Addison-Wesley, 2002).

Schema Design

There are several very good resources for schema design. The list below shows a few I recommend.

"ASC X12 Reference Model for XML Design," Type 2 Technical Report, by the ANSI Accredited Standards Committee X12. Available online at www.x12.org. While focusing primarily on a semantic architecture for X12's work on XML, this report discusses several syntax-level issues and recommendations regarding document and schema design.

"UBL Naming and Design Rules," by the Naming and Design Rules subcommittee of OASIS's Universal Business Language Technical Committee. Available online at http://oasis-open.org/committees/ubl/ndrsc/#documents. This is a very good framework for document and schema design. The specification itself generally just defines the approach and doesn't discuss the advantages and disadvantages of various approaches. However, various working and position papers on the Web site may discuss background, issues, and alternative approaches.

"XML Schemas: Best Practices," developed collaboratively by The MITRE Corporation and members of the xml-dev list group, hosted at Roger Costello's xFront site. Available online at www.xfront.com/BestPracticesHomepage.html. You can't go wrong starting here.

"XML Technical Specification for Higher Education," by The XML Forum for Education of the Postsecondary Electronic Standards Council. Available online at www.pescxml.org/. This document offers the perspective of a group in one particular industry sector regarding document and schema design practices. It includes discussion of tradeoffs and several examples.


Chapter 13. Security, Transport, Packaging, and Other Issues

Even paranoids have real enemies.

I chose this particular popular observation to start the chapter because I think we need to get some perspective on security. I tend to think that a lot of security worries are way out of proportion, but there are some concerns that you ignore at your peril. You need to take everyone's opinions about security—including my opinions—with a few grains of salt. That said, this chapter discusses general issues related to exchanging XML documents with other organizations over the Internet.


Some General Observations about Security

It isn't enough just to create or process XML documents; you have to exchange them with other applications. When those applications are under the control of other organizations, we've moved into the area of what is broadly called "electronic commerce." Since that's the primary focus of my consulting practice, I can't quite close this book without saying a bit about it. However, I won't say very much. Most of the end user target audience for this book, and the people who develop applications for them, aren't given very many choices. They do what large customers or government agencies require. I would like to help you understand why those organizations are telling you to do certain things and, if they do give you any options, to help you make those choices.

At the bottom line, all the choices we have to make about moving documents over the Internet depend on security. So, to set some perspective, here are a few observations and facts about security.

  • It is easier for a store clerk, waitress, or someone who takes your catalog order by phone to steal your credit card number than it is for someone to pluck it off the Internet while in transit.

  • It is easier for someone to grab your credit card number from your mailbox or your trash than it is to pluck it off the Internet while in transit. (How many of you use locked mailboxes and shred your trash?)

  • It is easier for a hacker to break into a system where your credit information is stored than it is to pluck it off the Internet in transit.

  • In over ten years of working with EDI, I've never heard of a fraudulent EDI transmission of business data—no bogus invoices, no orders to be shipped to suspicious locations. Granted, it is harder to break into the third-party value-added networks used for EDI than it is to hack the Internet. However, without giving away any hints, it's not impossible.

  • The most popular e-mail programs offer support for X.509 digital certificates that support encryption and digital signatures. With all the sensitive e-mails that I exchange and agreements that are confirmed by e-mail, only two of my regular correspondents ever use digital certificates for e-mail.

Are people overly paranoid about security? Probably so, but don't try using that as a defense when you're prosecuted for not securing the confidentiality of personal information about patients or students. And if you try telling that to the folks at Wal-Mart, they'll probably drop you from their vendor list.


Dealing with Security

In this chapter we'll deal primarily with security of data while in transit. Security of data within your enterprise is certainly very important, but it can involve fundamentally different problems. The scope is different and the mechanisms to enforce security can be different. Referring to one of my observations above, you want to make sure that your system makes it very difficult for both internal and external hackers to steal sensitive information such as credit card data. The mechanisms for ensuring this type of security often involve firewalls, operating system and file system protections, database security, and individual user authentication. As we'll see, these can be very different from the mechanisms commonly used to secure data while in transit. In this chapter we'll focus on what happens between the firewalls, not on what happens within the firewall or how to keep the firewall from being penetrated.

Security Requirements and Exposure

What are we really concerned about when we talk about security? Independent of any technology, including computers, almost everyone agrees that there are certain common security requirements. However, they may not agree on what the specific requirements are. For some guidance about specific requirements we can look to the Information Security Technology Evaluation Criteria recommendation [ITSEC 1991]. It breaks security down into three broad requirements.

  1. Confidentiality: Certain types of information must be restricted to only those people who need to know it. Cryptology and encryption first come to mind for most people when they think about computer security.

  2. Integrity: Resources must be secure from malicious tampering that would compromise their usefulness. Transactions must be free from fraud.

  3. Availability: The resource that we're trying to secure must be available to those who require its use. In computer networks, denial-of-service attacks violate this aspect of security by flooding Web sites with so many bogus requests that they aren't able to service legitimate users. In the physical world, we want our car to be there, in the parking lot, when we're ready to go home.

In conducting electronic commerce we're usually most concerned with the integrity of transactions and next concerned with confidentiality. That isn't to say that availability isn't an issue, but it is such a general issue that it isn't very often a specific concern of electronic commerce (at least the type of business-to-business commerce discussed in this book).

When we speak of risks of any type, we need to put them into perspective not only in regard to what we stand to lose if the worst happens (that is, our exposure) but also in regard to the likelihood that the worst will happen. In common electronic commerce situations we're probably most concerned about fraud or violation of transaction integrity. We could be concerned about processing orders from bogus businesses from which we'll never receive payment or about paying bogus invoices for goods we never received. Some organizations are also concerned about trading partners that deny an electronic transaction took place or dispute the details of a transaction, but this is less common. What is usually at stake in these cases is a specific dollar amount. If we take an objective look at all these things, they really don't happen very much. While the exposure may be moderate, the likelihood is low.

We're also frequently concerned with confidential information getting into the wrong hands. We don't want competitors to know the details of products in development. We may not even want other organizations to know what we buy or sell and the prices involved. However, if you listen to the concerns most voiced by users, fraud is mentioned more than confidentiality.

Countermeasures and Remediation Strategies

There's a class of actions, normally referred to as countermeasures, that are taken in response to security threats. You can think of burglar bars on your windows or shredding your credit card statements as countermeasures. Remediation strategies apply to what you do after your security has been compromised. If we look at attacks on security, what you can do about them is generally determined by the timing of the attack.

  • Prevention: Preventing or deterring the attack before it occurs. We buy a home alarm system so that the burglar will hit the house next door rather than ours. We reinforce our door locks to make it harder for thieves to break down the door.

  • Detection: Finding out about the attack as or shortly after it occurs. Our home alarm system automatically dials the security company if a door or window is opened.

  • Remediation: Choosing what to do after the attack has happened. We assess whether anything has been burgled from our home. If something is missing, we file an insurance claim. We take what we've learned about our vulnerability and apply it as prevention for the next attack.

In all honesty, it seems to me that there has been way too much focus on preventing security breaches that have moderate exposure and relatively low risk of occurring. Resources might be better spent on reasonable detection and remediation. In addition, way too much money is spent on some types of prevention strategies. What is the point of requiring small trading partners to use strong algorithms for encryption and digital signatures if the machines they use to communicate with you are in open offices, left on and logged in 24 hours a day?

The best you can do (if you are given a choice) is to assess what your real exposures and risks are, and then try to strike a reasonable, cost-efficient balance between prevention, detection, and remediation. However, since most of the requirements imposed on you have to do with prevention countermeasures, let's look at some of the options you might be offered.

Is It a Requirement or Is It a Countermeasure?

For those of us who have spent much time studying nonfunctional requirements, it comes as no surprise that there isn't universal agreement about how requirements like security are decomposed. That is why I chose a somewhat authoritative source like ITSEC when I broke down security into confidentiality, integrity, and availability. You may not be surprised, therefore, to know that there is also frequent disagreement about what constitutes a requirement versus a design option or mechanism that satisfies the requirement. For example, many people might regard a password-protected login as a requirement. Others might regard it more strictly as an authentication mechanism for enforcing access control, which ultimately has to do with satisfying the requirements of confidentiality and integrity. In this book I'm not going to split hairs with this type of theoretical debate. To set a frame of reference, I'll try to follow the framework and approach defined in ITSEC. You don't have to agree with it, but it at least will lend some consistency to how we consider the issues.

Prevention Countermeasures

The two countermeasures you are most likely to implement have to do with authentication and encryption.

Authentication

Authentication is the means by which someone can verify that you really are who you say you are. Organizations use authentication to ensure that it is indeed you who has logged into their portal, or it is really your company that has sent them an XML document. Authentication can either be session based, lasting the duration of a communications session, or persistent. Session-based authentication is still most often implemented with user names and passwords. Unless you have significant concerns about someone masquerading as you in order to commit a fraudulent transaction, take this option if it's offered to you. An increasing number of organizations are using X.509 digital certificates with public/private key pairs for session authentication. Certificates are getting much easier to acquire than they used to be a few years ago, and software support for them has matured as well. Session-based security using a certificate at sign-on or session initiation is still a fairly low-cost choice. Many Web browsers support it.

Persistent authentication usually involves placing a so-called "digital signature" on a document. The signature is a string of bits generated by an algorithm that uses your private key and certain information about the document. It can be used to verify not only that the document came from you but also that it hasn't been tampered with. Persistent authentication, as yet, is seldom required in most common exchanges of business documents. We could certainly imagine the day when we might want to apply a digital signature to a legal contract. However, most companies at present don't require this level of authentication for common documents like purchase orders or invoices. Persistent authentication often requires more expensive software and sometimes more complicated management than session-based authentication. If you have a choice between persistent authentication and session-based authentication, again depending on how you view your own requirements, go for the session-based approach since it is generally easier to implement.

Encryption

This is the familiar scrambling of bits that most people first think about when it comes to computer security. What many people don't realize is that encryption, like authentication, can also be session based or persistent. The same X.509 certificates can be used for public/private key pairs. However, network protocols support session-based encryption in which the keys are negotiated dynamically and certificates are not needed. This is how most credit card transactions are conducted when people place orders from Web sites.

With one exception, session-based encryption is sufficient for most common exchanges of business documents. Once the data is behind the firewall, typical operating system, file system, or database access control mechanisms are usually sufficient for ensuring confidentiality. The exception when session-based encryption is not applicable is when using transport approaches like e-mail where there isn't an end-to-end session, as we'll discuss below.

If You're Given Choices

The simplest and lowest-cost strategy for both authentication and security is to log into a Web site using a user name and password and use the browser's built-in support for session security. This method is probably perfectly adequate for most common business document exchanges. If whomever you are trading documents with demands that you use X.509 certificates for this process, it still isn't too bad since most browsers also support them.

The other option is persistent authentication and security. You must have your own public/private key pair for these, and you most likely will need an X.509 certificate. You will also most likely need special purpose software. Your trading partner will probably give you some recommendations.


Transport

As we'll see, the specific method we use to transmit documents largely depends on the required level of security. On the other hand, the choice of transport can restrict the security options. However, the choices are often made with security as the primary consideration. Here are the most common options for transport.

  • HTTP Post and Get: This is most often used by someone sitting at a computer with a Web browser, uploading and downloading files. However, it can also be done from Perl scripts. Session-based security, in the form of HTTPS, is usually used for HTTP data transfers.

  • FTP: This is the common Internet File Transport Protocol. An ftp utility is provided now with nearly every computer shipped. There's also a flavor of FTP known as secure FTP that uses session-based security. Low-cost packages are available for secure FTP. If you use secure FTP you probably won't need to worry about persistent encryption or authentication. If you use plain-vanilla FTP you probably will need to use them.

  • SMTP: The Internet's Simple Mail Transfer Protocol, or common e-mail. Documents are usually sent as attachments using the Multipurpose Internet Mail Extensions (MIME) protocol, which we'll discuss in the section on packaging. Since most e-mail winds its way through the Internet over unsecured sessions, you'll most likely need to use persistent authentication and encryption measures. However, there are easy ways to do this, as we'll discuss shortly.

Another option may involve one or more of these transports. Some trading communities implement virtual private networks (VPNs) to establish secure sessions over the public Internet. In most such implementations connecting to the VPN usually requires an X.509 certificate. When VPN technology first started coming into vogue there was a great deal of interest in establishing these trading community–based VPNs. One example is the ANX network (originally the Automotive Network Exchange) at www.anx.com. However, after the initial surge of interest this type of VPN has generally not grown as fast as internal corporate VPNs. But don't be surprised if you are asked to join a VPN. Many VPN client software packages come with the required file transport and packaging features, so installing and using one shouldn't be overly difficult. Integrating it with your business applications may not be so easy.


Packaging

The key problem with bundling together a group of XML documents for transmission is that you can't just append them all together into one file. Doing so will result in one big, malformed document that most XML parsers will reject when they hit the root Element of the second document. Despite the fact that people have been aware of this problem since XML was first proposed for electronic commerce, there isn't yet a universally accepted solution.

If you've picked up on one of the themes of this chapter, you shouldn't be surprised to learn that your packaging options are somewhat dependent on your choice of transport. Here are a few options using proven, readily available technology.

  • MIME attachments: MIME allows several XML documents (or other types of objects) to be grouped together and attached to an e-mail message. There's also a flavor of MIME called S/MIME, for secure MIME, that supports persistent encryption and authentication. Many common e-mail clients support S/MIME using common encryption algorithms and X.509 certificates.

  • zip or UNIX tar: I'm really surprised that more people haven't been considering these options. It's very easy to package a number of XML documents into one zip or tar file. There's free software available for doing this. And on top of packaging the documents, you get compression too! Either of these options will work easily with all of the transport options listed in the previous section. What more could you ask for?

  • mput or mget with FTP: If you're using FTP you can circumvent the packaging problem by using these commands to transmit or receive multiple files in a directory.

In addition, some software utilities that provide persistent encryption can operate on several files in a directory at the same time, generating one encrypted file with all the documents.


Common Combinations for Security, Transport, and Packaging

The biggest news in this area in the last part of 2002 was that Wal-Mart decided to move from doing direct dial-up for EDI to Internet-based transport using the EDI over the Internet (EDIINT) standard developed by the Internet Engineering Task Force (IETF). There are two flavors of EDIINT, one based on SMTP (called AS1 for Applicability Statement 1) and the other based on HTTP (called AS2). Both use S/MIME. Wal-Mart is requiring that its vendors use AS2. Notice two important things about this development: (1) the choice of the relatively mature EDIINT using S/MIME for packaging and encryption, and (2) the decision to still use EDI and not XML. EDIINT can be used for payloads other than EDI, and there are several software packages available for it. More than likely, you can expect to be asked to use it in the next several years.

Here are a couple other approaches I've seen.

  • Secure HTTP: Session-based security using HTTP post and get and X.509 certificates. The U.S. Immigration and Naturalization Service is using this approach for schools reporting information on foreign students. One document is transmitted at a time, so there isn't a packaging issue.

  • Secure FTP: Session-based security using FTP and session-negotiated encryption. The Student Aid Internet Gateway of the U.S. Department of Education is using this approach for transmitting student loan data. Proprietary software is used for packaging and compression.

There are a few very easy ways to package and transport XML if your requirements are not very stringent. At the inaugural ebXML Work Group meeting in November 1999, the Business Application Software Developers Association (BASDA) demonstrated a very simple, workable approach with small business–oriented desktop business applications. This approach basically involved exporting XML documents to disk files, attaching them to e-mail messages using common e-mail clients, and then sending them over the public Internet in the clear. The recipient detached the XML document from the message, imported it into the business application where it was held in a suspense state for review, and then released it after review.

If you don't have great concerns about confidentiality or authentication, an approach like BASDA's could work just as well for you as it did in the demo. However, if you are concerned about these security aspects, many common e-mail clients allow you to attach an XML document to an e-mail message, then encrypt them both with S/MIME. The recipient can then decrypt and authenticate both. There are, however, a few disadvantages to this approach. One is that most e-mail clients bundle the e-mail message body and any attachments together as one S/MIME attachment. This might not be a problem, but with this method you can't independently apply a digital signature to an XML document. In addition, some e-mail clients (particularly automated ones in large companies) may not support this approach. It is primarily person-to-person rather than fully automated, which comes with its own set of problems. However, if you're part of a small company and want to exchange XML documents with other small companies (or even with people in large companies who will support doing this method), this is a cheap, easy solution for packaging and transport that uses software you probably already own.


Emerging Technologies

Everything we've discussed up to this point is commonly available and widely implemented in one way or another. (The only problem is that there isn't yet a widespread consensus on which of these pieces to use and for which requirements.) I can't close the discussion without giving you at least a preview of a few newer, less mature technologies. It's always hard to predict the future. However, my hunch is that it will be at least a few years before any of these find their way out into the mass market to the extent that EDI has (not that EDI has a big market, but this comparison at least gives us a frame of reference). Major players, both middleware vendors and big application vendors, are making investments in these technologies. Though none are widely implemented yet, it is likely that at least some or all will replace or eclipse some of the technologies discussed above.

Two transport technologies bear watching. The first of these is SOAP (which started out as an acronym for Simple Object Access Protocol but now is just SOAP). This XML-based protocol was originally designed as a mechanism for invoking remote object methods over the Internet in distributed object-oriented systems. The arguments for a method are packaged into a SOAP XML message and sent to a SOAP server. It takes apart the message, passes the arguments to the method, and then passes the results back to the caller in another SOAP message. This is very similar to the model of the familiar UNIX Remote Procedure Call (RPC). However, it didn't take people very long to figure out that SOAP could be used as a general method for transporting XML messages. A SOAP message has a header part and a body part. The header part allows for extension elements, which gives people the ability to insert routing and other control information. The body part is usually one complete XML document.

SOAP was developed by Microsoft and IBM. Although it is now being worked on by the W3C XML Protocol (XMLP) Working Group as part of the W3C's Web Services Activity, it has not yet been approved as a recommendation. It is, however, gaining a lot of notice as a key component of Web services.

A significant problem with SOAP is that it didn't in the beginning support more than one XML document in a SOAP wrapper. SOAP 1.1 with attachments uses MIME to append one or more XML documents to a SOAP message following the body part. However, the approach of using SOAP 1.1 with attachments has not yet been adopted by the W3C's XMLP group. While the long-term status of this approach is open to question, the number of vendors that support it is increasing.

The other emerging transport approach, the ebXML messaging service, uses SOAP 1.1 with attachments. A few commercial and open source implementations of the ebXML messaging service have appeared. In addition it has been adopted by other bodies such as RosettaNet. However, interest in all of ebXML seems to have been eclipsed in the marketplace by Web services. Only time will tell whether the messaging service or the rest of the ebXML framework gains acceptance. A more detailed discussion and a critical analysis of ebXML is available on my Web site at www.rawlinsecconsulting.com/ebXML.

There are several emerging technologies in the security area, but few yet need monitoring by end users or business application developers. A lot of them have to do with Web services and might be incorporated in middleware layers that few of us need to be concerned with for at least a few years, if ever. Others involve sophisticated and complex security models that go beyond the needs of most common current business practices. However, a few of these technologies do bear watching.

One such security technology is the W3C's XML Signature, sometimes referred to as XML DSIG. This is a way to apply a persistent digital signature to an XML instance document or parts of it (or strictly speaking, to any digital content, whether part of the document with the signature or not). As such, it satisfies various aspects of integrity, providing assurance that the signed object hasn't been tampered with and providing authentication of the signer. DSIG is the recommended digital signature approach in the ebXML messaging service. DSIG was approved in February 2002 and is being supported by an increasing number of vendors and APIs. I've not yet seen it used in customary exchange of common business documents, but I'm sure it will be the preferred method when a persistent signature is required.

The other security technology to watch is XML Encryption. This area has several facets and is covered by more than one W3C Recommendation. XML Encryption provides a standard approach for encrypting part or all of an XML document and representing the resulting cypher text as an XML Element. The XML Encryption Syntax and Processing Recommendation was approved in December 2002. As with XML DSIG, it is being supported by a growing number of APIs and vendors, but I've not seen it used yet. However, as the market matures you can probably expect it to become the preferred method when persistent encryption of XML is required.


What This Means for You

How you should make sense of this situation depends on whether you are an end user or a developer.

If you're an end user, you're most likely going to implement what your big trading partners require. If you're lucky, you won't have different customers implementing different technologies. To the extent that you are given choices, keep in mind the general considerations presented at the beginning of the chapter. With an understanding of the various technologies available to you, make your choice based on which option best fits your requirements.

If you develop business applications for the people mentioned in the previous paragraph, you should come away with the message that the situation is still very much in flux and probably will be for the lifetime of this book. Even though several mature, well-supported technologies solve the problems of security, transport, and packaging, there isn't yet a widespread convergence on a few common combinations. It is very likely that your user base will be required to implement several different approaches. Your most cost-effective strategy for supporting security, transport, and packaging requirements will probably be to not develop software yourself or build features directly into your applications. You may be better served and may better serve your users by partnering with appropriate vendors or adopting an open approach that lets users plug in packages of their choice.


Reference

ITSEC. 1991. Information Security Technology Evaluation Criteria (ITSEC): Harmonised Criteria of France, Germany, the Netherlands, the United Kingdom. London: Department of Trade and Industry. Available online at www.iwar.org.uk/comsec/resources/standards/itsec.htm, accessed March 27, 2003.


Appendix A. GNU General Public License

Version 2, June 1991

Copyright © 1989, 1991 Free Software Foundation, Inc.

59 Temple Place, Suite 330, Boston, MA 02111-1307 USA

Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed.


Preamble

The licenses for most software are designed to take away your freedom to share and change it. By contrast, the GNU General Public License is intended to guarantee your freedom to share and change free software—to make sure the software is free for all its users. This General Public License applies to most of the Free Software Foundation's software and to any other program whose authors commit to using it. (Some other Free Software Foundation software is covered by the GNU Library General Public License instead.) You can apply it to your programs, too.

When we speak of free software, we are referring to freedom, not price. Our General Public Licenses are designed to make sure that you have the freedom to distribute copies of free software (and charge for this service if you wish), that you receive source code or can get it if you want it, that you can change the software or use pieces of it in new free programs; and that you know you can do these things.

To protect your rights, we need to make restrictions that forbid anyone to deny you these rights or to ask you to surrender the rights. These restrictions translate to certain responsibilities for you if you distribute copies of the software, or if you modify it.

For example, if you distribute copies of such a program, whether gratis or for a fee, you must give the recipients all the rights that you have. You must make sure that they, too, receive or can get the source code. And you must show them these terms so they know their rights.

We protect your rights with two steps: (1) copyright the software, and (2) offer you this license which gives you legal permission to copy, distribute and/or modify the software.

Also, for each author's protection and ours, we want to make certain that everyone understands that there is no warranty for this free software. If the software is modified by someone else and passed on, we want its recipients to know that what they have is not the original, so that any problems introduced by others will not reflect on the original authors' reputations.

Finally, any free program is threatened constantly by software patents. We wish to avoid the danger that redistributors of a free program will individually obtain patent licenses, in effect making the program proprietary. To prevent this, we have made it clear that any patent must be licensed for everyone's free use or not licensed at all.

The precise terms and conditions for copying, distribution and modification follow.


Terms and Conditions for Copying, Distribution and Modification

0. This License applies to any program or other work which contains a notice placed by the copyright holder saying it may be distributed under the terms of this General Public License. The "Program", below, refers to any such program or work, and a "work based on the Program" means either the Program or any derivative work under copyright law: that is to say, a work containing the Program or a portion of it, either verbatim or with modifications and/or translated into another language. (Hereinafter, translation is included without limitation in the term "modification".) Each licensee is addressed as "you".

Activities other than copying, distribution and modification are not covered by this License; they are outside its scope. The act of running the Program is not restricted, and the output from the Program is covered only if its contents constitute a work based on the Program (independent of having been made by running the Program). Whether that is true depends on what the Program does.

1. You may copy and distribute verbatim copies of the Program's source code as you receive it, in any medium, provided that you conspicuously and appropriately publish on each copy an appropriate copyright notice and disclaimer of warranty; keep intact all the notices that refer to this License and to the absence of any warranty; and give any other recipients of the Program a copy of this License along with the Program.

You may charge a fee for the physical act of transferring a copy, and you may at your option offer warranty protection in exchange for a fee.

2. You may modify your copy or copies of the Program or any portion of it, thus forming a work based on the Program, and copy and distribute such modifications or work under the terms of Section 1 above, provided that you also meet all of these conditions:

  1. You must cause the modified files to carry prominent notices stating that you changed the files and the date of any change.

  2. You must cause any work that you distribute or publish, that in whole or in part contains or is derived from the Program or any part thereof, to be licensed as a whole at no charge to all third parties under the terms of this License.

  3. If the modified program normally reads commands interactively when run, you must cause it, when started running for such interactive use in the most ordinary way, to print or display an announcement including an appropriate copyright notice and a notice that there is no warranty (or else, saying that you provide a warranty) and that users may redistribute the program under these conditions, and telling the user how to view a copy of this License. (Exception: if the Program itself is interactive but does not normally print such an announcement, your work based on the Program is not required to print an announcement.)

These requirements apply to the modified work as a whole. If identifiable sections of that work are not derived from the Program, and can be reasonably considered independent and separate works in themselves, then this License, and its terms, do not apply to those sections when you distribute them as separate works. But when you distribute the same sections as part of a whole which is a work based on the Program, the distribution of the whole must be on the terms of this License, whose permissions for other licensees extend to the entire whole, and thus to each and every part regardless of who wrote it.

Thus, it is not the intent of this section to claim rights or contest your rights to work written entirely by you; rather, the intent is to exercise the right to control the distribution of derivative or collective works based on the Program.

In addition, mere aggregation of another work not based on the Program with the Program (or with a work based on the Program) on a volume of a storage or distribution medium does not bring the other work under the scope of this License.

3. You may copy and distribute the Program (or a work based on it, under Section 2) in object code or executable form under the terms of Sections 1 and 2 above provided that you also do one of the following:

  1. Accompany it with the complete corresponding machine-readable source code, which must be distributed under the terms of Sections 1 and 2 above on a medium customarily used for software interchange; or,

  2. Accompany it with a written offer, valid for at least three years, to give any third party, for a charge no more than your cost of physically performing source distribution, a complete machine-readable copy of the corresponding source code, to be distributed under the terms of Sections 1 and 2 above on a medium customarily used for software interchange; or,

  3. Accompany it with the information you received as to the offer to distribute corresponding source code. (This alternative is allowed only for noncommercial distribution and only if you received the program in object code or executable form with such an offer, in accord with Subsection b above.)

The source code for a work means the preferred form of the work for making modifications to it. For an executable work, complete source code means all the source code for all modules it contains, plus any associated interface definition files, plus the scripts used to control compilation and installation of the executable. However, as a special exception, the source code distributed need not include anything that is normally distributed (in either source or binary form) with the major components (compiler, kernel, and so on) of the operating system on which the executable runs, unless that component itself accompanies the executable.

If distribution of executable or object code is made by offering access to copy from a designated place, then offering equivalent access to copy the source code from the same place counts as distribution of the source code, even though third parties are not compelled to copy the source along with the object code.

4. You may not copy, modify, sublicense, or distribute the Program except as expressly provided under this License. Any attempt otherwise to copy, modify, sublicense or distribute the Program is void, and will automatically terminate your rights under this License. However, parties who have received copies, or rights, from you under this License will not have their licenses terminated so long as such parties remain in full compliance.

5. You are not required to accept this License, since you have not signed it. However, nothing else grants you permission to modify or distribute the Program or its derivative works. These actions are prohibited by law if you do not accept this License. Therefore, by modifying or distributing the Program (or any work based on the Program), you indicate your acceptance of this License to do so, and all its terms and conditions for copying, distributing or modifying the Program or works based on it.

6. Each time you redistribute the Program (or any work based on the Program), the recipient automatically receives a license from the original licensor to copy, distribute or modify the Program subject to these terms and conditions. You may not impose any further restrictions on the recipients' exercise of the rights granted herein. You are not responsible for enforcing compliance by third parties to this License.

7. If, as a consequence of a court judgment or allegation of patent infringement or for any other reason (not limited to patent issues), conditions are imposed on you (whether by court order, agreement or otherwise) that contradict the conditions of this License, they do not excuse you from the conditions of this License. If you cannot distribute so as to satisfy simultaneously your obligations under this License and any other pertinent obligations, then as a consequence you may not distribute the Program at all. For example, if a patent license would not permit royalty-free redistribution of the Program by all those who receive copies directly or indirectly through you, then the only way you could satisfy both it and this License would be to refrain entirely from distribution of the Program.

If any portion of this section is held invalid or unenforceable under any particular circumstance, the balance of the section is intended to apply and the section as a whole is intended to apply in other circumstances.

It is not the purpose of this section to induce you to infringe any patents or other property right claims or to contest validity of any such claims; this section has the sole purpose of protecting the integrity of the free software distribution system, which is implemented by public license practices. Many people have made generous contributions to the wide range of software distributed through that system in reliance on consistent application of that system; it is up to the author/donor to decide if he or she is willing to distribute software through any other system and a licensee cannot impose that choice.

This section is intended to make thoroughly clear what is believed to be a consequence of the rest of this License.

8. If the distribution and/or use of the Program is restricted in certain countries either by patents or by copyrighted interfaces, the original copyright holder who places the Program under this License may add an explicit geographical distribution limitation excluding those countries, so that distribution is permitted only in or among countries not thus excluded. In such case, this License incorporates the limitation as if written in the body of this License.

9. The Free Software Foundation may publish revised and/or new versions of the General Public License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns.

Each version is given a distinguishing version number. If the Program specifies a version number of this License which applies to it and "any later version", you have the option of following the terms and conditions either of that version or of any later version published by the Free Software Foundation. If the Program does not specify a version number of this License, you may choose any version ever published by the Free Software Foundation.

10. If you wish to incorporate parts of the Program into other free programs whose distribution conditions are different, write to the author to ask for permission. For software which is copyrighted by the Free Software Foundation, write to the Free Software Foundation; we sometimes make exceptions for this. Our decision will be guided by the two goals of preserving the free status of all derivatives of our free software and of promoting the sharing and reuse of software generally.

NO WARRANTY

11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION.

12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

END OF TERMS AND CONDITIONS


How to Apply These Terms to Your New Programs

If you develop a new program, and you want it to be of the greatest possible use to the public, the best way to achieve this is to make it free software which everyone can redistribute and change under these terms.

To do so, attach the following notices to the program. It is safest to attach them to the start of each source file to most effectively convey the exclusion of warranty; and each file should have at least the "copyright" line and a pointer to where the full notice is found.

<one line to give the program's name and a brief idea of what it does.>
Copyright (C) <year>  <name of author>

    This program is free software; you can redistribute it and/or modify
    it under the terms of the GNU General Public License as published by
    the Free Software Foundation; either version 2 of the License, or
    (at your option) any later version.

    This program is distributed in the hope that it will be useful,
    but WITHOUT ANY WARRANTY; without even the implied warranty of
    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
    GNU General Public License for more details.

    You should have received a copy of the GNU General Public License
    along with this program; if not, write to the Free Software
    Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA

Also add information on how to contact you by electronic and paper mail.

If the program is interactive, make it output a short notice like this when it starts in an interactive mode:

Gnomovision version 69, Copyright (C) year name of author

Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type 'show w'.

This is free software, and you are welcome to redistribute it

under certain conditions; type 'show c' for details.

The hypothetical commands 'show w' and 'show c' should show the appropriate parts of the General Public License. Of course, the commands you use may be called something other than 'show w' and 'show c'; they could even be mouse-clicks or menu items—whatever suits your program.

You should also get your employer (if you work as a programmer) or your school, if any, to sign a "copyright disclaimer" for the program, if necessary. Here is a sample; alter the names:

Yoyodyne, Inc., hereby disclaims all copyright interest in the program

'Gnomovision' (which makes passes at compilers) written by James Hacker.

<signature of Ty Coon>, 1 April 1989

Ty Coon, President of Vice

This General Public License does not permit incorporating your program into proprietary programs. If your program is a subroutine library, you may consider it more useful to permit linking proprietary applications with the library. If this is what you want to do, use the GNU Library General Public License instead of this License.


Appendix B. Pseudocode Conventions

This appendix presents the pseudocode conventions used in this book. The pseudocode is based on Programming Design Language (PDL) by Caine and Gordon [1975]. The main changes from their presentation of PDL is that I have made the assignment and some other operators explicit and made specific references to the DOM.

While PDL was intended to be implementation dependent, I use specific DOM interface methods where they have been defined. Where the DOM specifies an attribute but doesn't state a specific function to set or get the value of the attribute, I just used the set or get terminology. In addition, where the DOM recommendation lists a specific set of constant values (for example, the nodeType list with ELEMENT_NODE, TEXT_NODE, and so on), I used the specified constant. One of the effects of this approach is that the pseudocode tends to be more detailed and implementation oriented (such as methods returning null pointers) than is often the case with pseudocode.


Assignment Operator

Assignment: <-

Example:

iDollarsYouOweMe <- 5000


Mathematical Operators

Addition: +

Subtraction: -

Multiplication: *

Division: /


Equality and Relational Operators

Equals: =

Less than: <

Less than or equal: <=

Greater than: >

Greater than or equal: =>


IF Selection Structures

Simple if-then:

IF condition
  statement
ENDIF

Example:

IF you pay your money
  you take your chances
ENDIF

If-then-else:

IF condition
  statement
ELSE
  statement
ENDIF

Example:

IF you pay your money
  you take your chances
ELSE
  you keep your money
ENDIF

Compound if-then-else:

IF condition
  statement
ELSEIF condition
  statement
ELSE
  statement
ENDIF

Example:

IF you pay your money
  you take your chances
ELSEIF I pay my money
  I take my chances
ELSE
  we both keep our money
ENDIF


Repetition Structure

Do loop:

DO continuation condition or for collection or range
  statement
ENDDO

Examples:

DO while you owe me money
  Send you a bill every month
ENDDO

DO for everyone who owes me money
  Send a bill every month
ENDDO


Case Selection Structure

Case statement:

DO CASE of variable
  case1:
    statement
    BREAK
  case2:
    statement
  other:
    statement
ENDDO

The PDL DO CASE construct is modified with the addition of a BREAK. This allows cases to fall through to the next case if there is no BREAK.

Example:

DO CASE of days overdue
  30:
    Send nice letter
    BREAK
  60:
    Send not nice letter
    BREAK
  90:
    Send nastygram
    BREAK
  120:
    Refer to collection agency
  other:
    Mourn
ENDDO


Appendix C. COM Essentials for the Non-COM Programmer

I use COM in this book not because I want to but because I have to. Having made the decision to use MSXML for the C++ implementation, I have to use COM. MSXML is a COM server component. The applications developed in this book are clients of the services it offers. In this appendix I'll try to give you just enough COM so that you can comfortably do client-side COM programming using MSXML. COM is very complex and this appendix doesn't even scratch the surface. In addition, with .NET out and about, COM is legacy technology. So, you probably don't want to spend any more time learning it than you have to.

Let's start at the beginning with how to tell the compiler we're using MSXML and the choices that gives us. Then we'll discuss initialization and a bit more about resource management. After touching on errors and exceptions, we'll finish up with some COM data types and helper classes.


#import and Smart Pointers

These two lines are included in all our C++ routines that use any of the DOM interfaces:

#import <msxml4.dll>
using namespace MSXML2;

This is a Visual C++ compiler directive that tells the compiler we're going to use MSXML. It must be written exactly as shown, on two lines.

The #import directive does more just than fulfill the general purpose of including information from a type library (similar to the way the #include directive brings in header files): It also creates so-called "smart pointer" interfaces for the COM interfaces defined in the type library being imported. Smart pointers are native C++ class wrappers that hide a lot of the messy details of creating and managing COM objects. One of the nicest things about them is that since they are native C++ classes, their destructors get called when the smart pointers go out of scope. Their destructors handle all the details of cleaning up the use of the COM object. Since they overload the indirection operator (->), we can use them just like any other pointer.

Smart pointers are particularly important to us because in MSXML's DOM implementation, all our DOM objects are accessed via pointers. We can reference a NodeList by using a regular pointer like this:

IXMLDOMNodeList * pNodeList;

or we can use the smart pointer approach like this:

IXMLDOMNodeListPtr pNodeList;

We use both types of pointers in the same way. The following code works with both declarations.

while (pNodeList->item[iNodes] != NULL)

However, the difference comes when we are finished using the interface. With dumb pointers, for our COM clients to be good citizens and help manage resources properly they must call the interface's AddRef and Release methods. With smart pointers this is done automatically.

The MSXML SDK code examples use both the dumb and smart pointer approaches. Following the KISS principle, I only use smart pointers. There may be cases in loops or in argument passing where this approach is not the most efficient, but again we're valuing simplicity over any miniscule effect on performance.

Note that I have adopted the convention of using an sp prefix with smart pointers.


Initialization and Resource Management

Since a lot of our resource management issues are made simpler by using smart pointers to MSXML's DOM interfaces, we really need to pay attention to only a few things.

  • CoInitialize(NULL), returns HRESULT: In every program that uses a COM component the COM library must be initialized using this call. By convention and following the sample code from Microsoft and other authors, I call this early in the main line of the main routine. Values other than NULL can be specified if you are, for example, working with a multi threaded application.

  • CoUninitialize(), returns void: This closes the COM library. Every CoInitialize call should be matched with a CoUninitialize call. Again, by convention and following the sample code from Microsoft and other authors, I call this in the main line of the main routine right before the return.

  • CreateInstance(SBCS clsidString), returns HRESULT: This creates a COM object. In our use of MSXML we create an object that represents a DOM document. In (almost) all our code the clsidString we use is the value returned by the uuidof keyword for DOMDocument40. CreateInstance is a method of the smart pointer class for the IXMLDOMDocument interface. There is a corresponding CoCreateInstance call for dumb pointers if you are using them.

  • Release(), returns void: This releases a COM interface. We will usually let smart pointers handle this, but in rare cases we may use it directly so we can reuse a pointer.

CAUTION Make Sure to Release Resources and Close Down Cleanly!

Until I figured out what was happening, I consistently got an unhandled exception with an access violation at 0xC0000005 in, of all places, the return statement in my main routine. This started happening when, like a good doobie, I put in the CoUninitialize call to close down the COM library. The exception finally went away after I called the Release methods on all my IXMLDOMMumblePtr objects. I finally made things simpler by just declaring the IXMLDOMMumblePtr objects in the try block instead of my usual place at the beginning of the routine. Exiting the try block made them go out of scope, called their destructors, and cleaned things up for me without having to invoke Release.

Commenting out the CoUninitialize call also made the exception go away. However, I'm a bit dubious and too cautious to rely solely on the C++ return statement and program termination to properly clean up everything. I've seen enough weirdness and instability in the various Windows operating systems to want to avoid risking the introduction of any more such problems. In addition, if you are using MSXML in only one routine of a complex application that will keep running when you're finished with XML, you really want to clean up properly.

CAUTION Call the Correct Release Method!

If you're going to call a Release method, make sure you call the right one. This is another bit of COM complexity. As noted earlier, our smart pointers can be used like regular C++ pointers, but they aren't simple pointers at all. Although they point to DOM objects, they are themselves objects with properties and methods. Adding to the confusion is that both the pointer and the pointee have Release methods. You want to call the Release method in the Ptr object and not the one in the thing it points to. For example, call spDocInput.Release() instead of spDocInput->Release(). Doing the former will work. Doing the latter may result in an unhandled exception at program termination. You can identify these by a report of an access violation at an address other than 0xC0000005 when you return in your main routine.


Exceptions and Errors

Well-behaved COM server objects don't usually throw exceptions back to their clients, but they can. So, to be cautious it's a good idea to put most of your COM-related code in a try block and follow it with a catch block for COM errors. COM errors are encapsulated in the Visual C++ compiler helper class _com_error.

Other than that, you need to know about HRESULT. This is a 32-bit data structure used to return status information about calls to methods in the COM APIs. You can't just check on a nonzero return value to see whether or not a call failed; it is a more complex data structure than that. Visual C++ provides some macros to check on HRESULT. I use FAILED and SUCCEEDED. Both return booleans. You'll see that the book's code also checks on some more specific return values associated with the MSXML DOM implementation.

If you want to do by-the-book COM programming, you will check for the HRESULT return from every method call to a COM interface. We could do this, but the code would be littered with status checks that in 99.9 percent of the cases would never execute the failure branch. I instead rely primarily on schema validation to enforce some expectations about the input XML documents we'll be processing and check for NULL pointers where applicable.


COM Classes and Data Types for C++

This is a brief list of the nonnative C++ COM data types and helper classes you are likely to have to deal with when using MSXML.

  • VARIANT, encapsulated as a COM helper class in C++ as _variant_t: This data type is the most awkward. It's a structure with several fields. The first one is a VARTYPE that specifies the type of VARIANT, that is, the data type of the data stored in the VARIANT. The last one is a union of several field types that contains the value of the VARIANT. There are only a few DOM methods that use a VARIANT, and for each I show how to set up the VARIANT.

  • BSTR Binary String data type, encapsulated as a COM helper class in C++ as _bstr_t: COM uses this in several places, but we mostly use it to represent character data in a VARIANT. A BSTR can be cast as TEXT, TCHAR, and LPTSTR to handle character sets other than the default. To keep things simple, we'll just use the BSTR whenever possible. Visual C++ usually is perfectly happy treating a _bstr_t as a char *.

VARIANT in the MSXML DOM Implementation

We most often have to deal with the VARIANT data type in the DOM get and set NodeValue methods. In the W3C DOM Recommendation these have different return and argument values depending on the Node type. For example, getNodeValue on a Text Node returns the text associated with the parent Element. When called on a Processing Instruction Node it returns the entire content excluding the target. When called on almost anything else it returns a NULL. From the perspective of someone developing a DOM API, using a VARIANT is a perfectly logical choice. However, it doesn't make life very easy for someone using the API. Java's JAXP is much easier to use in this regard, and so is the DOM implementation in System.XML in .NET.

If you find something in the book's code or in the MSXML sample code that isn't listed here, you can usually learn more about it by performing an index or search in the MSDN Library (available online at http://msdn.microsoft.com/library or from online help in Visual Studio).


Bibliography

    References

    Resources


References

Cafiero, Bill. 2002. Remarks while leading a panel discussion at the semimonthly meeting of the Dallas/Fort Worth Electronic Commerce Forum, July 18, in Irving, Texas.

Caine, S., and E. K. Gordon. 1975. "PDL—a Tool for Software Design." In the American Federation of Information Processing Societies' Proceedings of the 1975 National Computer Conference, Anaheim, CA, 19–22 May 1975. Montvale, NJ: AFIPS Press, pp. 271–276. Available online at http://www.cfg.com/pdl81/pdlpaper.html, accessed July 11, 2002.

Cline, Marshall. 2002. "C++ FAQ Lite." Available online at http://www.atd.ucar.edu/software/c++-faq/newbie.html#faq-29.7, accessed March 27, 2003..

Costello, Roger. 2003. "Global versus Local" in "XML Schemas: Best Practices." Available online at http://www.xfront.com/GlobalVersusLocal.pdf, accessed March 31, 2003..

exolab. 2002. "Castor XML Schema Support." Available online at http://castor.exolab.org/xmlschema.html, accessed September 7, 2002..

ITSEC. 1991. Information Security Technology Evaluation Criteria (ITSEC): Harmonised Criteria of France, Germany, the Netherlands, the United Kingdom. London: Department of Trade and Industry. Available online at http://www.iwar.org.uk/comsec/resources/standards/itsec.htm, accessed March 27, 2003..

Kimberley, Paul. 1991. Electronic Data Interchange. New York: McGraw Hill.

Martin, John C. 1991. Introduction to Languages and the Theory of Computation, 2nd ed. Boston, MA: WCB/McGraw-Hill.

Shaw, Mary, and David Garlan. 1996. Software Architecture—Perspectives on an Emerging Discipline. Upper Saddle River, NJ: Prentice Hall.

Skonnard, Aaron, and Martin Gudgin. 2002. Essential XML Quick Reference. Boston, MA: Addison-Wesley.

World Wide Web Consortium (W3C). 2002. Document Object Model main page. Available online at http://www.w3.org/DOM/, accessed July 10, 2002.

———. 2000. Canonical XML Version 1.0. Available online at http://www.w3.org/TR/xml-c14n, accessed May 17, 2003..

———. 1999. XSL Transformations, Version 1.0, W3C Recommendation, November 16. Available online at http://www.w3.org/TR/1999/REC-xslt-19991116, accessed January 26, 2003.


Resources

Apache Software Foundation: http://www.apache.org.

"ASC X12 Reference Model for XML Design," Type 2 Technical Report, by the ANSI Accredited Standards Committee X12: http://www.x12.org.

Babel Blaster, an open source project at SourceForge: http://babelblaster.sourceforge.net.

Compilers—Principles, Techniques, and Tools, by Alfred V. Aho, Ravi Sethi, and Jeffrey D. Ullman (Reading, MA: Addison-Wesley, 1988).

DOM Level 2 Core Specification, November 13, 2000, by the World Wide Web Consortium: http://www.w3.org/DOM/DOMTR#dom2.

DOM Reference, Microsoft XML Core Services (MSXML) 4.0, by Microsoft Corporation: http://msdn.microsoft.com/xml.

Java API for XML Processing Specification 1.2, by Sun Microsystems: http://java.sun.com/xml.

javax.xml.transform package documentation in the Java API for XML Processing Specification 1.2, by Sun Microsystems: http://java.sun.com/xml.

Microsoft's XML Resources: http://msdn.microsoft.com/xml.

Perl interpreters for various operating systems, as open source, available from Active State: http://www.activeware.com.

Rawlins EC Consulting, supplemental materials and information for this book: http://www.rawlinsecconsulting.com/booksupplement.

Requests for Comments, by the Internet Engineering Task Force of the Internet Society; base URL: http://www.ietf.org/rfc/.

Uniform Resource Identifiers (URI), Generic Syntax, RFC 2396, August 1998, rfc2396.txt

Uniform Resource Locators (URL), RFC 1738, December 1994, rfc1738.txt

URN Namespace Definition Mechanisms, RFC 2611, June 1999, rfc2611.txt

URN Syntax, RFC 2141, May 1997, rfc2141.txt

Sun Microsystem's Java Technology and XML: http://java.sun.com/xml.

transformNodeToObject method documentation in Microsoft XML Core Services (MSXML) 4.0, by Microsoft Corporation: http://msdn.microsoft.com/xml.

"UBL Naming and Design Rules," by the Naming and Design Rules subcommittee of OASIS's Universal Business Language Technical Committee: http://oasis-open.org/committees/ubl/ndrsc/#documents.

Unicode information: http://www.unicode.org.

World Wide Web Consortium: http://www.w3.org.

X12.5 Interchange Control Structures, Release 004010, December 1997, by the Accredited Standards Committee X12 of the American National Standards Institute; available for purchase from the Data Interchange Standards Association: http://www.x12.org.

X12.6 Application Control Structure, Release 004010, December 1997, by the Accredited Standards Committee X12 of the American National Standards Institute; available for purchase from the Data Interchange Standards Association: http://www.x12.org.

Xerces2 API Javadoc, by Apache Software Foundation: http://xml.apache.org.

XML and SQL—Developing Web Applications, by Daniel K. Appelquist (Boston, MA: Addison-Wesley, 2002).

The XML Cover Pages: http://xml.coverpages.org.

XML Path Language (XPath), Version 1.0, W3C Consortium Recommendation: http://www.w3.org/TR/1999/REC-xpath-19991116.

XML Schema Recommendation, by World Wide Web Consortium; base URL: http://www.w3.org/TR/.

Part 0: Primer, xmlschema-0.html

Part 1: Structures, xmlschema-1.html

Part 2: Datatypes, xmlschema-2.html

"XML Schemas: Best Practices," developed collaboratively by The MITRE Corporation and members of the xml-dev list group, hosted at Roger Costello's xFront site: http://www.xfront.com/BestPracticesHomepage.html.

"XML Technical Specification for Higher Education," by The XML Forum for Education of the Postsecondary Electronic Standards Council: http://www.pescxml.org/.

XSL-List listserv, an open forum on XSL hosted by Mulberry Technologies; subscription instructions: http://www.mulberrytech.com/xsl/xsl-list/.


Credits

    Chapter 1

    Chapter 4

    Chapter 5

    Chapter 6

    Chapter 8

    Chapter 9

    Chapter 12


Chapter 1

Cafiero, Bill. 2002. Remarks while leading a panel discussion at the semi-monthly meeting of the Dallas/Fort Worth Electronic Commerce Forum, July 18, in Irving, Texas.


Chapter 4

Cooley, Mason. 1993. City Aphorisms, Eleventh Selection. New York.


Chapter 5

Toynbee, Arnold Joseph. 1969. Experiences. Oxford, U.K.: Oxford University Press.


Chapter 6

Thoreau, Henry David. 1906. The Writings of Henry David Thoreau. Boston, MA: Houghton Mifflin.


Chapter 8

Fleischman, John. 1985. Letter to the editor. New York Times, June 7.


Chapter 9

Kimberley, Paul. 1991. Electronic Data Interchange. New York: McGraw Hill.


Chapter 12

Muir, John. 1911. My First Summer in the Sierra. Boston, MA: Houghton Mifflin.


Team LiB

Brought to You by

Like the book? Buy it!