Data Integration Suite – Flat File Support

In addition to supporting most relational databases and a wide array of ever-evolving EDI standards, the Data Integration Suite also provides built-in support for many common flat file formats. Additionally, using Data Integration Suite's powerful extensibility mechanism developers can build custom XML converters designed for specific file conversion needs.

Overall, Data Integration Suite can provide support a for wide variety of different flat file formats which can be processed by either built-in XML converters or user-defined, custom XML converters, including the following:

• Flat Files – such as comma- and tab-separated value files
• Fixed-width Files – like dumps from databases or certain EDI-like formats
• Tagged Files – which contain multiple row types within a single data stream
• Hybrid Files – those that contain mixtures of the above qualities

Built-in Flat File Conversion

Specifically, the following file types are supported out-of-the-box:

Additionally, using Data Integration Suite, you can create custom XML conversions for extensions to standard file formats or even proprietary file formats.

Custom Flat File Conversion

Data Integration Suite’s capabilities for custom Flat File to XML conversion understand more than just textual data types. Often a file will contain binary information in any number of specialized formats — from hardware-specific types like BCD or IEEE Floats and Doubles to cross-platform standards such as COBOL Comp3 or ISO 8601 Date Time. Rather than having to resort to extension functions, using Data Integration Suite allows you to specify the exact native data type and it will handle the conversion for you. Supported data types include:

• BCD (Binary Coded Decimal)
• Float
• Binary (including the W3C Schema types base64Binary and hexBinary)
• Boolean (including support for null or unknown values)
• Integer (32-bit integers)
• Byte (8-bit integers)
• Long (64-bit integers)
• Comp3 (the COBOL internal format; "IBM Packed")
• Number (unlimited-precision numbers)
• Date (in multiple languages)
• Short (16-bit integers)
• DateTime
• String
• Decimal (from System.Decimal on .Net)
• Time
• Double
• Zoned ("IBM Zoned" mainframe datatype)

Pattern Matching in Flat Files

Sophisticated pattern-matching technology lets you decode different rows within the same file using different record layouts. Perhaps you have master-detail records, or a header-body-trailer layout. Or perhaps the file contains many types, similar to the way EDI files are designed. Using the pattern-matching technology, specific markers within rows can be used to select which conversion rules should apply to which rows.

Flat File Conversion Performance

The Data Integration Suite has been designed for streaming environments. This means that it never has to hold the entire file in memory. It only needs to process a single row of data at a time. This keeps the memory requirements low.

Since the internals are designed for a streaming world, it only makes sense to expose those interfaces to the outside world. On the Java platform, the StAX platform is supported for a 'pull' API, and on .net the XmlReader interface is supported. Both platforms use 'string interning,' a technique that replaces costly string comparisons with machine-speed integer comparisons. In Java, the SAX interface is also supported for 'push' processing.

Download a free trial of Data Integration Suite today!