MFL+JAXB versus pure Java parsing

Disclaimer:   this test was done in Eclipse using WLXT. The test proves to be INCREDIBLY slower compared to running the same MFL transformation inside OSB. In fact, inside OSB reading 418 binary files (5 KB each) and writing them in XML format took roughly 8 seconds - a LOT better than the 155 seconds using WLST.
I am curious now to understand where lies the HUGE difference in performance of the Format Builder libraries when run in OSB and in Eclipse.

Morale: don't benchmark MFL using a out of container test.

I have 417 files containing "binary" messages, fixed length fields, to be parsed into Java objects. Each file is some 5 KB each. Total some 2 MB.

Doing the transformation:
Binary -> XML (with MFL) followed by
XML -> Java (with JAXB)

inside Eclipse takes a whopping 157 seconds, out of which 101 in MFL and 56 in JAXB.
The same transformation run on a OSB proxy service takes some 5 seconds (!!!).

Doing the transformation Binary -> Java directly in Java takes 3.5 seconds - that is some 50 times faster. I assume the memory garbage produced would be also immensely lower by parsing directly in Java.

The Java parser took 4 hours to build and test. MFL took 2 hours to make and 2 hours to troubleshoot.

Oracle MFL tutorial

All the documentation is here:
http://download.oracle.com/docs/cd/E13160_01/wli/docs10gr3/pdf/fbhelp.pdf

Some examples here:
http://download.oracle.com/docs/cd/E13159_01/osb/docs10gr3/userguide/testing.html#wp1041203

Excellent tutorial here

http://biemond.blogspot.com/2008/12/flat-file-to-xml-with-oracle-service.html


In a OSB Project, do New, Other, Oracle Service Bus, MFL. Enter a name, like MQFeedParser.
You get this:

GEOMQFeed is the "Message Format", i.e. the "Root Element".

From here you can either insert a Group or a Field. A Group can contain other Groups and/or Fields.
A Group can be OPTIONAL or REQUIRED
A Group can repeat itself
You can define a Group once and use Group References to instantiate it multiple times.

"Choice of Children" means that only one of the fields specified is allowed.

Group Occurrence is self-explanatory.

A TAGGED GROUP is a Group whose start is preceded by a specific value, like NEWGROUP:ACME (the tag being NEWGROUP: and the group name being ACME)


________________________________


A good way to test the MFL is to write a File Transport PS with MFL input

For instance this:

<?xml version='1.0' encoding='windows-1252'?>
<!DOCTYPE MessageFormat SYSTEM 'mfl.dtd'>
<!--   Enter description of the message format here.   -->
<MessageFormat name='SampleMFL' version='2.02'>
    <StructFormat name='Employees' delimOptional='y' repeat='*'>
        <FieldFormat name='Name' type='String' delimOptional='y' length='12' strlenInChars='n' codepage='windows-1252'/>
        <FieldFormat name='Age' type='String' delimOptional='y' length='3' strlenInChars='n' codepage='windows-1252'/>
    </StructFormat>
</MessageFormat>

when your input is this
Pierre01 345Pierre02 346

should return you

<?xml version="1.0" encoding="UTF-8"?>
<SampleMFL>
  <Employees>
    <Name>Pierre01</Name>
    <Age>345</Age>
  </Employees>
  <Employees>
    <Name>Pierre01</Name>
    <Age>345</Age>
  </Employees>
</SampleMFL> 

If the input data doesn't match the MFL you get:

com.bea.wli.sb.sources.TransformException: Failed to transform MFL content from binary to XML

Caused By: <MFLException>
<ErrorMessage>Did not completely process all input data.</ErrorMessage>
<Details>
<Detail>
<Name>ErrorCode</Name>
<Value>-3</Value>
</Detail>
<Detail>
<Name>DataOffset</Name>
<Value>15</Value>
</Detail>
</Details>
</MFLException>
at com.bea.nonxml.common.MFLException.create(MFLException.java:221)
at com.bea.nonxml.common.MFLException.create(MFLException.java:337)
at com.bea.nonxml.readers.NonXMLReaderVisitor.nextToken(NonXMLReaderVisitor.java:155)
at com.bea.nonxml.readers.TokenNonXMLReader.nextToken(TokenNonXMLReader.java:44)
at com.bea.binxml.v2.readers.SaxTokenReader$ExtendedTokenReaderWrapper.nextToken(SaxTokenReader.java:335)
at com.bea.binxml.v2.readers.SaxTokenReader.process(SaxTokenReader.java:189)
at com.bea.binxml.v2.readers.SaxTokenReader.process(SaxTokenReader.java:154)
at com.bea.wli.common.mfl.MflExecutor.executeXmlObject(MflExecutor.java:126)
at com.bea.wli.sb.resources.mfl.MflExecutor.executeXmlObject(MflExecutor.java:98)
at com.bea.wli.sb.stages.transform.MflTransformImpl.executeXmlObject(MflTransformImpl.java:84)
at com.bea.wli.sb.sources.MFLSource.getInstance(MFLSource.java:113)
at com.bea.wli.sb.sources.MFLTransformer.transform(MFLTransformer.java:75)
at com.bea.wli.sb.sources.MetaTransformer.doTransform(MetaTransformer.java:138)
at com.bea.wli.sb.sources.MetaTransformer.transform(MetaTransformer.java:89)




This http://forums.oracle.com/forums/thread.jspa?threadID=945164 should help,
"try to change the message type from text to binary"

Here
C:\bea11\Oracle_OSB11\lib\modules\com.bea.alsb.resources.mfl.jar

is the MflExecutor


There is this tool WLXT

http://download.oracle.com/docs/cd/E13214_01/wli/docs70/classdocs/com/bea/wlxt/WLXT.html

to unittest MFL.
You should include in the CLASSPATH : com.bea.wli.transform.fbruntime_1.1.0.0.jar and weblogic.jar

File binaryfile = new File("c:/geodata.dat");
WLXT wlxt = new WLXT();
URL mflDocumentName = new URL("file:c:/geodata.mfl");
FileInputStream in = new FileInputStream(mybinaryfile);
Document doc = wlxt.parse(mflDocumentName, in, null);
String xml = WLXT.getXMLText(doc, 0, 2);
System.out.println(xml);

MFL and Excel

I don't think anybody in his right state of mind would actually use the Format Builder graphical tool to write MFL files.

In fact the MFL file is a simple XML file, I found it very convenient to stick the structure of my binary message in an Excel file, and generate the XML using simple Excel formulas.

Most of the time you use only a few MFL options so it's pretty trivial to generate automatically the XML file.

I wish I had the time to develop some sort of DSL or something.... the MFL.DTD is here

http://download.oracle.com/docs/cd/E13182_01/elink/elinkii/v10/usergd/appendb.htm

Here I generated the equivalent XSD:

http://javatoolsforweblogic.googlecode.com/svn/trunk/MFL/MFL.xsd

FML and MFL

When working with Tuxedo one gets confused between FML and FML.


The way I understand it is that:

FML is a data format standard proprietary to Tuxedo.
MFL is a METADATA about an arbitrary binary data format.



FML

http://en.wikipedia.org/wiki/Tuxedo_%28software%29

Flexible buffer formats

Tuxedo applications can utilize a variety of message formats depending upon the type of data that is to be passed. One of the most popular formats is the FML buffer format which is much like a binary XML or ASN.1 format. FML buffers can contain an arbitrary number of named fields of arbitrary type. Fields can be repeated and nested. As it is a self describing binary format, the processing of fields incurs very little overhead in comparison to the parsing necessary to support something like XML. VIEW buffers are essentially records, C structures, or COBOL copybooks. A VIEW buffer has an external description which allows Tuxedo to access the fields within it if necessary for things like data dependent routing. Other buffer formats include XML, CARRAY (opaque binary data), STRING, and MBSTRING (a string buffer containing multibyte characters.) Tuxedo can automatically and transparently convert FML buffers to and from XML buffers.

There is also support for user-developed buffer types (for example JamFlex buffers defined by Tuxedo version of Panther RAD toolset).



MFL

http://download.oracle.com/docs/cd/E13171_01/alsb/docs25/consolehelp/mfls.html

A Message Format Language (MFL) document is a specialized XML document used to describe the layout of binary data. It is a BEA proprietary language used to define rules to transform formatted binary data into XML data. An MFL document conforms to the mfl.dtd, which includes elements and attributes used to describe each field of data, as well as groupings of fields (groups), repetition, and aggregation.

When you create a business services or proxy services of Messaging Service type, you can select MFL types as the request message type or the response message type of the service.

You use BEA Format Builder to create MFLs. When you define the hierarchy of a binary record, the layout of fields, and the grouping of fields and groups, the information is saved as an MFL document that can then be used to perform run-time translations. An MFL document can also be used in Format Builder to generate the corresponding DTD that describes its content model. To learn how to use the Format Builder, see the Format Builder Online Help.