Integration Challenges - Part 1

Linking different systems together is a very common task for IT projects. As more and more COTS (Commercial Off The Shelf) systems are used the need to link them together increases.

When I think about these issues I usually split it down it a number of areas.

• Protocol Transformation
• Routing
• Data Transformation
• Enrichment
• Orchestration

There are many challenges in this work but increasingly the technologies such as WebSphere Message Broker and Biztalk are helping to resolve some of them. These tools, helped by the move to more standards based network and transmission protocols are solving the Protocol Transformation issue, such as converting between different network protocols. For example there is less and less SNA or other proprietary protocols; TCP/IP is almost ubiquitous.

They are also getting better at routing but only in the last few years have they really addressed the bi-directional nature of request-reply routing. It is okay to route a message, possibly fanning it out on the way, but getting all the answers back together is still a challenge.

Orchestration is where a single message, or transaction, can exist for extended periods of time. It may move in and out of several applications during the life of the business process being transformed and routed along the way. This is again being address through the business process management tools which are now at the point of being industrial strength.

Data enrichment is where the message passing through the integration architecture has something added to it. For example, the message coming in may have a product code but not a description. If this is needed by another application before it can process the message then the integration solution can lookup the details and add them to the message.

It is worth mentioning that these tools can be very expensive. While I have used these sorts of tools on many occasions it either requires extensive stakeholder management, or fortunate commercial circumstances, to get them adopted. They really require a strategic investment to make them happy in anything but the most cash rich and technically savvy organisation.

However the challenge remains as the demand for integration continues to rise while the technologies solved the issues above we are still left with the really thorny one - data transformation. This is the killer issue. This is the one that takes the most time and effort to solve; this is the one that you cannot take your eye off. As soon as you ignore it, it will wreck your nice, elegant architecture. To be fair to the technology firms it is also the one which is more specific to your problem, making it more difficult for them to provide generic solutions.

In the next article I will look into this issue in more detail.

To be continued.