Integration - not only a topic of technology
If you’ve ever tried to get more than a few IT systems to communicate with each other, you’re familiar with all the problems involved. Over time, you end up with an almost organic web of interdependencies. Obscure, poorly documented point-to-point connections (or interfaces), involving a bewildering collection of formats, protocols, and technologies, make up an infrastructure that only the most courageous dare tamper with.
One of the reasons for this is that most of the integration work has been done as part of the various projects dealing with developing and implementing the systems being integrated. This way, no coherent view on system integration exists. Each interface will be designed and implemented from scratch and choice of methods and technology will be entirely up to the people involved in the different projects.
As more and more demand is placed on the IT infrastructure to be flexible and responsive to new business demands (agile is a popular buzzword for this), the spaghetti approach becomes less and less satisfying. And if you’re going for SOA (Service Oriented Architecture), the importance increases even more.
Towards a Solution
Some years back, a way of dealing with this problem emerged in the shape of an architectural mode called Hub and Spoke. This architecture was built around the notion of having one central piece of software (the Hub) deal with all the complexities of inter-system communication. A new class of software, called Message Brokers, entered the stage, providing the technology to build those hubs.
Deploying this kind of middleware is, in itself, not a guaranteed way to integration nirvana. Although you now have a bunch of software components that can help you connect your systems with each other, nothing stops you from keeping the point-to-point mind-set, and you may end up creating a somewhat more coherent and smaller bowl of spaghetti. It will still lack the consistency that is required if you want control and transparency.
The Enterprise Service Bus (ESB)
These days, we refer to the hub as the ESB, or Enterprise Service Bus, and we clearly define its responsibilities:
Connectivity. The ESB should be able to transport data using one of a clearly defined small set of protocols, such as HTTP or some message queuing protocol (e.g. JMS).
Protocol Conversion. The ESB should be able to convert data sent over one protocol, so it can be forwarded over another.
Data Transformation. Different systems will send and receive data in different formats, such as fixed length flat files or XML. The ESB should be able to transform (or map) data freely between such formats.
Routing. A key feature of any sound integration architecture is loose coupling, meaning that the sending application should know as little as possible about the receiving application. Therefore, no data is sent directly between sender and receiver, but is always routed by the ESB.
These activities are collectively called Message Mediation. As simple as it may seem, designing and implementing an ESB around these four key capabilities is complex and requires careful design. Without a solid methodology, the full potential of the ESB will not be realized.
Methodology Key Features
After having completed a number of integration projects, you come to the not-so-startling conclusion that a lot of the stuff you do is done in pretty much the same way regardless of the systems involved. You map from one format to the other. You connect the ESB to various systems by reading and writing files in directories, or reading and writing records to database tables. You document your work. You deploy similar artifacts in similar ways to the same environments (e.g. test environment and production environment).
A sound methodology should capture those general components and make them available to all people involved in integration projects. Examples of such components include reusable software, document templates, procedures, and roles.
If you think you’ve seen this before, you may well be right. RUP (Rational Unified Process) is an example of such a methodology. RUP, however, is an all-purpose software engineering discipline, not immediately applicable to systems integration work (or any other specific software engineering endeavor, for that matter). What we focus on with Baseline are the specific features of a systems integration development methodology.