1.       Scope & purpose

This document intends to provide an overview of Enterprise Service Bus (ESB). It also identifies high-level requirements and then goes on to propose a typical architecture for such an Enterprise Bus.

2.       Introduction

Last few years has seen a tremendous growth in business integration. Companies have focused on achieving this integration as quickly as possible, while at the same time safeguarding the return on exiting IT investments as much as possible. In past the problem has proven to be a dual edged sword; with the need to offer better and efficient service in the ever changing business world pushing for end to end integration and the ever increasing cost of available integration options pulling in reverse direction.

Recent years, however have seen emergence of standards for nearly all touch points in integration domain and have greatly simplified the enterprise integration problem. Enterprise Service Bus is the culmination of these developments and an ideal candidate for an enterprise integration backbone. This document captures the definition of ESB in terms of its functions and characteristics and attempts to lay down a possible high-level architecture of such an Enterprise Bus.

3.       Functions of ESB

The concept of ESB has evolved out of the principles of Service Oriented Architecture (SOA). To put it simply, SOA is a federation of loosely coupled services delivering a certain business solution. Loose coupling means that the connection can be established without being tailored to specific functionality embedded in the applications to be connected. Loose coupling is in sharp contrast to existing enterprise architectures in which tight coupling of resources became the primary means of delivering operating efficiencies.

SOA imposes certain requirements on an Enterprise Service Bus; these requirements are captured in the following sections

3.1.       Transport Services

The communication between these loosely coupled services can be as simple as marshalling data or as complex as involving invocation of complex business rules within these services.

For SOA to be effective loose coupling is the key implying that each service should expose interfaces that are well defined and consistent and services are self-contained.

Both of these requirements can be satisfied by wrapping the service creating a standard abstraction or insulation layer on top of application offering the service hiding all its details about the implementation language, operating system or the platform it is running on.

Application can then appear on the network as a set of services and it is possible to plug all these services into a single information bus. The process of integration can then be turned into the software equivalent of building a computer: i.e; merely plug different applications into the bus, and presto, they can all talk to each another and all possible sets of integrations are possible.

However for this seamless “plug and play” integration to be possible the Transport Services must meet certain pre-requisite:

Transport Language: In order for a service-oriented architecture to really take off, the language used to wrap the applications must have universal acceptance. This interface language must support both simple exchanges of data as well as remote procedure invocation. It should also be flexible enough to absorb any changes in the underlying application interface definition while exposing a common client interface to connect to these services. Having a homogenous mechanism to connect to a service allows enterprises to leverage the ‘language’ knowledge across different set of services, making it a repeatable experience.

A widely accepted XML based Web Services Definition Language (WSDL) is a prime candidate for such an interface language. Another possible albeit a less common option is J2EE Connector Architecture (JCA), which can be used to encapsulate and expose any EIS system including ERP, mainframe transaction processing and database.

Legacy systems are of great importance to any integration solution, due to the fact that a lot of a business value is currently contained within legacy operations and components. Perhaps top of the list for these legacy components are leading transaction-processing environments such as CICS and IMS. Any company with an IBM mainframe is almost certainly using CICS or IMS to run a major portion of its mission critical operations, connectivity to these systems and the interfaces exposed by them therefore is also a desired (although not required) feature of an ESB

In order to keep abreast with the emerging standards, support for WS-Routing and WS-Referral is also a desired feature in any best of breed ESB. WS-Routing is a simple, stateless, SOAP-based protocol for routing SOAP messages in an asynchronous manner over a variety of transports like TCP, UDP, and HTTP. With WS-Routing, the entire message path for a SOAP message (as well as its return path) can be described directly within the SOAP envelope. It supports one-way messaging, two-way messaging such as request/response and peer-to-peer conversations, and long running dialogs. WS-Referral is a protocol that enables the routing strategies used by SOAP nodes in a message path to be dynamically configured.

Transport Channel: Just like the transport language, communication channel or the transport bus should also have wide acceptance. Prime candidate for such a bus is the Internet with its HTTP and SMTP protocol.  Other possible options are the most commonly used message queuing software like JMS, MQ Series and TIBCO Rendezvous.

Additional connectivity channels that although are not explicitly required in an ESB implementation nevertheless are rather valuable to have because of their potentially wide application, these include SMIME, FTP, JRMI, CORBA, JDBC, DCOM. And for the EDI world AS1 (Applicability Statement 1), AS2 (Applicability Statement 2) and OFTP (ODETTE File Transfer Protocol)

Transport Paradigm: Given the variety of the transport channels and language options and wealth of services available for integration, ESB must provide some form of communication pipe to support multiple connectivity paradigms. At the very least the pipe must support asynchronous communication, a key to successfully model the essentially asynchronous business world.

A collaborating application may also communicate synchronously, either to respond to a request for a service or to request a service from another application. It might be acting as a consumer of certain set of data (pull) or may be pushing data out to whoever is interested in it. There might also be a need to hold messages (store-and-forward) in situations where the level of availability of the service is variable.

Transport requirement of ESB can therefore be summarized as follows:

Transport Services:

Transport Language:

1. Ability to connect to services with following types of exposed interfaces
   • Web Services (WSDL, SOAP, SwA and DIME)
   • J2EE Connector Architecture (JCA)
   • CICS
   • IMS
2. Support or WS-Routing and WS-Referral

Transport Channel:

1. Ability to connect to services over following types of communication channels
   • HTTP/HTTPS
   • SMTP
   • JMS
   • MQ Series
   • TIBCO Rendezvous
   • SMIME
   • FTP
   • JRMI
   • CORBA
   • JDBC
   • DCOM
   • AS1 (EDI)
   • AS2 (EDI)
   • OFTP (EDI)

Transport Paradigm:

2. Ability to support following types of communications
   • Asynchronous (Publish/Subscribe)
   • Synchronous (Request/Reply or Client/Server; acting either as client or server)
   • Store-and-Forward

3.2.       Transformation and Orchestration Services

In pre-SOA days enterprise systems were complex, monolithic behemoths. The complexity of these systems made them difficult to change. Modification, dictated by every changing business needs were costly and time consuming. IT therefore became the bottleneck in an organization way to adapt to the changing market conditions.

SOA with it collection of loosely coupled software components resulted in what is usually referred as ‘real-time enterprise’; an enterprise which is capable of having its IT infrastructure quickly evolve and adapt to the changing business world around it. An important aspect of ESB is therefore the ability of not only quickly wiring but also the ability of quickly re-wiring the collaborating systems.

One of the major challenges in wiring of varied systems is their usual inability to conform to a standard data structure. Data from one application is typically not easily understood by another service and must be transformed before it is handed off to the consumer. An ESB must therefore offer an adequate level of data transformation capabilities shielding any individual service from the implementation details of any other service.

Transformation is usually a two-step process; parsing the data to a canonical format and mapping it to data format of consumer. An optional validation process might also precede parsing. Mapping can be one-to-one (one source to one destination) or many-to-one, and may also involve enrichment of the source data either conditionally or unconditionally.

Parsing, validation and mapping all require reference data structures, which are usually kept in a shared repository. Transformation engine should therefore have some mechanism of connecting to this repository, at both design and runtime and linking to the applicable data structures and transformation rules/maps.

The implementation usually manifests as something based on XML standards like XSLT, XQuery and XPath, and makes use of XML schemas to validate, parse and transform the data from one format to another.

The proliferation of EDI systems places a substantial demand for EDI data handling capabilities in any integration framework. This although not an absolute must for an ESB still adds significant value to the product if it offers the ability to parse, validate and transform various formats of EDI, also a plus is the capability of parsing and transforming fixed length or delimited flat files.

On an equally important footing parsing and conversion to and from COBOL copybook data becomes an important asset when communicating with Main Frame legacy systems.

Orchestration services provide another part of ESB core functionality. For SOA to mimic a business process it must be able to specify the flow from one service to another. This flow can be either ‘static’ or ‘dynamic’ based on the content or the result from the previous step. ESB therefore must be capable of providing intelligent routing based on the content of the payload being passed from one step to another and the business rules or organizational policies acting on that content.

Another aspect of intelligent routing is the ability to respond and route around failure points in the network, the ability to seamlessly consume a redundant member of a service set, incase of failure of the primary one (either planned or unplanned) or its inability to deliver as per the SLA committed for the process. A possible architecture to achieve this may involve process orchestration services using the SLA Implementation Services to identify the best possible route and the appropriate service producer complying with the process SLA constraints.

Process orchestration service may also need to coordinate with security services to implement service specific data security/encryption and access authorization requirements.

Ability to test out a process before actual deployment is also an important desired aspect of orchestration services. Orchestration engine should therefore be capable of differentiating between a test execution (process ping) and actual process execution and should be able to indicate to the Connectivity Service to either block the transaction commit to the participating services and/or simulate a particular response of if so desired by the test scenario.

Transaction is another important aspect of process orchestration. Transactions are a fundamental concept in building reliable distributed applications. A transaction is a mechanism to insure all the participants in an application achieve a mutually agreed outcome. Traditionally, transactions have held the following properties collectively referred to as ACID:

• Atomicity: If successful, then all the operations happen, and if unsuccessful, then none of the operations happen.
• Consistency: The application performs valid state transitions at completion.
• Isolation: The effects of the operations are not shared outside the transaction until it completes successfully
• Durability: Once a transaction successfully completes, the changes survive failure.

SOA requires the same coordination behavior provided by a traditional transaction mechanism to control the operations and outcome of an application. However it also requires the capability to handle the coordination of processing outcomes or results from multiple services, in a more flexible manner. This requires more relaxed forms of transactions — those that do not strictly have to abide to the ACID properties–such as collaborations, Workflow, Realtime processing, etc. Additionally, there is a need to group collaborating services into applications that require some form of correlation, but do not necessarily require transactional behavior. In the Web Service world the emerging standards like WS-Coordination and WS-Transaction specifications provide a WSDL definition for such coordinated behavior.

ESB can also support transaction management through IMS and CICS connectivity offered by the transport services.

Workflow, an important aspect of orchestration, includes the ability to model business processes that involves human interaction; typical examples are mortgage application and insurance underwriting. Workflow applications are typically long running processes that may span for days or weeks, persistence of the process state then becomes an important requirement to support workflows. In a typical workflow end user interaction with the process is governed by their role and the authorization level assigned to this role, the role-authorization relationship management therefore also becomes a desired feature of an ESB supporting workflow applications.

True to the spirit of adhering to the standards support of BPEL (Business Process Execution Language) a specification that describes a business process is also a desired feature in an ESB.

Transformation Services:

Support Data formats:

1. XML
2. Delimited or Fixed Length file
3. COBOL CopyBook
4. EDI

Validation:

1. Optionally validate the data against the acceptable data structure and raise error in case of non-conformance
2. Ability to connect to shared repository and link to the appropriate data structure for validation

Parsing:

1. Ability to connect to shared repository and link to the appropriate data structure for parsing.
2. Conversion of source data to canonical format for mapping and further processing

Mapping:

1. Mapping of data from one structure to another
2. One-to-one and many-to-one mapping options
3. Ability to connect to shared repository and link to the appropriate data structure and transformation rules for mapping.
4. Enrichment of source data using concatenation/truncation functions
5. Support for conditional mapping.

Orchestration Services:

Process Orchestration

1. Intelligent routing based on the content of the payload and organizational policies/business rules and or on the service provider state/required SLA’s
2. Implementation of service specific data security and access authorization during process orchestration.
3. Runtime linking of process engine to repository of shared policies and business rules
4. Support for ‘process ping’
5. Support for representing the process as BPEL artifact.

Transaction Services

1. WS-Coordination and WS-Transaction implementation
2. Support for legacy transaction through CICS and IMS

Workflow Services

1. Support for long running transactions
2. Long term persistence of process state
3. Ability for end user to interact with the process
4. Integration with enterprise user store for roles and authorization definition or support for such function within ESB offering

3.3.       Connectivity Services/Tools

ESB must quickly and easily enable components to be attached to the bus for use by any authorized services. Various forms of transport that an ESB should support are discussed in section 3.1, this allows a wide breath of applications to readily drop into an ESB and start participating in a business process orchestration. More often there is however also a need to include the legacy applications in a SOA that does not expose one of the standard interfaces, or if it does the architecture calls for standardization to a specific type of transport service. The way this can be achieved is by providing ‘wrapping’ tools. The principle behind wrapping is that software is put in place to surround the legacy component in such a way that from the outside the legacy code looks and behaves like any other ESB component whereas from the inside the legacy code is not aware that it is participating in an ESB solution.

The solution can be in the form of API exposed in widely common languages (Java, C, C#, C++) capable of providing standard abstraction on top of application offering the services.

ESB can also offer tools which complements these API set and make this integration and configuration a very visual experience. The configuration services offered by these tools include but not limited to publishing the service specification exposed by the application, any ACL attached to it, SLA guaranteed by the application, data security requirement of the application etc. Necessary hooks are obviously needed between the Connectivity Service/Tools and the Meta-data Management Service

Working in conjunction with the orchestration service, connectivity services/tools can also contribute to end-to-end integration test scenario execution (Process Ping) by either suppressing the transaction to pass through to the end point or by simulating a particular response back to the consumer of the service.

Connectivity Services/Tools:

1. Java, C++, C# API
2. GUI based tools for
   1. Wrapping legacy system interfaces exposing them as one of the ESB component
   2. Publication of interfaces exposed by the application to the central repository.
   3. Specification of ACL attached with the application, SLA constraints, data security requirement and publication of these attributes to he central repository
3. Support for test scenario execution by suppressing/simulating response from the service provider.

3.4.       Meta-Data Management Service

An important characteristic of ESB is to provide the ability of quickly finding and connecting with the most appropriate resource, where the definition of appropriateness vary from process to process and depends on factors like scalability, reliability, performance, SLA commitment, data security /encryption and security requirements of the service. The need therefore is for a shared repository, which not only holds the information necessary to connect and consume the provided services but also persist the meta-information expressing the resource appropriateness. This shared repository can also act as the storage for global organization policies/business rules and common vocabulary that are common across all the processes.

This repository is just but one component of the Meta-Data Management Services, full gamut includes the corresponding enabling technologies providing ability to search the repository for the appropriate resources, automatic management of service location and interface definition within the repository and hooks into the Management and Monitoring Services to obtain the real-time statistic on the offered services. The meta-data being managed by this service may also include information about services payload data structure.

The consumer of the Meta-Data Management Service includes the Orchestration Service; which may use the stored information for intelligent routing, payload encryption and ACL, the Connectivity Service; for dynamic run time binding to the services and Monitoring and Management Service; for comparing the actual SLA against the committed ones.

Use of WS-Addressing to provide transport-neutral mechanisms of defining Web services end points is another area that an ideal ESB should support.

Meta-Data Management Service:

1. Repository for meta-data like
   1. Service location
   2. Service Signature
   3. Payload data structure
   4. Committed SLA
   5. Data Security/Encryption requirement
   6. Access Control
2. Meta-Data Management Service
   1. Standard hooks to the Service Bus
   2. Search ability against defined parameters
   3. Hooks to the Management and Monitoring System for real time statistics
   4. Access functions to service attributes (SLA, data security/encryption requirements)
3. Support for standards, like WS-Addressing

3.5.       Security Services

By its very nature, SOA presents a unique challenge on ESB as far as the security services are concerned. The need to connect variety of application and systems, which in most likelihood are fundamental to business, both within and across enterprise boundaries means that security is of major importance. There is also a realistic instinctive fear that improved connectivity means that sensitive information might now be intercepted or modified by malicious users. This is therefore a fundamental issue that any best-of-breed ESB should address.

Security concerns with integration projects centers around following areas:

• Access to component or services
• Exposure of information as it passes from one component to another.

The universal connectivity theme behind ESBs naturally causes concerns over access to the components hooked into the ESB, particularly if these are being made available to external companies. This makes it very important to restrict usage only to those who are authorized. There are number of security frameworks available in the market that can be used to carry out this authorization task and to control access based on the level of authority granted by the system, multiple security frameworks must therefore be understood and implemented by ESB. Commonly used framework for authentication and authorization include; use of Security Assertions Markup Language (SAML), LDAP, Active Directory etc. An ESB must be capable of either incorporating such mechanism within its offering or hooking to a corporate authentication and authorization system. Other authentication systems commonly used include Kerberos and X.509 address authentication. The framework also needs to expand and include mechanism of specifying and implementing service authority to access another service.

The issue of exposure of information is also a sensitive one. The concerns here are two fold – has anyone been able to see the information as it passed along the connectivity pipe from one component to another, and has anyone been able to alter it as it was transferred. To address the visibility of information, the most common tactic is some form of encryption capability so that sensitive information can be encrypted before transfer and decrypted on receipt. The payload can be encrypted in its entirety using SSL. However in some situations, this may not be sufficient. For instance different recipients may be authorized to view only certain attributes of the transmitted data. Use of SSL has its shortcomings where intermediaries are involved between source and final destination of the data, the data might need to decrypted increasing the chance of it intercepted and maliciously modified. SSL based security at best addresses point-to-point security. More complex solutions need end-to-end security baked in. Other security standards commonly in XML data world are XML Signature and XML Encryption. XML Signature and XML encryption describes ways of encrypting and signing the whole or part of XML messages.

WS-Security an emerging XML based security standard specifically for Web Services addresses how to maintain a secure context over a multi-point message path. WS-Security and WS Security Addendum addresses security by leveraging existing standards and specifications, it adds to existing specifications a framework to embed these mechanisms into a SOAP message.

Security Services:

1. Authentication and authorization to access component and services
   1. Ability to hook up to corporate authentication and authorization systems e.g; LDAP, Active Directory
   2. Support for SAML, Kerberos and X.509
2. Limiting information exposure and malicious alteration
   1. Support for transport level encryption like SSL
   2. Support for XML Encryption and XML Signature
   3. WS-Security framework

3.6.       SLA Implementation Services

Given that an ESB is likely to become any IT cornerstone of business operations, it is of paramount importance that an ESB provide the optimal levels SLA, ensuring guaranteed performance, reliability and scalability under all operating conditions and changing IT structure. Changes in the IT structure can either be planned or unplanned. Unplanned changes may include performance spikes triggered by either high traffic volume or transmission of very large amount of data. SLA implementation service has to cope with these situations while maintaining an acceptable level of performance and response time. Ability to route around failure point is another important ESB characteristics, as discussed earlier this intelligent routing by the orchestration engine requires the SLA services to identify the best possible route and possible also a redundant source of service.

SLA services therefore must be capable of actively monitoring a given ESB network, gather matrices and compare them against the committed SLA of the process. It then needs to coordinate with the Transport service to dynamically modify the route and/or destination of the request. SLA services also need to interact with the Monitoring and Management system providing them with the relevant data for generating alerts. Support for WS-ReliableMessaging, which describes a protocol that allows messages to be delivered reliably between distributed applications in the presence of software component, system, or network failures, is also a desired feature of SLA services.

Design time contribution of SLA Services include building SLA model as part of the process interaction by evaluating requirements and reviewing them against SLA constraints of different interfaces and identifying the points of failure and offline services and tagging them as unavailable in the meta-data Repository.

SLA Implementation Services:

1. Build SLA model as part of the process interaction by evaluating requirements and reviewing them against SLA constraints of different interfaces
2. Identification of service status and flagging offline services in the Meta-Data Repository
3. Monitor SLA compliance for the integration processes
4. Dynamic routing of the process against failure points in the network and failed nodes, handling fail-over and load-balancing
5. Feeding the Monitoring and Management System with data to generate alerts
6. Support for WS-ReliableMessaging

3.7.       Deployment Services

ESB solution must be deployable on widest variety of platforms and in location most appropriate for the situation. In some cases, deploying on a service provider or consumer is appropriate and desirable. In other cases deploying in the network between service providers and consumers is more appropriate.

An important facet of a SOA is to quickly embrace and adapt to a changing business need. However these intentional IT changes may cause unexpected impact to the service network. The planned changes can include introduction of new services, bring system up or down for maintenance or change in either the location or contract of the service. Impact of each of this change although manifest itself differently from slow response to outright service outage the end result on the business translates in dissatisfied customers, lost orders and unrealized. revenue. The need therefore is to effectively manage this change to eliminate or minimize the impact on the business.

In order to effectively deal with this problem, there must be a facility within Deployment Services to:

Describe the structure of the service network – the interdependencies of the various applications collaborating to deliver the business solution.

Assess the impact of the change – identifying the potential performance degradation of introducing new application and/or recognizing the ‘ripple effect’ change in moving or changing the contract of an existing service

Insulate the application from the negative impact while still allowing the change.

Deployment services must also be able to provide information to SLA, Meta-Data and Monitoring Services about a possible outage of system component so that suitable mitigation measures can be taken to minimize the effect on the business operation.

Deployment Services:

1. Support for distributed deployment
2. Ability to access the impact of change and insulate the application from negative effect of this change
3. Hooks to the SLA, Meta-Data and Monitoring services

3.8.       Monitoring and Management Service

Service change, as mentioned in section 3.7 might be planned, but it can also occur suddenly or unexpectedly, translating to same effect on business as dissatisfied customers, lost orders and unrealized revenue. The need therefore is to quickly detect, identify and rectify bottleneck in the service network before it explodes into a much bigger and unmanageable problem.

In order to effectively deal with this problem, there must be a facility within Monitoring and Management service to:

Detect that there is a service network problem – that the network of services is deviating from its normal operation range.

Identify the root cause of the problem – which application, service and operation is causing the bottleneck

Insulate the downstream applications from the root failure – minimizing the cascade of the problem to rest of the system and provide opportunity to address the underlying problem.

Monitoring and management service is usually the central hub of all activities once an application has been rolled out in production. It therefore should offer single point of access to all the operational and management data related to the processes and the components of the ESB. Access to this management hub can either be limited to within the corporate network or it may also be available from outside the firewall over the web. This service should be capable of providing the business view of the transaction as well as process mapping on the physical component. System should be capable of providing real time statistics on the state of process, committed and real performance metrics of the participating applications and should also have the ability to perform on demand or schedule ‘health check’ of the constituent applications and components.

Monitoring and Management system must also allow the ability to define rules for alerts and annunciations and should also have the ability to capture and report on historical logs of user specified metrics and transactions. It should also allow the ability to search these historical logs and transaction records and optionally replay/resubmit them.

Another desirable feature is hooks to and from other enterprise monitoring and management systems like Tivoli etc.

Monitoring and Management Service:

1. Ability to test a process integration scenario on a routine schedule to ensure uptime of all resources (Process Ping)
2. Provide business view for all active processes and instances
3. Provide administrate/technical view for active processes and instances
4. Consolidate network, server and process state databases to provide a unified view process network

1. Allows monitoring data corresponding to process, SLA and process performance related information.
2. Offers the ability to define rules for alerting and provides visual indication and historical logs of there alerts
3. Monitor the health of applications participating in the process by conducting scheduled or on demand ‘process ping’
4. Hooks to and from other enterprise management systems
5. Historical logs of measured metrics
6. Transaction history, search and resubmission capability.

3.9.       Administration Services

SOA being the pinnacle of decoupled architecture, it is only but natural that ESB itself should be modeled around the same principles and ideas. It may also then become necessary to segregate the administration and management responsibilities of various ESB components along defined hierarchies. The Administration Services can implement just that kind of mechanism by providing a single point of ACL controlled access to all management functions of various ESB services.  It should be possible to specify the granularity of access either vertically across each service down to individual functional level or horizontally across process boundaries; i.e. grouping the management of ESB services on a per process basis. System should also allow implementation of a tiered hierarchy of roles for association with the management functions.

These administration services should also be able to interface with the corporate user store for obtaining user credentials and /or the roles assigned to them as well as their authorization levels.

Administration Services:

1. Single point of controlled access for all management functions
2. Vertical or horizontal specification of the controlled access.
3. Support for tiered ACL.
4. Ability to hook to corporate user store for roles, authorization levels and user credentials.

3.10.   Presentation Services

Presentation services are responsible for providing the user experience. Best of breed ESB typically has a single point of interface, which consolidates all user accessible functions. Access to these functions is managed by the Security and Administration services and offers a customized view to the user based on their authorization level.

An ESB can offer multiple flavors of presentation service ranging from thick client with strictly local access to web interface for remotely interacting with the system. Some of the basic function of presentation service may include

1. Transformation and Orchestration
   • Rich set of process widgets/toolset providing a visual process orchestration and data transformation experience.
   • Support definition of “macros” or process/sub-process templates and their reuse.
   • Allow to real time process debug, break point, tracing facility and payload content inspection facility in combination with process ping
   • Import and export of process definition
2. Connectivity
   • Visual ‘wrapping’ of legacy interface
   • Ability to specify process SLA and relate them to payload content.
3. Meta-Data Management
   • Ability to View, search and modify the end point definition, polices and vocabularies
   • Schema creation, editing and managing tools
4. Security Services
   • ???
5. SLA Implementation
   • ???
6. Deployment
   • ???
7. Monitoring and Management
   • ???