Going Beyond Project-Driven SOA

However, most of the organizations we've worked with are taking a project-driven approach to SOA - namely, addressing tactical integration and composite application requirements with SOA tools. So where does this leave them? Fortunately, companies don't necessarily have to do SOA for the same reasons, or do enterprise SOA. Each of us can do SOA in our own way and still benefit from it.

How?

You need to answer two key questions:

  1. Why are you doing SOA? This question isn't limited to those who haven't dipped their toe into the SOA pond.
  2. How are you going to do SOA? How are you going to make sure you don't do it willy-nilly?

This article reflects our experiences - synthesized from working with CIOs, IT managers, and architects - of the ways SOA practitioners are addressing these issues.

Why Are You Doing SOA?
When architects and developers talk about SOA, they discuss Web Services-enabling applications; interoperability through standards; reducing integration pain; and the ease of use of new-generation SOA tools and platforms, compared to using the legacy integration brokers. On the other hand, business people often think their investments in SOA are delivering agility, reduced IT costs, and reduced business risk - if IT people have articulated this clearly to them. If not, they see SOA as the panacea for the long change cycles and lack of responsiveness of IT. As IT people, how do we ensure that we can deliver on the objectives of the business and ensure that expectations are adequately managed? Or, how do the aforementioned magical SOA benefits actually manifest themselves? Surely not just from buying the right tools, however important that is.

Let's look at SOA benefits from a business perspective and see how IT helps make them a reality:

These SOA benefits are shown in Figure 1. These business benefits can be mapped to supporting IT benefits, which, in turn, are enabled by what we term SOA enablers. Each of these enablers is realized through tasks, activities, or technologies. Figure 2 is the overall picture of the IT benefits, SOA enablers and supporting tasks, activities, and technologies.

You are doing integration between your customer relationship management (CRM) and your financials application with an SOA platform, and your aim is to enable greater business agility, which depends very heavily on how flexible its IT is. IT flexibility, in turn, is determined by the extensibility of existing applications and systems - how easily they can be extended to incorporate new business requirements.

Extensibility in applications isn't enough because tightly coupling applications can have a ripple effect of changes and is, in most cases, very costly and time-consuming to manage. So, managing the impact of change becomes a key enabler of IT flexibility. Designing the right level of abstraction and an appropriate service-layering architecture can help reduce the cascading effect of change. Loosely coupled, and even completely decoupled, applications help reduce the impact of change. An important factor in determining how changes are handled is the appropriate use of tools and technologies. Especially in an SOA environment with a wide set of tools and infrastructures that may provide overlapping functionality, it's important to understand clearly the design patterns for which each tool is most suitable.

Beyond IT flexibility, agility is also determined by how quickly and cost-effectively new changes and requirements are handled without sacrificing quality. Moving from creating thousands of lines of code - the traditional approach - to more of an assembly model can dramatically reduce the overall time and cost required for delivering new applications. In a service-oriented approach, this typically means reusing services from a well-defined service portfolio, orchestrating them, using the right set of tools to provide the required business functionality, and capturing business policies using a business rules engine (see our previous article "Building Flexible Business Processes Using BPEL and Rules").

Looking at Figure 3, you can see the SOA enablers (purple) and tasks, activities, and technologies (pink) needed to deliver agility for SOA projects. Aspects such as canonical data models and service portfolios are often missing from integration projects - limiting business agility.

The more SOA enablers and supporting tasks, activities, and technologies are in place, the more closely you'll realize your higher-level IT and business objectives. Many organizations use this benefits map to plan their SOA journeys and ensure that both business and IT are on the same page about what investments will actually deliver

It's not enough to decide on SOA - you also need to know why you're following this path - hence the need to be really certain of why you are doing SOA. One large inter-governmental organization we spoke to recently said that the key SOA benefit it was looking for was easier information exchange enabled through interoperable systems. From its perspective, the key SOA benefit it wanted was interoperability - and it was single-mindedly focused on that.

Clearly, the more SOA benefits you want to attain in a given time frame, the more investment, from a financial and organizational perspective, is required. Of course, everyone wants agility, but what's required in the short term?

What Approach Are You Going to Take to SOA?
Once you've decided on what SOA will mean to your organization, you should put together a SOA strategy and supporting roadmap for delivering on these SOA objectives.

Based on our experience with numerous customers that have undertaken SOA initiatives, we have identified three distinct SOA strategies:

  1. A project-driven approach in which SOA considerations are confined to the context of an individual project.
  2. An infrastructure-driven approach in which the focus is on building out the utility/foundation services and an SOA platform that is reused across projects.
  3. An enterprise-driven approach in which there's a wider view to leverage SOA for business responsiveness that typically involves building a portfolio of reusable services.

The three approaches are depicted in Figure 4. The project-driven approach can deliver successful SOA projects you can incrementally build on as long as the right level of buy-in and appropriate governance and portfolio planning are included; don't be fooled into thinking that anything other than an enterprise-driven approach is inadequate. An example of this is a Fortune 500 company that undertook a $250,000 pilot project and used it to learn valuable lessons that helped it in subsequent enterprise-driven SOA initiatives.

Let's look at each of these approaches in greater detail.

Enterprise-Driven Approach
Organizations engaged in the enterprise-driven approach typically have an active enterprise architecture group defining architectural standards, an SOA center of excellence for disseminating best practices, buy-in and involvement from businesspeople in the SOA initiatives, a managed portfolio of services, and governance focused on delivering business results through enterprise architecture and SOA as a discipline within EA.

Enterprise-driven SOA, or rather the capabilities it epitomizes, are a necessity for projects that involve process automation and span application silos and lines of business or that are aimed at consolidating applications across various departments. The enterprise-driven approach requires significant upfront investment to establish the required SOA framework. Because this outlay can engender a major ROI justification challenge, few companies are engaged in it.

What is a distinction between enterprise-driven and enterprise wide approaches? In the enterprise-driven approach, many of the planning and architectural aspects normally associated with enterprise wide SOA - such as canonical data models, architectural standards, blueprints, a service funding model, service portfolio planning, and service and process owners - are restricted to a domain in the enterprise. The enterprise is segmented into non-overlapping domains, each of which establishes its own boundary. A domain typically crosses multiple application silos and provides a significant and meaningful context in which services can be independently standardized and governed, architectural standards are defined and enforced, a service portfolio can be planned and built, and there is consensus (within the domain).

The key to taking the enterprise-driven approach is to adjust the scope of the domain so that it's manageable and doable. The domain may end up being a department, or multiple departments, but the key consideration is that it's feasible and you can pull it off. For more on the domain inventory pattern, click here

An enterprise-driven approach provides the most benefits because it's enterprise (or rather domain)-focused. This broad focus means that the initiative will be complemented by both planning and actual implementation work to ensure that services are created within an enterprise service portfolio plan. A service portfolio is defined as a collection of related services that are independently standardized and governed. Services in this context are treated as an enterprise resource, whereas units of business logic have typically been designed to be used in the application silos in which they were implemented. Creating a comprehensive service portfolio that includes business services - encompassing entity services, such as a customer service, with associated create, read, update, and delete (CRUD) operations, and task services, such as credit eligibility - and utility services will go a long way toward enhancing the ease and speed of new application development. Application development efforts can be substantially reduced because development activities in an enterprise-driven SOA organization would involve assembling services to provide business functions, rather than writing large amounts of new code.

Unfortunately, enterprise-driven SOA can fail if the organization is not aligned with the SOA initiative. For example, an organizational culture that doesn't encourage sharing can severely inhibit the expected SOA benefits. A comprehensive governance framework that addresses technology, project execution, data, architecture, operations, finance, portfolios, and people and skills is a vital foundation for the overall SOA success.

When we talk to customers, we typically refer to the enterprise-driven approach as requiring capabilities at Level 3 of the Oracle Level 5 SOA Maturity Model. For those of you who are interested in knowing the kind of capabilities required for adopting an enterprise-driven approach, you may want to refer the SOA Maturity Model found at Cheat Sheet. Suffice it to say that a lot of stars need to align to execute the enterprise-driven model successfully. To increase the chances of success with the enterprise-driven approach, cast the domain net wide enough to cross application silos but narrow enough to provide a meaningful context.

Infrastructure-Driven Approach
The infrastructure-driven approach primarily delivers benefits only for utility services (also known as foundation services). This is because in the infrastructure-driven approach, services built in a reasoned and well-planned manner are meant to provide a solid foundation of software utilities, such as logging, exception management, and e-mail. Generally, this means that strategic portfolio planning as well as architectural and design policies will typically be limited in scope to building out and deploying these foundation services. It's very likely that services built outside of this initiative still won't be governed by these policies, standards, and best practices, meaning reduced overall interoperability, and lowered reuse.Good candidate projects for the infrastructure-driven approach are integration (especially dealing with master data management), consolidation, and mainframe migration or modernization. To ensure success with this approach, it's important to standardize on the technology platform on which these services will be delivered. Although using industry standards can help provide level, out-of-the-box interoperability, it's also vital to consider following well-defined design standards to help sustain it across changes and evolution in industry standards. One of the most important aspects is to instill the practice of designing for reusability. Of course, all this requires support from management, especially in terms of budgeting for building out the SOA platform, utility service development, and an operational model for services that takes reuse into account - after all, there's little point in getting reuse if the reused services fail to perform! Because these types of services don't directly provide business value or benefits, it can also be a challenge to justify ROI beyond reuse of a common technology platform.

Fortunately, the infrastructure-driven approach will provide valuable experience in building, deploying, and managing shared resources. You'll also learn lessons about issues and challenges involving access control, security, availability, and overall quality of service in a more controlled environment. What you won't get is a high degree of reuse and interoperability across-the-board in services created by non-infrastructure-based projects - yes, you'll get your e-mail or logging service use everywhere, but you may end up getting four different customer services, each with different operations and developed by different groups. This redundancy arises because none of these customer services were designed to be delivered as reusable services. As a result, the cost savings will be limited to reusing utility services and a common SOA platform. As mentioned earlier, because the scope isn't very broad, any projects done outside this scope won't benefit from the governance, best practices, and other elements employed for infrastructure-driven SOA.

The infrastructure-driven approach requires capabilities at Level 2 in the Oracle SOA Maturity Model. Refer to the Cheat Sheet to learn more about what's required.

The Project-Driven Approach
As the name suggests, all the deliverables are driven by requirements specific to a project. Typically, involvement of the business is limited to mandating its needs and rarely includes collaborating with IT to define business processes and services.

Examples of project-driven SOA typically include integration scenarios. For instance, consider an order-to-cash business process that requires data to be moved between a CRM system and an enterprise resource planning (ERP)-based order management system. This type of integration is not new and has been around for years using a range of technologies, from file-based batch integration to message-oriented middleware (MOM)-based integration. They are all rigid, point-to-point in nature, and highly proprietary. All of these factors meant that these solutions are costly to deliver and maintain. Using SOA-based tools such as BPEL engines or Enterprise Service Bus (ESB), it's much easier to build sustainable and flexible standards-based integration. Note that benefits derived from these projects are generally limited to ease of development and standardization of integration.

If a project uses a more standards-based SOA tool and platform to perform data integration, it's likely that any services created are very specific to the project. If the project creates any reusable assets, it's probably by accident, especially in the absence of appropriate planning and governance. Even if none of the services are reusable, these projects help establish a technology foundation as well as enhance IT skill sets.

The downside of the project-driven SOA approach is that it generally creates a new batch of silos, albeit with a service-oriented flavor. Instead of reducing the long-term integration effort, it lets integration become an ever-present headache. It's obvious that to be able to capitalize on your project-driven SOA successes, you need to plan on transitioning either to infrastructure-driven or enterprise-driven SOA. Which path you take will depend on your capabilities, your organizational readiness, and the degree of buy-in from business and management. However, project-driven SOA is a good place to start. David Chappell, Oracle's vice-president and chief technologist for SOA, touches on the importance of project selection in his blog.

The project-driven approach requires capabilities at Level 1 of our SOA Maturity Model. Refer to the Cheat Sheet for more details. Generally Level 1 capability requirements aren't onerous.

What Will Each of These Approaches Get You?
Given these three distinct approaches to adopting SOA, it's essential to be aware of the approach you're taking and articulate that to the line-of-business IT team. So now that we know the key SOA benefits from both the business and IT perspectives, as well as the enablers of those benefits, we can map them to the three SOA adoption patterns outlined previously. This mapping is shown in Figure 5.

The table makes it clear that maximum benefits are gained through the enterprise-driven approach. The corresponding benefits of the other approaches are more limited. The takeaway from Figure 5 is that it's hard to get reuse (and we've seen this in practice) with the project-driven approach, which typically presupposes little central planning or coordination across projects.

Of course, many of you who've done some projects with SOA tools will say there's no need to do anything beyond the project-driven approach to get a catalog of services (that are actually reused): just buy some tools, get each project to use them, perhaps get the EA group (if you have one) or architects' collective to define some interface standards, and then set everyone loose to build their own services. With this approach, you'll have hundreds of services before you know it. This service proliferation typically means that you'll end up with a bunch of services that overlap, because there's no central planning that addresses which services should be built, when, and by whom (governance). If that's what your vision of SOA is, that's great, but be aware of which of the SOA benefits you will typically not be able to deliver.

Without a planned approach that pays particular attention to reuse, you end up with several siloed applications, even though they were supposed to be built by use of services. The reason they are siloed is that the overarching service planning and governance aspects were absent, which resulted in production of very few or, in many cases, no reusable assets (more on this later). In other words, if projects are charged with producing specific reusable services in predictable ways, is there any hope that they will deliver anything of use to anyone else? In the long run, these service-based siloed applications may end up adding to the overall IT maintenance cost, rather than helping reduce the cost via reuse and increased flexibility.

The choice is really clear: let each project do its own thing in its own way without any oversight whatsoever, albeit with modern tools built for SOA (which isn't totally bad), or look for a better way to do those projects so that they end up contributing assets to the department/enterprise. And that better way involves incorporating some planning early on to identify reusable business services.

Toward Reusable Business Services
If our goal is to elevate reuse to a point where the business can directly benefit from our efforts, we need our projects to deliver business services. Thinking about and planning for business services is also important for capitalizing on your success with the initial project-driven SOA initiatives and transitioning to the next level: either infrastructure-driven or enterprise-driven SOA.

What are business services?

There are several definitions, and they vary widely, depending on the context. In the context of SOA, services that provide functions related to one or more business processes and can be described with purely business semantics can be classified as business services. These types of services are typically composed from other, more fine-grained services, such as utility services. As we all know, a service is a logical grouping of semantically similar operations. For example, you may have a service named Customer with CRUD operations. Business services typically provide higher reuse value than utility, entity, data-oriented, and connector services.

There are essentially two ways of getting business services: (1) proactively prepare for them by putting together a service portfolio plan or (2) live by the law of the jungle - namely, let each IT group build services that populate the repository and can be reused by anyone. Although the latter requires no coordination between projects or departments (a huge advantage), the problem with it is that you invariably end up with an excessive number of services that frequently overlap and may not be useful beyond the project that implemented them. This service proliferation ultimately increases the costs and complexity of governance greatly. For example, you may end up with several services for customers, each of which has the following attributes and necessities:

Taking care of these duplicate responsibilities is a major headache and a costly exercise. So as we've been describing, you have to choose between pain now (planning to avoid the overlap and redundancy of services) or pain later, in the form of much higher governance costs and complications.

At a recent Gartner conference, a presenter spoke of an extreme case of duplicated effort. A major banking firm had a CIO with four direct reports, and he had mandated that they move toward SOA, having made it clear to everyone that he was retiring in two years. The four direct reports were in fierce competition for his job and, as a result, didn't work together or even communicate well between groups. Instead they wound up building four distinct SOA-based solutions that solved the same set of problems. Since then the CIO has retired, two of the direct reports moved on to other companies, and now the bank is in the process of consolidating the four SOA projects into one, paying way more in terms of time and money to undo the three extra SOA projects than it did for the initial investment.

Of course, one of the ways of ensuring that you don't get service proliferation is to put together a service portfolio plan - a key activity in the enterprise-driven SOA approach. Service portfolio planning is the iterative process of developing an enterprise-wide service portfolio. A service portfolio planning exercise helps in identifying and normalizing candidate services. Doing this at the enterprise (or domain) level with major involvement of the business, or at least within a significant domain of the organization, assists in finding services and their constituent capabilities that are aligned to where the business is going and are hence more likely to be reused in multiple projects. A service portfolio plan also helps in creating service blueprints that projects can use when building new services.

Typically, the first iteration of this exercise produces a portfolio of services that are mostly conceptual in nature. As projects get delivered, they will build out the services they need. These new services will conform to the specifications in the service portfolio, thereby ensuring that they're not project-specific. Over time the service portfolio will migrate from a purely conceptual exercise to one populated with living, breathing services. Creating a service portfolio plan is crucial to realizing key SOA benefits such as reuse, ease of development, and enhanced ROI. Please don't ignore it. Or, rather, don't ignore it for long.

Each of the three adoption approaches has its own pros and cons: the enterprise-driven (or domain-driven) approach requires buy-in and organizational alignment up front, but the services developed in this context will be reused without much effort and cost, ultimately leading to a reduced governance burden. The other approaches have less upfront impact but can impose a greater governance burden later on - a fact you have to live with.

Now Back to the Real World
A reality that all of us, however sold we are on SOA, have to live with is that most organizations today are taking the project-driven approach to SOA, typically addressing immediate integration or composite application requirements by using a service-oriented approach. How do you execute the project-driven approach to SOA correctly? Here's our checklist:

1. Decide what you need from SOA in the medium to long-term. No doubt you've heard the expression "think strategically, act tactically" applied to SOA. There are many benefits to using SOA tools at the project level (see Figure 5). Not everyone needs to be at Level 5 in the SOA maturity model straightaway, if at all. So assess which SOA benefits best fit your business, understand your constraints (funds, corporate culture) and build a plan that will enable you to deliver those benefits - perhaps even as part of existing projects. An SOA maturity model is helpful not only in pinpointing existing capabilities but also in identifying the end state and what needs to be put into place to get there. So think about what you need in terms of SOA benefits. Do you really need the benefits of the enterprise-driven approach? Are you, or rather, is the business willing to make the investment funds available to make it happen? Not everyone needs to win the Super Bowl and certainly not within 12 months. Be realistic and chart out your goals.

2. If you're taking the project-driven approach, partake in a few activities to get better results. Of course, the priority for any project is to solve the business problem at hand. If SOA helps deliver the project in a better way or results in a more maintainable and flexible system, that's great. But not all projects are suitable for SOA. For projects that are suitable, you can still get some of the benefits of enterprise SOA, such as interoperability, by adopting architectural principles - application reference architecture, canonical data models, agreement on standards (and versions of standards) that will be supported both in projects and across projects - at the project level.

Technology does matter, so select a standards-based, integrated, and flexible toolset that enables you to deliver on the promise of SOA and use those tools in the right way. For example, we have often seen developers get carried away and implement all forms of business logic in BPEL. Finally, avoid restrictive policies. Everyone says governance is vital to SOA, but you really don't need that much of it when you adopt the project-driven approach - we've often seen companies spend many cycles enacting restrictive policies that stunt their progress with SOA. Exert the minimum amount of control required.

3. Projects can contribute reusable services that are enterprise resources. If you're keen to get on the path to building a service portfolio then within the context of individual projects attempt to identify potentially reusable services: either utility services, such as an e-mail notification service; business services such as entity services (a customer service, for example); or business services such as an invoices service that provides operations related to invoice processing. The art of service identification is beyond the scope of this article, but suffice it to say that there are several techniques for identifying services that are bottom-up ("Which of our APIs can we factor into reusable services?"); top-down ("what are the domains in our business and therefore what are some potential services we are likely to need in each domain), or what we call middle-out, which delivers services by modeling several business processes and then aggregating semantically related operations into logical business services. There are also some industry blueprints for services, such as for insurance, although we've found that they offer a good starting point but require much work to realize in usable implementations.

Whatever technique you use, bear in mind that it's best to build services that have immediate use within a project. In other words, don't build a service portfolio for its own sake - such as by service-enabling a bunch of APIs - without a view to which projects the services will be used in and what the future business and, hence, IT requirements will be. Avoid building speculative functionality that may never end up being used or that's built in such a way that it won't be useful later on. Tactically delivered services in which you add reusable functionality run the risk of wasting time and effort and adding complexity while leading to little reuse, unless you know where they will be reused. To avoid this gold-plating problem create a service portfolio plan (a blueprint for your service inventory) and then use the projects as a means to accommodate/build services within that blueprint. Finally, develop an operational model for reusable service before any actual reuse happens - there are few things that stunt reuse more than poor service performance. So make sure you can ensure service levels as services are subject to greater traffic.

4. Seize opportunities for driving reuse, but don't force it. Even if you're taking the project-driven approach, it makes sense to get some kind of a cross-project view of what's going on, even in the context of a single department or group. It's often not rocket science to find opportunities for building reusable services. For example, we have seen multi-channel projects - Web, bricks and mortar, B2B - that have been great candidates for building reusable components. Consolidation efforts also offer ripe pickings for SOA. Many corporations drive consolidation of redundant and overlapping systems or have a legacy migration strategy. What better way to consolidate than to use SOA principles and build new applications based on reusable services? Business process management (BPM) projects are also good candidate projects for identifying reusable services, because they involve businesspeople who can help identify those business services and typically have high-level management backing, which can promote building and using services that support business processes that are being automated and managed.

5. When the time is right, start building a pool of resources that are reused across projects. By resources, we mean people, infrastructure, and services. If SOA nirvana (enterprise SOA) is your ultimate goal, organizational barriers have to be gradually eroded to enable greater collaboration across groups, pooling of resources, and some centralized planning, such as from an enterprise architecture perspective. So when the time is right, plan a migration from the project-driven approach to the enterprise-driven approach. For IT-driven organizations, transitioning from the project-driven approach to the infrastructure-driven approach, as a stepping point to the enterprise-driven approach, works well. All you need to do is pool a few people together and share a technology platform and related assets across a few projects. At the same time, building a few reusable foundation/utility services will show good results. From an architectural perspective, to improve interoperability, it's pretty important to have the enterprise architecture group define standards that should be adhered too, and ideally this group should be involved in scoring projects for their adherence to these standards. With careful planning, it's possible to seize synergies and increase your SOA maturity and related capabilities.

Conclusion
You can accomplish SOA by doing the same kind of things you did before, but with a better, more standards-based toolset. The more of the strategic SOA benefits you expect or need to deliver, the more you need to make some changes in the way you execute projects. You will also end up doing new things, such as service portfolio planning. The project-driven approach is a reasonable place to start, especially if you're new to SOA - and that's where we see most initiatives focused today. Anything you can do to increase the SOA sophistication and maturity of your organization will pay handsome benefits in terms of future project deliveries. If the SOA benefits you're delivering require a portfolio of services, realize that unless you make a concerted effort to build a portfolio of reusable services, it's unlikely that one will emerge.

If you feel that making the transition to an infrastructure-driven or enterprise-driven approach makes sense and is feasible for your organization, you'll want to build a roadmap that outlines what you will do to get there. The road map will typically include some project deliverables as well as broader initiatives related to delivering SOA, such as an SOA center of excellence if you are taking the enterprise-driven approach. A great way to build a roadmap is to use an SOA maturity model to assess where you are today and then chart out where you need to be over a multi-year period. What you want to be able to say is something like "We are at Level 1 today and, given our business goals, IT goals, and projects, we aim to be at Level 2 in 24 months. These are the things we have to do to get there." Finally, don't be perturbed by what other people are doing with SOA. Be clear about what you want from it, and progressively chart a path to get there. As Yogi Berra said, "If you don't know where you are going, you'll wind up somewhere else." Please don't end up somewhere else. Do SOA right!

Acknowledgments
The authors would like to thank Dave Chappell, Manas Deb, Thomas Erl, Harish Gaur, and Markus Zirn, who diligently read drafts of this article and provided valuable feedback.

© 2008 SYS-CON Media