BPMN is the de facto standard for process modeling, but many leading modeling tools, particularly those incorporated within high-end business process analysis (BPA) suites, have so far been reluctant to adopt it. Now that appears to be changing.

Recently IDS Scheer announced that ARIS, generally considered the leading standalone BPA suite, would be supporting the full BPMN notation in the v7.0.2 service release this spring. Announcement of BPMN support was tucked into their press release on new simulation capabilities based on Lanner's technology. IDS Scheer will provide their own BPMN serialization using the "ARIS Markup Language"; rather than XPDL or BPDM, asserting that customers are not asking for a standards-based serialization. Also, they are currently working on a mapping between EPC (ARIS's standard process modeling notation) and BPMN.

The announcement was surprising to me, since at Process World IDS Scheer's CTO, Dr Wolfram Jost, was generally dismissive of BPMN's richness compared to EPC and the rest of ARIS. Now, however, IDS Scheer appears to warming to BPMN as an emerging standard… not for modeling but for executable process design! The following is their current statement on BPMN:

In general, IDS Scheer will follow the trend that actually takes place with BPMN. On the execution layer, we expect that BPMN will become the accepted standard. It is specifically designed for this use case, including all the constraints necessary to describe a valid and executable process. For typical questions, in the area of BPD/BPA, the current BPMN specification does not provide the necessary broadness. The EPC notation, with its link into many other topics (BSC, risk management, compliance management, data modelling, BI, EA, etc.), is a mature industry standard which will have the rights to exist also in the future. All recent customers and partner projects have shown this very clearly. While BPMN is used on the technical-oriented BPM layer only, EPCs and many other methods are used on business-oriented BPM layers. Our strategy is to establish a bridge between those two layers via our tools and integrated methods.

Also, IBM is expected to make an announcement at their upcoming Impact event in 2 weeks regarding support for BPMN in WebSphere Business Modeler. It is not clear whether this is a wholehearted move to BPMN or just support for it as an alternate secondary notation as IDS Scheer is doing. Either way, it's a significant endorsement that I believe solidifies BPMN's place as the emerging standard for process modeling.

WebSphere Modeler is an interesting case, because it tries to be both a standalone BPA tool and an integrated component of the WebSphere BPM Suite. Even though IBM's own guy, Steve White, was editor of the BPMN 1.0 spec, the company had been betting that UML 2.0 would emerge as the eventual winner in the process modeling notation standards battle. (Funny, I never even knew there was a battle, or that UML was even in the running.) Either IBM is throwing in the towel, or they think that with BPDM, UML has co-opted BPMN anyway at the serialization level. But it really doesn't matter. With support from both BPA and BPMS vendors, it looks like BPMN is at last established as the only significant multivendor standard for the process modeling notation.

I’m in the process of updating my 2006 BPMS Report series on BPMInstitute.org to the new and improved 2007 version. A major change from last year is a beefed-up evaluation scoring. I’ve discovered that many users (and most vendors) are happy to skip the 25-page walkthrough of the product and go straight to the scorecard at the end. Which product “won”? I haven’t figured out the presentation - it will probably be some 2-dimensional thing like the Forrester Wave or Gartner MQ - but I’m close to having a finished scoring methodology. It’s probably asking for trouble, but I’m publishing it right here so that you can comment upon it.

The basic plan is this. I define 4 process types: Task Routing (basic workflow), Production Workflow, Case Management (emphasizes content, collaboration, and unstructured processes), and Integration-Centric. The characteristics of each type are explained in the report overview, but most of you can imagine what they are. Each BPMS is scored against all 4 process types using 12 sets of criteria, but the weightings of each set may differ from one process type to the next. Also, the capabilities affecting the individual criteria may be process type-specific.

Here are the 12 sets of criteria, things I’m looking for in each (some are process type-specific), and the percentage weighting of the set for each process type:

1. Architecture and Environment (Weightings: task routing 10%, production workflow 10%, case management 8%, integration-centric 10%)
· Unified environment for workflow, integration, rules, BAM
· Minimal programming required
· Scalable
· Clustering, hi availability
· Leverage J2EE or equiv platform services
· Web-hostable/SaaS-compatible runtime
· Support for standards
· RAD/iterative support
· Component discovery and reuse

2. Modeling (Weightings: task routing 10%, production workflow 10%, case management 4%, integration-centric 5%)
· Usable by business process analysts (not developers)
· Support for BPMN (full support)
· Integration of BPA (ARIS, ProForma, MEGA, etc)
· Stay in sync w/design (shared or roundtripping)
· KPI modeling
· Model repository, publishing
· Support for industry models (SCOR, ITIL, etc)
· Team collaboration
· Simulation

3. Human Workflow (design) (Weightings: task routing 10%, production workflow 10%, case management 8%, integration-centric 5%)
· Dynamic task assignment
· Flexible routing
· Form design
· Screenflow design
· Worklist design (columns for business data)
· Internationalization support

4. User Experience (runtime) (Weightings: task routing 15%, production workflow 7%, case management 13%, integration-centric 5%)
· Ajax web forms
· Offline task participation
· Guided task performance
· Portal design and integration
· Reassign/delegate task
· Production workflow support (getNext item)
· User sort/filter/query worklist
· Instance status tracking

5. Content/Collaboration/Case Management (Weightings: task routing 10%, production workflow 7%, case management 13%, integration-centric 5%)
· OOTB attachment support in task UI
· Doc upload, annotate, edit
· Viewers
· Checkin checkout, versioning
· Integ with 3d party ECM
· Archiving
· Retention mgmt
· Team room support
· Discussion, chat, presence detect
· Process knowledge repository
· Case mgmt support

6. Business Rule Management (Weightings: task routing 5%, production workflow 7%, case management 8%, integration-centric 10%)
· Rule repository
· Rule design
· Rule maintenance app
· Rule change impact analysis
· Rule engine
· Rule-process integration

7. Integration (Weightings: task routing 5%, production workflow 7%, case management 8%, integration-centric 10%)
· Adapters – Introspectng/self-generating; packaged - Mainframe, pkg app, b2b, .Net etc
· Async integration, callback, reliable messaging, WS-Addressing, security etc
· Data transformation mapping, XSLT/XQuery engine
· ESB, mediation
· Registry/repository
· Metadata library
· B2B support – trading partner gateway, EDI

8. Events and Exceptions (Weightings: task routing 5%, production workflow 6%, case management 13%, integration-centric 10%)
· Event listeners and adapters
· Full bpmn event support
· Wait for event
· Interrupt by event + exception flow
· Manual suspend/resume instance
· Error propagation, handling without programming
· BAM events and actions
· Transaction rollback and compensation

9. Performance Mgmt/BAM (Weightings: task routing 5%, production workflow 10%, case management 4%, integration-centric 10%)
· Metrics and kpis – OOTB and User-defined
· Dashboard design - Charts, reports, alerts and notifications
· OLAP-style breakdowns
· Bam rules and actions
· Drilldown to root cause
· Instance monitoring
· Optimization actions
· Predictive analytics, e.g. expected finish

10. Governance (Weightings: task routing 5%, production workflow 10%, case management 4%, integration-centric 10%)
· Enterprise repository of models and components
· Link processes to goals and KPIs
· Role based access control and authorization
· Version control of model and implementation components
· Change request, approval, and implementation workflows
· Audit trail on component changes
· Change impact analysis

11. Solutions and Services (Weightings: task routing 5%, production workflow 6%, case management 4%, integration-centric 10%)
· Industry solutions – documented, QA’ed, supported
· Partner industry solutions
· Professional services

12. Installed/reference customers (Weightings: task routing 15%, production workflow 10%, case management 13%, integration-centric 10%)
· Task routing implementations
· Production workflow implementations (volume, industry)
· Case management implementations
· Integration-centric implementations (volume, industry)

Products will be scored in each of the 12 categories from 0-5, as in the Forrester Wave, based on the bullets listed here (as amended); in some cases the scoring will be process type-specific. Then these scores will be weighted as shown here (or as amended) for each category. One or more categories may be split off to form the second axis of the final result, as Gartner and Forrester both do in theirs.

So there you have it. If you see something missing or improperly weighted here, please let me know, either by comment or by private email.

Just received a note from Phil Gilbert of Lombardi, a key contributor to the BPDM effort in OMG, that says:

[I] wanted to let you know that the OMG Architecture Board voted to approve the BPDM spec today. There are actually 2 more small hurdles instead of 1 more as I told you earlier. But these are 99.9% certain to approve specs that have passed the Architecture Board review. Apparently these take several weeks calendar time as the boards that approve aren’t on the TC calendar, they have their own.

In any event, it appears that a major milestone for the industry has been passed: a specification for a business process metamodel (as opposed to UML-defined process) is poised to achieve standards status and have the backing (and implementation) of process platform and modeling vendors. This will insure a standards-based way in which BPMN models can be exchanged, and both standards are driven by the same organization, allowing for unprecedented alignment. In fact, at this meeting, the next major version of BPMN is being discussed and it is expected that the focus will be on using that effort to merge the BPMN and BPDM specs, so that there will be one modeling spec, and that spec will have it’s explicit notation and its explicit metamodel.

The first step was to define the notation, and now we’ve defined the metamodel.

I think end-user companies are the winners today. There is now clarity around the implementation of process at the business level, and that clarity is in the form of business-based process notions, not IT- and systems-based notions of process.

This message was an update amplifying a previous one that announced BPDM approval by the Domain Task Force:

… this morning in San Diego at the OMG Technical Meeting, the Domain Task Force voted FOR approval of BPDM! This is the second major hurdle of any spec. The first is the RFP which kicks off the submission team… that was done long ago for BPDM. The second is the DTF vote (which occurred today). The third is the review and vote by the Architecture Board, which will happen Friday of this week. Then the OMG Board ratifies the AB decision within the next 30 days or so.

While it is rare, the AB can reject or send back a spec for more work, but in general the vote of the DTF is the more difficult barrier because of the diversity of companies (and interests). The AB is more focused on the technical implementation of the specification and less concerned with the business issues that may be reflected in the votes of all the member companies.

Phil also replied to my questions with some answers from Lombardi and MEGA representatives on the BPDM submission team:

Q. How would you create serialized BPMN from BPDM?

A. We are updating the EMF plugins. This will allow creation of BPDM models using Java. Additionally there will be a very basic editor (not usable by business people) that will allow a developer to create BPDM models and to look at the produced xml. The goal is to have these plugins updated for Monday 26 March. This will also be discussed on Sunday by the submission team. Most likely we’ll do a presentation about the serialization of BPMN using BPDM and we’ll show the actual XML that is produced.

Q. Could xsd be derived from this xmi document?

A. Yes. EMF provides such capability. XSD most likely will not be part of the submission, but a separate document that will be distributed freely.

Q. Do you expect OMG to provide one?

A. Yes. During the meeting we will create and distribute the XSD.

Q. Can standard xml validation be used?

A. Yes. Using the provided XSD.

Q. Can I use XSLT [to transform and validate models]?

A. Yes. However XSLT is not very suitable for performing model transformations. For object models there are better languages that can transform models. One such example is QVT. On the Eclipse.org you can find other languages also.

Q. Is there some example that can be used to understand BPDM?

A. We are working on an example that demonstrates BPDM features. This is incorporated in the html documentation that is generated. Most likely it will need additional explanations, to be provided after the approvals.

My comment on Keith Swenson’s XPDL-BPEL apples-and-oranges post and the failure of XPDL to fill the vacuum left by OMG in the BPMN specification stirred up an interesting response from Keith that reinvigorates the discussion and helps clear the air. But he still frames the discussion in terms of portability of executable designs rather than portability of models (i.e. abstracted from implementation details). In the XPDL vs BPEL discussion, this is appropriate, but in the discussion of BPMN portability it misses a fundamental point.

He says:

Bruce has criticized XPDL for inability to take an executable process from one vendor product, and bring it to another vendor product, and guarantee is it understood. He is right. There is no guarantee that a process drawn in one product, saved in XPDL, will be immediately executable in another product. This is not because XPDL fails to capture the semantics, but instead a failure to (1) be able to unambiguously capture those semantics in standard BPMN, as well as (2) a failure of the receiving tool to understand the same semantics that the sending tool transmitted.

In the first sentence above, replace “an executable process” with “a process model.” And there you have the essential problem, because I would say BPMN is able to capture the semantics adequately, but BPMN tools simply do not adhere to the spec, and users don’t know how to create diagrams correctly. The tool vendors’ own sample diagrams violate the spec time after time, something I’ve blogged about repeatedly in the past, and — more important — the tools (the ones I’ve looked at, at least) do not adequately validate the diagrams against the rules defined in the spec.

In trying to prove his point that the semantics required for model portability are missing from BPMN diagrams, Keith unwittingly proves mine instead, that the problem is diagrams often violate the spec and the tools don’t seem to care. He says:

Lets say there was a product called “Vulcan Mind-Meld” which used BPMN to express the diagrams that have meaning to this product. BPMN defines what each of the symbols mean, but the real meaning, the real semantics comes from the way that the symbols are composed together. Mind-Meld would guide you as you draw this diagram, making sure that you do not put anything together in a way that is nonsensical. The author of this diagram is making an expression that has meaning in Mind-Meld. Here is a possible diagram which is consistent with the BPMN specification:

Now, imagine the Vulcans had followed BPMN best practices and applied user-meaningul labels to the various activities, sequence flows, gateways, and events, so it would be clearer the conditions under which the process followed one path versus another. (In BPMN modeling it is actually these labels, not the attributes — those are mostly for BPEL generation — that clarify the process semantics.) That would go a long way toward clarifying what those Vulcans had in their melded minds. To the extent it’s still unclear, it’s because this diagram, which even a guy as smart as Keith says is perfectly good BPMN, contains at least 3 errors.

1. You have 2 sequence flows coming into Casino from potentially concurrent paths. You need an OR-merge there, or else you potentially have 2 tokens going through the Casino activity for one process instance. Not absolutely forbidden by BPMN, but best practice to avoid this. It will break virtually any simulation engine or process engine, and probably not what the user had in mind.

2. Casino, having no sequence flow out, is what the spec calls an end activity. If start events are used in the diagram (they are here), an end activity must have a sequence flow to an end event.

3. Casino has a Compensation intermediate event with a sequence flow coming out of it. That’s invalid. A Compensation intermediate event has an association to a single compensation activity, not a sequence flow.

If I can spot these things easily, why didn’t Keith’s BPMN tool?

For the past few weeks I have been grading exercises submitted by students in my Process Modeling with BPMN training. My decision to include hands-on with a tool and exercises submitted for grading turned out to be a good one — students say the exercises really drive home how to use BPMN in a way that the lecture material simply can’t. And what I’ve learned from the grading is that students frequently make diagram errors that could, in principle, be caught by a BPMN validator. The tool I use, Process Modeler for Visio from itp commerce, has a modest amount of BPMN validation - just enough to make sure the simulation engine can follow the process — and a stricter validation (overly so I think) of diagrams with BPEL export, reflecting more the constraints of BPEL’s block orientation than anything about BPMN. But almost all the errors I find are allowed by the tool. If I can see them, surely a validation routine should be able to spot them as well. This would change the nature of my training, perhaps, but… that’s another subject.

Finally we get to the real reason for this post, which is the long-awaited BPDM — the official metamodel for BPMN from OMG. You can see it, if you want to, at modeldriven.org. I was looking for a schema, but instead it’s this funky XMI thing. It’s not even schema-valid XMI (that’s ironic, isn’t it?). It fails XMI import as published on modeldriven.org, but if you patch it up with the right root elements — thanks to Marlon Dumas for doing that — you can at least import it into a UML tool. Yes that’s right, UML. The schemas that Marlon was able to generate don’t shed much light on what the structure of a BPMN serialization would look like, and that bugged me last week.

But I think Keith’s post here crystalized the thing that is really bothering me. It’s not portability of models between tools. I suppose whether they have to use Eclipse or whatever, the tool vendors will figure out that piece. What I realize I was actually looking for in BPDM was a more precise statement of the rules of BPMN, so they could be reflected in tool-based validation. I had the feeling that those rules were somehow buried in the XMI, which I can’t decode, unlike schema, which I can. But on further consideration, I see that it’s highly unlikely that a schema could possibly represent all the rules, such as if you have this kind of flow object over here, that constrains what is allowed for this other flow object over there. Of course, if you had a schema you could write XSLT that would do the validation based on rules extracted from the narrative of the metamodel. But that’s a story for another day.

Keith Swenson is one of the true superheroes of BPM, and a pioneer in the development of interoperability standards. Known for his stalwart defense of XPDL, he periodically feels called upon to insist that XPDL does not compete with BPEL… then usually adding that XPDL is actually better. But I’ve always felt that Keith obscures the real difference between XPDL and BPEL and their relationships to the “real” BPM standard, which is BPMN.

The latest fracas started a couple days ago when Keith claimed victory in the non-war from the fact that 8 of the 12 vendors in the top 3 quadrants of the Gartner MQ support XPDL. Even though a number of those vendors also support BPEL, at least as an interchange format for automated fragments of the process, it is fair to say that vendor support for XPDL is probably more widespread than vendor support for BPEL. So let’s stipulate that, no problem.

An anonymous commenter to that post said, paraphrasing, “Yeah I work for one of those vendors on your list, and for us XPDL is a checkoff item and actually BPEL is more important as a standard.” Well, that fried one of Keith’s circuit boards and led to yesterday’s apples-and-oranges post, where he once again tries to explain why XPDL is orthogonal to BPEL (but still sort of better).

It’s true that XPDL and BPEL do different things, and Keith does describe the essential difference, but he couches it in language that slants the implications to users. His example is a reasonable one, a BPMN diagram of a simple split and join. Keith describes the BPEL representation, which conveys the process semantics of the concurrency and join, as “lossy” and “one-way” because it does not capture — as XPDL does — the precise shapes of the BPMN activities, gateways, and events, the bends in the arrows, etc.

In other words, XPDL captures the diagram, while BPEL captures the process semantics. Keith dismisses the latter as just the information an “execution engine” would need to know. Technically that’s true of BPEL, I suppose. But which of these best represents the process model? The part that Keith glosses over is a process diagram is not the same as a process model. The argument over whether BPEL or XPDL is more “portable” is based on different interpretations of what portable means. If you mean the same process semantics can be executed on two different engines, then BPEL is more portable. If you mean that the same diagram can be created in two different tools, then XPDL — especially if you allow the target tool to ignore the graphical details that don’t carry over.

Which aspect of portability is more valuable? It depends on what you’re trying to do. If you’re just trying to glue together tool A and tool B, XPDL has more flexibility. The freedom to ignore the parts that don’t map exactly is implicit. Of course, you would need a side agreement between vendor A and vendor B to make the thing work, but let’s not talk about such details.

With BPEL you don’t have the freedom to ignore elements you don’t support. BPEL is BPEL and you have to support everything in the spec. The rest are called proprietary extensions. They live in their own namespaces, and a valid criticism of BPEL 1.1 is that real processes need too many of them. It’s a bit better in BPEL 2.0, but human tasks, subprocesses, and other basics still require extensions in 2.0, such as the nearly mythical BPEL4People.

So let’s get to the real question. BPMN is a modeling notation — more than just a diagram, since each element has defined process semantics, abstracted from implementation details — but BPMN has no official XML schema, i.e. no interchange format. XPDL 2.0 was explicitly created to capture all the elements of BPMN 1.0 for interchange, but — here’s the part that Keith omits — from a diagram portability perspective, not a model portability perspective. That’s because OMG (actually this dates back to BPMI) never defined which BPMN elements and attributes, and their associated process semantics, have to be supported by a “compliant” tool. Intermediate events? Compensation? Workflow engines traditionally didn’t support those, so they are conveniently left out of BPMN tools from many vendors. I’m not sure what they do when they import XPDL that has such elements. Maybe drop them on the floor, with apologies.

My view is that preserving BPMN shapes, colors, and line bends, while ignoring process semantics that don’t fit in the other tool, is not a particularly useful accomplishment. Each tool usually has its own graphical representation of model elements, anyway, so I can’t imagine the graphical details are really preserved in reality.

Bottom line is that neither XPDL nor BPEL today meets the real need of the BPM community, which is a portable serialization of process models — not diagrams, models — that is independent of implementation architecture. OMG is supposedly developing that based on BPDM, its formal metamodel for BPMN, now nearing finalization. I said last spring at OMG Think Tank that in BPDM’s absence, XPDL had a window of opportunity to become the de facto serialization standard for BPMN. But by focusing on diagrams not models, and positioning itself versus BPEL not BPDM, XPDL has let that window close. They might argue that adding BPMN compliance rules and semantics to XPDL is not their job but OMG’s. But that was in fact the opportunity, soon to disappear.

Here’s the puzzling part. I’ve actually seen a draft of BPDM, and looked there without success for any sign of a BPMN schema. Actually I found the thing near-incomprehensible; there was something about MOF and XMI but not a schema. It made me wonder whether BPDM would actually include a schema for BPMN, or just some kind of production rules that ensure conformance to the BPDM metamodel. If you know the answer to this question, please comment to this post. If OMG does not publish a BPMN schema, I see more consternation in BPM-Land and a second chance for XPDL to get it right. If BPDM produces a schema and a list of must-support BPMN semantics, then I predict next year Keith will forget about BPEL. He’ll be writing about why XPDL is different from BPMN… but sort of better.

- Bruce Silver, http://69.36.189.101/wordpress/