| 
“Collaborative
Transactions”
by John Tibbetts
and Barbara Bernstein
This last guest editorial
of 2004 stretches the limits of collaboration by suggesting that
rather than collaboration being tied to the interactions of two
or more people, that people interacting with a transactional system
can also be collaborative. It is an interesting idea, and the authors
have implemented several of these systems over the last decade and
have now made some of the components of this system available as
open source software… David Coleman
Nobody who has used a good
piece of collaboration technology needs further convincing: This
is a great way to take advantage of the skills and knowledge spread
throughout an organization, and to institutionalize successful review
procedures. Computers and networks have clearly been a great enabler
of collaboration. Collaboration as defined by Collaborative Strategies
(CS) is between two or more people. But what happens when our systems
get smart enough to act as a collaborative agent? This article looks
at a critical area of enterprise processes that are not usually
considered for collaboration…transaction-oriented processes.
A transaction is a piece
of official business with real-world results: Placing an order,
issuing a check, scheduling a shipment, recalculating a grade-point
average, adjusting the metrics on the manufacturing floor. Hashing
out the details of such pieces of business is often a multi-person
job, but computers tend to handle transactions in a way that makes
them profoundly collaboration-resistant. Anyone who turns from brainstorming
over a blueprint to submitting a departmental requisition notices
the difference immediately.
Enterprises would dearly
love to increase the collaboration potential of their business applications,
and there have been isolated pockets of success. We ourselves have
built several collaborative transactional systems over the past
10 years, and have come to some conclusions about what such systems
need to do and how they should be organized. These insights have
now been captured in a set of open-source software components called
the WorkThru Collaboration Framework, which may finally bring “collaborative
transactions” within everyone's reach.
Understanding Transactions
In the IT world, a piece
of work transacts when the information about it gets officially
recorded in the enterprise database. Usually a single transaction
involves multiple database updates: accept order, schedule delivery,
debit inventory, email confirmation, bill customer, credit commission
to salesman, etc. These updates involve valuable enterprise assets,
so they have to be done accurately, synchronously, and completely.
It is not surprising, then,
that the procedures for updating enterprise data are carefully controlled
and highly formalized. Every business application religiously observes
the “transactional disciplines,” a set of procedures developed during
the mainframe era to protect data against corruption. These disciplines
require a user who wants to propose a transaction to engage in a
real-time conversation with the database. This usually involves
filling out an electronic form and submitting it to the transaction-processing
software for consideration. If anything about the data is incomplete
or unexpected, the transaction fails and the user, most likely,
has to start over. If the user takes too long, the session times
out. Clearly these systems are computer-centric rather than people
or process-centric.
"Out-of-channel
collaboration"
The transactional disciplines
make little concession to the human side of the process. They want
work that is complete, error-free, and ready right now. But, obviously,
many transactions only get completed through people-to-people collaboration.
Just like a documents, these complex transactions often need input
from multiple departments, or may require some research, discussion
and negotiation. Estimates may get firmed up and rough proposals
refined. And then there is the review and sign-off process.
This entire collaboration
phase, alas, has to take place “out of channel.” It is completely
disconnected from the application that will evaluate and process
the transaction, and is often accomplished without any system help
at all. To a degree that would shock followers of Collaborative
Strategies, people working on business transactions still rely on
phone calls, emails, and conversations around the water cooler).
If they get the benefit of any system intelligence, it comes from
personal productivity software that is not usually directly integrated
with the core application. Only when the decisions are made and
the work is complete does someone open the actual application and
record the results. It's like collaborating on a document that can't
be typed into a word processing program until it has been perfected
on paper.
This is not only inefficient,
it is risky! Potentially high-stakes business deals are under discussion,
and the discussion is literally off the record. Important conversations
vanish as soon as you close your IM session. The thoughts of colleagues
disappear into the recesses of your email folders. The result is
that decisions are often anonymous, and do not have a specific audit
trail. The process cannot be reconstructed. Identifying where a
piece of data came from, or who changed it and at what point, is
impossible.
Work in Progress
Bridging the collaboration/transaction
divide is not impossible—many custom-built applications have managed
it, though with considerable effort. For example, certain on-line
marketplaces let participants open up an IM-style chat for negotiation.
The challenge is to make collaborative transactions widely available
without extra effort to any business application that wants them.
What we propose here is a generic approach that suits any kind of
business process where database information assets get updated.
It does not loosen up the transaction disciplines—information assets
are, if anything, more highly valued than ever.
This involves inserting a
new phase—perhaps a new space is a better way to think
about it—that breaks the direct connection between user and database,
holds work during its formation phase, and lets users collaborate
with each other and with the system to get the work assembled,
perfected, and cleared for commit (“commit” means cleared to be
recorded in the database).
From a user's point of view,
it looks as if those on-line forms that structure proposed transactions
have been disconnected from the database, enriched with behind-the-scenes
intelligence (promoting secure and effective collaboration, among
other things), made easy to change and easy to route.
Turning to Objects
From a developer's point
of view, each enriched form is in fact a software object—one of
those data-function bundles that much contemporary software is constructed
from. The object approach is to turn a process into a
thing . Objects are particularly useful as they can be
composed of both program actions and data. In addition their attributes
can be applied to a whole class of objects or can be inherited by
objects made up of other objects .Thus, the transient activity of
proposing a transaction turns into a concrete transactional proposal—an
evolving piece of work that can be saved, circulated, revised, and
reconnected to the database when it is ready.
This object is the reusable
core of the WorkThru framework. We call it a WIP, because it represents
a piece of Work-in-Progress. With partially-completed, ongoing work
embodied in software, transaction-facing collaboration starts to
look plausible.
The WIP object opens up
a collaborative preparation phase within the context of
a transactional application. Since the WIP is disconnected from
the back end, its life can last for a few seconds or for many weeks.
It can be saved in a partially-completed state as many times as
necessary. The WIP as a transaction-collaboration object can move
among collaborators, collecting not only additions and modifications
but also comments and suggestions. It can temporarily tolerate errors
until it finds somebody to correct them. Eventually, when the work
that it contains is ready, the WIP docks up against the database
and unloads its contents.
In one sense, a WIP is the
equivalent of the “document” in classical collaboration—something
that the group can look at together, work on collectively, and improve
incrementally. But documents tend to be content-heavy and function-light.
They are most often manipulated from the outside, and much of their
groupware capabilities come from the environment in which they circulate.
By contrast, a WIP allows less leeway with its content, but surrounds
this content with richer functionality. Most of its collaborative
behavior comes built right into its infrastructure.
Built In Collaboration
Technology
Collaborating within a transactional
context looks a lot like collaborating on anything else, with a
few significant differences that we will point out as we go along.
The capabilities required (and the ones that are built into the
WIP object) include these:
- Versioning :
Like a document, a transactional proposal may go through many
drafts. It is very helpful to have all these drafts handy, both
as a record of how the work evolved and as a chance to return
to an earlier version if something goes off track. A new version
of a WIP gets created every time a change is made, and all of
these versions travel around with the WIP so any user can look
through them. There is unlimited undo/redo.
- Change log :
In a collaborative environment, where many people touch a single
piece of work, it is critical to know who did what. Every version
of a WIP is digitally signed and time-stamped, creating a non-repudiable
record of where each piece of data came from and who is responsible
for every change made to it. If a question arises about how a
certain transaction was arrived at, including the specifics of
who did what when, there is a complete change log that provides
the answer.
- Access Control
: Collaboration raises confidentiality and authorization
concerns. Not every collaborator may be entitled to see and/or
interact with the entire form—it's true of documents and equally
true of evolving business deals. Fine-grained access control is
a must. Each WIP—or any of its sections, pages, or individual
fields—can be “trained” about who to display itself to and who
to accept changes from. This can be flexible and dynamic. For
example, an application may be arranged so that someone holding
the role of “Supervisor” can see the proposed bonuses
for all employees in her department but can adjust the
bonuses only of her direct reports.
- Annotations:
Collaboration means communication, and communication leading to
business deals should be in context and on the record. What's
needed is an annotation mechanism that encourages users to “scribble”
freely and preserves all these scribbles for future reference.
In a WIP, every field (and every page, and the form as a whole)
accepts electronic “stick-on notes” where users can pose questions,
provide explanations, give instructions, or otherwise elaborate
on the data. These notes travel with the WIP. They are clearly
marked, and can be opened, read, and added to by other collaborators.
Often this leads to threaded discussions where many people weigh
in. Every contribution is signed and dated, and the entire conversation
is saved.
- Workflow :
Classical workflow usually involves a separate piece of software
that moves documents around in a pre-determined way. WIPs, more
self-aware and function-rich than documents, can take a more active
role in their own travels. This kind of grounds-up workflow (also
called “agent-based workflow”) can be very powerful and flexible.
Here is how the WorkThru version works: When it is created, a
WIP is provided with a list of “needs” that will have to be satisfied
before it will be ready to commit—pieces of data to be collected,
attachments, approvals, etc.—along with a list of people (by role)
who can address each of these needs. When the circumstances arise
that cause a need to become active (a trip has been initiated:
“book a flight”), the WIP seeks out the appropriate user; once
that need is satisfied, a new need arises (“reserve a rental car”)
and the WIP moves on to the next person. A WIP can even analyze
its own contents and adjust the workflow accordingly—for example,
if the value in the “Airfare” field is above $1200, a “get manager
approval” need may become active and the WIP will take a detour.
By consulting a fairly simple set of internal rules, a WIP can
do the right thing in a wide range of circumstances, some unforeseeable.
- Error handling
: Errors stop most transactional applications cold. Often
work cannot continue as long as there is an error or a un-filled-in
field. Transactional sessions abort regularly for just this reason;
it is the only way to protect the data in the database. A database-independent
WIP object, however, can be more forgiving, handling data errors
the way documents handle spell-check errors. The problem is brought
to the user's attention, and the user can choose to make the correction
immediately or can defer it and keep working. A WIP with errors
in it has no problem being saved or sent on (perhaps to someone
who knows the right answer). All errors have to be fixed, and
all required fields filled in, before the WIP can be submitted
to the database, but until then they form a kind of to-do list:
marked, indexed, and scannable.
A
Real-World Example
Here is an example of a threaded
discussion kicked off by a “soft” error detected by the system.
The screen shot comes from a clinical trials application currently
in use by a major medical center. It shows one of many possible
UI implementations of the WorkThru error-handling and annotation
capabilities.
The entire light-blue area
is a “pop-open” that is not usually visible; that is, question 6
normally follows directly after question 5. The note/error area
pops open immediately above a question field when a user clicks
on the pyramid next to the question number. A gray pyramid (as next
to questions 5 and 7) means that the note/error area is empty but
ready to be typed into. A blue pyramid means that there is one or
more user-written annotation attached to the field. A red pyramid
(as next to question 6) means there is one or more system-generated
error messages on that field.
The red X in the top right-hand
corner closes the pop-open and returns the form to normal. The links
at the bottom of the pop-open enable searching the form for additional
notes and error messages.
The System as Collaborator
Error-handling is an area
where transactions can be even more collaborative than documents.
When the topic of discussion consists of structured data rather
than free form text, the system already contains a lot of intelligence
about what kinds of values are acceptable. When this intelligence
is used to signal work to be done, rather than to kill the transaction,
it can be a valuable help. The system becomes another collaborator—a
kind of superhuman proofreader checking not just for typos and misspellings,
but for data that is of the wrong type (no integers in the Name
field!), the wrong length (no 2-digit Area Code!), or incompatible
with other data (this Zip Code isn't in this State). If the developer
chooses to write sophisticated validation rules, a WIP can also
make sure that data values fall within ranges that make sense (Heart
Rate Per Minute must be higher than 50 and lower than 100) and align
with company policy (Percentage Raise cannot be more than 9%).
A WIP's ability to spot errors
can be very specific. It can distinguish between values that are
impossible and those that are just suspicious. The first case is
a “hard” error, a value that is unacceptable and must be changed.
The second case is a “soft” error which is a value that may be correct
but is so unusual that is has to be explicitly OK'd by someone in
authority. For example, a Blood Pressure value of 350/50 is incompatible
with human life and is certainly an error; a blood pressure value
of 180/90 is possible but should be brought to the attention of
the doctor. When a soft error is overridden and the data accepted,
the system notes who made the OK, in case a question about
the data arises in the future.
Conclusion
Collaboration technology
is so powerful a productivity-enhancer that transactional applications
cannot afford to forego it any longer. Enterprises need to find
a way to take advantage of the knowledge and talents of multiple
people within the context of their core business applications. The
WorkThru Collaboration Framework, coming soon to an open-source
site near you, provides a way to add collaborative functionality
to any transactional application. If it succeeds, and business tasks
routinely pass through a WIP phase on their way to updating the
database, collaboration tools will extend their influence into new
areas of the organization, supporting better decisions, increasing
accountability, improving productivity and adding to the quality
of the work.
John Tibbetts and Barbara
Bernstein are partners in the consulting firm Kinexis, specializing
in applying object technology to transaction-focused workflow and
collaborative systems. John Tibbetts is the creator of the WorkThru
Collaboration Framework, which will make its open-source debut early
in 2005. Learn more at www.kinexis.com or by contacting us directly
at (415) 558-9277.
|
