Compensational Transaction - maintaining Integrity in SOA

Transaction processing techniques play a major role in preserving data consistency in critical areas of computing. The reliability provided through transactional guarantees are required in many types of applications, found in for instance workflow systems, mobile systems, and lately also in SOA like web services based systems.

Web services provide interoperable application-to-application communication, allowing new applications to leverage existing software functions in a platform independent fashion. The transactional behaviour of a function accessed through a web service depends on the underlying implementation of the web service. Often a database system will provide the required local transactional behaviour. However, when an application combines multiple web services in order to complete a given task, coordination of the participating web services is required in order to preserve data consistency.
Traditionally two phased commit (2PC) based protocols have been used to achieve such coordination (e.g. X/Open DTP, CORBA OTS). Because of the loosely-coupled nature and autonomy requirements of web services, 2PC-based protocols may however not be appropriate in this environment. This could be because of the following reasons:

• The application uses multiple non-extended-architecture (XA) resources.
• The application uses more than one atomic transaction, for example, enterprise beans that have Requires new as the setting for the Transaction field in the container transaction deployment descriptor.
• The application does not run under a global transaction.

A web service is either a stand-alone service or a composite service relying on other web services to perform its task. Individual web service invocations may commit early without further coordination, provided that the effect of the invocation can be semantically reversed at some later point by executing a compensating transaction. Typically, compensating transactions are not focused in the design of web service transaction models, and implementation of this functionality is left to the application developer, here web service developer.

This is especially relevent in case of long-running transactions that avoid locks on non-local resources, use compensation to handle failures, potentially aggregate smaller atomic transactions. In contrast to rollback in ACID transactions, compensation restores the original state, or an equivalent, and is business-specific. The compensating action for making a hotel reservation is canceling that reservation, possibly with a penalty.

Interaction between web services, are typically handled through conversational transactions, that involve participation from several web services. The unit of business at each web service represents a subtransaction, also called component transaction.
The transactional behaviour of a single subtransaction is typically provided locally by an underlying database system. Additionally, transactional behaviour of the conversational transaction must be guaranteed through coordination and management of the set of subtransactions. If one or more subtransactions abort, the conversational transaction may or may not need to be cancelled depending on the business logic of the service. It is totally up to the web service starting the conversation to decide if all-or-nothing semantics should be enforced.
Within classical transaction processing, dependent transactions would have to be aborted if the dependent-upon transaction aborted. Resorting to cascading rollback of dependent transactions is however not acceptable in web services, both since dependent transactions may themselves be committed, and also since rollback may be disallowed by autonomous web services. Autonomous web services typically consider the results of a committed subtransaction as final and durable.

Here, the Compensating Transaction enters the stage - to semantically undo the results of the early committed subtransactions. A compensating transaction preserves database integrity without aborting other transactions.

Compensation Transaction -> semantically undoes the partial effects of a transaction T without performing cascading abort of dependent transactions, restoring the system to a consistent state.
The web services designer is responsible for determining the compensation rules, which are used to dynamically generate compensating transactions during runtime.

This can be automated by:

• In BPEL, using invocable Compensating Transaction

  

•  Coordinator protocols like OASIS Business Transaction Processing, and WS-CAF - mediate the successful completion or use of compensation in a long-running transaction.
• Including a Rules Engine (which does on an Event if a Condition satisfied does an Action)
• Custom solution using database as a rules repo(using triggers for events)

Also, read related blogs:

ACID properties

Snapshot Isolation

Transactions - ACID

set of properties that guarantee database transactions are processed reliably ie. Transaction

A - atomicity, 
C - consistency, 
I - isolation, 
D - durability

 For example, a transfer of funds from one bank account to another, even though that might involve multiple changes (such as debiting one account and crediting another), is a single transaction.

Atomic - requires that database modifications must follow an "all or nothing" rule. Each transaction is said to be atomic.

Consistent - ensures that any transaction the database performs will take it from one consistent state to another. The consistency rule applies only to integrity rules that are within its scope. Mechanisms include:

• abort the transaction, rolling back to the consistent, prior state;

• delete all records that reference the deleted record (this is known as cascade delete); or,

• nullify the relevant fields in all records that point to the deleted record.

Isolation - requirement that other operations cannot access data that has been modified during a transaction that has not yet completed.
The question of isolation occurs in case of concurrent transactions (multiple transactions occurring at the same time). Each transaction must remain unaware of other concurrently executing transactions, except that one transaction may be forced to wait for the completion of another transaction that has modified data that the waiting transaction requires. If the isolation system does not exist, then the data could be put into an inconsistent state => leads to dirty reads.

Durability - ability of the DBMS to recover the committed transaction updates against any kind of system failure (hardware or software).
guarantee that once the user has been notified of a transaction's success the transaction will not be lost, the transaction's data changes will survive system failure, and that all integrity constraints have been satisfied, so the DBMS won't need to reverse the transaction. Many DBMSs implement durability by writing transactions into a transaction log that can be reprocessed to recreate the system state right before any later failure. A transaction is deemed committed only after it is entered in the log.

Achieving ACID:

• Locking vs multiversioning
• Distributed transactions

Locking vs multiversioning 
Locking means that the transaction marks the data that it accesses so that the DBMS knows not to allow other transactions to modify it until the first transaction succeeds or fails. The lock must always be acquired before processing data, including data that are read but not modified. Non-trivial transactions typically require a large number of locks, resulting in substantial overhead as well as blocking other transactions. For example, if user A is running a transaction that has to read a row of data that user B wants to modify, user B must wait until user A's transaction completes. Two phase locking is often applied to guarantee full isolation.
In Multiversioning, database provides each reading transaction the prior, unmodified version of data that is being modified by another active transaction. This allows readers to operate without acquiring locks. I.e., writing transactions do not block reading transactions, and readers do not block writers. Going back to the example, when user A's transaction requests data that user B is modifying, the database provides A with the version of that data that existed when user B started his transaction. User A gets a consistent view of the database even if other users are changing data. Snapshot isolation is similar to multiversioning where reference done on a snapshot of the data.
Distributed transactions
2-phase commit is commonly used solution. Here,  in the first phase, one node (the coordinator) interrogates the other nodes (the participants) and only when all reply that they are prepared does the coordinator, in the second phase, formalize the transaction.
Transaction
unit of work performed within a enterprise system a database and treated in a coherent and reliable way independent of other transactions.
Follows ACID properties.

Also, refer related blogs:
Snapshot Isolation
Compensation Transaction

Snapshot Isolation in concurrent Transactions

Snapshot isolation is a guarantee that all reads made in a transaction will see a consistent snapshot of the database (in practice it reads the last committed values that existed at the time it started), and the transaction itself will successfully commit only if no updates it has made conflict with any concurrent updates made since that snapshot.
In practice snapshot isolation is implemented within multiversion concurrency control (MVCC), where generational values of each data item (versions) are maintained.
Snapshot isolation is called "serializable" mode in Oracle.
Eg:
If V1 and V2 are two balances held by a single person, Phil. The bank will allow either V1 or V2 to run a deficit, provided the total held in both is never negative (i.e. V1 + V2 ≥ 0). Both balances are currently $100. Phil initiates two transactions concurrently, T1 withdrawing $200 from V1, and T2 withdrawing $200 from V2.
If the database guaranteed serializable transactions, the simplest way of coding T1 is to deduct $200 from V1, and then verify that V1 + V2 ≥ 0 still holds, aborting if not. T2 similarly deducts $200 from V2 and then verifies V1 + V2 ≥ 0. Since the transactions must serialize, either T1 happens first, leaving V1 = -$100, V2 = $100, and preventing T2 from succeeding (since V1 + (V2 - $200) is now -$200), or T2 happens first and similarly prevents T1 from committing.
Under snapshot isolation, however, T1 and T2 operate on private snapshots of the database: each deducts $200 from an account, and then verifies that the new total is zero, using the other account value that held when the snapshot was taken. Since neither update conflicts, both commit successfully, leaving V1 = V2 = -$100, and V1 + V2 = -$200.
Snapshot isolation present this.
If built on MVCC, snapshot isolation allows transactions to proceed without worrying about concurrent operations, and more importantly without needing to re-verify all read operations when the transaction finally commits. The only information that must be stored during the transaction is a list of updates made, which can be scanned for conflicts fairly easily before being committed.

• Materialize the conflict: Add a special conflict table, which both transactions update in order to create a direct write-write conflict.
• Promotion: Have one transaction "update" a read-only location (replacing a value with the same value) in order to create a direct write-write conflict (or use an equivalent promotion, e.g. Oracle's SELECT FOR UPDATE).

Materialize the conflict: by adding a new table which makes the hidden constraint explicit, mapping each person to their total balance. Phil would start off with a total balance of $200, and each transaction would attempt to subtract $200 from this, creating a write-write conflict that would prevent the two from succeeding concurrently. This approach violates the normal form.
Alternatively, we can promote one of the transaction's reads to a write. For instance, T2 could set V1 = V1, creating an artificial write-write conflict with T1 and, again, preventing the two from succeeding concurrently. This solution may not always be possible.

What is MVCC?
- provides access to implement transactional memory (shared memory allowing a concurrent group of load and store instructions to execute in an atomic way).

For detailed study:
Snapshot Isolation

related blog entries:
ACID Properties
Compensation Transaction

Adabas - grand-old database beast

ADABAS (acronym for Adaptable DAta BAse System) - primary database management system; vendor=Software AG. One of the fastest OLTP db.
Features: offers 24x7 functioning, Parallel Sysplex support, real-time replication capability, SQL and XML access and other leading edge capabilities.
Historically, ADABAS was used in conjunction with Software AG's programming language NATURAL, so that many legacy applications (eg: Mainframe) that use ADABAS as a database on the back-end are also developed with NATURAL as well.
Best suited for very high volumes of data processing or in high transaction online analytical processing environments.
Proven to be very successful in providing efficient access to data and maintaining the integrity of the database. ADABAS is now widely used in applications that require very high volumes of data processing or in high transaction online analytical processing environments.

Technical Info:
Inverted list database - content based indexing of records => quicker search but slower storage

• Files - major organizational unit (similar to ~tables)
• Records - content unit within the organizational unit (~rows)
• Fields - components of a content unit (~columns)
• No embedded SQL engine; popular external query mechanism ADASql
• Search facilities may use indexed fields or non indexed fields or both
• No implicit referential integritiy constraint => parent-child relations must be maintained by application code
• Supports two methods of denormalization: repeating groups in a record ("periodic groups"); and multiple value fields in a record ("multi-value fields")

Netstring format

A netstring is a formatting method for byte strings that uses a declarative notation to indicate the size of the string.
Netstrings store the byte length of the data that follows, making it easier to unambiguously pass text and byte data between programs that could be sensitive to values that could be interpreted as delimiters or terminators (such as a null character).

Eg: the text "hello world!" encodes as:
12:hello world!,
And an empty string as:
0:,

Since the format is easy to generate and to parse, it is easy to support by programs written in different programming languages. In practice, netstrings are often used to simplify exchange of bytestrings, or lists of bytestrings. For example, see its use in the Simple CGI and the Quick Mail Queuing Protocol (QMQP).
Netstrings avoid complications that arise in trying to embed arbitrary data in delimited formats. For example, XML may not contain certain byte values and requires a nontrivial combination of escaping and delimiting, while generating multipart MIME messages involves choosing a delimiter that must not clash with the content of the data.
Note that since netstrings pose no limitations on the contents of the data they store, netstrings can not be embedded verbatim in most delimited formats without the possibility of interfering with the delimiting of the containing format.
In the context of network programming it is potentially useful that the receiving program is informed of the size of the data that follows, as it can allocate exactly enough memory and avoid the need for reallocation to accommodate more data.

SOA Composite Editor for JDeveloper - installation

Oracle SOA Composite Editor - JDeveloper extension for SOA technologies: SOA Composite Assembly, BPEL PM, Mediator, Human Task, Business Rules, Adapters.


    1. Help->Check for updates
       In the Update Wizard, select Search Update Centersand ensure Oracle Fusion Middleware Products is checked. Select Oracle SOA Composite Editor and click on Next to begin downloading.
        If this doesn't work add a new update site with link: 
http://www.oracle.com/ocom/groups/public/@otn/documents/webcontent/156082.xml
       and it'll show the editor for updating JDev


    2. From file - Direct download file at: 
 In the Update Wizard, http://www.oracle.com/ocom/groups/public/@otn/documents/webcontent/156082.xml
        and choose install from file radio option. This file is about 230MB.


References
Installing JDev for BPM

BPELscript - javascript-like language for BPEL

BPELscript is a language to specify BPEL processes. It provides a compact syntax inspired by scripting languages such as JavaScript and Ruby and a full coverage of all features provided by BPEL. A programming language which omits the XML-overhead of BPEL but offers the same features as BPEL.
BPELscript provides:

1. a compact syntax inspired by scripting languages such as JavaScript and Ruby
2. the full coverage of all features provided by BPEL
3. a translation from & to WS-BPEL 2.0
4. The translation to WS-BPEL 2.0 ensures that BPELscript can be executed on all workflow engines supporting WS-BPEL 2.0.

Eg:

namespace pns = "http://example.com/loan-approval/";  

namespace lns = "http://example.com/loan-approval/wsdl/"; 

@type "http://schemas.xmlsoap.org/wsdl/"  

import lServicePT = lns::"loanServicePT.wsdl"; 

@suppressJoinFailure  

process pns::loanApprovalProcess {  

    partnerLink customer = (lns::loanPartnerLT, loanService, null),  

    approver = (lns::loanApprovalLT, null, approver),  

    assessor = (lns::riskAssessmentLT, null, assessor);  

    try {  

        parallel { 

            @portType "lns::loanServicePT" @createInstance  

            request = receive(customer, request);  

            signal(receive-to-assess, [$request.amount < 10000]);  

            signal(receive-to-approval, [$request.amount >= 10000]);  

        } and {  

            join(receive-to-assess); 

            @portType "lns::riskAssessmentPT"  

            risk = invoke(assessor, check, request);  

            signal(assess-to-setMessage, [$risk.level = 'low']);  

            signal(assess-to-approval, [$risk.level != 'low']);  

        } and {  

            join(assess-to-setMessage);  

            approval.accept = "yes";  

            signal(setMessage-to-reply);  

        } and {  

            join(receive-to-approval, assess-to-approval); 

            @portType "lns::loanApprovalPT"  

            approval = invoke(approver, approve, request);  

            signal(approval-to-reply);  

        } and {  

            join(approval-to-reply, setMessage-to-reply); 

            @portType "lns::loanServicePT"  

            reply(customer, request, approval);  

        }

    }  

    @faultMessageType "lns::errorMessage"  

    catch(lns::loanProcessFault) { |error| 

        @portType "lns::loanServicePT" @fault "unableToHandleRequest"  

        reply(customer, request, error);  

    }  

}

how it works:

It uses a translator based on the ANTLR v3 (ANother Tool for Language Recognition) parser generator which is based on a predicated-LL(*) parsing strategy. This solution uses the implicit tree structure behind the input sentences to construct an abstract syntax tree (AST) which is a highly processed and condensed version of the input. The translator maps each input sentence of the source language to an output sentence by embedding actions (e.g. code) within the grammar or tree. This actions will be executed according to its position within the grammar or tree. To support an easy handling of implicit declarations the translation is broken down into multiple passes.

References:

BPELscript: A Simplified Script Syntax for WS-BPEL 2.0

Translating WS-BPEL 2.0 to BPELscript and Vice Versa
BPELScript Grammer