WebSphere Business Monitor: troubleshooting events not flowing through a monitor model

One of the most common problems in WebSphere Business Monitor runtime administration is events not making it all the way to Dashboards (Portal or Business Space). Applications are running normally and emitting events that should be input to WebSphere Business Monitor (colloquially known as BAM), but BAM produces no output: no instance data is seen in dashboards. I’ve recently had an opportunity to troubleshoot 2 BAM installations in different organizations on consecutive days and I was faced with this problem in both cases.

Here are some things to check if you come across this situation:
1. A good first step in problem determination if you have access to monitor database is to check tables in the monitor model schema. There are few tables in this schema, so the task is easy. Here’s the list of tables from one simple model I developed:

There is one table per monitoring context (the one which name starts with MCT), plus tables for incoming, consumed and processed events, tables for KPIs and, in this case, one more service table. If this is the first time you attempt to run a model, you are just interested in cardinalities (number of rows in tables). Otherwise, you’d be looking at the number of recent rows.
If there is no data in any tables, events are not being routed to BAM. If there are rows in incoming_events table only, events are sent to BAM, but monitor model is not processing events. In the end, if the model works correctly, you should see one row in the monitoring context (MCT_*) table per, well, monitoring context created by original events.

2. Check that your monitor model is startable. Inspect SystemOut.log of the server running your model’s moderator module.
This message indicates successful startup of the monitor model:
[12/2/09 13:53:53:375 EST] 00000011 ConsumerDaemo I com.ibm.wbimonitor.mm.TestModel3.20081024161629.moderator.ConsumerDaemonHandlerImpl startDaemon() CWMRT3005I: The Monitor Model "TestModel3 20081024161629" is starting consumption on this server or cluster member in SERIAL_ST mode with reordering=false from PRIMARY_JMS_QUEUE.

On the other hand, this message indicates failure:
CWMRT2009W: The MM application is not in a startable state. This is usually because lifecycle steps are not complete.

If you see this error, perform step 3.

3. Check that mandatory lifecycle steps have been performed. Log on to admin console, go to Applications -> Monitor Models -> your model -> click on version -> make sure that all lights are green.
Monitor version deployment - lifecycle steps complete OK
“Schema created” is the only lifecycle step that impacts operational data flow through monitor model.
If “Schema created” step is red, database schema creation for the model did not complete. Click on “Manage schema” link on the right. If your database environment is not restrictive and monitor database user has administrative permissions, you can create schema by clicking “Run Create Schema Script” button. In case of restrictive databases, such as DB2 for z/OS, this button is not even enabled. You will need to export the DDL by clicking “Export Create Schema Script” and work with your DBA to create and configure schema.

4. Check that CEI distribution is active
Click on your model version, and check the text under “CEI distribution mode”.
Monitor model version CEI distribution is ActiveIt should be active AND it should be in the right mode. In most cases (with exception of test environment), you likely are using queue bypass, in which case CEI distribution mode should read “Active (monitor model queue bypass)”.
To change distribution mode, click on “Change CEI distribution mode” link on the right.
CEI distribution is Active with Queue bypass
Select desired value from “Target” drop down and click OK. Restart the target CEI server/cluster!

5. Check that correct CEI server has been configured as event source. This is a common problem in complex multiple cluster topologies.
Click on your model (not version) and then click on “Change CEI configuration”.
CEI source server selection
Check the table under “Event group profile list name” at the bottom of the properties page. In case of a complex topology with multiple CEI servers in the cell, all CEI-enabled clusters/servers will be listed individually. Make sure the right CEI server is selected (checkbox ticked). If you need to make a change, follow this procedure. First, checkboxes are made inactive (unavailable) if at least one model version has active CEI distribution. Deactivate CEI distribution for all model versions (change distribution mode to Inactive) as described in previous step. Wait for a minute for the change to become effective. Then come back to this property page – those checkboxes will become enabled. Select the right cluster/server, click Apply and then change distribution mode to Active for all model versions as described in step 4.

Update on Alphablox in WebSphere Business Monitor

Earlier this year I blogged about installing Alphablox as part of WebSphere Business Monitor. Real production environments require clustering, and it was difficult to accomplish. Using DB2 on z/OS for data repository was particularly daunting task. Mike Killelea commented on my original post, noting that IBM disclaimed support for this scenario.

This time of the year, IBM is preparing version 7 of its BPM stack for release. From what I saw, Alphablox is much better integrated into Business Monitor (BAM). Remember, until now you had to run a separate Alphablox installer (with the exception of non-production-grade standalone profile). Now, ABX install is fully integrated. I specifically inquired about z/OS database support and was told that it is ON. I’ll post an update when I learn more.

Coexistence of WebSphere Business Monitor and Process Server 6.2

UPDATE 11/12: The fix for the issue described below is included in WBM 6.2 Fixpack 2, now publicly available.

If you installed WebSphere Business Monitor (aka BAM) version 6.2 into the same directory as Process Server or WESB 6.2, you may run into problems with overlapping OSGI plugins.
Your WPS/WESB modules and mediations could fail with JXPath error like this:

[5/18/09 19:25:21:744 EDT] 0000005f ExceptionUtil E CNTR0020E: EJB threw an unexpected (non-declared) exception during invocation of method "transactionRequiredActivitySessionNotSupported" on bean "BeanId(TestFanoutApp#TestFanoutEJB.jar#Module, null)". Exception data:
Mediation primitive failure:
Mediation primitive: FanOut1
Component: TestFanout
Module: TestFanout
com.ibm.wsspi.sibx.mediation.MediationBusinessException: CWSXM3752E: Error using XPath expression [Ljava.lang.Object;@7d6e7d6e to locate repeating element: org.apache.commons.jxpath.JXPathException: No value for xpath: /body/print/input/orders. This has been reported by the following entity: ID=FanOut1,Request,print,PrintOrder,http://TestFanout/PrintOrder,TestFanout,TestFanout
at com.ibm.ws.sibx.mediation.primitives.util.ExceptionHelper.newMediationBusinessException(ExceptionHelper.java:128)
at com.ibm.ws.sibx.mediation.primitives.fan.FanOutMediation.locateRepeatingElement(FanOutMediation.java:726)
at com.ibm.ws.sibx.mediation.primitives.fan.FanOutMediation.performNonAggregationMediate(FanOutMediation.java:593)
at com.ibm.ws.sibx.mediation.primitives.fan.FanOutMediation.mediate(FanOutMediation.java:265)
at com.ibm.ws.sibx.scax.mediation.engine.JavaMediationPrimitive.performInvocation(JavaMediationPrimitive.java:630)
at com.ibm.ws.sibx.scax.mediation.engine.JavaMediationPrimitive.invoke(JavaMediationPrimitive.java:352)
at com.ibm.ws.sibx.scax.mediation.engine.MediationPrimitive.invokeConnections(MediationPrimitive.java:318)
at com.ibm.ws.sibx.scax.mediation.engine.JavaMediationPrimitive.fireOutputTerminals(JavaMediationPrimitive.java:728)
at com.ibm.ws.sibx.scax.mediation.engine.JavaMediationPrimitive.performInvocation(JavaMediationPrimitive.java:650)
at com.ibm.ws.sibx.scax.mediation.engine.JavaMediationPrimitive.invoke(JavaMediationPrimitive.java:352)
at com.ibm.ws.sibx.scax.mediation.engine.MediationPrimitive.invokeConnections(MediationPrimitive.java:318)
at com.ibm.ws.sibx.scax.mediation.engine.Input.invoke(Input.java:138)
at com.ibm.ws.sibx.scax.mediation.engine.RequestFlow.invokeFlow(RequestFlow.java:132)
at com.ibm.ws.sibx.scax.mediation.engine.MediationFlow.invokeRequestFlow(MediationFlow.java:145)
at com.ibm.wsspi.sibx.mediation.flow.ejb.MediationFlowBean.invokeRequestFlow(MediationFlowBean.java:231)
at com.ibm.wsspi.sibx.mediation.flow.ejb.EJSLocalStatelessTestFanout_c53bef64.invokeRequestFlow(EJSLocalStatelessTestFanout_c53bef64.java:127)
at com.ibm.ws.sibx.scax.mediation.component.ejb.EJBMediationFlowComponentImpl.invokeRequestFlow(EJBMediationFlowComponentImpl.java:223)
at com.ibm.ws.sibx.scax.runtime.handler.MFCImplementationHandler.processMessage(MFCImplementationHandler.java:199)
at com.ibm.ws.sca.internal.message.impl.MessageDispatcherImpl.processMessageWithPCI(MessageDispatcherImpl.java:715)
at com.ibm.ws.sca.internal.message.impl.MessageDispatcherImpl.processMessage(MessageDispatcherImpl.java:1167)
at com.ibm.ws.sca.internal.message.impl.ManagedMessageImpl.process(ManagedMessageImpl.java:843)
at com.ibm.wsspi.sca.ejb.module.impl.ModuleSessionBean.processUOWMessage(ModuleSessionBean.java:336)
at com.ibm.wsspi.sca.ejb.module.impl.ModuleSessionBean.transactionRequiredActivitySessionNotSupported(ModuleSessionBean.java:315)
at com.ibm.wsspi.sca.ejb.module.EJSLocalStatelessModule_43132892.transactionRequiredActivitySessionNotSupported(EJSLocalStatelessModule_43132892.java:233)
at com.ibm.ws.sca.internal.uow.handler.UOWStrategyImpl.transactionGlobalActivitySessionFalse(UOWStrategyImpl.java:311)
at com.ibm.ws.sca.internal.uow.handler.JoinUOWHandler.processMessage(JoinUOWHandler.java:165)

Or you may have classloading problems when trying to work with monitor models.

Same issues may occur with fixpack 1 of both products (WPS/WESB and WBMonitor
This is happening because WPS/WESB and WBM/BAM each comes with its own version of JXPath libraries. A fix for this issue will soon be publicaly available from the monitor team. It would restrict visibility of JXPath library packaged with WBM to monitor code.
If your symptoms match, please ask IBM support about JR33245.