Wednesday, December 17, 2014

WSO2 BAM - APIM error - Failed to write data to database

When viewing statistics of APIs in WSO2 API Manager 1.7.0 via WSO2 BAM, you may have come across the following issue. [1]

As you might know, AM sends events about requests to APIs to WSO2 BAM. And, BAM stores these data in Cassandra storage which gets later by Hive analytics scripts. For AM, we have the am_stats_analyzer. After analyzing, the summarized information gets written to a RDBMS instance. The summarized db could be mysql, oracle or even a in-memory H2 DB.


This error [1] comes when a column in a summarized database table is too small to store a given value. In this, it's the resourcePath of an API. By default, the size for the resourcePath column is set to VARCHAR(100). If the resourcePath of an API is longer than 100 characters, then this error will be thrown.


The 'resourcePath' is defined in the API_Resource_USAGE_SUMMARY table in the summary db.

  • If the system is already up and running, the summarized mysql tables are already created. Therefore, we need to alter the tables to modify column lengths. You can use the following steps for that.

1. Since the issue is in a RDBMS such as mysql, you first need to log-in to console where you can execute SQL statements.
2. Then, execute the following statement.

alter table API_Resource_USAGE_SUMMARY modify resourcePath MEDIUMTEXT

  • If the summarized tables are not created yet, then you could go and and modify the table creation script in the am_stats_analyzer.
  1. Open the API_Manager_Analytics.tbox.
  2. In there, you will find am_stats_analyzer analytics script.
  3. Open that, and look for the hive.jdbc.table.create.query, which has the following summary table creation sql statement.
 CREATE TABLE API_Resource_USAGE_SUMMARY ( api VARCHAR(100), version VARCHAR(100),apiPublisher VARCHAR(100) , consumerKey VARCHAR(100),resourcePath VARCHAR(100) ,context VARCHAR(100),
        method VARCHAR(100), total_request_count INT, hostName VARCHAR(100), year SMALLINT, month SMALLINT, day SMALLINT, time VARCHAR(30), PRIMARY KEY(api,version,apiPublisher,consumerKey,context,method,time))

Change the type of resourcePath from VARCHAR(100) to MEDIUMTEXT. Save the script.

ERROR {} - Failed to write data to database {}
com.mysql.jdbc.MysqlDataTruncation: Data truncation: Data too long for column 'resourcePath' at row 1
        at com.mysql.jdbc.MysqlIO.checkErrorPacket(
        at com.mysql.jdbc.MysqlIO.checkErrorPacket(
        at com.mysql.jdbc.MysqlIO.sendCommand(
        at com.mysql.jdbc.MysqlIO.sqlQueryDirect(
        at com.mysql.jdbc.ConnectionImpl.execSQL(
        at com.mysql.jdbc.PreparedStatement.executeInternal(
        at com.mysql.jdbc.PreparedStatement.executeUpdate(
        at com.mysql.jdbc.PreparedStatement.executeUpdate(
        at com.mysql.jdbc.PreparedStatement.executeUpdate(
        at org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(
        at org.apache.hadoop.hive.ql.exec.Operator.process(
        at org.apache.hadoop.hive.ql.exec.Operator.forward(
        at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(
        at org.apache.hadoop.hive.ql.exec.Operator.process(
        at org.apache.hadoop.hive.ql.exec.Operator.forward(
        at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(
        at org.apache.hadoop.hive.ql.exec.Operator.process(
        at org.apache.hadoop.hive.ql.exec.Operator.forward(
        at org.apache.hadoop.hive.ql.exec.GroupByOperator.forward(
        at org.apache.hadoop.hive.ql.exec.GroupByOperator.processAggr(
        at org.apache.hadoop.hive.ql.exec.GroupByOperator.processOp(
        at org.apache.hadoop.hive.ql.exec.Operator.process(
        at org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(
        at org.apache.hadoop.mapred.ReduceTask.runOldReducer(

Tuesday, December 2, 2014

Fixing 101505 WSO2 Connection close issues


The error code 101505 means the HTTP connection initiated by the ESB has been terminated at the backend. According to the log you can determine whether message has been completely written to the backend prior to the connection close (See [Status SendingCompleted : true] in the log.) Users will usually get an error message such as the following.

fault: {
code: "101505"
type: "Status report"
message: "Runtime Error"
description: "Connection close For : x.x.x.x For Request : Axis2Request [Message ID : urn:uuid:84445830-ed4e-5c15-cd87-bb5b21869e3f] [Status Completed : false] [Status SendingCompleted : true]"

Possible fix:

Some Back-end servers do not support persistent connections. So, they tend close down the connections intermediately. If this is the case, then we should disable the Keep-Alive connections at ESB side for such back-ends.

By default Keep-Alive is enabled in WSO2 ESB/APIM. You can switch off this per proxy/api by adding the following property mediator.

<property name="NO_KEEPALIVE" value="true" scope="axis2"/> 

But If you need to disable Keep-Alive globally for all the connections initiated from ESB, then that configuration should be specified in transport configuration files.

So, if you are using NIO transport, then add the following property to ESB_HOME/repository/conf/ But, if you are using Passthru transport, then add the following property to ESB_HOME/repository/conf/


Saturday, November 29, 2014

Restrict WSO2 Carbon management console access

The WSO2 Carbon products provides users with a browser based management console to administer the WSO2 products. All the WSO2 products comes with a management console to make the administration easier. When hosting the products ready for production, it's imperative to restrict the access to the mgt console from the external network.

Following steps shows how to restrict the access to WSO2 Carbon management console by IP.

Carbon itself gets added to the Tomcat runtime as a webapp. The "/carbon" is a servlet under it. Hence, we can use the Tomcat servlet filter, org.apache.catalina.filters.RemoteAddrFilter to restrict access.

1. Open CARBON_HOME/repository/conf/tomcat/carbon/WEB-INF/web.xml
2. Then, add the filter, org.apache.catalina.filters.RemoteAddrFilter, as follows. This filter can be used to restrict access by IP. Following is an example valve configuration to restrict the access only to localhost (considering both IPv4 and IPv6).

    <filter-name>Remote Address Filter</filter-name>

    <filter-name>Remote Address Filter</filter-name>

For parameter value reference, see -

Tuesday, August 5, 2014

JSF webapps fronted with reverse proxy

JSF web applications can be fronted by a reverse proxy just like any other applications, except when the reverse proxy has a different context path compared to the application server that this JSF webapp is deployed to.

Look at the following HTML content of a JSF Forms application. The 'action' URL of the form begins with a forward-slash '/'. This URL is actually a absolute url without the hostname. So, this is not a relative URL. This is a problem if your reverse proxy has a different context. Since the JSF webapp is not aware of the context path of the reverse proxy, the JSF form action will fail.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "">
<html xmlns="">
  <h2>Welcome to Marks Calculator</h2>
  <form id="j_id_4" name="j_id_4" method="post" 
    <table cellspacing="10">
          <td>Subject 1 marks:</td>
            <input id="j_id_4:j_id_8" name="j_id_4:j_id_8" type="text" value="2" />
          <td>Subject 2 marks:</td>
            <input id="j_id_4:j_id_b" name="j_id_4:j_id_b" type="text" value="4" />
    <input id="j_id_4:j_id_c" name="j_id_4:j_id_c" type="submit" value="Calculate" />
    <input type="hidden" name="j_id_4_SUBMIT" value="1" />
    <input type="hidden" name="javax.faces.ViewState" id="javax.faces.ViewState" value="wpq2AMaIFASZtUJ+IghBD2X2mhXm5MAFTDfdI20attDvNPOF " />

Now, let's look at how the Apache HTTPD configuration might look like. Do not add forward-slash at the end of ProxyPass/ProxyPassReverse since the Location, '/as', do not have a forward-slash at the end either.

<Location /as>
    Order Deny,Allow
    Deny from none
    Allow from all
    ProxyPass http://localhost:9763 retry=0 timeout=5
    ProxyPassReverse http://localhost:9763

Apache is configured to forward requests that come to http://localhost/as/ to http://localhost:9763/.

Now, you can access the webapp by using the request URL - http://localhost/as/jsf-custom-viewhandler-marks/index.jsf. Now, because of the JSF form 'action' URL, the form submit will try to call http://localhost/jsf-custom-viewhandler-marks/index.jsf which is non-existent.

Now, I hope you understand what's the problem here. In my view, this is a limitation of JSF. So, let's look at how to fix this.


Implementing A ViewHandler

To fix this, you need to implement javax.faces.application.ViewHandler, and register it to your webapp. In this custom view handler, we will set the action url as a relative url rather than a absolute url without the host name.

Needed dependencies -
commons-lang 2.6
commons-logging 1.1


import org.apache.commons.lang.StringUtils;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;

import javax.faces.application.ViewHandler;
import javax.faces.application.ViewHandlerWrapper;
import javax.faces.context.FacesContext;
import javax.servlet.http.HttpServletRequest;
import java.util.List;
import java.util.Map;

public class CustomViewHandler extends ViewHandlerWrapper {
  private static final Log log = LogFactory.getLog(CustomViewHandler.class);

  private ViewHandler wrappped;

  public CustomViewHandler(ViewHandler wrappped) {
    this.wrappped = wrappped;


  public ViewHandler getWrapped() {
    return wrappped;

  public String getActionURL(FacesContext context, String viewId) {
    String url =  super.getActionURL(context, viewId);
    log.debug("The getActionURL: " + url);
    return addContextPath(context, url);

  public String getRedirectURL(FacesContext context, String viewId, Map<String,
          List<String>> parameters, boolean includeViewParams) {
    String url =  super.getRedirectURL(context, viewId, parameters, includeViewParams);
    log.debug("The getRedirectURL: " + url);
    return url;

  public String getResourceURL(FacesContext context, String path) {
    String url = super.getResourceURL(context, path);
    log.debug("The getResourceURL: = " + url);
    return addContextPath(context, url);

  private String addContextPath(FacesContext context, String url) {
      final HttpServletRequest request = ((HttpServletRequest) 
      String result = url;
      if (url.startsWith("/")) {
          int subpath = StringUtils.countMatches(getPath(request), "/") - 1;
          String pathPrefix = "";
          if (subpath > 0) {
              while (subpath > 0) {
                  pathPrefix += "/..";
              pathPrefix = StringUtils.removeStart(pathPrefix, "/");
          result = pathPrefix + result;
      return result;

    private String getPath(final HttpServletRequest request) {
        try {
            return StringUtils.replace(new URI(request.getRequestURI()).getPath(), "//", "/");
        } catch (final URISyntaxException e) {
            return StringUtils.EMPTY;


In the addContextPath private method, we modify the original url, and replace it with a relative url.

Then, build this class into a jar and pack it into WEB-INF/lib of your web application. You can find the source and a Apache Maven build file at the end of this article.

Now, you need to register this ViewHandler with your web application. For that, open your faces-config.xml and add your view handler as follows. The faces-config.xml is usually placed under WEB-INF/ folder in the webapp.


Now, the faces-config.xml might look as follows.

<?xml version='1.0' encoding='UTF-8'?>
<faces-config xmlns=""





Setting the sessionCookiePath

Some JSF webapps depends on sessions and JSESSIONID cookie. Usually, JSF sets this cookie to the webapp path. But if the reverse proxy has a different context, then the cookie fails to get stored in the browser properly. In my case, I use EJB with the JSF webapp, and I needed to fix this issue. Otherwise, I faced the below exception.

To fix this, you need to add context.xml to your webapp, and set the sessionCookiePath attribute. We can configure the JSESSIONID cookie path via this configuration file. In our case, we can set it as follows. See the tomcat documentation for more details.

<Context sessionCookiePath="/as/jsf-custom-viewhandler-marks">

There, the path has taken into account the reverse proxy configuration path "/as".

Now you are all set. You can use the given Custom View Handler sample code in your own JSF webapps. I have also posted a sample JSF webapp that uses EJB. The sample app can be deployed Apache TomEE.

You can find a sample JSF Custom View Handler code here -

You can find the sample webapp here -

Tuesday, May 27, 2014

Invoke STS Service in WSO2 IS using CXF STS Client

Following guide describes how to invoke the STS Service in WSO2 Identity Server using a CXF STS client. The STS Client is configured via Spring.

As I have noticed, CXF STS client fails to invoke the STS service if you use the wso2carbon-sts wsdl to generate the client. So, I have followed a different approach.


WSO2 Identity Server 4.6.0 or latter
Apache CXF 2.7.x


What I'm going to show you is using the CXF client via a simple stand-alone java class with Spring.


1. First, you need to define the spring bean. Let's say the file is named wssec-sts-bean.xml.

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns=""
       xsi:schemaLocation="    ">

<bean id="{}wso2carbon-stsHttpsSoap12Endpoint.sts-client"
    <constructor-arg ref="cxf"/>
    <!--<property name="wsdlLocation" value="https://localhost:9443/services/wso2carbon-sts?wsdl"/>-->
    <property name="Location"
    <property name="serviceName" value="{}wso2carbon-sts"/>
    <property name="endpointName" value="{}wso2carbon-stsHttpsSoap12Endpoint"/>

    <property name="properties">
            <entry key="ws-security.username" value="admin"/>
            <!--<entry key="ws-security.password" value="admin"/>-->
            <entry key="ws-security.callback-handler" value="com.cxf.sts.ClientCallbackHandler"/>
            <entry key=""
            <entry key="ws-security.encryption.username" value="wso2carbon"/>
            <entry key="ws-security.sts.applies-to"

NOTES: Please note that the wsdlLocation has NOT been set. Instead, we set the Location attribute which is the SOAP 1.2 STS https endpoint. Other attributes are the usual attributes that we normally set.

2. For the property in the spring bean config, you have to  point to a properties file that has the encryption properties. The contents of would look like follows.

3. We need one more configuration file to store the ws-policy of the STS service. First, you need to secure the STS service in IS as described in this documentation -
Now, if you browse to the wso2carbon-sts WSDL, you will see the policy that is applied to your service. Extract the policy element from the WSDL. It starts with <wsp:Policy ...>. If you secure the service with UsernameToken, then the policy would like below. Place it in a file called sts.policy.xml.

<wsp:Policy xmlns:wsp="" xmlns:wsu="" wsu:Id="UTOverTransport">
    <wsp:ExactlyOne  xmlns:wsp="" xmlns:wsu="" >
            <sp:TransportBinding xmlns:sp="">
                        <sp:HttpsToken RequireClientCertificate="false"></sp:HttpsToken>
            <sp:SignedSupportingTokens xmlns:sp="">
                    <sp:UsernameToken sp:IncludeToken=""></sp:UsernameToken>

Now all the configuration files are in place. Now, we can go ahead, and write java code to retrieve the sts tokens.

STS Service Invocation

1. First, we need to set the trust-store of wso2 is as a system property since we are invoking a https endpoint. And, then, we will read the spring beans file.

System.setProperty("", "wso2carbon");

2. Next, we should read the spring bean file, wssec-sts-bean.xml, and get the bean we created.

ApplicationContext ctx = new FileSystemXmlApplicationContext(
STSClient sts = (STSClient) ctx.

3. Now, we have the STSClient instance. We need to set some attributes to the client before we can request security tokens. One is the policy we just extracted from the WSDL.

//parse the ut policy xml, and get a DOM element
File f = new File("src/main/resources/sts.policy.xml");
Element stsPolicy = loadPolicy(f.getAbsolutePath());


4. Then, we can go ahead and request sts tokens.

SecurityToken samlToken = 
        "/Issue", null);

//convert the token dom element to string
String token = ((DOMImplementationLS) samlToken.getToken().getOwnerDocument().getImplementation()).

With these steps, you should be able to invoke STS service in WSO2 Identity Server using a CXF client. You can download the full source of this from here -



1. Start WSO2 Identity Server 4.6.0.
2. Secure STS Service with UsernameToken as described in this documentation -
3. Extract the client,
4. Run the client as follows.

mvn clean install -Pclient

Friday, May 2, 2014

Different ways to Revert GIT commits

In GIT, there are different requirements to revert changes. This includes reverting local changes in your working copy, revert a commit already made, rewrite already pushed GIT history etc. I'll show you how to do these in this blog. The `git log` I have used can be found at the end of this article.

  • To temporarily go back to a old revision, fool around, then come back to where you are, then you should checkout the desired commit.
git checkout 1f5d1eb7e

If you want to make some commits while you are in this older revision:
git checkout -b old_revision_branch 1f5d1eb7e

But, if you want to completely remove the commits you have made since a given revision, then there are two routes to go:

1. If you have pushed any commits:
#This will destroy all your local modifications:
git reset --hard 1f5d1eb7e

#But, if you have local work to keep:
git stash
git reset --hard 1f5d1eb7e
git stash pop
#This saves the modifications (git stash), and re-applies (git stash pop) it on top of the reset copy. You might get merge conflicts.

2. If you have already pushed the work:
Then you might not want to actually remove all the commits which will effectively rewrite the history. But what you can do is you can revert the commits. With git, revert has a very specific meaning: create a commit with a reverse patch to cancel it out. This way, you don't rewrite any history.

# This will create three separate commits
git revert 2ce3f91c4 0cc5599c5 3e5f8805a

# It also takes ranges. This will revert the last two commits:
git revert HEAD~2..HEAD

#Finally, commit
git commit

`git log`

commit 3e5f8805aa06d80d50f130107d48e4ec67289719
Author: Kasun Gajasinghe <>
Date:   Wed Apr 30 02:14:20 2014 +0530

    cleaning up cxf ws-discovery

commit 0cc5599c57f443a937affcd2045617d8660a58e4
Author: Kasun Gajasinghe <>
Date:   Wed Apr 30 01:51:51 2014 +0530

    moving APIScanner class to correct package

commit 2ce3f91c4262749ca262ddc0bf75924f13bbb485
Author: Kasun Gajasinghe <>
Date:   Wed Apr 30 01:45:12 2014 +0530

    Adding support to read the WSDL 1.1 wsdl:portType from the JAX-WS annotations.

commit 1f5d1eb7ef6152de32f5c8282e765f3e8aaa6468
Author: chanikag <>
Date:   Mon Apr 28 08:37:07 2014 +0530

    upgrade data-bridge conmponent to revision 201003

commit 64958bc9df641b3628d6d88ed39acb5e37eae688
Author: Chamath Gunawardana <>
Date:   Fri Apr 25 12:56:20 2014 +0530

    Updating pom

This was generously inspired from a Stackoverflow answer at

Monday, February 10, 2014

Resolve activation Module not found issue in WSO2 ESB 4.8.0

When you were upgrading your WSO2 ESB pfrom 4.7.0 or older version to 4.8.0, you might notice the following issue.  Caused by: org.wso2.carbon.CarbonException: Axis Module not found for : activation-2.1.2

This error happens if you migrate the ESB_HOME/repository/deployment/server/servicemetafiles/ folder to the ESB 4.8.0 or a later version. The reason is that activation module is not shipped with ESB anymore, but the servicemetafiles from the older version might still contain references to activation module. Since ESB cannot find that module, it throws the following error, and the proxy service fails to deploy.

To resolve this issue, you have to edit the metafiles to remove references to activation module. There, you might notice, the references like follows. The version might differ.

         <module name="activation" version="2.1.2" type="engagedModules"/>

Simply remove this line from the metafile, and then the proxy services will deploy successfully. You can create a script to do this if you have several proxy services / axis2 services.

[2014-02-10 13:02:09,156] ERROR - PersistenceMetaDataDeployer Unable to handle service initialization. Service: myProxyService
org.wso2.carbon.core.persistence.PersistenceException: Unable to handle service initialization. Service: myProxyService
    at org.wso2.carbon.core.persistence.AbstractPersistenceManager.handleExceptionWithRollback(
    at org.wso2.carbon.core.persistence.ServicePersistenceManager.handleExistingServiceInit(
    at org.wso2.carbon.core.persistence.file.deployer.PersistenceMetaDataDeployer.deploy(
    at org.apache.axis2.deployment.repository.util.DeploymentFileData.deploy(
    at org.apache.axis2.deployment.DeploymentEngine.doDeploy(
    at org.apache.axis2.deployment.repository.util.WSInfoList.update(
    at org.apache.axis2.deployment.RepositoryListener.update(
    at org.apache.axis2.deployment.RepositoryListener.checkServices(
    at org.apache.axis2.deployment.RepositoryListener.startListener(
    at org.apache.axis2.deployment.scheduler.SchedulerTask.checkRepository(
    at org.wso2.carbon.core.deployment.CarbonDeploymentSchedulerTask.runAxisDeployment(
    at java.util.concurrent.Executors$
    at java.util.concurrent.FutureTask$Sync.innerRunAndReset(
    at java.util.concurrent.FutureTask.runAndReset(
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(
    at java.util.concurrent.ScheduledThreadPoolExecutor$
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(
    at java.util.concurrent.ThreadPoolExecutor$
Caused by: org.wso2.carbon.CarbonException: Axis Module not found for : activation-2.1.2
    at org.wso2.carbon.core.persistence.AbstractPersistenceManager.getExistingAxisModule(
    at org.wso2.carbon.core.persistence.ServicePersistenceManager.handleExistingServiceInit(
    ... 20 more