Disable Chunking in APIs of WSO2 API Cloud

Sometimes it may be required to disable chunking in the API requests done via WSO2 API Cloud. Some back-end servers does not accept chunked content and therefore you may need to disable chunking.

In a Standalone WSO2 APIM, this can be done by editing the velocity template as described in this blog post. However in WSO2 API Cloud it is not possible to edit the velocity template.

This can be done with the use of a custom mediation extension which will disable chunking, as described below.

The corresponding sequence would be like;

<?xml version="1.0" encoding="UTF-8"?>
<sequence xmlns="http://ws.apache.org/ns/synapse"
          name="disable-chunking">
        <property name="DISABLE_CHUNKING" value="true" scope="axis2" />
</sequence>

You can find a sample sequence (disable-chunking.xml) which you can upload to the Governance registry of the API Cloud using the management console UI of Gateway (https://gateway.api.cloud.wso2.com/carbon).

You can get the user name from the top right corner of the Publisher and then enter your password and log in. Once you are logged in select Resources (left hand side of the Management console) and click on Browse and then navigate to /_system/governance/apimgt/customsequences registry location. Since this sequence need to be invoked in the In direction (or the request path) navigate to the in collection. Click Add Resource and upload the XML file of the sequence configuration and add it.  (Note: Once you add the sequence it might take up-to 15 minutes until it is deployed into the publisher)

In Publisher select the required API and go to edit wizard by clicking edit and then navigate into Manage section. Click on Sequences check box and select the sequence which disable chunking, from the In Flow. After that Save and Publish the API.

Now invoke the API. If you investigate your requests, you will notice that chunking is disabled.

Chunking Enabled

POST customerservice/customers HTTP/1.1
Content-Type: text/xml; charset=ISO-8859-1
Accept: text/xml
Transfer-Encoding: chunked
Host: 192.168.77.1:8080
Connection: Keep-Alive
User-Agent: Synapse-PT-HttpComponents-NIO

Chunking Disabled

POST customerservice/customers HTTP/1.1
Content-Type: text/xml; charset=ISO-8859-1
Accept: text/xml
Content-Length: 42
Host: 192.168.77.1:8080
Connection: Keep-Alive
User-Agent: Synapse-PT-HttpComponents-NIO

Highest performing and Fastest ESB - WSO2 ESB 4.8.1

The latest performance study for WSO2 ESB was released. It is based on the latest verion of WSO2 ESB - 4.8.1.

It describes stabilization activities performed to WSO2 ESB and compares performance against a number of leading open source ESBs.

Observations

Results shows how the latest WSO2 ESB convincingly outperforms all other open-source ESBs. This also indicates that it is the fastest open source ESB available currently.

You can find the article from here: ESB Performance Round 7.5

So don't wait...Try the fastest ESB from here...! It will only take couple of minutes to get it up and running...!

Using Email address as the Username in WSO2 Identity server (Email based authentication)

It is a common scenario in organizations to use email address as the user name to log into various organizational applications/systems. If you are using WSO2 Identity Server (or any WSO2 server)  in your enterprise solutions and if you want to enable email as the user name, in this post I will explain how you can configure it to use email as the user name.

I will use the default LDAP instance which ships with IS to keep the users. You can use either your own LDAP instance or Active Directory instance instead of the default LDAP that ships with IS. Refer this for configuring an external LDAP or Active Directory to work with WSO2 Identity Server.

You can download WSO2 Identity Server from here.

Extract it and make sure you have set the environment correctly in order to run the server. If you need more information on running WSO2 Identity Server you can refer this.

I will use IS_HOME to refer the extracted location of the server.

Step 1: Open carbon.xml in IS_HOME/repository/conf  and uncomment

       <EnableEmailUserName>true</EnableEmailUserName>

Step 2: Open user-mgt.xml in IS_HOME/repository/conf and do the following changes

First you have to modify the Configuration section as bellow;

If you need to add the admin user to the LDAP/AD set AddAdmin to ‘true’. If you are using an user already existing in the LDAP/AD set this as false.

                <AddAdmin>true</AddAdmin>

UserName  → If you need to use an user already available in the AD/LDAP specify that users email here. Or else if you need to add an admin (as in above step) specify the required admin email

               <UserName>admin@wso2.com</UserName>

Passsword → if the admin is a already existing user keep the value as it is. Do not change it. Or else if you are adding the admin, specify a password here. Later change the password in the AD/LDAP itself while keeping the older or a dummy password here.

              <Password>admin</Password>

Refer this for more information on the above.

Next you have to find the correct UserStoreManager section which is being used (or uncommented in default) and modify the below properties.(By default this will have the configurations required to connect with the Internal LDAP)

UserNameListFilter → (&amp;(objectClass=identityPerson)(mail=*))

UserNameSearchFilter → (&amp;(objectClass=identityPerson)(mail=?))

UserNameAttribute → mail

UsernameJavaRegEx → ^[_A-Za-z0-9-\+]+(\.[_A-Za-z0-9-]+)*@[A-Za-z0-9-]+(\.[A-Za-z0-9]+)*(\.[A-Za-z]{2,})$

Comment  out UserDNPattern and RoleDNPattern

       <!--Property name="UserDNPattern">uid={0},ou=Users,dc=wso2,dc=org</Property>
       <Property name="RoleDNPattern">cn={0},ou=Groups,dc=wso2,dc=org</Property-->


Thats it. Now start the server and you can login with the admin user name and password you provided. Make sure you change the password though Management console or by LDAP it self.

You can download the user-mgt.xml which I used with WSO2 Identity Server 4.6.0 for your reference.

Start a WSO2 Server as a Linux service

Sometimes there can be a requirement to start WSO2 Servers as Linux services. For example to start the server automatically in the boot-up and {start|stop|restart} as a normal Linux service later. This can be achieved easily with a small startup script which can be added to Linux startup.

If you need to run a service in the boot up, first you have to create a startup script and add it to the boot sequence. There are three main parts in the script; start, stop and restart. The basic structure of a startup service script is as follows;   

#!/bin/bash
 
case “$1″ in
start)
   echo “Starting Service”
;;
stop)
   echo “Stopping Service”
;;
restart)
   echo “Restarting Service”
;;
*)
   echo $”Usage: $0 {start|stop|restart}”
exit 1
esac

Following script is a startup script written for WSO2 Application Server 5.2.0 with the above format.

#! /bin/sh
export JAVA_HOME="/usr/lib/jvm/jdk1.7.0_07"

startcmd='/opt/WSO2/wso2as-5.2.0/bin/wso2server.sh start > /dev/null &'
restartcmd='/opt/WSO2/wso2as-5.2.0/bin/wso2server.sh restart > /dev/null &'
stopcmd='/opt/WSO2/wso2as-5.2.0/bin/wso2server.sh stop > /dev/null &'

case "$1" in
start)
   echo "Starting WSO2 Application Server ..."
   su -c "${startcmd}" sumedha
;;
restart)
   echo "Re-starting WSO2 Application Server ..."
   su -c "${restartcmd}" sumedha
;;
stop)
   echo "Stopping WSO2 Application Server ..."
   su -c "${stopcmd}" sumedha
;;
*)
   echo "Usage: $0 {start|stop|restart}"
exit 1
esac

In this script the server will be started as the user 'sumedha' rather than the root.

    i.e. :  su -c "${startcmd}" sumedha

Then you have to add this script to /etc/init.d/ directory. Optionally you can only add a symbolic link to the script in /etc/init.d/ while keeping the actual script in a separate place. Also you have to make the script executable.

Lets say my script is 'appserver' and it is in /opt/WSO2/, then the command for adding it to /etc/init.d/ will be as follows;

    Make executable :   sudo chmod a+x /opt/WSO2/appserver
    Add a link to /etc/inin.d/ :   sudo ln -snf /opt/WSO2/appserver /etc/init.d/appserver

After that you have to install this script to respective run levels. This can be done by update-rc.d command. For above script it is as follows;

    sudo update-rc.d appserver defaults

If defaults is used then update-rc.d  will  make the service  to  start  in runlevels 2,3,4,5 and to stop in runlevels 0,1,6.

Now when you boot the system, the App Server will be up and running. 

You can {start|stop|restart} it by 'service appserver {start|stop|restart} '. You have to give the password of the user (or root) who was used to start the service.

Hope this is helpful..!

A Simple way to log time durations in WSO2 ESB

In this post I will explain a simple mechanism to log a time duration in WSO2 ESB. Already there is a great and powerful way of getting statistics of WSO2 ESB which is capable of providing runtime statistical information from sequences, endpoints, proxies and view it through the ESB management console. This will help you to analyze response times, counts and many more performance related options very effectively.

But lets assume, for debugging purposes (or any other purpose) you need a quick and simple way of measuring time durations. For example imagine that you have a proxy service which calls to a back end and returns the response from the back end. Lets say you need to get a rough idea on the time taken to call the back end and get the response. This kind of a scenario can be simply achieved by the use of property mediator, script mediator and a log mediator.

There is a Synapse Message Context Property which can retrieve the current time is milliseconds (i.e. the difference, measured in milliseconds, between the current time and midnight, January 1, 1970 UTC). We can set this value to a property (at the desired point of the message flow) and access it when ever we need.

For example, following is how to set the time to the property 'TIME' and log it later;

<property name="TIME" expression="get-property('SYSTEM_TIME')" scope="default" type="LONG"/>
...
<log>
 <property name="Time : " expression="get-property('TIME')"/>
</log>
...

Like wise we can store the current system time in milliseconds, at desired points in the message flow into different properties. After that we can use a Script mediator to read these values and perform the required calculations.

For example if we have set two time values in propeties 'TIME_1' and 'TIME_2' we can get the difference between those as follows;

<script language="js">
 var time1 = mc.getProperty("TIME_1");
 var time2 = mc.getProperty("TIME_2");
 var timeTaken = time2 - time1;
 print("Time Duration :  " + timeTaken + " ms ");
 mc.setProperty("RESPONSE_TIME", timeTaken);
</script>

Also we can set the result to another property (mc.setProperty("RESPONSE_TIME", timeTaken);) which can be used in the message flow (for example to log the time with the Log mediator).

Therefore this will provide a simple mechanism to log a time in WSO2 ESB. A sample proxy with this kind of a scenario is as follows;

<proxy xmlns="http://ws.apache.org/ns/synapse"
       name="SampleTimeProxy"
       transports="https,http"
       statistics="disable"
       trace="disable"
       startOnLoad="true">
   <target>
      <inSequence>
         <property name="TIME_1"
                   expression="get-property('SYSTEM_TIME')"
                   scope="default"
                   type="LONG"/>
      </inSequence>
      <outSequence>
         <send/>
         <property name="TIME_2"
                   expression="get-property('SYSTEM_TIME')"
                   scope="default"
                   type="LONG"/>
         <script language="js">
              var time1 = mc.getProperty("TIME_1");
              var time2 = mc.getProperty("TIME_2");
              var timeTaken = time2 - time1;
              print("--------------  " + timeTaken + " ms  -----------------");
              mc.setProperty("RESPONSE_TIME", timeTaken);
         </script>
         <log>
            <property name="Time Duration in ms: " expression="get-property('RESPONSE_TIME')"/>
         </log>
      </outSequence>
      <endpoint>
         <address uri="http://localhost:8080/axis2/services/SimpleStockQuoteService"/>
      </endpoint>
   </target>
   <publishWSDL uri="http://localhost:8080/axis2/services/SimpleStockQuoteService?wsdl"/>
   <description/>
</proxy>

Hope this will be helpfull...!

Clustering WSO2 ESB - Part 2 - Distributed worker-manager setup with one manager and two workers

In the previous post I explained how you can cluster two WSO2 ESB servers where  one ESB node will be playing a dual role as a worker as well as a manager. The setup which I am going to explain in this post is similar to the above, but it has three ESB servers where one server will be dedicated only for management purposes while the other two will be dedicated workers.

As the previous one, this setup also can be used as a Active/Passive setup where the active-passiveness can be decided by the load balancer based on a predefined mechanism(echo test or a heartbeat mechanism). 

The overall setup will contain three ESB servers (nodes), in which one server will act as the Manager while other two will act as the workers. The Manager will be used for deploying artifacts (CAR files and proxies) and managing the cluster.

The  synchronization  of artifacts between the servers will be achieved through the SVN based Deployment Synchronizer.

The overall view of the cluster will be as in the following figure.

Configuring the Manager

Download and extract the WSO2 ESB distribution (referred as MANAGER_HOME ).

axis2.xml configuration

Clustering   should  be  enabled  at  axis2  level  in  order  for  management  node  to  communicate with  the   worker   nodes.  
Open  MANAGER_HOME/repository/conf/axis2/axis2.xml  and  update the clustering configuration as follows:

<clustering class="org.apache.axis2.clustering.tribes.TribesClusteringAgent"  enable="true">
<parameter name="membershipScheme">wka</parameter>

Specify the cluster domain as follows;

<parameter name="domain">wso2.esb.domain</parameter>

Uncomment localmemberhost element in axis2.xml and specify the IP address (or host name) to be exposed to members of the cluster. This address can be an internal address used only for clustering.

<parameter name="localMemberHost">127.0.0.1</parameter>

Define the port through which other nodes will contact this member. If several instances are running in the same host make sure to use ports which will not conflict.

<parameter name="localMemberPort">4001</parameter>

Comment out the static or well­-known members, because this will be the well­-known member. (i.e do not need to have a list of members)
Add a new property "subDomain" under properties and set it to "mgt" to denote that this node belongs to management subdomain.  

<property name="subDomain" value="mgt"/> 

carbon.xml configuration

If multiple WSO2 Carbon ­based products are running in same host, to avoid possible port conflicts, the port offset of MANAGER_HOME/repository/conf/carbon.xml should be changed as follows. Ignore this if you are using three separate hosts.

<Offset>1</Offset>

Configure the Deployment Synchronizer as follows. Make sure both AutoCommit and
AutoCheckout are ‘true’

<DeploymentSynchronizer>
     <Enabled>true</Enabled>
     <AutoCommit>true</AutoCommit>
     <AutoCheckout>true</AutoCheckout>
     <RepositoryType>svn</RepositoryType>
     <SvnUrl>http://svnrepo.example/repos/esb</SvnUrl>
     <SvnUser>USERNAME</SvnUser>
     <SvnPassword>PASSWORD</SvnPassword>
     <SvnUrlAppendTenantId>true</SvnUrlAppendTenantId>
</DeploymentSynchronizer>

Start the ESB manager node

 ./bin/wso2server.sh

You can access the management console by: https://localhost:9444/carbon/ within the managers node. (Use the correct port according to the port offset which was defined earlier) 

Add a sample proxy though the console and check whether it is getting committed to the SVN correctly. Proxy will be committed to SvnUrl/-1234/synapse-configs/default/proxy-services/ if you created the proxy as the admin.

Configuring the Worker node 1

Download and extract the WSO2 ESB distribution(referred as WORKER_HOME).

axis2.xml configuration

Clustering   should  be  enabled  at  axis2  level  in  order  for  management  node  to  communicate with  the  worker  nodes. Open  WORKER_HOME/repository/conf/axis2/axis2.xml and update the clustering configuration as follows:

<clustering class="org.apache.axis2.clustering.tribes.TribesClusteringAgent" enable="true">
<parameter name="membershipScheme">wka</parameter>

Specify the cluster domain as follows;

<parameter name="domain">wso2.esb.domain</parameter>

Define  the  port  through  which  other  nodes  will  contact  this  member.  If several instances are running in the same host make sure to use ports which will not conflict.

<parameter name="localMemberPort">4002</parameter>

Add managers(Node 1) IP address (or host name) and the port as the well known member.

• hostName: defined as localMemberHost in managers axis2.xml

• port: defined as localMemberPort in managers axis2.xml.

Make  sure  this  address  can  be  accessed  through  the  workers  host. If  you  use  host name, map it to managers IP in the ‘etc/hosts’ file of the workers machine.

<members>
   <member>
   <hostName>127.0.0.1</hostName>
      <port>4001</port>
   </member>
</members> 

Add a new property "subDomain" under properties and set it to "worker" to denote that this node belongs to worker subdomain.  

<property name="subDomain" value="worker"/> 

carbon.xml configuration

If multiple WSO2 Carbon­ based products are running in same host, to avoid possible port conflicts, the port offset of WORKER_HOME/repository/conf/carbon.xml should be changed as follows. Ignore this if you are using three separate hosts.

<Offset>2</Offset>

Configure the Deployment Synchronizer as follows. Make sure AutoCommit is set to ‘false’ and AutoCheckout to ‘true’

<DeploymentSynchronizer>
     <Enabled>true</Enabled>
     <AutoCommit>false</AutoCommit>
     <AutoCheckout>true</AutoCheckout>
     <RepositoryType>svn</RepositoryType>
     <SvnUrl>http://svnrepo.example/repos/esb</SvnUrl>
     <SvnUser>USERNAME</SvnUser>
     <SvnPassword>PASSWORD</SvnPassword>
     <SvnUrlAppendTenantId>true</SvnUrlAppendTenantId>
</DeploymentSynchronizer> 

Start the ESB worker node. The workerNode system property must be set to true when starting the workers. 
Therefore worker should be started as follows (in linux),

./bin/wso2server.sh -DworkerNode=true 

Configuring Worker node 2 

With the consideration of the following points, follow the exact steps as the configuration the worker node 1.

• Decide on a port Offset value and update the port Offset element in WORKER_HOME/repository/conf/carbon.xml file. If you are using separate hosts ignore this.

• Change the localMemberPort value of WORKER_HOME/repository/conf/axis2/axis2.xml to a port that is not used already.

Start the ESB worker node. The workerNode system property must be set to true when starting the workers in a cluster. Therefore worker should be started as follows,

./bin/wso2server.bat -DworkerNode=true

How to test the setup..

• Start the servers as described above. 

• Go to the Management Console https://localhost:9444/carbon

• Add a sample proxy and save it. Add a log mediator to the inSequence so that logs will be displayed in the workers terminals

• Observe cluster messages(through the terminal or log) which is send by the manager and received by the worker. Worker will then synchronize with the SVN and deploy the proxy

• Send a request to the end point through the load balancer. Load Balancer should point to the active nodes end point. For example, for external clients, the EP of the proxy should be as  

http://{Load_Balancer_Mapped_URL_for_worker}/services/{Sample_Proxy_Name}

• In Active workers logs you will see the proxy being invoked (You need to add a log mediator as in number 3 in the proxy to see this)

Clustering WSO2 ESB - Part 1 - Distributed worker-manager setup with one node acting the dual role, as a manager as well as a worker

There are several ways of clustered deployment of WSO2 ESB (or any WSO2 server). You can find more about ESB clustering from here. The deployments introduced in that document is fronted by WSO2 ELB (Elastic Load Balancer). In this blog post I will explain the minimum configurations required to configure WSO2 ESB in a worker/manager distributed setup without having a WSO2 ELB at front so that any load balancer can be used. This setup can be also used as a Active/Passive setup where the active-passiveness can be decided by the load balancer based on a predefined mechanism(echo test or a heartbeat mechanism). 

In this setup one ESB node will be playing a dual role as a worker as well as a manager. The  overall  setup  will contain  two  ESB  servers  (nodes), in which  one  server (node 1) will act as  both  Manager  and  Worker while the other server(node 2) will act as only a worker. Node 1 will be used for deploying artifacts (CAR files and proxies) and managing the cluster.

The  synchronization  of artifacts between the servers will be achieved through the SVN based Deployment Synchronizer.

The overall view of the cluster will be as in the following figure.

Configuring the Manager/Worker Node (node 1)

Download and extract the WSO2 ESB distribution (referred as MANAGER_HOME ).

axis2.xml configuration

Clustering   should  be  enabled  at  axis2  level  in  order  for  management  node  to  communicate with  the   worker   nodes.  
Open  MANAGER_HOME/repository/conf/axis2/axis2.xml  and  update the clustering configuration as follows:

<clustering class="org.apache.axis2.clustering.tribes.TribesClusteringAgent"  enable="true">
<parameter name="membershipScheme">wka</parameter>

Specify the cluster domain as follows;

<parameter name="domain">wso2.esb.domain</parameter>

Uncomment localmemberhost element in axis2.xml and specify the IP address (or host name) to be exposed to members of the cluster. This address can be an internal address used only for clustering.

<parameter name="localMemberHost">127.0.0.1</parameter>

Define the port through which other nodes will contact this member. If several instances are running in the same host make sure to use ports which will not conflict.

<parameter name="localMemberPort">4001</parameter>

Comment out the static or well­-known members, because this will be the well­-known member. (i.e do not need to have a list of members)

carbon.xml configuration

If multiple WSO2 Carbon ­based products are running in same host, to avoid possible port conflicts, the port offset of MANAGER_HOME/repository/conf/carbon.xml should be changed as follows. Ignore this if you are using three separate hosts.

<Offset>1</Offset>

Configure the Deployment Synchronizer as follows. Make sure both AutoCommit and
AutoCheckout are ‘true’

<DeploymentSynchronizer>
     <Enabled>true</Enabled>
     <AutoCommit>true</AutoCommit>
     <AutoCheckout>true</AutoCheckout>
     <RepositoryType>svn</RepositoryType>
     <SvnUrl>http://svnrepo.example/repos/esb</SvnUrl>
     <SvnUser>USERNAME</SvnUser>
     <SvnPassword>PASSWORD</SvnPassword>
     <SvnUrlAppendTenantId>true</SvnUrlAppendTenantId>
</DeploymentSynchronizer>

Start the ESB manager node(./bin/wso2server.sh ).

You can access the management console by: https://localhost:9444/carbon/ within the managers node. (Use the correct port according to the port offset which was defined earlier) 

Add a sample proxy though the console and check whether it is getting committed to the SVN correctly. Proxy will be committed to SvnUrl/-1234/synapse-configs/default/proxy-services/ if you created the proxy as the admin.

Configuring the worker node (Node 2)

Download and extract the WSO2 ESB distribution(referred as WORKER_HOME).

axis2.xml configuration

Clustering   should  be  enabled  at  axis2  level  in  order  for  management  node  to  communicate with  the  worker  nodes. Open  WORKER_HOME/repository/conf/axis2/axis2.xml and update the clustering configuration as follows:

<clustering class="org.apache.axis2.clustering.tribes.TribesClusteringAgent" enable="true">
<parameter name="membershipScheme">wka</parameter>

Specify the cluster domain as follows;

<parameter name="domain">wso2.esb.domain</parameter>

Define  the  port  through  which  other  nodes  will  contact  this  member.  If several instances are running in the same host make sure to use ports which will not conflict.

<parameter name="localMemberPort">4002</parameter>

Add managers(Node 1) IP address (or host name) and the port as the well known member.

• hostName: defined as localMemberHost in managers axis2.xml

• port: defined as localMemberPort in managers axis2.xml.

Make  sure  this  address  can  be  accessed  through  the  workers  host. If  you  use  host name, map it to managers IP in the ‘etc/hosts’ file of the workers machine.

<members>
   <member>
   <hostName>127.0.0.1</hostName>
      <port>4001</port>
   </member>
</members> 

carbon.xml configuration

If multiple WSO2 Carbon­ based products are running in same host, to avoid possible port conflicts, the port offset of WORKER_HOME/repository/conf/carbon.xml should be changed as follows. Ignore this if you are using three separate hosts.

<Offset>2</Offset>

Configure the Deployment Synchronizer as follows. Make sure AutoCommit is set to ‘false’ and AutoCheckout to ‘true’

<DeploymentSynchronizer>
     <Enabled>true</Enabled>
     <AutoCommit>false</AutoCommit>
     <AutoCheckout>true</AutoCheckout>
     <RepositoryType>svn</RepositoryType>
     <SvnUrl>http://svnrepo.example/repos/esb</SvnUrl>
     <SvnUser>USERNAME</SvnUser>
     <SvnPassword>PASSWORD</SvnPassword>
     <SvnUrlAppendTenantId>true</SvnUrlAppendTenantId>
</DeploymentSynchronizer> 

Start the ESB worker node. The workerNode system property must be set to true when starting the workers. 
Therefore worker should be started as follows (in linux),

./bin/wso2server.sh -DworkerNode=true 

How to test the setup..

• Start the servers as described above. 

• Go to the Management Console https://localhost:9444/carbon

• Add a sample proxy and save it. Add a log mediator to the inSequence so that logs will be displayed in the workers terminals

• Observe cluster messages(through the terminal or log) which is send by the manager and received by the worker. Worker will then synchronize with the SVN and deploy the proxy

• Send a request to the end point through the load balancer. Load Balancer should point to the active nodes end point. For example, for external clients, the EP of the proxy should be as  

http://{Load_Balancer_Mapped_URL_for_worker}/services/{Sample_Proxy_Name}

• In Active nodes logs you will see the proxy being invoked (You need to add a log mediator as in number 3 in the proxy to see this)