CARGO With TLS/SSL and Spring Security

In this post, I discuss how to configure Cargo to start Tomcat using TLS/SSL. To quote the CARGO project, "Cargo is a thin wrapper that allows you to manipulate Java EE containers in a standard way." Cargo can be used as part of an Integration Test strategy, where Cargo will manage starting the J2EE container, deploying an application, and stopping the server after the integration tests have completed.

In this environment we are using Tomcat 7.0.26, JDK 1.7.0_03, ANT 1.8.4 and Cargo 1.3.1.

Background

As I discussed in my previous post, we had been using Grizzly - Jersey's test container - to test our web services. As you may recall, Grizzly will start the container just before a test method, execute the test method, and then shut down the container after the test method has finished. If you have a large number of tests, this can become problematic for a few reasons. We found that starting and stopping the container between each test not only drastically increased the time it took to run our tests, but also masked a huge problem which caused our application to become non-responsive.  Additionally, Grizzly has a substantial footprint and to use it, you must increase the heap as well as the permGen space. In my opinion, Grizzly should NOT be used to run a large number of tests, rather it should be used to spot check web services.

It was obvious that we needed to adopt another approach to testing our web services - creating real world integration tests in addition to testing the individual web service - that would keep our container running throughout the duration of the tests. We decided to use CARGO to manage the test container.

Configuration

Our application can run in basic authorization using TLS/SSL leveraged by Spring Security. The problem was to configure CARGO such that it would leverage Spring Security to run in basic authorization using TLS/SSL. 

The following assumes that you are already familiar with two-way SSL, have already created your certificates/key pairs and have placed them in their respective truststore/keystores . If not, please refer to the following link: Tomcat SSL How-to. If you are not already familiar with the CARGO ANT tasks, you can refer to the following link CAROG-ANT Tasks for more information.  

Here is an example of how to configure CARGO/ANT to run web service tests in basic authorization using TLS/SSL:


    <target name="cargostart" depends="init">
        <delete dir="${tomcatconfig.dir}"/>
        <mkdir dir="${tomcatlog.dir}"/>
        <mkdir dir="${tomcatconfig.dir}"/>
        <echo message="Starting Cargo..."/>
        <!-- Property needs to be passed in as a system property -->
        <echo message="Using tomcat.home = ${tomcat.home} "/>
        <echo message="Using war = ${capf.war}"/>
        <cargo containerId="tomcat7x" home="${tomcat.home}" output="${tomcatlog.dir}/output.log"
               log="${tomcatlog.dir}/cargo.log" action="start">
            <configuration  home="${tomcatconfig.dir}" type="standalone">
                <property name="cargo.logging" value="medium"/>
                <!-- For installed container o.c.c.c.tomcat.Tomcat7xInstalledLocalContainer -->
                <property name="cargo.hostname" value="${test.hostname}"/>
                <property name="cargo.servlet.port" value="${test.hostname.port}"/>
                <property name="cargo.protocol" value="https"/>

                <property name="cargo.tomcat.connector.clientAuth" value="basic"/>
                <property name="cargo.tomcat.connector.keyAlias" value="${ssl.key.alias}"/>
                <property name="cargo.tomcat.connector.keystoreFile" value="${javax.net.ssl.keyStore}"/>
                <property name="cargo.tomcat.connector.keystorePass" value="${javax.net.ssl.keyStorePassword}"/>
                <property name="cargo.tomcat.connector.keystoreType" value="${javax.net.ssl.keyStoreType}"/>
                <property name="cargo.tomcat.connector.sslProtocol" value="https"/>
                <property name="cargo.tomcat.connector.truststoreFile" value="${javax.net.ssl.trustStore}"/>
                <property name="cargo.tomcat.connector.truststorePass" value="${javax.net.ssl.trustStorePassword}"/>
                <property name="cargo.tomcat.connector.truststoreType" value="${javax.net.ssl.trustStoreType}"/>
                <property name="cargo.tomcat.httpSecure" value="true"/>
                <property name="cargo.springSecurityFilterChain" value="org.springframework.web.filter.DelegatingFilterProxy"/>
                <deployable type="war" file="${myWar.war}" />
            </configuration>
        </cargo>


    </target>

    <target name="cargo-stop">
        <cargo containerId="tomcat7x" home="${tomcat.home}" output="${tomcatlog.dir}/output.log"
               log="${tomcatlog.dir}/cargo.log" action="stop"/>
            <configuration  home="${tomcatconfig.dir}" type="standalone">
                <property name="cargo.logging" value="medium"/>
...
    </target>


Of course, you'll need to supply all of the properties - i.e. "${javax.net.ssl.trustStore}" - with valid values.

Cargo will take a copy of the Tomcat container as defined by the "${tomcat.home}" property and place it into the "tomcatconfig.dir." One caveat is that a secure port must be configured in your "server.xml." The "server.xml" that comes with the Tomcat distribution will not suffice, it will need to be modified to configure a "connector" as shown below:

<connector address="${fully.qualified.hostname}" clientauth="want" connectiontimeout="20000" keyalias="${ssl.key.alias}" keystorefile="${javax.net.ssl.keyStore}" keystorepass="${javax.net.ssl.keyStorePassword}" keystoretype="${javax.net.ssl.keyStoreType}" maxsparethreads="75" maxthreads="150" minsparethreads="25" port="1443" protocol="HTTP/1.1" scheme="https" secure="true" sslenabled="true" sslprotocol="TLS" truststorefile="${javax.net.ssl.trustStore}" truststorepass="${javax.net.ssl.trustStorePassword}" truststoretype="${javax.net.ssl.trustStoreType}">
</connector>

Although I have been able to get this to work in an development environment, I have yet to configure Jenkins to allow integration tests to be run using TLS/SSL. At this point, the Cargo ping application cannot connect to the Tomcat container to signal that it is up and ready for tests. I suspect that it is a problem with the client's truststore and that I need to pass in the necessary "javax.net.ssl" properties when ANT/Jenkins starts up. I have posted this question on CARGO's user email list and should have some insight on how to do this soon. This will be the subject of upcoming posts.

In the interim, the integration tests now run in an development environment without interruption and the integration tests will provide some insight in to how the application will perform with some "typical" operations.

The Importance of Adhering to Application Boundaries in Enterprise Applications

Introduction

In software engineering, multi-tier architecture refers to an architecture that divides components of the application into logical units which serve a specific purpose (Multitier Architecture). Consider an application which is intended to administer a larger system via several RESTful APIs.  In such an application we have three distinct layers that perform a specific function: 1) the data access layer, 2) the business logic or service layer and 3) the presentation layer. Using this paradigm makes the application easier to understand and maintain. When we stray from this architecture and components bleed from one unit to another, unwanted circumstances may arise and you run the risk of a costly refactor down the line.

The Problem

I encountered such a situation when I was tasked with finding a reason why after a period of time, an application would stop processing requests  then start responding after about 5 minutes.

The application was a multi-tier RESTful web service that uses a very typical tool suite, namely Spring, Hibernate, Jersey and Tomcat; the application was built using Maven. The application used the C3P0 connection pool.  The XML payloads were representations of DTOs and some fields were simply URI references to other DTOs instead of a full-blown serialized objects.

We were using Grizzly - Jersey's sample test container which starts Tomcat before each test method, runs and completes the test, and then stops Tomcat. If there are a large number of integration tests, this approach will take too long. Grizzly is not capable of starting before the test case, so we chose to abandon it and go with the CARGO approach of starting Tomcat before the test suite, then shut it down after the test suite has completed.

As a result, we had uncovered the situation described above. We suspected the problem was with the persistence layer.

Approach

Classpath

We decided to start investigating this issue by first looking at the classpath. We had just moved from ANT to Maven as our build tool, and we had not fully tuned our pom. So starting there was a good approach. We did find classpath conflicts and fixed them, but that did not resolve the issue. The next step was to investigate the connection pool.

C3P0 Connection Pool

As noted above, we were using C3P0 which comes with a handy JMX extension to monitor its health. We connected to the JMX server using "jconsole" which comes with your JDK. As we would run our tests, we would observe the the number of busy connections rose very quickly until it reached its pool limit. In normal circumstances, one connection would be busy and then it would be quickly returned to the connection pool after a transaction had been completed.

We had tried using several properties to force C3P0 to return the connection to the connection pool after the transaction had been committed, but those yielded the same result. The next option was to review the transactional boundaries.

@Transactional

As noted above, the application used Hibernate and Spring to manage the persistence layer. The @Transactional annotation was used to demarcate the transactional boundaries. We found a couple of persistence calls that were not wrapped in a transaction, but the problem still existed.

Investigate Datasource

Our next step was to configure the datasource in such a way that it would throw an exception when a database connection had been orphaned and was then evicted from the connection pool. We decided to use the Tomcat JDBC Connection Pool and use the following settings configured through Spring:

    <bean id="dataSource" class="org.apache.tomcat.jdbc.pool.DataSource" destroy-method="close">
        <property name="dataSource" value="javax.sql.DataSource"/>
        <property name="driverClassName" value="${connection.db.driver.class}"/>
        <property name="url" value="${connection.db.connection.url}"/>
        <property name="username" value="${connection.db.username}"/>
        <property name="password" value="${connection.db.password}"/>
        <property name="maxIdle" value="10"/>
        <property name="maxActive" value="100"/>
        <property name="maxWait" value="1000"/>
        <property name="removeAbandoned" value="true"/>
        <property name="removeAbandonedTimeout" value="300"/>
        <property name="logAbandoned" value="true"/>
        <property name="testWhileIdle" value="true"/>
        <property name="testOnBorrow" value="true"/>
        <property name="testOnReturn" value="true"/>
        <property name="validationQuery" value="select 1"/>
        <property name="testWhileIdle" value="true"/>
        <property name="timeBetweenEvictionRunsMillis" value="1200000"/>
        <property name="minEvictableIdleTimeMillis" value="1800000"/>
        <property name="numTestsPerEvictionRun" value="5"/>
        <property name="defaultAutoCommit" value="false"/>

        <property name="jmxEnabled" value="true"/>
    </bean>

When we ran our tests we quickly found the culprit. An exception was thrown in an JAXB adapter that as part of its deserialization process, was trying to query the database by grabbing a static reference to a Hibernate Session from the persistence layer.

What Went Wrong?

This Post from StackOverflow explains the difference between obtaining a hibernate session using "openSession" and "getCurrentSession." Using the former, you are responsible for closing it after using it. Using the latter retrieves the  session from the current context (in this case "thread'), and once the operation has been completed, is returned to context by Hibernate.

The persistence layer lives in a separate thread from the thread that serializes and deserializes the XML payloads. As a result, a session was created in one thread, handed off to another thread, and then abandoned. The connection associated with this session was also abandoned. This resulted in exhausting the connection pool.

Additionally, the DAO layer was hacked to allow a static instance of the DAO, and hence an instance of the Hibernate Session to escape the persistence layer and be used directly by other layers in the application!

The Fix

Any and all references to the Hibernate Session in the JAXB layer - e.g. the JAXB adapter - were removed and the presentation layer was decoupled from the data access layer. The object URI references in the XML payloads were now managed down another layer, into our service objects where our DTOs were hydrated. Using this approach resulted in sessions no longer being abandoned and the connection pool used only one connection throughout the entire test suite!

Conclusions

The main lesson learned here is to stick to your architectural standards; don't use your objects in layers where they absolutely don't belong. And if you do need access to those layers, use APIs in the proper layers to get what you need.

Most importantly, If you have to hack Spring - or any tool - to get what you need, then you need to step back and take a deeper look at to what you are doing. If you have to hack it, you're doing something wrong! I've learned that lesson the hard way, and it pains me to see other people make the same mistake.

In my next blog, I talk about implementing CARGO in using Spring Security and Basic Authorization for use in long running integration tests. I'll cover how to configure CARGO using both ANT and Maven.