The Importance of Adhering to Application Boundaries in Enterprise Applications

Introduction

In software engineering, multi-tier architecture refers to an architecture that divides components of the application into logical units which serve a specific purpose (Multitier Architecture). Consider an application which is intended to administer a larger system via several RESTful APIs.  In such an application we have three distinct layers that perform a specific function: 1) the data access layer, 2) the business logic or service layer and 3) the presentation layer. Using this paradigm makes the application easier to understand and maintain. When we stray from this architecture and components bleed from one unit to another, unwanted circumstances may arise and you run the risk of a costly refactor down the line.

The Problem

I encountered such a situation when I was tasked with finding a reason why after a period of time, an application would stop processing requests  then start responding after about 5 minutes.

The application was a multi-tier RESTful web service that uses a very typical tool suite, namely Spring, Hibernate, Jersey and Tomcat; the application was built using Maven. The application used the C3P0 connection pool.  The XML payloads were representations of DTOs and some fields were simply URI references to other DTOs instead of a full-blown serialized objects.

We were using Grizzly - Jersey's sample test container which starts Tomcat before each test method, runs and completes the test, and then stops Tomcat. If there are a large number of integration tests, this approach will take too long. Grizzly is not capable of starting before the test case, so we chose to abandon it and go with the CARGO approach of starting Tomcat before the test suite, then shut it down after the test suite has completed.

As a result, we had uncovered the situation described above. We suspected the problem was with the persistence layer.

Approach

Classpath

We decided to start investigating this issue by first looking at the classpath. We had just moved from ANT to Maven as our build tool, and we had not fully tuned our pom. So starting there was a good approach. We did find classpath conflicts and fixed them, but that did not resolve the issue. The next step was to investigate the connection pool.

C3P0 Connection Pool

As noted above, we were using C3P0 which comes with a handy JMX extension to monitor its health. We connected to the JMX server using "jconsole" which comes with your JDK. As we would run our tests, we would observe the the number of busy connections rose very quickly until it reached its pool limit. In normal circumstances, one connection would be busy and then it would be quickly returned to the connection pool after a transaction had been completed.

We had tried using several properties to force C3P0 to return the connection to the connection pool after the transaction had been committed, but those yielded the same result. The next option was to review the transactional boundaries.

@Transactional

As noted above, the application used Hibernate and Spring to manage the persistence layer. The @Transactional annotation was used to demarcate the transactional boundaries. We found a couple of persistence calls that were not wrapped in a transaction, but the problem still existed.

Investigate Datasource

Our next step was to configure the datasource in such a way that it would throw an exception when a database connection had been orphaned and was then evicted from the connection pool. We decided to use the Tomcat JDBC Connection Pool and use the following settings configured through Spring:

    <bean id="dataSource" class="org.apache.tomcat.jdbc.pool.DataSource" destroy-method="close">
        <property name="dataSource" value="javax.sql.DataSource"/>
        <property name="driverClassName" value="${connection.db.driver.class}"/>
        <property name="url" value="${connection.db.connection.url}"/>
        <property name="username" value="${connection.db.username}"/>
        <property name="password" value="${connection.db.password}"/>
        <property name="maxIdle" value="10"/>
        <property name="maxActive" value="100"/>
        <property name="maxWait" value="1000"/>
        <property name="removeAbandoned" value="true"/>
        <property name="removeAbandonedTimeout" value="300"/>
        <property name="logAbandoned" value="true"/>
        <property name="testWhileIdle" value="true"/>
        <property name="testOnBorrow" value="true"/>
        <property name="testOnReturn" value="true"/>
        <property name="validationQuery" value="select 1"/>
        <property name="testWhileIdle" value="true"/>
        <property name="timeBetweenEvictionRunsMillis" value="1200000"/>
        <property name="minEvictableIdleTimeMillis" value="1800000"/>
        <property name="numTestsPerEvictionRun" value="5"/>
        <property name="defaultAutoCommit" value="false"/>

        <property name="jmxEnabled" value="true"/>
    </bean>

When we ran our tests we quickly found the culprit. An exception was thrown in an JAXB adapter that as part of its deserialization process, was trying to query the database by grabbing a static reference to a Hibernate Session from the persistence layer.

What Went Wrong?

This Post from StackOverflow explains the difference between obtaining a hibernate session using "openSession" and "getCurrentSession." Using the former, you are responsible for closing it after using it. Using the latter retrieves the  session from the current context (in this case "thread'), and once the operation has been completed, is returned to context by Hibernate.

The persistence layer lives in a separate thread from the thread that serializes and deserializes the XML payloads. As a result, a session was created in one thread, handed off to another thread, and then abandoned. The connection associated with this session was also abandoned. This resulted in exhausting the connection pool.

Additionally, the DAO layer was hacked to allow a static instance of the DAO, and hence an instance of the Hibernate Session to escape the persistence layer and be used directly by other layers in the application!

The Fix

Any and all references to the Hibernate Session in the JAXB layer - e.g. the JAXB adapter - were removed and the presentation layer was decoupled from the data access layer. The object URI references in the XML payloads were now managed down another layer, into our service objects where our DTOs were hydrated. Using this approach resulted in sessions no longer being abandoned and the connection pool used only one connection throughout the entire test suite!

Conclusions

The main lesson learned here is to stick to your architectural standards; don't use your objects in layers where they absolutely don't belong. And if you do need access to those layers, use APIs in the proper layers to get what you need.

Most importantly, If you have to hack Spring - or any tool - to get what you need, then you need to step back and take a deeper look at to what you are doing. If you have to hack it, you're doing something wrong! I've learned that lesson the hard way, and it pains me to see other people make the same mistake.

In my next blog, I talk about implementing CARGO in using Spring Security and Basic Authorization for use in long running integration tests. I'll cover how to configure CARGO using both ANT and Maven.