Slow performance with CXF and Jetty

I am, quite admittedly, far from an expert with CXF. When I was tasked with upgrading an old CXF 2.x / Jetty 8.x service to CXF 3/Jetty 9, I hoped it would be a fairly simply task of modifying a pom and perhaps a few configuration points and being done. Unfortunately, I found myself needing to upgrade quite a few pieces, dependencies and customized connectors. But even that wasn’t the biggest problem.

Once all the dependencies, security and customized pieces were completed and all the tests were passed, we were ready to place the service into a non-production environment. This environment is far from isolated, as it receives on average 150+ TPS for a single instance. But, all the tests were completed so no problem, right? Unfortunately, from the moment it was started, there were issues.

CPU usage skyrocketed, calls that took milliseconds now took (literally) minutes and the log was filled with Broken Pipes. Thread dumps showed a plethora of stacktraces leading back to XMLStreamWriterImpl and java.lang.String.toCharArray. It seems something was getting stuck outputting the XML responses, but all of the elements in the stack trace were third party – none of it really had anything to do with our application code.

Searches on StackOverflow and other places turned up nothing. At first, we couldn’t replicate in a standalone environment until finally – we could. Turns out that after checking nearly everything, the classpath was the issue. After more searching, we had transitive dependencies pulling in the jaxp-ri library, which appears to have been taking precedence over what did actually work properly, which was StAX. So if you happen to ever find yourself in a similar situation, make sure to check those transitive dependencies for rogue elements!

As a side note, I’m still not 100% clear on exactly why this performed so abysmally with the jaxp-ri library. If anyone happens to have any insight into that, as my time for such endeavors is often lacking, feel free to drop in a comment or two.