jasonb on Wednesday, February 17, 2010

A Better Tomcat for Ubuntu and Debian

28

As mentioned in my last post, I have recently spent some time improving the Tomcat package on the Ubuntu and Debian Linux distributions. This post goes into more detail on those changes.

For quite some time I have been studying the Tomcat startup and shutdown procedures, and trying to improve the reliability, security, and user experience on Linux. I noticed that the Ubuntu and Debian init scripts were starting Tomcat via the JSVC service runner, which is known to shut down Tomcat abruptly. JSVC also implements unreliable restarts, such that it is possible for the init script to be unable to restart Tomcat. Instead, the init script should start Tomcat by running Tomcat’s catalina.sh script, and that script should in turn run the Java binary, so that the init script can properly handle any issues with starts, stops, or restarts.

However, JSVC is also used to allow binding the Tomcat JVM to privileged server port numbers (port numbers lower than 1024), such as the standard HTTP and HTTPS ports 80 and 443. Without JSVC, the Ubuntu Tomcat 6 init script would need to use some other mechanism to allow Tomcat to bind to privileged ports. As a replacement for binding via JSVC, I proposed using either Linux Capabilities (the setcap command), or the authbind package.

It was difficult to decide whether to use Linux Capabilities or to allow the Tomcat JVM to bind to privileged ports. The most elegant way appeared to be to use Linux Capabilities, because that’s completely out of the execution path of the Tomcat JVM, and it’s built into Linux. But after trying it, I found that the Hotspot JVM currently does not support Linux Capabilities — I tried it on several different Linux distributions and found that Hotspot just doesn’t support it. I filed JDK bug 6919633 about that, which has been accepted by Sun as an original JDK bug report. This means we can’t use setcap with Sun Java, for now. Next, I turned to authbind. Authbind is written specifically to allow a process that is not running as root to bind to one or more privileged ports. It turned out to work well, and that’s now what the Debian Tomcat 6 package uses.

After the above research and testing, and now that I am a Debian committer, here are the changes that I just finished implementing, or helped to implement:

  • JSVC is no longer used by the package. Instead, Tomcat is invoked via the stock catalina.sh script. Any script code in the init script that was used to generate the Java startup arguments has been removed because catalina.sh performs that work.
  • Authbind is now the standard method for binding Tomcat to ports lower than 1024. There is a new option (disabled by default) to enable authbind in /etc/default/tomcat6. Enabling authbind means that Tomcat will be allowed to bind to privileged ports, if Tomcat or its webapps are configured to do so, and if authbind is configured to allow it. Enabling authbind also presumes the use of IPv4, since authbind only works with IPv4. The permission for Tomcat to use privileged port numbers is now configured at Tomcat package install time. Tomcat itself can be run entirely as an unprivileged user for this reason, tightening the scope of escalated privileges, improving server security.
  • The security manager now defaults to the disabled state. It is commented that way in /etc/default/tomcat6. Having it disabled by default (but still supported in the code) means that fewer Tomcat package users will have difficulty getting Tomcat working in the default configuration.
  • Reliable restarts are now implemented in the init script. A stop command sends a SIGTERM to the Tomcat JVM, running Tomcat’s shutdown hook. The init script monitors the process after sending this signal and will exit right away when Tomcat’s JVM exits. If Tomcat’s JVM does not exit, the init script waits for 20 seconds, monitoring the JVM process. At the end of the 20 seconds, the init script sends a SIGKILL to the JVM and waits up to 5 additional seconds for the JVM to quit. Of course, you can easily change these timeouts to suit your use of Apache Tomcat by modifying the number in the init script. If the JVM process will not quit (kernel I/O hang problem, or similar), the init script prints a failure message saying that Tomcat will not shut down. Tomcat can now be reliably and gracefully restarted via the init script. Graceful shutdowns mean that no webapp data is lost due to the Tomcat JVM being forcefully killed. Reliable restarts means that it is sufficient to script a “service tomcat6 restart” command, and you can be confident that the currently running Tomcat will be shut down gracefully and that a new Tomcat process will be started only after the first one is gone. Implementing graceful stops means that webapps have the appropriate amount of time to persist their application data to disk before the Tomcat JVM exits.
  • Tomcat now sends STDOUT and STDERR to its usual stock log file, $CATALINA_BASE/logs/catalina.out (for example, /var/log/tomcat6/catalina.out in this package) alongside all of the other Tomcat log files, instead of logging STDOUT and STDERR to syslog.
  • The Debian Tomcat 6 package is now up to date with the latest stable release of Tomcat, 6.0.24. This fixes a number of important issues, as I described in a recent blog entry .

These improvements have been implemented in time for the Ubuntu Lucid Lynx (10.04) and Debian Squeeze (6.0) releases.

MuleSoft’s contributions to Tomcat have significantly improved the reliability and usability of both the Ubuntu and Debian Tomcat 6 packages, and we hope to do the same for Tomcat 7, which is expected to be released around summer 2010. We’re happy to have contributed these open source changes!

No related posts.


28 Responses to “A Better Tomcat for Ubuntu and Debian”

Owen Jacobson February 17th, 2010, 8:53 am

But does it include the rebadged tomcat distribution of DBCP yet? I realize it’s a demented practice, but the setup for container-managed DB connections in the Tomcat docs doesn’t work as-is under Ubuntu’s packages, because someone decided to strip out those classes.

Jason Brittain February 17th, 2010, 10:30 am

@Owen: I’m not familiar with that problem, but that’s probably because I did not see it when I reviewed the reported bugs: http://bugs.debian.org/cgi-bin/pkgreport.cgi?package=tomcat6 Thanks for mentioning it because it raises awareness, but in general if you’d like us to investigate and fix a problem, please file a bug against the tomcat6 package here: http://www.debian.org/Bugs/Reporting

Julien Dubois February 17th, 2010, 4:22 pm

I’m using iptables for binding my app server to port 80 (OK, I use Jetty and not Tomcat, but that’s the same issue in the end). This works great on Ubuntu (at least on Amazon EC2, using the standard 32-bit ubuntu 9.10 image). Here is my command :
sudo /sbin/iptables -t nat -I PREROUTING -p tcp –dport 80 -j REDIRECT –to-port 8080
Why didn’t you talk about this option? Is there some issue with this method?

jasonb February 17th, 2010, 4:59 pm

@Julien: I have also used iptables for years to remap port 80 to port 8080. In fact, I documented how to do that in the book Tomcat: The Definitive Guide, 2nd Edition (see Chapter 2: Configuring Tomcat, page 44). After using this method in production for quite some time, I’ve settled on the following two-rule iptables remapping:

# iptables -t nat -I PREROUTING -p tcp –dst 192.168.1.100 –dport 80 -j DNAT –to 192.168.1.100:8080
# iptables -t nat -I OUTPUT -p tcp –dst 192.168.1.100 –dport 80 -j DNAT –to 192.168.1.100:8080

The first rule is for remapping connections originating outside the machine, and the second rule is for remapping connections originating inside the machine.

Now, there are some drawbacks to this approach. If it works fine for you, then it is probably what you should use! But, since iptables is really the built-in Linux software firewall, many Tomcat users and sysadmins do not want to touch it for security reasons. Also, iptables isn’t necessarily the easiest thing to use, overall, and the rule syntax is rather cryptic, which is another set of reasons why people reject this solution. Then another reason why it is easily rejected is because it is difficult to do a good/intuitive job of combining the Tomcat init script with the setting of these rules at runtime, and setting the rules before Tomcat runtime (for example at Tomcat package installation time) makes a different set of people object. One thing I like about this solution was that it is completely out of the execution path of Tomcat. That fact is something that others actually find inelegant about it because it is working and they cannot find it! To them it is like the port is magically remapped — they have no idea what iptables is and so they wouldn’t know to look at the iptables rules — and they find that frustrating. Another thing I liked about the iptables solution is that it appears not to affect HTTP performance. But, others are convinced (without performing any of their own benchmarks first) that it adversely affects performance, so they reject it as a solution for that reason. IMO, none of these reasons not to use it have merit, except perhaps for anyone who doesn’t know how to use iptables, and even then they could learn how. :) So, if you like it, go ahead and use it — it works great. But, I’m convinced more people will find the authbind solution more acceptable.

Owen Jacobson February 18th, 2010, 9:21 pm

@Jason Wow, I’m surprised there’s no bug for this. I’ll see about putting together a proper repro case for it and reporting it. I just sort of assumed — my bad. :)

Owen Jacobson February 18th, 2010, 9:21 pm

Incidentally — this is awesome.

Alex February 19th, 2010, 2:25 am

Nice work: thanks to you guys for your work

Amir Laher February 19th, 2010, 2:33 am

Jason, this is fantastic news.
If only I could use Ubuntu at work. It’s already so much easier to set up Sun Java on Ubuntu than it is on CentOS.
Incidentally we’re using IPTABLES for port-binding – it works really well for us. I also wrote a custom init script for switching user and calling catalina.sh.
The only part I’m unhappy about is the occasional obstinate servlet which won’t let tomcat shut down without a kill -9.

I would love to plagiarise your init script – particularly the graceful kill part – for use on our custom tc installation on CentOS.
What would be your feelings on this? If positive, please could you point me in the direction of your source code.
Thanks

Javi February 22nd, 2010, 2:58 pm

Hi,

If jsvc is not working somebody should change the instructions which are used to install tomcat if people doesn’t like tomcat from package (I really always want to install from package):
http://tomcat.apache.org/tomcat-6.0-doc/setup.html

It seems like it is the recommended method, isn’t it ?

Thanks

Torstein Krause Johansen February 22nd, 2010, 11:26 pm

Hi there,

thank you so much for your hard work, this is highly appreciated! I know many sites running with the vanilla distribution of Tomcat on Debian and Ubuntu, mostly because the package has been very old (only tomcat 5.5 the stable pools) and because of the jsvc wrapper.

Thanks again,

-Torstein

Jason Brittain March 1st, 2010, 10:41 am

@Amir: You’re in luck. I have also written a fully-relocatable Tomcat RPM package, and it includes an init script that implements reliable restarts. You can find my RPM and source snapshots here: http://www.webdroid.org/archives/tomcat-package/ Those RPMs are built from an Ant build file I wrote that implements a highly customizable Tomcat RPM package builder.. the source for that I’m trying to decide where to host at the moment. If you’re interested, let me know.

Jason Brittain March 1st, 2010, 10:52 am

@Javi: Yes, I do think it would be a good idea to change that particular public Tomcat documentation page so that it doesn’t recommend that new Tomcat users start out using jsvc. There are probably people who have different opinions about jsvc, however. I encourage you to discuss that on the tomcat-user mailing list, and also here if you’d like.

Matt March 10th, 2010, 7:25 am

Jason, I have upgraded to the latest Tomcat6 packages and changed all of my startup scripts with the new one, to use the catalina.sh script to start and stop. I had to change 6 scripts since we are using multiple instances of tomcat but it wasn’t a biggie. Authbind is working great, only binding to port 443, BUT there isn’t great documentation on how to even get Authbind working, that was the hardest part for me. Anyways, I have one unresolved issue. Since we are not using jsvc anymore, as you stated there is no stdout and no stderr, so it’s all going to /logs/catalina.out. That’s fine but this is running in a production environment and I REALLY need it to go to SYSLOG. I tried a few things but it hasn’t worked yet. Any ideas on this?

Fabrizio April 10th, 2010, 10:24 am

Hi Jason,

just a quick question…. Do you have a new version script to be able to build an own custom Tomcat RPM package?
I got the script from tomcat-6.0.20-0-src.tar.gz.(from Fri, 03 Jul 2009) Inside I have found a version with a TODO.txt with many new features. Is a new version availabe to be downloaded?

Thanks

Best regards,
Fabrizio

Ludovic Claude April 29th, 2010, 4:06 pm

@Owen: I left this repackaging of DBCP in Tomcat, mostly because nobody complained about it in the package tomcat6 bug reports and it seems to work. If you have an issue with this, raise a bug, it should not be too difficult to switch back to the Tomcat version of DBCP. I guess that previous package maintainers chose Commons DBCP because Debian tries to reuse existing libraries as much as possible, as packaging is quite a costly operation.

Terry May 13th, 2010, 7:45 am

Hi, Looking at the out-of-the-box debian config for tomcat6 in squeeze – catalina.out never gets rotated, and tomcat crashes on the 2gig file limit. is this really better than the previous logrotate methods?

Its like we are going backwards with out of the box logging and tomcat.

Jason Brittain May 15th, 2010, 6:31 pm

@Terry: You’re right that if you’re going to be writing that much data to the logs you need to have something configured to rotate your logs. I think very few people would run into that problem because catalina.out doesn’t normally grow very fast — it would usually take on the order of a year or longer for the log to get that big. I don’t see any bug filed about this on the tomcat6 package:

http://bugs.debian.org/cgi-bin/pkgreport.cgi?package=tomcat6

Also, I think that for those who really do want their Tomcat log messages mixed in with their system logs, there should be an easy configuration way to send the log messages to syslog instead.. though I don’t think that’s the best solution for simply rotating Tomcat’s own logs — I like using logrotate for that.

veghead May 26th, 2010, 8:22 am

Could you post a link to the init script itself? Those of us running other Linux distros would be very interested in seeing your implementation…

bobmanc June 16th, 2010, 11:59 am

Thanks for this Jason. I just updated from karmic to lucid and everything seems to work the same except for one thing. when I try to upload a file using common fileupload I now see an error.

ERROR : Jun 16, 2010 14:53:16 [CommonsLogger.java:27] : Unable to parse request
org.apache.commons.fileupload.FileUploadException: Processing of multipart/form-data request failed. javax.servlet.context.tempdir/upload_2bd97087_129421bad1c__8000_00000000.tmp (No such file or directory)

the javax.servlet.context.tempdir points to /var/lib/tomcat6/work/Catalina/localhost/myapp which exists and to test I was able to create a new file there in my code.

This wan’t happening with the karmic build of tomcat. as far as I know everything else is the same. Any insight on this would be greatly appreciated.

bobmanc June 29th, 2010, 2:15 pm

SOLVED: Turns out the issue was with Struts2 feeding fileupload a bad directory. If tomcat was root it worked anyway. Not sure how it worked in Karmic.

Topher LaFata July 23rd, 2010, 3:29 pm

A note. Hopefully this makes it into some documentation somewhere.

If using authbind with tomcat6 and apr connectorsto run on ports below 1024 -Djava.net.preferIPv4Stack=true wont affect the apr stuff. Since authbind will not work with IPv6 it will seem like authbind is not working with tomcat even though you have it configured correctly.

You have to either compile apr without IPv6 support ( the apt-get libtcnative-1 has it enabled by default)

or disable IPv6 system wide by adding :

net.ipv6.conf.all.disable_ipv6=1

to /etc/sysctl.conf

This drove me crazy for a while.

Rodrigo March 2nd, 2011, 4:26 am

Hello Jason, I’m having trouble with my installation. I’m using the authbind and everything. But I cant override the Xxmx128m vm argument with my own. I tried modifying the /etc/init.d/tomcat6 script, the setenv.sh, and the defaults.template with my own Xmx750m but I got no result.

This is how my tomcat is running.

tomcat6 7788 4.1 21.7 759436 182340 ? Sl Mar01 0:30 /usr/lib/jvm/java-6-openjdk/bin/java -Djava.util.logging.config.file=/var/lib/tomcat6/conf/logging.properties -Djava.awt.headless=true -Xmx128m -XX:+UseConcMarkSweepGC -Djava.net.preferIPv4Stack=true -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.endorsed.dirs=/usr/share/tomcat6/endorsed -classpath /usr/share/tomcat6/bin/bootstrap.jar -Dcatalina.base=/var/lib/tomcat6 -Dcatalina.home=/usr/share/tomcat6 -Djava.io.tmpdir=/tmp/tomcat6-tmp org.apache.catalina.startup.Bootstrap start
root 19026 0.0 0.0 6020 664 pts/1 S+ 00:06 0:00 grep java

Thanks.

ps. When you think tomcat 7 debian can be ready ?

thanks for all your work.
Rodrigo

Casey Watson March 7th, 2011, 11:18 am

There is a lot of bad advice out there. What you mentioned in the article is a much better solution.

Editing /etc/default/tomcat6

AUTHBIND=yes

Corné March 9th, 2011, 2:36 am

Hi Jason,

First of all, thanks for improving the Tomcat 6 startup and shutdown. I’m glad JSVC is no longer used and shutdowns are safe and solid.

However, I’m wondering if and when there will be an update of the tomcat6 package for Ubuntu 10.4 Lucid Lynx (LTS)?
The problem is that 6.0.24 contains a critical bug causing a JVM deadlock (https://issues.apache.org/bugzilla/show_bug.cgi?id=48694). So it’s truly important to upgrade to 6.0.27 or higher (6.0.32 is already the latest version).

Another, more general question, that I maybe should ask elsewhere: what if a new tomcat6 Ubuntu package is not released into the repository (soon enough)? Is there a way to upgrade the apt installation with the Apache Tomcat binary release manually, or even better, can I create an tomcat6 apt package myself?

I hope to hear from you soon.

Regards,
Corné

Java Applications on Privileged Ports | Stefan Reuter January 10th, 2012, 4:12 am

[...] approach works with any Java application and is not limited to ApacheDS. Have a look at A Better Tomcat for Ubuntu and Debian by MuleSource to see how they are using authbind without Java Service Wrapper to make Tomcat run on [...]

Jan van der Meer June 19th, 2012, 4:35 am

How to use AUTHBIND in a Tomcat non service setup, where the file /etc/default/tomcat7 file just is not available? Many people just unzip the downloaded file.

Robert January 9th, 2014, 12:39 am

Hi,
Unfortunately both files HRData.xsd and HRDataService.wsdl are no longer available for download. Could you provide another location for them please?

Thank you

Leave a Comment