Within a multi-threaded Java application running on Cloud Foundry we started seeing the following error during our performance tests:
The article also describes how to resolve the error on a Linux VM, by increasing the number of user processes using ulimit -u. This is a scale-up solution and Cloud Foundry is a scale-out environment thus the higher user process limits are not made available within containers rendering this solution a non-starter.
James Bayer came up with a neat hack to work around this using websocketd so I used his hack to create an interactive shell into the Linux Container.
Using the interactive Shell we can easily observe the max user processes available to us is 512:
To test this I threw together a basic Java application to exhaust the available threads. Running the application in Cloud Foundry I see the following:
So lets look at the max user processes available on the VM:
It makes sense to leave some head room in each container so we will aim to utilise a maximum of 90% of the available threads. This leaves us with a target maximum of 450 threads per Linux Container. When more threads are required we will scale out to multiple containers.
[App/0] ERR java.lang.OutOfMemoryError: unable to create new native threadDoing some basic troubleshooting the testers observed the following:
- Increasing the memory allocated to the application did not resolve the error.
- Running the same Java application on a Linux VM we did not see the error.
Research
This article on plumber.eu does a great job of describing the unable to create new native thread error. In short the error occurs when we exhaust the available user processes and then continue trying to create processes from within Java.The article also describes how to resolve the error on a Linux VM, by increasing the number of user processes using ulimit -u. This is a scale-up solution and Cloud Foundry is a scale-out environment thus the higher user process limits are not made available within containers rendering this solution a non-starter.
Investigation
For us the solution needed to be scale-out run multiple Linux Containers each with fewer threads. To implement this solution effectively we needed to know the following:- How many user processes were available within each Linux Container.
- How many of those processes are available for us to use as Java threads.
Determining the number of user processes
On a Linux VM to determine the number of user processes available to a user we simply run ulimit -a and look at the max user processes value. However on Cloud Foundry developers do not have an interactive Shell into the Linux Container.James Bayer came up with a neat hack to work around this using websocketd so I used his hack to create an interactive shell into the Linux Container.
Using the interactive Shell we can easily observe the max user processes available to us is 512:
$ whoami vcap $ ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 1024 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 16384 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 512 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited
Determining how many of user processes are available for Java threads
At a guess from the max user processes processes value we should be able to run about 500 threads within each Linux container (this will vary based on your Stack, we are currently using Ubuntu Lucid).To test this I threw together a basic Java application to exhaust the available threads. Running the application in Cloud Foundry I see the following:
tastle@TASTLE-M-W03P tgobbler (master) $ cf logs tgobbler --recent Connected, dumping recent logs for app tgobbler in org COL-DEV-FT / space Development as taste... [DEA] OUT Starting app instance (index 0) with guid 34c88014-fdf0-4f25-b555-b58291d6f879 [App/0] OUT Threads started: 1 ... [App/0] OUT Threads started: 490 [App/0] ERR Exception in thread "main" java.lang.OutOfMemoryError: unable to create new native thread ... [DEA] OUT Starting app instance (index 0) with guid 34c88014-fdf0-4f25-b555-b58291d6f879 [App/0] OUT Threads started: 1 ... [App/0] OUT Threads started: 490 [App/0] ERR Exception in thread "main" java.lang.OutOfMemoryError: unable to create new native thread [App/0] ERR at java.lang.Thread.start0(Native Method) [App/0] ERR at java.lang.Thread.start(Thread.java:714) [App/0] ERR at org.trastle.App.main(App.java:12)So we can see the following:
- The application starts a first time, then consumes 490 threads, then dies.
- Cloud Foundry notices, then restarts the failing application.
- The application starts a second time, then consumes 490 threads, then dies.
Why do we not see this error on an Ubuntu Linux VM?
The Stack we use in Cloud Foundry is Ubuntu Linux so testing this on a Debian VM we should be able to reproduce the failure. However our testers had already observed this was not the case.So lets look at the max user processes available on the VM:
tastle@tastle-debian:~$ ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 3824 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 3824 virtual memory (kbytes, -v) unlimited file locks (-x) unlimitedThere are roughly 7x the user processes available to us on on the VM explaining why we could not see the issue. This is largely because the VM is tuned for a lower level of multi-tenancy the the Cloud Foundry Linux Container. In order to reproduce the error on the VM we can reduce the max user processes
tastle@tastle-debian:~$ ulimit -u 512 tastle@tastle-debian:~$ ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 3824 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 512 virtual memory (kbytes, -v) unlimited file locks (-x) unlimitedRunning the same test application now we see the following:
tastle@tastle-debian:~$ java -jar tgobbler-1.0.jar ... Threads started: 500 Exception in thread "main" java.lang.OutOfMemoryError: unable to create new native threadThe result largely similar to what we saw in the Cloud Foundry Linux Container. The difference exists because the user processes available represent everything running as that user not just the Java application.
Resolution
We now know we will always be able to create roughly 490 Java threads within a single Cloud Foundry Linux Container.It makes sense to leave some head room in each container so we will aim to utilise a maximum of 90% of the available threads. This leaves us with a target maximum of 450 threads per Linux Container. When more threads are required we will scale out to multiple containers.
