Wednesday, October 1, 2014

Debugging java.lang.OutOfMemoryError: unable to create new native thread on Cloud Foundry

Within a multi-threaded Java application running on Cloud Foundry we started seeing the following error during our performance tests:
[App/0]  ERR java.lang.OutOfMemoryError: unable to create new native thread
Doing some basic troubleshooting the testers observed the following:
  • Increasing the memory allocated to the application did not resolve the error.
  • Running the same Java application on a Linux VM we did not see the error.
So the question arises what is different between a Cloud Foundry Linux Container and a Linux VM that precipitates these errors?

Research

This article on plumber.eu does a great job of describing the unable to create new native thread error. In short the error occurs when we exhaust the available user processes and then continue trying to create processes from within Java.
The article also describes how to resolve the error on a Linux VM, by increasing the number of user processes using ulimit -u. This is a scale-up solution and Cloud Foundry is a scale-out environment thus the higher user process limits are not made available within containers rendering this solution a non-starter.

Investigation

For us the solution needed to be scale-out run multiple Linux Containers each with fewer threads. To implement this solution effectively we needed to know the following:
  • How many user processes were available within each Linux Container.
  • How many of those processes are available for us to use as Java threads.
Additionally we wanted to understand why our testers had not seen the error on the Linux VM.

Determining the number of user processes

On a Linux VM to determine the number of user processes available to a user we simply run ulimit -a and look at the max user processes value. However on Cloud Foundry developers do not have an interactive Shell into the Linux Container.
James Bayer came up with a neat hack to work around this using websocketd so I used his hack to create an interactive shell into the Linux Container.

Using the interactive Shell we can easily observe the max user processes available to us is 512:
$ whoami 
vcap

$ ulimit -a
core file size (blocks, -c) 0 
 data seg size (kbytes, -d) unlimited 
 scheduling priority (-e) 0 
 file size (blocks, -f) unlimited 
 pending signals (-i) 1024 
 max locked memory (kbytes, -l) 64 
 max memory size (kbytes, -m) unlimited 
 open files (-n) 16384 
 pipe size (512 bytes, -p) 8 
 POSIX message queues (bytes, -q) 819200 
 real-time priority (-r) 0 
 stack size (kbytes, -s) 8192 
 cpu time (seconds, -t) unlimited 
 max user processes (-u) 512 
 virtual memory (kbytes, -v) unlimited 
 file locks (-x) unlimited  

Determining how many of user processes are available for Java threads

At a guess from the max user processes processes value we should be able to run about 500 threads within each Linux container (this will vary based on your Stack, we are currently using Ubuntu Lucid).
To test this I threw together a basic Java application to exhaust the available threads. Running the application in Cloud Foundry I see the following:
tastle@TASTLE-M-W03P tgobbler (master) $ cf logs tgobbler --recent
Connected, dumping recent logs for app tgobbler in org COL-DEV-FT / space Development as taste...
[DEA]     OUT Starting app instance (index 0) with guid 34c88014-fdf0-4f25-b555-b58291d6f879
[App/0]   OUT Threads started: 1
...
[App/0]   OUT Threads started: 490
[App/0]   ERR Exception in thread "main" java.lang.OutOfMemoryError: unable to create new native thread
...
[DEA]     OUT Starting app instance (index 0) with guid 34c88014-fdf0-4f25-b555-b58291d6f879
[App/0]   OUT Threads started: 1
...
[App/0]   OUT Threads started: 490
[App/0]   ERR Exception in thread "main" java.lang.OutOfMemoryError: unable to create new native thread
[App/0]   ERR  at java.lang.Thread.start0(Native Method)
[App/0]   ERR  at java.lang.Thread.start(Thread.java:714)
[App/0]   ERR  at org.trastle.App.main(App.java:12)
So we can see the following:

  1. The application starts a first time, then consumes 490 threads, then dies.
  2. Cloud Foundry notices, then restarts the failing application.
  3. The application starts a second time, then consumes 490 threads, then dies.
After a few more runs it became clear that 490 threads was a consistent max.

Why do we not see this error on an Ubuntu Linux VM?

The Stack we use in Cloud Foundry is Ubuntu Linux so testing this on a Debian VM we should be able to reproduce the failure. However our testers had already observed this was not the case.
So lets look at the max user processes available on the VM:
tastle@tastle-debian:~$ ulimit -a
 core file size          (blocks, -c) 0
 data seg size           (kbytes, -d) unlimited
 scheduling priority             (-e) 0
 file size               (blocks, -f) unlimited
 pending signals                 (-i) 3824
 max locked memory       (kbytes, -l) 64
 max memory size         (kbytes, -m) unlimited
 open files                      (-n) 1024
 pipe size            (512 bytes, -p) 8
 POSIX message queues     (bytes, -q) 819200
 real-time priority              (-r) 0
 stack size              (kbytes, -s) 8192
 cpu time               (seconds, -t) unlimited
 max user processes              (-u) 3824
 virtual memory          (kbytes, -v) unlimited
 file locks                      (-x) unlimited
There are roughly 7x the user processes available to us on on the VM explaining why we could not see the issue. This is largely because the VM is tuned for a lower level of multi-tenancy the the Cloud Foundry Linux Container. In order to reproduce the error on the VM we can reduce the max user processes
tastle@tastle-debian:~$ ulimit -u 512
tastle@tastle-debian:~$ ulimit -a
 core file size          (blocks, -c) 0
 data seg size           (kbytes, -d) unlimited
 scheduling priority             (-e) 0
 file size               (blocks, -f) unlimited
 pending signals                 (-i) 3824
 max locked memory       (kbytes, -l) 64
 max memory size         (kbytes, -m) unlimited
 open files                      (-n) 1024
 pipe size            (512 bytes, -p) 8
 POSIX message queues     (bytes, -q) 819200
 real-time priority              (-r) 0
 stack size              (kbytes, -s) 8192
 cpu time               (seconds, -t) unlimited
 max user processes              (-u) 512
 virtual memory          (kbytes, -v) unlimited
 file locks                      (-x) unlimited
Running the same test application now we see the following:
tastle@tastle-debian:~$ java -jar tgobbler-1.0.jar
...
Threads started: 500
Exception in thread "main" java.lang.OutOfMemoryError: unable to create new native thread
The result largely similar to what we saw in the Cloud Foundry Linux Container. The difference exists because the user processes available represent everything running as that user not just the Java application.

Resolution

We now know we will always be able to create roughly 490 Java threads within a single Cloud Foundry Linux Container.
It makes sense to leave some head room in each container so we will aim to utilise a maximum of 90% of the available threads. This leaves us with a target maximum of 450 threads per Linux Container. When more threads are required we will scale out to multiple containers.

Update: Changes to the Java Buildpack

After discussing this issue, the Cloud Foundry team have very kindly integrated a call to ulimit -a and df -h into the kill script used in the Java Buildpack. The output of this script will also be surfaced over Loggregator making the information readily available.



Extending Cloud Foundry with Open Source Toys

A talk that I gave with my colleague Matt Johnson describing how we have extended Cloud Foundry using Open Source tools including: Jenkins, Sensu, CollectD and Graphite.

Presented at the London Cloud Foundry User Group September 2014 meetup.


Thursday, September 25, 2014

Taking blobs from BOSH releases

When developing BOSH releases we quite often take packages from existing BOSH releases to save time. Finding the binary blobs you need for the package (from the .blobs directory) is a pain, the files in this directory are all UUIDs making them difficult to work with.

To resolve this issue I wrote a small function for my .bashrc to copy the blobs to a new directory and reverse the UUID naming process that takes place when you upload the blobs using BOSH.

The usage pattern for the function is as follows:

you@yourpc $ git clone git@github.com:FreightTrain/sensu-client-boshrelease.git
you@yourpc $ cd sensu-client-boshrelease
you@yourpc $ bosh create release
...
you@yourpc $ bosh-blobs-realize
...

I hope someone finds this useful.

Troy

Sunday, June 15, 2014

Cloud Foundry Summit 2014 Lighting Talk

At this year's CF Summit, I was given the opportunity to speak with my colleague Matt Johnson about using Cloud Foundry and BOSH at Cisco. The talk covers some of the pain points we had adopting Cloud Foundry, how things have improved and the open source automation pipeline we built to reliably deploy and test our Cloud Foundry installations.

The slides from the talk are up on SlideShare if you want to take a look.

Cloud Foundry Summit 2014

Tuesday, March 4, 2014

Setting the Disk Quota for your Cloud Foundry apps

Setting the disk quota using a manifest

It is possible to set the disk quota for an application using the application's `manifest.yml`. The required property is disk_quota:
---
applications:
- name: testapp-large-disk
  memory: 512M
  instances: 1
  disk_quota: 3072
  path: .

Setting the disk quota manually

There is currently no CLI command to set the disk quota for an app, however it is supported by the Cloud Foundry REST API. This means you can set the disk quota using the `cf curl` command.

Example commands

$ cf curl /v2/apps

# Find the metadata.guid for the app you want to increase the disk quota of

$ cf curl /v2/apps/[APP-GUID-HERE] -X PUT -d '{"disk_quota":2048}'
Note: You will need to set the `disk_quota` to the number of megabytes of disk quota you require for your application. This must be less than the maximum allowed disk quota or the command will fail.