不是全部的OutofMemory exception都是内存问题。。。
前几天有个客户的site报了下面错误：
[ERROR][thread ] Could not start thread Timer-72025. Resource temporarily unavailable
Exception in thread "Timer-72024" java.lang.OutOfMemoryError: Resource temporarily unavailable in tsStartJavaThread
(lifecycle.c:1097).
Attempting to allocate 2G bytes
There is insufficient native memory for the Java
Runtime Environment to continue.
Possible reasons:
The system is out of physical RAM or swap space
In 32 bit mode, the process size limit was hit
Possible solutions:
Reduce memory load on the system
Increase physical memory or swap space
Check if swap backing store is full
Use 64 bit Java on a 64 bit OS
Decrease Java heap size (-Xmx/-Xms)
Decrease number of Java threads
Decrease Java thread stack sizes (-Xss)
Disable compressed references (-XXcompressedRefs=false)
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:640)
at java.util.Timer.<init>(Timer.java:137)
at java.util.Timer.<init>(Timer.java:106)
at ......
at java.util.TimerThread.mainLoop(Timer.java:512)
at java.util.TimerThread.run(Timer.java:462)
第一反应以为是内存不足，于是dump了heap回来看。发现系统仅仅占用了80M的内存，远远系统内存绰绰有余，那么为啥会出现这样的情况呢？查看了出错的代码。发现代码启动了非常多的线程，遂想起曾经遇到过类似的问题，weblogic里面启动的线程太多。超出了linux的最大限制，也是会出这个看起来内存泄漏的日志。查看系统的限制：
sc-1@vECE-222 limits.d # ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 38610 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 8192 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 10240 cpu time (seconds, -t) unlimited max user processes (-u) 1024 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited发现最大的进程数是1024，注意：linux里面计算线程和进程是用一样计算的，这里说的max user processes是包含线程的。
使用下面代码重现问题：
public class ThreadExample {
public static void main(String[] args){
System.out.println(Thread.currentThread().getName());
for(int i=0; i< 2048; i++){
new Thread("" + i){
public void run(){
System.out.println(Thread.currentThread().getName() + " running");
try {
Thread.sleep(60000);
} catch (InterruptedException e) {
e.printStackTrace();
}
System.out.println(Thread.currentThread().getName() + " ending");
}
}.start();
}
}
}
运行和输出：
sc-1@ECE30CP3_Z1 etmdaha # su -p occas -c "/opt/jrockit-jdk/bin/java ThreadExample"
Main Thread
0 running
5 running
//
836 running
[ERROR][thread ] Could not start thread 837. Resource temporarily unavailable
Exception in thread "Main Thread" java.lang.OutOfMemoryError: Resource temporarily unavailable in tsStartJavaThread (lifecycle.c:1096).
Attempting to allocate 3G bytes
There is insufficient native memory for the Java
Runtime Environment to continue.
Possible reasons:
The system is out of physical RAM or swap space
In 32 bit mode, the process size limit was hit
Possible solutions:
<strong>Reduce memory load on the system
Increase physical memory or swap space
Check if swap backing store is full
Use 64 bit Java on a 64 bit OS
Decrease Java heap size (-Xmx/-Xms)
Decrease number of Java threads
Decrease Java thread stack sizes (-Xss)
Disable compressed references (-XXcompressedRefs=false)
</strong>
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:640)
at ThreadExample.main(ThreadExample.java:6)
linux系统资源有各种限制：内存限制。栈限制，文件数目限制，线程限制。要小心陷进这些问题。
# ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 79010 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 10240 cpu time (seconds, -t) unlimited max user processes (-u) 1024 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited
这里有两篇非常好的文章：
http://people.redhat.com/alikins/system_tuning.html#threads
http://kavassalis.com/2011/03/linux-and-the-maximum-number-of-processes-threads/