问题描述
Java程序运行过程中抛出java.lang.OutOfMemoryError: unable to create new native thread，如下所示：
[java] view plain copy
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:691)
at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949)
at java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1017)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1163)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
[java] view plain copy
Caused by: java.lang.OutOfMemoryError
at java.util.zip.ZipFile.open(Native Method)
at java.util.zip.ZipFile.(ZipFile.java:214)
at java.util.zip.ZipFile.(ZipFile.java:144)
at java.util.jar.JarFile.(JarFile.java:153)
at java.util.jar.JarFile.(JarFile.java:117)
从JVM层面去解决
减小thread stack的大小
JVM默认thread stack的大小为1024，这样当线程多时导致Native virtual memory被耗尽，实际上当thread stack的大小为128K 或 256K时是足够的，所以我们如果明确指定thread stack为128K 或 256K即可，具体使用-Xss，例如在JVM启动的JVM_OPT中添加如下配置
[java] view plain copy
-Xss128k
减小heap或permgen初始分配的大小
如果JVM启动的JVM_OPT中有如下配置
[java] view plain copy
-Xms1303m -Xmx1303m -XX:PermSize=256m -XX:MaxPermSize=256m
我们可以删除或减小初始化最小值的配置，如下
[java] view plain copy
-Xms256m -Xmx1303m -XX:PermSize=64m -XX:MaxPermSize=256m
[java] view plain copy
-Xmx1303m -XX:MaxPermSize=256m
升级JVM到最新的版本
最新版本的JVM一般在内存优化方面做的更好，升级JVM到最新的版本可能会缓解测问题
从操作系统层面去解决
使用64位操作系统
如果使用32位操作系统遇到unable to create new native thread，建议使用64位操作系统
增大OS对线程的限制
在Linux操作系统设定nofile和nproc，具体编辑/etc/security/limits.conf添加如下：
[html] view plain copy
soft nofile 2048
hard nofile 8192
[html] view plain copy
soft nproc 2048
hard nproc 8192
如果使用Red Hat Enterprise Linux 6，编辑/etc/security/limits.d/90-nproc.conf，添加如下配置：
[html] view plain copy
cat /etc/security/limits.d/90-nproc.conf
- soft nproc 1024
root soft nproc unlimited
user - nproc 2048
JVM Crash抛出如下信息：
[java] view plain copy
There is insufficient memory for the Java Runtime Environment to continue.
Native memory allocation (malloc) failed to allocate 813056 bytes for Chunk::new
An error report file with more information is saved as:
/home/kylin/work/brms/brms-standalone-5.3.1/jboss-as/bin/hs_err_pid26819.log
hs_err_pid26819.log文件内容如下：
[java] view plain copy
There is insufficient memory for the Java Runtime Environment to continue.
Native memory allocation (malloc) failed to allocate 813056 bytes for Chunk::new
Possible reasons:
The system is out of physical RAM or swap space
In 32 bit mode, the process size limit was hit
Possible solutions:
Reduce memory load on the system
Increase physical memory or swap space
Check if swap backing store is full
Use 64 bit Java on a 64 bit OS
Decrease Java heap size (-Xmx/-Xms)
Decrease number of Java threads
Decrease Java thread stack sizes (-Xss)
Set larger code cache with -XX:ReservedCodeCacheSize=
This output file may be truncated or incomplete.
Out of Memory Error (allocation.cpp:328), pid=26819, tid=1286601584
JRE version: 7.0_21-b11
Java VM: Java HotSpot(TM) Server VM (23.21-b01 mixed mode linux-x86 )
Failed to write core dump. Core dumps have been disabled. To enable core dumping, try “ulimit -c unlimited” before starting Java again
————— T H R E A D —————
Current thread (0x4cff4c00): JavaThread “C2 CompilerThread0” daemon [_thread_in_native, id=26829, stack(0x4ca7f000,0x4cb00000)]
Stack: [0x4ca7f000,0x4cb00000], sp=0x4cafd9a0, free space=506k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x722359] VMError::report_and_die()+0x199
V [libjvm.so+0x2e8ef2] report_vm_out_of_memory(char const*, int, unsigned int, char const*)+0x72
V [libjvm.so+0x14ddad] Chunk::operator new(unsigned int, unsigned int)+0x10d
V [libjvm.so+0x14dde9] Arena::grow(unsigned int)+0x29
V [libjvm.so+0x3d5fe2] PhaseIFG::init(unsigned int)+0x1c2
V [libjvm.so+0x23835c] PhaseChaitin::Register_Allocate()+0x75c
V [libjvm.so+0x2a2991] Compile::Code_Gen()+0x3b1
V [libjvm.so+0x2a5250] Compile::Compile(ciEnv*, C2Compiler*, ciMethod*, int, bool, bool)+0xbd0
V [libjvm.so+0x220f16] C2Compiler::compile_method(ciEnv*, ciMethod*, int)+0x176
V [libjvm.so+0x2a9f6a] CompileBroker::invoke_compiler_on_method(CompileTask*)+0x33a
V [libjvm.so+0x2ab037] CompileBroker::compiler_thread_loop()+0x417
V [libjvm.so+0x6ddd98] compiler_thread_entry(JavaThread*, Thread*)+0x18
V [libjvm.so+0x6e4944] JavaThread::thread_main_inner()+0xf4
V [libjvm.so+0x6e4ad1] JavaThread::run()+0x161
V [libjvm.so+0x5e1091] java_start(Thread*)+0x111
C [libpthread.so.0+0x6a2e] abort@@GLIBC_2.0+0x6a2e
阿里云 Java环境 每小时在根目录下生成 hs_err_pid*.log ，求教！
错误日志
OS环境：阿里云 1核 1G内存；
安装软件：安装了 JDK 1.7 TOMCAT 7.X；
部署内容：部署了 java web 工程，工程正常运行；
问题：每小时在跟目录下生成一个 hs_err_pid.log
已经采取的措施：
1、配置服务器Swap
2、配置tomcat jvm 内存 （bin/catalina.sh）
未果，求牛人指点。
文件内容如下：
#
There is insufficient memory for the Java Runtime Environment to continue.
pthread_getattr_np
Possible reasons:
The system is out of physical RAM or swap space
In 32 bit mode, the process size limit was hit
Possible solutions:
Reduce memory load on the system
Increase physical memory or swap space
Check if swap backing store is full
Use 64 bit Java on a 64 bit OS
Decrease Java heap size (-Xmx/-Xms)
Decrease number of Java threads
Decrease Java thread stack sizes (-Xss)
Set larger code cache with -XX:ReservedCodeCacheSize=
This output file may be truncated or incomplete.
#
Out of Memory Error (os_linux_x86.cpp:718), pid=29229, tid=140505843455744
#
JRE version: Java(TM) SE Runtime Environment (7.0_67-b01) (build 1.7.0_67-b01)
Java VM: Java HotSpot(TM) 64-Bit Server VM (24.65-b04 mixed mode linux-amd64 compressed oops)
Failed to write core dump. Core dumps have been disabled. To enable core dumping, try “ulimit -c unlimited” before starting Java again
#
————— T H R E A D —————
Current thread (0x00007fca0c08a800): JavaThread “Service Thread” daemon [_thread_new, id=29237, stack(0x0000000000000000,0x0000000000000000)]
Stack: [0x0000000000000000,0x0000000000000000], sp=0x00007fca10e34880, free space=137212737746k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x99eb8a]
V [libjvm.so+0x49721b]
V [libjvm.so+0x8237ca]
V [libjvm.so+0x823815]
V [libjvm.so+0x952614]
V [libjvm.so+0x958dd4]
V [libjvm.so+0x81f988]
————— P R O C E S S —————
Java Threads: ( => current thread )
=>0x00007fca0c08a800 JavaThread “Service Thread” daemon [_thread_new, id=29237, stack(0x0000000000000000,0x0000000000000000)]
0x00007fca0c088000 JavaThread “C2 CompilerThread1” daemon [_thread_blocked, id=29236, stack(0x00007fca10e36000,0x00007fca10f37000)]
0x00007fca0c085800 JavaThread “C2 CompilerThread0” daemon [_thread_blocked, id=29235, stack(0x00007fca10f37000,0x00007fca11038000)]
0x00007fca0c084000 JavaThread “Signal Dispatcher” daemon [_thread_blocked, id=29234, stack(0x00007fca11038000,0x00007fca11139000)]
0x00007fca0c064800 JavaThread “Finalizer” daemon [_thread_blocked, id=29233, stack(0x00007fca11139000,0x00007fca1123a000)]
0x00007fca0c062800 JavaThread “Reference Handler” daemon [_thread_blocked, id=29232, stack(0x00007fca1123a000,0x00007fca1133b000)]
0x00007fca0c008800 JavaThread “main” [_thread_in_vm, id=29230, stack(0x00007fca12195000,0x00007fca12296000)]
Other Threads:
0x00007fca0c05e000 VMThread [stack: 0x00007fca1133b000,0x00007fca1143c000] [id=29231]
VM state:not at safepoint (normal execution)
VM Mutex/Monitor currently owned by a thread: ([mutex/lock_event])
[0x00007fca0c006bc0] PeriodicTask_lock - owner thread: 0x00007fca0c008800
Heap
def new generation total 4800K, used 343K [0x00000000eb400000, 0x00000000eb930000, 0x00000000f0750000)
eden space 4288K, 8% used [0x00000000eb400000, 0x00000000eb455e40, 0x00000000eb830000)
from space 512K, 0% used [0x00000000eb830000, 0x00000000eb830000, 0x00000000eb8b0000)
to space 512K, 0% used [0x00000000eb8b0000, 0x00000000eb8b0000, 0x00000000eb930000)
tenured generation total 10624K, used 0K [0x00000000f0750000, 0x00000000f11b0000, 0x00000000fae00000)
the space 10624K, 0% used [0x00000000f0750000, 0x00000000f0750000, 0x00000000f0750200, 0x00000000f11b0000)
compacting perm gen total 21248K, used 2132K [0x00000000fae00000, 0x00000000fc2c0000, 0x0000000100000000)
the space 21248K, 10% used [0x00000000fae00000, 0x00000000fb015148, 0x00000000fb015200, 0x00000000fc2c0000)
No shared spaces configured.
Card table byte_map: [0x00007fca117bd000,0x00007fca11864000] byte_map_base: 0x00007fca11063000
Polling page: 0x00007fca1332b000
Code Cache [0x00007fca09000000, 0x00007fca09270000, 0x00007fca0c000000)
total_blobs=156 nmethods=0 adapters=126 free_code_cache=48779Kb largest_free_block=49950080
Compilation events (0 events):
No events
GC Heap History (0 events):
No events
Deoptimization events (0 events):
No events
Internal exceptions (1 events):
Event: 0.027 Thread 0x00007fca0c008800 Threw 0x00000000eb410380 at /HUDSON/workspace/7u-2-build-linux-amd64/jdk7u67/1368/hotspot/src/share/vm/prims/jni.cpp:3991
Events (10 events):
Event: 0.058 loading class 0x00007fca11701880
Event: 0.058 loading class 0x00007fca11701880 done
Event: 0.058 loading class 0x00007fca1170df80
Event: 0.058 loading class 0x00007fca1170df80 done
Event: 0.058 loading class 0x00007fca116c8ee0
Event: 0.058 loading class 0x00007fca116c8ee0 done
Event: 0.058 Thread 0x00007fca0c084000 Thread added: 0x00007fca0c084000
Event: 0.058 Thread 0x00007fca0c085800 Thread added: 0x00007fca0c085800
Event: 0.059 Thread 0x00007fca0c088000 Thread added: 0x00007fca0c088000
Event: 0.059 Thread 0x00007fca0c08a800 Thread added: 0x00007fca0c08a800
…..
资源暂时不可用错误(Out of memery)完美解决方案
首先说一下环境：
centos6.3
业务环境：weiboyi
情况说明一下：
为了适应公司业务发展需要，公司新购买了一批新机器，作为weiboyi的新环境，安装了比较新的操作系统：通统：centos6.3
一开始硬件（Dell R410）主板坏了，更换后再进行各种测试。
测试前是用root用户启动业务，没有什么问题。但是后来在用非root用户启动时出现种问题。java报如下错误：
There is insufficient memory for the Java Runtime Environment to continue.
Cannot create GC thread. Out of system resources.
Possible reasons:
The system is out of physical RAM or swap space
In 32 bit mode, the process size limit was hit
Possible solutions:
Reduce memory load on the system
Increase physical memory or swap space
Check if swap backing store is full
Use 64 bit Java on a 64 bit OS
Decrease Java heap size (-Xmx/-Xms)
Decrease number of Java threads
Decrease Java thread stack sizes (-Xss)
Set larger code cache with -XX:ReservedCodeCacheSize=
This output file may be truncated or incomplete.
#
Out of Memory Error (gcTaskThread.cpp:46), pid=50128, tid=140649625663232
同时运行java -version也报错
执行一个ls也会报资源暂时不足
害得我又是查看系统的/etc/security/limit.conf 又是搞/etc/sysctl.conf 然后换JDK版本等等，搞得精疲力尽，到处找相关的资料！
搞了几天没有搞定，今天再接着进行这方面的研究，在以前的生产环境：centos5.5下可以正常使用的系统参数时突然发现：新系统的max user processes 只有1024 而旧系统默认为278528
你M呀，问题是出在这里了，再在/etc/security/下一看。centos6多出来一个limits.d目录，下面有个文件： 90-nproc.config
此文件内容：
Default limit for number of user’s processes to prevent
accidental fork bombs.
See rhbz #432903 for reasoning.
- soft nproc 1024
这里限制了1024呀，果断注释。
再运行业务程序，问题解决！汗一下，centos6加这个搞什么呢，没有搞清楚！