• Linux系统OOM killer机制详解


    介绍:

    Linux下面有个特性叫OOM killer(Out Of Memory killer),会在系统内存耗尽的情况下出现,选择性的干掉一些进程以求释放一些内存。广大从事Linux方面的IT农民工兄弟或多或少都有遇到过吧。

    现象:

    主机SSH突然无法连接,主机重启要过很久,重启后主机正常,登录查看日志信息,会有大量的 " Out of Memory: Killed process xxx 、Out of Memory: Killed process xxx "

    案例:

    kernel: lowmem_reserve[]: 0 0 0 0

    kernel: Normal free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no

    kernel: lowmem_reserve[]: 0 0 0 0

    kernel: HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no

    kernel: lowmem_reserve[]: 0 0 0 0

    kernel: DMA: 0*4kB 1*8kB 1*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 2040kB

    kernel: DMA32: 22*4kB 4*8kB 1*16kB 1*32kB 0*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 4008kB

    kernel: Normal: empty

    kernel: HighMem: empty

    kernel: 8917 pagecache pages

    kernel: Swap cache: add 96509, delete 88464, find 14295/17401, race 0+4

    kernel: Free swap = 0kB

    kernel: Total swap = 262136kB

    kernel: Free swap: 0kB

    kernel: 262144 pages of RAM

    kernel: 8801 reserved pages

    kernel: 45869 pages shared

    kernel: 8045 pages swap cached

    kernel: Out of memory: Killed process 330, UID 500, (mysqld).

    kernel: httpd invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0

    kernel:

    kernel: Call Trace:

    kernel: [<ffffffff802c2be6>] out_of_memory+0x8b/0x203

    kernel:[<ffffffff8020fab8>] __alloc_pages+0x27f/0x308

    kernel:[<ffffffff80213a38>] __do_page_cache_readahead+0xc8/0x1af

    kernel:[<ffffffff802142a9>] filemap_nopage+0x14c/0x360

    kernel:[<ffffffff80208e9d>] __handle_mm_fault+0x444/0x144f

    kernel:[<ffffffff80234691>] lock_sock+0xa7/0xb2

    kernel:[<ffffffff80232068>] release_sock+0x13/0xbe

    kernel:[<ffffffff8022c1bd>] local_bh_enable+0x9/0x9c

    kernel:[<ffffffff80234691>] lock_sock+0xa7/0xb2

    kernel:[<ffffffff80267d94>] do_page_fault+0xf72/0x131b

    kernel: [<ffffffff8041f85e>] move_addr_to_user+0x5d/0x78

    kernel: [<ffffffff8041ff45>] sys_accept+0x18c/0x1d2

    kernel: [<ffffffff8026082b>] error_exit+0x0/0x6e

    kernel:

    kernel: Mem-info:

    kernel: DMA per-cpu:

    kernel: cpu 0 hot: high 0, batch 1 used:0

    kernel: cpu 0 cold: high 0, batch 1 used:0

    kernel: cpu 1 hot: high 0, batch 1 used:0

    kernel: cpu 1 cold: high 0, batch 1 used:0

    kernel: DMA32 per-cpu:

    kernel: cpu 0 hot: high 186, batch 31 used:46

    kernel: cpu 0 cold: high 62, batch 15 used:12

    kernel: cpu 1 hot: high 186, batch 31 used:25

    kernel: cpu 1 cold: high 62, batch 15 used:14

    kernel: Normal per-cpu: empty

    kernel: HighMem per-cpu: empty

    kernel: Free pages: 6040kB (0kB HighMem)

    kernel: Active:120445 inactive:113583 dirty:0 writeback:0 unstable:0 free:1510 slab:4042 mapped-file:548 mapped-anon:234724 pagetables:7852

    kernel: DMA free:2040kB min:32kB low:40kB high:48kB active:0kB inactive:0kB present:9060kB pages_scanned:0 all_unreclaimable? yes

    kernel: lowmem_reserve[]: 0 994 994 994

    kernel: DMA32 free:4000kB min:4016kB low:5020kB high:6024kB active:481780kB inactive:454332kB present:1018080kB pages_scanned:1718135 all_unreclaimable? yes

    以上的日志信息标红处就是出现 OOM Killer(Out Of Memory killer)。

    原理:

    linux oom-killer是一种自我保护机制,当系统分配不出内存时(触发条件)会触发这个机制,由操作系统在己有进程中挑选一个占用内存较多,回收内存收益最大的进程kill掉来释放内存。

    系统为每个进程做评估(/proc/<pid>/oom_score中数值最大的进程被kill掉)。

    详见oom_killer.c

    /**
    * badness – calculate a numeric value for how bad this task has been
    * @p: task struct of which task we should calculate
    * @uptime: current uptime in seconds
    *
    * The formula used is relatively simple and documented inline in the
    * function. The main rationale is that we want to select a good task
    * to kill when we run out of memory.
    *
    * Good in this context means that:
    * 1) we lose the minimum amount of work done
    * 2) we recover a large amount of memory
    * 3) we don’t kill anything innocent of eating tons of memory
    * 4) we want to kill the minimum amount of processes (one)
    * 5) we try to kill the process the user expects us to kill, this
    * algorithm has been meticulously tuned to meet the principle
    * of least surprise … (be careful when you change it)
    */

    Linux下有3种Overcommit的策略(参考内核文档:vm/overcommit-accounting),可以在/proc/sys/vm/overcommit_memory配置。取0,1和2三个值,默认是0。

    0:启发式策略,比较严重的Overcommit将不能得逞,比如你突然申请了128TB的内存。而轻微的Overcommit将被允许。另外,root能Overcommit的值比普通用户要稍微多些。

    1:永远允许Overcommit,这种策略适合那些不能承受内存分配失败的应用,比如某些科学计算应用。

    2:永远禁止Overcommit,在这个情况下,系统所能分配的内存不会超过swap+RAM*系数(/proc/sys/vm/overcmmit_ratio,默认50%,你可以调整),如果这么多资源已经用光,那么后面任何尝试申请内存的行为都会返回错误,这通常意味着此时没法运行任何新程序。

  • 相关阅读:
    打开任务管理器的hang住的进程直接右键抓取dmp文件
    Excel匯入匯出整理
    ShellExecute 命令行参数最长2068
    一个判断是否为质数的正则表达式
    visualstudio 比较和merge工具命令行配置
    C++ 螺旋队列
    寻找连通线,参考八后算法递归,可用于验证码去除连续干扰线
    DES 加解密
    记个地址
    CF1383C String Transformation 2 题解
  • 原文地址:https://www.cnblogs.com/youngerchina/p/5624592.html
Copyright © 2020-2023  润新知