计算Spark StorageMemory Heap内存

计算Spark StorageMemory Heap内存

tag: Spark, Spark Memory, Spark Storage Memory

2021-04-23 21:26:25 星期五
version: spark-2.4.5
Executor 进程 org.apache.spark.executor.CoarseGrainedExecutorBackend
Driver 进程 org.apache.spark.deploy.SparkSubmit

CMS GC型

1. 找到Jvm进程 pid
   Executor/Driver process 通过端口找: netstat -tunpl | grep $port
   Driver process 通过Jar包找: ps -ef | grep jarFileName
   Executor/Driver process 通过applicationId查找: ps -ef | grep ${application-id}

2. 查询Executor内存

jmap -heap $pid
# 此处为计算方便将Executor的Xms设为最大值2048m 并设置关闭OffHeapMemory

结果:

Attaching to process ID 59755, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.77-b03

using thread-local object allocation.
Parallel GC with 53 thread(s)

Heap Configuration:
   MinHeapFreeRatio         = 0
   MaxHeapFreeRatio         = 100
   MaxHeapSize              = 2147483648 (2048.0MB)
   NewSize                  = 715653120 (682.5MB)
   MaxNewSize               = 715653120 (682.5MB)
   OldSize                  = 1431830528 (1365.5MB)
   NewRatio                 = 2
   SurvivorRatio            = 8
   MetaspaceSize            = 21807104 (20.796875MB)
   CompressedClassSpaceSize = 1073741824 (1024.0MB)
   MaxMetaspaceSize         = 17592186044415 MB
   G1HeapRegionSize         = 0 (0.0MB)

Heap Usage:
PS Young Generation
Eden Space:
   capacity = 537395200 (512.5MB)
   used     = 142312432 (135.71971130371094MB)
   free     = 395082768 (376.78028869628906MB)
   26.481894888528963% used
From Space:
   capacity = 89128960 (85.0MB)
   used     = 89115992 (84.98763275146484MB)
   free     = 12968 (0.01236724853515625MB)
   99.98545029584099% used
To Space:
   capacity = 89128960 (85.0MB)
   used     = 0 (0.0MB)
   free     = 89128960 (85.0MB)
   0.0% used
PS Old Generation
   capacity = 1431830528 (1365.5MB)
   used     = 163379752 (155.81107330322266MB)
   free     = 1268450776 (1209.6889266967773MB)
   11.41055095592989% used

17279 interned Strings occupying 1729944 bytes.

根据JVM计算出Spark Storage Memory 为
(Eden + From + PS Old)
=> ((537395200 + 89128960+ 1431830528 -300 * 1024 * 1024)) * 0.6
= 1046269132.8B
向下取整得 1046269132B

3. 获取Spark 应用的Storage Memory(因为ui界面Storage Memory是 按照 1KB = 1000B计算所以有差异)

curl http://appDriverHost:4040/api/v1/applications/app-id/allexecutors | grep maxMemory

% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  "maxMemory" : 1046269132,  #Executor
  ...

1046269132B 即是Spark ui界面上的 Storage Memory(该值是最大值, 即JVM申请全部内存的情况下的内存量)

---

G1 GC型

1. ...

2.  jmap -heap $pid
   

结果:

Attaching to process ID 174736, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.77-b03

using thread-local object allocation.
Garbage-First (G1) GC with 8 thread(s)

Heap Configuration:
   MinHeapFreeRatio         = 40
   MaxHeapFreeRatio         = 70
   MaxHeapSize              = 8589934592 (8192.0MB)
   NewSize                  = 1363144 (1.2999954223632812MB)
   MaxNewSize               = 5152702464 (4914.0MB)
   OldSize                  = 5452592 (5.1999969482421875MB)
   NewRatio                 = 2
   SurvivorRatio            = 8
   MetaspaceSize            = 21807104 (20.796875MB)
   CompressedClassSpaceSize = 1073741824 (1024.0MB)
   MaxMetaspaceSize         = 17592186044415 MB
   G1HeapRegionSize         = 2097152 (2.0MB)

Heap Usage:
G1 Heap:
   regions  = 4096
   capacity = 8589934592 (8192.0MB)
   used     = 3097300200 (2953.8156509399414MB)
   free     = 5492634392 (5238.184349060059MB)
   36.05731995776296% used
G1 Young Generation:
Eden Space:
   regions  = 815
   capacity = 3472883712 (3312.0MB)
   used     = 1709178880 (1630.0MB)
   free     = 1763704832 (1682.0MB)
   49.21497584541063% used
Survivor Space:
   regions  = 31
   capacity = 65011712 (62.0MB)
   used     = 65011712 (62.0MB)
   free     = 0 (0.0MB)
   100.0% used
G1 Old Generation:
   regions  = 694
   capacity = 2078277632 (1982.0MB)
   used     = 1323109608 (1261.8156509399414MB)
   free     = 755168024 (720.1843490600586MB)
   63.663756354184734% used

26551 interned Strings occupying 2726760 bytes.

根据JVM计算出Spark Storage Memory 为
(Eden + From + PS Old)
=> (8589934592 -300 *1024 *1024) * 0.6
= 4965217075.2B
向下取整得 4965217075B

3. 获取Spark 应用的Storage Memory

curl http://hadoop-master:4040/api/v1/applications/app-20210423171912-0326/allexecutors | grep maxMemory
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  6662  100  6662    0     0   667k      0 --:--:-- --:--:-- --:--:--  722k
  "maxMemory" : 4965217075, # Executor
  "maxMemory" : 455501414,  # Driver

4965217075B 即是Spark ui界面上的 Storage Memory(该值是最大值, 即JVM申请全部内存的情况下的内存量)