节点二storage 内存太高好像被系统kill了,现在storage down 了。
看了下dmesg, executor-pri3-2 这个被kill 了,翻了下 之前有过类似的的schedule 问题 2.0版本一个奇怪操作引发coredump - #18,来自 sworduo
[29296.431825] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
[29296.433521] [ 462] 0 462 9765 1129 23 0 0 systemd-journal
[29296.435560] [ 494] 0 494 11421 209 24 0 -1000 systemd-udevd
[29296.437315] [ 614] 0 614 13883 101 27 0 -1000 auditd
[29296.439233] [ 641] 0 641 11042 88 27 0 0 qemu-ga
[29296.441001] [ 642] 81 642 14559 165 34 0 -900 dbus-daemon
[29296.442927] [ 643] 0 643 1097 35 8 0 0 acpid
[29296.444567] [ 644] 0 644 6596 76 18 0 0 systemd-logind
[29296.446312] [ 645] 999 645 153058 1860 61 0 0 polkitd
[29296.448144] [ 647] 0 647 5404 77 16 0 0 irqbalance
[29296.450325] [ 653] 998 653 29452 115 30 0 0 chronyd
[29296.452199] [ 729] 0 729 137545 1032 86 0 0 NetworkManager
[29296.454267] [ 970] 0 970 181836 3250 25 0 0 node_exporter
[29296.456456] [ 971] 0 971 143569 3325 97 0 0 tuned
[29296.458253] [ 1214] 0 1214 22427 259 42 0 0 master
[29296.460040] [ 1240] 89 1240 22470 250 43 0 0 qmgr
[29296.461720] [ 1309] 0 1309 56680 486 46 0 0 rsyslogd
[29296.463367] [ 1314] 0 1314 285265 4237 42 0 0 elkeid-agent
[29296.465180] [ 1358] 0 1358 31596 156 19 0 0 crond
[29296.466960] [ 1425] 0 1425 28225 256 57 0 -1000 sshd
[29296.468582] [ 1426] 0 1426 27551 33 11 0 0 agetty
[29296.470228] [ 1428] 0 1428 27551 34 10 0 0 agetty
[29296.472064] [ 1671] 0 1671 25726 514 51 0 0 dhclient
[29296.474004] [ 1886] 0 1886 271750 71372 296 0 0 nebula-metad
[29296.475740] [ 1896] 0 1896 156429 71996 206 0 0 nebula-graphd
[29296.477556] [ 2399] 0 2399 40377 344 81 0 0 sshd
[29296.479332] [ 2401] 0 2401 28886 108 14 0 0 bash
[29296.480899] [ 2543] 0 2510 45081322 31527927 79735 0 0 executor-pri3-2
[29296.482541] [ 2945] 0 2945 40412 358 82 0 0 sshd
[29296.484046] [ 2947] 0 2947 28886 107 14 0 0 bash
[29296.485518] [ 2993] 0 2993 18063 188 40 0 0 sftp-server
[29296.487146] [ 3022] 89 3022 22453 249 43 0 0 pickup
[29296.488840] Out of memory: Kill process 2543 (executor-pri3-2) score 930 or sacrifice child
[29296.490471] Killed process 2543 (executor-pri3-2), UID 0, total-vm:180325288kB, anon-rss:126109248kB, file-rss:2460kB, shmem-rss:0kB
