php-fpm7.1 mmap/munmap(非常)在虚拟化系统上的性能下降(大页) [英] php-fpm7.1 mmap/munmap (very) slow performance on virtualized systems (hugepage)
问题描述
我的php-fpm进程在Ubuntu 14.04 LTS(Nginx服务器，MariaDB数据库)上遇到性能问题.
strace -f $(pidof php-fpm7.1 | sed 's/\([0-9]*\)/\-p \1/g')
给我
<... epoll_wait resumed> {}, 1, 1000) = 0
[pid 32533] epoll_wait(8, {}, 1, 103) = 0
[pid 32533] epoll_wait(8, <unfinished ...>
[pid 32535] mmap(NULL, 2097152, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fd933fdd000
[pid 32535] munmap(0x7fd933fdd000, 2097152) = 0
[pid 32535] mmap(NULL, 4190208, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fd933dde000
[pid 32535] munmap(0x7fd933dde000, 139264) = 0
[pid 32535] munmap(0x7fd934000000, 1953792) = 0
[pid 32535] madvise(0x7fd933e00000, 2097152, MADV_HUGEPAGE) = 0
[pid 32533] <... epoll_wait resumed> {}, 1, 897) = 0
[pid 32533] epoll_wait(8, {}, 1, 1000) = 0
[pid 32533] epoll_wait(8, {}, 1, 1000) = 0
[pid 32533] epoll_wait(8, <unfinished ...>
[pid 32535] mmap(NULL, 2097152, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fd933c00000
[pid 32535] madvise(0x7fd933c00000, 2097152, MADV_HUGEPAGE) = 0
[pid 32533] <... epoll_wait resumed> {}, 1, 1000) = 0
[pid 32533] epoll_wait(8, {}, 1, 1000) = 0
[pid 32533] epoll_wait(8, {}, 1, 1000) = 0
[pid 32533] epoll_wait(8, {}, 1, 1000) = 0
[pid 32533] epoll_wait(8, <unfinished ...>
[pid 32535] mmap(NULL, 2097152, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fd933a00000
[pid 32535] madvise(0x7fd933a00000, 2097152, MADV_HUGEPAGE) = 0
[pid 32533] <... epoll_wait resumed> {}, 1, 1000) = 0
[pid 32533] epoll_wait(8, <unfinished ...>
[pid 32535] open("/usr/share/zoneinfo/UTC", O_RDONLY) = 7
[pid 32535] fstat(7, {st_mode=S_IFREG|0644, st_size=118, ...}) = 0
[pid 32535] read(7, "TZif2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0", 20) = 20
[pid 32535] lseek(7, 0, SEEK_SET) = 0
[pid 32535] mmap(NULL, 118, PROT_READ, MAP_SHARED, 7, 0) = 0x7fd946835000
[pid 32535] close(7) = 0
[pid 32535] munmap(0x7fd946835000, 118) = 0
[pid 32535] pwrite(5, "_sf2_attributes|a:2:{s:14:\"_secu"..., 979, 0) = 979
[pid 32535] close(5) = 0
[pid 32535] mmap(NULL, 2097152, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fd933200000
[pid 32535] madvise(0x7fd933200000, 2097152, MADV_HUGEPAGE) = 0
我尝试使用php-fpm7.0，PHPMod7.1，但存在相同的问题.
对于数据量较小的请求，CPU的利用率高达100％.
配置是默认配置.
在重复的实例上，php5.6-fpm运行良好.
可能相关的 PHP脚本一直在执行mmap/munmap
我尝试启用大页面 https://wiki.debian.org/Hugepages
cat /proc/meminfo | grep Huge给了我
AnonHugePages: 108544 kB
HugePages_Total: 512
HugePages_Free: 497
HugePages_Rsvd: 50
HugePages_Surp: 0
Hugepagesize: 2048 kB
但仍然是同一问题.
我试图启用/禁用OPCache，也设置了opcache.huge_code_pages=0，没有结果.在 http://php.net/
我不确定100％是否在这里遇到相同的问题，但这是我在搜索Stack Overflow时可以找到的最接近的问题
我的发现是在具有以下规格的虚拟机上进行的.
CPU :16
RAM :16GB
操作系统:Ubuntu 16.04.4 LTS
容器:使用块设备的具有ZFS文件系统的LXD
PHP版本:7.1
我正在运行一些运行MariaDB，Nginx和PHP-FPM 7.1的LXD容器.这些是开发环境，所以我只是访问服务器的流量.运行的应用程序是使用Symfony框架构建的.
我注意到页面加载时间非常慢.我会持续等待一分钟，以便页面以开发人员模式加载.我也没有花太多时间来确认这一点，但是CLI脚本也感觉很慢.我尝试调整各种PHP设置(realpath缓存，opcache的开/关/各种设置等)，但没有任何效果.
我可以始终跟踪一个PHP-FPM进程，并看到一个似乎慢的系统调用.所有其他调用都会轻而易举，但是在整个过程的生命周期中，它多次陷入下一个调用中.
madvise(0x7f4dcca00000, 2097152, MADV_HUGEPAGE) = 0
长话短说，通过禁用THP，我能够彻底改变此应用程序的性能.我在LXD主机上运行了以下命令，并且页面加载时间像黑夜一样改变了.
sysctl -w vm.nr_hugepages=0
echo never > /sys/kernel/mm/transparent_hugepage/enabled
我知道Redis建议禁用THP，因为与写时复制相关的性能问题.我也知道ZFS文件系统也可以写时复制，所以也许这个问题是相关的?
My php-fpm process is facing performance issues on Ubuntu 14.04 LTS (Nginx server, MariaDB database).
strace -f $(pidof php-fpm7.1 | sed 's/\([0-9]*\)/\-p \1/g')
Gave me
<... epoll_wait resumed> {}, 1, 1000) = 0
[pid 32533] epoll_wait(8, {}, 1, 103) = 0
[pid 32533] epoll_wait(8, <unfinished ...>
[pid 32535] mmap(NULL, 2097152, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fd933fdd000
[pid 32535] munmap(0x7fd933fdd000, 2097152) = 0
[pid 32535] mmap(NULL, 4190208, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fd933dde000
[pid 32535] munmap(0x7fd933dde000, 139264) = 0
[pid 32535] munmap(0x7fd934000000, 1953792) = 0
[pid 32535] madvise(0x7fd933e00000, 2097152, MADV_HUGEPAGE) = 0
[pid 32533] <... epoll_wait resumed> {}, 1, 897) = 0
[pid 32533] epoll_wait(8, {}, 1, 1000) = 0
[pid 32533] epoll_wait(8, {}, 1, 1000) = 0
[pid 32533] epoll_wait(8, <unfinished ...>
[pid 32535] mmap(NULL, 2097152, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fd933c00000
[pid 32535] madvise(0x7fd933c00000, 2097152, MADV_HUGEPAGE) = 0
[pid 32533] <... epoll_wait resumed> {}, 1, 1000) = 0
[pid 32533] epoll_wait(8, {}, 1, 1000) = 0
[pid 32533] epoll_wait(8, {}, 1, 1000) = 0
[pid 32533] epoll_wait(8, {}, 1, 1000) = 0
[pid 32533] epoll_wait(8, <unfinished ...>
[pid 32535] mmap(NULL, 2097152, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fd933a00000
[pid 32535] madvise(0x7fd933a00000, 2097152, MADV_HUGEPAGE) = 0
[pid 32533] <... epoll_wait resumed> {}, 1, 1000) = 0
[pid 32533] epoll_wait(8, <unfinished ...>
[pid 32535] open("/usr/share/zoneinfo/UTC", O_RDONLY) = 7
[pid 32535] fstat(7, {st_mode=S_IFREG|0644, st_size=118, ...}) = 0
[pid 32535] read(7, "TZif2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0", 20) = 20
[pid 32535] lseek(7, 0, SEEK_SET) = 0
[pid 32535] mmap(NULL, 118, PROT_READ, MAP_SHARED, 7, 0) = 0x7fd946835000
[pid 32535] close(7) = 0
[pid 32535] munmap(0x7fd946835000, 118) = 0
[pid 32535] pwrite(5, "_sf2_attributes|a:2:{s:14:\"_secu"..., 979, 0) = 979
[pid 32535] close(5) = 0
[pid 32535] mmap(NULL, 2097152, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fd933200000
[pid 32535] madvise(0x7fd933200000, 2097152, MADV_HUGEPAGE) = 0
I tried with php-fpm7.0, PHPMod7.1 but same issues.
CPU is up to 100% on requests with a small amount of data.
Configurations are the default ones.
On a duplicated instance php5.6-fpm works well.
Edit: Possibly related PHP script keeps doing mmap/munmap
Edit: I tried to enable hugepages https://wiki.debian.org/Hugepages
A cat /proc/meminfo | grep Huge gave me
AnonHugePages: 108544 kB
HugePages_Total: 512
HugePages_Free: 497
HugePages_Rsvd: 50
HugePages_Surp: 0
Hugepagesize: 2048 kB
but still the same issue.
Edit: I tried to enable/disable OPCache, also set opcache.huge_code_pages=0, no results. There is no documentation about hugepages on http://php.net/
I'm not 100% sure if we are facing the same issue here, but it's the closest thing I could find in my search of Stack Overflow for my issue.
My findings were on a virtual machine with the following specs.
CPUs: 16
RAM: 16GB
OS: Ubuntu 16.04.4 LTS
Containers: LXD with ZFS filesystem using block devices
PHP Version: 7.1
I'm running a few LXD containers running MariaDB, Nginx, and PHP-FPM 7.1. These were dev environments, so I was only traffic hitting the server. The application running was built with the Symfony framework.
I was noticing extremely slow page load times. I would consistently wait over a minute for the page to load in dev mode. I didn't spend as much time confirming this as well, but CLI scripts also felt quite slow. I tried tweaking all kinds of PHP settings (realpath caches, opcache on/off/various settings, etc) but nothing worked.
I could consistently strace one of the PHP-FPM processes and see one system call that appeared to be slow. All the other calls would breeze by, but it was getting stuck on the following call many times throughout the process's lifespan.
madvise(0x7f4dcca00000, 2097152, MADV_HUGEPAGE) = 0
Long story short, I was able to drastically change the performance of this application by disabling THP. I ran the following commands on the LXD host and the page load times changed like night and day.
sysctl -w vm.nr_hugepages=0
echo never > /sys/kernel/mm/transparent_hugepage/enabled
I know that Redis recommends disabling THP because of performance issues related to copy-on-write. I also know ZFS filesystems also does copy-on-write, so maybe this issue is related?
这篇关于php-fpm7.1 mmap/munmap(非常)在虚拟化系统上的性能下降(大页)的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
