1mega的大数组导致CPU占用率高? [英] Big array of size 1mega caused high CPU?

问题描述

我有一个多线程服务器应用程序.该应用程序从套接字接收数据，然后处理这些数据，例如拆包，添加到数据队列等，其功能如下.经常调用此功能.有一条select语句，如果发现有数据，它将调用此函数来接收):

I have a multithreaded server application. This application receives data from sockets then handles these data like unpacking package, adding to data queue, etc, the function is as below. This function is called frequently. There is a select statement and if it finds there is data it will call this function to receive):

         //the main function used to receive 
         //file data from clients
         void service(void){
              while(1){
                   ....
                   struct timeval timeout;
                   timeout.tv_sec = 3;

                   ...
                   ret = select(maxFd+1, &read_set, NULL, NULL, &timeout);
                   if (ret > 0){
                       //get socket from SocketsMap
                       //if fd in SocketsMap and its being set
                       //then receive data from the socket
                       receive_data(fd);
                   }
              }
         } 

         void receive_data(int fd){
              const int ONE_MEGA = 1024 * 1024;

              //char *buffer = new char[ONE_MEGA]; consumes much less CPU
              char buffer[ONE_MEGA]; // cause high CPU 
              int readn = recv(fd, buffer, ONE_MEGA, 0);

              //handle the data
         }

我发现上面的资源消耗过多的CPU，通常是80％到90％，但是如果我从堆中创建缓冲区，则CPU仅为14％.为什么?

I found the above consumes too much CPU -- usually 80% to 90%, but if I create the buffer from heap instead the CPU is only 14%. Why?

[更新]
添加了更多代码

[update]
Added more code

[update2]
最糟糕的是，我还编写了另一个简单的数据接收服务器和客户端.服务器只是从套接字接收数据，然后将其丢弃.两种类型的空间分配工作原理几乎相同，CPU使用率没有太大差异.在有问题的多线程服务器应用程序中，我什至将进程堆栈大小重置为30M，使用数组仍然会导致问题，但是从堆中分配可以解决该问题.我不知道为什么.

[update2]
The stangest thing is that I also wrote another simple data-receiving server and client. The server simply receives data from sockets then discard it. Both types of space allocating works almost the same, no big difference in CPU usage. In the multithreaded server application which has the problem, I even reset the process stack size to 30M, using array still results in the problem, but allocating from heap solves it. I don't know why.

关于"sizeof(buffer)"，感谢您指出这一点，但是我100％确信这不是问题，因为在我的应用程序中，我不使用sizeof(buffer)，而是ONE_MEGA(1024 * 1024).

Regarding the "sizeof(buffer)", thanks for pointing out this, but I am 100% sure that it is not the problem, because in my application I don't use sizeof(buffer), but ONE_MEGA (1024*1024) instead.

顺便说一句，尽管我不确定它是否有用，但还有一件事需要提及. 使用较小的数组(例如"char缓冲区[1024]")替换数组；这也大大降低了CPU使用率.

By the way, there is one more thing to mention though I am not sure it's useful or not. Replacing the array with a smaller one such as "char buffer[1024]; also decreases the cpu usage dramatically.

[update3]
所有套接字都处于非阻塞模式.

[update3]
All sockets are in non-blocking mode.

推荐答案

我只是这样写的:

#include <iostream>
#include <cstdio>

using namespace std;

static __inline__ unsigned long long rdtsc(void)
{
    unsigned hi, lo;
    __asm__ __volatile__ ("rdtsc" : "=a"(lo), "=d"(hi));
    return ( (unsigned long long)lo)|( ((unsigned long long)hi)<<32 );
}

const int M = 1024*1024;

void bigstack()
{
    FILE *f = fopen("test.txt", "r");
    unsigned long long time;
    char buffer[M];

    time = rdtsc();
    fread(buffer, M, 1, f);
    time = rdtsc() - time;
    fclose(f);
    cout << "bs: Time = " << time / 1000 << endl;
}


void bigheap()
{
    FILE *f = fopen("test.txt", "r");
    unsigned long long time;
    char *buffer = new char[M];

    time = rdtsc();
    fread(buffer, M, 1, f);
    time = rdtsc() - time;
    delete [] buffer;
    fclose(f);
    cout << "bh: Time = " << time / 1000 << endl;
}



int main()
{
    for(int i = 0; i < 10; i++)
    {
    bigstack();
    bigheap();
    }
}

输出是这样的:

bs: Time = 8434
bh: Time = 7242
bs: Time = 1094
bh: Time = 2060
bs: Time = 842
bh: Time = 830
bs: Time = 785
bh: Time = 781
bs: Time = 782
bh: Time = 804
bs: Time = 782
bh: Time = 778
bs: Time = 792
bh: Time = 809
bs: Time = 785
bh: Time = 786
bs: Time = 782
bh: Time = 829
bs: Time = 786
bh: Time = 781

换句话说，从堆的堆栈进行分配绝对没有区别.开始时少量的缓慢"与预热缓存"有关.

In other words, allocating from the stack of the heap makes absolutely no difference. The small amount of "slowness" in the beginning has to do with "warming up the caches".

我非常相信，您的代码在两者之间行为不同的原因还有其他原因-也许simonc说:sizeof buffer是问题吗?

And I'm fairly convinced that the reason your code behaves differently between the two is something else - maybe what simonc says: sizeof buffer is the problem?

这篇关于1mega的大数组导致CPU占用率高?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！