由于周期性的sendto()/recvfrom()延迟而导致的UDP套接字的周期性延迟尖峰，对于Linux RT-PREEMPT系统是C ++ [英] Periodic latency spikes from UDP socket caused by periodic sendto()/recvfrom() delay, C++ for Linux RT-PREEMPT system
问题描述
我已经设置了两个Raspberry Pi，以使用UDP套接字，一个作为客户端，一个作为服务器.内核已使用RT-PREEMPT(4.9.43-rt30 +)进行了修补.客户端充当对服务器的回显，以允许计算往返延迟(RTL).目前，服务器端正在使用10Hz的发送频率，它有2个线程:一个用于向客户端发送消息，另一个用于从客户端接收消息.使用循环调度将线程设置为具有95的调度优先级.
I have setup two Raspberry Pis to use UDP sockets, one as the client and one as the server. The kernel has been patched with RT-PREEMPT (4.9.43-rt30+). The client acts as an echo to the server to allow for the calculation of Round-Trip Latency (RTL). At the moment a send frequency of 10Hz is being used on the server side with 2 threads: one for sending the messages to the client and one for receiving the messages from the client. The threads are setup to have a schedule priority of 95 using Round-Robin scheduling.
服务器构造一条消息，其中包含发送消息的时间和开始发送消息以来的时间.该消息从服务器发送到客户端，然后立即返回到服务器.从客户端收到消息后，服务器将计算往返延迟，然后将其存储在.txt文件中，以用于使用Python进行绘图.
The server constructs a message containing the time the message was sent and the time past since messages started being sent. This message is sent from the server to the client then immediately returned to the server. Upon receiving the message back from the client the server calculates the Round-Trip Latency and then stores it in a .txt file, to be used for plotting using Python.
问题在于，当我分析图形时，我注意到RTL中存在周期性的峰值.图片的顶部图形: RTL延迟和sendto()+ recvfrom()时间.在图例中，我使用RTT而不是RTL.这些峰值与服务器端sendto()和recvfrom()调用中所示的峰值直接相关.关于我的应用程序非常依赖一致性的任何建议，如何消除这些尖峰?
The problem is that when analysing the graphs I noticed there is a periodic spike in the RTL. The top graph of the image:RTL latency and sendto() + recvfrom() times. In the legend I have used RTT instead of RTL. These spikes are directly related to the spikes shown in the server side sendto() and recvfrom() calls. Any suggestion on how to remove these spikes as my application is very reliant on consistency?
我尝试过并注意到的事情:
Things I have tried and noticed:
- 所发送消息的大小无效.我尝试了较大的消息(1024个字节)和较小的消息(0个字节)，并且周期性延迟不改变.这对我来说，这不是缓冲区问题，因为没有东西填满?
- 发送消息的频率确实起着很大的作用，如果频率增加一倍，则延迟峰值的发生频率是两倍.这是否表明某些东西正在填充，而它清空了sendto()/recvfrom()函数时会遇到延迟?
- 使用setsockop()更改缓冲区大小无效.
- 我尝试了很多其他设置(MSG_DONTWAIT等)，但无济于事.
我绝不是套接字/C ++编程/Linux方面的专家，所以我提出的任何建议都将不胜感激.以下是用于创建套接字并启动服务器线程以发送和接收消息的代码.下面是用于从服务器发送消息的代码，如果您需要其余的信息，请告诉我，但是现在我的关注点是围绕sendto()函数引起的延迟.如果您还有其他需要，请告诉我.谢谢.
I am by no means an expert in sockets/C++ programming/Linux so any suggestions given will be greatly appreciated as I am out of ideas. Below is the code used to create the socket and start the server threads for sending and receiving the messages. Below that is the code for sending the messages from the server, if you need the rest please let me know but for now my concern is centred around the delay caused by the sendto() function. If you need anything else please let me know. Thanks.
thread_priority = priority;
recv_buff = recv_buff_len;
std::cout << del << " Second start-up delay..." << std::endl;
sleep(del);
std::cout << "Delay complete..." << std::endl;
master = socket(AF_INET, SOCK_DGRAM, IPPROTO_UDP);
// master socket creation
if(master == 0){// Try to create the UDP socket
perror("Could not create the socket: ");
exit(EXIT_FAILURE);
}
std::cout << "Master Socket Created..." << std::endl;
std::cout << "Adjusting send and receive buffers..." << std::endl;
setBuff();
// Server address and port creation
serv.sin_family = AF_INET;// Address family
serv.sin_addr.s_addr = INADDR_ANY;// Server IP address, INADDR_ANY will
work on the server side only
serv.sin_port = htons(portNum);
server_len = sizeof(serv);
// Binding of master socket to specified address and port
if (bind(master, (struct sockaddr *) &serv, sizeof (serv)) < 0) {
//Attempt to bind master socket to address
perror("Could not bind socket...");
exit(EXIT_FAILURE);
}
// Show what address and port is being used
char IP[INET_ADDRSTRLEN];
inet_ntop(AF_INET, &(serv.sin_addr), IP, INET_ADDRSTRLEN);// INADDR_ANY
allows all network interfaces so it will always show 0.0.0.0
std::cout << "Listening on port: " << htons(serv.sin_port) << ", and
address: " << IP << "..." << std::endl;
// Options specific to the server RPi
if(server){
std::cout << "Run Time: " << duration << " seconds." << std::endl;
client.sin_family = AF_INET;// Address family
inet_pton(AF_INET, clientIP.c_str(), &(client.sin_addr));
client.sin_port = htons(portNum);
client_len = sizeof(client);
serv_send = std::thread(&SocketServer::serverSend, this);
serv_send.detach();// The server send thread just runs continuously
serv_receive = std::thread(&SocketServer::serverReceive, this);
serv_receive.join();
}else{// Specific to client RPi
SocketServer::clientReceiveSend();
}
以及用于发送消息的代码:
And the code for sending the messages:
// Setup the priority of this thread
param.sched_priority = thread_priority;
int result = sched_setscheduler(getpid(), SCHED_RR, ¶m);
if(result){
perror ("The following error occurred while setting serverSend() priority");
}
int ched = sched_getscheduler(getpid());
printf("serverSend() priority result %i : Scheduler priority id %i \n", result, ched);
std::ofstream Out;
std::ofstream Out1;
Out.open(file_name);
Out << duration << std::endl;
Out << frequency << std::endl;
Out << thread_priority << std::endl;
Out.close();
Out1.open("Server Side Send.txt");
packets_sent = 0;
Tbegin = std::chrono::high_resolution_clock::now();
// Send messages for a specified time period at a specified frequency
while(!stop){
// Setup the message to be sent
Tstart = std::chrono::high_resolution_clock::now();
TDEL = std::chrono::duration_cast< std::chrono::duration<double>>(Tstart - Tbegin); // Total time passed before sending message
memcpy(&message[0], &Tstart, sizeof(Tstart));// Send the time the message was sent with the message
memcpy(&message[8], &TDEL, sizeof(TDEL));// Send the time that had passed since Tstart
// Send the message to the client
T1 = std::chrono::high_resolution_clock::now();
sendto(master, &message, 16, MSG_DONTWAIT, (struct sockaddr *)&client, client_len);
T2 = std::chrono::high_resolution_clock::now();
T3 = std::chrono::duration_cast< std::chrono::duration<double>>(T2-T1);
Out1 << T3.count() << std::endl;
packets_sent++;
// Pause so that the required message send frequency is met
while(true){
Tend = std::chrono::high_resolution_clock::now();
Tdel = std::chrono::duration_cast< std::chrono::duration<double>>(Tend - Tstart);
if(Tdel.count() > 1/frequency){
break;
}
}
TDEL = std::chrono::duration_cast< std::chrono::duration<double>>(Tend - Tbegin);
// Check to see if the program has run as long as required
if(TDEL.count() > duration){
stop = true;
break;
}
}
std::cout << "Exiting serverSend() thread..." << std::endl;
// Save extra results to the end of the last file
Out.open(file_name, std::ios_base::app);
Out << packets_sent << "\t\t " << packets_returned << std::endl;
Out.close();
Out1.close();
std::cout << "^C to exit..." << std::endl;
推荐答案
我已经解决了问题.它不是ARP表，因为即使禁用了ARP功能，也会出现周期性的峰值.禁用ARP功能后，延迟将只有一个峰值，而不是一系列延迟峰值.
I have sorted out the problem. It was not the ARP tables as even with the ARP functionality disabled there was a periodic spike. With the ARP functionality disabled there would only be a single spike in latency as opposed to a series of latency spikes.
事实证明，我使用的线程有问题，因为CPU上只有两个线程，一次只能处理一个线程.正在发送信息的一个线程正受到正在接收信息的第二个线程的影响.我改变了很多线程优先级(发送优先级高于接收，接收高于发送，发送等于接收)无济于事.我现在购买了具有4个内核的Raspberry Pi，并且将发送线程设置为在内核2上运行，而接收线程在内核3上运行，以防止线程相互干扰.这不仅消除了延迟峰值，而且减少了我的设置的平均延迟.
It turned out to be a problem with the threads I was using as there were two threads on a CPU only capable of handling one thread at a time. The one thread that was sending the information was being affected by the second thread that was receiving information. I changed the thread priorities around a lot (send priority higher than receive, receive higher than send and send equal to receive) to no avail. I have now bought a Raspberry Pi that has 4 cores and I have set the send thread to run on core 2 while the receive thread runs on core 3, preventing the threads from interfering with each other. This has not only removed the latency spikes but also reduced the mean latency of my setup.
这篇关于由于周期性的sendto()/recvfrom()延迟而导致的UDP套接字的周期性延迟尖峰，对于Linux RT-PREEMPT系统是C ++的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
