在嵌入式Linux系统中查找延迟问题（停滞） [英] Finding latency issues (stalls) in embedded Linux systems

问题描述

我有一个在Atmel AT91SAM9260EK板上运行的嵌入式Linux系统，其中有两个进程以实时优先级运行。管理器进程使用POSIX消息队列周期性地ping工作进程来检查工作进程的运行状况。通常，往返ping大约需要1ms，但是偶尔需要更长的时间 - 大约800ms 。没有其他进程以较高的优先级运行。

似乎停顿可能与日志记录（syslog）有关。如果我停止记录问题似乎消失了。但是，如果日志文件位于JFFS2或NFS上，则不会有任何差异。没有其他进程正在写入磁盘 - 只是syslog。

我可以使用哪些工具来帮助我跟踪为什么这些档位正在发生？我已经知道了latencytop，并且会使用它。有些其他工具可能更有用吗？

一些细节：

• 内核版本：2.6.32.8


• libc（syslog函数）：uClibc 0.9.30.1


• syslog：busybox 1.15.2


• 未配置交换空间[在编辑中添加]


• 根文件系统在tmpfs上（从initramfs加载）[在编辑中添加]

解决方案

问题是（如你所说）syslogd。当您的进程以RT优先级运行时，syslogd不是。此外，syslogd不会锁定其堆，并且可以（并且将被内核调出），特别是对于非常少的客户。

您可以尝试的是：

• 启动另一个线程来管理优先级队列，让该线程与syslog通信。然后记录将只是获取一个锁并将某些东西插入到列表中。只有两个订阅者，你不应该花很多时间获取互斥体。




• 不使用syslog，实现自己的日志记录（基本上是第一个建议，减去与syslog通话）

我有一个类似的问题，我第一次尝试修复它是将syslogd本身修改为锁定它的堆。那是一场灾难。然后我尝试了rsyslogd，这改进了一些，但我仍然有随机的延迟峰值。我最终只是使用优先级队列来实现我自己的日志记录，以帮助确保更多的关键消息实际上是先写的。

注意，如果你不使用交换），修复这个的最短路径可能是实现自己的日志记录。

I have an embedded Linux system running on an Atmel AT91SAM9260EK board on which I have two processes running at real-time priority. A manager process periodically "pings" a worker process using POSIX message queues to check the health of the worker process. Usually the round-trip ping takes about 1ms, but very occasionally it takes much longer - about 800ms. There are no other processes that run at a higher priority.

It appears the stall may be related to logging (syslog). If I stop logging the problem seems to go away. However it makes no difference if the log file is on JFFS2 or NFS. No other processes are writing to the "disk" - just syslog.

What tools are available to me to help me track down why these stalls are occurring? I am aware of latencytop and will be using that. Are there some other tools that may be more useful?

Some details:

• Kernel version: 2.6.32.8
• libc (syslog functions): uClibc 0.9.30.1
• syslog: busybox 1.15.2
• No swap space configured [added in edit]
• root filesystem is on tmpfs (loaded from initramfs) [added in edit]

解决方案

The problem is (as you said) syslogd. While your process is running at a RT priority, syslogd is not. Additionally, syslogd does not lock its heap and can (and will) be paged out by the kernel, especially with very few 'customers'.

What you could try is:

• Start another thread to manage a priority queue, have that thread talk to syslog. Logging would then just be acquiring a lock and inserting something into a list. Given only two subscribers, you should not spend a lot of time acquiring the mutex.

• Not using syslog, implement your own logging (basically the first suggestion, minus talking to syslog).

I had a similar problem and my first attempt to fix it was to modify syslogd itself to lock its heap. That was a disaster. I then tried rsyslogd, which improved some but I still got random latency peaks. I ended up just implementing my own logging using a priority queue to help ensure that more critical messages were actually written first.

Note, if you are not using swap (at all), the shortest path to fixing this is probably implementing your own logging.

这篇关于在嵌入式Linux系统中查找延迟问题（停滞）的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！