如何以跨系统的方式将进程仅绑定到物理内核? [英] How to bind a process to only physical cores in a cross system way?

问题描述

我正在使用 一个项目，每次您将线程数加倍，您会增加 40% 到 60% 的开销.由于超线程将性能提高到最多 30%，这意味着程序在超线程系统上的运行速度比单线程模式要慢.

I’m using a project where each time you double the number of threads, you add between 40% to 60% overhead. As hyperthreading increases performance to a maximum of 30% this means, the program runs slower than in single thread mode on hyperthreaded systems.

第一步似乎很简单.

• 通过len(os.sched_getaffinity(0))

通过z3参数限制线程数.

Restrict the number of threads through z3 parameters.

使用 os.sched_setaffinity(0,mask) 将线程绑定到物理内核.

Bind the threads to physical cores using os.sched_setaffinity(0,mask).

为 platform.machine() 中不包含 Intel 或 amd 的系统启用 smt 解决方案.

Leave smt solutions enabled for systems not containing Intel or amd inside platform.machine().

然而，这样做会产生几个问题.

However several problems arise for doing this.

• 如何知道系统是否启用了超线程?

• How to know if the system has hyperthreading enabled?

在使用os.sched_setaffinity(0,mask)之前，如何知道哪些cpu核数是物理的还是逻辑的?

Before using os.sched_setaffinity(0,mask), how to know which cpu core numbers are physical or logical?

问题是该程序目前通过 python3 支持多种平台:所有 Unixes，以及 Windows 和 Osx 以及 Openvms，同时不要忘记 PyPy.

The problem is the program currently supports a wide number of platforms through python3: all Unixes, as well as Windows and Osx and Openvms while not forgetting PyPy.

修复问题的任何补丁都不应产生新进程，也不应添加未包含的依赖项，也不应放弃对上述某些平台的支持.

Any patch to fix the problem shouldn’t spawn a new process nor add a non-included dependency nor drop support for some of the platforms above.

有什么干净的方法可以解决这个问题?

What can be a clean way to fix this?

推荐答案

loky 库 包含一个相当便携的解决方案.它确实产生了一个进程，然后缓存了结果——所以它不像你多次产生一个进程.鉴于这是支持 sklearn 等流行库的解决方案，我猜它几乎和它得到的一样好.

The loky library contains a fairly portable solution to this. It does spawn a process, and then caches the result -- so it's not like you're spawning a process more than once. Given this is the solution which backs popular libraries like sklearn, I would guess that it's almost as good as it gets.

这篇关于如何以跨系统的方式将进程仅绑定到物理内核?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！