如何在Python中将不同的进程固定到单个cpu内核 [英] How to pin different processes to individual cpu cores in Python

问题描述

我正在编写一个Python程序，该程序将从文本文件中加载单词列表，然后尝试将每个单词的存档解压缩.如果不使用所有cpu内核，那将不是很严重.由于存在GIL，如果我没记错的话，Python线程不是一个好选择.

I'm writing a Python program that will load a wordlist from a text file and then try unzipping an archive with each word. It wouldn't be serious if it didn't make use of all cpu cores. Because of the GIL, threading in Python isn't a great option if I'm not mistaken.

所以我想获取cpu_cores的数量，拆分单词列表，并使用multiprocessing.process模块​​在不同的进程中处理单词列表的不同部分.

So I want to get the number of cpu_cores, split the wordlist and use the multiprocessing.process module to process different parts of the wordlist in different processes.

但是，每个进程都会自动固定到cpu内核吗?如果没有，是否可以手动固定它们?

But would every process get pinned to a cpu core automatically? If not, is there a way to pin them manually?

推荐答案

您可以通过导入使用Pythons multiprocessing import multiprocessing as mp并使用mp.cpu_count()找出处理器的数量，并且可以在大多数平台上使用.

You can use Pythons multiprocessing by importing import multiprocessing as mp and find out the number of processors by using mp.cpu_count() and should work on most platforms.

要在特定的CPU内核上启动程序/进程(在Linux中)，可以使用 taskset ，并使用此指南作为参考.

To launch programs/processes on specific CPU cores (in linux) you can use taskset and use this guide as a reference.

另一种跨平台的解决方案是对Python使用 psutil 包.

An alternative cross-plattform solution would be to use the psutil package for python.

但是我建议您使用线程/进程池方法，因为我认为您应该让操作系统将任务分配给每个cpu/核心.您可以查看如何在python多处理中利用所有内核如何解决这个问题.

However i would suggest you go with a thread/process pooling approach as in my opinion you should let the operating system assign tasks to each cpu/core. You can look at How to utilize all cores with python multiprocessing on how to approach this problem.

这篇关于如何在Python中将不同的进程固定到单个cpu内核的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！