如何增加 cpu 内核的数量及其在 python 中的使用 [英] How to increase numbers of cpu cores and their usage in python

问题描述

我编写了一个计算成本很高的 python 脚本.在远程 linux 服务器上运行代码(通过 putty)，需要一个星期才能完成.

I've written a python script with heavy computational costs. After running the code in a remote linux server(via putty), it takes one week to get finished.

我们的远程服务器有 32 个 CPU 内核和 64 GB 内存.我需要通过使用更多数量的 cpu 核心在更短的时间内运行我的脚本，但我不知道如何为我的脚本使用更多数量的 cpu 核心来增加.

Our remote server has 32 cpu cores and 64 GB of Ram memory. I need to run my script in a shorter time by using more numbers of cpu cores but I don't know exactly how to increase using more numbers of cpu cores for my script.

我想知道是否有任何 python 代码要添加到我的 python 脚本的第一部分或末尾部分，用于添加 cpu 内核的数量、内存使用情况等，以便能够在更短的时间内运行脚本.获取脚本结果后，我还需要返回到默认状态.

I would like to know if there is any python code to add at the first or end part of my python script for adding numbers of cpu cores, memory usage and etc to be able to run the script in a shorter time. I also need to return to the default state after getting my script result.

有没有办法在 Python 中实现这一点?

Is there any way to achieve this in Python?

推荐答案

你可以看看 multiprocessing lib (https://docs.python.org/2/library/multiprocessing.html)，但您必须管理要在进程之间共享的数据.

You can take a look on multiprocessing lib (https://docs.python.org/2/library/multiprocessing.html) but you will have to manage data to share between processes.

您还可以查看线程库 (https://docs.python.org/2/library/threading.html)，如果您有很多 I/O，这可能很有效.

You can also take a look on threading lib (https://docs.python.org/2/library/threading.html) which may be efficient if you have lot of I/O.

最后你使用 asyncio lib (python 3.4+ https://docs.python.org/3/library/asyncio.html)，在我看来，实现起来更复杂，但让您有更多能力明确管理代码的行为.

Finally you use asyncio lib (python 3.4+ https://docs.python.org/3/library/asyncio.html) which is, in my opninion, more complicated to implement but give you more ability to manage explicitly the behavior of your code.

这是一个很大的问题，您可能会找到很多关于此问题的文档.这真的取决于你的代码真正做了什么，你最好给出你的代码片段.

This is a large question, you may find lot of documentation on this issue. It really depends on what your code really do, you should better give a snippet of your code.

这篇关于如何增加 cpu 内核的数量及其在 python 中的使用的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！