Python动态写入大文件，避免100％CPU使用率 [英] Python Write dynamically huge files avoiding 100% CPU Usage

问题描述

我正在

I am parsing a huge CSV approx 2 GB files with the help of this great stuff. Now have to generate dynamic files for each column in a new file where column name as file name. So I written this code to write the dynamic files:

def write_CSV_dynamically(self, header, reader):
  """
  :header - CSVs first row in string format
  :reader - CSVs all other rows in list format  
  """

  try:
    headerlist =header.split(',') #-- string headers 
    zipof = lambda x, y: zip(x.split(','), y.split(','))
    filename = "{}.csv".format(self.dtstamp)
    filename = "{}_"+filename
    filesdct = {filename.format(k.strip()):open(filename.format(k.strip()), 'a')\
    for k in headerlist}
    for row in reader:
      for key, data in zipof(header, row):
        filesdct[filename.format(key.strip())].write( str(data) +"\n" )
    for _, v in filesdct.iteritems():
      v.close()
  except Exception, e:
    print e

现在使用100% CPU可以花大约50秒的时间来写这些大文件.因为服务器上还有其他繁重的工作.我想阻止我的程序仅使用10％到20％的CPU并写入这些文件.无论花费10到15分钟. 如何优化我的代码，以使其限制10-20％的CPU使用率.

Now its taking around 50 secs to write these huge files using 100% CPU.As there are other heavy things running on my server. I want to block my program to use only 10% to 20% of the CPU and write these files. No matter if it takes 10-15 mins. How can I optimize my code, so that it should limit 10-20% CPU usage.

推荐答案

有多种方法可以实现:

• 很好的过程-简单明了.

cpulimit -只需将脚本和cpu的使用作为参数传递:

cpulimit - just pass your script and cpu usage as parameters:

cpulimit -P/path/to/your/script -l 20

cpulimit -P /path/to/your/script -l 20

Python的资源程序包可从脚本中设置限制.请记住，它与绝对CPU时间一起工作.

Python's resource package to set limits from the script. Bear in mind it works with absolute CPU time.

这篇关于Python动态写入大文件，避免100％CPU使用率的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！