在写CSV时将列添加到CSV [英] Add columns to CSV while writing the CSV

问题描述

我正在写一个csv中的以下数据：

I am writing on the fly the following data in a csv:

name first file parsed                    
STEP ID  ELEMENT_ID  Fatigue SW  Fatigue F1  Fatigue F3
Step 10  10000       1.30E-07    1.51E-06    2.15E-06

当我完成解析第一个文件，并开始第二个我想添加更多的列如下：

when I finish to parse the first file, and start the second I would like to add more columns as follows:

name first file parsed                                   name first file parsed
STEP ID  ELEMENT_ID  Fatigue SW  Fatigue F1  Fatigue F3  Fatigue SW  Fatigue F1  Fatigue F3
Step 10  10000       1.30E-07    1.51E-06    2.15E-06    1.30E-07    1.51E-06    2.15E-06

我正在阅读的文件是大量的2Gb，所以我不能创建列表，我需要写我正在解析。

The files I am reading in are massive 2Gb, so I cannot afford to create lists, I need to write as I am parsing.

任何建议？

推荐答案

您可以使用以下上下文管理器来替换一个文件：

You can use the following context manager to make replacing a file a little easier:

from contextlib import contextmanager
import io
import os


@contextmanager
def inplace(filename, mode='r', buffering=-1, encoding=None, errors=None,
            newline=None, backup_extension=None):
    """Allow for a file to be replaced with new content.

    yields a tuple of (readable, writable) file objects, where writable
    replaces readable.

    If an exception occurs, the old file is restored, removing the
    written data.

    mode should *not* use 'w', 'a' or '+'; only read-only-modes are supported.

    """

    # move existing file to backup, create new file with same permissions
    # borrowed extensively from the fileinput module
    if set(mode) & set('wa+'):
        raise ValueError('Only read-only file modes can be used')

    backupfilename = filename + (backup_extension or os.extsep + 'bak')
    try:
        os.unlink(backupfilename)
    except os.error:
        pass
    os.rename(filename, backupfilename)
    readable = io.open(backupfilename, mode, buffering=buffering,
                       encoding=encoding, errors=errors, newline=newline)
    try:
        perm = os.fstat(readable.fileno()).st_mode
    except OSError:
        writable = open(filename, 'w' + mode.replace('r', ''),
                        buffering=buffering, encoding=encoding, errors=errors,
                        newline=newline)
    else:
        os_mode = os.O_CREAT | os.O_WRONLY | os.O_TRUNC
        if hasattr(os, 'O_BINARY'):
            os_mode |= os.O_BINARY
        fd = os.open(filename, os_mode, perm)
        writable = io.open(fd, "w" + mode.replace('r', ''), buffering=buffering,
                           encoding=encoding, errors=errors, newline=newline)
        try:
            if hasattr(os, 'chmod'):
                os.chmod(filename, perm)
        except OSError:
            pass
    try:
        yield readable, writable
    except Exception:
        # move backup back
        try:
            os.unlink(filename)
        except os.error:
            pass
        os.rename(backupfilename, filename)
        raise
    finally:
        readable.close()
        writable.close()
        try:
            os.unlink(backupfilename)
        except os.error:
            pass

ccsv 使用inplace（csvfilename，'rb'）添加列的模块：

Use this with the csv module to add columns:

with inplace(csvfilename, 'rb') as (infh, outfh):
    reader = csv.reader(infh)
    writer = csv.writer(outfh)

    for row in reader:
        row += ['new', 'column']
        writer.writerow(row)

这篇关于在写CSV时将列添加到CSV的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！