加快python的struct.unpack [英] Speed up python's struct.unpack

问题描述

我正在尝试加快脚本速度.它基本上会读取带有Velodyne的Lidar HDL-32信息的pcap文件，并允许我获取X，Y，Z和Intensity值.我已经使用python -m cProfile ./spTestPcapToLas.py对脚本进行了概要分析，并且在我的readDataPacket()函数调用中花费了最多的时间.在小型测试(80 MB文件)中，解压缩部分花费了大约56％的执行时间.

I am trying to speed up my script. It basically reads a pcap file with Velodyne's Lidar HDL-32 information and allows me to get X, Y, Z, and Intensity values. I have profiled my script using python -m cProfile ./spTestPcapToLas.py and it is spending the most amount of time in my readDataPacket() function calls. In a small test (80 MB file) the unpacking portion takes around 56% of the execution time.

我这样调用readDataPacket函数(chunk是指pcap文件):

I call the readDataPacket function like this (chunk refers to the pcap file):

packets = []
for packet in chunk:
    memoryView = memoryview(packet.raw())
    udpDestinationPort = unpack('!h', memoryView[36:38].tobytes())[0]

    if udpDestinationPort == 2368:
        packets += readDataPacket(memoryView)

readDataPacket()函数本身的定义如下:

The readDataPacket() function itself is defined like this:

def readDataPacket(memoryView):
    firingData = memoryView[42:]    
    firingDataStartingByte = 0    
    laserBlock = []

    for i in xrange(firingBlocks):
        rotational = unpack('<H', firingData[firingDataStartingByte+2:firingDataStartingByte+4])[0]        
        startingByte = firingDataStartingByte+4
        laser = []
        for j in xrange(lasers):   
            distanceInformation = unpack('<H', firingData[startingByte:(startingByte + 2)])[0] * 0.002
            intensity = unpack('<B', firingData[(startingByte + 2)])[0]   
            laser.append([distanceInformation, intensity])
            startingByte += 3
        firingDataStartingByte += 100
        laserBlock.append([rotational, laser])

    return laserBlock

关于如何加快流程的任何想法?顺便说一句，我在X，Y，Z，强度计算中使用numpy.

Any ideas on how I can speed up the process? By the way, I am using numpy for the X, Y, Z, Intensity calculations.

推荐答案

Numpy使您可以非常快速地完成此操作.在这种情况下，我认为最简单的方法是使用 ndarray 直接构造函数:

Numpy lets you do this very quickly. In this case I think the easiest way is to use the ndarray constructor directly:

import numpy as np

def with_numpy(buffer):
    # Construct ndarray with: shape, dtype, buffer, offset, strides.
    rotational = np.ndarray((firingBlocks,), '<H', buffer, 42+2, (100,))
    distance = np.ndarray((firingBlocks,lasers), '<H', buffer, 42+4, (100,3))
    intensity = np.ndarray((firingBlocks,lasers), '<B', buffer, 42+6, (100,3))
    return rotational, distance*0.002, intensity

这将返回单独的数组，而不是嵌套列表，这应该更易于进一步处理.作为输入，它需要一个buffer对象(在Python 2中)或任何公开缓冲区接口的对象.不幸的是，这取决于您的Python版本(2/3)可以确切使用哪些对象.但是这种方法非常快:

This returns separate arrays instead of the nested list, which should be much easier to process further. As input it takes a buffer object (in Python 2) or anything that exposes the buffer interface. Unfortunately, it depends on your Python version (2/3) what objects you can use exactly. But this method is very fast:

import numpy as np

firingBlocks = 10**4
lasers = 32
packet_raw = np.random.bytes(42 + firingBlocks*100)

%timeit readDataPacket(memoryview(packet_raw))
# 1 loop, best of 3: 807 ms per loop
%timeit with_numpy(packet_raw)
# 100 loops, best of 3: 10.8 ms per loop

这篇关于加快python的struct.unpack的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！