while 循环读取文件速度极慢 [英] while loop extremely slow read file

问题描述

我有一个 while 循环，它读取 ftp 日志文件并将其放入一个数组中，这样我就可以搜索该数组并匹配/搜索流.不幸的是，while 循环需要花费很长时间才能通过文件，它是一个非常大的文件，但必须有另一种更快的方法来执行此操作.

I have a while loop that that reads in a ftp log file and puts it into an array so I'll be able to search through the array and match up/search for a flow. Unfortunately the while loop is taking forever to get through the file, it is a very large file but there must be another faster way of doing this.

# read file into array for original search results
while read FTP_SEARCH
do
ogl_date[count]=`echo $FTP_SEARCH | awk '{print $1, $2}'`
ogl_time[count]=`echo $FTP_SEARCH | awk '{print $3}'`
ogl_server[count]=`echo $FTP_SEARCH | awk '{print $4}'`
ogl_id[count]=`echo $FTP_SEARCH | awk '{print $5}'`
ogl_type[count]=`echo $FTP_SEARCH | awk -F '[' '{print $1}' | awk '{print $5}'`
ogl_pid[count]=`echo $FTP_SEARCH | awk -F'[' '{print $2}' | awk -F']' '{print $1}'`
ogl_commands[count]=`echo $FTP_SEARCH | awk '{
    for(i = 6; i <= NF; i++) 
        print $i;
    }'`

let "count += 1"

done < /tmp/ftp_search.14-12-02


Dec  1 23:59:03 sslmftp1 ftpd[4152]: USER xxxxxx  
Dec  1 23:59:03 sslmftp1 ftpd[4152]: PASS password  
Dec  1 23:59:03 sslmftp1 ftpd[4152]: FTP LOGIN FROM 172.19.x.xx [172.19.x.xx], xxxxxx  
Dec  1 23:59:03 sslmftp1 ftpd[4152]: PWD  
Dec  1 23:59:03 sslmftp1 ftpd[4152]: CWD /test/data/872507/  
Dec  1 23:59:03 sslmftp1 ftpd[4152]: TYPE Image`
Dec  1 23:59:03 sslmftp1 ftpd[4152]: PASV
Dec  1 23:59:04 sslmftp1 ftpd[4152]: NLST
Dec  1 23:59:04 sslmftp1 ftpd[4152]: FTP session closed
Dec  1 23:59:05 sslmftp1 ftpd[4683]: USER xxxxxx 
Dec  1 23:59:05 sslmftp1 ftpd[4683]: PASS password
Dec  1 23:59:05 sslmftp1 ftpd[4683]: FTP LOGIN FROM 172.19.1.24 [172.19.x.xx], xxxxxx 
Dec  1 23:59:05 sslmftp1 ftpd[4683]: PWD
Dec  1 23:59:05 sslmftp1 ftpd[4683]: CWD /test/data/944837/
Dec  1 23:59:05 sslmftp1 ftpd[4683]: TYPE Image

推荐答案

• 您不需要保留迭代器来添加到数组.你可以简单地做 array+=(item)(not array+=item).
• 获取输入中的列就像使用带有多个目标变量的 read 一样简单.作为奖励，最后一个变量获得第 N 个词和所有后续词.参见 help [r]ead.
• You don't need to keep an iterator to add to arrays. You can simply do array+=(item) (not array+=item).
• Getting the columns in the input is as simple as using read with multiple target variables. As a bonus, the last variable gets the Nth word and all subsequent words. See help [r]ead.

这节省了大量的叉子，但我还没有测试它有多快.

This saves a ton of forks, but I haven't tested how fast it is.

ogl_date=()
[...]
ogl_commands=()

while read -r date1 date2 time server id type pid commands
do
    ogl_date+=("$date1 $date2")
    [...]
    ogl_commands+=("$commands")
done < /tmp/ftp_search.14-12-02

这篇关于while 循环读取文件速度极慢的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！