unix.Script中的性能调整非常慢 [英] performance tuning in unix.Script is very slow

问题描述

我们有一个名为data_mask的函数.它接受字符串并屏蔽它.在我们的文件中，我们希望根据用户提供的输入来屏蔽任何列.为此，我们将文件的列分为三部分.

We have a function named as data_mask. it accept string and mask it. in our file we want to mask any column based on the input given by user. for that we are splitting the columns of file in 3 parts.

while read p; do

  if [[ $line -le $skip_line ]]; then
    echo "$p" >> $outputfile
  else
    pre_str=`echo $p | cut -d'|' -f1-$((colnum - 1))`
    column_value=`echo $p | cut -d'|' -f$colnum`
    post_str=`echo $p | cut -d'|' -f$((colnum + 1))-$totalcol`
    echo "column_value=$column_value"
     maskvalue=$(data_mask "$column_value")
    echo $pre_str"|"$maskvalue"|"$post_str >> $outputfile


  fi

  line=$((line + 1))

  done <$temp_outputfile

此处colnum是我们通过传递用户输入的列的值而调用的user.data_mask函数输入的数字. 假设用户输入5，则pre_str将存储从1到4的列.column_value将存储需要屏蔽的列. post_str将列存储在被屏蔽的列之后.在这里，我们拆分列值，然后将其串联.任何更好的方法.这需要很多时间.

here colnum is the number input by user.data_mask function we are calling by passing the value of column entered by user. Suppose user enter 5 then pre_str will store the column from 1 to 4. column_value will store the column which needs to be masked. post_str will store the column after the masked column. here we are splitting the column value and then concatenating it. Any better way to do it. it is taking lot of time.

我们可以通过awk或sed更改以下几行吗?我们正在分割每一行，并逐行读取数据.

Can we change the below lines through awk or sed. We are splitting everyline and reading data line by line.

 pre_str=`echo $p | cut -d'|' -f1-$((colnum - 1))`
    column_value=`echo $p | cut -d'|' -f$colnum`
    post_str=`echo $p | cut -d'|' -f$((colnum + 1))-$totalcol`
    echo "column_value=$column_value"
     maskvalue=$(data_mask "$column_value")
    echo $pre_str"|"$maskvalue"|"$post_str >> $outputfile

下面是示例输入:-

11|Shrut|consultant
12|wipro|company
13|capgemini|IT

如果用户输入2，则输出应为

if user enters 2 then output should be

11|sqmbr|consultant
12|itzaw|company
13|khvlipkoi|IT

用于屏蔽的算法被写入到函数data_mask中.我们只需要更改上面的代码. 下面是我们的data_mask函数.

Algorithm for masking is written in the function data_mask. we have to change only the above code. Below is the our data_mask function.

data_mask() {

  col_val=$1
  l_ret_str=""
  l_an=0
  l_lp=0
  l_mod=0
  absnum=0
  austart=65
  auend=90
  aclsize=26
  alstart=97
  alend=122
  nstart=48
  nend=57
  nclsize=10

  l_lp=`expr length "$col_val"`
  if [[ $l_lp -ne 0 ]]; then
    for i in `eval "echo {1..$l_lp}"`
    do
      single_char=$(SUBSTR "$col_val" $i)
      ascii_num_val=$(ASCII "$single_char")
      l_mod=$((l_mod+ascii_num_val))
    done

    l_mod=$((l_mod % nclsize))

    for i in `eval "echo {1..$l_lp}"`
    do
      single_char=$(SUBSTR "$col_val" $i)
      ascii_num_val=$(ASCII "$single_char")
      l_an=$ascii_num_val
      tempvar=$((l_an - l_lp - l_mod - i))
      absnum=$(ABS $tempvar)
      if [[ $l_an -ge $austart && $l_an -le $auend ]]; then
        tempmodval=$((absnum % aclsize))
        tempasciival=$((austart + tempmodval))
        l_ret_str=$l_ret_str$(CHR $tempasciival)
      elif [[ $l_an -ge $alstart && $l_an -le $alend ]]; then
        tempmodval=$((absnum % aclsize))
        tempasciival=$((alstart + tempmodval))
        l_ret_str=$l_ret_str$(CHR $tempasciival)
      elif [[ $l_an -ge $nstart && $l_an -le $nend ]]; then
        tempmodval=$((absnum % nclsize))
        tempasciival=$((nstart + tempmodval))
        l_ret_str=$l_ret_str$(CHR $tempasciival)
      else
        tempmodval=$((absnum % nclsize))
        tempasciival=$((austart + tempmodval))
        l_ret_str=$l_ret_str$(CHR $tempasciival)
      fi

    done
  fi
  echo "$l_ret_str"
}

预先感谢

推荐答案

听起来这就是您要寻找的内容:

It sounds like this is what you're looking for:

$ cat tst.sh
data_mask() { printf '%s\n' "${1//?/_}"; }

col="$1"
file="$2"
skip=2

sep='|'
tail +"$(( skip + 1 ))" "$file" |
cut -f "$col" -d "$sep" |
while IFS= read -r val; do
    data_mask "$val"
done |
awk -F"$sep" -v OFS="$sep" -v col="$col" -v skip="$skip" '
    NR==FNR { a[NR]=$0; next }
    FNR > skip { $col = a[FNR-skip] }
    { print }
' - "$file"

我的data_mask()只是将每个字符转换为下划线-显然将其替换为您的真实函数.

My data_mask() just converts every char to an underscore - obviously replace it with your real function.

如前所述，如果您在awk中重写data_mask()(看起来像是一个微不足道的任务)，那么其余部分也可以完全在awk中完成，然后执行速度实际上将提高100倍.

As previously stated if you re-write data_mask() in awk (which looks like it'd be a trivial task) then the rest can also be done entirely in awk and then the execution will speed up literally 100-fold.

这篇关于unix.Script中的性能调整非常慢的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！