如何使用jq将数组拆分为块? [英] How to split an array into chunks with jq?

问题描述

我有一个非常大的JSON文件，其中包含一个数组.是否可以使用 jq 将此数组拆分为固定大小的几个较小的数组?假设我的输入是这样的: [1,2,3,4,5,6,7,8,9,10] ，我想将其拆分为3个元素长的块. jq 的期望输出为:

I have a very large JSON file containing an array. Is it possible to use jq to split this array into several smaller arrays of a fixed size? Suppose my input was this: [1,2,3,4,5,6,7,8,9,10], and I wanted to split it into 3 element long chunks. The desired output from jq would be:

[1,2,3]
[4,5,6]
[7,8,9]
[10]

实际上，我的输入数组有将近300万个元素，所有的UUID.

In reality, my input array has nearly three million elements, all UUIDs.

推荐答案

由于CédricConnes，以下 window/3 的面向流的定义(github:connesc)，概括了 _nwise ，并说明了避免了使用拳击技术的拳击技术"流结束标记，因此可以使用如果流包含非JSON值 nan .定义还包括根据 window/3 的 _nwise/1 的内容.

The following stream-oriented definition of window/3, due to Cédric Connes (github:connesc), generalizes _nwise, and illustrates a "boxing technique" that circumvents the need to use an end-of-stream marker, and can therefore be used if the stream contains the non-JSON value nan. A definition of _nwise/1 in terms of window/3 is also included.

window/3 的第一个参数被解释为流.$ size是窗口大小，$ step指定要跳过的值数.例如，

The first argument of window/3 is interpreted as a stream. $size is the window size and $step specifies the number of values to be skipped. For example,

window(1,2,3; 2; 1)

产量:

[1,2]
[2,3]

窗口/3和_nsize/1

def window(values; $size; $step):
  def checkparam(name; value): if (value | isnormal) and value > 0 and (value | floor) == value then . else error("window \(name) must be a positive integer") end;
  checkparam("size"; $size)
| checkparam("step"; $step)
  # We need to detect the end of the loop in order to produce the terminal partial group (if any).
  # For that purpose, we introduce an artificial null sentinel, and wrap the input values into singleton arrays in order to distinguish them.
| foreach ((values | [.]), null) as $item (
    {index: -1, items: [], ready: false};
    (.index + 1) as $index
    # Extract items that must be reused from the previous iteration
    | if (.ready | not) then .items
      elif $step >= $size or $item == null then []
      else .items[-($size - $step):]
      end
    # Append the current item unless it must be skipped
    | if ($index % $step) < $size then . + $item
      else .
      end
    | {$index, items: ., ready: (length == $size or ($item == null and length > 0))};
    if .ready then .items else empty end
  );

def _nwise($n): window(.[]; $n; $n);

来源:

https://gist.github.com/connesc/d6b87cbacae13d4fd58763724049da58

这篇关于如何使用jq将数组拆分为块?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！