从bash中以竖线分隔的键和值使用jq创建JSON [英] Create JSON using jq from pipe-separated keys and values in bash

问题描述

我正在尝试从bash中的字符串创建一个json对象.字符串如下.

I am trying to create a json object from a string in bash. The string is as follows.

CONTAINER|CPU%|MEMUSAGE/LIMIT|MEM%|NETI/O|BLOCKI/O|PIDS
nginx_container|0.02%|25.09MiB/15.26GiB|0.16%|0B/0B|22.09MB/4.096kB|0

输出来自docker stats命令，我的最终目标是将自定义指标发布到AWS CloudWatch.我想将此字符串格式化为json.

The output is from docker stats command and my end goal is to publish custom metrics to aws cloudwatch. I would like to format this string as json.

{
    "CONTAINER":"nginx_container",
    "CPU%":"0.02%", 
    ....
}

我以前使用过jq命令，似乎在这种情况下应该可以正常工作，但是我还没有能够提出一个好的解决方案.除了使用sed或awk对变量名进行硬编码和索引编制以外.然后从头开始创建一个json.任何建议，将不胜感激.谢谢.

I have used jq command before and it seems like it should work well in this case but I have not been able to come up with a good solution yet. Other than hardcoding variable names and indexing using sed or awk. Then creating a json from scratch. Any suggestions would be appreciated. Thanks.

推荐答案

前提条件

对于以下所有内容，假定您的内容位于名为s的shell变量中:

Prerequisite

For all of the below, it's assumed that your content is in a shell variable named s:

s='CONTAINER|CPU%|MEMUSAGE/LIMIT|MEM%|NETI/O|BLOCKI/O|PIDS
nginx_container|0.02%|25.09MiB/15.26GiB|0.16%|0B/0B|22.09MB/4.096kB|0'

什么(现代jq)

# thanks to @JeffMercado and @chepner for refinements, see comments
jq -Rn '
( input  | split("|") ) as $keys |
( inputs | split("|") ) as $vals |
[[$keys, $vals] | transpose[] | {key:.[0],value:.[1]}] | from_entries
' <<<"$s"

操作方式(现代jq)

这需要非常新的(可能是1.5?)jq来工作，并且是密集的代码块.要对其进行分解:

How (modern jq)

This requires very new (probably 1.5?) jq to work, and is a dense chunk of code. To break it down:

• 使用-n可以防止jq自己读取stdin，而让input和inputs可以读取整个输入流，前者只能读取一行，而后者读取所有剩余的行. (对于原始输入，-R导致读取文本行而不是JSON对象).
• 使用[$keys, $vals] | transpose[]，我们将生成[key, value]对(以Python的方式，将两个列表压缩).
• 使用{key:.[0],value:.[1]}，我们将每对[key, value]组合成{"key": key, "value": value}
形式的对象
• 使用from_entries，我们将这些对组合到包含这些键和值的对象中.

• Using -n prevents jq from reading stdin on its own, leaving the entirety of the input stream available to be read by input and inputs -- the former to read a single line, and the latter to read all remaining lines. (-R, for raw input, causes textual lines rather than JSON objects to be read).
• With [$keys, $vals] | transpose[], we're generating [key, value] pairs (in Python terms, zipping the two lists).
• With {key:.[0],value:.[1]}, we're making each [key, value] pair into an object of the form {"key": key, "value": value}
• With from_entries, we're combining those pairs into objects containing those keys and values.

这将在比上述版本更旧的jq上使用，并且对于本机jq解决方案可能更难以解决的情况是一种容易采用的方法:

This will work with a significantly older jq than the above, and is an easily adopted approach for scenarios where a native-jq solution can be harder to wrangle:

{
   IFS='|' read -r -a keys # read first line into an array of strings

   ## read each subsequent line into an array named "values"
   while IFS='|' read -r -a values; do

    # setup: positional arguments to pass in literal variables, query with code    
    jq_args=( )
    jq_query='.'

    # copy values into the arguments, reference them from the generated code    
    for idx in "${!values[@]}"; do
        [[ ${keys[$idx]} ]] || continue # skip values with no corresponding key
        jq_args+=( --arg "key$idx"   "${keys[$idx]}"   )
        jq_args+=( --arg "value$idx" "${values[$idx]}" )
        jq_query+=" | .[\$key${idx}]=\$value${idx}"
    done

    # run the generated command
    jq "${jq_args[@]}" "$jq_query" <<<'{}'
  done
} <<<"$s"

---

操作方式(外壳辅助)

上面调用的jq命令类似于:

---

How (shell-assisted)

The invoked jq command from the above is similar to:

jq --arg key0   'CONTAINER' \
   --arg value0 'nginx_container' \
   --arg key1   'CPU%' \
   --arg value1 '0.0.2%' \
   --arg key2   'MEMUSAGE/LIMIT' \
   --arg value2 '25.09MiB/15.26GiB' \
   '. | .[$key0]=$value0 | .[$key1]=$value1 | .[$key2]=$value2' \
   <<<'{}'

...将每个键和值带外传递(这样就将其视为文字字符串而不是解析为JSON)，然后分别进行引用.

...passing each key and value out-of-band (such that it's treated as a literal string rather than parsed as JSON), then referring to them individually.

以上任何一种都会发出:

Either of the above will emit:

{
  "CONTAINER": "nginx_container",
  "CPU%": "0.02%",
  "MEMUSAGE/LIMIT": "25.09MiB/15.26GiB",
  "MEM%": "0.16%",
  "NETI/O": "0B/0B",
  "BLOCKI/O": "22.09MB/4.096kB",
  "PIDS": "0"
}

---

为什么

简而言之:因为可以保证生成有效的JSON作为输出.

---

Why

In short: Because it's guaranteed to generate valid JSON as output.

将以下示例视为打破更幼稚方法的示例:

Consider the following as an example that would break more naive approaches:

s='key ending in a backslash\
value "with quotes"'

当然，这是意外情况，但是jq知道如何处理它们:

Sure, these are unexpected scenarios, but jq knows how to deal with them:

{
  "key ending in a backslash\\": "value \"with quotes\""
}

...而一个不理解JSON字符串的实现很容易最终发出:

...whereas an implementation that didn't understand JSON strings could easily end up emitting:

{
  "key ending in a backslash\": "value "with quotes""
}

这篇关于从bash中以竖线分隔的键和值使用jq创建JSON的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！