如何使用jq和Bash将JSON拼合为Bash关联数组，其中Key = Selector? [英] How to Flatten JSON using jq and Bash into Bash Associative Array where Key=Selector?

问题描述

作为 Flatten Arbitrary JSON的后续操作，我正在寻找平坦的结果，使其适合进行查询并更新回原始JSON文件.

As a follow-up to Flatten Arbitrary JSON, I'm looking to take the flattened results and make them suitable for doing queries and updates back to the original JSON file.

动机:我正在编写Bash(4.2+)脚本(在CentOS 7上)，该脚本使用JSON选择器/过滤器作为键将JSON读取到Bash关联数组中.我对关联数组进行处理，最后，我想用这些更改来更新JSON.

Motivation: I'm writing Bash (4.2+) scripts (on CentOS 7) that read JSON into a Bash associative array using the JSON selector/filter as the key. I do processing on the associative arrays, and in the end I want to update the JSON with those changes.

前面的解决方案使我接近这个目标.我认为有两件事是不会做的:

The preceding solution gets me close to this goal. I think there are two things that it doesn't do:

1. 它不引用需要引用的键.例如，com.acme键需要加引号，因为它包含特殊字符.
2. 数组索引未以可用于查询原始JSON的形式表示.

1. It doesn't quote keys that require quoting. For example, the key com.acme would need to be quoted because it contains a special character.
2. Array indexes are not represented in a form that can be used to query the original JSON.

现有解决方案

以上解决方法是:

$ jq --stream -n --arg delim '.' 'reduce (inputs|select(length==2)) as $i ({};
[$i[0][]|tostring] as $path_as_strings
    | ($path_as_strings|join($delim)) as $key
    | $i[1] as $value
    | .[$key] = $value
)' input.json

例如，如果input.json包含:

{
   "a.b":
   [
      "value"
   ]
}

然后输出为:

{
  "a.b.0": "value"
}

真正需要的是

本来可以得到改善:

{
  "\"a.b\"[0]": "value"
}

但是我真正想要的是输出格式，以便可以直接在Bash程序中获取它(暗示将数组名称作为参数传递给jq):

But what I really want is output formatted so that it could be sourced directly in a Bash program (implying the array name is passed to jq as an argument):

ArrayName['"a.b"[0]']='value'  # Note 'value' might need escapes for Bash

我希望上面具有更易理解的语法，而不是更通用的语法:

I'm looking to have the more human-readable syntax above as opposed to the more general:

ArrayName['.["a.b"][0]']='value'

我不知道jq是否可以处理所有这一切.我目前的解决方案是从前面的解决方案中获取输出，并将其后处理为所需的形式.这是正在进行的工作:

I don't know if jq can handle all of this. My present solution is to take the output from the preceding solution and to post-process it to the form that I want. Here's the work in process:

#!/bin/bash
Flatten()                                                                                                                                                                                                                       
{
local -r OPTIONS=$(getopt -o d:m:f: -l "delimiter:,mapname:,file:" -n "${FUNCNAME[0]}" -- "$@")
eval set -- "$OPTIONS"

local Delimiter='.' MapName=map File=
while true ; do
   case "$1" in
   -d|--delimiter)   Delimiter="$2"; shift 2;;
   -m|--mapname)     MapName="$2"; shift 2;;
   -f|--file)        File="$2"; shift 2;;
   --)               shift; break;;
   esac
done

local -a Array=()
readarray -t Array <<<"$(
   jq -c -S --stream -n --arg delim "$Delimiter" 'reduce (inputs|select(length==2)) as $i ({}; .[[$i[0][]|tostring]|join($delim)] = $i[1])' <<<"$(sed 's|^\s*[#%].*||' "$File")" |
   jq -c "to_entries|map(\"\(.key)=\(.value|tostring)\")|.[]" |
   sed -e 's|^"||' -e 's|"$||' -e 's|=|\t|')"

if [[ ! -v $MapName ]]; then
   local -gA $MapName
fi

. <(
   IFS=$'\t'
   while read -r Key Value; do
      printf "$MapName[\"%s\"]=%q\n" "$Key" "$Value"
   done <<<"$(printf "%s\n" "${Array[@]}")"
)
}
declare -A Map
Flatten -m Map -f "$1"
declare -p Map

输出:

$ ./Flatten.sh <(echo '{"a.b":["value"]}')
declare -A Map='([a.b.0]="value" )'

推荐答案

1)jq已经完成了Turing，所以这只是要使用哪个锤子的问题.

1) jq is Turing complete, so it's all just a question of which hammer to use.

2)

本来可以得到改善:

An improvement would have been:

{ "\" a.b \"[0]":值" }

{ "\"a.b\"[0]": "value" }

使用辅助功能可以很容易地实现这些目标:

That is easily accomplished using a helper function along these lines:

def flattenPath(delim):
  reduce .[] as $s ("";
    if $s|type == "number" 
    then ((if . == "" then "." else . end) + "[\($s)]")
    else . + ($s | tostring | if index(delim) then "\"\(.)\"" else . end)
    end );

3)

我对关联数组进行处理，最后我想用这些更改来更新JSON.

I do processing on the associative arrays, and in the end I want to update the JSON with those changes.

这表明您可能提出了 xy问题.但是，如果您确实确实要序列化和反序列化某些JSON文本，那么使用jq的自然方法是使用leaf_paths，如以下序列化/反序列化功能所示:

This suggests you might have posed an xy-problem. However, if you really do want to serialize and unserialize some JSON text, then the natural way to do so using jq is using leaf_paths, as illustrated by the following serialization/deserialization functions:

# Emit (path, value) pairs
# Usage: jq -c -f serialize.jq input.json > serialized.json
def serialize: leaf_paths as $p | ($p, getpath($p));


# Usage: jq -n -f unserialize.jq serialized.json
def unserialize:
  def pairwise(s):
    foreach s as $i ([]; 
      if length == 1 then . + [$i] else [$i] end;
      select(length == 2));
  reduce pairwise(inputs) as $p (null; setpath($p[0]; $p[1]));

如果使用bash，则可以使用readarray(mapfile)将路径和值读取到单个数组中，或者如果要更轻松地区分路径和值，则可以(例如)使用如下所示的方法:

If using bash, you could use readarray (mapfile) to read the paths and values into a single array, or if you want to distinguish between the paths and values more easily, you could (for example) use the approach illustrated by the following:

i=0
while read -r line ; do
  path[$i]="$line"; read -r line; value[$i]="$line"
  i=$((i + 1))
done < serialized.json

但是还有许多其他选择.

But there are many other alternatives.

这篇关于如何使用jq和Bash将JSON拼合为Bash关联数组，其中Key = Selector?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！