使用jq用另一个json文件中的对象覆盖json文件 [英] overwrite a json file with objects from another json file using jq
问题描述
已修改:
你好，我在合并2个文件时遇到问题，我有2个具有以下结构的json文件:
Hello I'm having problems on merging 2 files basically I have 2 json files with this structure:
[
{
"uri": "some/url.feature",
"id": "safety-tests",
"keyword": "Feature",
"name": "Safety Tests",
"description": "Some description",
"line": 2,
"tags": [
{
"name": "@sometag",
"line": 1
}
],
"elements": [
{
"id": "some-element-id",
"keyword": "Scenario Outline",
"name": ": Some scenario name",
"description": "",
"line": 46,
"type": "scenario",
"tags": [
{
"name": "@sometag",
"line": 1
},
{
"name": "@someothertag",
"line": 31
}
],
"before": [
{
"match": {
"location": "some/test/file.rb:201"
},
"result": {
"status": "passed",
"duration": 15000
}
},
{
"match": {
"location": "some/other/file.rb:5"
},
"result": {
"status": "passed",
"duration": 1722192000
}
}
],
"steps": [
{
"keyword": "Given ",
"name": "Some step name",
"line": 46,
"output": [
"Some output"
],
"match": {
"location": "some/other/path/to/other/file.rb:137"
},
"result": {
"status": "passed",
"duration": 989158000
}
},
{
"keyword": "When ",
"name": "some other step",
"line": 46,
"output": [
"WARNING: static wait for 1 seconds."
],
"match": {
"location": "some/other/path/to/other/file.rb:80"
},
"result": {
"status": "passed",
"duration": 2700052000
}
},
{
"keyword": "And ",
"name": "Some other name",
"line": 46,
"match": {
"location": "some/other/path/to/other/file.rb:38"
},
"result": {
"status": "passed",
"duration": 954225000
}
},
{
"keyword": "Then ",
"name": "Some other step name",
"line": 46,
"match": {
"location": "some/other/path/to/other/file.rb:2"
},
"result": {
"status": "passed",
"duration": 38792000
}
},
{
"keyword": "And ",
"name": "And again some other step name",
"line": 46,
"match": {
"location": "some/other/path/to/other/file.rb:2"
},
"result": {
"status": "passed",
"duration": 39268000
}
},
{
"keyword": "And ",
"name": "Some other step name",
"line": 46,
"match": {
"location": "some/other/path/to/other/file.rb:2"
},
"result": {
"status": "passed",
"duration": 55637000
}
},
{
"keyword": "And ",
"name": "Some other step name",
"line": 46,
"match": {
"location": "some/other/path/to/other/file.rb:2"
},
"result": {
"status": "passed",
"duration": 38375000
}
},
{
"keyword": "When ",
"name": "Some other step name",
"line": 46,
"match": {
"location": "some/other/path/to/other/file.rb:12"
},
"result": {
"status": "passed",
"duration": 751416000
}
},
{
"keyword": "And ",
"name": "Some other step name",
"line": 46,
"match": {
"location": "some/other/path/to/other/file.rb:2"
},
"result": {
"status": "passed",
"duration": 28043000
}
},
{
"keyword": "Then ",
"name": "Some other step name",
"line": 46,
"match": {
"location": "some/other/path/to/other/file.rb:20"
},
"result": {
"status": "passed",
"duration": 5204000
}
}
],
"after": [
{
"match": {
"location": "some/other/path/to/other/file.rb:91"
},
"result": {
"status": "passed",
"duration": 20000
}
},
{
"match": {
"location": "some/other/path/to/other/file.rb:52"
},
"result": {
"status": "passed",
"duration": 5585000
}
},
{
"match": {
"location": "some/other/path/to/other/file.rb:27"
},
"result": {
"status": "passed",
"duration": 168146000
}
},
{
"match": {
"location": "some/other/path/to/other/file.rb:428"
},
"result": {
"status": "passed",
"duration": 62000
}
}
]
},
{
"id": "some-element-id",
"keyword": "Scenario Outline",
"name": ": Some scenario name",
"description": "",
"line": 46,
"type": "scenario",
"tags": [
{
"name": "@sometag",
"line": 1
},
{
"name": "@someothertag",
"line": 31
}
],
"before": [
{
"match": {
"location": "some/test/file.rb:201"
},
"result": {
"status": "passed",
"duration": 15000
}
},
{
"match": {
"location": "some/other/file.rb:5"
},
"result": {
"status": "passed",
"duration": 1722192000
}
}
],
"steps": [
{
"keyword": "Given ",
"name": "Some step name",
"line": 46,
"output": [
"Some output"
],
"match": {
"location": "some/other/path/to/other/file.rb:137"
},
"result": {
"status": "passed",
"duration": 989158000
}
},
{
"keyword": "When ",
"name": "some other step",
"line": 46,
"output": [
"WARNING: static wait for 1 seconds."
],
"match": {
"location": "some/other/path/to/other/file.rb:80"
},
"result": {
"status": "passed",
"duration": 2700052000
}
},
{
"keyword": "And ",
"name": "Some other name",
"line": 46,
"match": {
"location": "some/other/path/to/other/file.rb:38"
},
"result": {
"status": "passed",
"duration": 954225000
}
},
{
"keyword": "Then ",
"name": "Some other step name",
"line": 46,
"match": {
"location": "some/other/path/to/other/file.rb:2"
},
"result": {
"status": "passed",
"duration": 38792000
}
},
{
"keyword": "And ",
"name": "And again some other step name",
"line": 46,
"match": {
"location": "some/other/path/to/other/file.rb:2"
},
"result": {
"status": "passed",
"duration": 39268000
}
},
{
"keyword": "And ",
"name": "Some other step name",
"line": 46,
"match": {
"location": "some/other/path/to/other/file.rb:2"
},
"result": {
"status": "passed",
"duration": 55637000
}
},
{
"keyword": "And ",
"name": "Some other step name",
"line": 46,
"match": {
"location": "some/other/path/to/other/file.rb:2"
},
"result": {
"status": "passed",
"duration": 38375000
}
},
{
"keyword": "When ",
"name": "Some other step name",
"line": 46,
"match": {
"location": "some/other/path/to/other/file.rb:12"
},
"result": {
"status": "passed",
"duration": 751416000
}
},
{
"keyword": "And ",
"name": "Some other step name",
"line": 46,
"match": {
"location": "some/other/path/to/other/file.rb:2"
},
"result": {
"status": "passed",
"duration": 28043000
}
},
{
"keyword": "Then ",
"name": "Some other step name",
"line": 46,
"match": {
"location": "some/other/path/to/other/file.rb:20"
},
"result": {
"status": "passed",
"duration": 5204000
}
}
],
"after": [
{
"match": {
"location": "some/other/path/to/other/file.rb:91"
},
"result": {
"status": "passed",
"duration": 20000
}
},
{
"match": {
"location": "some/other/path/to/other/file.rb:52"
},
"result": {
"status": "passed",
"duration": 5585000
}
},
{
"match": {
"location": "some/other/path/to/other/file.rb:27"
},
"result": {
"status": "passed",
"duration": 168146000
}
},
{
"match": {
"location": "some/other/path/to/other/file.rb:428"
},
"result": {
"status": "passed",
"duration": 62000
}
}
]
}
]
}
]
其中每个元素上的 elements 可以包含任意数量的对象.这些是黄瓜的测试结果，因此文件A通常比文件B包含更多的元素，因为文件B是文件A中失败测试的重新运行.
where elements can contain any number of objects on either file. These are test results from cucumber so usually file A contains more elements than file B since file B is a re-run of the failed tests in file A.
例如.如果在第一遍中我们进行了100次测试，则文件A elements 数组将包含100个具有上述格式的对象.但是，如果在这100个测试中有50个失败，则文件B elements 数组将包含50个对象.我要做的是用文件B覆盖文件A的 elements 数组，只是添加了在这两者中重复的元素.
For example. If on the first pass we ran 100 tests, file A elements array will contain 100 objects of with the format described above. However if from those 100 tests 50 of them failed, file B elements array will contain 50 objects. What I want to do is to overwrite file A elements array with file B's just adding the elements that repeat in both. Something like
如果文件A有
"elements":[{a:1, b:2, c:3, d:2, e:9, f:4}]
文件B具有
"elements":[{d:5}]
我希望新文件具有
"elements":[{a:1, b:2, c:3, d:5, e:9, f:4}]
到目前为止，我已经拥有
So far I've got
jq '.[].elements' path/to/file/b > path/to/new/file
jq --argfile file path/to/new/file '.[].elements += $file' path/to/file/b
这会将文件B中包含在文件A中的 elements 数组中的 elements 数组中的所有文件放在一起，但不会删除其中的重复对象.
That puts together whatever file B contains in the elements array within the elements array in file A but doesn't remove the duplicated object inside of it.
我尝试使用 unique ，但不知道如何使用它.有什么想法吗?
I tried to use unique but no clue on how to use it. Any ideas?
在这里得到一些答复后，我得到了
After a few responses here I got
jq --argfile b ~/Desktop/cucumber-rerun.json '.[0].elements[4] *= $b[0].elements[0]' ~/Desktop/cucumber.json
可以工作，因为在我的实际示例中，我知道文件A中的元素4是我要用文件B中的1唯一覆盖的元素.但是，这对我不起作用，因为两个文件都是自动生成的并且对象的顺序是未知的.
to works since, in my actual example, I knew the element 4 in file A is the one I want to overwrite with the 1 and only element in file B. However that doesn't work for me since both files are autogenerated and the order of the objects is unknown.
我希望有一个命令，可以将两个文件进行比较，并自动检测A和B中的重复对象，然后用B中的对象覆盖A中的对象
I'd like to have a command that sees both files compared them and autodetects repeated objects from A and B and overwrite those in A with those in B
推荐答案
Here is a solution which uses Object Multiplication. Assuming your data is in A.json and B.json:
$ jq -M --argfile b B.json '.[0].elements[0] *= $b[0].elements[0]' A.json
产生
[
{
"uri": "https://someurl.com",
"id": "some-id",
"keyword": "SomeKeyword",
"name": "Some Name",
"description": "Some description for that test result",
"line": 2,
"tags": [
{
"name": "@sometag",
"line": 1
}
],
"elements": [
{
"a": 5,
"b": 2
}
]
}
]
如果您的数组包含更多数据，则很容易推广这种方法，但是您需要了解如何识别相应的元素.
This approach is easily generalized if your arrays contain more data but you'll need to understand how corresponding elements should be identified.
关于修订后的问题，这是一个过滤器，该过滤器用具有相同 .id的 B.json 的相应对象更新 A.json 的对象.代码>:
Regarding the revised question, here is a filter which updates objects of A.json with corresponding objects of B.json having the same .id:
def INDEX(stream; idx_expr):
reduce stream as $row ({};
.[$row|idx_expr| if type != "string" then tojson else . end] |= $row);
def merge_by_id(a;b):
if b then INDEX(a[];.id) * INDEX(b[];.id) | map(.) else a end;
INDEX($b[];.id) as $i
| map( .elements = merge_by_id(.elements; $i[.id].elements) )
例如，如果上述过滤器位于 filter.jq 中，则 A.json 包含修订后的示例数据，而 B.json 包含
For example if the above filter is in filter.jq, A.json contains the revised sample data and B.json contains
[
{
"id": "safety-tests",
"elements": [
{
"id": "some-element-id",
"description": "updated description"
}
]
}
]
命令
$ jq -M --argfile b B.json -f filter.jq A.json
生成结果
[
{
"uri": "some/url.feature",
"id": "safety-tests", <------ top level .id
...
"elements": [
{
"id": "some-element-id", <------ element .id
"keyword": "Scenario Outline",
"name": ": Some scenario name",
"description": "updated description", <------ updated value
"line": 46,
"type": "scenario",
...
请注意，上述解决方案假定 A.json 中元素的 .id 是唯一的，否则 merge_by_id 不会产生所需的输出.在这种情况下，以下过滤器就足够了:
Note that the above solution assumes the .id of the elements in A.json are unique otherwise merge_by_id won't produce the desired output. In that case the following filter should suffice:
def INDEX(stream; idx_expr):
reduce stream as $row ({};
.[$row|idx_expr| if type != "string" then tojson else . end] |= $row);
(INDEX($b[];.id) | map_values(INDEX(.elements[];.id))) as $i
| map( $i[.id] as $o | if $o then .elements |= map($o[.id]//.) else . end )
此过滤器仅要求 B.json 中的对象的 .id 是唯一的.如果 A.json 和 B.json 中都可能存在非唯一元素，则需要更复杂的映射.
This filter only requires the .id of the objects in B.json to be unique. If it's possible for there to be non-unique elements in both A.json and B.json then a more sophisticated mapping then this one will be required.
以下是带有注释的过滤器版本:
Here is a version of the filter with comments:
def INDEX(stream; idx_expr):
reduce stream as $row ({};
.[$row|idx_expr| if type != "string" then tojson else . end] |= $row);
# first create a lookup table for elements from B.json
( # [{id:x, elements:[{id:y, ...}]}]
INDEX($b[];.id) # -> {x: {id:x, elements:[{id:y, ...}]}..}
| map_values(INDEX(.elements[];.id)) # -> {x: {y: {id:y, ...}}}
) as $i
# update A.json objects
| map( # for each object in A.json
$i[.id] as $o # do we have updated values from B.json ?
| if $o then .elements |= map($o[.id]//.) # if so then replace corresponding elements
else . end # otherwise leave object unchanged
)
这篇关于使用jq用另一个json文件中的对象覆盖json文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
