使用Akka Stream流式传输巨大的json [英] Streaming huge json with Akka Stream

问题描述

我有一个用json平板产生大量http响应的问题，其中只有一部分是兴趣点. 我无法更改响应结构. 这是一个例子

I have a problem of huge http response with a json slab, where only portion is point of interest. I cannot change the response structure. here is an example

{
  "searchString": "search",
  "redirectUrl": "",
  "0": {
    "numRecords": 123,
    "refinementViewModelCollector": {},
//    Lots of data here
    "results": [
      {
        "productCode": "123",
        "productShortDescription": "Desc",
        "brand": "Brand",
        "productReview": {
          "reviewScore": 0
        },
        "priceView": {
          "salePriceDisplayable": false,
        },
        "productImageUrl": "url",
        "alternateImageUrls": [
          "url1"
        ],
        "largeProductImageUrl": "url4",
        "videoUrl": ""
      },
      {
        "productCode": "124",
        "productShortDescription": "Desc",
        "brand": "Brand",
        "productReview": {
          "reviewScore": 0
         },
        "priceView": {
          "salePriceDisplayable": false,
        },
        "preOrder": false,
        "productImageUrl": "url",
        "alternateImageUrls": [
          "url1"
        ],
        "largeProductImageUrl": "url4",
        "videoUrl": ""
      }
    ]
    //lots of data here
  }
}

我的兴趣点是results Jason Array中的条目，但它们位于json的中间

My point of interest is entries in results Jason Array, but the are sitting in the middle of json

我这样创建了一个小型的Play WS Client:

I created a small Play WS Client like this:

val wsClient: WSClient = ???
val ret = wsClient.url("url").stream()
ret.flatMap { response =>
  response.body.via(JsonFraming.objectScanner(1024))
    .map(_.utf8String)
    .runWith(Sink.foreach(println))
}

这将不起作用，因为它将整个json平板作为Json对象.我需要跳过一些数据，直到"results":条目出现在流中，然后开始解析条目并跳过所有其余的内容. 任何想法如何做到这一点?

this will not work because it will take whole json slab as Json object. I need to skip some data until "results": entry appear in the stream, then start parsing entries and skip all the rest. Any ideas how to do this?

推荐答案

有些解析器支持将其解析为流.举一个很好的例子，看看这个Circe例子 https://github .com/circe/circe/tree/master/examples/sf-city-lots

There are parsers that support parsing as a stream. For a good example check out this Circe example https://github.com/circe/circe/tree/master/examples/sf-city-lots

这篇关于使用Akka Stream流式传输巨大的json的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！