如何将数据存储区备份转换为JSON(包括地图和数组)? [英] How to convert datastore backup to JSON including maps and arrays?

问题描述

我在我的应用程序中使用了Firestore，我想将整个数据库导出为json.我导出了Firestore数据库(使用gcloud firestore导出)，并将备份下载到计算机上.

I am using firestore in my app and I want to export the whole database as a json. I did an export of my firestore database (with gcloud firestore export) and I downloaded the backup to my computer.

我用于解析output-x文件的代码是

My code for parsing the output-x files is this

import io
import json
import sys

sys.path.append('/Users/riterrani/Downloads/google-cloud-sdk/platform/google_appengine')
from google.appengine.api.files import records
from google.appengine.datastore import entity_pb
from google.appengine.api import datastore

def default(obj):
  """Default JSON serializer."""
  import calendar, datetime

  if isinstance(obj, datetime.datetime):
    if obj.utcoffset() is not None:
      obj = obj - obj.utcoffset()
    millis = int(
      calendar.timegm(obj.timetuple()) * 1000 +
      obj.microsecond / 1000
    )
    return millis
  raise TypeError('Not sure how to serialize %s' % (obj,))


items = []


f = open('data.json', 'w')
for fileIndex in range(0, 8):
  raw = open('output-' + str(fileIndex), 'r')
  reader = records.RecordsReader(raw)
  for record in reader:
    entity_proto = entity_pb.EntityProto(contents=record)
    entity = datastore.Entity.FromPb(entity_proto)
    # print entity
    items.append(entity)
    print "Writing " + str(len(items)) + " items to file"
    f.write(json.dumps(entity, default=default, encoding='latin-1'))
    f.write("\n")


f.close()

该脚本正在运行，但是所有属于Firestore Map的属性的编码均不正确

The script is working, but all the attributes that are firestore Maps have bad encoding

{"environment_changes": ["j\u0004j\u0000r\u0000z\u0014\u001a\u0004date \u0000*\n\u001a\b20191101z.\u001a\u0007changes \u0001*!\u001a\u001fEnvironmentChangeType.new_setupz\u00c1\u0001\b\u0013\u001a\u000benvironment \u0000*\u00ad\u0001\u001a\u00aa\u0001j\u0004j\u0000r\u0000z\u0014\u001a\fexposureTime \u0000*\u0002\b\u0012z&\u001a\u0004type \u0000*\u001c\u001a\u001aEnvironmentTypeEnum.indoorz\u0010\u001a\u0004name \u0000*\u0006\u001a\u0004TenrzO\b\u0013\u001a\u0006lights \u0001*A\u001a?j\u0004j\u0000r\u0000z \u001a\u0004type \u0000*\u0016\u001a\u0014LightingTypeEnum.hpsz\u0012\u001a\u0007wattage \u0000*\u0005\u001a\u0003600\u0082\u0001\u0000\u0082\u0001\u0000\u0082\u0001\u0000", "j\u0004j\u0000r\u0000z\u0014\u001a\u0004date \u0000*\n\u001a\b20191101z0\u001a\u0007changes \u0001*#\u001a!EnvironmentChangeType.name_changez6\u001a\u0007changes \u0001*)\u001a'EnvironmentChangeType.exposition_changez\u00c1\u0001\b\u0013\u001a\u000benvironment \u0000*\u00ad\u0001\u001a\u00aa\u0001j\u0004j\u0000r\u0000z\u0014\u001a\fexposureTime \u0000*\u0002\b\u0018z&\u001a\u0004type \u0000*\u001c\u001a\u001aEnvironmentTypeEnum.indoorz\u0010\u001a\u0004name \u0000*\u0006\u001a\u0004TentzO\b\u0013\u001a\u0006lights \u0001*A\u001a?j\u0004j\u0000r\u0000z \u001a\u0004type \u0000*\u0016\u001a\u0014LightingTypeEnum.hpsz\u0012\u001a\u0007wattage \u0000*\u0005\u001a\u0003600\u0082\u0001\u0000\u0082\u0001\u0000z\u00ca\u0001\b\u0013\u001a\u0014original_environment \u0000*\u00ad\u0001\u001a\u00aa\u0001j\u0004j\u0000r\u0000z\u0014\u001a\fexposureTime \u0000*\u0002\b\u0012z&\u001a\u0004type \u0000*\u001c\u001a\u001aEnvironmentTypeEnum.indoorz\u0010\u001a\u0004name \u0000*\u0006\u001a\u0004TenrzO\b\u0013\u001a\u0006lights \u0001*A\u001a?j\u0004j\u0000r\u0000z \u001a\u0004type \u0000*\u0016\u001a\u0014LightingTypeEnum.hpsz\u0012\u001a\u0007wattage \u0000*\u0005\u001a\u0003600\u0082\u0001\u0000\u0082\u0001\u0000\u0082\u0001\u0000", "j\u0004j\u0000r\u0000z\u0014\u001a\u0004date \u0000*\n\u001a\b20191117z6\u001a\u0007changes \u0001*)\u001a'EnvironmentChangeType.exposition_changez\u00c1\u0001\b\u0013\u001a\u000benvironment \u0000*\u00ad\u0001\u001a\u00aa\u0001j\u0004j\u0000r\u0000z\u0014\u001a\fexposureTime \u0000*\u0002\b\u0012z&\u001a\u0004type \u0000*\u001c\u001a\u001aEnvironmentTypeEnum.indoorz\u0010\u001a\u0004name \u0000*\u0006\u001a\u0004TentzO\b\u0013\u001a\u0006lights \u0001*A\u001a?j\u0004j\u0000r\u0000z \u001a\u0004type \u0000*\u0016\u001a\u0014LightingTypeEnum.hpsz\u0012\u001a\u0007wattage \u0000*\u0005\u001a\u0003600\u0082\u0001\u0000\u0082\u0001\u0000z\u00ca\u0001\b\u0013\u001a\u0014original_environment \u0000*\u00ad\u0001\u001a\u00aa\u0001j\u0004j\u0000r\u0000z\u0014\u001a\fexposureTime \u0000*\u0002\b\u0018z&\u001a\u0004type \u0000*\u001c\u001a\u001aEnvironmentTypeEnum.indoorz\u0010\u001a\u0004name \u0000*\u0006\u001a\u0004TentzO\b\u0013\u001a\u0006lights \u0001*A\u001a?j\u0004j\u0000r\u0000z \u001a\u0004type \u0000*\u0016\u001a\u0014LightingTypeEnum.hpsz\u0012\u001a\u0007wattage \u0000*\u0005\u001a\u0003600\u0082\u0001\u0000\u0082\u0001\u0000\u0082\u0001\u0000"], "lights": ["j\u0004j\u0000r\u0000z \u001a\u0004type \u0000*\u0016\u001a\u0014LightingTypeEnum.hpsz\u0012\u001a\u0007wattage \u0000*\u0005\u001a\u0003600\u0082\u0001\u0000"],}

我如何才能解码为具有JSON?

How I can decode this to have a JSON?

我不是python开发人员，我从获得了代码这里

I am not a python dev, I got the code from here

推荐答案

这不是编码问题.

似乎您的嵌套对象和列表仍为LevelDB格式，您可以执行递归函数来解析实体的每个级别.

It seems that your nested objects and lists are still in LevelDB format, you can do a recursive function to parse every level of your entities.

这篇关于如何将数据存储区备份转换为JSON(包括地图和数组)?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！