在JSON对象中存储和发送原始文件数据 [英] Storing and sending raw file data within a JSON object
问题描述
我正在寻找一种方法来传输任何文件类型的原始文件数据与任何可能的内容（我的意思是文件和文件内容都是用户生成的）两种方式在Backbone中使用xhr / ajax调用对Django后端的前端。
编辑：也许问题仍然不清楚......
如果在IDE（例如Sublime）中打开文件，则可以查看和编辑包含该文件的实际代码。我正在尝试将 THAT 原始内容放入JSON中，以便我可以发送到浏览器，它可以被修改，然后发回。
<我发布了这个问题，因为我的印象是因为这些文件的内容可以有效地使用任何编码语言，只是字符串化内容并发送似乎就像一个易于破解或利用的脆弱解决方案。内容可以包含任意数量的
'， {和} 似乎会破坏JSON格式的字符，并且转义这些字符会在代码中留下有效破坏它们的工件（不是吗？）。 如果这个假设是错误的，这也是一个可以接受的答案（只要你能指出我忽略的是什么）。
我正在开发的项目是一个基于浏览器的IDE，它将从服务器接收完整的文件结构。用户可以添加/删除文件，编辑这些文件的内容，然后将其更改保存回服务器。发送/接收都必须通过ajax / xhr调用处理。
- 在Backbone中，每个文件都被实例化为模型并存储在Collection中。文件的内容将作为属性存储在模型中。
- 理想情况下，文件内容仍然可靠地抛出所有应用程序在进行更改时进行ropriate事件。
- 不应将提取内容分解为与文件模型其余部分的单独调用。我想使用单个save / fetch调用来发送/接收包含原始内容的文件。
需要Underscore的解决方案/ jQuery很好，如果有专门用于管理原始文件数据的东西，我可以引入额外的库。
有趣的问题。执行此操作所需的代码将非常复杂，对不起，我不提供示例，但您似乎是一个不错的程序员，应该能够实现下面提到的内容。
关于通过JSON发送原始数据，你需要做的就是使JSON安全而不破坏你的代码是通过使用Python的json.dumps& amp;中的stringyfying来逃避特殊字符。 JavaScript的JSON.stringyfy。 [1]
如果您担心某种形式的基本防篡改，那么您的数据的光编码将符合目的，除了让客户端和服务器使用JSON传输来回传递每会话令牌，以确保JSON不是来自恶意地址。
如果要检查数据的端到端完整性，请生成md5校验和并将其发送到JSON中然后在到达时生成另一个md5并与JSON中的一个进行比较。
Base64 编码：您的数据大小将增长33％因为它编码四个字符来表示三个字节的数据。
Base85 ：将四个字节编码为五个字符，并将数据增加25％ ，但在Python中使用比Base64更多的处理开销。这是数据大小的8％，但是以处理开销为代价。此外，它不是字符串安全的双和&单引号，尖括号和&符号不能在JSON内部使用，因为它使用所有95个可打印的ASCII字符。在JSON传输之前需要进行stringyfied。 [2]
yEnc 只有2-3％的开销（取决于数据中相同字节的频率），但是由不切实际的缺陷排除（见[3]）。
ZeroMQ Base-85 ，又名 Z85 。它是Base85的字符串安全变体，数据开销为25％，优于Base64。将其粘贴到JSON中不需要使用stringyfying。我强烈推荐这种编码算法。 [4] [5] [6]
如果你只发送小文件（比如几KB），那么二进制到文本转换的开销将会可以接受。对于大到几Mbs的文件，将它们增长25-33％可能是不可接受的。在这种情况下，您可以尝试在发送之前压缩它们。 [7]
您也可以使用multipart / form-data将数据发送到服务器，但我看不出它是如何双向工作的。
UPDATE
总之，这是我的解决方案的算法：
发送数据
-
生成会话令牌并将其存储为
登录（服务器）时的关联用户，或从会话cookie（客户端）检索 -
为数据生成MD5哈希以进行完整性检查使用Z85对原始数据进行编码，以添加一些基本的防篡改和JSON友好性。
-
将上述内容置于JSON内并在请求时发送POST。
接收
-
从POST中抓取JSON
-
从存储中检索关联用户（服务器）的会话令牌，或从会话cookie（客户端）检索。
-
为收到的数据生成MD5哈希，并在收到的JSON中对MD5进行测试，拒绝或有条件接受。
-
Z85-解码接收到的JSON中的数据，以获取原始数据，并根据需要存储在文件或数据库（服务器）或GUI / IDE（客户端）中的处理/显示。
参考文献
[2] JSON字符串中的二进制数据。比Base64更好的东西
[3] https://en.wikipedia.org/wiki/YEnc
[4] http://rfc.zeromq.org/spec:32
[5] Z85实施在C / C ++中 https://github.com/artemkin/z85
[6] Z85 Python实现 https://gist.github.com / minrk / 6357188
[7] JavaScript zip库 http://stuk.github.io/jszip/
[8] JavaScript Gzip SO Gzip的JavaScript实现
I'm looking for a way to transfer the raw file data of any file-type with any possible content (By that I mean files and file-content are all user generated) both ways using xhr/ajax calls in a Backbone front-end against a Django back-end.
EDIT: Maybe the question is still unclear...
If you open a file in an IDE (such as Sublime), you can view and edit the actual code that comprises that file. I'm trying to put THAT raw content into a JSON so I can send to the browser, it can be modified, and then sent back.
I posted this question because I was under the impression that because the contents of these files can effectively be in ANY coding language that just stringify-ing the contents and sending it seems like a brittle solution that would be easy to break or exploit. Content could contain any number of ', ", { and } chars that would seem to break JSON formatting, and escaping those characters would leave artifacts within the code that would effectively break them (wouldn't it?).
If that assumption is wrong, THAT would also be an acceptable answer (so long as you could point out whatever it is I'm overlooking).
The project I'm working on is a browser-based IDE that will receive a complete file-structure from the server. Users can add/remove files, edit the content of those files, then save their changes back to the server. The sending/receiving all has to be handled via ajax/xhr calls.
- Within Backbone, each "file" is instantiated as a model and stored in a Collection. The contents of the file would be stored as an attribute on the model.
- Ideally, file content would still reliably throw all the appropriate events when changes are made.
- Fetching contents should not be broken out into a separate call from the rest of the file model. I'd like to just use a single save/fetch call for sending/receiving files including the raw content.
Solutions that require Underscore/jQuery are fine, and I am able to bring in additional libraries if there is something available that specializes in managing that raw file data.
Interesting question. The code required to implement this would be quite involved, sorry that I'm not providing examples, but you seem like a decent programmer and should be able to implement what's mentioned below.
Regarding the sending of raw data through JSON, all you would need to do to make it JSON-safe and not break your code is to escape the special characters by stringyfying using Python's json.dumps & JavaScript's JSON.stringyfy. [1]
If you are concerned about some form of basic tamper-proofing, then light encoding of your data will fit the purpose, in addition to having the client and server pass a per-session token back and forth with JSON transfers to ensure that the JSON isn't forged from a malicious address.
If you want to check the end-to-end integrity of the data, then generate an md5 checksum and send it inside your JSON and then generate another md5 on arrival and compare with the one inside your JSON.
Base64 encoding: The size of your data would grow by 33% as it encodes four characters to represent three bytes of data.
Base85: Encodes four bytes as five characters and will grow your data by 25%, but uses much more processing overhead than Base64 in Python. That's a 8% improvement in data size, but at the expense of processing overhead. Also it's not string safe as double & single quotation marks, angle brackets, and ampersands cannot be used unescaped inside JSON, as it uses all 95 printable ASCII characters. Needs to be stringyfied before JSON transport. [2]
yEnc has as little as 2-3% overhead (depending on the frequency of identical bytes in the data), but is ruled out by impractical flaws (see [3]).
ZeroMQ Base-85, aka Z85. It's a string-safe variant of Base85, with a data overhead of 25%, which is better than Base64. No stringyfying necessary for sticking it into JSON. I highly recommended this encoding algorithm. [4] [5] [6]
If you're sending only small files (say a few KB), then the overhead of binary-to-text conversion will be acceptable. With files as large as a few Mbs, it might not be acceptable to have them grow by 25-33%. In this case you can try to compress them before sending. [7]
You can also send data to the server using multipart/form-data, but I can't see how this will work bi-directionally.
UPDATE
In conclusion, here's my solution's algorithm:
Sending data
Generate a session token and store it for the associated user upon login (server), or retrieve from the session cookie (client)
Generate MD5 hash for the data for integrity checking during transport.
Encode the raw data with Z85 to add some basic tamper-proofing and JSON-friendliness.
Place the above inside a JSON and send POST when requested.
Reception
Grab JSON from POST
Retrieve session token from storage for the associated user (server), or retrieve from the session cookie (client).
Generate MD5 hash for the received data and test against MD5 in received JSON, reject or accept conditionally.
Z85-decode the data in received JSON to get raw data and store in file or DB (server) or process/display in GUI/IDE (client) as required.
References
[1] How to escape special characters in building a JSON string?
[2] Binary Data in JSON String. Something better than Base64
[3] https://en.wikipedia.org/wiki/YEnc
[4] http://rfc.zeromq.org/spec:32
[5] Z85 implementation in C/C++ https://github.com/artemkin/z85
[6] Z85 Python implementation of https://gist.github.com/minrk/6357188
[7] JavaScript zip library http://stuk.github.io/jszip/
[8] JavaScript Gzip SO JavaScript implementation of Gzip
这篇关于在JSON对象中存储和发送原始文件数据的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
