动态创建模型字段的替代方法 [英] Alternatives to dynamically creating model fields

问题描述

我正在尝试构建一个Web应用程序，用户可以在其中上传文件（特别是 MDF 文件格式）并以各种图表的形式查看数据。这些文件可以包含任意数量的基于时间的信号（各种数字数据类型），用户可以随意命名信号。

I'm trying to build a web application where users can upload a file (specifically the MDF file format) and view the data in forms of various charts. The files can contain any number of time based signals (various numeric data types) and users may name the signals wildly.

我保存数据的想法涉及两个步骤：

My thought on saving the data involves 2 steps:

1. 维护 主表 作为索引，以将此类元信息保存为文件名称，上传者，时间等。每次上传新文件时都会添加记录（行）。


2. 创建新表格（我将其称为 数据表 ），对于每个上传的文件，该表中的每一列将是一个信号（第一列为时间戳）。

1. Maintain a master table as an index, to save such meta information as file names, who uploaded it, when, etc. Records (rows) are added each time a new file is uploaded.
2. Create a new table (I'll refer to this as data tables) for each file uploaded, within the table each column will be one signal (first column being timestamps).

这带来的问题是我无法为数据表预定义 Model ，因为字段的数量，名称和数据类型在几乎所有上传的文件中都会有所不同。

This brings the problem that I can't pre-define the Model for the data tables because the number, name, and datatype of the fields will differ among virtually all uploaded files.

我知道一些有助于构建运行时动态模型的库，但是它们都是过时的，基本上都对它们有疑问得到零答案。因此，尽管付出了很多努力，但我什至不确定自己的方法是完成自己想做的事情的最佳方法。

I'm aware of some libs that help to build runtime dynamic models but they're all dated and questions about them on SO basically get zero answers. So despite the effort to make it work, I'm not even sure my approach is the optimal way to do what I want to do.

我也遇到了此 Postgres特定模型字段，可以嵌套数组（我认为适合基于2D时间的信号列表）。从理论上讲，我可以解析原始的上传文件并构造一个数组，并将所有数据基本上保存在一个字段中。不知道数据大小的限制，这对于以后的查询也可能是一场噩梦，因为创建图表通常一次只占用几列信号，而总共要接收多达数百个信号。

I also came across this Postgres specifc model field which can take nested arrays (which I believe fits the 2-D time based signals lists). In theory I could parse the raw uploaded file and construct such an array and basically save all the data in one field. Not knowing the limit of size of data, this could also be a nightmare for the queries later on, since to create the charts it usually takes only a few columns of signals at a time, compared to a total of up to hundreds of signals.

所以我的问题是：

是否有更好的方法来组织数据存储？以及如何？

Is there a better way to organize the storage of data? And how?

任何见识都会受到赞赏！

Any insight is greatly appreciated!

推荐答案

如果每个用户的字段名称，编号和数据类型将有所不同，因此您不需要ORM。您需要的是查询生成器或SQL字符串组成，例如 Psycopg 。您将以编程方式为用户和上传文件的每种组合（如果它们不同）创建一个表，并以编程方式插入记录。

If the name, number and datatypes of the fields will differ for each user, then you do not need an ORM. What you need is a query builder or SQL string composition like Psycopg. You will be programatically creating a table for each combination of user and uploaded file (if they are different) and programtically inserting the records.

使用postgresql可能是一个不错的选择，您也可以在阵列上创建GIN索引以加快查询速度。

Using postgresql might be a good choice, you might also create a GIN index on the arrays to speed up queries.

但是，如果您主要使用时间序列数据，则可以使用时序数据库，Prometheus更具意义。

However, if you are primarily working with time-series data, then using a time-series database like InfluxDB, Prometheus makes more sense.

这篇关于动态创建模型字段的替代方法的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！