读取/写入NetworkX图形对象 [英] Read/Write NetworkX Graph Object

问题描述

我正在尝试处理具有数亿个节点的超大规模NetworkX Graph对象.我希望能够将其写入文件，以免消耗我所有的计算机内存.但是，我需要不断搜索现有节点，更新边缘等.

I am trying to deal with a super-massive NetworkX Graph object with hundreds of millions of nodes. I'd like to be able to write it to file as to not consume all my computer memory. However, I need to constantly be searching across existing nodes, updating edges, etc.

是否有一个好的解决方案?我不确定它如何与 http://networkx.lanl.gov/reference提供的任何文件格式一起使用/readwrite.html

Is there a good solution for this? I'm not sure how it would work with any of the file formats provided on http://networkx.lanl.gov/reference/readwrite.html

我能想到的唯一解决方案是将每个节点存储为单独的文件，并引用文件系统中的其他节点-这样，打开一个节点进行检查不会使内存过载.是否存在现有的文件系统来处理大量数据(例如PyTables)而无需编写自己的样板代码?

The only solution i can think of is to store each node as a separate file with references to other nodes in the filesystem - that way, opening one node for examination doesn't overload the memory. Is there an existing filesystem for large amounts of data (e.g. PyTables) to do this without writing my own boilerplate code?

推荐答案

如果已将其构建为NetworkX图，则该图已经在内存中.对于这么大的图表，我猜想您将不得不执行与建议的单独文件类似的操作.但是，不是使用单独的文件，而是使用数据库来存储每个节点以及节点之间的多对多连接.换句话说，您将拥有一个节点表和一个边表，然后要查询特定节点的邻居，您只需查询两端都具有该特定节点的任何边.这应该很快，尽管我不确定您是否能够在不首先在内存中构建整个网络的情况下利用NetworkX的分析功能.

If you've built this as a NetworkX graph, then it will already be in memory. For this large of a graph, my guess is you'll have to do something similar to what you suggested with separate files. But, instead of using separate files, I'd use a database to store each node with many-to-many connections between nodes. In other words you'd have a table of nodes, and a table of edges, then to query for the neighbors of a particular node you could just query for any edges that have that particular node on either end. This should be fast, though I'm not sure if you'll be able to take advantage of NetworkX's analysis functions without first building the whole network in memory.

这篇关于读取/写入NetworkX图形对象的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！