R spdep巨型权重矩阵 [英] R spdep giant weight matrix
问题描述
我是空间统计的新手,我正在尝试为R中美国所有人口普查区域创建一个空间权重矩阵.大约有74000个区域.
I'm new to spatial statistics, and I'm trying to create a spatial weight matrix for all Census tracts in the US in R. There are around 74000 tracts.
基于美国人口普查老虎文件,我创建了所有区域的shapefile,然后执行了此操作(使用spdep
包):
Based on US Census Tiger Files, I created a shapefile of all tracts, and then did (using the spdep
package):
#Create adjacency matrix
am = poly2nb(us)
is.symmetric.nb(am)
这很好,尽管我很大.
下一步:
am = nb2mat(am, style="B",zero.policy=T)
哪个给我这个错误:
Error: cannot allocate vector of size 40.9 Gb
很明显,我的笔记本电脑无法处理40.9 Gb的内存.我尝试在AWS EC2云上执行此操作,但是要获得那么多的内存,我需要避免一个非常大的实例,因为我是云计算领域的新手,所以我想使用免费的T2.微型沙箱(最大可容纳1 GiB的内存),直到我准备花钱在更大的机器上为止.如果我可以将权重矩阵转换为稀疏矩阵,我想我可以处理它,但是我不知道该怎么做.我试图做这样的事情:
Obviously my laptop cannot handle 40.9 Gb of memory. I tried doing this on AWS EC2 cloud, but to get that much memory I'd need to get a very large instance which I'd like to avoid since I'm totally new at cloud computing and would rather play in the free T2.micro sandbox (max up to 1 GiB of memory) until I'm ready to spend some cash on a bigger machine. If I could turn the weight matrix into a sparse matrix I think I'd be able to handle it, but I don't know how to do that. I tried doing something like this:
Wmat<-Matrix(nb2mat(am, style="B",zero.policy=T),sparse=TRUE)
但是在创建稀疏矩阵之前,它仍然需要所有内存来执行nb2mat命令.
But it still needs all the memory to do the nb2mat command before creating the sparse matrix.
有解决方案吗?
推荐答案
确定有点晚了.但是我想我只是想出了一个解决方案.我有一个71k * 71k矩阵的类似情况.
Sure it's a bit late. But I think I just figured out a solution. I have a similar situation with a 71k*71k matrix.
我刚刚修改了nb2mat函数,以使用bigmemory库中的big.matrix.我们需要定义两个新功能:
I just reworked the nb2mat function to use big.matrix from the bigmemory library. We need to define two new functions:
my_nb2mat = function (neighbours, glist = NULL, style = "W", zero.policy = NULL)
{
if (is.null(zero.policy))
zero.policy <- get("zeroPolicy", envir = .spdepOptions)
stopifnot(is.logical(zero.policy))
if (!inherits(neighbours, "nb"))
stop("Not a neighbours list")
listw <- nb2listw(neighbours, glist = glist, style = style,
zero.policy = zero.policy)
res <- my_listw2mat(listw)
attr(res, "call") <- match.call()
res
}
my_listw2mat = function (listw)
{
require(bigmemory)
n <- length(listw$neighbours)
if (n < 1)
stop("non-positive number of entities")
cardnb <- card(listw$neighbours)
if (any(is.na(unlist(listw$weights))))
stop("NAs in general weights list")
#res <- matrix(0, nrow = n, ncol = n)
res <- big.matrix(n, n, type='double', init=NULL)
options(bigmemory.allow.dimnames=TRUE)
for (i in 1:n) if (cardnb[i] > 0)
res[i, listw$neighbours[[i]]] <- listw$weights[[i]]
if (!is.null(attr(listw, "region.id")))
row.names(res) <- attr(listw, "region.id")
res
}
在此处调用新的my_nb2mat函数:
Call the new my_nb2mat function here:
a=my_nb2mat(neighbours = out, style='W',zero.policy =F )
注意:对我来说,bigmemory库似乎只能在R \ R-2.15.3中工作.
Note: bigmemory library only seems to work in R\R-2.15.3 for me.
这篇关于R spdep巨型权重矩阵的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!