无法使用parLapply打开连接错误 [英] cannot open the connection error with parLapply
问题描述
我使用parLapply在4核计算机上成功处理了一些数据,如下所示:
I successfully processed some data on a 4 core computer using parLapply, using a code like below:
require("parallel")
setwd("C:/...")
file_summary<-read.table("file_summary",header=T,sep=" ",stringsAsFactors=F)
new_filename<-sub(".asc",".dat",file_summary$filename)
file_list<-list.files()
myfunction <- function(k) {
x<-M$x[k]
y<-M$y[k]
for (i in 1:length(file_summary[,1])) {
if ( # logical condition on x and y ) {
new_file<-new_filename[i]
new_data<-read.table(new_file,header=T,sep=" ")
eval<-matrix(,nrow=length(new_data$x),ncol=1)
for (j in 1:length(new_data$x)) {
eval[j]<-(new_data$x[j]-x)^2+(new_data$y[j]-y)^2
}
index<-which(eval == max(eval))
out<-c(new_data$x[index],new_data$y[index],new_data$mean[index],new_data$S[index])
}
rm(eval)
gc()
}
return(out)
}
n_tasks <- length(M$x)
n_cores <- 8
Cl = makeCluster(n_cores, type = "PSOCK")
clusterExport(Cl, "M")
clusterExport(Cl, "file_summary")
clusterExport(Cl, "new_filename")
clusterExport(Cl, "file_list")
Results <- parLapply(Cl, c(1:n_tasks), myfunction)
stopCluster(Cl)
,现在使用完全相同的代码,相同的数据和目录结构(即路径),我试图在8核计算机上运行分析以进一步加快分析速度.但是,在我第一次尝试时,出现以下错误:
and now using exactly the same code and the same data and directory structure (ie. paths), I am trying to run the analysis on a 8 core machine to speed it up further. However, at my first attempt I get the following error:
8 nodes produced errors; first error: cannot open the connection
我试图清除RAM(对于非R进程),看是否有帮助,但没有.有什么建议吗?
I tried to clear the RAM a bit (for non R processes) to see if it helped, but it didnt. Any suggestions?
推荐答案
我在"myfunction"中看到的唯一可以生成无法打开连接"错误的操作是"read.table".您可能需要在调用read.table之前添加以下权限:
The only operation that I see in "myfunction" that can generate a "cannot open the connection" error is "read.table". You might want to add the following right before calling read.table:
if (! file.exists(new_file))
stop(paste(new_file, "does not exist"))
检查工作人员是否具有读取文件的权限可能也很有用:
It might also be useful to check that the worker has permission to read the file:
if (file.access(new_file, mode=4) == -1)
stop(paste("no read permission on", new_file))
排除这些问题似乎很值得.
It seems worthwhile to rule out these problems.
这篇关于无法使用parLapply打开连接错误的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!