Powershell FTP发送大文件System.OutOfMemoryException [英] Powershell FTP Send large file System.OutOfMemoryException

查看：795
本文介绍了Powershell FTP发送大文件System.OutOfMemoryException的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！

问题描述

我在这里有一个基于脚本的脚本：使用PowerShell上传FTP文件



对于小文件，这一切都可以正常工作，但我试图用它来创建我们用于将访问mdb文件导出到客户端的过程，

我的第一个测试涉及一个10MB的文件，我在Get-Content阶段遇到了一个System.OutOfMemoryException



powershell ISE在获取尝试期间运行接近2GIG。



下面是一个完整的脚本示例（非常温和，我对它很新颖）： 

  ##### 
＃控制脚本的用户变量
 #### ＃
 
＃重新尝试连接多少次
 $ connectionTries = 5 
＃秒之间的时间间隔
 $ connectionTryInterval = 300 
＃在哪里记录输出
 $ logFile =D：\MyPath\ftplog.txt
＃在归档前最大日志文件大小（以KB为单位）
 $ logFileMaxSize = 500 
 
＃格式化日期部分，用于传输特定文件
＃这附加到文件名库。以替代
 $ datePart =
＃文件名的基本部分
 $ fileNameBase =Myfile
＃文件扩展名
 $ fileExtension = .mdb
＃源文件的位置（请包括尾部反斜杠）
 $ sourceLocation =D：\MyPath\
 
＃位置和证书目标ftp服务器
 $ userName =iamafish
 $ password =ihavenofingers
 $ ftpServer =10.0.1.100
 
 ###### 
＃主要脚本
 ##### 
 
＃如果有一个日志文件，并且它比声明的限制长，那么将它存档为当前时间戳
 if （test-path $ logfile）
 {
 if（$（（get-item $ logFile）.Length / 1kb）-gt $ logFileMaxSize）
 {
 write-host $ （归档日志到ftplog_+（get-date -format yyyyMMddhhmmss）+.txt）
重命名项目$ logFile $（ftplog_+（get-date -format yyyyMMddhhmmss）+.txt ）
} 
} 
 
＃启动新的日志条目
＃添加内容$ logFile_______________ ____________________________________________
＃write-host $ logEntry 
 
 #contruct源文件和目标uri 
 $文件名= $ fileNameBase + $ datePart + $ fileExtension 
 $ sourceFile = $ sourceLocation + $ fileName 
 $ sourceuri =ftp：//+ $ ftpServer +/+ $ fileName 
 
 
＃创建一个FTPWebRequest对象来处理与ftp服务器
 $ ftprequest = [System.Net.FtpWebRequest] :: create（$ sourceuri）
 
＃为请求的认证连接设置网络凭证
 $ ftprequest.Credentials = New-Object System.Net.NetworkCredential（$ username，$ password）
 
 $ ftprequest.Method = [System.Net.WebRequestMethods + Ftp] :: UploadFile 
 $ ftprequest.UseBinary = $ true 
 $ ftprequest.KeepAlive = $ false 
 
 $ succeeded = $ true 
 $ errorMessage =
 
＃在文件中读入以上传一个字节数组
 trap [exception] {
 $ script：succeeded = $ false 
 $ script：errorMessage = $ _。Exception.Message 
 Add-Content $ logFile $（（get-Date -formatyyyy-MM-dd hh：mm：ss）+| 1 |+ $ _。Exception.Message）
＃write-host $ logEntry $ b $＃write-host $（TRAPPED：+ $ _。Exception.GetType（）。FullName）
＃write-host $（TRAPPED：+ $ _。Exception.Message） 
出口
} 
＃-ea 1强制错误被捕获
 $ content = gc -en byte $ sourceFile -ea 1 
 
 
 $ try = 0 
 
 do {
 trap [System.Net.WebException] {
 $ script：succeeded = $ false 
 $ script：errorMessage = $ _。Exception.Message 
 Add-Content $ logFile $（（get-Date -formatyyyy-MM-dd hh：mm：ss）+| 1 |+ $ _。Exception.Message） 
＃write-host $ logEntry 
＃write-host $（TRAPPED：+ $ _。Exception.GetType（）。FullName）
 $ script：try ++ 
 start- sleep（$ try -le $ connectionTries）-a $ connectionBryInterval 
 continue 
} 
 $ ftpresponse = $ ftprequest.GetResponse（）
 
} -a nd（-not $ succeeded））
 
 if（$ succeeded）{
 
 Add-Content $ logFile $（（get-Date -formatyyyy-MM-dd hh ：mm：ss）+| 0 |+Starting file transfer。）
＃获取请求流并将字节写入它
 $ rs = $ ftprequest.GetRequestStream（）$ $ b $ rs.Write（$ content，0，$ content.Length）
＃请务必自行清理
 $ rs.Close（）
 $ rs.Dispose（） 
 $ content.Close（）
 $ content.Dispose（）
 Add-Content $ logFile $（（get-Date -formatyyyy-MM-dd hh：mm：ss） +| 0 |+Transfer complete。）
＃write-host $ logEntry 
}

我不能将代码放在注释中，所以感谢来自keith的指针我将文件acces位向下移动到了底部，以便像这样将其链接到另一个。

  trap [例外] {
 $ script：succeeded = $ false 
 $ script：errorMessage = $ _。Exception.Message 
 Add-Content $ logFile $（（get-Date -formatyyyy-MM-d （）
 $ sourceStream.Close（）
 $ sourceStream.Dispose（）
 ＃write-host $（（get-Date -formatyyyy-MM-dd hh：mm：ss）+| 1 |试图打开文件|+ $ _。Exception.Message）
 #write -$ $（TRAPPED：+ $ _。Exception.GetType（）。FullName）
 exit 
 
 $ sourceStream = New-Object IO.FileStream（$（New-Object System .IO.FileInfo $ sourceFile），[IO.FileMode] :: Open）
 [byte []] $ readbuffer = New-Object byte [] 1024 
 
＃获取请求流，并写入字节
 $ rs = $ ftprequest.GetRequestStream（）
 do {
 $ readlength = $ sourceStream.Read（$ readbuffer，0,1024）
 $ rs .write（$ readbuffer，0，$ readlength）
} while（$ readlength -ne 0）

我只需要弄清楚为什么我会得到：使用0参数调用GetResponse的异常：无法访问已处理的对象。每隔一次运行
它。这是在ISE中运行它的一个怪癖吗，还是我在做初始声明或最终处置时做了一些严重错误？



完成后，我会发布完整的最终脚本，因为我认为它会生成一个非常坚固的ftp导出示例，包含错误陷印和日志记录。





好的，这里是完整的脚本。 Dispose被编辑出来，但有或没有它在5分钟内运行该脚本将要么给我一个消息，我不能使用一个处理对象或告诉我，getResponse（）产生了一个错误（226）文件传输（在ISE中运行） 。虽然这在正常操作中不会成为问题，但我想正确地登录FTP会话并清理脚本末尾的资源，并确保根据需要正确声明它们。

  ##### 
＃控制脚本的用户变量
 ##### 
 
＃多次连接将被重新尝试
 $ connectionTries = 5 
＃以秒为单位的时间间隔
 $ connectionTryInterval = 1 
＃在哪里记录输出
 $ logFile =D：\MyPath\ftplog.txt
＃归档前的最大日志文件大小（以KB为单位）
 $ logFileMaxSize = 500 
 #log到文件或控制台 - ＃true =登录到文件#false =登录到控制台
 $ logToFile = $ false 
 
＃格式化日期部分用于传输特定文件
＃这是附加到文件名的基础。以代替
 $ datePart =
＃文件名的基本部分
 $ fileNameBase =MyFile
＃文件扩展名
 $ fileExtension = .mdb
＃源文件的位置（请包括尾部反斜杠）
 $ sourceLocation =D：\MyPath\
 
＃位置和证书目标ftp服务器
 $ userName =iamafish
 $ password =ihavenofingers
 $ ftpServer =10.0.1.100
 
 ###### 
＃主脚本
 ##### 
 
函数logEntry（$ entryType，$ section，$ message）
 {
＃只需创建一个用于记录到控制台以进行测试
＃$ entryType：0 =成功，1 =错误
＃$部分：脚本日志条目从
＃$ message生成的部分：日志消息
 
＃这是管道分离以适应我的标准MSSQL链接平面文件架构，以便于查询
 $ logString =$（get-Date -formatyyyy-MM- dd hh：mm：ss）| $ entryType | $ section | $ message
 
 if（$ script：logtoFile）
 {
 Add-Content $ logFile $ logString 
} 
 else 
 {
 write-host $ logString $ b $如果有一个日志文件，并且它比声明的限制长，那么将它存档为当前时间戳
 if（test-path $ logfile）$（$）
 
 
 $ b b $ b {
 if（$（（get-item $ logFile）.Length / 1kb）-gt $ logFileMaxSize）
 {
 write-host $（归档日志到ftplog_+ （get-date -format yyyyMMddhhmmss）+.txt）
 rename-item $ logFile $（ftplog_+（get-date -format yyyyMMddhhmmss）+.txt）
 New-Item $ logFile -type file 
} 
} 
 else 
 {
 New-Item $ logFile -type file 
} 
 
 
 #contruct源文件和目标uri 
 $ fileName = $ fileNameBase + $ datePart + $ fileExtension 
 $ sourceFile = $ sourceLocation + $文件名
 $ destination =ftp：// + $ ftpServer +/+ $ fileName 
 
 
＃检查源文件是否存在
 if（（test-path $ sourceFile）-eq $ false）
 {
 logEntry 1Check Source File$（File not found：+ $ sourceFile）
 Exit 
} 
 
 
＃创建一个FTPWebRequest对象来处理到ftp服务器的连接
 $ ftpRequest = [System.Net.FtpWebRequest] :: create（$ destination）
 
＃设置请求的认证连接的网络凭证
 $ ftpRequest.Credentials = New-Object System.Net.NetworkCredential（$ username，$ password）
 $ ftpRequest.Method = [ System.Net.WebRequestMethods + Ftp] :: UploadFile 
 $ ftpRequest.UseBinary = $ true 
 $ ftpRequest.KeepAlive = $ false 
 
 $ succeeded = $ true 
 $ try = 1 
 $ b $ {
 trap [例外] {
 $ script：succeeded = $ false 
 logEntry 1检查FTP连接$ _。异常。 Message 
 $ script：try ++ 
 start-sleep -s $ connectionTryInterval 
 continue 
} 
 $ ftpResponse = $ ftpRequest.GetResponse（）
 
} （$ try -le $ connectionTries）和（-not $ succeeded））
 $ b $ if（$成功）{
 logEntry 0连接到FTP成功
 
 
＃打开文件流到源文件
 trap [Exception] {
 logEntry 1Check File Connection$ _。Exception.Message 
 $ sourceStream。 Close（）
 $ ftpResponse.Close（）
 exit 
 
 $ sourceStream = New-Object IO.FileStream（$（New-Object System.IO.FileInfo $ sourceFile） ，[IO.FileMode] :: Open）
 [byte []] $ readbuffer = New-Object byte [] 1024 
 $ b logEntry 0启动文件传输成功
＃获取请求流，并将字节写入它
 $ rs = $ ftpRequest.GetRequestStream（）
 do {
 $ readlength = $ sourceStream.Read（$ readbuffer，0,1024 ）
 $ rs.Write（$ readbuffer，0，$ readlength）
} while（$ readlength -ne 0）
 
 logEntry 0传输完成成功
＃一定要清理后十二s 
 $ rs.Close（）
＃$ rs.Dispose（）
 $ sourceStream.Close（）
＃$ sourceStream.Dispose（）
 
} 
 $ ftpResponse.Close（）





< 

  logEntry 0开始文件传输成功
＃获取请求流，并将字节写入它
 $ rs = $ ftpRequest.GetRequestStream（）
 do {
 $ readlength = $ sourceStream.Read（$ readbuffer，0 ，1024）
 $ rs.Write（$ readbuffer，0，$ readlength）
} while（$ readlength -ne 0）
 $ rs.Close（）
＃start- sleep -s 2 
 
 trap [Exception] {
 $ script：succeeded = $ false 
 logEntry 1检查FTP连接$ _。Exception.Message 
 continue 
} 
 $ ftpResponse = $ ftpRequest.GetResponse（）


解决方案

与使用Get-Content将整个文件读入内存不同，尝试一次一个块地读取它，而w引用它到FTP请求流。我会使用其中一个较低级别的.NET文件流API来读取数据。诚然，你不会认为10MB会造成内存问题。

另外，确保在创建请求流并写入请求流后获得响应。获取响应流是上传数据。从文档：


当使用FtpWebRequest对象到
将文件上传到服务器时，您必须
写入通过调用
GetRequestStream方法或其
异步对应方法，
BeginGetRequestStream和
EndGetRequestStream方法获得请求
流的文件内容。在发送请求之前，您必须
写入流并关闭
流。

请求通过
发送到服务器，调用GetResponse方法或
异步对象，
BeginGetResponse和EndGetResponse
方法。当请求的操作
完成时，FtpWebResponse对象返回
。 FtpWebResponse对象
提供操作
的状态以及从
服务器下载的任何数据。


I have a script partially based on the one here: Upload files with FTP using PowerShell

It all works absolutely fine with tiny files but I am trying to use it to make the process we use for exporting access mdb files to clients that only have ftp more robust.

My first test involved a 10MB file and I ran into a System.OutOfMemoryException at the Get-Content stage

The powershell ISE was running to nearly 2GIG usage during the get attempt.

Here is a full script sample (Be gentle. I am fairly new to it):

#####
# User variables to control the script
#####

# How many times connection will be re-tried
$connectionTries = 5
#time between tries in seconds
$connectionTryInterval = 300
#Where to log the output
$logFile = "D:\MyPath\ftplog.txt"
#maximum log file size in KB before it is archived
$logFileMaxSize = 500

#formatted date part for the specific file to transfer
#This is appended to the filename base. Leave as "" for none
$datePart = ""
#base part of the file name
$fileNameBase = "Myfile"
#file extension
$fileExtension = ".mdb"
#location of the source file (please include trailing backslash)
$sourceLocation = "D:\MyPath\"

#location and credentials of the target ftp server    
$userName = "iamafish"
$password = "ihavenofingers"
$ftpServer = "10.0.1.100"

######
# Main Script
#####

#If there is a log file and it is longer than the declared limit then archive it with  the current timestamp
if (test-path $logfile)
{
    if( $((get-item $logFile).Length/1kb) -gt $logFileMaxSize)
    {
        write-host $("archiving log to ftplog_" + (get-date -format yyyyMMddhhmmss) +     ".txt")
        rename-item $logFile $("ftplog_" + (get-date -format yyyyMMddhhmmss) + ".txt")
    }
}

#start new log entry
#Add-Content $logFile "___________________________________________________________"
#write-host $logEntry

#contruct source file and destination uri
$fileName = $fileNameBase + $datePart + $fileExtension
$sourceFile = $sourceLocation + $fileName
$sourceuri = "ftp://" + $ftpServer + "/" + $fileName


# Create a FTPWebRequest object to handle the connection to the ftp server
$ftprequest = [System.Net.FtpWebRequest]::create($sourceuri)

# set the request's network credentials for an authenticated connection
$ftprequest.Credentials = New-Object System.Net.NetworkCredential($username,$password)

$ftprequest.Method = [System.Net.WebRequestMethods+Ftp]::UploadFile
$ftprequest.UseBinary = $true
$ftprequest.KeepAlive = $false

$succeeded = $true
$errorMessage = ""

# read in the file to upload as a byte array
trap [exception]{
    $script:succeeded = $false
    $script:errorMessage = $_.Exception.Message
    Add-Content $logFile $((get-Date -format "yyyy-MM-dd hh:mm:ss") + "|1|" +    $_.Exception.Message)
    #write-host $logEntry
    #write-host $("TRAPPED: " + $_.Exception.GetType().FullName)
    #write-host $("TRAPPED: " + $_.Exception.Message)
    exit
}
#The -ea 1 forces the error to be trappable
$content = gc -en byte $sourceFile -ea 1


$try = 0

do{
    trap [System.Net.WebException]{
        $script:succeeded = $false
        $script:errorMessage = $_.Exception.Message
        Add-Content $logFile $((get-Date -format "yyyy-MM-dd hh:mm:ss") + "|1|" +    $_.Exception.Message)
        #write-host $logEntry
        #write-host $("TRAPPED: " + $_.Exception.GetType().FullName)
        $script:try++
        start-sleep -s $connectionTryInterval
        continue
        }
        $ftpresponse = $ftprequest.GetResponse()

} while(($try -le $connectionTries) -and (-not $succeeded))

if ($succeeded) { 

    Add-Content $logFile $((get-Date -format "yyyy-MM-dd hh:mm:ss") + "|0|" +    "Starting file transfer.")
    # get the request stream, and write the bytes into it
    $rs = $ftprequest.GetRequestStream()
    $rs.Write($content, 0, $content.Length)
    # be sure to clean up after ourselves
    $rs.Close()
    $rs.Dispose()
    $content.Close()
    $content.Dispose()
    Add-Content $logFile $((get-Date -format "yyyy-MM-dd hh:mm:ss") + "|0|" +    "Transfer complete.")
    #write-host $logEntry
}

I can't put code in comments so, thanks to pointers from keith I have moved the file acces bit down to the bottom to link it with the other like so..

trap [Exception]{
    $script:succeeded = $false
    $script:errorMessage = $_.Exception.Message
    Add-Content $logFile $((get-Date -format "yyyy-MM-dd hh:mm:ss") + "|1|Check File Connection|" + $_.Exception.Message)
    $sourceStream.Close()
    $sourceStream.Dispose()
    #write-host $((get-Date -format "yyyy-MM-dd hh:mm:ss") + "|1|Attempt to open file|" + $_.Exception.Message)
    #write-host $("TRAPPED: " + $_.Exception.GetType().FullName)
    exit
}
$sourceStream = New-Object IO.FileStream ($(New-Object System.IO.FileInfo $sourceFile),[IO.FileMode]::Open)
[byte[]]$readbuffer = New-Object byte[] 1024

# get the request stream, and write the bytes into it
$rs = $ftprequest.GetRequestStream()
do{
    $readlength = $sourceStream.Read($readbuffer,0,1024)
    $rs.Write($readbuffer,0,$readlength)
} while ($readlength -ne 0)

I just need to work out why I get: Exception calling "GetResponse" with "0" argument(s): "Cannot access a disposed object. every other time I run it. Is this a quirk of running it in the ISE or am I doing somethign drasically wrong with either initial declaration or final disposing?

I'll post the full final script when done since I think it will make a nice sturdy ftp export example with error trapping and logging.

OK, here is the full script. Dispose is edited out but with or without it runnign the script within 5 minutes will either get me a message that I cannot use a disposed opject or tell me that the getResponse() has produced an error (226) File transfered (running in ISE). Whilst this will not be a problem during normal opperation I would like to correctly log oout of the FTP session and clean the resources at the end of the script and ensure I am correctly declaring them as needed.

#####
# User variables to control the script
#####

# How many times connection will be re-tried
$connectionTries = 5
#time between tries in seconds
$connectionTryInterval = 1
#Where to log the output
$logFile = "D:\MyPath\ftplog.txt"
#maximum log file size in KB before it is archived
$logFileMaxSize = 500
#log to file or console - #true=log to file, #false = log to console
$logToFile=$false

#formatted date part for the specific file to transfer
#This is appended to the filename base. Leave as "" for none
$datePart = ""
#base part of the file name
$fileNameBase = "MyFile"
#file extension
$fileExtension = ".mdb"
#location of the source file (please include trailing backslash)
$sourceLocation = "D:\MyPath\"

#location and credentials of the target ftp server
$userName = "iamafish"
$password = "ihavenofingers"
$ftpServer = "10.0.1.100"

######
# Main Script
#####

function logEntry($entryType, $section, $message)
{
    #just to make a one point switch for logging to console for testing
    # $entryType: 0 = success, 1 = Error
    # $section: The section of the script the log entry was generated from
    # $message: the log message

    #This is pipe separated to fit in with my standard MSSQL linked flat file schema for easy querying
    $logString = "$(get-Date -format "yyyy-MM-dd hh:mm:ss")|$entryType|$section|$message"

    if($script:logtoFile)
    {
        Add-Content $logFile $logString
    }
    else
    {
        write-host $logString
    }
}

#If there is a log file and it is longer than the declared limit then archive it with the current timestamp
if (test-path $logfile)
{
    if( $((get-item $logFile).Length/1kb) -gt $logFileMaxSize)
    {
        write-host $("archiving log to ftplog_" + (get-date -format yyyyMMddhhmmss) + ".txt")
        rename-item $logFile $("ftplog_" + (get-date -format yyyyMMddhhmmss) + ".txt")
        New-Item $logFile -type file
    }
}
else
{
    New-Item $logFile -type file
}


#contruct source file and destination uri
$fileName = $fileNameBase + $datePart + $fileExtension
$sourceFile = $sourceLocation + $fileName
$destination = "ftp://" + $ftpServer + "/" + $fileName


#Check if the source file exists
if ((test-path $sourceFile) -eq $false)
{
    logEntry 1 "Check Source File" $("File not found: " + $sourceFile)
    Exit
}


# Create a FTPWebRequest object to handle the connection to the ftp server
$ftpRequest = [System.Net.FtpWebRequest]::create($destination)

# set the request's network credentials for an authenticated connection
$ftpRequest.Credentials = New-Object System.Net.NetworkCredential($username,$password)
$ftpRequest.Method = [System.Net.WebRequestMethods+Ftp]::UploadFile
$ftpRequest.UseBinary = $true
$ftpRequest.KeepAlive = $false

$succeeded = $true
$try = 1

do{
    trap [Exception]{
        $script:succeeded = $false
        logEntry 1 "Check FTP Connection" $_.Exception.Message
        $script:try++
        start-sleep -s $connectionTryInterval
        continue
        }
        $ftpResponse = $ftpRequest.GetResponse()

} while(($try -le $connectionTries) -and (-not $succeeded))

if ($succeeded) {
    logEntry 0 "Connection to FTP" "Success"


    # Open a filestream to the source file
    trap [Exception]{
        logEntry 1 "Check File Connection" $_.Exception.Message
        $sourceStream.Close()
        $ftpResponse.Close()
        exit
    }
    $sourceStream = New-Object IO.FileStream ($(New-Object System.IO.FileInfo $sourceFile),[IO.FileMode]::Open)
    [byte[]]$readbuffer = New-Object byte[] 1024

    logEntry 0 "Starting file transfer" "Success"
    # get the request stream, and write the bytes into it
    $rs = $ftpRequest.GetRequestStream()
    do{
        $readlength = $sourceStream.Read($readbuffer,0,1024)
        $rs.Write($readbuffer,0,$readlength)
    } while ($readlength -ne 0)

    logEntry 0 "Transfer complete" "Success"
    # be sure to clean up after ourselves
    $rs.Close()
    #$rs.Dispose()
    $sourceStream.Close()
    #$sourceStream.Dispose()

}
$ftpResponse.Close()

Example of trying to trap the Transfer OK response at the end:

logEntry 0 "Starting file transfer" "Success"
# get the request stream, and write the bytes into it
$rs = $ftpRequest.GetRequestStream()
do{
    $readlength = $sourceStream.Read($readbuffer,0,1024)
    $rs.Write($readbuffer,0,$readlength)
} while ($readlength -ne 0)
$rs.Close()
#start-sleep -s 2

trap [Exception]{
    $script:succeeded = $false
    logEntry 1 "Check FTP Connection" $_.Exception.Message
    continue
}
$ftpResponse = $ftpRequest.GetResponse()

解决方案

Rather than read the whole file into memory using Get-Content, try reading it in a chunk at a time and writing it to the FTP request stream. I would use one of the lower level .NET file stream APIs to do the reading. Admittedly, you wouldn't think a 10MB would pose a memory problem though.

Also, make sure you get the response after geting the request stream and writing to it. The get of the response stream is what uploads the data. From the docs:

When using an FtpWebRequest object to upload a file to a server, you must write the file content to the request stream obtained by calling the GetRequestStream method or its asynchronous counterparts, the BeginGetRequestStream and EndGetRequestStream methods. You must write to the stream and close the stream before sending the request.

Requests are sent to the server by calling the GetResponse method or its asynchronous counterparts, the BeginGetResponse and EndGetResponse methods. When the requested operation completes, an FtpWebResponse object is returned. The FtpWebResponse object provides the status of the operation and any data downloaded from the server.

这篇关于Powershell FTP发送大文件System.OutOfMemoryException的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！

查看全文
相关文章
其他开发最新文章
热门教程
热门工具
登录 关闭
扫码关注1秒登录
发送“验证码”获取 | 15天全站免登陆