Powershell以块读取文件 [英] Powershell read file in chunks

问题描述

我在Powershell中编写了一个脚本，它通过FTP传输文件，使用它可以正常工作：

  $ content = [System.IO.File] :: ReadAllBytes（$ backup_app_data）
  

但是，文件大小达到2Gb，通过抛出一个错误，说这种方法仅限于读取文件大小。

$ b

调用ReadAllBytes带有1参数：该文件是
太长，此操作目前仅限于支持少于
的文件，而不是大小为2 GB的文件。

现在，我试图修改我的脚本并使用 阅读 ，根据我的理解，这将允许我以块的形式读取文件，然后编写这些块等等。

不幸的是，尽管我修改了脚本d似乎正在工作，因为在我的FTP位置创建了一个文件，所以没有数据被传输，脚本也没​​有发生错误。

只有一个在目标文件夹中创建的0kb文件和脚本结束。

我试图做一些调试，但我似乎无法找到问题。以下是我目前使用的代码。

  $ file ='G：\Backups\DATA_backup_2016_09_05.zip'
 $ dest_name ='DATA_backup_2016_09_05.zip'
 
 $ ftp = [System.Net.FtpWebRequest] :: Create（$ ftp + $ dest_name）
 $ ftp = [System.Net。 FtpWebRequest] $ ftp 
 $ ftp.Method = [System.Net.WebRequestMethods + Ftp] :: UploadFile 
 $ ftp.Credentials = new-object System.Net.NetworkCredential（$ user，$ pass）
 
 $ ftp.UseBinary = $ true 
 $ ftp.UsePassive = $ true 
 
＃决定文件大小
 $ file_size =（Get- Item $ file）.length 
 $ chunk_size = 512mb 
 
 $ bytes_to_read = $ file_size 
 $ iterations = $ file_size / $ chunk_size 
 $ iter = 0 
 
 $ fstream = [System.IO.FileStream] 
 [byte []] $ byte_array 
 
 while（$ bytes_to_read> 0）{
 if（$迭代> 1）{
 $ content = $ fstream.Read（$ byte_array，$ iter，$ chunk_size）
} 
 else {
 $ content = $ fstream.Read（ $ b yte_array，$ iter，$ bytes_to_read）
} 
 
 $ ftp.ContentLength = $ content.Length 
 
 $ rs = $ ftp.GetRequestStream（）
 $ rs.Write（$ content，0，$ content.Length）
 
＃让循环进行
 $ iter = $ iter + 1 
 $ iterations = $ iterations  - 1 
 $ bytes_to_read = $ bytes_to_read  -  $ chunk_size 
} 
 
 $ rs.Close（）
 $ rs.Dispose（）
  

任何帮助表示感谢。

>因此，根据评论部分中的建议并以 答案 为例可能存在重复 问题 ，我设法找到解决方案

以下是我通过FTP传输的代码为3.74Gb的.zip文件。

  $ ftp_addr =ftp：//ftp.som erandomwebsite.com/folder1/
 $ user =usr
 $ pass =passwrd
 $ bufSize = 256mb 
 
＃...缺少代码其中我确定了我想要FTP的文件...＃
 
＃初始化连接到FTP 
 $ ftp = [System.Net.FtpWebRequest] :: Create（$ ftp_addr + $ backup_file + .zip）
 $ ftp = [System.Net.FtpWebRequest] $ ftp 
 $ ftp.Method = [System.Net.WebRequestMethods + Ftp] :: UploadFile 
 $ ftp.Credentials =新对象System.Net.NetworkCredential（$ user，$ pass）
 
 $ ftp.Timeout = -1 #infinite timeout 
 $ ftp.ReadWriteTimeout = -1 #infinite timeout 
 
 $ ftp.UseBinary = $ true 
 $ ftp.UsePassive = $ true 
 $ b $ requestStream = $ ftp.GetRequestStream（）
 $ fileStream = [System。 IO.File] :: OpenRead（$ file_to_ftp）
 $ chunk = New-Object byte [] $ bufSize 
 
 while（$ bytesRead = $ fileStream.Read（$ chunk，0，$ bufsize））{
 $ requestStream.write（$ chunk，0，$ bytesRead）
 $ requestStream.Flush（）
} 
 
 $ FileStream.Cl ose（）
 $ requestStream.Close（）
  

到目前为止，这段代码完美无缺地工作大约多次（20+）次尝试。我希望它能帮助别人！

I've had a script written in Powershell which transferred a file via FTP which worked fine by using:

$content = [System.IO.File]::ReadAllBytes($backup_app_data)

But this stopped working once the file size reached 2Gb, by throwing an error saying that this method is limited to reading files up to this size.

Exception calling "ReadAllBytes" with "1" argument(s): "The file is too long. This operation is currently limited to supporting files less than 2 gigabytes in size.

Now, I'm trying to modify my script and use an alternate approach with Read, which from what I understand will allow me to read the file in chunks and then write those chunks and so on.

Unfortunately although the script I've modified seems to be working, since a file is created on my FTP location, there is no data being transferred and no error is thrown by the script.

There is just a 0kb file created in the destination folder and the script ends.

I've tried to do some debugging, but I can't seem to find the issue. Below is the code I'm currently using.

$file = 'G:\Backups\DATA_backup_2016_09_05.zip'
$dest_name = 'DATA_backup_2016_09_05.zip'

$ftp = [System.Net.FtpWebRequest]::Create($ftp+$dest_name)
$ftp = [System.Net.FtpWebRequest]$ftp
$ftp.Method = [System.Net.WebRequestMethods+Ftp]::UploadFile
$ftp.Credentials = new-object System.Net.NetworkCredential($user, $pass)

$ftp.UseBinary = $true
$ftp.UsePassive = $true

# determine the size of the file
$file_size = (Get-Item $file).length
$chunk_size = 512mb

$bytes_to_read = $file_size
$iterations = $file_size / $chunk_size
$iter = 0

$fstream = [System.IO.FileStream]
[byte[]] $byte_array 

while ($bytes_to_read > 0){
    if($iterations > 1) {
        $content = $fstream.Read($byte_array, $iter, $chunk_size)
        }
    else {
        $content = $fstream.Read($byte_array, $iter, $bytes_to_read)
    }

    $ftp.ContentLength = $content.Length

    $rs = $ftp.GetRequestStream()
    $rs.Write($content, 0, $content.Length)

    # keep the loop going
    $iter = $iter + 1
    $iterations = $iterations - 1
    $bytes_to_read = $bytes_to_read - $chunk_size
}

$rs.Close()
$rs.Dispose()

Any help is appreciated.

解决方案

So, based on the suggestions in the comment section and using as an example the answer from the possible duplicate question, I managed to find the solution myself.

Below is the code I used to transfer via FTP a .zip archive which was 3.74Gb.

$ftp_addr = "ftp://ftp.somerandomwebsite.com/folder1/"
$user = "usr"
$pass = "passwrd"
$bufSize = 256mb

# ... missing code where I identify the file I want to FTP ... #

# Initialize connection to FTP
$ftp = [System.Net.FtpWebRequest]::Create($ftp_addr + $backup_file + ".zip")
$ftp = [System.Net.FtpWebRequest]$ftp
$ftp.Method = [System.Net.WebRequestMethods+Ftp]::UploadFile
$ftp.Credentials = new-object System.Net.NetworkCredential($user, $pass)

$ftp.Timeout = -1              #infinite timeout
$ftp.ReadWriteTimeout = -1     #infinite timeout

$ftp.UseBinary = $true
$ftp.UsePassive = $true

$requestStream = $ftp.GetRequestStream()
$fileStream = [System.IO.File]::OpenRead($file_to_ftp)
$chunk = New-Object byte[] $bufSize

while ( $bytesRead = $fileStream.Read($chunk, 0, $bufsize) ){
    $requestStream.write($chunk, 0, $bytesRead)
    $requestStream.Flush()
}

$FileStream.Close()
$requestStream.Close()

So far, this code has worked flawlessly for about multiple (20+) attempts. I hope it helps others!

这篇关于Powershell以块读取文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！