大文件加密逐行 [英] Large file encryption line by line

问题描述

我'开发'了一个应用程序用于加密给定文本文件的ID。应用程序采用大文本文件，要加密的ID的长度，密码和填充字符串。应用程序正常工作，直到文本文件的行数大于200万。我相信这是由于某些内容存储在内存中而未发布。



请查看下面我正在使用的代码以及示例数据：

I have 'developed' an application is used to encrypted an ID of a given text file. The application takes a large text file, the length of the ID to be encrypted, a password and a padding string. The application works fine until the number of lines of the text file is greater than 2 million. I believe this is due to something being stored in memory and not released.

Please find the code below that I am currently using along with sample data:

Imports System
Imports System.IO
Imports System.Text
Imports System.Security.Cryptography
Module RijndaelMemoryExample
    Function encryptStringToBytes_AES(ByVal plainText As String, ByVal Key() As Byte, ByVal IV() As Byte) As Byte()
        If plainText Is Nothing OrElse plainText.Length <= 0 Then
            Throw New ArgumentNullException("plainText")
        End If
        If Key Is Nothing OrElse Key.Length <= 0 Then
            Throw New ArgumentNullException("Key")
        End If
        If IV Is Nothing OrElse IV.Length <= 0 Then
            Throw New ArgumentNullException("Key")
        End If

        Dim aesAlg As RijndaelManaged = Nothing

        Dim msEncrypt As MemoryStream = Nothing
        Try
            aesAlg = New RijndaelManaged()
            aesAlg.Key = Key
            aesAlg.IV = IV

            Dim encryptor As ICryptoTransform = aesAlg.CreateEncryptor(aesAlg.Key, aesAlg.IV)
            msEncrypt = New MemoryStream()
            Using csEncrypt As New CryptoStream(msEncrypt, encryptor, CryptoStreamMode.Write)
                Using swEncrypt As New StreamWriter(csEncrypt)
                    swEncrypt.Write(plainText)
                End Using
            End Using
        Finally
            If Not (aesAlg Is Nothing) Then
                aesAlg.Clear()
            End If
        End Try
        Return msEncrypt.ToArray()
    End Function
End Module

Module Module1

    Public Class RijndaelSimple

        Public Shared Function Encrypt(ByVal plainText As String, _
                                        ByVal passPhrase As String, _
                                        ByVal saltValue As String, _
                                        ByVal hashAlgorithm As String, _
                                        ByVal passwordIterations As Integer, _
                                        ByVal initVector As String, _
                                        ByVal keySize As Integer) _
                                        As String
            Dim initVectorBytes As Byte()
            initVectorBytes = Encoding.ASCII.GetBytes(initVector)
            Dim saltValueBytes As Byte()
            saltValueBytes = Encoding.ASCII.GetBytes(saltValue)
            Dim plainTextBytes As Byte()
            plainTextBytes = Encoding.UTF8.GetBytes(plainText)
            Dim password As Rfc2898DeriveBytes
            password = New Rfc2898DeriveBytes(passPhrase, _
                                                saltValueBytes, _
                                                passwordIterations)
            Dim keyBytes As Byte()
            keyBytes = password.GetBytes(keySize / 8)
            Dim symmetricKey As RijndaelManaged
            symmetricKey = New RijndaelManaged()
            symmetricKey.Mode = CipherMode.CBC
            Dim encryptor As ICryptoTransform
            encryptor = symmetricKey.CreateEncryptor(keyBytes, initVectorBytes)
            Using memoryStream As MemoryStream = New MemoryStream()
                Using cryptoStream As CryptoStream = New CryptoStream(memoryStream, _
                                                encryptor, _
                                                CryptoStreamMode.Write)
                    cryptoStream.Write(plainTextBytes, 0, plainTextBytes.Length)
                    cryptoStream.FlushFinalBlock()
                    Dim cipherTextBytes As Byte()
                    cipherTextBytes = memoryStream.ToArray()
                    memoryStream.Close()
                    cryptoStream.Close()
                    Dim cipherText As String
                    cipherText = Convert.ToBase64String(cipherTextBytes)
                    Encrypt = cipherText
                    cryptoStream.Clear()
                    cryptoStream.Dispose()
                End Using
                memoryStream.Flush()
                memoryStream.Dispose()
            End Using
        End Function

    End Class

    Public Function Process(ByVal strPass As String, ByVal strData As String, ByRef strType As String)
        Dim plainText As String
        Dim cipherText As String
        Dim passPhrase As String
        Dim saltValue As String
        Dim hashAlgorithm As String
        Dim passwordIterations As Integer
        Dim initVector As String
        Dim keySize As Integer
        plainText = "Hello, World!"
        passPhrase = strPass ' can be any string
        saltValue = "s@1tValue" ' can be any string
        hashAlgorithm = "SHA1" ' can be "MD5"
        passwordIterations = 1 ' can be any number
        initVector = "@1B2c3D4e5F6g7H8" ' must be 16 bytes
        keySize = 256 ' can be 192 or 128
        strType = "Encrypt"
        plainText = strData ' original plaintext
        cipherText = RijndaelSimple.Encrypt(plainText, _
                                            passPhrase, _
                                            saltValue, _
                                            hashAlgorithm, _
                                            passwordIterations, _
                                            initVector, _
                                            keySize)
        Process = cipherText
    End Function

'Option Explicit On
'Option Strict On
Imports System
Imports System.IO
Imports System.Data
Public Class Form1
    Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click
        Dim FILE_NAME As String = txtInpath.Text
        If Not File.Exists(FILE_NAME) Then
            MsgBox("file_name does not exist. - " & FILE_NAME)
            Exit Sub
        End If
        If Len(txtLength.Text) = 0 Then
            MsgBox("Please enter the length of the Unique Identifier")
            Exit Sub
        End If
        If Len(txtPad.Text) = 0 Then
            MsgBox("Please enter a Padding String")
            Exit Sub
        End If
        If Len(txtPass.Text) = 0 Then
            MsgBox("Please enter a Password")
            Exit Sub
        End If

        StartEncrypt(FILE_NAME) ', i, tempInFileName, splitInFileName)


    End Sub


    Sub StartEncrypt(ByVal FILE_NAME As String) ', ByVal i As Integer, ByVal tempInFileName As String, ByVal splitInFileName() As String)

        Dim tempInFileName As String
        Dim splitOutFileName() = Split(txtOutpath.Text, ".")

        Dim strTopLineIn As String
        tempInFileName = FILE_NAME


        Using sr As StreamReader = File.OpenText(tempInFileName)

            Using sw As StreamWriter = New StreamWriter(splitOutFileName(0) & "_out." & splitOutFileName(1))
                Dim strLineIN As String, strLineOUT As String
                Dim strCHIN As String
                Dim strType As String
                Dim strPad As String = txtPad.Text
                Dim strLength As Integer = Int(txtLength.Text)
                strTopLineIn = sr.ReadLine()
                sw.WriteLine(strTopLineIn)
                strType = "Encrypt"
                Do While Not sr.EndOfStream
                    strLineIN = sr.ReadLine
                    strCHIN = Mid(strLineIN, 1, strLength)
                    strLineOUT = Process(txtPass.Text, (strPad & strCHIN), strType)
                    strLineOUT = strLineOUT & Mid(strLineIN, (strLength + 1), 7990)
                    sw.WriteLine(strLineOUT)
                Loop

            End Using

        End Using
        Return
    End Sub

    Private Sub Button2_Click_1(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button2.Click
        Me.Close()
    End Sub
    Private Sub OpenFileDialog1_FileOk_1(ByVal sender As System.Object, ByVal e As System.ComponentModel.CancelEventArgs) Handles OpenFileDialog1.FileOk
        Dim strm As System.IO.Stream
        strm = OpenFileDialog1.OpenFile()
        txtInpath.Text = OpenFileDialog1.FileName.ToString()
        If Not (strm Is Nothing) Then

            strm.Close()

        End If
    End Sub
    Private Sub Button3_Click_1(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button3.Click
        OpenFileDialog1.Title = "Please Select a File"
        OpenFileDialog1.Filter = "All Files|*.*|CSV Files|*.csv|Text Files|*.txt"
        OpenFileDialog1.ShowDialog()
        If txtInpath.Text = "" Then
            'do nothing
        Else
            txtOutpath.Text = Mid(txtInpath.Text, 1, Len(txtInpath.Text) - 4) & "_Out.csv"
        End If

    End Sub
    Private Sub Button4_Click(ByVal sender As System.Object, ByVal e As System.EventArgs)
        OpenFileDialog2.Title = "Please Select a File"
        OpenFileDialog2.ShowDialog()
    End Sub
    Private Sub OpenFileDialog2_FileOk(ByVal sender As System.Object, ByVal e As System.ComponentModel.CancelEventArgs) Handles OpenFileDialog2.FileOk
        Dim strm As System.IO.Stream
        strm = OpenFileDialog2.OpenFile()
        txtOutpath.Text = OpenFileDialog2.FileName.ToString()
        If Not (strm Is Nothing) Then

            strm.Close()

        End If
    End Sub

    Private Sub cboxPassShow_CheckedChanged(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles cboxPassShow.CheckedChanged
        If txtPass.PasswordChar = "*" Then
            txtPass.PasswordChar = ""
        Else
            txtPass.PasswordChar = "*"
        End If
        txtPass.Refresh()
    End Sub

    Private Sub cboxPadShow_CheckedChanged(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles cboxPadShow.CheckedChanged
        If txtPad.PasswordChar = "*" Then
            txtPad.PasswordChar = ""
        Else
            txtPad.PasswordChar = "*"
        End If
        txtPad.Refresh()
    End Sub

    Private Sub txtLength_validate(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles txtLength.KeyPress, Me.MouseMove
        If txtLength.Text = Nothing Then
            Button1.Enabled = False
        Else
            Button1.Enabled = True
        End If
    End Sub

End Class
    End Module

样本数据：



输入数据示例：



| ID | FName | SName |号码| Pcode |

------------------------------------------ ---------------------------

| 5364gfyue8 |杰森|年轻| 65782947 | BT753HY |

------------------------------------------ ---------------------------

| 8472hgiek5 |本|上衣| 87385942 | BT439ES |

------------------------------------------ ---------------------------

| 0243hgqwl4 |菲利普|摩尔| 73894769 | BT641II |



输出数据示例：



| ID | FName | SName |号码| Pcode |

------------------------------------------ ---------------------------

| 85hftFHDYTE * = jcg |杰森|年轻| 65782947 | BT753HY |

------------------------------------------ ---------------------------

| 5jgj59GUF55 = hf4！|本|上衣| 87385942 | BT439ES |

------------------------------------------ ---------------------------

| jhIOPfd5GF ^ 83 = 3c |菲利普|摩尔| 73894769 | BT641II |

Sample data:

Example of input data:

| ID | FName | SName | Number | Pcode |
---------------------------------------------------------------------
|5364gfyue8| Jason | Young | 65782947 | BT753HY |
---------------------------------------------------------------------
|8472hgiek5| Ben | Tops | 87385942 | BT439ES |
---------------------------------------------------------------------
|0243hgqwl4| Philip | Moore | 73894769 | BT641II |

Example of output data:

| ID | FName | SName | Number | Pcode |
---------------------------------------------------------------------
|85hftFHDYTE*=jcg| Jason | Young | 65782947 | BT753HY |
---------------------------------------------------------------------
|5jgj59GUF55=hf4!| Ben | Tops | 87385942 | BT439ES |
---------------------------------------------------------------------
|jhIOPfd5GF^83=3c| Philip | Moore | 73894769 | BT641II |

推荐答案

这篇关于大文件加密逐行的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！