最强大的使用Java读取文件或流的方法（防止DoS攻击） [英] Most Robust way of reading a file or stream using Java (to prevent DoS attacks)

问题描述

目前我有以下代码用于阅读 InputStream 。我将整个文件存储到 StringBuilder 变量中，然后处理该字符串。

  public static String getContentFromInputStream（InputStream inputStream）
 // public static String getContentFromInputStream（InputStream inputStream，
 // int maxLineSize，int maxFileSize）
 {
 
 StringBuilder stringBuilder = new StringBuilder（）; 
 BufferedReader bufferedReader = new BufferedReader（new InputStreamReader（inputStream））; 
 String lineSeparator = System.getProperty（line.separator）; 
 String fileLine; 
 
 boolean firstLine = true; 
 try {
 //期待一些检查行大小限制的函数。 
 //例如：逐个字符地读取字符数组并在循环中检查
 // linesize，直到遇到换行符。 
 //如果传递最大行大小限制然后抛出异常
 //如果遇到换行符将char数组追加到StringBuilder 
 //后追加检查StringBuilder的大小
 //如果文件大小超过最大文件限制，则抛出异常
 
 fileLine = bufferedReader.readLine（）; 
 
 while（fileLine！= null）{
 if（！firstLine）stringBuilder.append（lineSeparator）; 
 stringBuilder.append（fileLine）; 
 fileLine = bufferedReader.readLine（）; 
 firstLine = false; 
} 
} catch（IOException e）{
 // TODO：抛出或处理异常
} 
 // TODO：关闭流
 
返回stringBuilder.toString（）; 
 
} 
  

该代码用于安全团队和收到以下评论：

1. BufferedReader.readLine 易受DOS影响（拒绝服务）攻击（无限长度的行，不包含换行/回车的巨大文件）




2. 的资源耗尽StringBuilder 变量（包含大于可用内存的数据的文件的情况）

下面是我能想到的解决方案：

1. 创建 readLine的替代实现方法（ readLine（int limit）），它检查no。读取的字节数如果超过指定的限制，则抛出自定义异常。




2. 逐行处理文件而不完整加载文件。 （纯非Java解决方案:)）

请建议是否有任何现有的库实现上述解决方案。
还建议任何替代解决方案，它们提供更强大的功能，或者比实现方案更方便实现。虽然性能也是一个主要要求，但安全性是第一位的。

解决方案

更新后的答案

您希望避免各种DOS攻击（在线路上，文件大小等）。但是在函数的最后，你试图将整个文件转换成一个 String ！假设您将行限制为8 KB，但如果某人向您发送包含两个8 KB行的文件会发生什么？行读取部分将通过，但是当你最终将所有内容组合成一个字符串时，字符串将阻塞所有可用内存。

因此，最后你将所有内容转换为一个字符串，限制行大小无关紧要，也不安全。你必须限制文件的整个大小。

其次，你基本上要做的是，你试图以块的形式读取数据。所以你正在使用 BufferedReader 并逐行阅读。但是你要做的是什么，以及你最终想要的是什么 - 是一种逐段阅读文件的方式。而不是一次只读一行，为什么不一次读取2 KB？

BufferedReader - by它的名字 - 里面有一个缓冲区。您可以配置该缓冲区。假设你创建一个缓冲区大小为2 KB的 BufferedReader ：

  BufferedReader reader = new BufferedReader（...，2048）; 
  

现在，如果您传递给 InputStream BufferedReader 有100 KB的数据， BufferedReader 将自动读取2 KB的数据。因此它将读取流50次，每次2 KB（50x2KB = 100 KB）。同样，如果使用10 KB缓冲区大小创建 BufferedReader ，它将读取输入10次（10x10KB = 100 KB）。

BufferedReader 已经完成了以块为单位读取文件的工作。因此，您不希望在其上方添加额外的逐行图层。只关注最终结果 - 如果你的文件结尾太大（>可用内存） - 你最后如何将它转换为 String ？ / p>

更好的方法是将事物作为 CharSequence 传递。这就是Android的功能。在整个Android API中，您将看到它们在任何地方都返回 CharSequence 。由于 StringBuilder 也是 CharSequence 的子类，Android将在内部使用 String ，或者 StringBuilder 或基于输入的大小/性质的一些其他优化的字符串类。因此，一旦您阅读了所有内容，您就可以直接返回 StringBuilder 对象，而不是将其转换为 String 。这对大数据更安全。 StringBuilder 还在其中维护相同的缓冲区概念，它将在内部为大字符串分配多个缓冲区，而不是一个长字符串。

总体来说：

• 限制整体文件大小，因为你要在某个时候处理整个内容。忘记限制或拆分行


• 读入块

使用Apache Commons IO，这里是如何将 BoundedInputStream 中的数据读入 StringBuilder ，拆分2 KB块而不是行：

  // import org.apache.commons.io.output.StringBuilderWriter; 
 // import org.apache.commons.io.input.BoundedInputStream; 
 // import org.apache.commons.io.IOUtils; 
 
 BoundedInputStream boundedInput = new BoundedInputStream（originalInput，< max-file-size>）; 
 BufferedReader reader = new BufferedReader（new InputStreamReader（boundedInput），2048）; 
 
 StringBuilder输出= new StringBuilder（）; 
 StringBuilderWriter writer = new StringBuilderWriter（output）; 
 
 IOUtils.copy（读者，作家）; //从reader=>复制数据作家
返回输出; 
  

原始答案

使用 BoundedInputStream 来自 Apache Commons IO 库。您的工作变得更加容易。

以下代码可以满足您的需求：

  public static String getContentFromInputStream（InputStream inputStream）{
 inputStream = new BoundedInputStream（inputStream，< number-of-bytes>）; 
 //其余代码全部相同
  

你只需简单地包装 InputStream ，带有 BoundedInputStream ，并指定最大大小。 BoundedInputStream 将负责将读数限制为最大尺寸。

或者你可以这样做创建阅读器：

  BufferedReader bufferedReader = new BufferedReader（
 new InputStreamReader（
 new BoundedInputStream（inputStream， < no-of-bytes>）
）
）; 
  

基本上我们在这里做的是，我们限制在<$ c的读取大小$ c> InputStream 图层本身，而不是在读取线条时这样做。所以你最终得到了一个可重用的组件，如 BoundedInputStream ，它限制了在InputStream层的读取，你可以在任何你想要的地方使用它。

编辑：添加脚注

编辑2：根据评论添加更新的答案

Currently I have the below code for reading an InputStream. I am storing the whole file into a StringBuilder variable and processing this string afterwards.

public static String getContentFromInputStream(InputStream inputStream)
// public static String getContentFromInputStream(InputStream inputStream,
// int maxLineSize, int maxFileSize)
{

    StringBuilder stringBuilder = new StringBuilder();
    BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(inputStream));
    String lineSeparator = System.getProperty("line.separator");
    String fileLine;

    boolean firstLine = true;
    try {
        // Expect some function which checks for line size limit.
        // eg: reading character by character to an char array and checking for
        // linesize in a loop until line feed is encountered.
        // if max line size limit is passed then throw an exception
        // if a line feed is encountered append the char array to a StringBuilder
        // after appending check the size of the StringBuilder
        // if file size exceeds the max file limit then throw an exception

        fileLine = bufferedReader.readLine();

        while (fileLine != null) {
            if (!firstLine) stringBuilder.append(lineSeparator);
            stringBuilder.append(fileLine);
            fileLine = bufferedReader.readLine();
            firstLine = false;
        }
    } catch (IOException e) {
        //TODO : throw or handle the exception
    }
    //TODO : close the stream

    return stringBuilder.toString();

}

The code went for a review with the Security team and the following comments were received:

1. BufferedReader.readLine is susceptible to DOS (Denial of Service) attacks (line of infinite length, huge file containing no line feed/carriage return)

2. Resource exhaustion for the StringBuilder variable (cases when a file containing data greater than the available memory)

Below are the solutions I could think of:

1. Create an alternate implementation of readLine method (readLine(int limit)), which checks for the no. of bytes read and if it exceeds the specified limit, throw a custom exception.

2. Process the file line by line without loading the file in entirety. (pure non-Java solution :) )

Please suggest if there are any existing libraries which implement the above solutions. Also suggest any alternate solutions which offer more robustness or are more convenient to implement than the proposed ones. Though performance is also a major requirement, security comes first.

解决方案

Updated Answer

You want to avoid all sorts of DOS attacks (on lines, on size of the file, etc). But in the end of the function, you're trying to convert the entire file into one single String!!! Assume that you limit the line to 8 KB, but what happens if somebody sends you a file with two 8 KB lines? The line reading part will pass, but when finally you combine everything into a single string, the String will choke all available memory.

So since finally you're converting everything into one single String, limiting line size doesn't matter, nor is safe. You have to limit the entire size of the file.

Secondly, what you're basically trying to do is, you're trying to read data in chunks. So you're using BufferedReader and reading it line-by-line. But what you're trying to do, and what you really want at the end - is some way of reading the file piece by piece. Instead of reading one line at a time, why not instead read 2 KB at a time?

BufferedReader - by its name - has a buffer inside it. You can configure that buffer. Let's say you create a BufferedReader with buffer size of 2 KB:

BufferedReader reader = new BufferedReader(..., 2048);

Now if the InputStream that you pass to BufferedReader has 100 KB of data, BufferedReader will automatically read it 2 KB at at time. So it will read the stream 50 times, 2 KB each (50x2KB = 100 KB). Similarly, if you create BufferedReader with a 10 KB buffer size, it will read the input 10 times (10x10KB = 100 KB).

BufferedReader already does the work of reading your file chunk-by-chunk. So you don't want to add an extra layer of line-by-line above it. Just focus on the end result - if your file at the end is too big (> available RAM) - how are you going to convert it into a String at the end?

One better way is to just pass things around as a CharSequence. That's what Android does. Throughout the Android APIs, you will see that they return CharSequence everywhere. Since StringBuilder is also a subclass of CharSequence, Android will internally use either a String, or a StringBuilder or some other optimized string class based on the size/nature of input. So you could rather directly return the StringBuilder object itself once you've read everything, rather than converting it to a String. This would be safer against large data. StringBuilder also maintains the same concept of buffers inside it, and it will internally allocate multiple buffers for large strings, rather than one long string.

So overall:

• Limit the overall file size since you're going to deal with the entire content at some point. Forget about limiting or splitting lines
• Read in chunks

Using Apache Commons IO, here is how you would read data from a BoundedInputStream into a StringBuilder, splitting by 2 KB blocks instead of lines:

// import org.apache.commons.io.output.StringBuilderWriter;
// import org.apache.commons.io.input.BoundedInputStream;
// import org.apache.commons.io.IOUtils;

BoundedInputStream boundedInput = new BoundedInputStream(originalInput, <max-file-size>);
BufferedReader reader = new BufferedReader(new InputStreamReader(boundedInput), 2048);

StringBuilder output = new StringBuilder();
StringBuilderWriter writer = new StringBuilderWriter(output);

IOUtils.copy(reader, writer); // copies data from "reader" => "writer"
return output;

Original Answer

Use BoundedInputStream from Apache Commons IO library. Your work becomes much more easier.

The following code will do what you want:

public static String getContentFromInputStream(InputStream inputStream) {
  inputStream = new BoundedInputStream(inputStream, <number-of-bytes>);
  // Rest code are all same

You just simply wrap your InputStream with a BoundedInputStream and you specify a maximum size. BoundedInputStream will take care of limiting reads up to that maximum size.

Or you can do this when you're creating the reader:

BufferedReader bufferedReader = new BufferedReader(
  new InputStreamReader(
    new BoundedInputStream(inputStream, <no-of-bytes>)
  )
);

Basically what we're doing here is, we're limiting the read size at the InputStream layer itself, rather than doing that when reading lines. So you end up with a reusable component like BoundedInputStream which limits reading at the InputStream layer, and you can use that wherever you want.

Edit: Added footnote

Edit 2: Added updated answer based on comments

这篇关于最强大的使用Java读取文件或流的方法（防止DoS攻击）的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！