使用Web应用程序上传大文件 [英] Upload of huge file using a web application

问题描述

给定目标的环境并不是现有的，因此，我无法尝试，只能依靠分析！

我的目标可以分为以下几个截然不同的步骤：
$ b $ ol

使用哑巴上传大文件（upto 100GB ），上传文件页面 - 由于用户需要（愚蠢的）前端，并且不愿意ftp文件等，所以没有任何改变。

提供上述的web应用程序前端将托管在一台低端机器上 - 2GB内存和40GB硬盘，这个Web应用程序将不会在本地机器上存储大文件的任何部分，但必须快速将其写入到高端远程Linux机器上

对于每一步，我都强调了我的方法，疑虑和疑问：

• 我把这个线程搞糊涂了，因为我是规划中g创建一个愚蠢的web应用程序使用Spring MVC与上传页面 - 我需要进入HTML5等或一个简单的Web应用程序就足够了吗？ >鉴于2GB内存，Web应用程序将获得小于1GB的。如果代码没有严格写入，恐怕会出现'OutOfMemoryError'错误 - 我必须确保从流中取出一个小块，例如10MB，并且一次只读一次，并写入远程Linux机器的文件中。
  假设我在Controller Servlet的doPost（...）中，我做了一些关于如何进行并弄糊涂的阅读：
  
  

   < （HttpServletRequest请求，HttpServletResponse 
   *响应）
   * / 
  保护无效doPost（HttpServletRequest请求，
  ）HttpServletResponse响应）throws ServletException，IOException {
   // TODO自动生成的方法存根
   
   InputStream fis = request.getInputStream（）; 
   int i = 0; 
   $ b / *方法 -  1：普通的老字节方法* / 
  套接字套接字=新套接字（192.168.90.20，22）; 
   OutputStream remoteOpStream = socket.getOutputStream（）; （（i = fis.read（））！= -1）{
   remoteOpStream.write（i）; 
   
   while 
   
   $ b $ * b $ b $ * b 
   $ * $方法2：NIO * / 
   ByteBuffer byteBuff = ByteBuffer.allocate（10000） ; / *读取10MB数据* / 
   
   ReadableByteChannel rdbyc = Channels.newChannel（request 
   .getInputStream（））; 
   
   File remoteFile = new File（192.168.90.20/Remote_Linux_Folder）; / * 
   * Dunno 
   * how 
   * to 
   * create 
   * a 
   *文件
   * a 
   *远程
   * Linux 
   *机器
   * / 
   FileOutputStream remoteFos = new FileOutputStream（remoteFile）; 
   FileChannel writableChannel = remoteFos.getChannel（）; 
   
   while（true / * dunno如何循环，直到所有的数据都被读取了！）/ $ {
   rdbyc.read（byteBuff）; 
   writableChannel.write（byteBuff）; 
   
   $ b $ *清理* / 
   
  } 
    

我需要某种方法，其中本地机器上的数据存储是最小的 - 代码只是从输入流中读取n个字节，相同的远程计算机

我相信NIO是要走的路，但我无法建立我将如何进行 - 请指导一下。

解决方案

我会实现一个FixedSizeQueue和popAll（） QueueStream到另一台计算机。可能有双缓冲，只是为了缓解网络/带宽问题。

The environment for the given objective is not currenly available, hence, I'm not able to try out things and have to rely on the analysis only !

My objective can be broken into the following distinct steps :

1. Uploading huge files(upto 100GB) using a dumb 'Upload File' page - there is no escape from this as the users want a (dumb)front-end and are not willing to ftp the file etc.
2. The web application which provides the above front end will be hosted on a low-end machine - 2GB RAM and 40GB HDD and this web application WILL NOT STORE any part of the huge file on the local machine but must 'quickly' write it to a high-end remote Linux machine

For each step, I'm highlighting my approach,concerns and queries :

• I referred this thread which confused me as I was planning to create a dumb web application using Spring MVC with an upload page - do I need to go into the HTML5 etc. or a simple web application will suffice?

• Given the 2GB RAM, the web application will get less than 1GB of it. I'm afraid that an 'OutOfMemoryError' is probable if the code is not written strictly - I have to ensure that from the stream, a small chunk, say 10MB must be read at a time and written to the remote Linux machine's file. Assuming that I am in the Controller Servlet's doPost(...), I did some reading about how to proceed and got confused :

        /**
           * @see HttpServlet#doPost(HttpServletRequest request, HttpServletResponse
           *      response)
           */
          protected void doPost(HttpServletRequest request,
                  HttpServletResponse response) throws ServletException, IOException {
              // TODO Auto-generated method stub
  
              InputStream fis = request.getInputStream();
              int i = 0;
  
              /* Approach - 1 : Plain old byte-by-byte method */
              Socket socket = new Socket("192.168.90.20", 22);
              OutputStream remoteOpStream = socket.getOutputStream();
  
              while ((i = fis.read()) != -1) {
                  remoteOpStream.write(i);
              }
  
              /* clean-up */
  
              /* Approach - 2 : NIO */
              ByteBuffer byteBuff = ByteBuffer.allocate(10000);/* read 10MB of data */
  
              ReadableByteChannel rdbyc = Channels.newChannel(request
                      .getInputStream());
  
              File remoteFile = new File("192.168.90.20/Remote_Linux_Folder");/*
                                                                               * Dunno
                                                                               * how
                                                                               * to
                                                                               * create
                                                                               * a
                                                                               * File
                                                                               * on a
                                                                               * remote
                                                                               * Linux
                                                                               * machine
                                                                               */
              FileOutputStream remoteFos = new FileOutputStream(remoteFile);
              FileChannel writableChannel = remoteFos.getChannel();
  
              while (true/* dunno how to loop till all the data is read! */) {
                  rdbyc.read(byteBuff);
                  writableChannel.write(byteBuff);
              }
  
              /* clean-up */
  
          }
  

I need some way wherein the data storage on the local machine is minimal - the code simply reads n bytes from the input stream and writes the same to a remote machine

I believe NIO is the way to go but I'm not able to establish as to how I must proceed - please guide about the same.

解决方案

I would implement a FixedSizeQueue and popAll() the data out of the QueueStream to another computer. Probably have it double buffered, just to provide the cushioning for network/bandwidth problems.

这篇关于使用Web应用程序上传大文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！