在NodeJS服务器之间发送文件最有效的方法是什么？ [英] What is the most efficient way of sending files between NodeJS servers?
问题描述
- 服务器A是一个API服务器，它接收请求并将其保存到MongoDB数据库（文件存储为缓冲区及其详细信息作为其他字段） / li>
- 服务器F提供表单，处理表单并将表单的数据发送到服务器A.
在接收服务器是Express API的两个NodeJS服务器之间发送文件最有效的方法是什么？文件大小在哪里很重要？
1。 HTTP方式
如果我发送的文件是PDF文件（不超过50mb），通过HTTP将整个内容作为字符串发送是否有效？
算法如下：
- 服务器F使用< a href =https://www.npmjs.com/package/multer =nofollow noreferrer> https://www.npmjs.com/package/multer 并保存文件 然后，服务器F读取此文件，并通过 https://github.com发出HTTP请求/请求/请求以及文件的一些细节
- 服务器A收到此请求，并将文件内容从字符串转换为缓冲区，并将文件与文件一起保存在MongoDB中详细信息。
在此算法中，服务器A（存储到MongoDB时）和服务器F（当它发送到服务器A）已将文件读入内存，两台服务器之间的请求大小与文件大小相同。 （是否有50Mb请求？）
然而，需要考虑的一点是，使用这种方法 - 我将使用API的ExpressJS风格进行整个过程将与路由中的 / list ， / details 请求的其余部分保持一致。我喜欢一致性。
2。 Socket.IO方式
与此算法相反，我探索了 https://github.com/nkzawa/socket.io-stream 的方式，破坏了服务器A上HTTP API的一致性（因为socket.io事件的处理程序是定义不在路由中，但是具有 var server = http.createServer（app）; ）的文件。
服务器F在 routes / some_route.js 中处理表单数据：
router.post（'/'，multer（{dest：'./uploads/'}).single('file'），function（req， res）{
var api_request = {};
api_request.name = req.body.name;
//将其他字段添加到api_request ...
var has_file = req.hasOwnProperty（'file'）;
var io = require（'socket.io-client'）;
var transaction_sent = false;
var socket = io.connect（'http：// localhost：3000'）;
socket.on（'connect'，function（）{
console.log（socket connected to 3000）;
if（transaction_sent === false）{
var ss = require（'socket.io-stream'）;
var stream = ss.createStream（）;
ss（socket）.emit（'transaction new'，stream，api_request）;
if（has_file）{
var fs = require（'fs' ;
var filename = req.file.destination + req.file.filename;
console.log（'发送文件：'，文件名）;
fs.createReadStream（filename）.pipe（stream）;
}
如果（！has_file）{
console.log（'send without file'）;
}
transaction_sent = true;
//通过socket获取响应
socket.on（'transaction new sent'，function（data）{
console.log（'response from 3000：'，data
console.log（'Closed socket to 3000'）;
}）;
}
}）;
}）;
我说我会处理PDF文件， 50MB。但是，如果将来使用这个程序发送较大的文件，socket.io是一个更好的处理1GB文件的方式，因为它使用流？
此方法发送文件和细节，但我是这个图书馆的新人，不知道是否应该用于这个目的，或者如果有更好的方法利用它。
最终想法
我应该探索哪些其他方法？
- 应该我通过SCP发送文件，并发出HTTP请求，其中包含文件的详细信息，包括我发送的文件，从而分离文件和API请求的协议。
- 我应该始终使用流，因为他们不将整个文件存储到内存中？ （这是他们的工作方式，对吗？）
- 此 https：/ /github.com/liamks/Delivery.js ？
参考资料：
- 文件/数据传输在两个node.js服务器之间这让我尝试socket-stream方式。
-
有很多方法可以实现这一点，但没有太多的做法吧！
套接字io和wesockets是有效的，当您使用浏览器，但由于您没有，没有必要。
您可以尝试的第一种方法是使用nodejs的内置Net模块，ba在这种情况下，它将在服务器之间建立tcp连接并传递数据。
你还应该记住，你需要发送大量的数据而不是整个文件， net模块的 socket.write 方法似乎非常适合您的情况检查： https://nodejs.org/api/net.html
但根据文件大小和并发性，内存消耗可能相当大。
如果您在两台服务器上都运行linux，甚至可以使用简单的linux命令scp
nohup scp -rpC / var / www / httpdocs / * remote_user@remote_domain.com：/ var / www / httpdocs&
您甚至可以使用Windows到Linux或其他方式。
http：//www.chiark。 greenend.org.uk/~sgtatham/putty/download.html
Windows的客户端scp是pscp.exe
希望这有帮助！
Introduction
Say that on the same local network we have two Node JS servers set up with Express: Server A for API and Server F for form.
- Server A is an API server where it takes the request and saves it to MongoDB database (files are stored as Buffer and their details as other fields)
- Server F serves up a form, handles the form post and sends the form's data to Server A.
What is the most efficient way to send files between two NodeJS servers where the receiving server is Express API? Where does the file size matter?
1. HTTP Way
If the files I'm sending are PDF files (that won't exceed 50mb) is it efficient to send the whole contents as a string over HTTP?
Algorithm is as follows:
- Server F handles the file request using https://www.npmjs.com/package/multer and saves the file
- then Server F reads this file and makes an HTTP request via https://github.com/request/request along with some details on the file
- Server A receives this request and turns the file contents from string to Buffer and saves a record in MongoDB along with the file details.
In this algorithm, both Server A (when storing into MongoDB) and Server F (when it was sending it over to Server A) have read the file into the memory, and the request between the two servers was about the same size as the file. (Are 50Mb requests alright?)
However, one thing to consider is that -with this method- I would be using the ExpressJS style of API for the whole process and it would be consistent with the rest of the app where the /list, /details requests are also defined in the routes. I like consistency.
2. Socket.IO Way
In contrast to this algorithm, I've explored https://github.com/nkzawa/socket.io-stream way which broke away from the consistency of the HTTP API on Server A (as the handler for socket.io events are defined not in the routes but the file that has var server = http.createServer(app);).
Server F handles the form data as such in routes/some_route.js:
router.post('/', multer({dest: './uploads/'}).single('file'), function (req, res) {
var api_request = {};
api_request.name = req.body.name;
//add other fields to api_request ...
var has_file = req.hasOwnProperty('file');
var io = require('socket.io-client');
var transaction_sent = false;
var socket = io.connect('http://localhost:3000');
socket.on('connect', function () {
console.log("socket connected to 3000");
if (transaction_sent === false) {
var ss = require('socket.io-stream');
var stream = ss.createStream();
ss(socket).emit('transaction new', stream, api_request);
if (has_file) {
var fs = require('fs');
var filename = req.file.destination + req.file.filename;
console.log('sending with file: ', filename);
fs.createReadStream(filename).pipe(stream);
}
if (!has_file) {
console.log('sending without file.');
}
transaction_sent = true;
//get the response via socket
socket.on('transaction new sent', function (data) {
console.log('response from 3000:', data);
//there might be a better way to close socket. But this works.
socket.close();
console.log('Closed socket to 3000');
});
}
});
});
I said I'd be dealing with PDF files that are < 50Mb. However, if I use this program to send larger files in the future, is socket.io a better way to handle 1GB files as it's using stream?
This method does send the file and the details across but I'm new to this library and don't know if it should be used for this purpose or if there is a better way of utilizing it.
Final thoughts
What alternative methods should I explore?
- Should I send the file over SCP and make an HTTP request with file details including where I've sent it- thus, separating the protocols of files and API requests?
- Should I always use streams because they don't store the whole file into memory? (that's how they work, right?)
- This https://github.com/liamks/Delivery.js ?
References:
- File/Data transfer between two node.js servers this got me to try socket-stream way.
- transfer files between two node.js servers over http for HTTP way
There are plenty of ways to achieve this , but not so much to do it right !
socket io and wesockets are efficient when you use them with a browser , but since you don't , there is no need for it.
The first method you can try is to use the builtin Net module of nodejs, basically it will make a tcp connection between the servers and pass the data.
you should also keep in mind that you need to send chunks of data not the entire file , the socket.write method of the net module seems to be a good fit for your case check it : https://nodejs.org/api/net.html
But depending on the size of your files and concurrency , memory consumption can be quite large.
if you are running linux on both servers you could even send the files at ground zero with a simple linux command called scp
nohup scp -rpC /var/www/httpdocs/* remote_user@remote_domain.com:/var/www/httpdocs &
You can even do this with windows to linux or the other way.
http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html
the client scp for windows is pscp.exe
Hope this helps !
这篇关于在NodeJS服务器之间发送文件最有效的方法是什么？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
