将TensorflowJS模型保存到Firebase存储的最佳实践? [英] Best practice to save a TensorflowJS model to firebase storage?
问题描述
似乎有可能进行发布"基于http请求的保存.
Seems possible to do a "post" http request based save.
https://www.tensorflow.org/js/guide/save_load
https://firebase.google.com/docs/functions/http-events
担心我可能会使其复杂化，是否有一种简单的方法可以在客户端js中创建json文件(也许也避免保存到本地存储中?)，然后使用put函数简单地上传到存储中? https://firebase.google.com/docs/storage/web/upload-文件
Worried that I may be overcomplicating this, is there a simple way to perhaps create the json files in the client side js (maybe avoiding saving into local storage as well?), then simply upload to storage using put function? https://firebase.google.com/docs/storage/web/upload-files
在这些领域经验不足，找不到针对此场景的特定内容.
Not very experienced in these areas and could not find anything specific to this scenario.
调查:
似乎这是在使用云功能 从HTTP函数将图像文件写入Firebase存储
seems this is using cloud functions Write image file to Firebase Storage from HTTP function
更新:
当前正在研究使用基于内置Tensorflowjs HTTP请求的保存(具有权重的bin文件和模型结构json文件)，该保存将由云功能处理，而不是确定这是最好的方法还是行得通，但到目前为止，我仍然是最好的领导者.现在也正在研究如何通过http请求发送文件夹名称?
Currently looking into using built in Tensorflowjs HTTP request based save (both bin file with weights and model structure json file) that will be handled by a cloud function, not sure if this is the best way to go or if it will work, but best lead I have so far. Also now looking into how to send folder name through http request?
推荐答案
我的客户端Tensorflow JS请求通过HTTP请求保存到云存储.也许有更好的方法可以做到这一点，但这至少对我有用.
My client side Tensorflow JS request to save to cloud storage via HTTP request. There is probably a better way to do this but this at least has worked for me.
var api = 'https://us-central1-your-uniquelocation.cloudfunctions.net/uploadModel';
var res = await model.save(tf.io.browserHTTPRequest(api,
{method: 'POST', headers:{'Authorization':'test','Content-Type': 'multipart/form-data; boundary=----WebKitFormBoundaryVoB3gVGBQHG0btQR'}}));
console.log(res);
然后将 index.js用于我的云函数
const functions = require('firebase-functions');
var admin = require("firebase-admin");
var serviceAccount = require("./service-account-credentials.json");
admin.initializeApp({
credential: admin.credential.cert(serviceAccount),
//databaseURL: "https://yoururl.firebaseio.com",
storageBucket: "yoururl.appspot.com"
});
var bucket = admin.storage().bucket("yoururl.appspot.com");
const path = require('path');
const os = require('os');
const fs = require('fs');
const Busboy = require('busboy');
exports.uploadModel = functions.https.onRequest((req, res) => {
res.set('Access-Control-Allow-Origin', '*');
res.set('Access-Control-Allow-Credentials', 'true');
if (req.method === 'OPTIONS') {
// Send response to OPTIONS requests
console.log("send response 204");
res.set('Access-Control-Allow-Methods', 'GET');
res.set('Access-Control-Allow-Headers', 'Authorization');
res.set('Access-Control-Max-Age', '3600');
res.status(204).send();
}
console.log("metho--->"+req.method);
console.log(req.headers);
const busboy = new Busboy({headers: req.headers});
const tmpdir = os.tmpdir();
// This object will accumulate all the fields, keyed by their name
const fields = {};
// This object will accumulate all the uploaded files, keyed by their name.
const uploads = {};
// This code will process each non-file field in the form.
busboy.on('field', (fieldname, val) => {
// TODO(developer): Process submitted field values here
console.log(`Processed field ${fieldname}: ${val}.`);
fields[fieldname] = val;
});
const fileWrites = [];
//res.status(200).send();
// This code will process each file uploaded.
busboy.on('file', (fieldname, file, filename) => {
// Note: os.tmpdir() points to an in-memory file system on GCF
// Thus, any files in it must fit in the instance's memory.
console.log(`Processed file ${filename}`);
const filepath = path.join(tmpdir, filename);
uploads[fieldname] = filepath;
const writeStream = fs.createWriteStream(filepath);
file.pipe(writeStream);
// File was processed by Busboy; wait for it to be written to disk.
const promise = new Promise((resolve, reject) => {
file.on('end', () => {
writeStream.end();
});
writeStream.on('finish', resolve);
writeStream.on('error', reject);
});
fileWrites.push(promise);
});
// Triggered once all uploaded files are processed by Busboy.
// We still need to wait for the disk writes (saves) to complete.
busboy.on('finish', () => {
Promise.all(fileWrites).then(() => {
// TODO(developer): Process saved files here
for (const name in uploads) {
const file = uploads[name];
bucket.upload(file).then(()=>{
fs.unlinkSync(file);
console.log("saved");
});
}
res.send();
});
});
busboy.end(req.rawBody);
});
更新:
Update:
增加了通过http请求发送文件夹名称的功能，这也许再一次不完美，但是可以正常工作，最初很难弄清楚.
Added functionality to send folder names through the http request as well, once again probably not perfect but works, originally pretty hard to figure out.
针对我的云功能的新 index.js ，适用于那些希望通过http请求发送自定义文件夹名称的人.
New index.js for my cloud functions for those who want custom folder names sent through http request.
const functions = require('firebase-functions');
var admin = require("firebase-admin");
var serviceAccount = require("./service-account-credentials.json");
admin.initializeApp({
credential: admin.credential.cert(serviceAccount),
storageBucket: "YOURURL.appspot.com"
});
var bucket = admin.storage().bucket();
const path = require('path');
const os = require('os');
const fs = require('fs');
const Busboy = require('busboy');
exports.uploadModel2s = functions.https.onRequest((req, res) => {
res.set('Access-Control-Allow-Origin', '*');
res.set('Access-Control-Allow-Credentials', 'true');
if (req.method === 'OPTIONS'){
console.log("send response 204");
res.set('Access-Control-Allow-Methods', 'GET');
res.set('Access-Control-Allow-Headers', 'Authorization,ModelId');
res.set('Access-Control-Max-Age', '3600');
return res.status(204).send();
}
console.log("method ---> "+req.method);
const modelid = req.headers['modelid'];
console.log("modelid ---> "+modelid);
console.log(req.headers);
const busboy = new Busboy({headers: req.headers});
const tmpdir = os.tmpdir();
const fields = {};
const uploads = {};
busboy.on('field', (fieldname, val) => {
console.log(`Processed field ${fieldname}: ${val}.`);
fields[fieldname] = val;
});
const fileWrites = [];
busboy.on('file', (fieldname, file, filename) => {
console.log(`Processed 2 file ${modelid} ${filename}`);
const filepath = path.join(tmpdir,filename);
uploads[fieldname] = filepath;
console.log('PATH: '+filepath);
const writeStream = fs.createWriteStream(filepath);
file.pipe(writeStream);
const promise = new Promise((resolve, reject) => {
file.on('end', () => {
writeStream.end();
});
writeStream.on('finish', resolve);
writeStream.on('error', reject);
});
fileWrites.push(promise);
});
busboy.on('finish', () => {
Promise.all(fileWrites).then(() => {
for (const name in uploads){
const file = uploads[name];
const destpath = "/m_"+modelid+"/"+name;
console.log("SAVE ME HERE: "+destpath);
bucket.upload(file,{ destination: destpath,public: true }).then(() => {
if(name == "model.weights.bin"){
fs.unlinkSync(file);
console.log("save weights");
res.send();
}else{
console.log("save model");
}
});
}
});
});
busboy.end(req.rawBody);
});
新客户代码
var api = 'https://yourproject.cloudfunctions.net/uploadModel2s';
var res = await model.save(tf.io.browserHTTPRequest(api,
{method: 'POST', headers:{'Authorization':'test','ModelId':m}}));
安全风险
为了加快开发速度，我在上传时使用了 public:true ，因此我可以轻松访问文件以进行下载，但这绝对是不安全的.
In order to speed development I am using public: true on upload so I can access file easily for download, however this is definitely not secure.
这篇关于将TensorflowJS模型保存到Firebase存储的最佳实践?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
