是否可以与Google Compute Engine实例进行实时通信? [英] Is it possible to perform real-time communication with a Google Compute Engine instance?

问题描述

我想在笔记本电脑上运行一个程序(凉亭模拟器)，然后将图像数据流发送到GCE实例，该实例将通过对象检测网络运行，并以接近真实的速度发送回笔记本电脑.时间.这样的设置可行吗?

I would like to run a program on my laptop (Gazebo simulator) and send a stream of image data to a GCE instance, where it will be run through an object-detection network and sent back to my laptop in near real-time. Is such a set-up possible?

我现在最好的想法是，对于每张图片:

My best idea right now is, for each image:

1. 将图像另存为JPEG
2. 将JPEG流式传输到云存储桶
3. 从我的GCE实例访问存储桶并将文件传输到实例
4. 在我的python脚本中，将JPEG图像转换为numpy数组并通过对象检测网络运行
5. 将检测结果保存到文本文件中并传输到Cloud Storage存储桶
6. 从笔记本电脑访问存储桶并下载检测结果文件
7. 将检测结果文件转换为numpy数组以进行进一步处理

1. Save the image as a JPEG on my personal machine
2. Stream the JPEG to a Cloud Storage bucket
3. Access the storage bucket from my GCE instance and transfer the file to the instance
4. In my python script, convert the JPEG image to numpy array and run through the object detection network
5. Save the detection results in a text file and transfer to the Cloud Storage bucket
6. Access the storage bucket from my laptop and download the detection results file
7. Convert the detection results file to a numpy array for further processing

这似乎有很多步骤，我很好奇是否有加快速度的方法，例如减少保存和加载操作的次数或以更好的格式传输图像.

This seems like a lot of steps, and I am curious if there are ways to speed it up, such as reducing the number of save and load operations or transporting the image in a better format.

推荐答案

如果您的问题是是否可以建立这样的系统并实时执行这些操作?"那么我认为答案是肯定的.如果您的问题是如何减少执行上述操作的步骤数"，那么我不确定我能帮上忙，请教这里的一位专家，迫不及待想听听答案！

If your question is "is it possible to set up such a system and do those actions in real time?" then I think the answer is yes I think so. If your question is "how can I reduce the number of steps in doing the above" then I am not sure I can help and will defer to one of the experts on here and can't wait to hear the answer!

我已经实现了一个系统，该系统与您描述的用于外汇交易算法研究的系统类似(例如，将数据从我的笔记本电脑上传到存储设备，计算引擎工作人员提取数据并进行处理，将结果发布回存储设备以及我从笔记本电脑上下载了编译结果.

I have implemented a system that I think is similar to what you describe for research of Forex trading algorithms (e.g. upload data to storage from my laptop, compute engine workers pull the data and work on it, post results back to storage and I download the compiled results from my laptop).

我使用了 Google PubSub 体系结构-如果您已经对此有所了解，则表示歉意.它允许程序之间的近实时消息传递.例如，您可以在笔记本电脑上执行代码循环，以扫描寻找新图像的文件夹.当它们出现时，它会自动将文件上传到存储桶中，一旦存储在存储桶中，它就会向实例发送一条消息，告诉他们有新文件要处理，或者您可以使用更改通知"功能Google存储桶.实例可以完成工作，将结果发送回存储设备，并向笔记本电脑上运行的代码发送通知，表明工作已完成且可以提取结果.

I used the Google PubSub architecture - apologies if you have already read up on this. It allows near-realtime messaging between programs. For example you can have code looping on your laptop that scans a folder that looks out for new images. When they appear it automatically uploads the files to a bucket and once theyre in the bucket it can send a message to the instance(s) telling them that there are new files there to process, or you can use the "change notification" feature of Google Storage buckets. The instances can do the work, send the results back to the storage and send a notification to the code running on your laptop that work is done and results are available for pick-up.

请注意，我在上面为我的项目进行了设置，遇到了一些问题，以至于我放弃了PubSub.原因是用于PubSub的Python客户端库仅支持异步"消息提取，这似乎意味着订阅者将从队列中提取多个消息并并行处理它们.有一些功能可以帮助管理内置在API中的消息的流控制"，但是即使实现了这些功能，我也无法使其按我想要的方式工作.对于我的特定应用程序，我想按顺序处理所有内容，一次处理一个文件，因为对我来说很重要的一点是，我必须清楚实例正在执行的操作及其执行的顺序.Google搜索中有多个线程，StackOverflow和Google小组讨论了使用队列，类，为特定实例分配特定任务等问题的变通办法，尽管我尝试过，但即使这些也给我带来了问题.其中一些链接是:

Note that I set this up for my project above and encountered problems to the point that I gave up with PubSub. The reason was that the Python Client Library for PubSub only supports 'asynchronous' message pulls, which seems to mean that the subscribers will pull multiple messages from the queue and process them in parallel. There are some features to help manage 'flow control' of messages built into the API, but even with them implemented I couldn't get it to work the way I wanted. For my particular application I wanted to process everything in order, one file at a time because it was important to me that I'm clear what the instance is doing and the order its doing it in. There are several threads on google search, StackOverflow and Google groups that discuss workarounds for this using queues, classes, allocating specific tasks for specific instances, etc which I tried, but even these presented problems for me. Some of these links are:

使用Python客户端API在PubSub中运行同步提取和发布pubsub问题一次发送一条消息，如果您愿意的话，还有更多消息！

Run synchronous pull in PubSub using Python client API and pubsub problems pulling one message at a time and there are plenty more if you would like them!

您可能会发现，如果图像的处理速度相对较快，顺序并不是太重要，并且您不介意并行处理多个事情的实例并不能真正解决您的问题.

You may find that if the processing of an image is relatively quick, order isn't too important and you don't mind an instance working on multiple things in parallel that my problems don't really apply to your case.

仅供参考，我最后只是对我的工作人员实例"进行了一个简单的循环，每30秒扫描一次任务列表"存储区，或者寻找要处理的新文件的方式，但是显然，这并不是真正的-您最初正在寻找的时间方法.祝你好运！

FYI, I ended up just making a simple loop on my 'worker instances' that scans the 'task list' bucket every 30 seconds or whatever to look for new files to process, but obviously this isn't quite the real-time approach that you were originally looking for. Good luck!

这篇关于是否可以与Google Compute Engine实例进行实时通信?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！