在Google App Engine上运行gcsfuse时出现问题 [英] Problem running gcsfuse on Google App Engine

问题描述

我正在尝试在App Engine Flexible上运行Airflow Webserver，但是要使其正常工作，我需要安装一个GCS存储桶.我正在使用自定义运行时. 我这样做的原因是为了获得App Engine与IAP一起提供的安全端点.

I am trying to run Airflow Webserver on App Engine Flexible however for it to work I need a mounted GCS bucket. I am using custom runtime. The reason why I am doing it is to get a secured endpoint that app Engine provides together with IAP.

我的app.yaml是一个简单的文件，带有服务名称，环境和运行时 我的Dockerfile有很多apt-get install，在CMD中有gcsfuse安装和运行气流Web服务器，这没什么大不了的.

My app.yaml is a simple file with service name, env and runtime My Dockerfile is a lots of apt-get installs and in CMD there is gcsfuse mounting and running airflow webserver, it is not a big deal.

在App Engine中尝试使用gcsfuse时遇到的错误是:

The error I am getting when trying to use gcsfuse in App Engine is:

daemonize.Run: readFromProcess: sub-process: mountWithArgs: mountWithConn: Mount: mount: running fusermount: exit status 1

stderr:
fusermount: fuse device not found, try 'modprobe fuse' first

我知道Google Composer存在，但是对于我的需求来说太贵了.因此，我更喜欢在GAE上使用调度程序和Web服务器创建一个VM，共享一个GCS存储桶，类似于Composer提供的功能，但没有HA以及我想运行的简单东西的所有疯狂费用.

I know that Google Composer exists but it is way too expensive for my needs. So I prefer to create a VM with a scheduler and webserver on GAE, sharing a GCS bucket, similar to what Composer gives but without all that HA and insane cost for simple things I want to run.

我正在App Engine上进行搜索，到目前为止，由于某种原因，我找到的所有答案都提到了GKE.

I am searching to do this in App Engine, all the answers I have found so far mention GKE for some reason.

我知道这是一个特权问题，但是在App Engine中，我看不到任何设置特权的选项，这样做的方式将非常有帮助.

I know it is a privilege problem, however in App Engine I do not see any option to set privileges, a way to do it would be very helpful.

甚至可以在App Engine上做我想做的事吗?

Is is even possible to do what I want to do on App Engine?

推荐答案

这是可能的.我将向您展示如何手动执行操作，您可能需要利用shell脚本来处理多个实例.

This is possible. I'll show you how to do it manually, you might need to utilize shell script to deal with multiple instances.

1. 定义本手册中使用的几个变量

service=YOUR_APPENGINE_VERSION
version=YOUR_APPENGINE_VERSION
project=PROJECTID

1. 获取实例列表

gcloud app instances list  --project $project
SERVICE  VERSION          ID                                VM_STATUS  DEBUG_MODE
default  ***************  instance-id-1                     RUNNING    YES
default  ***************  instance-id-2                     RUNNING

1. ssh进入一个实例

gcloud app instances ssh instance-id-1 --service $service --version $version --project $project

1. 获取图像ID

docker ps | grep gaeapp | awk '{print $2}'

您将获得一个图像ID

you will get an imageid

1. 获取gaeapp
的环境

1. get env of gaeapp

docker exec gaeapp env
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=*****
GAE_MEMORY_MB=614
GAE_INSTANCE=****
GAE_SERVICE=default
PORT=8080
GCLOUD_PROJECT=*****
GAE_VERSION=*****
GOOGLE_CLOUD_PROJECT=*****

1. 使用特权重新启动gaeapp

docker rm -f gaeapp
docker run --privileged -d -p 8080:8080 --name gaeapp -e GAE_MEMORY_MB=614 -e GAE_INSTANCE=instance-id-1 -e GAE_SERVICE=$service -e PORT=8080 -e GCLOUD_PROJECT=$project -e GAE_VERSION=$version -e GOOGLE_CLOUD_PROJECT=$project $imageid

1. 输入gaeapp(假设您已安装gcsfuse并具有服务帐户密钥json:/test-service-account.json)

$ docker exec -it gaeapp bash
[in gaeapp] # GOOGLE_APPLICATION_CREDENTIALS=/test-service-account.json gcsfuse BUCKET /mnt/
Using mount point: /mnt
Opening GCS connection...
Opening bucket...
Mounting file system...
File system has been successfully mounted.

这篇关于在Google App Engine上运行gcsfuse时出现问题的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！