Google计算引擎为hadoop仪表板添加防火墙规则 [英] google compute engine add firewall rule for hadoop dashboard

问题描述

我使用bdutil安装hadoop集群（而不是单击来部署）。我无法访问locahost上的作业跟踪器页面：50030 / jobtracker.jsp（ https ：//cloud.google.com/hadoop/running-a-mapreduce-job ）

我在本地使用lynx而不是从客户端进行检查浏览器（所以本地主机，而不是外部IP）

我在bdutil的配置文件中的设置是

MASTER_UI_PORTS = （'8088''50070''50030'）

但在执行防火墙规则列表后，部署hadoop集群后，我得到以下内容：

NAME NETWORK SRC_RANGES RULES SRC_TAGS TARGET_TAGS

default-allow-http默认值0.0.0.0/0 tcp：80，tcp：8080 http-server p>

default-allow-https默认0.0.0.0/0 tcp：443 https-server

default-allow-icmp默认0.0.0.0/0 icmp

默认允许内部默认值10.240.0.0/16 tcp：1-65535，udp：1-65535，icmp

default-allow-rdp默认值0.0 .0.0 / 0 tcp：3389

default-allow-ssh default 0.0.0.0/0 tcp：22

现在我没有看到规则列表中的端口50030。为什么这样？

所以我运行一个命令来添加它们（手动）

gcloud compute防火墙规则创建allow-http --description允许传入的http。 --allow tcp：50030 --format json

现在它被添加，我可以在防火墙规则列表命令的输出中看到。

但是当我做lynx locahost：50030 / jobtracker.jsp时，我无法连接。然后，我运行一个hadoop工作，以便有一些输出来查看，然后运行lynx命令，但仍然无法连接。

有人可以告诉我我要去哪里在这个完整的过程中出错？

解决方案

一个短暂的IP 是外部IP。短暂IP和静态IP之间的区别在于静态IP可以被重新分配给另一个虚拟机实例，而当实例被销毁时，临时IP被释放。通过Web UI或gcloud命令行工具，可以将短暂IP升级为静态IP。

您可以通过查询元数据来获取主机的外部IP API位于 http://169.254.169.254/0.1/meta-data/network 。响应将是一个JSON文档，看起来像这样（为了清楚起见，打印出来的文字很漂亮）：

  {
networkInterface ：[
 {
network：projects / 852299914697 / networks / rabbit，
ip：10.129.14.59，
accessConfiguration：[
 {
externalIp：107.178.223.11，
type：ONE_TO_ONE_NAT
} 
] 
} 
] 
} 
  

防火墙规则命令看似合理，但您可能想要选择更具描述性的名称。如果我看到 allow-http 的规则，我会认为它意味着端口80.您可能还想将其限制为放置在Hadoop仪表板实例上的目标标记;按照书面规定，您的规则将允许该端口访问当前项目中的所有实例。

I installed hadoop cluster using bdutil (instead of click to deploy). I am not able to access job tracker page at locahost:50030/jobtracker.jsp (https://cloud.google.com/hadoop/running-a-mapreduce-job)

I am checking it locally using lynx instead of from my client browser (so localhost instead of external ip)

My setting in my config file for bdutil is

MASTER_UI_PORTS=('8088' '50070' '50030')

but after deploying the hadoop cluster when I do firewall rules list I get following

NAME NETWORK SRC_RANGES RULES SRC_TAGS TARGET_TAGS

default-allow-http default 0.0.0.0/0 tcp:80,tcp:8080 http-server

default-allow-https default 0.0.0.0/0 tcp:443 https-server

default-allow-icmp default 0.0.0.0/0 icmp

default-allow-internal default 10.240.0.0/16 tcp:1-65535,udp:1-65535,icmp

default-allow-rdp default 0.0.0.0/0 tcp:3389

default-allow-ssh default 0.0.0.0/0 tcp:22

Now I dont see port 50030 in the list of rules. Why so?

so I run a command to add them (manually)

gcloud compute firewall-rules create allow-http --description "Incoming http allowed." --allow tcp:50030 --format json

Now it gets added and I can see in the output of firewall-rules list command.

But still when I do lynx locahost:50030/jobtracker.jsp I get unable to connect. Then, I run a hadoop job so that there is some output to view and then run lynx command but still see unable to connect.

Can someone tell me where I am going wrong in this complete process?

解决方案

An ephemeral IP is an external IP. The difference between an ephemeral IP and a static IP is that a static IP can be reassigned to another virtual machine instance, while an ephemeral IP is released when the instance is destroyed. An ephemeral IP can be promoted to a static IP through the web UI or the gcloud command-line tool.

You can obtain the external IP of your host by querying the metadata API at http://169.254.169.254/0.1/meta-data/network. The response will be a JSON document that looks like this (pretty-printed for clarity):

{
   "networkInterface" : [
      {
         "network" : "projects/852299914697/networks/rabbit",
         "ip" : "10.129.14.59",
         "accessConfiguration" : [
            {
               "externalIp" : "107.178.223.11",
               "type" : "ONE_TO_ONE_NAT"
            }
         ]
      }
   ]
}

The firewall rule command seems reasonable, but you may want to choose a more descriptive name. If I saw a rule that said allow-http, I would assume it meant port 80. You may also want to restrict it to a target tag placed on your Hadoop dashboard instance; as written, your rule will allow access on that port to all instances in the current project.

这篇关于Google计算引擎为hadoop仪表板添加防火墙规则的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！