无法使用gethue/hue docker映像配置HDFS地址 [英] cannot configure HDFS address using gethue/hue docker image

问题描述

我正在尝试从gethue/hue获取Hue docker映像，但似乎忽略了我给他的配置，总是在localhost而不是我要他寻找的docker容器上寻找HDFS.

I'm trying to get the Hue docker image from gethue/hue, but it seems to ignore the configuration I give him and always look for HDFS on localhost instead of the docker container I ask him to look for.

以下是一些上下文:

1. 我正在使用以下docker compose启动HDFS集群:

 hdfs-namenode:
    image: bde2020/hadoop-namenode:1.1.0-hadoop2.7.1-java8
    hostname: namenode
    environment:
      - CLUSTER_NAME=davidov
    ports:
      - "8020:8020"
      - "50070:50070"
    volumes:
      - ./data/hdfs/namenode:/hadoop/dfs/name
    env_file:
      - ./hadoop.env


  hdfs-datanode1:
    image: bde2020/hadoop-datanode:1.1.0-hadoop2.7.1-java8
    depends_on:
      - hdfs-namenode
    links:
      - hdfs-namenode:namenode
    volumes:
      - ./data/hdfs/datanode1:/hadoop/dfs/data
    env_file:
      - ./hadoop.env

这会启动来自BigDataEurope的图像，这些图像已经正确配置，包括:

This launches images from BigDataEurope, which are already properly configured, including:

- the activation of webhdfs (in /etc/hadoop/hdfs-site.xml):
  - dfs.webhdfs.enabled set to true
- the hue proxy user (in /etc/hadoop/core-site.xml): 
  - hadoop.proxyuser.hue.hosts set to *
  - hadoop.proxyuser.hue.groups set to *

我按照他们的说明启动色相:

首先，我在docker容器内启动一个bash提示:

First, I launch a bash prompt inside the docker container:

docker run -it -p 8888:8888 gethue/hue:latest bash

然后，我修改desktop/conf/pseudo-distributed.ini以指向正确的hadoop节点"(在我的情况下是地址为172.30.0.2的docker容器:

Then, I modify desktop/conf/pseudo-distributed.ini to point to the correct hadoop "node" (in my case a docker container with the address 172.30.0.2:

[hadoop]

  # Configuration for HDFS NameNode
  # ------------------------------------------------------------------------
  [[hdfs_clusters]]
    # HA support by using HttpFs

    [[[default]]]
      # Enter the filesystem uri
      fs_defaultfs=hdfs://172.30.0.2:8020

      # NameNode logical name.
      ## logical_name=

      # Use WebHdfs/HttpFs as the communication mechanism.
      # Domain should be the NameNode or HttpFs host.
      # Default port is 14000 for HttpFs.
      ## webhdfs_url=http://172.30.0.2:50070/webhdfs/v1

      # Change this if your HDFS cluster is Kerberos-secured
      ## security_enabled=false

      # In secure mode (HTTPS), if SSL certificates from YARN Rest APIs
      # have to be verified against certificate authority
      ## ssl_cert_ca_verify=True

然后我使用以下命令启动hue(仍在hue容器中):

And then I launch hue using the following command (still inside the hue container):

./build/env/bin/hue runserver_plus 0.0.0.0:8888

然后，我将浏览器指向localhost:8888，创建一个新用户(在本例中为"hdfs")，然后启动HDFS文件浏览器模块.然后，我收到以下错误消息:

I then point my browser to localhost:8888, create a new user ('hdfs' in my case), and launch the HDFS file browser module. I then get the following error message:

Cannot access: /user/hdfs/.
HTTPConnectionPool(host='localhost', port=50070): Max retries exceeded with url: /webhdfs/v1/user/hdfs?op=GETFILESTATUS&user.name=hue&doas=hdfs (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 99] Cannot assign requested address',))

有趣的一点是，即使我修改了它的配置文件使其指向172.30.0.2，它仍然尝试连接到 localhost (这当然是行不通的).

The interesting bit is that it still tries to connect to localhost (which of course cannot work), even though I modified its config file to point to 172.30.0.2.

谷歌搜索问题，我发现了另一个配置文件:desktop/conf.dist/hue.ini.我尝试修改此设置并再次启动色相，但结果相同.

Googling the issue, I found another config file: desktop/conf.dist/hue.ini. I tried modifying this one and launching hue again, but same result.

有人知道我该如何正确配置色相吗?

Does any one know how I could correctly configure hue in my case?

预先感谢您的帮助.

此致

洛朗.

推荐答案

您的一次性docker run命令与docker-compose容器不在同一网络上.

Your one-off docker run command is not on the same network as the docker-compose containers.

您将需要类似的内容，将[projectname]替换为在其中启动docker-compose up的文件夹

You would need something like this, replacing [projectname] with the folder you started docker-compose up in

docker run -ti -p 8888:8888 --network="[projectname]_default" gethue/hue bash

我建议还将Docker Compose用于desktop/conf/下可以简单指定

I would suggest using Docker Compose also for the Hue container and volume mount for a INI files under desktop/conf/ that you can specify simply

fs_defaultfs=hdfs://namenode:8020

(因为您将hostname: namenode放在了撰写文件中)

(since you put hostname: namenode in the compose file)

您还需要取消注释WebHDFS行以使更改生效

You'll also need to uncomment the WebHDFS line for your changes to take affect

所有INI文件都已合并到conf色相文件夹中

All INI files are merged in the conf folder for Hue

这篇关于无法使用gethue/hue docker映像配置HDFS地址的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！