JupyterHub产生的Jupyter Notebook容器没有外部网络访问权限 [英] No outside network access for Jupyter Notebook container spawned by JupyterHub

问题描述

所以，这就是我想要实现的目标:

So, here is what I am trying to achieve:

• Jupyterhub服务器
• 在访问并且您未登录时会带您到另一个Web服务器(在Django中自定义编码)
• 该Web服务器使用OAuth验证用户身份
• 然后生成一个笔记本容器.
• 此笔记本容器必须预先填充一个令牌，该令牌由烘焙到笔记本Docker映像中的自定义库使用，以针对服务进行身份验证.
• 笔记本容器需要能够与Web服务器通信以进行进一步的交互，例如检索结果等.

除最后一部分外，我或多或少地实现了这一目标.我正在启动笔记本服务器，但无法访问外界.它只能访问Jupyter Hub(这就是它起作用的原因！)，而不能访问其他任何内容.

I have more or less achieved this except for the last part. I am getting a notebook server started but it has no access to the outside world. It can only access the Jupyter Hub (that's why it works!) and nothing else.

这是我与DockerSpawner相关的Jupyter Hub配置(由于这些功能按预期工作，因此我省略了OAuth设置.

Here is my Jupyter Hub config relevant to the DockerSpawner (I'm leaving out the OAuth settings since these work as expected.

# Tell JupyterHub that we want Docker Spawner to be used.
c.JupyterHub.spawner_class = 'dockerspawner.DockerSpawner'

# And what image should be used by the Docker Spawner
c.DockerSpawner.image = 'jupyter/scipy-notebook:7a0c7325e470'

# The Hub must listen on all interfaces.
c.JupyterHub.hub_ip = '0.0.0.0'

# And this should be the address of the Hub API
c.JupyterHub.hub_connect_ip = 'jupyterhub'

# Ask containers to connect to this network so that they can
# communicate with the Hub.
c.DockerSpawner.network_name = 'djangodockerjupyterdemo_default'

# And let's not make a mess, remove user containers when done.
c.DockerSpawner.remove = True

# We need to set the Notebook Directory
notebook_dir = '/home/jovyan/work'
c.DockerSpawner.notebook_dir = notebook_dir

# Need to tell where to mount the volumes.
c.DockerSpawner.volumes = { 'jupyterhub-user-{username}': notebook_dir }

请注意，由于项目目录的名称是docker-compose，因此正在创建djangodockerjupyterdemo_default. (我知道这不是最好的做法，但现在我只是希望有一个简单的示例可以工作.)

Please note that djangodockerjupyterdemo_default is being created by docker-compose thanks to the name of the project directory being such. (I know this is not the best thing to do but right now I'm just hoping to have a bare minimal example working.)

这是我的docker-compose:

Here is my docker-compose:

version: "2"

services:
  database:
      image: "mysql:5.6"
      volumes:
      - ./data:/var/lib/mysql
      environment:
      - MYSQL_ROOT_PASSWORD=test123
      - MYSQL_DATABASE=oauthserver
      - MYSQL_USER=oauthadmin
      - MYSQL_PASSWORD=test123
  webapp:
    image: auth_server:latest
    volumes:
      - ./:/app
    links:
      - database:database
    environment:
      - PYTHONUNBUFFERED=1
      - ENV=DEV
      - DATABASE_HOST=database
      - DATABASE_USER=oauthadmin
      - DATABASE_DBNAME=oauthserver
      - DATABASE_PASSWORD=test123
    hostname: oauthserver.ddi.in
  jupyterhub:
    image: "jupyterhub:test"
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:rw"
      - "./jupyterhub:/srv/jupyterhub"
    environment:
      - OAUTH2_AUTHORIZE_URL=http://oauthserver.ddi.in:8000/o/authorize
      - OAUTH2_TOKEN_URL=http://oauthserver.ddi.in:8000/o/token/
    hostname: jhtest.ddi.in
    links:
      - webapp:oauthserver.ddi.in

我使用 https://hub.docker.com/r/defreitas /dns-proxy-server 来访问JupyterHub服务器，方法是说" http://jhtest.ddi. in:8000 ".

I use https://hub.docker.com/r/defreitas/dns-proxy-server to access the JupyterHub server by saying "http://jhtest.ddi.in:8000".

现在，一旦容器装满，我可以确认以下内容:

Now, once the containers are up, here is what I can confirm:

• docker exec放入webapp或jupyterhub容器，然后wget从Internet上的某个位置进行文件复制.
• docker exec插入生成的Jupyter笔记本容器中并执行相同操作.尝试从笔记本内部使用requests.get()也是一样.

• docker execing into webapp or jupyterhub containers and then wgeting a file from some place on the Internet works.
• docker execing into the spawned Jupyter notebook container and doing the same doesn't. Same goes with trying to use requests.get() from inside the notebook.

如何使生成的笔记本进入外界?这对我的用例至关重要(我敢肯定，这是合理的期望).

How can I make the spawned notebook access the outside world? It's critical for my use case (and I'm sure a reasonable expectation).

PS:我注意到几乎没有关于自定义Django应用程序的OAuth JupyterHub设置的示例.我希望公开发布我的示例，希望它可以作为Jupyter Hub文档中的资源.

PS: I notice there are hardly any examples covering OAuth JupyterHub setup with a custom Django application out there. I hope to publish my example publicly and hopefully it can constitute as a resource on the Jupyter Hub docs.

推荐答案

因此我能够找到解决方案.我在下面总结一下.

So I was able to find the solution. I summarize it below.

docker-compose.yml的调整包括将network_mode: bridge添加到所有服务.这允许容器从本质上访问外部世界.但是，这样做的代价是容器无法通过简单的服务名称引用自动相互通信.但这可以使用链接轻松解决.

Adjustments to docker-compose.yml include adding a network_mode: bridge to all the services. This allows the containers to essentially access the outside world. The cost of doing so however is that the containers cannot automatically talk to each other via simple service name reference. But this can easily be solved using links.

下一个调整是将DockerSpawner配置为创建使用默认桥接网络而不是某些其他网络的容器.与此相关的设置包括:

The next adjustment was to configure the DockerSpawner to create containers that use the default bridge network instead of some other network. The settings that help with this include:

c.DockerSpawner.network_name = 'bridge'
c.DockerSpawner.use_internal_ip = True
c.DockerSpawner.extra_host_config = {'network_mode': 'bridge'}

此外，由于笔记本无法使用服务名称发现主JupyterHub，因此将c.JupyterHub.hub_connect_ip调整为JupyterHub服务的主机名.请注意，使用我的问题中提到的dns-proxy-server有助于将主机名解析为容器IP.

Also, since it is not possible for the notebook to discover the main JupyterHub using service name, I adjust the c.JupyterHub.hub_connect_ip to the hostname of the JupyterHub service. Note that the use of a dns-proxy-server mentioned in my question helps resolve the hostname to the container IP.

希望这可以帮助某个人.我很快将在我的博客上发布整个Django-OAuth-JupyterHub示例.

Hope this helps someone out there. I will be posting the whole Django-OAuth-JupyterHub example soon on my blog.

如上所述，我写了一篇博客文章，描述如何使JupyterHub使用OAuth和Django对用户进行身份验证.这是链接: https://vkaustubh.github.io/blog/geek/2020-02-08-integrating-jupytethub-with-django.html

As mentioned above, I have written a blog post describing how to make JupyterHub authenticate users using OAuth with Django. Here is the link: https://vkaustubh.github.io/blog/geek/2020-02-08-integrating-jupytethub-with-django.html

这篇关于JupyterHub产生的Jupyter Notebook容器没有外部网络访问权限的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！