在 Heroku 上使用 docker，scrapy 飞溅 [英] Using docker, scrapy splash on Heroku

问题描述

我有一个爬虫爬虫，它使用在 Docker localhost:8050 上运行的飞溅来在爬取之前呈现 javascript.我正在尝试在 heroku 上运行它，但不知道如何配置 heroku 以在运行我的网站之前启动 docker 以运行 splash:scrapy crawl abc dyno.非常感谢任何指南！

I have a scrapy spider that uses splash which runs on Docker localhost:8050 to render javascript before scraping. I am trying to run this on heroku but have no idea how to configure heroku to start docker to run splash before running my web: scrapy crawl abc dyno. Any guides is greatly appreciated!

推荐答案

据我所知，您正在期待:

From what I gather you're expecting:

• 通过 Docker 容器在 Heroku 上运行的 Splash 实例
• 在 Heroku dyno 中运行的 Web 应用程序(Scrapy spider)

• 确保您可以安装 docker CLI 和 heroku CLI
• 如 Heroku 的 Container Registry - Pushing 中所见现有图像:
  • 确保安装了 docker CLI 和 heroku CLI
  • heroku 容器:登录
  • docker tag scrapinghub/splash registry.heroku.com//web
  • docker push registry.heroku.com//web
  • 测试应用程序:heroku open -a .这应该允许您在 Heroku 主机上的端口 8050 上看到此应用程序名称的 Splash UI.
    • 您可能需要确保 $PORT 设置适当，因为 EXPOSE docker 配置不受重视(https://devcenter.heroku.com/articles/container-registry-and-runtime#dockerfile-commands-and-运行时)
    • Ensure you can have docker CLI and heroku CLI installed
    • As seen in Heroku's Container Registry - Pushing existing image(s):
      • Ensure docker CLI and heroku CLI are installed
      • heroku container:login
      • docker tag scrapinghub/splash registry.heroku.com/<app-name>/web
      • docker push registry.heroku.com/<app-name>/web
      • To test the application: heroku open -a <app-name>. This should allow you to see the Splash UI at port 8050 on the Heroku host for this app name.
        • You may need to ensure $PORT is set appropriately as the EXPOSE docker configuration is not respected (https://devcenter.heroku.com/articles/container-registry-and-runtime#dockerfile-commands-and-runtime)
        • 将您的应用程序配置为指向 :8050.Scrapy 蜘蛛现在应该能够向之前运行的 Splash 实例发出请求.
        • Configure your application to point to <app-host-name>:8050. And the Scrapy spider should now be able to request to the Splash instance previously run.

        这篇关于在 Heroku 上使用 docker，scrapy 飞溅的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！