使用docker，在Heroku上刮擦飞溅 [英] Using docker, scrapy splash on Heroku

问题描述

我有一个使用splash的抓爬式蜘蛛，该爬虫在Docker localhost：8050上运行，以在抓取之前呈现JavaScript。我正在尝试在heroku上运行此程序，但不知道如何配置heroku以在运行我的Web之前启动docker运行启动程序：scrapy crawl abc dyno。

I have a scrapy spider that uses splash which runs on Docker localhost:8050 to render javascript before scraping. I am trying to run this on heroku but have no idea how to configure heroku to start docker to run splash before running my web: scrapy crawl abc dyno. Any guides is greatly appreciated!

推荐答案

根据我的收集，您期望的是：

From what I gather you're expecting:

• 通过Docker容器在Heroku上运行的Splash实例


• 在Heroku dyno中运行的Web应用程序（Scrapy spider）

• 确保您可以拥有 docker CLI和 heroku CLI已安装


• 在Heroku的容器注册表-推送现有图像：
  
  
  
  
  • 确保已安装 docker CLI和 heroku CLI
  
  
  • heroku container：login
  
  
  • docker tag scrapinghub / splash Registry.heroku。 com /< app-name> / web
  
  
  • docker push Registry.heroku.com/<app-name>/web
  
  
  • 测试应用程序阳离子： heroku open -a< app-name> 。这应该允许您在此应用程序名称的Heroku主机上的端口8050上看到Splash UI。
    
    
    
    
    • 您可能需要确保将 $ PORT 正确设置为 EXPOSE 不遵守docker配置（ https://devcenter.heroku.com/articles/container-registry-and-runtime#dockerfile-commands-and-runtime ）
    
    
    • Ensure you can have docker CLI and heroku CLI installed
    • As seen in Heroku's Container Registry - Pushing existing image(s):
      • Ensure docker CLI and heroku CLI are installed
      • heroku container:login
      • docker tag scrapinghub/splash registry.heroku.com/<app-name>/web
      • docker push registry.heroku.com/<app-name>/web
      • To test the application: heroku open -a <app-name>. This should allow you to see the Splash UI at port 8050 on the Heroku host for this app name.
        • You may need to ensure $PORT is set appropriately as the EXPOSE docker configuration is not respected (https://devcenter.heroku.com/articles/container-registry-and-runtime#dockerfile-commands-and-runtime)
        
        
        • 配置您的应用程序以使其指向< app-host-name>：8050 。现在Scrapy蜘蛛应该可以请求以前运行的Splash实例。
        
        
        • Configure your application to point to <app-host-name>:8050. And the Scrapy spider should now be able to request to the Splash instance previously run.

        这篇关于使用docker，在Heroku上刮擦飞溅的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！