将一个巨大的字符串作为参数传递给 Sidekiq 工人是个好主意吗? [英] Is it a good idea to pass a huge string as an argument to Sidekiq worker?

问题描述

我正在开发一个 scraper，它可以浏览网站并在 Sidekiq 工作器中解析其中的特定部分.想象一下这样一种情况:scraper 访问一个包含我感兴趣的 10 个元素的网站，并且每个元素都在 Sidekiq 中排队.目前我将元素的源代码作为参数传递，稍后将在 Nokogiri 中加载.我的问题是 - 将一个巨大的字符串作为参数传递给 Sidekiq 工作人员是个好主意吗?字符串长度总是在 77,000-80,000 个字符之间，所以它真的很大.或者我应该把它存储在一个临时表中并在Nokogiri加载之前找到特定的记录?

I'm working on a scraper which goes through websites and parses specific parts of them in Sidekiq workers. Imagine a situation when the scraper visits a website which contains 10 elements that I'm interested in and each of them is queued in Sidekiq. At the moment I pass the source code of the element as an argument which is loaded in Nokogiri later on. My question is - is it a good idea to pass a huge string as an argument to the Sidekiq worker? The string length is always between 77,000-80,000 characters so it's really huge. Or should I store it in a temporary table and find the specific record before loading by Nokogiri?

推荐答案

我建议将字符串存储在 S3(或任何其他对象存储)上，并使用返回的 URL 获取字符串并处理作业.

I would recommend storing the string on S3(or any other object store) and use the returned URL to fetch the string and process the job.

这样你就可以保证一个小型的Redis服务器可以支持很多并发的sidekiq作业并且不会耗尽内存.

This way you can ensure that a small Redis server can support many concurrent sidekiq jobs and will not go out of RAM.

这篇关于将一个巨大的字符串作为参数传递给 Sidekiq 工人是个好主意吗?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！