减少sidekiq作业的执行时间 [英] Reduce the execution time of jobs of sidekiq
问题描述
我目前正在开发一个涉及在Rails服务器上同步联系人的应用程序.我正在使用redis服务器和sidekiq在后台执行联系人同步.我的数据库是mongodb，我正在使用Mongoid gem作为ORM.工作流程如下:
I am currently working on an app which involves syncing of contacts on rails server. I am using redis server and sidekiq for performing contact syncing in the background. My database is mongodb and I am using mongoid gem as ORM. Workflow is a follows:
- 电话上的联系人通过应用程序传递到rails服务器，然后在rails服务器上，它在redis服务器中排队.
- 现在，cron作业会触发连接到Redis的sidekiq并完成作业.
sidekiq的一项工作如下:
One Job of sidekiq is as follows:
- 它具有联系人数组(大小最多3000个).
- 它必须处理所有这些联系人.通过处理，我的意思是对数据库进行插入查询.
现在的问题是，sidekiq需要花费大量时间才能完成工作.平均需要50-70秒才能完成工作.
Now the problem is that sidekiq takes insane amount of time to complete the job. On average it takes 50-70 sec to complete the job.
以下是相关文件
sidekiq.yml
sidekiq.yml
# Sample configuration file for Sidekiq.
# Options here can still be overridden by cmd line args.
# sidekiq -C config.yml
:verbose: true
:concurrency: 5
:logfile: ./log/sidekiq.log
:pidfile: ./tmp/pids/sidekiq.pid
:queues:
- [new_wall, 1]#6
- [contact_wall, 1]#7
- [email, 1]#5
- [gcm_chat, 1]#5
- [contact_address, 1]#7
- [backlog_contact_address, 5]
- [comment, 7]
- [default, 5]
mongoid.yml
mongoid.yml
development:
# Configure available database sessions. (required)
sessions:
# Defines the default session. (required)
default:
# Defines the name of the default database that Mongoid can connect to.
# (required).
database: "<%= ENV['DB_NAME']%>"
# Provides the hosts the default session can connect to. Must be an array
# of host:port pairs. (required)
hosts:
- "<%=ENV['MONGOD_URL']%>"
#username: "<%= ENV['DB_USERNAME']%>"
#password: "<%= ENV['DB_PASSWORD']%>"
options:
#pool: 12
# Change the default write concern. (default = { w: 1 })
# write:
# w: 1
# Change the default consistency model to primary, secondary.
# 'secondary' will send reads to secondaries, 'primary' sends everything
# to master. (default: primary)
# read: secondary_preferred
# How many times Moped should attempt to retry an operation after
# failure. (default: The number of nodes in the cluster)
# max_retries: 20
# The time in seconds that Moped should wait before retrying an
# operation on failure. (default: 0.25)
# retry_interval: 0.25
# Configure Mongoid specific options. (optional)
options:
# Includes the root model name in json serialization. (default: false)
# include_root_in_json: false
# Include the _type field in serializaion. (default: false)
# include_type_for_serialization: false
# Preload all models in development, needed when models use
# inheritance. (default: false)
# preload_models: false
# Protect id and type from mass assignment. (default: true)
# protect_sensitive_fields: true
# Raise an error when performing a #find and the document is not found.
# (default: true)
# raise_not_found_error: true
# Raise an error when defining a scope with the same name as an
# existing method. (default: false)
# scope_overwrite_exception: false
# Use Active Support's time zone in conversions. (default: true)
# use_activesupport_time_zone: true
# Ensure all times are UTC in the app side. (default: false)
# use_utc: false
test:
sessions:
default:
database: db_test
hosts:
- localhost:27017
options:
read: primary
# In the test environment we lower the retries and retry interval to
# low amounts for fast failures.
max_retries: 1
retry_interval: 0
production:
# Configure available database sessions. (required)
sessions:
# Defines the default session. (required)
default:
# Defines the name of the default database that Mongoid can connect to.
# (required).
database: "<%= ENV['DB_NAME']%>"
# Provides the hosts the default session can connect to. Must be an array
# of host:port pairs. (required)
hosts:
- "<%=ENV['MONGOD_URL']%>"
username: "<%= ENV['DB_USERNAME']%>"
password: "<%= ENV['DB_PASSWORD']%>"
pool: 10
options:
# Configure Mongoid specific options. (optional)
options:
Model.rb
def retry_save_contact_dump(c_dump_id)
c_dump = ContactDump.where(_id: c_dump_id, status: ContactDump::CONTACT_DUMP_CONS[:ERROR]).first
return false if c_dump.blank?
user = User.where(_id: c_dump.user_id).first
puts "retry_save_contact_dump"
user.save_contacts_with_name(c_dump.contacts)
c_dump.status = ContactDump::CONTACT_DUMP_CONS[:PROCESSED]
c_dump.error_msg = ""
c_dump.save
rescue => e
c_dump.status = ContactDump::CONTACT_DUMP_CONS[:CANTSYNC]
c_dump.error_msg = e.message
c_dump.save
end
def save_contacts_with_name(c_array)
m_num = Person.get_number_digest(self.mobile_number.to_s)
c_array.each do |n|
next if m_num == n["hash_mobile_number"]
p = Person.where(h_m_num: n["hash_mobile_number"]).first_or_create
save_friend(p) #if p.persisted?
p.c_names.create(name: n["name"], user_id: self.id)
end
end
ContactDump.rb
ContactDump.rb
class ContactDump
include Mongoid::Document
include Mongoid::Timestamps::Created
include Mongoid::Timestamps::Updated
field :contacts, type: Array
field :status, type: Integer, default: 0
field :user_id, type: BSON::ObjectId
field :error_msg, type: String
CONTACT_DUMP_CONS = {FRESH: 0, PROCESSED: 1, ERROR: 2, CANTSYNC: 3}
end
如何加快作业处理速度?我尝试了在sidekiq.yml和mongoid.yml池中增加sidekiq并发性的排列，但是没有帮助.
How can I speed up the processing of jobs? I tried with permutation of increasing concurrency of sidekiq in sidekiq.yml and pool of mongoid.yml, but no help.
whatsApp和其他消息传递应用程序如何处理联系人同步?
How do whatsApp and other messaging apps deal with contact syncing?
如果需要其他信息，请询问.谢谢.
If some other info is required, please ask. Thanks.
如果无法回答这个问题，谁能建议我其他方法来同步Rails服务器上的联系人.
If not possible to answer this question, can anyone please suggest me other ways to sync the contacts on the rails server.
推荐答案
对救援的索引.
class ContactDump
index({status: 1})
end
class Person
index({h_m_num: 1})
end
Person可能需要更多索引，具体取决于您的Person.get_number_digest所做的事情.
Person might need more indexes depending on what your Person.get_number_digest does.
添加索引后运行
rake db:mongoid:create_indexes
此外，请删除puts，即使您看不到输出，也不需要您的工人，并且看跌期权严重影响您的表现！
Also, do remove the puts, you don't need that on your worker and puts is hitting your performance badly, even when you can't see the output!
这篇关于减少sidekiq作业的执行时间的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
