8000 start_heartbeat can die while worker is still running · Issue #1883 · resque/resque · GitHub
[go: up one dir, main page]
Skip to content

start_heartbeat can die while worker is still running #1883

New issue
New issue
Open
#1884
Open
start_heartbeat can die while worker is still running#1883
#1884
@arianf

Description

@arianf
arianf
opened on Oct 19, 2023

Our Redis instances has been having some connectivity issues (which we are resolving as well), however I noticed something interesting in our Resque logs. It's possible for the heartbeat to die, while the worker remains running.

In our Resque logs I see this for the worker that is still running but no longer listed within Resque.workers

#<Thread:0x00007f974e3b8ef8 /usr/local/bundle/ruby/3.2.0/gems/resque-2.4.0/lib/resque/worker.rb:527 run> terminated with exception (report_on_exception is true):
/usr/local/bundle/ruby/3.2.0/gems/hiredis-client-0.12.0/lib/redis_client/hiredis_connection.rb:48:in `rescue in initialize': Failed to connect to helm-redis-primary:6379 (ECONNREFUSED) (Redis::CannotConnectError)
	from /usr/local/bundle/ruby/3.2.0/gems/hiredis-client-0.12.0/lib/redis_client/hiredis_connection.rb:44:in `initialize'
	from /usr/local/bundle/ruby/3.2.0/gems/redis-client-0.12.0/lib/redis_client.rb:664:in `new'
	from /usr/local/bundle/ruby/3.2.0/gems/redis-client-0.12.0/lib/redis_client.rb:664:in `block in connect'
	from /usr/local/bundle/ruby/3.2.0/gems/redis-client-0.12.0/lib/redis_client/middlewares.rb:12:in `connect'
	from /usr/local/bundle/ruby/3.2.0/gems/redis-client-0.12.0/lib/redis_client.rb:663:in `connect'
	from /usr/local/bundle/ruby/3.2.0/gems/redis-client-0.12.0/lib/redis_client.rb:657:in `raw_connection'
	from /usr/local/bundle/ruby/3.2.0/gems/redis-client-0.12.0/lib/redis_client.rb:624:in `ensure_connected'
	from /usr/local/bundle/ruby/3.2.0/gems/redis-client-0.12.0/lib/redis_client.rb:223:in `call_v'
	from /usr/local/bundle/ruby/3.2.0/gems/redis-5.0.6/lib/redis/client.rb:73:in `call_v'
	from /usr/local/bundle/ruby/3.2.0/gems/redis-5.0.6/lib/redis.rb:167:in `block in send_command'
	from /usr/local/bundle/ruby/3.2.0/gems/redis-5.0.6/lib/redis.rb:166:in `synchronize'
	from /usr/local/bundle/ruby/3.2.0/gems/redis-5.0.6/lib/redis.rb:166:in `send_command'
	from /usr/local/bundle/ruby/3.2.0/gems/redis-5.0.6/lib/redis/commands/server.rb:178:in `time'
	from /usr/local/bundle/ruby/3.2.0/gems/redis-namespace-1.9.0/lib/redis/namespace.rb:479:in `call_with_namespace'
	from /usr/local/bundle/ruby/3.2.0/gems/redis-namespace-1.9.0/lib/redis/namespace.rb:354:in `block (2 levels) in <class:Namespace>'
	from /usr/local/bundle/ruby/3.2.0/gems/resque-2.4.0/lib/resque/data_store.rb:96:in `server_time'
	from /usr/local/bundle/ruby/3.2.0/gems/resque-2.4.0/lib/resque/worker.rb:492:in `heartbeat!'
	from /usr/local/bundle/ruby/3.2.0/gems/resque-2.4.0/lib/resque/worker.rb:529:in `block (2 levels) in start_heartbeat'
	from /usr/local/bundle/ruby/3.2.0/gems/resque-2.4.0/lib/resque/worker.rb:528:in `loop'
	from /usr/local/bundle/ruby/3.2.0/gems/resque-2.4.0/lib/resque/worker.rb:528:in `block in start_heartbeat'
/usr/local/bundle/ruby/3.2.0/gems/hiredis-client-0.12.0/lib/redis_client/hiredis_connection.rb:48:in `rescue in initialize': Failed to connect to helm-redis-primary:6379 (ECONNREFUSED) (RedisClient::CannotConnectError)
	from /usr/local/bundle/ruby/3.2.0/gems/hiredis-client-0.12.0/lib/redis_client/hiredis_connection.rb:44:in `initialize'
	from /usr/local/bundle/ruby/3.2.0/gems/redis-client-0.12.0/lib/redis_client.rb:664:in `new'
	from /usr/local/bundle/ruby/3.2.0/gems/redis-client-0.12.0/lib/redis_client.rb:664:in `block in connect'
	from /usr/local/bundle/ruby/3.2.0/gems/redis-client-0.12.0/lib/redis_client/middlewares.rb:12:in `connect'
	from /usr/local/bundle/ruby/3.2.0/gems/redis-client-0.12.0/lib/redis_client.rb:663:in `connect'
	from /usr/local/bundle/ruby/3.2.0/gems/redis-client-0.12.0/lib/redis_client.rb:657:in `raw_connection'
	from /usr/local/bundle/ruby/3.2.0/gems/redis-client-0.12.0/lib/redis_client.rb:624:in `ensure_connected'
	from /usr/local/bundle/ruby/3.2.0/gems/redis-client-0.12.0/lib/redis_client.rb:223:in `call_v'
	from /usr/local/bundle/ruby/3.2.0/gems/redis-5.0.6/lib/redis/client.rb:73:in `call_v'
	from /usr/local/bundle/ruby/3.2.0/gems/redis-5.0.6/lib/redis.rb:167:in `block in send_command'
	from /usr/local/bundle/ruby/3.2.0/gems/redis-5.0.6/lib/redis.rb:166:in `synchronize'
	from /usr/local/bundle/ruby/3.2.0/gems/redis-5.0.6/lib/redis.rb:166:in `send_command'
	from /usr/local/bundle/ruby/3.2.0/gems/redis-5.0.6/lib/redis/commands/server.rb:178:in `time'
	from /usr/local/bundle/ruby/3.2.0/gems/redis-namespace-1.9.0/lib/redis/namespace.rb:479:in `call_with_namespace'
	from /usr/local/bundle/ruby/3.2.0/gems/redis-namespace-1.9.0/lib/redis/namespace.rb:354:in `block (2 levels) in <class:Namespace>'
	from /usr/local/bundle/ruby/3.2.0/gems/resque-2.4.0/lib/resque/data_store.rb:96:in `server_time'
	from /usr/local/bundle/ruby/3.2.0/gems/resque-2.4.0/lib/resque/worker.rb:492:in `heartbeat!'
	from /usr/local/bundle/ruby/3.2.0/gems/resque-2.4.0/lib/resque/worker.rb:529:in `block (2 levels) in start_heartbeat'
	from /usr/local/bundle/ruby/3.2.0/gems/resque-2.4.0/lib/resque/worker.rb:528:in `loop'
	from /usr/local/bundle/ruby/3.2.0/gems/resque-2.4.0/lib/resque/worker.rb:528:in `block in start_heartbeat'
/usr/local/bundle/ruby/3.2.0/gems/hiredis-client-0.12.0/lib/redis_client/hiredis_connection.rb:45:in `connect_tcp': Connection refused (Errno::ECONNREFUSED)
	from /usr/local/bundle/ruby/3.2.0/gems/hiredis-client-0.12.0/lib/redis_client/hiredis_connection.rb:45:in `initialize'
	from /usr/local/bundle/ruby/3.2.0/gems/redis-client-0.12.0/lib/redis_client.rb:664:in `new'
	from /usr/local/bundle/ruby/3.2.0/gems/redis-client-0.12.0/lib/redis_client.rb:664:in `block in connect'
	from /usr/local/bundle/ruby/3.2.0/gems/redis-client-0.12.0/lib/redis_client/middlewares.rb:12:in `connect'
	from /usr/local/bundle/ruby/3.2.0/gems/redis-client-0.12.0/lib/redis_client.rb:663:in `connect'
	from /usr/local/bundle/ruby/3.2.0/gems/redis-client-0.12.0/lib/redis_client.rb:657:in `raw_connection'
	from /usr/local/bundle/ruby/3.2.0/gems/redis-client-0.12.0/lib/redis_client.rb:624:in `ensure_connected'
	from /usr/local/bundle/ruby/3.2.0/gems/redis-client-0.12.0/lib/redis_client.rb:223:in `call_v'
	from /usr/local/bundle/ruby/3.2.0/gems/redis-5.0.6/lib/redis/client.rb:73:in `call_v'
	from /usr/local/bundle/ruby/3.2.0/gems/redis-5.0.6/lib/redis.rb:167:in `block in send_command'
	from /usr/local/bundle/ruby/3.2.0/gems/redis-5.0.6/lib/redis.rb:166:in `synchronize'
	from /usr/local/bundle/ruby/3.2.0/gems/redis-5.0.6/lib/redis.rb:166:in `send_command'
	from /usr/local/bundle/ruby/3.2.0/gems/redis-5.0.6/lib/redis/commands/server.rb:178:in `time'
	from /usr/local/bundle/ruby/3.2.0/gems/redis-namespace-1.9.0/lib/redis/namespace.rb:479:in `call_with_namespace'
	from /usr/local/bundle/ruby/3.2.0/gems/redis-namespace-1.9.0/lib/redis/namespace.rb:354:in `block (2 levels) in <class:Namespace>'
	from /usr/local/bundle/ruby/3.2.0/gems/resque-2.4.0/lib/resque/data_store.rb:96:in `server_time'
	from /usr/local/bundle/ruby/3.2.0/gems/resque-2.4.0/lib/resque/worker.rb:492:in `heartbeat!'
	from /usr/local/bundle/ruby/3.2.0/gems/resque-2.4.0/lib/resque/worker.rb:529:in `block (2 levels) in start_heartbeat'
	from /usr/local/bundle/ruby/3.2.0/gems/resque-2.4.0/lib/resque/worker.rb:528:in `loop'
	from /usr/local/bundle/ruby/3.2.0/gems/resque-2.4.0/lib/resque/worker.rb:528:in `block in start_heartbeat'

If this thread dies, there is nothing that will restart it?

resque/lib/resque/worker.rb

Lines 527 to 533 in 2f9d080

@heartbeat_thread = Thread.new do
loop do
heartbeat!
signaled = @heartbeat_thread_signal.wait_for_signal(Resque.heartbeat_interval)
break if signaled
end
end

Maybe each time it runs a new job it should check if the heartbeat thread is still alive, and if not start a new heartbeat.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions
      0