Alex's Slip-box

These are my org-mode notes in sort of Zettelkasten style

Sidekiq stuff

Created on November 15, 2021

Updated on April 16, 2024

Tags: rails, ruby

:ID: 57F3BD52-A75D-433A-A07F-CB1E06875C84

For a deep ass dive, read this https://www.mikeperham.com/how-sidekiq-works/

# Starting it up

sidekiq -q default -q foo

sidekiq --help shows you all the startup options.

# Poking around in the console

See Mike Perham’s A Tour of the Sidekiq API post. This has most of what I need.

# Running a job in the console (when sidekiq isn’t running)

job = Sidekiq::Queue.new('somequeue').first
job.klass.constantize.new.perform(*job.args)

A Queue is kind of enumerable. You can convert it to_a and find the job if you need. Otherwise just, queue.clear to empty it out and start over.

# Scheduled jobs

See also https://www.rubydoc.info/github/mperham/sidekiq/Sidekiq/ScheduledSet 

irb(main):001:0> scheduled_set = Sidekiq::ScheduledSet.new
=> #<Sidekiq::ScheduledSet:0x00000000090f6a50 @name="schedule", @_size=1>

# Add a scheduled job to the queue

See also https://www.rubydoc.info/github/mperham/sidekiq/Sidekiq/SortedEntry 

r = Sidekiq::ScheduledSet.new
r.first.add_to_queue

# Retry Sets

r = Sidekiq::RetrySet.new

# Enqueuing in bulk

See also https://www.rubydoc.info/github/mperham/sidekiq/Sidekiq%2FClient:push_bulk 

Sidekiq::Client.push_bulk('class' => SomeWorkerClass, 'args' => args)

args is a 2D array.

# Middleware

https://github.com/mperham/sidekiq/wiki/Middleware

# Unique Jobs (ENT)

https://github.com/mperham/sidekiq/wiki/Ent-Unique-Jobs

  • Jobs are unique for the duration set OR until the job completes successfully.
  • If job retries beyond the specified duration, another job can be pushed.

# Manually deleting unique job locks

Try using the redis-cli. Something like… 

redis-cli --scan --pattern "*unique*" | xargs redis-cli DEL

# Scheduled jobs

https://github.com/mperham/sidekiq/wiki/Scheduled-Jobs

Scheduled jobs (including retry queue) are polled around every 5 seconds by default. This can be configured with the average_scheduled_poll_interval config property.

# Rate Limiting (ENT)

https://github.com/mperham/sidekiq/wiki/Ent-Rate-Limiting

There’s several options, here is one example.

# Window Limiting Example

class SomeWorker
  include Sidekiq::Worker

  def perform
    # Limit 5 per 1 second window. If limit is reached, job thread will pause
    # (sleep) for 1 second before trying again. If still limited, job will be
    # rescheduled linearly up to 20 times until finally going to retry queue.
    limiter = Sidekiq::Limiter.window('test_limit_worker', 5, :second, wait_timeout: 1)
    limiter.within_limit do
      puts "****** RUNNING THE WORKER: at #{Time.current} ********"
      raise if rand(9).zero? # Simulate some job errors
    end
  end
end

# Poison Pills

Jobs that cause the process to crash are called “poison pills”. You can find these in the logs (Killed poison pill) See also https://github.com/sidekiq/sidekiq/wiki/Reliability#poison-pills

# Batches

https://github.com/mperham/sidekiq/wiki/Batches

Gives you a collection of jobs that can be monitored as a group. And has callbacks for success and complete. 

class BatchWorker
  include Sidekiq::Worker

  BATCH_DESCRIPTION = 'Test Limiter Batch'

  def perform
    batch = Sidekiq::Batch.new
    batch.description = BATCH_DESCRIPTION
    batch.jobs do
      100.times do
        SomeWorker.perform_async
      end
    end
  end
end

# Batch Sets

Pragmatically get info about current batches (this will only show batches that are not complete) 

Sidekiq::BatchSet.new.each { |batch| p batch }
  • Batches can be deleted programmatically: batch.delete

# Capsules

Introduced in Sidekiq v7. This should let you define concurrency options per queue. https://github.com/mperham/sidekiq/blob/v7.0.0/docs/capsule.md#sidekiq-70-capsules

# Add-ons

See also Ruby gems

  • sidekiq-grouping: combines individually enqueued jobs into one job with a single argument. Useful for converting a bunch of single requests to a single bulk request.

# Setup

For redis configuration just set REDIS_URL env var. see https://github.com/sidekiq/sidekiq/wiki/Using-Redis for more options.

# Testing

To run jobs inline in a test: 

require 'sidekiq/testing'

RSpec.describe 'My Test' do
  before do
    Sidekiq::Testing.inline!
  end

  after do
    Sidekiq::Testing.fake!
  end
end

Search Results