Jets Job Multiple Queues

Deploy Multiple Queues

To have Jets deploy and create multiple Queue

config/jets/deploy.rb

Jets.deploy.configure do
  config.job.enable = true

  config.job.default_queue.lambda.reserved_concurrency = 5

# Add additional queues. Note: default queue is always created
  config.job.additional_queues = %w[urgent low_priority]

  # Adjust settings for each queue
  config.job.queues.low_priority.lambda.reserved_concurrency = 2
  config.job.queues.urgent.lambda.memory_size = 2048
  config.job.queues.urgent.lambda.reserved_concurrency = 10
end

This creates 2 additional queues. The default queue is always created. In the example, there’ll be 3 queues.

default urgent low_priority

Each queue has its own Lambda Function for processing. This allows you to individually control settings like reserved_concurrency and memory_size. You can manually use jets concurrency:set to tune the concurrency and then codify the settings.

Example Job with Queue

Here’s an example job

app/jobs/mailman_job.rb

class MailmanJob < ApplicationJob
  queue_as :urgent

  def perform(*args)
    puts "Delivering: #{args}"
  end
end

When you call the job, it will send it to the specific SQS queue.

$ rails console
> MailmanJob.perform_later("mail") # => urgent queue
> MailmanJob.set(queue: "low_priority").perform_later("candy") # => low_priority queue

Reference

The table below covers each setting. Each option is configured with config.OPTION. The config. portion is not shown for conciseness. IE: logger.level vs config.logger.level.

Name Default Description
job.additional_queues [] Example: %w[urgent low_priority]
job.default_queue.lambda.reserved_concurrency 5 Reserved concurrency to use for Lambda function associated with default queue.
job.default_queue.properties {} Override the SQS Queue properties. See Generated Function SQS Queue and AWS::SQS::Queue.
job.enable false Enable Jets to create Job resources to handle ActiveJob work, like SQS Queue.
job.queue_defaults.lambda.memory_size 1536 Default reserved concurrency for memory_size lambda functions. Can be overwritten individually with config.job.queues.NAME.lambda.memory_size
job.queue_defaults.lambda.properties {} Default properties for the additional_queues lambda functions. Can be overwritten individually with config.job.queues.NAME.lambda.PROPERTY
job.queue_defaults.lambda.reserved_concurrency 5 Default reserved concurrency for additional_queues lambda functions. Can be overwritten individually with config.job.queues.NAME.lambda.reserved_concurrency
job.queue_defaults.lambda.timeout 900 Default timeout for additional_queues lambda functions. Can be overwritten individually with config.job.queues.NAME.lambda.timeout
job.queue_defaults.properties {} Default properties for the additional_queues. Can be overwritten individually with config.job.queues.NAME.PROPERTY. See AWS::SQS::Queue
job.queues.NAME.memory_size 1536 An example of how you can set individual additional queue the memory_size setting.
job.queues.NAME.reserved_concurrency 5 An example of how you can set individual additional queue the reserved_concurrency setting.

See Full Config Reference