quebert

KO | EN

gitlite — search

#javascript #react #typescript #llm #golang #go #android #machine-learning #deep-learning #linux #kubernetes #pytorch #security #ios

(★ 35)

A pluggable worker queue framework inspired by Beanstalk. This puppy is used in a production Rails environment at Poll Everywhere.

quebert Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Quebert

[![Build Status](https://travis-ci.org/polleverywhere/quebert.svg?branch=master)](https://travis-ci.org/polleverywhere/quebert) [![Code Climate](https://codeclimate.com/github/polleverywhere/quebert/badges/gpa.svg)](https://codeclimate.com/github/polleverywhere/quebert)

Quebert is a ruby background worker library that works with the very fast and simple [beanstalkd](http://kr.github.io/beanstalkd/) deamon.

## Why Quebert?

Because it has really low latency. Other Ruby queuing frameworks, like [DJ](https://github.com/collectiveidea/delayed_job) or [Resque](https://github.com/resque/resque), have to poll their queue servers periodicly. You could think of it as a "pull" queue. Quebert is a "push" queue. It maintains a persistent connection with beanstalkd and when is enqueued, it's instantly pushed to the workers and executed.

[Sidekiq](http://sidekiq.org) uses Redis's "push" primitives so it has low latency, but it doesn't support class reloading in a development environment. Sidekiq is also threaded, which means there are no guarantees of reliability when running non-threadsafe code.

[Backburner](https://github.com/nesquena/backburner) is very similar to Quebert. It offers more options for concurrency (threading, forking, etc.) than Quebert but lacks pluggable back-ends, which means you'll be stubbing and mocking async calls.

## Who uses it?

Quebert is a serious project. It's used in a production environment at [Poll Everywhere](https://www.polleverywhere.com/) to handle everything from SMS message processing to account downgrades.

## Features

* **Multiple back-ends** (InProcess, Sync, and Beanstalk)
* **Rails/ActiveRecord integration** similar to async_observer
* **Pluggable exception handling** (for Hoptoad integration)
* **Run workers with pid, log, and config files**. These do not daemonize (do it yourself punk!)
* **Custom hooks** that may be called before, after & around jobs are run

Some features that are currently *missing* that I will soon add include:

* Rails plugin support (The AR integrations have to be done manually today)
* Auto-detecting serializers. Enhanced ClassRegistry to more efficiently look up serializers for objects.

## How to use

There are two ways to enqueue jobs with Quebert: through the Job itself, provided you set a default back-end for the job, or put it on the backend.

### Supported Ruby Versions

Quebert officially is supported to run on the currently supported versions of MRI.
This includes versions >= `2.4.3`. Have a look at the `.travis.yml` configuration file to see all Ruby versions we support.

### Jobs

Quebert includes a Job class so you can implement how you want certain types of Jobs performed.

```ruby
Quebert.backend = Quebert::Backend::InProcess.new

class WackyMathWizard < Quebert::Job
  def perform(*nums)
    nums.inject(0){|sum, n| sum = sum + n}
  end
end
```

You can either drop a job in a queue:

```ruby
Quebert.backend.put WackyMathWizard.new(1, 2, 3)
```

Or drop it in right from the job:

```ruby
# Run job right away!
WackyMathWizard.new(4, 5, 6).enqueue
# Run a lower priority job in 10 seconds for a max of 120 seconds
WackyMathWizard.new(10, 10, 10).enqueue(ttr: 120, priority: 100, delay: 10)
```

Then perform the jobs!

```ruby
Quebert.backend.reserve.perform # => 6
Quebert.backend.reserve.perform # => 15
Quebert.backend.reserve.perform # => 30
```

### Rails integration

config/quebert.yml:

```yaml
development:
  backend: beanstalk
  host: localhost:11300
  queue: myapp-development
test:
  backend: sync
# etc.
```

config/initializers/quebert.rb:

```ruby
Quebert.config.from_hash(Rails.application.config.quebert)
Quebert.config.logger = Rails.logger
```

### Global Job hooks

Quebert has support for providing custom hooks to be called before, after & around your jobs are being run.
A common example is making sure that any active ActiveRecord database connections are put back on the connection pool after a job is done:

```ruby
Quebert.config.after_job do
  ActiveRecord::Base.clear_active_connections!
end

Quebert.config.before_job do |job|
  # all hooks take an optional job argument
  # in case you want to do something with that job
end

Quebert.config.around_job do |job|
  # this hook gets called twice
  # once before & once after a job is performed
end
```

### Beanstalk Job hooks

Jobs can define their own business logic that will get called surrounding Beanstalk events:

```ruby
class FooJob < Quebert::Job
  def around_bury
    # custom pre-bury code
    yield
    # custom post-bury code
  end
end
```

Supported Beanstalk event hooks: `around_bury`, `around_release`, `around_delete`

### Async sender

Take any ol' class and include the Quebert::AsyncSender.

```ruby
Quebert.backend = Quebert::Backend::InProcess.new

class Greeter
  include Quebert::AsyncSender::Class

def initialize(name)
    @name = name
  end

def sleep_and_greet(time_of_day)
    sleep 10000 # Sleeping, get it?
    "Oh! Hi #{name}! Good #{time_of_day}."
  end

def self.budweiser_greeting(name)
    "waaazup #{name}!"
  end
end

walmart_greeter = Greeter.new("Brad")
```

Remember the send method in ruby?

```ruby
walmart_greeter.send(:sleep_and_greet, "morning")
# ... time passes, you wait as greeter snores obnoxiously ...
# => "Oh! Hi Brad! Good morning."
```

What if the method takes a long time to run and you want to queue it? async.send it!

```ruby
walmart_greeter.async.sleep_and_greet("morning")
# ... do some shopping and come back later when the dude wakes up
```

Quebert figures out how to serialize the class, throw it on a worker queue, re-instantiate it on the other side, and finish up the work.

```ruby
Quebert.backend.reserve.perform # => "Oh! Hi Brad! Good morning."
# ... Sorry dude! I'm shopping already
```

Does it work on Class methods? Yeah, that was easier than making instance methods work:

```ruby
Quebert.async.budweiser_greeting("Coraline")
Quebert.backend.reserve.perform # => "waazup Coraline!"
```

* Only basic data types are included for serialization. Serializers may be customized to include support for different types.

### Backends

* Beanstalk: Enqueue jobs in a beanstalkd service. The workers run in a separate process. Typically used in production environments.
* Sync: Perform jobs immediately upon enqueuing. Typically used in testing environments.
* InProcess: Enqueue jobs in an in-memory array. A worker will need to reserve a job to perform.

### Multiple queues

To start a worker pointed at a non-default queue (e.g., a Quebert "tube"), start the process with `-q`:

```sh
bundle exec quebert -q other-tube
```

Then specify the queue name in your job:

```ruby
class FooJob < Quebert::Job
  def queue
    "other-tube"
  end

def perform(args)
    # ...
  end
end
```

### Setting job defaults

A `Quebert::Job` is a Plain Ol' Ruby Object. The defaults of a job, including its `ttr`, `queue_name`, and `delay` may be overridden in a super class as follows:

```ruby
# Assuming you're in Rails or using ActiveSupport
class FooJob < Quebert::Job
  def ttr
    5.minutes
  end

def delay
    30.seconds
  end

def queue_name
    "long-running-delayed-jobs"
  end

def perform(args)
    # ...
  end
end
```

Take a look at the [`Quebert::Job` class](https://github.com/polleverywhere/quebert/blob/master/lib/quebert/job.rb) code for more details on methods you may ovveride.

quebert

quebert Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

quebert

quebert Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README