Queues allow you to defer/postpone the processing of a time consuming task until a later time. Laravel makes it a breeze!

Laravel provides a single API to work with different queue back-ends. The subject may be overwhelming, so lets break it down piece by piece. I’d like to illustrate the problem using a common example, as I find it the best way to learn when there’s lot of new terminology involved.

Scenario

Imagine that you have a website where visitors can register for an account, perhaps an eCommerce or social network website. A visitor is currently on the registration page and has submitted the registration form. It stands to reason that, among other stuff, we want the following to happen:

// store user's data into the database
// send a welcome email to the user
// return "thank you" page

Skipping the details, lets just suppose there’s some PHP in the background that takes care of sanitizing the input and storing the user into the database.

After inserting the user into the database, the script now has to send him a welcome email and, since PHP executes code line-by-line top-to-bottom, the user will see the “thank you” page only after the email has been sent. Even though sending email sometimes happens very fast, it sometimes takes ages for this to happen. Why make the user wait?

Queues to the rescue!

That’s where queue services come into play. I’ll use terminology that’s specific to Laravel’s queue API.

A queue is just a list/line of things waiting to be handled in order, starting from the beginning. When I say things, I mean jobs.

I won’t cover a lot about jobs in this post, but you should know that if you want to push a job (dispatch it) onto the queue, the job must implement Illuminate\Contracts\Queue\ShouldQueue interface. To learn more about jobs and how to create them, visit official docs.

In Laravel, a job that should be queued must implement the Illuminate\Contracts\Queue\ShouldQueue interface.

What if we take the process of sending the email and shove it into a job, and then push that job onto the queue instead? This approach now differs at the second step:

// store user's data into the database
$this->dispatch(new SendWelcomeEmail($user)); // push a SendWelcomeEmail job onto the queue
// return "thank you" page

Instead of returning the “thank you” response after the email has been sent, we now return the response after the job has been pushed onto the queue. This way, user has to wait only as long as it takes for the job to be pushed, as opposed to waiting for an email to be actually sent.

Executing jobs

Okay, you got it. There’s this queue/list and there are these jobs that get pushed onto the queue. But when and how do these jobs get executed? When does the welcome email actually gets sent?

In Laravel, there is this intimidating thing called Queue Listener. Queue Listener is nothing more than a long-running process that listens to and runs the jobs from the queue.

Technically, a worker command is being created in the background, but to avoid overwhelming you with too many new words at once, there will be a whole chapter about it later in the article. Long story short - if Queue Listener never existed, none of the jobs from the queue would ever get executed.

We start the Queue Listener by running the following command from the terminal:

php artisan queue:listen

If your server crashes, so will the Queue Listener stop. You should configure a process monitor that will automatically restart the Queue Listener. Supervisor is a great process monitor for the Linux operating system.

If there were already jobs on the queue, it will just go on and do them one-by-one.

Storing jobs

But where are these jobs stored? All we’ve learned so far is that they’re pushed onto this queue, but what is it exactly?

As mentioned earlier - a queue is just a list of jobs that are waiting to be executed. Don’t think of queue as anything else but a list of jobs!

Okay, where is this list stored anyway? How do we push the jobs onto this list? We use queue drivers!

Queue drivers

A queue driver is a concrete implementation of the Illuminate\Contracts\Queue\Queue interface. It is responsible for managing the jobs, that is - storing and retrieving the jobs from our queue.

There are several drivers that ship with Laravel, and you can create one yourself if that’s what you want (will cover it in another article).

For example - we can store jobs in the database. Laravel even provides database driver out of the box! You only have to do two simple steps:

  • set environment variable QUEUE_DRIVER to database (or just shove it in .env file)
  • run the php artisan queue:table and php artisan migrate from the terminal

The latter will create migration for table that will hold the jobs, conveniently called “jobs”, and run the migration.

Believe it or not - literally that’s all you have to do! You can run Queue Listener and everything will work.

You might want to create a migration for failed jobs as well. You can do so by running queue:failed-jobs Artisan command.

In this article I won’t be covering any other specific drivers that ship with Laravel. You can also create your own queue drivers. Bear in mind your queue implementation has to adhere to Illuminate\Contracts\Queue\Queue contract.

Listener and Workers explained

When you run queue:listen Artisan command, the Illuminate\Queue\Listener::listen() method will eventually get triggered:

public function listen($connection, $queue, $delay, $memory, $timeout = 60)
{
    $process = $this->makeProcess($connection, $queue, $delay, $memory, $timeout);

    while (true) {
        $this->runProcess($process, $memory);
    }
}

More specifically, the $process that is created is actually a Worker process. Even more specifically, the Worker process is actually Symfony’s Process object that calls the queue:work command once it has been started.

The while(true) basically says “run forever”. Thus, the listen command runs as long as you want it to (or until it runs out of memory), running the runProcess($process) method over and over.

This is what the runProcess() method looks like:

public function runProcess(Process $process, $memory)
{
    $process->run(function ($type, $line) {
        $this->handleWorkerOutput($type, $line);
    });

    if ($this->memoryExceeded($memory)) {
        $this->stop();
    }
}

Basically, it does two things. It runs the process and it checks if the memory limit has been exceeded. You can set the memory limit by providing a --memory option when starting the listener, but by default it’s 128 megabytes. If the memory has been exceeded, the listener gets stopped so a process manager can re-start it with a clean slate of memory (given you have a configured process manager to do so).

If all this is still somewhat confusing, read on the step-by-step guide.

Doing it step-by-step

Lets go through all the listener-worker fuss one more time, step-by-step.

When we run the queue:listen, the following things occur:

  1. Listener::listen() method is triggered, which creates a new instance of Symfony’s Process (which is a call to queue:work Artisan command), and stores it in $process variable
  2. runProcess($process) called in the infinite loop (for the 1st time)
  3. run() method is being triggered on the Process (which starts the queue:work Artisan command at this point)
  4. The queue:work command eventually runs the Worker::pop() method, that either runs the next job available, or sleeps if there are no jobs
  5. After the job has been finished or, if there wasn’t any job available, worker has finished sleeping, a check is being made on the Listener class whether memory has been exceeded - if the memory has been exceeded, the listener just stops here and there are no more steps
  6. runProcess($process) called in the infinite loop (for the 2nd time)
  7. And so on…

As stated in step #4, the worker command essentially runs Worker::pop() method, which looks like this:

public function pop($connectionName, $queue = null, $delay = 0, $sleep = 3, $maxTries = 0)
{
    $connection = $this->manager->connection($connectionName);

    $job = $this->getNextJob($connection, $queue);

    if (! is_null($job)) {
        return $this->process(
            $this->manager->getName($connectionName), $job, $maxTries, $delay
        );
    }

    $this->sleep($sleep);

    return ['job' => null, 'failed' => false];
}

So as you can see, all it does is try to get the next job off of queue. If there is nothing on the queue, it will sleep for whatever time you’ve specified it to (--sleep option when running the listener).

There were some comments in the community about this being a cron job, so hopefully all this illustrates a bit better what actually happens in the background.

Tips and tricks

Daemon Worker

You might as well try to run queue:work with --daemon option for forcing the queue worker to continue processing jobs without ever re-booting the framework. This results in a significant reduction of CPU usage when compared to the queue:listen command, but at the added complexity of needing to drain the queues of currently executing jobs during your deployments.

Daemon queue workers do not restart the framework before processing each job. Therefore, you should be careful to free any heavy resources before your job finishes. For example, if you are doing image manipulation with the GD library, you should free the memory with imagedestroy() when you are done.

Similarly, your database connection may disconnect when being used by long-running daemon. You may use the DB::reconnect method to ensure you have a fresh connection.

Failed Jobs

Sometimes things don’t go as planned, meaning your queued jobs will fail. It happens to everyone, don’t worry. Laravel includes a simple way to specify the maximum number of times a job should be attempted. After a job has exceeded this amount of attempts, it will be inserted into a failed_jobs table.

When running Queue Listener, you may specify the maximum number of times a job should be attempted using the --tries option:

php artisan queue:listen --tries=3

You also may define failed method directly on a job class, which will get triggered when a failure occurs.

class SendWelcomeEmail extends Job implements SelfHandling, ShouldQueue
{
    public function failed()
    {
        // Called when the job is failing...
    }
}

Leverage DispatchesJobs trait

As long as your controllers extend Laravel’s App\Http\Controllers\Controller you can dispatch jobs easily using the $this->dispatch($job) syntax. If you want to dispatch a job from somewhere other than your controllers, you should use Illuminate\Foundation\Bus\DispatchesJobs trait.

Multiple Queues and Workers

You can have different queues/lists for storing the jobs. You can name them however you want, such as “images” for pushing image processing tasks, or “emails” for queue that holds jobs specific to sending emails.

You can also have multiple workers, each working on a different queue if you want. You can even have multiple workers per queue, thus having more than one job being worked on simultaneously. Bear in mind having multiple workers comes with a CPU and memory cost.

Look it up in the official docs, it’s pretty straightforward.

Conclusion

This was a brief overview of how queues work in Laravel. Most of this also does apply to other PHP frameworks (or even other languages), but the API and approaches are different.

Hopefully this article was helpful to you and you’ve gotten the grasp of what queues are and how they fit into the web development world.

Should you have any further questions, feel free to join the discussion below.