Understanding the Queue API

Drupal’s Queue API allows developers to add items to a queue that can be processed later. This is ideal for tasks that cannot be completed within a single page load or require delayed execution. By processing items in manageable batches, you ensure a balanced load on your server and prevent timeouts.

Step 1: Create the Queue Worker Plugin

To create a queue worker in Drupal, define a plugin class that specifies how each queued item should be processed. In your custom module (e.g., custom_module), create a PHP file under the src/Plugin/QueueWorker directory.

Here’s an example of a simple queue worker plugin:

<?php

namespace Drupal\custom_module\Plugin\QueueWorker;

use Drupal\Core\Queue\QueueWorkerBase;

/**

 * Processes tasks for the custom queue.

 *

 * @QueueWorker(

 *   id = "custom_queue_worker",

 *   title = @Translation("Custom Queue Worker"),

 *   cron = {"time" = 60}

 * )

 */

class CustomQueueWorker extends QueueWorkerBase {

  /**

   * Processes a single item in the queue.

   *

   * @param object $data

   *   The data object representing the queued item.

   */

  public function processItem($data) {

    \Drupal::logger('custom_module')->info('Processing item: @item', ['@item' => $data->name]);

    // Here you can add logic to process the item.

  }

}

In this code:

  • The @QueueWorker annotation registers this class as a queue worker with an ID (custom_queue_worker) and title.
  • The cron annotation tells Drupal to check and process this queue every 60 seconds during cron runs.
  • The processItem() method processes each individual item. In this example, each processed item is logged to the Drupal logger, but you can replace this with any processing logic.

Step 2: Adding Items to the Queue

Now that you have a queue worker set up, you can add items to the queue programmatically. You can do this in a form submission handler, a custom page, or any other part of your module.

$queue = \Drupal::queue('custom_queue_worker');

for ($i = 1; $i <= 100; $i++) {

  $data = (object) ['name' => "Task $i"];

  $queue->createItem($data);

}

In this example, we add 100 items to the queue, each represented by an object with a unique name (Task 1, Task 2, etc.). The createItem() function enqueues each item for future processing by the custom_queue_worker.

Step 3: Batch Processing the Queue

You can process the queue in batches either through Drupal’s cron or manually using Drush.

Automatic Processing with Cron

When the cron annotation is set, Drupal will automatically process items in the queue each time cron runs. You can configure the cron frequency in Drupal’s administrative settings (/admin/config/system/cron). Each time cron runs, Drupal processes a batch of items, reducing the queue load over time.

Manual Processing with Drush

For immediate batch processing, you can use Drush, a command-line tool for managing Drupal. Run the following command to process the queue items manually:

drush queue:run custom_queue_worker

This command will process items in the custom_queue_worker queue in batches until the queue is empty. Running the command is helpful for testing the queue or processing items outside of scheduled cron runs.

Benefits of Queues and Batch Processing

Using the Queue API with batch processing allows Drupal sites to handle high volumes of tasks while maintaining performance and reliability. Instead of processing every item immediately (which could cause server overload), tasks are deferred to the queue, allowing for controlled, incremental processing. This helps avoid timeouts, prevents server strain, and improves overall user experience, as long-running operations are handled in the background.

Conclusion

Drupal’s Queue API, combined with batch processing, is an effective approach for handling large workloads and deferred tasks. By setting up a queue worker, adding tasks to the queue, and processing them in manageable batches, you can efficiently manage resource-intensive tasks in Drupal, ensuring a stable and responsive website.