Trace Laravel Queue Workers with DataDog

Hey guys,

This topic is for anyone that uses Laravel & DataDog for their projects.

By default, DataDog won't do the APM trace for the Queue Workers (only HTTP layer). So we need a little bit of hacking, let's get started.

Before walking into the tech, I assume the ddtrace extension is installed & enabled from your servers.

Use the trace_method

Yeah, the first thing we need to do is update your code, create a simple class eg: DataDogQueueTracer.

Then invoke the trace_method from DD lib.

use Illuminate\Queue\Jobs\JobName;
use function DDTrace\trace_method;
use DDTrace\SpanData;
use Illuminate\Contracts\Queue\Job;

class DataDogQueueTracer
{
    public function register(): void
    {
        if (!extension_loaded('ddtrace')) {
            return;
        }

        trace_method(
            'Illuminate\Queue\Worker',
            'process',
            function (SpanData $span, array $args) {
                /** @var string $connectionName */
                /** @var Job $job */
                [$connectionName, $job, $options] = $args;
                $payload = $job->payload();

                $span->resource = JobName::resolve($job->getName(), $payload);

                // you can change these values below
                $span->name = 'laravel.queue.worker';
                $span->type = 'workers';
                $span->service = 'queue-workers'; // will show up as APM's service

                $span->meta = [
                    'jobPayload' => json_encode($payload),
                    'connection' => $connectionName,
                    'queue' => $job->getQueue(),
                ];
            }
        );
    }
}

From the meta above, I push the job's payload, connection name & queue name to DD too, would come in handy for the debugging or create any dashboard widgets.

Then, register it from your AppServiceProvider@register method

// AppServiceProvider.php

public function register(): void
{
    if ($this->app->runningInConsole()) {
        try {
            $this->app->make(DataDogQueueTracer::class)->register();
        } catch (Throwable) {
            Log::info('Unable to register DD queue traces');
        }
    }
}

That's it for the code-wise.

Server configuration

We need these 3 ENVs in order to enabling the DD trace under CLI (yeah Queue worker is running under CLI, we're all know that)

DD_TRACE_CLI_ENABLED=1
DD_TRACE_GENERATE_ROOT_SPAN=0
DD_TRACE_AUTO_FLUSH_ENABLED=1

Run the Queue Worker manually:

DD_TRACE_CLI_ENABLED=1 \
DD_TRACE_GENERATE_ROOT_SPAN=0 \
DD_TRACE_AUTO_FLUSH_ENABLED=1 \
php artisan queue:work

Or setup for supervisor

[program:queue-workers]
process_name=%(program_name)s_%(process_num)02d
command=php /home/myuser/myprojects/artisan queue:work
autostart=true
autorestart=true
stopasgroup=true
killasgroup=true
user=myuser
numprocs=10
redirect_stderr=true
stdout_logfile=/home/myuser/queue.log
stopwaitsecs=3600
environment=DD_TRACE_CLI_ENABLED=1,DD_TRACE_GENERATE_ROOT_SPAN=0,DD_TRACE_AUTO_FLUSH_ENABLED=1

Note: you need to reload and restart your supervisor after changed the configurations.

Finally

Let's dispatch some queue msgs and wait a bit, your APM traces from the Queue Workers would be up and available in the next 1~2 mins.

End

Well that's about it. Hope this topic will help you a bit to set everything up.

When the tasks under the Queue are being traced, you can easily determine and find the slowness / bottleneck and fasten it up.

Cheers!