Seth Phat
Seth Phat Blog

Follow

Seth Phat Blog

Follow
Trace Laravel Queue Workers with DataDog

Photo by Pauline Loroy on Unsplash

Trace Laravel Queue Workers with DataDog

Seth Phat's photo
Seth Phat
·Jan 2, 2023·

2 min read

Hey guys,

This topic is for anyone that uses Laravel & DataDog for their projects.

By default, DataDog won't do the APM trace for the Queue Workers (only HTTP layer). So we need a little bit of hacking, let's get started.

Before walking into the tech, I assume the ddtrace extension is installed & enabled from your servers.

Use the trace_method

Yeah, the first thing we need to do is update your code, create a simple class eg: DataDogQueueTracer.

Then invoke the trace_method from DD lib.

use Illuminate\Queue\Jobs\JobName;
use function DDTrace\trace_method;
use DDTrace\SpanData;
use Illuminate\Contracts\Queue\Job;

class DataDogQueueTracer
{
    public function register(): void
    {
        if (!extension_loaded('ddtrace')) {
            return;
        }

        trace_method(
            'Illuminate\Queue\Worker',
            'process',
            function (SpanData $span, array $args) {
                /** @var string $connectionName */
                /** @var Job $job */
                [$connectionName, $job, $options] = $args;
                $payload = $job->payload();

                $span->resource = JobName::resolve($job->getName(), $payload);

                // you can change these values below
                $span->name = 'laravel.queue.worker';
                $span->type = 'workers';
                $span->service = 'queue-workers'; // will show up as APM's service

                $span->meta = [
                    'jobPayload' => json_encode($payload),
                    'connection' => $connectionName,
                    'queue' => $job->getQueue(),
                ];
            }
        );
    }
}

From the meta above, I push the job's payload, connection name & queue name to DD too, would come in handy for the debugging or create any dashboard widgets.

Then, register it from your AppServiceProvider@register method

// AppServiceProvider.php

public function register(): void
{
    if ($this->app->runningInConsole()) {
        try {
            $this->app->make(DataDogQueueTracer::class)->register();
        } catch (Throwable) {
            Log::info('Unable to register DD queue traces');
        }
    }
}

That's it for the code-wise.

Server configuration

We need these 3 ENVs in order to enabling the DD trace under CLI (yeah Queue worker is running under CLI, we're all know that)

DD_TRACE_CLI_ENABLED=1
DD_TRACE_GENERATE_ROOT_SPAN=0
DD_TRACE_AUTO_FLUSH_ENABLED=1

Run the Queue Worker manually:

DD_TRACE_CLI_ENABLED=1 \
DD_TRACE_GENERATE_ROOT_SPAN=0 \
DD_TRACE_AUTO_FLUSH_ENABLED=1 \
php artisan queue:work

Or setup for supervisor

[program:queue-workers]
process_name=%(program_name)s_%(process_num)02d
command=php /home/myuser/myprojects/artisan queue:work
autostart=true
autorestart=true
stopasgroup=true
killasgroup=true
user=myuser
numprocs=10
redirect_stderr=true
stdout_logfile=/home/myuser/queue.log
stopwaitsecs=3600
environment=DD_TRACE_CLI_ENABLED=1,DD_TRACE_GENERATE_ROOT_SPAN=0,DD_TRACE_AUTO_FLUSH_ENABLED=1

Note: you need to reload and restart your supervisor after changed the configurations.

Finally

Let's dispatch some queue msgs and wait a bit, your APM traces from the Queue Workers would be up and available in the next 1~2 mins.

End

Well that's about it. Hope this topic will help you a bit to set everything up.

When the tasks under the Queue are being traced, you can easily determine and find the slowness / bottleneck and fasten it up.

Cheers!

 
Share this