Hey guys,
This topic is for anyone that uses Laravel & DataDog for their projects.
By default, DataDog won't do the APM trace for the Queue Workers (only HTTP layer). So we need a little bit of hacking, let's get started.
Before walking into the tech, I assume the ddtrace
extension is installed & enabled from your servers.
Use the trace_method
Yeah, the first thing we need to do is update your code, create a simple class eg: DataDogQueueTracer
.
Then invoke the trace_method
from DD lib.
use Illuminate\Queue\Jobs\JobName;
use function DDTrace\trace_method;
use DDTrace\SpanData;
use Illuminate\Contracts\Queue\Job;
class DataDogQueueTracer
{
public function register(): void
{
if (!extension_loaded('ddtrace')) {
return;
}
trace_method(
'Illuminate\Queue\Worker',
'process',
function (SpanData $span, array $args) {
/** @var string $connectionName */
/** @var Job $job */
[$connectionName, $job, $options] = $args;
$payload = $job->payload();
$span->resource = JobName::resolve($job->getName(), $payload);
// you can change these values below
$span->name = 'laravel.queue.worker';
$span->type = 'workers';
$span->service = 'queue-workers'; // will show up as APM's service
$span->meta = [
'jobPayload' => json_encode($payload),
'connection' => $connectionName,
'queue' => $job->getQueue(),
];
}
);
}
}
From the meta
above, I push the job's payload, connection name & queue name to DD too, would come in handy for the debugging or create any dashboard widgets.
Then, register it from your AppServiceProvider@register
method
// AppServiceProvider.php
public function register(): void
{
if ($this->app->runningInConsole()) {
try {
$this->app->make(DataDogQueueTracer::class)->register();
} catch (Throwable) {
Log::info('Unable to register DD queue traces');
}
}
}
That's it for the code-wise.
Server configuration
We need these 3 ENVs in order to enabling the DD trace under CLI (yeah Queue worker is running under CLI, we're all know that)
DD_TRACE_CLI_ENABLED=1
DD_TRACE_GENERATE_ROOT_SPAN=0
DD_TRACE_AUTO_FLUSH_ENABLED=1
Run the Queue Worker manually:
DD_TRACE_CLI_ENABLED=1 \
DD_TRACE_GENERATE_ROOT_SPAN=0 \
DD_TRACE_AUTO_FLUSH_ENABLED=1 \
php artisan queue:work
Or setup for supervisor
[program:queue-workers]
process_name=%(program_name)s_%(process_num)02d
command=php /home/myuser/myprojects/artisan queue:work
autostart=true
autorestart=true
stopasgroup=true
killasgroup=true
user=myuser
numprocs=10
redirect_stderr=true
stdout_logfile=/home/myuser/queue.log
stopwaitsecs=3600
environment=DD_TRACE_CLI_ENABLED=1,DD_TRACE_GENERATE_ROOT_SPAN=0,DD_TRACE_AUTO_FLUSH_ENABLED=1
Note: you need to reload and restart your supervisor after changed the configurations.
Finally
Let's dispatch some queue msgs and wait a bit, your APM traces from the Queue Workers would be up and available in the next 1~2 mins.
End
Well that's about it. Hope this topic will help you a bit to set everything up.
When the tasks under the Queue are being traced, you can easily determine and find the slowness / bottleneck and fasten it up.
Cheers!