Averages Lie at Scale

p50 says the median was fine.

At 2M events per day, the slow 1% is 20,000 bad experiences. SLAs live in p95 and p99.

If you only chart http_request_duration, you measure one room in a six-room house.

Technology Stack

Layer	Choice	Why
Logging	Structured JSON (Monolog)	Queryable fields, no vendor lock-in
Storage	CloudWatch / Loki / ELK	Search by `event_id`, aggregate latencies
Cache	Redis	Read gates after trace proves DB is hot
DB	MySQL + slow query log	Find N+1 and missing indexes
Queue timing	Laravel job `handle()` logs	Worker segment separate from HTTP
Dashboards	Grafana or CloudWatch Metrics	p50/p95/p99 per hop — not one blended line

Start with logs. Add APM when log queries become daily habit.

The Full Chain

One partner webhook is never one HTTP call.

Partner POST → API ingest
            → RabbitMQ publish
            → worker pickup (queue wait)
            → enrich + MySQL write
            → downstream notify
            → mark complete

Sample Dashboard — One Slow Event Cohort

Metrics derived from structured logs (event_id, hop, duration_ms):

Hop	p50	p95	p99	Notes
`ingest`	38ms	52ms	71ms	API looks healthy
`enqueue`	4ms	9ms	14ms	Not the problem
`queue_wait`	120ms	890ms	2.1s	Dominant gap
`process`	180ms	240ms	310ms	Acceptable
`downstream_notify`	400ms	2.8s	4.2s	Second gap — sync HTTP in worker
`complete`	12ms	18ms	25ms	Fine
end-to-end	780ms	3.4s	6.1s	What partner sees

API-only dashboard: 40ms green.
Pipeline dashboard: p99 at 6.1s red.

The bottleneck was queue wait and sync downstream calls — not the controller.

Trace One Webhook

Structured logs only. No APM required on day one.

// Ingest
Log::info('pipeline.hop', [
    'event_id' => $payload['id'],
    'hop' => 'ingest',
    'duration_ms' => $timer->ms(),
    'partner_id' => $payload['partner_id'],
]);

ProcessPartnerEvent::dispatch($payload['id'], $payload);

Log::info('pipeline.hop', [
    'event_id' => $payload['id'],
    'hop' => 'enqueued',
    'duration_ms' => $timer->ms(),
]);

// Worker
public function handle(PartnerEventProcessor $processor): void
{
    $t0 = hrtime(true);

    Log::info('pipeline.hop', [
        'event_id' => $this->eventId,
        'hop' => 'worker_start',
        'queue_wait_ms' => $this->queueWaitMs(), // job property set at dispatch
    ]);

    $processor->run($this->payload);

    Log::info('pipeline.hop', [
        'event_id' => $this->eventId,
        'hop' => 'processed',
        'duration_ms' => (hrtime(true) - $t0) / 1_000_000,
    ]);
}

Query:

filter hop IN ("ingest","enqueued","worker_start","processed")
| stats p50(duration_ms), p95(duration_ms), p99(duration_ms) by hop

Subtract timestamps for one event_id when debugging a single complaint.

Where Time Hides

Queue wait — too few queue:work processes or slow jobs blocking prefetch.

Sync HTTP inside workers — partner notify at 3s ties your p99 to their p99.

N+1 in consumers — fine at low volume, fatal at 2M/day.

Cache cold after deploy — Redis miss storm hits MySQL together.

Fix Order

Measure every hop — log event_id + hop + duration_ms
Fix dominant p95 gap — usually queue wait or external I/O
Then queries — indexes, batch inserts, eager loads in jobs
Then micro-opts — opcode cache, serializer tweaks

Skipping step 1 optimizes the wrong 40ms.

After Tracing — Targeted Fixes

Dashboard signal	Fix
`queue_wait` p95 high	Scale workers; reduce job time; lower prefetch
`downstream_notify` p99 high	Async notify + status polling; circuit breaker
`process` p95 high	MySQL slow log; batch writes; Redis read gate
`ingest` p99 high	Now worth profiling PHP — rarely first

Redis cut p95 roughly 45% on one read-heavy gate — after logs proved MySQL was the bottleneck, not Laravel routing.

Key Takeaways

API latency is one hop in pipeline latency.

Build per-hop p50/p95/p99 dashboards from structured logs.

Partner webhooks need event_id tracing from ingest to complete.

Fix the largest p95 gap first — usually queue wait or sync I/O.

Performance at scale is chain engineering, not endpoint tuning.