Ingest delays

Incident Report for Scout

Resolved

This incident has been resolved.
Posted May 22, 2025 - 20:27 MDT

Monitoring

Zookeeper corruption has been rooted out. Things appear healthier and catching up in all cases.
Posted May 22, 2025 - 15:05 MDT

Update

We are not to full resolution yet.
Posted May 22, 2025 - 11:43 MDT

Update

Throughput has improved although behavior of individual partitions remains a problem and is still causing delays in some cases.
Posted May 21, 2025 - 23:42 MDT

Identified

It has been a long day with kafka. We continue to experience instability, causing lag and dropped payloads.
Posted May 21, 2025 - 20:13 MDT

Monitoring

An alternate approach has been applied, we are watching.
Posted May 21, 2025 - 10:13 MDT

Identified

The initial fix was unsuccessful, certain accounts are now substantially delayed.
Posted May 21, 2025 - 09:08 MDT

Monitoring

A fix has been implemented and we are monitoring the results.
Posted May 21, 2025 - 08:33 MDT

Investigating

We are currently investigating this issue.
Posted May 21, 2025 - 07:55 MDT
This incident affected: Application Monitoring.