Live Crawler

Three intervals, and vendors usually quote only the smallest two. Change-to-detection is bounded by poll interval and is by far the largest. Detection-to-emission is typically under a second. Emission-to-delivery is a few hundred milliseconds.

We report detection_window_s on every event and publish p50 and p95 monthly, because the first interval is the one that determines whether you can act. A vendor quoting sub-second real time is describing their internals, not your latency.

No, and no responsible provider can. Ten thousand entities at 30-second intervals is 20,000 requests per minute against a source, sustained — that is a load test, and sources block it.

The design is tiered: a small priority set at sub-minute intervals, a working set at several minutes, the long tail hourly or daily. Tier sizing is the main scoping conversation and it usually shrinks the initial ask substantially while improving the result.

Thresholds per field so trivial movements do not emit, direction filters so you can take decreases only, oscillation damping, revert detection within a configurable window, and bulk-change summarisation so a category-wide reprice arrives as one event rather than three thousand.

Suppressed changes are still recorded with a suppressed_reason and available on request. Nothing is invisible; it just does not interrupt you. Tuning happens during the pilot against your thresholds.

Exponential backoff retry, then a dead letter store you can replay from. Event IDs are idempotent so redelivery does not double-process, and the delivery object records attempts and latency for auditing.

Silent event loss is the worst outcome in an event system, so we prefer at-least-once delivery with idempotency over at-most-once. Your consumer should be idempotent on event_id.

Not better, different, and it depends entirely on your response process. If nobody acts on a change within hours, sub-minute detection buys nothing and scheduled files are simpler and cheaper.

Events earn their cost when something acts quickly — a repricing engine, a campaign pause, a bid team. Many clients run both: events on the priority tier, scheduled files for the full dataset and analytics.

Yes, as event types. New listing appearance fires on first observation. Delisting requires confirmed absence across polls before emitting, because a single missed fetch is not a delisting.

That confirmation requirement adds latency to delisting events deliberately. Emitting a delisting on one absent poll would produce constant false positives from transient errors, which is worse than being a few polls slower.

Fewer than clients expect. It depends on the source's tolerance and our politeness floor per host. Some large sites sustain it on a small entity set; many do not.

We assess this per source during scoping and quote the achievable interval rather than a generic figure. Where sub-minute is not sustainable, saying so upfront is better than promising it and delivering minutes with an excuse.

Yes, and most clients do. The same pipelines feed events, API responses and scheduled files. Events handle the fast path; files handle the full population and analytics.

Running both on one contract also means the event stream and the dataset agree, which is not guaranteed when they come from different vendors with different collection times.

We quote individually, driven almost entirely by priority-tier size multiplied by poll frequency. The long tail is comparatively cheap; the sub-minute tier is where cost concentrates.

A focused priority set with tiered standard and long-tail coverage sits at the lighter end. Large priority tiers at sub-minute intervals sit considerably higher, and are usually not the right design. One scoping call, a free pilot stream on your own entities within 24 hours with latency percentiles reported, then a fixed monthly quote. Request a quote.