A reproducible protocol for Pinnacle odds API benchmarks

Protocol version 1.0. Use this method to collect and report comparable odds API measurements. This is a research protocol; measured service performance results are not yet available.

The central rule is simple: a fast response and a fresh price are different observations. Measure each with a defined clock, denominator, and scope. Publish only what the instrumentation can support.

Define the question before collecting data

Metric	Operational definition	Required evidence
REST round-trip latency	Monotonic time from dispatch until the complete response body is read	Request timing, status, bytes, connection policy, and timeout threshold
Observed update cadence	Time between distinct changes observed by the collector for a matched selection	Selection identity, change detection, receipt times, and sampling method
Source-to-client freshness	Receipt time minus a meaningful source-update timestamp	Verified timestamp semantics, clock assumptions, and timestamp resolution
Connection recovery	Time from detected interruption until defined usable state is restored	Disconnect detection, retry policy, replay or snapshot completion, and state-validity checks

These are this publication’s proposed definitions. Transport standards describe interfaces, not a service’s performance or data freshness. HTTP Semantics, WebSockets standard, SSE standard.

If there is no usable source timestamp, report that freshness is not measurable with the available fields. Do not replace it with request latency or an event’s scheduled start. A service-reported board age can be recorded under that label but cannot be promoted to independently measured per-price freshness.

Freeze a run manifest

Record the methodology version, collector commit, service, plan, account entitlements, endpoint, query scope, region, host environment, runtime, concurrency, and connection-reuse policy. Specify DNS and TLS treatment, response compression, warmup duration, measured start and end times, target sample count, and collection stop conditions.

Also record the authorized request budget, retry policy, timeouts, reconnect policy, and whether raw data may be retained or published. Keep the authorization evidence in a private review record; public manifests should contain a reference and permission status without credentials or personal details.

Predeclare the sampling schedule and market selection. A convenient successful endpoint should not stand in for a full multi-sport workload. A paid high-capacity plan should not be compared with a constrained trial without prominently identifying that difference.

Make workload equivalence explicit

Match the sport, event, bookmaker source, period, settlement rule, market, line, selection, and live/prematch state. Report unresolved mapping counts separately. An aggregator response containing many bookmakers is a different payload from a single-source snapshot; identify the scope instead of treating the two request times as interchangeable.

Run comparable workloads from the same region during overlapping periods when authorized. Alternate or randomize request order to avoid always favoring the first request in a changing market. Ensure the collector itself is not saturated. If the workloads cannot be made equivalent, present separate case studies rather than a universal winner.

Record clocks and missing observations

Use a monotonic clock for elapsed durations and timezone-bearing wall-clock timestamps for run provenance. Record the clock synchronization method and estimated uncertainty. RFC 3339 timestamps provide a representation, not a guarantee of synchronization. RFC 3339.

For freshness, document the source clock’s resolution and semantics. Clock skew can produce negative apparent age; flag it as a data-quality issue. Do not clamp it to zero or quietly drop it. If the uncertainty is comparable to the reported difference, the comparison cannot support that precision.

Persist scheduled attempts, successful responses, failed responses, timeouts, invalid payloads, throttled requests, missing fields, and unmatched markets. A timeout is a failed or censored observation with a known deadline, not an invented latency at that deadline. Report its count and threshold.

Summarize a distribution honestly

Report sample count, observation period, success rate, median, and relevant tail percentiles. State the percentile method; protocol 1.0 uses nearest rank: for sorted observations and percentile p, select the one-based rank ceil(p × n), clamped to the range 1 through n.

Latency percentiles should identify their denominator, such as successful validated requests. Put the failure rate next to them so a fast-success subset does not hide failures. Do not publish a p99 from a handful of requests as a stable service characteristic. Describe temporal clustering and coverage limitations rather than implying all samples are independent.

For reconnect behavior, define when state is usable again. The socket opening is not sufficient if the local board is still missing updates. Separate transport recovery from data-state recovery and explain any replay gaps.

Validate datasets before publication

Each dataset must carry units, run and observation timestamps, method version, source/provenance, and permission to publish. Validate nonnegative finite durations, valid timestamp formats, consistent service and run identities, declared units, and required metric-specific fields. Preserve missing values as missing.

Importers should reject inconsistent units or a dataset mislabeled as measured when it contains synthetic fixtures. Test fixtures belong in a separate directory and cannot feed production research charts, rankings, structured data, or aggregate statistics.

Before publishing results, require a complete manifest, permitted underlying evidence, reproducible calculations, explained exclusions, editorial review, and a stated limit on what the experiment establishes.

What to include with results

Every report should identify the authorized account scope and request limits, confirm publication rights, link the measurement method, and explain how the calculations were independently checked. Give readers enough context to reproduce the work and understand where its conclusions apply.

Read the transport guide for implementation tradeoffs and the normalization guide for comparable market identity. Neither replaces a measured dataset.