A benchmark is the acceptance gate between a promising spec sheet and a paid invoice: run a defined test, compare results against written KPIs, and keep the report. This page is a template, with acceptance KPIs, a test plan, failure-code classification, and a reporting structure, so the same methodology can be applied to any SMS gateway you evaluate.
Acceptance KPIs
Define targets before testing and compare every vendor on the same numbers. Delivered TPS measures messages delivered per minute rather than submitted; delivery rate measures the percentage confirmed by DLR; DLR accuracy checks that reports match actual delivery; latency at the 95th percentile matters for transactional traffic; failover time measures recovery when a SIM drops; and queue behavior checks stability under burst load.
Targets are yours to set from your workload, and a vendor's marketing numbers are not the acceptance standard.
| KPI | What it measures | Practical target |
|---|---|---|
| Delivered TPS | Messages delivered per minute | Your required rate plus headroom |
| Delivery rate | Percentage confirmed by DLR | Your market baseline |
| DLR accuracy | Reports match actual delivery | No unexplained gaps |
| Latency (p95) | Delivery time for transactional traffic | Seconds for OTP-type traffic |
| Failover time | Recovery when a SIM drops | Minutes, ideally automatic |
| Queue behavior | Stability under burst load | No message loss at tested peak |
The Test Plan
Load the gateway with your SIMs and a test list of numbers you control, send at increasing rates until delivered TPS stops rising, and record conditions with every run, including message length, encoding, SIM count, signal, and time of day. Run a stability window of several hours repeated across days, and test failure scenarios such as pulling a SIM or overloading the queue.
Sample size matters: a 10-minute test measures one moment, not a production pattern, so plan for at least a full business day.
The test script should be repeatable: a defined message template, a defined send-rate ladder, and a defined recording format. Repeatability is what makes the benchmark fair across vendors, because each supplier sees the same workload rather than a test tuned to their hardware.
The send-rate ladder deserves careful design. Start below the expected ceiling, increase in defined steps, and hold each step long enough to measure steady-state delivered TPS rather than a burst spike.
Record the step where delivered TPS stops rising and the step where delivery rate begins to fall, because those two points define the operating envelope: the first is the capacity ceiling, and the second is where pushing harder starts to cost delivery.
Failure-Code Classification
Classify every failed message by cause so the report says more than some failed. Transient failures such as network busy or expired deserve retry with backoff; permanent failures such as invalid numbers should never retry; SIM-related failures point to throttled, blocked, or out-of-balance cards; and route-related failures point to undelivered traffic via the upstream. The classification turns a delivery-rate number into a diagnosis.
The classification also tells you what to fix first. If most failures are SIM-related, the fix is fleet health; if most are content-related, the fix is copy and registration; if most are route-related, the fix is failover. Reporting the mix per run, not just a total, is what makes the benchmark actionable across vendors and across months.
The Test Report Template
Every benchmark report should contain the test dates and duration, hardware model and firmware, SIM plan and carrier details, message length and encoding, submitted versus delivered per minute, delivery-rate and latency percentiles, failover results, the failure-code breakdown, and the conditions recorded with each run. A report without conditions is not comparable, because the same gateway can look fast in one market and slow in another.
Standardize the report format across vendors so the comparison is direct: one table per run, one summary per vendor, and one acceptance sheet with the KPIs and pass-or-fail status. Procurement teams rarely have time to reconcile three different report styles, and a standard format is the difference between a clean comparison and a spreadsheet argument.
Keep the benchmark evidence after acceptance, because it serves two later purposes: the baseline for capacity reviews as traffic grows, and the reference for diagnosing a performance change months later. A gateway that delivered 500 TPS at acceptance and delivers 300 after a firmware update has a measurable regression, and the old report is the proof.
The benchmark is also the bridge to the sales conversation: a documented delivered-TPS figure per model, per SIM plan, and per market gives your procurement team and your future reseller customers a realistic planning number instead of a marketing ceiling. Publishing the conditions alongside the number is what keeps the claim credible and the decision defensible.
Finally, schedule the benchmark to be repeated, not remembered. Re-run it after major changes such as firmware updates, SIM plan changes, or new markets, because a baseline that is never refreshed becomes a historical artifact rather than a planning tool. The habit of re-testing is what turns a one-time acceptance exercise into a continuous capacity discipline.
The benchmark also validates the support relationship: a vendor that supports a buyer-led pilot, answers methodology questions in writing, and accepts the report as evidence is a vendor that will be useful after the sale. Include the support experience in the benchmark score, because acceptance is the first test of the partnership, not the last.
Finally, treat the benchmark as a shared document: give the supplier the report, invite their response, and include their comments in the final version. A vendor that engages with the evidence is one you can plan capacity with; one that dismisses it is a vendor you will fight during every later disagreement. The benchmark report is the first contract between the two sides.
The benchmark method also scales to the fleet: once the template exists, new gateways and new markets are measured with the same script, and the library of reports becomes the capacity record for the whole operation. A repeatable method is the difference between a one-time purchase test and a continuous capacity discipline.
And keep the KPIs realistic per market: the delivered-TPS figure that holds in one country with one carrier will differ elsewhere, so record the market with every result and compare within markets rather than across them. Market-aware benchmarks are the only fair comparison, and they are what make the acceptance number trustworthy.
With the market recorded and the conditions documented, the benchmark is complete: KPIs, plan, classification, report, and a decision. That is the whole template, and it is repeatable for every vendor, market, and gateway in the fleet, which is what makes the benchmark a durable procurement standard rather than a one-time exercise.
After the Benchmark
If the gateway meets the KPIs, keep the report as the acceptance record and the baseline for future capacity reviews. If it does not, the report tells you which layer failed and gives you evidence to raise with the supplier before payment. The same methodology applies to every vendor, which is what makes the benchmark a procurement tool rather than a one-time exercise.
Telarvo Expert Views
The benchmark report is the document that settles arguments. When the conditions are recorded and the KPIs are written, a failed acceptance test becomes a remediation conversation; without it, it becomes a dispute. Run the same test on your SIMs and carriers, and keep the report.
— Messaging Solutions Engineer, Telarvo Store
Validation note: KPI targets are planning guidance; set them from your workload and market baseline.
Conclusion
A benchmark with written KPIs, a repeatable plan, failure classification, and a conditions-backed report turns a purchase decision into an evidence-based decision.
Key Takeaways for B2B Buyers
Set KPIs before testing, run the same repeatable plan across vendors, classify failures by cause, and keep the report as the acceptance and baseline record.
Questions to Ask Before Committing
Ask how long an acceptance test should run, what happens if the gateway fails acceptance, and whether the supplier will support a pilot on your SIMs and carriers.
FAQs
How long should an acceptance test run?
At least a full business day, spread across days where possible, because carrier behavior varies by hour and a single burst measures only that moment.
What if the gateway fails acceptance?
Use the report to isolate the layer, give the supplier one chance to remediate with a re-test, and keep payment contingent on the documented result.
Do I need special tools to benchmark?
The gateway's software and API provide the core data, per-message DLR status and timing; add logging for the test conditions and you have a reproducible setup.