A case study is a marketing document that occasionally contains engineering evidence. The skill is separating the deployment detail, which can be checked and reused, from the outcome claim, which usually cannot.
This article sets out what a useful case study contains, how to tell detail from assertion, the questions to ask about any published example, and how to build an evidence record of your own when none of the published ones answer your question.
What a useful industrial gateway case study contains
Five elements, of which most published cases contain two.
The first is the situation before the deployment: what the operator was doing, with what equipment, and what specifically was not working. The second is the requirement, expressed in a measurable form such as a peak volume, a required response time, or an environmental condition. The third is the configuration actually installed, including the model, the port or slot count and how the devices were connected. The fourth is the result, with the measurement and the conditions it was taken under. The fifth is what went wrong, which is the element that makes the rest credible and the one that is almost always missing.
A case study with the first four elements is a useful document even if it omits the fifth, because the deployment detail can be compared against your own situation. A case study with only the second and fourth is a claim. The distinction is not about honesty, since a marketing department writes what it is asked to write; it is about what the document can be used for. Only the first kind can be used to make a decision.
| Element | What it looks like | Can it be reused |
|---|---|---|
| Situation before | Existing equipment and the specific problem | Yes, as a comparison for your own case |
| Requirement | A measurable target or constraint | Yes, as a sizing reference |
| Configuration | Models, counts and connectivity | Yes, as a design starting point |
| Result | A measurement with its conditions | Only if the conditions are stated |
| What went wrong | A problem and how it was resolved | Yes, and it is the most valuable part |
Outcome claims against deployment detail
One is a number about somebody else, the other is information you can use.
An outcome claim typically takes the form of a percentage improvement or a superlative: throughput doubled, downtime eliminated, reliability improved. These statements have three properties that make them unusable for a decision. They are usually relative to an unspecified baseline. They are measured under conditions that are not described. And they are chosen from the results that were favourable, because nobody publishes the deployment that performed as expected. None of that makes them false; it makes them uninformative.
Deployment detail behaves differently. If a case study states that a deployment used a specific model with a given number of ports across a given number of sites, that is a fact that can be compared against your own requirement. If it states that the ambient temperature in the room was a particular figure and the equipment was declared against a named environmental class, that is a fact that can be checked, using the classification defined in ETSI EN 300 019-1-3, one of the ETSI standards.
The useful habit is to read for the nouns and numbers and to discount the adjectives. A paragraph that says a solution transformed operations contains nothing usable. A paragraph that says twelve gateways were installed across four sites with two operators each and the peak was a stated figure contains four things worth writing down, even if the paragraph is less enjoyable to read.
Questions to ask about any published case
Six questions convert a case study into comparable information.
What was the baseline before the deployment, and was it measured or estimated? Under what conditions was the result obtained, including the ambient temperature, the operators and the load? Is the configuration described in enough detail to be reproduced, meaning models, counts and connectivity? Over what period was the result observed, since a two-week pilot and a two-year deployment say different things? What problem was encountered and how was it resolved? And can the deployment be discussed by somebody who was involved?
The last question is the one that most often reveals the difference between a documented deployment and a constructed one. Where a reference is available, a short conversation about the problems encountered is far more informative than the written result, because the problems are where the engineering actually happened. Where no reference is available, the honest conclusion is that the case study describes an outcome rather than a deployment.
It is worth asking those questions about any source, including material published by a supplier you are inclined to trust. The purpose is not to catch anybody out but to determine what can be reused, and a supplier who answers the questions precisely has given you something more valuable than the case study itself.
Metrics worth looking for
Four figures appear in good case studies and almost never in poor ones.
Peak throughput with the measurement interval stated, because a peak without an interval is meaningless: a figure per second and a figure per hour are different claims about the same device. Delivery latency from submission to confirmation, expressed as a distribution rather than an average, because the tail is what users notice. Availability expressed as the observed downtime over a stated period, rather than as a target. And the replacement rate, meaning how many units were replaced over what population and period, which is the closest thing to a reliability figure that a real deployment can produce.
The recording practices that make those figures usable are the same as for any operational record. Keeping an audit trail with defined retention and protecting it from alteration are described in NIST SP 800-92, and the management functions that a mature operation defines around fault and performance data are set out in ITU-T M.3400. A case study that reports a figure without saying how it was measured is the marketing equivalent of a log entry without a timestamp.
What to verify with a reference call
Half an hour with an operator is worth more than a published PDF.
Where a reference is offered, four questions make the call productive. What was the requirement the deployment had to meet, in the customer’s own words? What problem appeared after commissioning that had not been anticipated? How was it resolved, and how long did that take? And what would the customer do differently if starting again? The last question is the one that produces the most useful answer, because it is where the operational reality is condensed.
The conditions of the call matter too. A reference provided by a supplier is usually a satisfied customer, which is fine as long as the questions are about specifics rather than satisfaction. Asking whether the customer would recommend the product invites a polite answer; asking what failed in the first six months invites a real one. Reference calls are most valuable when they are treated as a technical interview rather than a testimonial.
Where no reference is available, a pilot occupies the same role and produces better evidence, because it is measured under your own conditions. The resistance to finding a case study that matches your situation is often better spent running a small deployment and recording what happens, using the same elements a good case study would contain.
Building your own evidence record
The most useful case study is the one you write about your own deployment.
Six items make a record that can be compared across suppliers and reused in the next procurement cycle: the requirement as stated before the purchase, the configuration installed with models and counts, the measurements from commissioning with their conditions, the problems encountered and how they were resolved, the availability and replacement figures observed over the first year, and the questions a supplier declined to answer. The last of those is the item most often omitted and the one that prevents the same uncertainty being re-created at the next purchase.
Recording the requirement before the purchase is what makes the record comparable. It is common to find after a deployment that the system works well but nobody can say what it was supposed to achieve, which makes it impossible to judge whether the choice was good. A requirement written down in measurable terms at the outset converts every later observation into evidence about that requirement rather than an impression of general satisfaction.
The general reason a host system needs the same discipline is covered in NIST SP 800-123, which is written about server hardening but applies to any device that is expected to run unattended for years. The resistibility expectations that such equipment should meet on its power and signal ports, described in ITU-T K.20, are the kind of thing a good internal record captures because it was verified at acceptance rather than assumed from a datasheet.
An evaluation checklist
The checklist turns a published case study into something that can be assessed, by recording which elements are present and which are missing. Each item is answered from the document itself rather than from an impression of it.
- Whether the case study states the situation before the deployment.
- Whether the requirement is expressed in measurable terms.
- Whether the configuration is described in enough detail to reproduce.
- Whether the result states the conditions and the measurement interval.
- Whether any problem is described, and how it was resolved.
- Whether a reference is available, and whether the call focuses on specifics.
- Whether the metrics reported include latency distribution, availability and replacement rate.
- Whether your own requirement was written down before the purchase.
- Whether your own commissioning measurements were recorded with their conditions.
Read for the numbers and the configurations, not the adjectives. Send the deployment detail you need in a comparable form to service@telarvo.com, or review the published configurations on the SMS gateway range and the SMS gateway solution page. Background on the company and the range is on the about page. Telarvo publishes the SK-SMS gateway range, the TYH modem pools and the TGW SMS machine on its product pages, and the configurations referenced above come from those listings.
FAQ
What should a useful gateway case study include?
Five elements: the situation before the deployment, a requirement expressed in measurable terms, the configuration actually installed, a result with the conditions it was measured under, and the problems encountered and how they were resolved. Most published case studies contain two of those. The measured result is only reusable when the conditions are stated alongside it. Record which elements are present before reading the conclusion, because the gaps change its weight.
Why are percentage improvements in case studies unreliable?
Because the baseline is usually unspecified, the conditions are not described, and the results are chosen from the favourable cases, since nobody publishes the deployment that performed as expected. Those statements are not necessarily false, but they cannot be used to make a decision. Deployment detail, such as models, counts and environmental conditions, can be reused and checked. Note the conditions with the figure, because a percentage without them cannot be reused.
What should I ask on a customer reference call?
Ask what requirement the deployment had to meet, in the words of the customer, what problem appeared after commissioning that had not been anticipated, how long it took to resolve, and what the customer would do differently if starting again. The last question produces the most useful answer. Treat the call as a technical interview rather than a testimonial, because asking whether the customer is satisfied invites a polite answer.
What if no case study matches my deployment?
Run a small pilot and record it in the same form a good case study would take: the requirement before the purchase, the configuration installed, the measurements with their conditions, the problems and their resolution, and the availability and replacement figures over time. That produces evidence measured under your own conditions, which is more useful than a published example from a different site.
{
“@context”: “https://schema.org”,
“@type”: “FAQPage”,
“mainEntity”: [
{
“@type”: “Question”,
“name”: “What should a useful gateway case study include?”,
“acceptedAnswer”: {
“@type”: “Answer”,
“text”: “Five elements: the situation before the deployment, a requirement expressed in measurable terms, the configuration actually installed, a result with the conditions it was measured under, and the problems encountered and how they were resolved. Most published case studies contain two of those. The measured result is only reusable when the conditions are stated alongside it. Record which elements are present before reading the conclusion, because the gaps change its weight.”
}
},
{
“@type”: “Question”,
“name”: “Why are percentage improvements in case studies unreliable?”,
“acceptedAnswer”: {
“@type”: “Answer”,
“text”: “Because the baseline is usually unspecified, the conditions are not described, and the results are chosen from the favourable cases, since nobody publishes the deployment that performed as expected. Those statements are not necessarily false, but they cannot be used to make a decision. Deployment detail, such as models, counts and environmental conditions, can be reused and checked. Note the conditions with the figure, because a percentage without them cannot be reused.”
}
},
{
“@type”: “Question”,
“name”: “What should I ask on a customer reference call?”,
“acceptedAnswer”: {
“@type”: “Answer”,
“text”: “Ask what requirement the deployment had to meet, in the words of the customer, what problem appeared after commissioning that had not been anticipated, how long it took to resolve, and what the customer would do differently if starting again. The last question produces the most useful answer. Treat the call as a technical interview rather than a testimonial, because asking whether the customer is satisfied invites a polite answer.”
}
},
{
“@type”: “Question”,
“name”: “What if no case study matches my deployment?”,
“acceptedAnswer”: {
“@type”: “Answer”,
“text”: “Run a small pilot and record it in the same form a good case study would take: the requirement before the purchase, the configuration installed, the measurements with their conditions, the problems and their resolution, and the availability and replacement figures over time. That produces evidence measured under your own conditions, which is more useful than a published example from a different site.”
}
}
]
}