Running Your Own Independent Speed Tests on 4G Bulk Modems

Published throughput figures for bulk modems are not comparable with each other, because they were produced under different conditions and describe different things. A test you run yourself is the only way to obtain a number that applies to your site, and it takes an afternoon to set up.

This article sets out what to hold constant, why submission and acceptance have to be measured separately, how to record the conditions alongside the results, and how to compare two devices without drawing a false conclusion.

Why independent speed tests for 4G bulk modems are needed

Because the conditions behind them are different and usually unstated.

A throughput figure for a modem pool depends on four things that vary between sites: the operator and the band in use, the signal conditions where the antenna is installed, the load the subscriptions will accept, and the way the test workload was constructed. Two devices tested in different places, on different operators, with different message lengths and different pacing, produce two numbers that cannot be subtracted from each other. The comparison they invite is not supported by the way they were obtained.

The second source of confusion is the definition. A claim of a certain rate may refer to the rate at which messages are submitted to the modules, the rate at which the modules confirm acceptance, or the rate at which delivery receipts arrive. Those three figures differ substantially in practice and none of them is wrong; they are simply different measurements. A test that does not state which of them it reports cannot be compared with anything, including another test by the same organisation.

The third is that submissions are cheap and confirmations are not. A device can accept commands at a rate far above the rate at which the network will confirm them, and reporting the submission rate as throughput produces a number that looks impressive and describes a queue rather than a service. The command and response definitions involved are standardised, and the field-level detail is described in 3GPP TS 27.005 and its ETSI TS 127 005 equivalent.

Figure	What it measures	Why it differs
Submission rate	How fast commands are accepted by the device	Bounded by the host, not the network
Acceptance rate	How fast the network acknowledges submissions	Bounded by the operator and the subscription
Delivery receipt rate	How fast recipients are reached	Bounded by the recipient and the network
Confirmed completion	How fast work is finished and accounted for	The figure that describes the service

What to hold constant: message set, pacing and interval

A test that changes more than one thing produces no conclusion.

Five parameters have to be fixed before a comparison means anything. The message set, including the length and the character set, because a message that requires concatenation occupies a port for longer than a single short message. The pacing policy, because a test with no pacing measures the device and a test with pacing measures the policy. The test interval, because a figure per second and a figure per minute describe different behaviours. The subscription set, because mixing operators within a test mixes their behaviour into the result. And the environment, because a device tested in a cool room and one tested in a warm rack are not being compared.

The parameters to vary deliberately are the ones whose effect you want to understand. A useful pattern is to fix everything except the pacing, and run the same workload at three different rates, which produces a curve showing how throughput responds to pacing. Then fix the pacing and vary the load, which shows where the device or the network becomes the constraint. Changing several parameters at once produces two numbers and no explanation.

Message content matters more than it appears. The identity used to route a message is defined in ITU-T E.164, and a test that sends to a different number each time measures routing work that a test sending to the same number does not. Both are legitimate tests; they simply measure different things, and the choice should be recorded.

Measuring submission and acceptance separately

The gap between the two is where the useful information lives.

Recording only the submission rate describes how fast the queue accepts work, which is a property of the device and the host. Recording the acceptance rate describes how fast the network will acknowledge it, which is a property of the operator and the subscription. The two together give the queue depth the deployment will carry, and the queue depth is what determines whether a message submitted at nine in the morning is delivered at nine in the morning or later.

The measurement is straightforward if the timestamps are captured. Record the time each message is submitted and the time each confirmation is received, and the difference between the two rates is the accumulation. Where the submission rate exceeds the acceptance rate, the queue grows for as long as the burst lasts, and the delivery time for the last message in the burst is the burst duration plus the drain time.

A third measurement belongs alongside them: the failure rate at each rate tested. Throughput claims that omit failures describe a device that accepted a workload rather than one that delivered it, and a device that fails a tenth of its sends at a high rate has not achieved that rate in any useful sense. Recording refusals, errors and retries alongside the rate is what turns a throughput figure into a capacity figure.

Recording conditions alongside results

A result without its conditions cannot be compared with anything, including a later result of your own.

Six conditions are worth recording for every test. The operator and the band, since the same device on a different network is a different measurement. The signal level at the time, since a marginal signal changes the rate and the retry count. The ambient and module temperatures, since a device that slows as it heats is behaving normally and one that slows without heating is not. The firmware version, since behaviour changes between releases. The pacing policy and the message set, since those are the parameters the test depends on. And the time of day, because network conditions vary.

The environmental conditions that the equipment is expected to operate within are classified for fixed installations in ETSI EN 300 019-1-3, one of the ETSI standards, and recording the room conditions against that class makes the result comparable with a later test in a different season. Without it, a rate measured in winter and a rate measured in summer appear to describe different devices.

The broader practice of keeping operational records that can be relied on is described in NIST SP 800-123. For a throughput test the relevant habit is simply that the conditions are written down at the time rather than reconstructed afterwards, since the second is unreliable and the first costs nothing.

Building a throughput curve

A curve is more useful than a maximum, and it is built from repeated runs.

The shape of a throughput curve for a modem pool is fairly consistent. At low rates, the measured rate tracks the requested rate closely, because nothing is constrained. As the requested rate rises, a point is reached where the measured rate stops tracking and begins to lag, which is where the network has become the constraint. Beyond that point, further increases in the requested rate raise the queue and the failure count without raising throughput, which is the region where the device is no longer delivering more work but is accumulating more risk.

The useful number to take from the curve is not the maximum but the knee: the highest requested rate at which the measured rate still tracks it and the failure count stays low. That is the rate the deployment can sustain, and it is the figure to compare between devices, because it is bounded by the network and the subscriptions rather than by the device’s ability to accept commands.

Building the curve requires several runs at different rates, which is why it is worth setting the test up properly once. Each run should use the same message set, the same subscriptions and the same duration, and each should record the conditions. Three or four points are usually enough to locate the knee, and the runs can be repeated later to see whether it has moved.

Comparing two devices fairly

Compare them at the same requested rate, with everything else fixed.

Two devices can be compared only if the workload, the subscriptions, the location, the antenna arrangement and the environment are identical, and if the measurements are taken close enough together that the network conditions are comparable. Where those conditions cannot be met, the comparison should not be made, because the result will reflect the conditions rather than the devices. That is the honest answer to a question that procurement conversations often assume away.

Where a side-by-side comparison is possible, the figures to compare are the confirmed completion rate at a fixed requested rate, the failure count, and the time to reach a steady state. The last of those is often overlooked and matters in practice: a device that takes ten minutes to reach its working rate behaves differently from one that reaches it in thirty seconds, and the difference is invisible in a maximum throughput figure.

The alternative to a side-by-side test is a sequence of tests against a fixed reference, run at the same site with the same conditions. A reference device that stays in place while candidates are tested one at a time controls for the network better than two devices running simultaneously, and it costs less equipment. The reference does not have to be the strongest device; it only has to be known.

Where the modem pool is one part of a larger deployment, the configuration of the SMS gateway that feeds it belongs in the same record. A change on that side alters the rate the modems are asked to sustain, so a throughput result is only comparable with an earlier one if the configuration upstream of the pool is the same.

A test protocol

The protocol fixes the workload, the subscriptions and the conditions before any run, so that the result describes the deployment rather than the session. It produces a small set of comparable figures from repeated runs, and it is written to be repeated later against the same baseline.

1. Fix the message set, including length and character set.
2. Fix the pacing policy and the subscription set for the whole test.
3. Record the operator, band, signal level and firmware version before starting.
4. Record ambient and module temperature at the start and end of each run.
5. Capture a submission timestamp and a confirmation timestamp for every message.
6. Run the same workload at three or four different requested rates.
7. Record refusals, errors and retries at each rate, not only the successful count.
8. Identify the knee of the curve: the highest rate where the measured rate still tracks.
9. Repeat the reference run later to check whether the knee has moved.

{
“@context”: “https://schema.org”,
“@type”: “FAQPage”,
“mainEntity”: [
{
“@type”: “Question”,
“name”: “Why can I not compare published throughput figures for different modems?”,
“acceptedAnswer”: {
“@type”: “Answer”,
“text”: “Because a throughput figure depends on the operator, band, signal conditions, subscription limits and the test workload used. Two devices tested in different places produce numbers that cannot be subtracted from each other. The definition also varies: the figure may describe submissions accepted by the device, acknowledgements from the network, or delivery receipts received, and those three differ substantially. Record the conditions with each figure, because none of them can be compared without the context.”
}
},
{
“@type”: “Question”,
“name”: “What should I hold constant in a bulk modem speed test?”,
“acceptedAnswer”: {
“@type”: “Answer”,
“text”: “Five things: the message set including length and character set, the pacing policy, the test interval, the subscription set, and the environment. Vary one parameter at a time, because changing several produces two numbers and no explanation. A useful pattern is to fix everything except pacing and run the same workload at three rates to see how throughput responds. Write down which parameter was varied, so that the result is attributable to a cause.”
}
},
{
“@type”: “Question”,
“name”: “What is the difference between submission rate and acceptance rate?”,
“acceptedAnswer”: {
“@type”: “Answer”,
“text”: “Submission rate is how fast the device accepts commands, which is bounded by the host. Acceptance rate is how fast the network acknowledges them, which is bounded by the operator and the subscription. The gap between them determines how deep the queue becomes during a burst, and therefore how late the last message in that burst is delivered. Keep both rates in one record, because the gap between them is what sizes the queue.”
}
},
{
“@type”: “Question”,
“name”: “Which throughput figure should I use for sizing?”,
“acceptedAnswer”: {
“@type”: “Answer”,
“text”: “The knee of the throughput curve: the highest requested rate at which the measured confirmed completion still tracks the request and the failure count stays low. The maximum figure is usually beyond that point, where further increases only raise the queue and the failure count. The knee is bounded by the network and the subscriptions, which is what a sizing decision actually depends on.”
}
}
]
}