From 5 Devices to 500: What Breaks When a Pilot Becomes a Product

Here is the pattern, and it repeats across agriculture, logistics, building services, mining and manufacturing.

A team builds five units. They work. The data lands in a dashboard, the demo goes well, the board is impressed, and someone signs off on a rollout to five hundred sites. Six months later the rollout has stalled at unit ninety, two engineers are spending their week on things nobody named in the plan, and the finance team is asking why a project that cost a certain amount for five units is not costing a hundred times that for five hundred. It is costing considerably more.

Nothing went wrong technically. The pilot simply measured the wrong thing. It measured whether the concept works. Production asks whether the concept works five hundred times, unattended, for five years, when nobody involved in the pilot is available.

Those are different questions, and they have different answers.

The cost curve nobody drew at the start

When people budget a rollout, they usually reason from unit cost. If a device costs a certain amount to build, then five hundred devices cost five hundred times that, less a volume discount. That part is roughly true, and it is also the smaller half of the problem.

The larger half is the engineering that a pilot legitimately defers. It stays close to zero while a human is compensating for it, then arrives suddenly, in steps, at thresholds that have nothing to do with how many devices you have built and everything to do with what you are now legally and operationally obliged to do.

That shape is the whole argument. The steps are fixed costs. Certification costs what it costs whether you certify for sixty units or six thousand. A production test fixture costs what it costs. A provisioning and update system costs what it costs. Spreading those over a larger fleet is the only lever you have, which is exactly why discovering them late is so damaging: you meet them at the worst possible fleet size, with the least possible schedule.

Part one: the software that was never there

1. Provisioning, or how a device knows who it is

Every device needs an identity: a unique serial, a certificate or key, a server address, a site configuration. In a pilot, an engineer plugs in a cable and types them in. It takes four minutes and nobody records it as work.

At five hundred units that same four minutes is more than thirty hours of factory labour, performed by someone whose job is assembly, not security. Worse, the credentials now pass through human hands, which means they exist in a spreadsheet somewhere, which means your fleet has a single file that compromises all of it.

Production provisioning means keys generated on the device or injected by an automated fixture, a per-unit identity that is unique and unguessable, a record written to a database at the moment of programming, and a device that arrives on site already knowing how to reach the platform without anyone typing anything. That is a real piece of software with a real build cost, and it is invisible until the day you need it.

2. Over-the-air updates, and the rollback you have not thought about

Five devices are all within driving distance. Five hundred are not. The moment the fleet exceeds what you can physically visit in a week, firmware updates stop being a convenience and start being the thing your entire support model rests on.

Most teams get as far as "we can push new firmware." Far fewer have answered the questions that decide whether the update system helps or destroys you:

• What happens if power fails halfway through a write? A device that cannot boot cannot be updated remotely, and a hundred of them is a truck roll.
• How does a device get back to the previous version by itself when the new one is broken?
• How do you release to ten devices, watch them, then release to the rest, rather than betting the fleet on one push?
• Are updates signed, and does the bootloader refuse anything unsigned? An unauthenticated update path is a remote code execution path.
• What does a device on a metered connection do about a large firmware image?

Dual-bank flash with automatic rollback, signed images and staged release is not gold plating. It is the difference between a bad release costing an afternoon and a bad release costing a field campaign.

3. Fleet visibility, or the difference between "it is broken" and "which one"

With five devices you know each one personally. You notice when the data stops. With five hundred, silence is normal: units go offline for legitimate reasons all the time, and the question stops being "is something wrong" and becomes "which of these forty quiet devices is actually a problem."

What you need is unglamorous and decisive. Every device reporting firmware version, uptime, connection quality, battery or supply voltage, and error counters. A view that sorts the fleet by health instead of by name. Alerting that fires on a pattern rather than a single event, because a single device dropping out at 2 am is weather and forty devices dropping out at 2 am is your platform.

Teams that skip this do not find out. They get a phone call from a customer, weeks after the fact, and then spend a fortnight reconstructing what happened from data that was never designed to answer the question.

4. The cloud behaves differently when everything reconnects at once

Five devices generate a trickle. Five hundred generate a trickle too, on average, and average is not the figure that matters.

The figure that matters is what happens when connectivity is restored after an outage. Every device reconnects in the same second, replays whatever it buffered, and your ingest tier meets its entire fleet simultaneously with a backlog attached. If each device retries on a fixed interval, they stay synchronised and hit you again in the same second, and again after that.

The fixes are well understood and almost never present in pilot code: randomised reconnect backoff so the fleet spreads itself out, rate limiting at the ingest edge, a queue between ingest and processing so a spike becomes latency instead of data loss, and idempotent writes so a replayed message does not corrupt a total. Retrofitting these into a running fleet is possible. Building them in at the start costs a fraction as much.

Part two: the physics and the paperwork

5. Connectivity economics, which quietly decide the business case

A pilot on a handful of SIMs feels free. Multiply by fleet size and contract length and connectivity becomes one of the largest recurring lines in the whole programme, and it is the line that determines whether the product has a margin.

Two habits change the outcome. First, measure actual bytes per device per day during the pilot rather than estimating them, then multiply properly. Second, look at what those bytes contain. Verbose formats, full payloads sent when nothing has changed, and per-message overhead that exceeds the reading itself are all normal in prototype code and all straightforwardly fixable. Sending deltas and batching sensibly frequently cuts volume by an order of magnitude, which can be the difference between a viable radio choice and an unviable one.

The choice between cellular, LoRaWAN, Wi-Fi and satellite is not only a technical question about range and power. It is a question about what the fleet costs to run for five years, and it is much easier to answer with a real measurement in hand. We have written a longer comparison in choosing IoT connectivity.

6. Certification, which sits on the critical path whether you planned for it or not

A pilot on your own site is one thing. Supplying devices into the Australian market is another, and compliance is not optional. Electromagnetic compatibility, radio spectrum requirements, electrical safety where relevant, and the labelling that goes with them all have to be satisfied before you can legally sell.

The trap is sequencing. Teams treat compliance as a final gate, so testing happens after the design is frozen. A failure at that point is not a small thing: it means a board revision, new samples, and a repeat test, on a schedule set by a lab's queue rather than yours. Bringing compliance into the conversation while the layout, antenna placement and enclosure are still changeable turns a restart into an adjustment. Our guide to RCM compliance for electronic products in Australia covers what applies and when.

7. Production test, because "it worked when we built it" does not scale

Five units get tested by the person who built them, who knows what right looks like. Five hundred get tested by someone who has never seen the product before, on a line, against a written procedure, in a fixed amount of time.

That difference demands a test fixture: something that powers the board, exercises every interface, checks that sensors read plausible values, confirms the radio transmits, writes the calibration constants and the identity, and gives a clear pass or fail with a logged result tied to the serial number. Without it, defects reach the field, and a defect in the field costs many multiples of a defect caught on the bench.

Calibration deserves its own mention. Analogue sensing that seems accurate across five units often is not accurate across five hundred, because component tolerances that averaged out in a small sample become a visible spread in a large one. Per-unit calibration at production test, stored on the device, is usually the cheapest way to fix a problem that is otherwise structural.

8. Supply chain, where lead times replace prices as the binding constraint

Prototype purchasing is convenient: single units from a distributor, next-day delivery, whatever is in stock. Volume purchasing is a different discipline. Minimum order quantities, lead times measured in months for some parts, lifecycle status that determines whether a component will still exist for the life of your product, and the pricing tiers that only appear at real quantities.

The practical requirement is a bill of materials that has been reviewed for availability rather than only for function: second sources identified for anything critical, lifecycle status checked, and long-lead items ordered against a forecast instead of an order. A single part with a long lead time and no alternative can hold an entire production run, and that risk is visible months in advance if anyone looks.

9. Field service, and the cost of not knowing why

Every fleet has failures. The variable is not whether they happen but what each one costs you.

A device that fails with no diagnostic information means someone travels to site, brings it back, and someone else attempts to reproduce a fault with no context. A device that records why it reset, what its supply voltage was doing, and what its last error was turns that into a remote diagnosis and often a remote fix. Design for the second case: retain diagnostics across reboots, make them retrievable remotely, and label units so that a serial number maps to a build, a firmware version and a site without anyone guessing.

The pattern behind all nine

Every item on this list is the same trade in a different costume. At pilot scale a human absorbs the work. At production scale nobody can, so the work has to become software, a fixture, a process or a document. It never disappears.

How to run a pilot that survives contact with production

The good news is that almost all of this is a sequencing problem rather than a money problem. A pilot designed with the rollout in mind costs slightly more and removes most of the cliff.

1. Decide the target fleet size before you build the pilot. Fifty, five hundred and fifty thousand are three different products. The number changes the radio, the provisioning approach, the test strategy and the cloud architecture. Guessing is fine. Not deciding is not.
2. Provision the pilot units the way you intend to provision production units. Even a crude automated flow proves the concept and stops manual habits becoming permanent.
3. Ship an over-the-air update to the pilot fleet deliberately, then roll one back. If you have not rolled back, you do not have an update system. You have a hope.
4. Measure real data volume for a fortnight. Then do the arithmetic for the full fleet across the contract term, and check the answer against your margin.
5. Get a compliance opinion while the design is still soft. An early conversation about antenna placement and layout is worth more than any amount of testing after the fact.
6. Run the pilot through one real environmental extreme. A full summer, a full winter, or a controlled equivalent. Thermal behaviour in particular is not something you can reason your way out of, as we covered in edge AI boards under real thermal load.
7. Write the production test procedure during the pilot, while the people who understand the product are still the people writing it.

Seven steps. Each of them is a few days of deliberate effort during a phase where the schedule is usually forgiving, and each removes a step from the cost curve at the point where the schedule is not.

Frequently asked questions

Do we need to redesign the hardware, or can we scale what the pilot used?

It depends what the pilot proved. If it proved the concept using development boards, jumper wiring and hand-soldered modules, expect a production design pass: a single board, qualified components, a real enclosure, and design choices made for manufacture rather than for speed. If it proved the implementation, with a purpose-built board and a considered bill of materials, you may only need refinement. The honest way to find out is a design review before the purchase order, not after.

How long does the pilot to production transition usually take?

The dominant terms are certification and manufacturing setup, and both are largely outside your control once started. A realistic plan reserves months rather than weeks and runs the workstreams in parallel: compliance and production test can be prepared while fleet software is being built. Sequential planning is what turns a manageable transition into a stalled one.

Can we scale the software first and deal with hardware later?

Partly, and it is often the right order. Provisioning, updates, fleet visibility and ingest behaviour can all be built and proven against a small fleet, which de-risks the largest software steps early. What you cannot defer is anything that touches the board itself, because certification and production test both depend on a frozen design.

What is the single most common thing teams miss?

Rollback. Almost everyone builds a way to push firmware. Far fewer build a way for a device to recover by itself when the firmware it just received does not work. That gap converts one bad afternoon into a field campaign, and it is the cheapest of all these problems to fix in advance.

We already stalled partway through a rollout. Is it recoverable?

Usually, and more cheaply than starting again. A stalled rollout normally has one or two binding constraints rather than nine, and identifying which ones are actually blocking you is a short piece of work. The units already in the field are an asset: they carry real data about how the design behaves, which is exactly what a production design pass needs.

Further reading

• IoT development services
• Hardware product development, concept to production
• PCB prototype to production: timelines and costs
• Choosing IoT connectivity: LoRaWAN, NB-IoT, Wi-Fi, cellular
• RCM compliance for electronic products in Australia
• How to write a hardware product brief

Incendio Solutions is a Melbourne engineering studio that builds edge AI, embedded systems, IoT and the software that runs them, from schematic to production line - 15 production programs shipped. If you have a pilot that works and a rollout that has not started, or has started and stopped, tell us where it is stuck and we will tell you straight what it takes to unstick it.