Connected devices: updates, connectivity loss and diagnostics
What happens during an outage and after reconnection: a worked storage budget, duplicate delivery, catch-up throughput and updates with a recovery path.
Connectivity is only one operating state
A device can be connected to Wi-Fi while still failing to deliver data to an application. The cause might be an unavailable server, name resolution failure or a rejected request. A connectivity icon therefore does not establish successful operation of the complete system. We need to know which stage actually succeeded.
Consider a hypothetical logger that measures once per second, stores each result locally and sends it to a server. The user needs both a current reading and historical data. During normal operation these needs are easy to conflate because everything arrives quickly. An outage reveals that freshness of the latest value and completeness of the archive are separate properties.
We will follow a day without connectivity, a restart and synchronisation after reconnection. The numbers are assumptions for the example, not specifications of an actual product. The goal is to define system behaviour before a failure forces improvisation.
States should describe behaviour
A useful model distinguishes startup, online operation, offline operation, recovery and updating. Each state needs entry and exit conditions and a list of available functions. Losing the server need not stop a logger’s measurements. For a controller, also define which decisions remain local and how it behaves without new commands.
In the diagram, “Online” means that data can be exchanged with the server, and “connection” refers to that entire path rather than Wi-Fi alone. The recovery branch shows communication being re-established. The drawing omits startup without a server, failed connection attempts and update failures; these need their own transitions in a real device. Queue recovery and configuration checks must also be included at startup. Not every fault should restart the entire device. Repeated resets can merely hide the cause and increase the risk of data loss.
Show measurement age or the time of the last successful delivery with an accurate label in the interface. The last known temperature can still be useful, but it should not look like a fresh reading. After reconnection, distinguish “network available” from “history synchronised”.
A day offline: how much storage is needed?
Assume each record occupies 32 B, including its identifier and time metadata. At one record per second, a day produces 86,400 records. Their total size is 86,400 · 32 B = 2,764,800 B, or approximately 2.64 MiB, where 1 MiB = 1,048,576 B.
That covers only the records. Allow for storage structure overhead, alignment, information identifying incomplete writes, free space for block management and the device’s other data. A memory nominally larger than 2.64 MiB is not automatically a one-day queue. The budget must reflect the actual storage design.
Capacity and endurance are separate constraints. Flash memory often cannot overwrite an arbitrary byte without first erasing a larger block. Updating the same queue pointer every second can wear one region much faster than the rest. Sequential writing, wear distribution and batching operations help, but require a deliberate trade-off: data held only in RAM can disappear on power loss.
For this logger, choose an explicit policy: when the buffer fills, discard the oldest unacknowledged records, retain the newest and record the identifiers of the discarded range. This establishes removal of local copies, not necessarily absence on the server: some records may have arrived even though their acknowledgements did not. This favours freshness over complete history; it is not a rule for every product. A system requiring a complete record would need a different response, such as stopping acquisition with an alarm. Silent overwriting without marking a possible gap invites incorrect interpretation. Actual missing records can be established by comparing identifiers with those the server accepted.
Restarting halfway through a write
Power loss can leave an incomplete record. On startup, the device must distinguish committed data from unfinished operations and restore the queue position. Merely storing the last index is insufficient if index and record updates can be interrupted at different times.
A practical structure might use a length, sequence number, checksum and completion marker, according to the memory’s properties. A checksum helps detect corruption but does not establish authenticity. The important point is a tested recovery rule: an incomplete record must not be accepted as valid, and previously committed records should remain available within the declared fault tolerance.
Test power removal during record writing, metadata updates and block reclamation. Use the actual storage medium and driver. A file’s behaviour on a computer does not prove that memory in a microcontroller behaves identically.
The server stored the data, but its response was lost
Suppose the device sends record 1042. The server stores it correctly, but the acknowledgement never arrives. The device sees a timeout and retries. Without protection, the server stores a second measurement even though only one physical measurement was taken.
The identifier should be assigned before the first transmission and remain unchanged through retries and restarts. One approach combines device identity with a persistently managed sequence number. If the counter restarts from zero, an additional session distinction is needed and must remain attached to restored older records. A timestamp alone may be insufficient, especially with an unreliable clock.
The backend, the part of the system running on the server, must couple duplicate recognition with committing the effect in one atomic operation: other requests and failures must not split those two steps. Storing a measurement can use a transaction and a uniqueness constraint on its identifier in the database. Separate “check whether it exists” and “store it” steps can fail under concurrent requests. Reusing an identifier with different content should expose a conflict rather than silently replace the measurement.
The device removes a record from its queue after acknowledgement of durable acceptance by the server application, according to the agreed response semantics. Transport-layer packet delivery does not necessarily mean that. If the device resets before recording acknowledgement locally, it will resend the record; the receiver should recognise the duplicate. Duplicate tracking must be retained long enough to cover possible late retries.
This is idempotent processing: repeated delivery of the same operation does not repeatedly create its effect. It does not mean that the network delivers every message exactly once. Amazon Builders’ Library explains the problem through lost responses and request identifiers.
Reconnection does not empty the queue immediately
After a one-day outage, the backlog contains 86,400 records, provided the buffer held them all. Assume the server durably accepts 10 distinct records/s while the device continues producing one new record/s. The queue shrinks by 9 records/s. Clearing it takes 86,400 / 9 = 9600 s, or 2 hours 40 minutes, at constant throughput and without further outages.
If sending achieves only one record/s, the backlog does not shrink at all. At a lower rate it continues growing. Ordering also needs a decision: sending only the oldest records delays a fresh view. In this example, reserve one of the ten places per second for the current record and the other nine for history. The backend orders data by measurement rather than arrival.
Retries should have a bounded cost, increasing intervals and a random timing spread known as jitter. After exhausting the current retry budget, the device returns to a quiet offline mode and tries again later. Hundreds of devices recovering together should not overwhelm the server with synchronised requests. Configuration or permission errors need a different response from temporary unavailability.
Every attempt consumes energy too. The battery budget in “The prototype works. What has actually been proven?” shows how longer activity changes lifetime. Connectivity recovery policy therefore belongs to power-system design as well. AWS: Timeouts, retries, and backoff with jitter describes practical ways to control retries.
Measurement time is not delivery time
A record delivered at 14:00 may have been produced in the morning. Store measurement time, server receipt time and time quality separately. After restarting without a retained clock, the device may know sample order and time since boot without knowing the correct calendar time.
A sequence number and monotonic time, an elapsed-time counter independent of calendar clock corrections, help reconstruct events within a session. Later clock synchronisation may allow earlier sample times to be estimated, but should not turn an estimate into an apparently certain measurement. Changing calendar time must not cause identifier collisions or reverse the logical ordering of events.
Updating needs a path back
Downloading a file securely does not complete an update. Before execution, the device should verify image authenticity and integrity, hardware compatibility and whether the version is permitted. A firmware image is a file containing the device program; authenticity means trusted origin, while integrity means its contents have not been changed without authorisation. An encrypted connection protects transport; trust in executable code also requires appropriate verification and key management. RFC 9019 describes architectures for these processes.
One example sequence downloads to an inactive area, checks the image, performs a trial boot and confirms successful operation. Confirmation should follow meaningful checks, such as reading configuration and starting measurements. Merely reaching the application’s first instruction is a weak criterion. Conversely, requiring server contact at all costs may reject healthy firmware during an ordinary network outage.
Two slots and reversion after an unconfirmed boot are one possible solution; MCUboot documents such a mechanism. In this mechanism, reversion after an unconfirmed boot occurs on the next reset. If the program hangs, a reset must be triggered, for example by a watchdog that monitors regular signs of program activity. Two slots alone do not ensure automatic escape from a hang. Memory cost and hardware architecture may require another recovery route. Recovery must remain consistent with security policy so that vulnerable versions cannot be run arbitrarily.
Configuration adds another complication. If new firmware irreversibly changes the stored data format, an older image may boot but fail to read it. The update strategy should include format versioning, compatible migration or retaining the copy needed for recovery. Update testing ends only after checking operation with the actual stored state, including after interrupted operations.
Diagnostics should explain the failure
Record hardware and firmware versions, reset cause, update stage, queue occupancy, locally discarded record count and the last successful acknowledgement. Mark confirmed gaps on the server separately from records whose delivery outcome is unknown. Events need time or ordering and a session identifier. A thousand identical “network error” messages cannot replace this context; aggregate repetitions and retain important state transitions.
Scroll horizontally if needed.
| Disturbance | Expected behaviour in this example | Diagnostic evidence |
|---|---|---|
| One day without a server | Acquisition continues and records enter the buffer | Oldest-record age and queue occupancy |
| Full buffer | Oldest data is discarded according to the chosen policy | Identifier range and locally discarded record count |
| Lost acknowledgement | The same record returns without duplicating its effect | Record identifier and retry outcome |
| Restart during a write | Incomplete record rejected; committed data recovered | Reset reason and queue recovery result |
| Failed trial boot | Controlled recovery of compatible firmware and configuration | Image versions, failure stage and reversion cause |
Do not record secrets or unnecessary personal data. Bound log size, retention and access. Diagnostics must work under failure without consuming storage intended for measurements. NISTIR 8259A treats updates and cybersecurity state awareness as device capabilities.
What to agree before installation
For each fault, define allowed data loss, recovery time and the message shown to the user. Introduce disturbances at different moments and in realistic combinations, such as restarting with a full queue. Check whether recorded information is sufficient to reproduce the problem without the device’s developer present.
Also establish responsibility for updates, the support period and device retirement: removing data and credentials and disconnecting it from services. A connected system is maintainable when its behaviour can be described both during normal operation and when communication, storage or an update fails to finish as intended.
Sources
- IETF RFC 9019 - an IoT firmware update architecture.
- MCUboot: Bootloader design - trial boot, confirmation and version recovery.
- Amazon Builders’ Library: Making retries safe with idempotent APIs - operation identifiers and retries.
- Amazon Builders’ Library: Timeouts, retries, and backoff with jitter - retry cost and timing.
- NISTIR 8259A - baseline IoT device cybersecurity capabilities.
- Buffer size and synchronisation time are original calculations under the stated assumptions, not guaranteed product specifications or certification.