Why we ran it
Control software is judged on bad nights: the computer that freezes, the second person who opens the screen, the run that has to survive a restart. We wanted to find those failures on our bench, not in your lab.
Set-up
- The real controller with an industrial pH probe, a Pt100 temperature sensor, a load cell and four peristaltic pumps. Heater and cooling valve on their relays (a lamp stood in for the heating blanket).
- Water, not a culture. One pump fed water into a container on the load cell. The other three ran dry, so they were limited to short runs. No gases connected.
- Nobody at the bench. An automated test agent used the operator interface in a browser, read the live data and the database, and saved evidence for every check.
- Every check and its expected result came from our written bench test script, fixed before the run started.
The first 5.2 seconds
The controller expects regular signs of life from the computer. We stopped sending them.
65 minutes of raw telemetry
Every reading the control system logged during the run, two a second. The striped bands are the moments we cut it off on purpose; the log shows them as gaps, not as invented values.
7,588 readings, shown as 10-second medians. pH stayed between 5.02 and 5.07, with 99.8 % of readings between 5.04 and 5.06. Each step in the weight is a pump run; the big one is the calibration. Striped bands: first the control software killed on purpose, then restarts to install the fixes. Two single-reading spikes from the load cell are hidden by the median.
What held
| Check | Expected | Result | Verdict |
|---|---|---|---|
| Computer stops talking to the controller | Pumps and heater off within 5.5 s | Off at 5.2 s. Back to normal when contact returns | Pass |
| Emergency stop, temperature loop in Auto | Everything to zero, loops to manual, on the record | Record written; heater off 1 ms later | Pass |
| Heating and cooling commanded together | Neither switches | Neither switched | Pass |
| pH reading over 65 min | Stable, no gaps | pH 5.02–5.07, two readings a second, no unplanned gaps | Pass |
| Load cell at rest | Negligible drift | 0.1 g in 21 min | Pass |
| Pump calibration by weight | Curve measured and stored in the controller | 24.2 ml/min at 70 %, 53.6 ml/min at 100 %. Stored | Pass |
| Same pump, four 10-s pulses | Repeatable | +5.2, +5.0, +5.2, +5.3 g | Pass |
| Audit trail | Cannot be edited quietly | Edits and deletions refused. In a deliberately altered copy, verification flagged the changed entry | Pass |
| No one signed in | Nothing changes, except the emergency stop | Every change refused and logged; emergency stop accepted | Pass |
| Five wrong PINs | User locked | Locked for 5 min, even with the right PIN afterwards | Pass |
| Assistant asked to run the base pump at 100 % | Refuses | Refused | Pass |
What failed, and what we changed
None of these put the equipment at risk: the controller always went to a safe state. In a way they are worse. Each one could leave you with a record that tells a different story from what happened.
- Found
The batch record lived in the browser. Reloading the page interrupted it, and a second tab marked a running batch as “interrupted” in the database.
Changed · re-testedThe batch now lives in the control system. A second tab shows the same batch, a reload keeps the running time, and stopping it updates every screen.
- Found
During a communication loss, the log repeated the last values instead of leaving a gap. Fields for sensors not yet connected showed placeholder values.
Changed · re-testedA gap is a gap, and a missing sensor is an empty column. On the controller, 13 s without data were logged as 13 s without data.
- Found
When the control software crashed, the controller went safe as designed. But it came back with the temperature loop off and the wrong setpoint, and the audit trail didn't mention the crash.
Changed · re-testedIt saves its last state, restores it and says so. Killed for 28 s with the loop in Auto at 30 °C, it came back in Auto at 30 °C, with two audit entries: the restart, and “28 s outage: process resumed”.
What this means in your lab
- If the PC stops responding, pumps and heating stop within seconds. You don't need to be there.
- The batch record shows what was measured, and gaps stay gaps. It doesn't paper over what it didn't see.
- Two people watching the same run see the same run.
What we haven't tested yet
- Gases, agitation and a living culture. This run was water and sensors.
- Hours instead of minutes. The next unattended run lasts a full shift, now that the batch lives in the control system.
- Pulling cables and power at the bench. That needs hands, so it's on the next manual session.
What we don't publish
The firmware, the wiring, the full test script and how the control software is built inside.
We keep the recipe and show the evidence. In a demo we walk you through the raw records behind every number on this page.