I once worked at an equipment company that made a dispenser which filled and discharged a chamber from the city water line for chemical mixing. It had to be very accurate. We designed a new and improved model that used optical water level sensors for 'full' and 'empty,' driven by an inexpensive chip that also operated the valves.
The unit worked perfectly at the factory, but soon after shipping we began to get angry complaints from customers that the unit was erratic. It would fail to drain, or hang up, or overfill. We dragged out the test stand and went over and over the thing, testing relentlessly. It was so simple we just couldn't see what could go wrong, and could not get anything to fail.
Finally we sent engineers into the field. After many false leads one of them noticed an extra unit in the water supply feed piping. When he asked about it, he was told it was an aspirator for premixing one chemical with the water, and that many of our customers used them. Of course, it was shut off during our tests. It didn't register at first, but after several more weeks of headscratching and devising some specific tests, the extra chemicals turned out to be the problem. The chemical shifted the index of refraction and surface tension of the water just enough that our optical sensors were getting fouled up at the mounting angles we used.
So after all that agony and gnashing of teeth, a simple change to a different type of level sensor cured the problem.
Dave B., Eastlake, Ohio,