In the ancient past (20 years ago), control systems used to be stand-alone systems, usually decoupled from the larger enterprise networks, using their own network cabling infrastructures (based on a variety of proprietary buses and protocols) and programming languages dedicated to control systems. But no more.

With the rise of “standard” IT technologies, interconnectivity and IT services also made it down to the plant floor. Early tests at CERN and elsewhere, however, showed that, with this rapprochement, control systems also inherited the genuine, inherent flaws, weaknesses and vulnerabilities of those IT technologies. But contrary to IT, control systems cannot be fixed and patched as quickly as IT systems. Maintenance cycles need to be respected, certifications need to be reissued, controls loops need to be revalidated.

Even so, more and more control system manufacturers integrated “IT” means into their products, despite the fact that their expertise lies in controls and not IT. Their brilliance lies in sophisticated (and technologically “beautiful”) data acquisition hardware, extremely accurate measurement devices like oscilloscopes, precise timing systems, etc. IT is not usually their turf. Similarly, and unsurprisingly, control system experts and operators centre their knowledge, skills and expertise on control systems. They are not necessarily network experts, web server managers, software programmers or database administrators. Still, both vendors/manufacturers and experts/operators embraced the benefits of IT without considering risks and mitigation measures.

The series of, so far, nine Control System Cybersecurity workshops attached and prior to the International Conference on Accelerator and Large Experimental Physics Control Systems (ICALEPCS) is supposed to raise awareness among the control system community and bring them together with cybersecurity experts in order to holistically and in-detail improve the cybersecurity of the control systems of accelerators and (large) experiments. While many accelerator labs and experiments have started to adopt standard cybersecurity protections, OT (operational technology) environments and their complex implementation and deployment requirements and timelines make it significantly more difficult to thoroughly adapt OT/control systems to achieve a more secure architecture and operation.

Nothing has changed?

Fast-forward 20 years and the control system world has not changed much in terms of complexity, timelines and security. In fact, control systems have become even more complex and interconnected, creating even more cross-dependencies and making the deployment of security measures even harder. In parallel, hardware manufacturers and vendors have now entirely embraced IT.

Recent penetration tests by the CERN Computer Security team of a plethora of control systems, embedded devices and the Internet of Things (IoT) have repeatedly brought to light basic areas of security sub-optimalisms:

• Commercial CCTV cameras stored their default passwords, secrets and private certificates in an unprotected fashion within their hardware while secure-chip technologies, i.e. so-called Secure Access Modules or SAM, already exist. In other embedded IoT devices, changing default passwords even turned out to be impossible, or access control was entirely missing such that anyone with network access was able to take full control.
• Intelligent, cloud-controlled lockers allow customers to open or close their lockers by scanning a QR code instead of using an RFID wristband or coin and key. Unfortunately, that corresponding cloud service came with a vulnerable set-up and allowed anyone on the internet to manipulate the open/close functionality of any subscribed locker worldwide. While the risk of theft is eventually controllable in this case, a similar vulnerability on defibrillators could be life-threatening. And a review of cloud-connected washing machines serving the CERN hostels is in the pipeline.
• Custom-designed PCB (Printed Circuit Board) modules for fast feedback control connected to the CERN intranet turned out to lack any level of robustness when receiving malformed or inappropriate network packages and, instead of just silently dropping those packages, completely broke down and stopped working. In some cases, restarting required the re-flashing of the local firmware.
• The authentication mechanism of a web-based SCADA (Supervisory Control and Data Acquisition) system was found to be flawed in such a way that it was possible to bypass the authentication process entirely.
• Furthermore, shared passwords are used across vast access control system infrastructures and transmitted unprotected and in clear text instead of using encryption as well as per-device credentials or certificates.
• In other cases, expert and operator passwords were accidentally exposed on public websites. Unfortunately, it turned out to be impossible to systematically change those passwords for operational reasons (“hardcoded passwords”) or to be certain where that password was actually used… Similarly, such passwords regularly end up in public Git/GitHub/GitLab repositories, their branches, forks and back-ups, and in the logs from their built pipelines.

But does that really matter? What is the real risk? Are criminals and thugs interested in breaking into control systems?  Or are security incidents linked to control systems the literal “black swans” – rarely appearing and more hyped than real? And if not, how often do those black swans appear? Let’s delve into that in the next Bulletin…

This is an abridged version of an article that first appeared in the proceedings of the ICALEPCS 2025 conference.

_______

Do you want to learn more about computer security incidents and issues at CERN? Follow our Monthly Report. For further information, questions or help, check our website or contact us at Computer.Security@cern.ch.