Danger
through ransomware
In May 2017, the beverage industry realised that production lines could be vulnerable to malware. The ransomware WannaCry infected tens of thousands of computers worldwide in a very short time and was used for ransom demands. Even if production lines were operated separately from the internet or used security software, the infection often occurred, resulting in expensive production downtime. Not only outdated operating systems, but also insecure communication protocols, unscheduled connections to the internet and devices from external service technicians are technical threats to production availability. Many plant operators often face the same problem: their production lines have grown over the years, the documentation is outdated and ultimately there is now a lack of clarity about which hardware was installed in the machines purchased and which software is running on them.
Network analysis
as a first step
A network analysis serves to improve the transparency of the production lines. This is one of the first steps towards securing existing production lines technically in the long term. During the network analysis, all existing documentation is first compiled and evaluated. This is followed by an on-site analysis by security specialists. Thereby, the central network components (such as switches and routers) and, in some cases, the cabling are examined. This results in current plans that show the physical and logical network structure. In addition, the network traffic between the production lines is analysed. If necessary, this can be supplemented with active scans of unknown devices in the networks or extensive analysis of individual systems. These steps result in a detailed overview of all communication participants in the production lines, their communication relationships and physical locations.
Based on these analysis steps, the plant operator receives a report with recommendations for action that meet the specific information security requirements of production.
This was also the case for a mineral water bottler where a network analysis was carried out at the beginning of 2021. During the evaluation of the documentation, logically segmented network areas were already revealed at the production line level, which is an important prerequisite for securing them. The recommendation in this case was to route all cross-line data traffic through existing firewalls and only allow absolutely necessary data traffic. Only by filtering the data traffic a real separation of the production lines is guaranteed. It is important to incorporate the manufacturer's knowledge of machine communication into the firewall’s set of rules.
Furthermore, in the course of the analysis, several undocumented network participants were found. Among them was a control system that could be assigned to a completely undocumented machine in the subsequent investigation. This machine had been retrofitted into the production line years ago and had existed "under the radar" ever since.
Another network analysis at a soft drink bottler, on the other hand, showed a conspicuous behaviour of an inspection machine. This machine tried unsuccessfully to connect to IP addresses on the internet. The following analysis of the system showed that a malware infection had occurred, which might be related to a service technician's notebook. However, no one had noticed this because there was no permanent monitoring of the network traffic. Successful communication with the internet would probably have allowed malicious code to be reloaded and could have led to the infection of further machines and even production downtime.
Special requirements
for cyber security
When technically securing production, the special requirements of cyber security has to be considered. For example, in the classic IT sector, anti-malware solutions are often used to scan files and directories for malware. Usually, these solutions cannot be used on production systems. Production systems often have limited hardware resources, have a special operating system or the signature database of the anti-malware solution can never be updated. In this case, so-called "application whitelisting" can help, which only allows explicitly defined files to be executed on the production system.
However, security measures on individual systems quickly reach their limits because the underlying operating system often cannot be updated at all or only very rarely. In some cases, no security updates have been published for the operating systems used for years, or security updates cannot be installed because network communication with other systems or the execution of special software is impaired. This does not only affect Human Machine Interfaces (HMIs), which are often the primary focus of customers. Controls, switches and servers are also present in production and have their own limitations when it comes to implementing security solutions.
Consequently, security solutions have to be found that do justice to the long lifespan, the special hardware and software and the general differences between IT and production.
Step by step
to secure production
In order to technically secure existing production lines, measures in the production network are particularly suitable. After a network analysis, large logical network areas can be divided into smaller segments, for example a division into individual production lines, a network segment for service technicians and further segments for comprehensive manufacturing execution systems (MES). The data traffic from these segments is then routed through central firewalls and filtered so that only explicitly necessary communication is permitted. This strict separation already greatly limits the spread of malware across lines.
The building blocks offered by Syskron Security complement each other perfectly to secure existing production lines against cyber attacks. See also: www.nist.gov/cyberframework
Nevertheless, malware can continue to infect and spread within network segments. To detect this at an early stage, a central anomaly detection can be used. Due to very stable communication patterns and relationships, it can easily learn the normal state in the network communication of a production environment and then generate corresponding messages in case of deviations. This, together with the results of the network analysis, creates a high level of transparency of the current security of the production lines.
Both firewall and anomaly detection additionally require an existing organisation and processes to operate these technical solutions permanently. It has to be defined who is allowed to change firewall rules, where this is documented and what happens if this solution fails. It also has to be clear who receives the anomaly detection messages, evaluates them and then initiates countermeasures. In this regard, Syskron Security operates a Security Operations Centre (SOC) to professionally evaluate security messages and support customers in responding. Medium-sized customers in particular appreciate this service, as it allows them to react more quickly and to draw on many years of security expertise in the industrial environment.
Last but not least, secure production includes the human factor. Even with a high technical security level and lived processes, people can be deliberately manipulated to deactivate or circumvent security measures. Therefore, it is important to continuously educate all employees about dangers to production systems and, above all, to show them how to recognise these dangers and react adequately. This can be achieved with awareness trainings that, for example, address the reporting channels for security incidents or use live demonstrations to show how real threats to information security are in the production area.
In the case of external staff, it further has to be ensured that rules of information security and data protection are complied with. In addition to contractual and organisational agreements, technical security solutions can have a complementary effect. A global soft drink bottler installed special scanning stations for removable media. The company uses USB sticks to back up projects and configure controls. Before a USB stick can be used in production, it has to be scanned at one of these stations. If there is malware on the medium, it can be removed immediately without endangering production. Each scan is logged. The scanning stations address the risk of infection with malware via removable media.
Secure production ultimately also includes a well-maintained Information Security Management System (ISMS) and a functioning emergency management system. The experts at Syskron Security have a decade of experience in these areas and advise a wide range of companies. In the course of emergency exercises with adapted scenarios, some customers first became aware of the recovery issues they had not yet thought about. If a ransomware infection does occur despite all the protective measures implemented, the trained processes help to restore production quickly and systematically.
Cyber Security
for successful digitalisation
For new production lines - especially in the context of digitalisation - cyber security has to be integrated as an elementary component, virtually from the very first minute. In this way, there is a chance to achieve a high level of technical security right from the start. The President of the German Federal Office for Information Security (BSI), Arne Schönbohm, said that information security is the prerequisite for successful digitalisation. (BSI - Press - "Digitisation and information security belong together" (bund.de))
A well-known approach here is "secure by design". This approach firmly assumes that there may be harmful activities and addresses them accordingly. For example, user account rights are limited to the necessary minimum, individual systems or services are isolated from each other and attention is paid to secure software development.
The experts at Syskron Security advise every operator of production facilities to approach providers and suppliers as early as possible to address the topic of cyber security as an important component of digitalisation. The earlier this happens, the easier and more cost-efficient it is to implement information security requirements, approaches and concepts. A high level of security significantly supports successful digitalisation in production.
In workshops, Syskron Security identifies threats to production with customers and develops tailored security solutions.
Conclusion
Cyber security from the first minute
Beverage production is not spared from cybercriminals. Unplanned production stoppages, lack of contingency plans and information leakage can hit unprepared companies hard. Therefore, it is important to plan for and implement cyber security in production lines from the very first minute. In particular, the differences between production systems and traditional IT must be addressed by security experts. Segmented network areas with filtering of data traffic, a separation of production networks from other networks, regular awareness trainings of employees and defined security processes are a good mix for a solid basic security in every production.
Authors: Benjamin Süß, Kuzman Kurtev - Syskron Security
