SCADA&HMI BY CFE MEDIA

SCADA & HMI S PRING ED I TI ON Contents 3 — From DCS to OCS: A change in process control system 9 — Inductive Automation Celebrates 20 Years 10 — Enterprise SCADA Systems Deliver Data Wherever It’s Needed 21 — IT/OT convergence combines industrial and laboratory automation 29 — Capabilities every HMI/SCADA system should have 30 — Build Better HMI/ SCADA Solutions with Edge Computing 40 — Gaining actionable insight with SCADA systems 44 — VTScada and the AquaNereda Wastewater Treatment Process 45 — The Cost of Failure - Why Mission-Critical Should Be Your Mission 53 — Effective process control system migration, Part 3: Poll results, answers 57 — Integrating a DCS into an existing process cell 2 From DCS to OCS: A change in process control system  Back to TOC Compared to a traditional distributed control system (DCS), an industrial optical bus control system (OCS) has a lower investment cost of automation, shorter installation time and reduces control system maintenance costs. C ompared with a traditional distributed control system (DCS), the use of the industrial optical bus control system (OCS) can lower automation investment costs, shorten project time, and reduce control system maintenance expenses. That was the answer given by Jinsong Huang, vice president of Hangzhou Hollysys Automation Co. Ltd., while discussing Hollysys’ OCS industrial optical bus control system. The DCS, since its birth in the 1970s, has many technical advantages and has become a standard in the fields of oil refining, petroleum, chemical industry, electric power, met- allurgy and other large-scale process control industries. Because of this, the DCS has not changed much in the past 30 or 40 years. Today, with the rapid development of technologies in information technology (IT), information and communication technology (ICT) and Internet of Things (IoT) have brought many societal changes. As optical communication technology and software have been introduced into DCS, the DCS has evolved into the OCS. In an OCS, the system architecture, the controller and input/output (I/O) connections have undergone qualitative changes, resulting in changes in project cost, space between cabinets and implementation cycles, outperforming a traditional DCS. What is OCS? Why can it change a traditional DCS? What technologies does it use? What benefits can it bring to users? Which applications is it suitable for? 3 From DCS to OCS: A change in process control system What is an OCS?  Back to TOC The biggest difference between OCS and DCS is optical fiber is used in OCS as the network transmission medium between the controller and I/O, instead of the copper-core twisted pair cable usually used in a DCS. In a traditional control system, the signal transmission of the field equipment to the control system needs to go through a series of intermediate links such as field junction boxes, cable trays, terminal cabinets, safety barrier cabinets and I/O cabinets. The OCS changes the system architecture of the traditional DCS. OCS field devices are connected to the remote controller through the industrial optical bus data transmission unit (iDTU) located in the field, eliminating a series of intermediate links. OCS industrial optical bus control system is composed of engineer station, operator station, redundant historical station, equipment management station, redundant control station, redundant industrial optical bus connection unit (RJU) and industrial optical bus intelligent data transmission unit (iDTU) and other equipment in which optical fiber is used for communication from iDTU to RJU and redundant control station. The redundant control station uses triple redundant links, a pair of Gigabit Ethernet, a pair of RS-485 links, and a set of general purpose I/O connections (GPIOs). It ensures the redundancy and disturbance-free, functional safety and stable control of the control system, and the mean time between failures of redundant switching can be increased by 22.2% compared with controllers of other systems. RJU is equivalent to the role of a switch. Each RJU can connect 16 or 32 iDTUs in a star shape, and adopts passive optical device technology. The physical iDTU is used to connect field devices and communicate with the control station through an industrial optical bus. 4 From DCS to OCS: A change in process control system The iDTU adopts a modular  Back to TOC design and has redundant The Plant Floor in Your Pocket Get an overview of your process at a glance. Control your SCADA with a swipe. virtual I/O (VIO) modules, redundant optical bus interface modules and redundant power supply modules. The VIO module supports six signal types: Analog input (AI), analog output (AO), digital input (DI), digital output (DO), process interface (PI) and Namur. iDTU also supports Modbus communication protocol, which can transmit field data to and third-party systems as a Modbus slave station. What technological innovations does OCS have? Installing OCS industrial optical bus control system See the live demo now. Scan this QR code with your phone or visit demo.ia.io/engineering is not as simple as replacing the original copper cables with optical fibers. OCS 5 From DCS to OCS: A change in process control system has patents for its technological innovation for signal input/output and transmission.  Back to TOC Huang said OCS integrates industrial optical bus and software-defined I/O. Industrial optical bus is a technological innovation application of Hollysys in OCS. Although the technology of using optical fiber to transmit the Ethernet protocol has been very mature, many technical difficulties still need to be overcome to transmit the industrial bus protocol on the optical fiber. Hollysys has developed the Onet redundant industrial optical bus based on optical communications experience in the control network Cnet on DCS. Different from the 7-layer architecture of Ethernet, Onet has only three layers: physical layer, data link layer and protocol layer. This concise layer improves the transmission efficiency. Onet complies with IEC 61158 international standard, JB/T 10308.3-2001 national standard and EN50170 European standard, supports star network topology, supports 128 node devices, communication rate is 24Mbps, transmission medium is single-mode fiber and transmission distance is 20 km (without relay). Software-defined I/O is a technology that has emerged in the DCS field in recent years, and Hollysys is well-prepared in this field. Software-defined I/O is embodied in two aspects. One is to define the channel type of I/O through software. All channels of I/O are consistent in hardware, and it is not necessary to configure different I/O modules for different signals. It is only necessary to set the channel to the same signal type as the field instrument output through the software. The other is to switch the backup channel through software that is the channel redundancy mode of N+1. When the I/O channel fails, it is only necessary to perform software switching on the engineer station, and switch the faulty channel to the standby channel, without the need for maintenance personnel to arrive at the fault site. 6 From DCS to OCS: A change in process control system What are the benefits of OCS?  Back to TOC The technical innovations of the OCS industrial optical bus control system have brought fundamental changes to the traditional process control system. The OCS also brings benefits to users. First, OCS brings a substantial reduction in initial project cost. Use of optical fiber communication between the control station and the field data acquisition unit saves money compared to expensive instrument cables. The cost of intermediate links, such as cable trays, is also reduced. Industrial optical bus technology is more cost-effective than Ethernet-based fiber-optic transmission. With elimination of many intermediate links, later control system maintenance cost and the cost of spare parts decreases. Second, the project implementation cycle of OCS is less. Thanks to the simplified OCS architecture, the construction volume is reduced. With software-defined I/O technology, the signal orchestration (marshalling) link between field devices and controllers in traditional engineering design is eliminated, shortening the project execution cycle. The standard engineering design simplifies the work processes, and the modularized standard chassis design shortens the supply and delivery cycle of manufacturers. Onsite guidance and installation can be implemented upon arrival, easing efforts for users, system designers and manufacturers. Third, the system’s reliability is higher. The application of industrial optical bus is superior in improving the anti-electromagnetic interference ability and lightning strike ability in the process of on-site signal transmission. One optical fiber can transmit at least 512 on-site signals, and the transmission distance can be as long as 20 km without relays, as mentioned. The OCS can realize redundant transmission of on-site signals, with higher transmission efficiency and stronger usability. 7 From DCS to OCS: A change in process control system How does OCS ensure the security of the system?  Back to TOC System availability is very important since process control systems often are used in petrochemical, chemical and other key applications that require high continuous production. Huang said the OCS has a special design for reliability on the Onet industrial optical bus, RJU and iDTU. In the Onet optical bus communication, redundant design is used between the slave controller and the iDTU, which ensures fast switching between the main and standby controllers. Also the entire optical bus link has real-time diagnosis functionality, which ensures communications reliability. Information security of industrial control systems is facing more severe challenges. Advanced persistent threat (APT) attacks occur frequently, ransomware technologies are constantly being upgraded and historical loopholes are being exploited. Huang said the OCS has implanted the latest information security technology into the control system. Trusted computing technology has been integrated into the OCS controllers. Different from the passive defense mechanisms of information security, such as firewalls and defense-in-depth of traditional industrial control systems, trusted computing technology realizes an active immune mechanism based on endogenous security. According to Jinsong Huang, the controller adopts a high-performance main processor and a co-processor based on the national secret algorithm, and achieves three information security defense strategies: static trusted startup, dynamic trusted measurement and trusted firmware update. Stone Shi Stone Shi is executive editor-in-chief, Control Engineering China. 8 Inductive Automation Celebrates 20 Years  Back to TOC  Inductive Automation Celebrates 20 Years To celebrate Inductive Automation turning 20, we wanted to tell our community what we do, why we do it, and about our commitment to fulfilling our company’s purpose. Meet our incredible company leaders and members from every division as they share stories of our growth and dedication to solve customers’ pain points over the years. 9 Enterprise SCADA Systems Deliver Data Wherever It’s Needed I f you look behind the scenes of almost any industrial organization, you’ll find a SCADA system at work. SCA- DA systems carry out essential day-to-day functions like controlling industrial processes; notifying operators about problems that could cause downtime; monitoring, gathering, and processing data; interacting directly with devices through HMI software; and recording events. Some SCADA systems are small and simple, but in today’s world larger and more complex SCADA installations are becoming more common and necessary. Whereas your standard SCADA system is used on the plant floor at one site, an enterprise SCADA system serves organizations with multiple sites, and provides one place where users can view all of the operational information across their various sites. When adding or upgrading an enterprise SCADA system, there are, of course, a number of factors for the organization to consider. Can it be deployed effectively at multiple sites, to the cloud, and in some combination thereof? Can it effectively manage data and projects from one central location? Can it grow and scale nimbly enough to keep the David Dudley Co-Director of Marketing Inductive Automation organization future-proof? Can it easily set up a remote connection to anything inside the enterprise? Can it add capabilities like reporting, alarming, and enterprise ad- Enterprise SCADA Systems Deliver Data Wherever It’s Needed  Back to TOC ministration to manage a fleet of installations? Can it increase the number of tags and connections without going over budget? Read on to see three case studies of real companies, each from a different industry, and learn how they found answers to these questions and went on to achieve success with ambitious enterprise SCADA projects. Energy Case Study: SB Energy SoftBank (SB) Energy is a leading utility-scale solar, energy storage, and technology platform. The company develops, constructs, owns and operates some of the largest and most technically advanced renewable projects across the United States. 11 Enterprise SCADA Systems Deliver Data Wherever It’s Needed Recently, SB Energy acquired a portfolio of photovoltaic (PV) solar plants across North  Back to TOC America and wanted a way to monitor and control the plants from its remote operations center in Redwood City, California. So SB Energy engaged system integration firm Vertech to provide a world-class, single-platform enterprise SCADA solution that could oversee the six solar plants and perform real-time data acquisition, data validation, data consolidation, site monitoring, and performance analysis. SB Energy requested that Inductive Automation’s Ignition platform be used for the project, at both the enterprise and site levels. “There were two major problems we were confident that Ignition would help us solve,” says Andy Singh, Director of Software Architecture at SB Energy. “One is, we knew we had a lot of data coming in from our power plants in the field, and so the performance of the software was critical — and Ignition shined in that regard. The other aspect was creating a modern UI; we didn’t want software that looked dated and up in the ‘90s, and Ignition had that capability to make a customized, beautiful UI that the end users love.” Scalability and visibility were also big considerations for the project. As Chris McLaughlin, SCADA & MES Specialist at Vertech, says, “With a single site, you can log in to your site’s SCADA and be able to see the content that you need. By the time you get up to two or three sites, it becomes more complex and it’s harder to figure out what’s happening across a few. By the time that you get to an entire fleet of sites, it’s a real problem. And so with Ignition, what it’s doing for you is that you can immediately log in and be able to see all of your lowest performing inverters across all of your sites or a real power trend for all of your sites in just one dashboard. Ignition is solving the problems of being able to see your content across an entire fleet.” 12 Enterprise SCADA Systems Deliver Data Wherever It’s Needed  Back to TOC Using Ignition, Vertech’s skilled team created a single-platform, multi-site management tool that allows SB Energy’s greenfield operations center full access to view and optimize performance at each remote site. The Ignition Perspective Module was the primary visualization tool for the application. Chad Waddoups, Senior Control Integrator at Vertech, says, “Ignition is a good fit for this project because of the ability to connect to multiple data sources, an unlimited tag count, and Perspective’s web-based SCADA is ideal for SB Energy’s business environment.” 13 Enterprise SCADA Systems Deliver Data Wherever It’s Needed The new solution was deployed in October 2021. Since then it has provided SB Energy  Back to TOC with complete visibility of all its sites, assets, and energy production performance. Now they can more easily maximize energy production and resolve maintenance issues with less time and resources. The dynamic, highly scalable new system also gives SB Energy a strong base for the future. Mining Case Study: Dolese Bros. Dolese Bros. is a large provider of aggregate material and cement throughout Oklahoma. Over the past several years, Dolese has upgraded many of its quarries with advanced automation, networking, and operational systems. A key component of this strategy is the deployment of the Ignition platform at each quarry. As the Ignition projects were rolled out for each quarry, Dolese wanted to take advantage of the Ignition Perspective Module’s mobile-responsive capabilities to allow plant operators to view the status of quarry operations anywhere. The initial goal of the project was to focus on downtime, OEE, and real-time production reporting. As more information became available, and Dolese began to see the power of providing information to everyone, they requested the ability to view even more. The plant’s legacy SCADA solutions allowed only a limited number of viewers to see what was going on with the equipment. To accomplish their goals, Dolese enlisted the help of an integration company named Industrial Networking Solutions (INS). Dolese asked INS to provide them with the capability for anyone in the organization to monitor processes from anywhere, and that everything be mobile-responsive. INS leveraged the Ignition Perspective Module to create a fully web-based, dynamic system with fast response time and with enough flexibility to include mobile devices. 14 Enterprise SCADA Systems Deliver Data Wherever It’s Needed The plants were using a spreadsheet to gather production data, so INS built a dy-  Back to TOC namic reusable page that allows the user to enter data that could not be automatically collected, such as item codes and downtime data. With this page, operators can view the data gathered from scales during shifts. INS built the page so that it could be reused from site to site. All that is needed is to populate a database table with the information about the various entities within the system, such as work centers, conveyors, or environmental parameters. Once the data table is populated, the page essentially builds itself with the use of query bindings that populate the table, and accordion and flex repeater components. Once the production run is started, the data is sent via MQTT to the cloud where it is integrated into the ERP system in real time. At the enterprise level, other users can view all this data over time using the downtime report. This report allows the user to select a work center or entity from the chart, and further drill into things such as a particular production run or a piece of equipment to view downtime data for that selection. Dolese initially began their Ignition journey wanting to capture downtime events and replace reporting spreadsheets with real-time reporting. Since then, multiple areas of integration were identified, transforming the original proof of concept into a holistic enterprise solution. This includes real-time inventory and production data, downtime tracking and reporting, and plant monitoring in real time for mobile interfaces. As Dolese scales their operations, they plan on using Ignition to integrate plant-to-plant transfer systems with RFID, real-time integrations to their data warehouse to help facilitate ticketing and sales data more efficiently, and begin migrating legacy control systems into Perspective HMIs. 15 Enterprise SCADA Systems Deliver Data Wherever It’s Needed  Back to TOC Oil & Gas Case Study: CHS Pipelines and Terminals CHS Pipelines and Terminals is a leading global agribusiness owned by farmers, ranchers, and cooperatives across the US. CHS used a legacy SCADA system for monitoring and controlling its oil pipeline. They wanted to move toward a modern solution that not only enabled the safe delivery of hydrocarbons that met industry best practices, such as high-performance and compliant displays, but could also enhance the operational capabilities of the controller and enhance the visibility of the system to enterprise and remote users. Additionally, they wanted a scalable solution that would deliver sustainable outcomes. 16 Enterprise SCADA Systems Deliver Data Wherever It’s Needed The integration firm Streamline Control worked with CHS to replace their legacy SCA-  Back to TOC DA with a new system built around open standards for communication (MQTT/SPVB) and utilizing Ignition at all levels of the Purdue Model. Streamline successfully executed all project activities including designing the SCADA architecture, developing high-performance HMI displays in the Ignition Vision Module, developing a robust and comprehensive tagging/asset structure for each asset location, building Perspective displays for enterprise users, integrating Ignition with leak detection systems, and developing applications in Ignition used for batch tracking, pig tracking and compliance. A key feature of this project was aligning the system on standards, including protocols, tag structures, naming conventions, security, and extensive templating using UDTs. There was also alignment on compliance-related measures including display characteristics, navigation, colors, and alarming. The solution is a single stack solution that provides robust and secure operation for the pipeline controllers and utilizes Ignition at the enterprise to expose operational data and to provide rich and meaningful displays to all users. It went live and into production in the fall of 2021. From the field to the boardroom, Ignition was used at all layers of the business. At the field or remote location, Ignition Edge is used to gather and contextualize operational data points using Sparkplug B, and interface with PLCs and Flow Computers. At the controller, control room or SCADA level, Ignition was used to visualize and integrate compliance and CRM-related activities into a unified operator experience following best practices around API 1175 and the High-Performance HMI. 17 Enterprise SCADA Systems Deliver Data Wherever It’s Needed MQTT/Middleware and Cirrus Link Modules were used to gather data not only for  Back to TOC pipeline controllers but also to provide a secure mechanism to expose all operational data points to the enterprise. At the enterprise level, an enterprise Ignition was built to gather all operational data using MQTT and Sparkplug B into contextualized, templated HTML5-based displays that are fit for desktop or mobile devices and for casual users (as opposed to exposing HMI displays that are fit for controllers). Using MQTT with Ignition modernized edge connectivity by providing a reliable and secure means of communication that also improved efficiency. Modifying devices and tags is very straightforward in the new system. Additionally, Ignition allows CHS to support multiple user groups to provide essential data to everyone who needs it, including people outside the pipeline control room, and everyone is able to have that same consistent and current data. The new solution provided CHS with new and improved capabilities: A secondary control center which was independent of the primary control center, integrated compliance-related activities and processes at the SCADA level, built-in reporting, and the ability through MQTT to make information meaningful so that the enterprise applications can begin utilizing the data right away. The Ignition Enterprise instance – which CHS calls the Mobile Business Interface, or MBI – leveraged all the hard work they put in standardizing the data structures at the edge and the MQTT infrastructure. This allowed for templated Perspective displays built for casual users, rather than for a control room. Data is now accessible, without 18 Enterprise SCADA Systems Deliver Data Wherever It’s Needed users reaching into SCADA for information, and more secure. Information is displayed  Back to TOC to users on the enterprise via a web browser or phone because of HTML5/CSS3. Jeremiah Henley, CTO of Streamline Control Solutions, says, “The Ignition SCADA system was a good choice for this project. The modern technology, along with proven security, configuration options, and scalability features, all align very well with our goals for this new system. Ignition allows us to support multiple user groups to provide that essential data to everyone that needs it and all with a secure segmented and consistent model. This includes people outside of the pipeline control room such as field technicians, business users, and managers. And everyone’s able to have that same consistent and current data.” 19 Enterprise SCADA Systems Deliver Data Wherever It’s Needed Ignition: The Unlimited Platform for Enterprise SCADA  Back to TOC As the foregoing case studies attest, there are a variety of reasons why innovative companies like these have selected Ignition as their enterprise SCADA solution. Ignition breaks down internal silos by bringing all of the enterprise’s devices and data together in one place. It allows SCADA, alarming, reporting, and more, to live on one platform, and it’s built on trusted IT standards including SQL, Python, MQTT and OPC UA. Ignition is easy to scale from a single client installation to an enterprise-wide system, and it lets companies expand access to an unlimited number of users. Its unlimited licensing model lets companies launch as many web-launched clients as needed for the price of one Ignition server license. Ignition comes with a powerful, unlimited, zero-install integrated design environment, and enables multiple people to work on the same project simultaneously. And when you need to roll out application changes, the updates are pushed down instantly from the server without requiring a restart, which reduces downtime. These are just some of the reasons why organizations are adopting Ignition for the enterprise. To learn more, visit https://inductiveautomation.com/solutions/enterprise. David Dudley David Dudley is co-director of marketing at Inductive Automation. 20 IT/OT convergence combines industrial and laboratory automation  Back to TOC Teamwork enabled the creation of a large testing lab in just nine months T he COVID-19 pandemic inspired creativity and improvisation in many areas of industry. It also encouraged stronger collaborations, one of which created a high-capacity testing lab in an extremely short period of time. System integrator DMC helped Ginkgo Bioworks quickly create a sample-processing lab for diagnostic testing for COVID-19. The lab went from concept to bio-validation in nine months. Once the project was completed, the lab had the capacity to process 100,000 sam- ples daily. The new Ginkgo Bioworks lab, located in Boston, has more than 40 automated work cells, including robots, programmable logic controllers (PLCs), lab equipment and vision inspection systems (see Figure 1). Each work cell can be set up for new tasks, allowing Ginkgo to use it for other large-scale work in the future. The lab is automated, with samples making their way through conveyors and work cells with no human intervention. DMC has offices in several cities in the U.S., including New York, Boston, Chicago, Houston, Dallas, Denver, St. Louis and Seattle. The engineering consulting firm focuses on software development and control systems in a variety of industries, including pharmaceutical, automotive, oil and gas, and food and beverage. Ginkgo Bioworks is based in Boston and is working to make biology easier to engineer on several fronts. This project leveraged the strengths of both companies. It was the first time Ginkgo had done automation at this scale, so it asked DMC to help. 21 IT/OT convergence combines industrial and laboratory automation  Back to TOC “In a large, automated facility like this one, we’re blending a couple of different forms of automation,” said Henri Girard, automation engineer with Ginkgo Bioworks (see Figure 2). “We’re trying to blend industrial automation, where you see a lot of robots Figure 1: The new Ginkgo Bioworks lab has more than 40 automated work cells, including robots, programmable logic controllers (PLCs), lab equipment and vision inspection systems. Courtesy: Inductive Automation and PLCs, and lab automation from the world of biology. And, to make these two communicate together, we needed a smart SCADA platform that can bridge the gap between these two worlds.” The supervisory control and data acquisition (SCADA) platform was crucial not only for that reason, but also because the requirements were constantly shifting as more was 22 IT/OT convergence combines industrial and laboratory automation learned about COVID-19 in the early months of the pandemic. Given the demands  Back to TOC on performance and the flexibility needed to keep up with the constantly changing requirements, DMC believed Ignition by Inductive Automation was a great fit for this project. It was also used for the human-machine interface (HMI) (see Figure 3). Up to the task “The SCADA platform really shined in processing the massive volume of data and commands and tag changes that happen and flow through a system like this,” said Paul Mangels of DMC. “It was really good at managing that high volume of data that something this complex requires.” The SCADA system also provided real-time dashboards for production data, reliable storage of event data and traceability data on all samples. Events occurred multiple times per second throughout the lab across the PLCs, and the SCADA system captured all the data. The lab instruments used a variety of different communication protocols, but the SCADA system tied things together with a consistent interface for users. “We went from initial concept to final bio-validation of the laboratory in about nine months,” said Jay LaFave, project engineer for DMC. “We couldn’t have done that without all the capabilities of the Ignition software.” “The pace of this project, I would say, is pretty unprecedented in the automation world,” Mangels said. “We had to move quickly because of COVID — and the knowledge we had was constantly changing, because the virus was changing and our science’s understanding of it was changing. The biological process we were try- 23 IT/OT convergence combines industrial and laboratory automation  Back to TOC ing to implement was evolving over the course of the project, so we needed a platform that would be flexible with us to stay up to date with those changes.” Figure 2: Ginkgo Bioworks is blending different forms of industrial automation such as robots, PLCs and lab automation from the biology world. Courtesy: Inductive Automation The aggressive schedule required teamwork among the two companies, with strong convergence of SCADA and laboratory technologies. Numerous activities had to occur simultaneously. “The time scale was extremely accelerated by trying to do things in parallel,” Girard said. “We were trying to do some of the design, some of the software work and some of the hardware installation, all in parallel, as the process itself was also changing.” 24 IT/OT convergence combines industrial and laboratory automation Concurrent development  Back to TOC “When we were creating this project, time was of Run your business-critical applications with no downtime – where you need them the essence and so, we needed to grow the number of engineers we had working on the SCADA Continually Available Edge Computing project,” LaFave said. “The Across industries and businesses, operations and IT leaders want to harness Industry 4.0 opportunities to gain new insight, achieve operational excellence, and operate more efficiently and safely. Edge Computing enables these advancements while solving inherent challenges of bandwidth, latency, and security at the edge. Stratus zero-touch and fault tolerant platforms provide teams with secure and highly-automated Edge Computing platforms, purpose-built for edge environments. With built-in virtualization, OT teams are able to run multiple applications concurrently, backed by self-protecting and self-monitoring features that eliminate unplanned downtime and ensure continuous availability for their operations. software’s robust support for concurrent development allowed us to have 10 different developers working on the project at the same time — all while production and testing was ongoing. The ability to do that in a seamless way was Up to 5,000 I/Os 100,000+ I/Os Up to 4 virtual machines 30+ virtual machines High availability or fault tolerant Intelligent uptime layer critical to us achieving the timeline on this project.” Fault tolerant UL Class I Div 2 certified DMC also chose the software because of its flexi- Simple | Protected | Autonomous www.stratus.com bility since many requirements weren’t fully known at the start of the project. 25 IT/OT convergence combines industrial and laboratory automation  Back to TOC The SCADA system evolved along with everything else. “There are a lot of different components to Figure 3: Henri Girard is shown with HMI. Courtesy: Inductive Automation the system,” Mangels said. “You have PLCs and device drivers talking to the lab equipment, various APIs from Ginkgo relating to inventory and moving parts through, and the SCADA system was right in the middle, helping to broker all that data and connect all the different parts of the lab together.” Traceability was also an important requirement. “The process events are an auditable record of what happened to each sample as it moved through the lab,” Mangels 26 IT/OT convergence combines industrial and laboratory automation said. “So, we used the SCADA system’s store-and-forward capabilities and transac-  Back to TOC tion groups to quickly capture queues of events and log them in a structured query language (SQL) database.” Capturing the huge volume of data was a challenge. “Each sample, as it went throughout the process, would produce upwards of 50 different unique data records,” LaFave said. “So, that equates to nearly half a million data rows per day, and the SCADA platform enabled us to capture all of that data in an effective and organized manner.” Ready for the future Both companies were pleased with the result. While a lot of it was uncharted territory, the collaboration was nonetheless successful, and it leaves Ginkgo with flexibility for future needs. “A lot of biology relies on small-scale liquid handling between different plates, between different reagents and these operations can be very time-consuming to do at the bench for scientists,” Girard said. “On the other hand, once we can leverage the power of automation to multiply the throughput of all these operations by orders of magnitude, we can have a lot more shots at the goal, we can handle a lot more projects at the same time, and we can learn a lot more about the underlying biology as well.” In addition to learning a lot along the way, engineers were also energized by the challenge. “This is probably the coolest project I’ve worked on in my time at DMC,” Mangels said. “We had a huge team of people. We worked with a whole bunch of 27 IT/OT convergence combines industrial and laboratory automation organizations. The Ginkgo team was fantastic. Just the number of bright people that  Back to TOC came into this project in different ways was amazing, and I definitely appreciated being a part of something and feeling like I was doing something to make a difference.” Ginkgo Bioworks also appreciated the team effort. “DMC’s been great to work with,” Girard said. “They have really taken on all the challenges that we’ve given them during this project, despite the super-accelerated timeline and the constantly changing process. We were all learning as we were building the facility, really flying the plane while we were building it.” Jim Meyers Jim Meyers is communications manager at Inductive Automation, creator of the Ignition industrial application platform for SCADA, HMI, IIoT and more. 28 Capabilities every HMI/SCADA system should have  Back to TOC  Capabilities every HMI/SCADA system should have The Edge is found where the action is, at, or near the industrial process. The introduction of the Edge is the most significant plantfloor advancement in industrial computing since the introduction of SCADA, 30 years ago. Edge Computing provides enormous value for HMI/SCADA systems through workload consolidation and operational resilience. Consolidating multiple physical assets into a single, more advanced Edge platform, allows companies to deploy, operate, and maintain a smaller number of devices. 29 Build Better HMI/SCADA Solutions with Edge Computing Running the finest PLCs and software on a conventional platform is no longer the best way to protect monitoring and control operations. Introduction Traditional human-machine interface (HMI) and supervisory control and data acquisition (SCADA) architectures for mission-critical automation control have served industry well, but there is always room for improvement. With digitalization comes an edge-computing-based approach that redefines what “good” looks like in a monitoring and control system architecture. Modern Edge Computing platforms solve many common challenges faced by engineers when deploying or operating HMI and SCADA in their automation and control systems. This paper describes what an improved and fortified industrial control system architecture looks like; the inherent advantages of Edge Computing for resolving persistent challenges; and proof points from an oil & gas industrial organization, materials manufacturer, and specialty chemical company whose control systems are benefitting from the transition to a single edge platform. You’ll also see how three different organizations implemented an Edge Computing control and automation ar- Build Better HMI/ SCADA Solutions with Edge Computing chitecture along with a modern distributed control system (DCS) and historian software  Back to TOC for process optimization. Visualizing the opportunity Edge Computing is a distributed computing framework that collects, processes, and stores crucial information close to the edge — where it is produced or consumed — rather than in a centralized server or data center. But what does that mean for an industrial control system infrastructure? Figure 1 reflects how Gartner’s representation of an Edge Computing topology is essentially the edge version of the Purdue Model for Industrial Control Systems. Figure 2 combines the two models to illustrate a complete edge portfolio, from the device gateway to compute edge to the enterprise. Figure 1. Gartner’s Edge Computing topology is essentially the edge version of the Purdue Model for Industrial Control Systems Source: Gartner. 31 Build Better HMI/ SCADA Solutions with Edge Computing  Back to TOC Adopting Edge Computing within your enterprise benefits automation control, specifically HMI and SCADA design, by improving your ability to protect mission-critical operations. A single edge platform can Figure 2. Combining the Gartner Edge Topology with the Purdue Model produces this diagram of an entire edge portfolio from device gateway and compute edge all the way up to the enterprise. It shows how it is possible to use Edge Computing platforms at various levels of the Gartner Edge Topology. bolster operational excellence to deliver extensive engineering, operation, and maintenance efficiencies as well as critical reliability, safety, and security improvements. Solving core challenges One of the most pressing challenges with typical automation and control architectures is turning the sheer volume of data generated by today’s industrial automation systems 32 Build Better HMI/ SCADA Solutions with Edge Computing into actionable information. Other significant challenges include modernizing legacy  Back to TOC infrastructure, streamlining operations by reducing disparate islands of automation, and eliminating unplanned downtime. The Edge Computing approach tackles these challenges by design. Edge Computing modernizes the infrastructure and streamlines operations by enabling the consolidation of multiple software solutions running on multiple individual computers into a single edge platform. A typical installation has HMI/SCADA software and advanced applications such as historians, manufacturing execution systems (MESs), batch, asset performance, engineering, and programming software installed in multiple computers. Edge Computing platforms offering built-in virtualization avoid this complexity and cost. In addition to embracing digitalization, decreasing islands of automation, and facilitating workload consolidation, edge platforms offer additional performance benefits such as mitigating delays from bandwidth and latency issues and simplifying maintenance complexities common with an expanding pool of hardware and software assets. Edge platforms that support remote and autonomous operations enable OT users to manage systems, removing the need to rely on IT resources. As more critical applications reside in edge platforms, built-in redundancy to protect against unplanned downtime and enable operational resilience increases in importance. Downsizing to a single redundant device where all the valuable automation and control software are installed actually increases reliability and simplifies security management, thus minimizing the risk of costly operational disruptions. Built-in redundan- 33 Build Better HMI/ SCADA Solutions with Edge Computing cy and preconfigured virtualization are especially important when the SCADA or HMI is  Back to TOC installed in a remote or hazardous environment. Delivering additional capabilities Some modern Edge Computing platforms are industrial grade, such as those provided by Stratus. Ruggedized, Class I Division 2 (CID2) certified platforms, such as Stratus’ ztC™® Edge Computing platform (see Figure 3), can be installed in hazardous locations, together with the programmable logic controllers (PLCs), in the same control panel without special accommodations for temperature, humidity, or vibration protection. Many Edge Computing devices have builtin protection from cyberattacks, such as: • Host-based firewalls for blacklisting or whitelisting IP addresses or domain names • Restricted USB ports to help prevent the spread Figure 3. Some modern Edge Computing platforms are industrial grade, such as the ruggedized, Class I Division 2 (CID2) certified Stratus ztC Edge Computing platform, which can be installed in hazardous locations, with the PLCs, in the same control panel without special accommodations for temperature, humidity or vibration protection. of malware • Role-based access controls to authorize specific users and groups • Secure communications protocols and Trusted Boot to thwart cyberattacks 34 Build Better HMI/ SCADA Solutions with Edge Computing  Back to TOC Scalability, extensibility, and standardization also are associated with advanced edge platforms, such as the smaller-capacity Stratus ztC Edge and large-capacity Stratus ftServer® platforms (see Figure 4). The best Edge Computing platforms scale well as new nodes and locations are added. They are extendable to accommodate new operations and control capabilities without a significant investment, and flexible to extend monitoring and control of the plant to mobile devices. Figure 4. Advanced edge platforms, such as the Stratus ftServer platform, provide scalability, extensibility and standardization. The best Edge Computing platforms scale well as new nodes and locations are added. Some of the most pressing challenges with typical automation and control architectures include demand for digitalization, disparate islands of automation, and downtime. The Edge Computing approach tackles these challenges by design. They also support standardization of all control into a single architecture and can meet non-redundant, high availability, or fault-tolerant needs. And the best edge platform can be easily installed, operated, and maintained by non-IT personnel. Optimization in action The following three companies, each with unique needs, found their solution in an Edge Computing control and automation architecture. See how they optimized their operations. 35 Build Better HMI/ SCADA Solutions with Edge Computing  Back to TOC 1. Streamline Innovations A Texas-based solution provider that focuses on eliminating hazardous emissions through technology, selected Stratus to bring AI-powered predictive maintenance, remote operation, and 99.5% uptime to remote oil field equipment. This included: • Stratus ztC Edge to develop, host, and protect proprietary hydrogen sulfide gas treating process • Inductive Automation Ignition HMI/SCADA Software • PostgreSQL Database and Python • Seeq AI Software Streamline Innovations created solutions for their customers that enabled real-time analysis, HMI, and tracking of 50 KPIs from 350 miles away. The company also reduced onsite staffing requirements by 66% and surpassed 95% customer SLA by achieving 99.5% uptime. Stratus, we have built smart plants that are “Using managed remotely by Artificial Intelligence and visible from any location 24/7. ” Dr. Peter Photos Chief Technology Officer, Streamline Innovations 36 Build Better HMI/ SCADA Solutions with Edge Computing Read the full case study to learn how by using Stratus Edge Computing, the company  Back to TOC brought intelligence and remote operations to complex process skids, enabled continuous operation, and surpassed customer expectations. 2. Rubberlite A West Virginia-based materials manufacturer chose Stratus Edge Computing platforms to simplify and consolidate IT infrastructure, standardize data collection, and ensure continuous availability of applications and reliability of data. This included: • Stratus ftServer to run application virtualization and fault tolerance eliminating downtime, centralizing data, and improving reliability • AVEVA™ System Platform DCS and data historian By improving data reliability and automation, Rubberlite drove significant improvement in efficiency and product quality, translating to business growth and customer satisfaction. The team reduced non-sellable material by 80% while increasing sales and production by 25%. Most dramatically, Rubberlite has experienced zero downtime since implementing a Stratus solution in 2016. partnership with Stratus has helped Rubberlite “Our evolve to where we are now, and it’s put us in the position to continue to grow and improve. ” Former Process Automation Engineer Rubberlite 37 Build Better HMI/ SCADA Solutions with Edge Computing Read the full case study to learn how Stratus Edge Computing enabled Rubberlite to  Back to TOC eliminate downtime, lower IT costs by 50%, and reduce non-sellable material by 80%. 3. Synthomer A London-based specialty chemical company embraced a fault-tolerant Edge Computing platform to centralize and optimize operations at a production plant in Italy and protect against failures and downtime. Standardizing the entire facility on the same system eliminated islands of automation and improved operational safety, resilience, and analytics. This included: • Stratus ftServer to protect the modernized operation and house all the software used to engineer, program, operate, and maintain the DCS • Rockwell Automation PlantPAx DCS • Rockwell Automation Allen-Bradley ControlLogix PACs • System integration by Progecta Upgrading the plant to run on a single Stratus edge platform and centralized PlantPAx DCS, in addition to replacing the plant’s previous disparate, multi-branded PLCs with Allen-Bradley ControlLogix PACs resulted in a 30% increase in production and millions of Euros of additional revenue. Throughout the three-year digital transformation project, production never stopped. Read the Synthomer case study to see how the company implemented a fault-tolerant Edge Computing platform to centralize and optimize operations. 38 Build Better HMI/ SCADA Solutions with Edge Computing Summary  Back to TOC The breadth and depth of proven Edge Computing advantages underscore the reasons it is time to turn the page on traditional automation control architectures. For mission-critical operations, leveraging benefits such as work consolidation, operational resilience, downtime protection, and standardization ensures a more powerful, efficient, and secure HMI/SCADA solution. About Stratus Technologies For leaders digitally transforming their operations in order to drive predictable, peak performance with minimal risk, Stratus ensures the continuous availability of business-critical applications by delivering zero-touch Edge Computing platforms that are simple to deploy and maintain, protected from interruptions and threats, and autonomous. For 40 years, we have provided reliable and redundant zero-touch computing, enabling global Fortune 500 companies and small-to-medium sized businesses to securely and remotely turn data into actionable intelligence at the Edge, cloud and data center – driving uptime and efficiency. For more information, please visit www.stratus.com or follow Stratus on Twitter @StratusAlwaysOn and LinkedIn @StratusTechnologies. 39 Gaining actionable insight with SCADA systems  Back to TOC Modern supervisory control and data acquisition (SCADA) solutions can offer actionable insight to allow for better monitoring and control of production processes. T o succeed in the food processing sector, requires a balancing act of nuances between perishable ingredients, supply chains, production, quality control, and effective workflows along with proper reporting to ensure compliance with both government and customer requirements. To ensure efficient and profitable operations, obtaining actionable insight into production is now imperative. One way to achieve this is by using supervisory control and data acquisition (SCADA) software in conjunction with human-machine interface (HMI) technology. SCADA systems collect and analyze production data and deliver it to the accompanying HMIs for intuitive visualization of real-time performance. Together these technologies allow companies to better monitor and control production. Likewise, it can provide diagnostic and alarm capabilities which result in reduced downtime and waste. With real-time performance data, companies can also adopt predictive and preventive maintenance capabilities which allows operations teams to make data-driven decisions about where and when maintenance should take place. It also offers improved traceability through the real-time tracking of production data. And, due to the nature 40 Gaining actionable insight with SCADA systems  Back to TOC of the food industry, it is essential to have operational flexibility and agility to ensure efficiency and scalability. This is especially important to help companies stay on top of the frequent changes that are common in this industry sector. A good example of a food manufacturing company Courtesy: CFE Media and Technology that has improved production through the use of real-time data insight driven by SCADA and HMI technology is Catania Oils. Using SCADA and data to improve operations Catania Oils is a manufacturer of non-GMO Project verified and organic oils for the ingredients, food service, and retail markets. The company needed help to utilize existing data to allow it to gain informational insight for plant-wide visualization into production. In particular, the company wanted to know the status of its lines – which lines are up and which lines are down. It also wanted to better understand the quantity of resources being used and specifically to know the tank levels to be able to order raw materials based on the availability of storage space. Other requirements were to be able to identify potential efficiency gains in its manufacturing processes by investigating overall equipment effectiveness 41 Gaining actionable insight with SCADA systems (OEE) and measuring key perfor-  Back to TOC mance indicators (KPIs); to have the capability of batch management and material handling; to track the quantity of additives for regulatory purposes; and to monitor the quantity of products produced. To achieve all of these goals, the company set out to find a solution that could capture data and generate meaningful displays to allow it to make data-driven decisions. Catania Oils went with a system integrator that chose to employ an web-based real-time HMI/SCADA suite along with a plant data historian. By using this technology, Catania Oils saw immediate benefits. The company is now able to better monitor and control production, including OEE and loss deployment, to improve produc- 42 Gaining actionable insight with SCADA systems tion efficiency. The key to improving operations was to focus the right resources on  Back to TOC the actual problems and to use OEE for gaining insight. As a metric, OEE provided a clear understanding of the difference between the quantity of sellable product that could be made versus the actual product that was made. The company also made efficiency gains by monitoring loss deployment. The use of SCADA software has enabled Catania Oils to determine how much product has been made, the loss of efficiency attributed to the fault of a machine or asset, and the loss of efficiency attributed to the process around the asset. By monitoring both OEE and loss deployment, it has gained improved awareness into potential losses of efficiency. Using the production data collected by the SCADA system and displayed on the HMI system, the company has been able to focus on the entire process, as opposed to a single machine fault or quality defect, to capitalize on greater efficiency gains. Control Engineering Europe 43 VTScada and the AquaNereda Wastewater Treatment Process  Back to TOC  VTScada and the AquaNereda Wastewater Treatment Process at Riviera Utilities Engineer Lee Kibler describes the principals and benefits of the AquaNereda Wastewater Treatment Process and how the utility implemented it using the VTScada automation platform. This case study originally aired as part of The Automation Village in October 2022. 44 The Cost of Failure - Why Mission-Critical Should Be Your Mission Explore the conditions that allow gaps to emerge in the most hardened process control systems W ith many things in life, we are on the brink of failure and not even know it. This is never truer than in mission-critical control systems like SCADA (Supervisory Control and Data Acquisition). If you are reading this, you probably think a lot about how to keep your system online. Rather than deep diving into technical best practices, let us look at how blind spots develop even when smart people are actively looking for them. “I’m your friendly neighborhood sinkhole detector,” said no one, ever. The Cost of Failure - Why Mission-Critical Should Be Your Mission Sinkholes are excellent examples of how potentially disastrous gaps emerge unno-  Back to TOC ticed. They seemingly appear without warning yet develop over long periods and leave plenty of clues. In 2016, a sinkhole erased a major intersection in Fukuoka, Japan. Remarkably, no one was injured. Staring into the massive hole, one could clearly see a variety of infrastructure that would have possibly shown signs that something was amiss. In the lead up to the collapse, water and gas utilities may have experienced losses of pressure. Telecoms may have noticed intermittent signal losses. The city may have filled more pavement cracks than usual. The problem was that there was no one who’s job it was to unify all this information. There was no friendly neighborhood sinkhole detector. The opposite of a sinkhole is… an iPhone? When Steve Jobs appeared on stage at the Macworld Conference & Expo in 2007, the audience was already expecting to see the world’s first iPhone. Most assumed it would simply be a flip phone grafted onto an iPod music player. Instead, what Jobs pulled from his pocket was a single unified platform that brought together Internet, email, accelerometer, microphone, camera, and a full app marketplace, all with centralized quality control of hardware and software. All features can be counted on to work together over time, with little involvement by the end user. Why Systems Fail - Architecture Single Points of Failure – This is one of the most common reasons systems go down. This may be a single hard drive, server, PLC, office location, or network. Trace the path from I/O, to the PLC, to the HMI, out to thin clients and alarm notifications. Identify individual components that can take down your whole system. 46 The Cost of Failure - Why Mission-Critical Should Be Your Mission Limited Levels of Redundancy – SCADA specifications typically require server  Back to TOC failover. The problem is that not all redundancy is equal. Most platforms only support two redundant servers. Worse, most use third-party Historians which require their own methodology for failover and synchronisation. Virtual Redundancy – Virtualized servers are an important tool for IT departments to manage systems. Developers can create multiple server instances, each with its own OS, running on a single physical computer. The obvious problem is that the physical computer is a single point of failure. Complex virtualized designs can also make it harder to spot points of failure. Make sure there is at least one backup physical server. Why Systems Fail - Cyber Attacks Distributed Denial of Service (DDoS) – For this common strategy, the attacker floods the target’s network with meaningless requests. One solution is to employ Virtual Private Networks (VPNs) between servers and remote I/O devices and to avoid using public IP addresses. You can also configure your firewall to reject excessive requests and accept requests from whitelisted computers at specific times. 47 The Cost of Failure - Why Mission-Critical Should Be Your Mission Ransomware – By tricking users into opening email links or inserting infected USB  Back to TOC drives (beware that nice camera you found in the parking lot), attackers encrypt a company’s data and sell them the decryption key. Avoiding ransomware requires training and vigilance. There is always a “first time” for new exploits. Assume Bad Guys are Already In – You can then begin to develop ways to limit the damage intruders can do once they are past your security. Recently, a company that provides IT solutions to American businesses and governmental organizations discovered that their software had been compromised. Hackers were able to access the networks of over 18,000 customers for weeks before being discovered. Why Systems Fail - Underestimated Cost of Data Recovery What happens if something does fail? In addition to the loss of real-time monitoring and control, what is the cost of recovering lost data? This can be hundreds of times more than the cost of the systems itself. a. Manual Syncing of Data - Assuming that there are backups to work from, it is often a long and cumbersome process to manually synchronize secondary computers or backed up databases. b. Data Loss - Data may be permanently lost, resulting in inaccurate reporting. This may have a knock-on effect, as these reports may be assumed to be correct leading to operational inefficiencies for years. c. Complexity of Procedures - The complexity of the restoration process itself can lead to errors in the re-inputting of data. This has a similar effect as that listed above. 48 The Cost of Failure - Why Mission-Critical Should Be Your Mission Building Resilience into Your System  Back to TOC System-wide redundancy - Many software platforms are limited in to a primary and a backup server. Look for a product that can provide unlimited levels of redundancy. Ensure that there is robust failover for all components like alarm notifications (email, SMS text message, voice-to-speech call out), thin clients, networks, etc. Also, if you have a redundant network is there an alarm to inform you if it fails? Real-time System Backup and Bi-directional Synchronization - Traditionally, SCADA systems are backed up offline or online. The former involves shutting down the system leaving operators blind and unable to manage alarms. The later can corrupt data during the process. Additionally, few platforms automatically sync historical data after failover. Often a separate backup methodology is required for third-party historians. Automating backups may require custom scripting. Manual backups are easily forgotten. 49 The Cost of Failure - Why Mission-Critical Should Be Your Mission Systems that support bi-directional synchronisation provide real-time synchronisation  Back to TOC of all the services that make up SCADA systems. In addition to the historian, this includes events, alarms, security, and application settings. This means each SCADA server can be an up-to-the-second copy of your whole application. No missed backups. Integrated Software Platforms - As we learned from the story of the sinkhole, gaps emerge overtime when disparate pieces are cobbled together. Many platforms use third-party products for core components such as Historians, alarm notifications, thin clients, and scripting. A single product ensures that everything works together seamlessly with new software versions. It also eliminates the risk that components are altered or discontinued by their manufacturers. Best of all, a unified approach means, one install, license agreement, training track and support contract. Application Version Control - Many system failures are the result of malicious acts by disgruntled workers or the unexpected consequences of innocent configuration. When things go wrong it is vital to identify who did what and to roll back to the last known working version immediately. While some SCADA providers support third-party version control, there are benefits to this being a native component such as the ability to automatically distribute the encrypted change list across all servers. Fast Response to Vulnerabilities from the Vendor – Software platforms regularly release new versions and features and often connect to devices developed long after applications are deployed. This ensures that security gaps will appear over time. The Industrial Control Systems Cyber Emergency Response Team (ICS-CERT) regularly conducts vulnerability analysis on products used in critical infrastructure. When they identify a potential security exploit, they contact the vendor who then has time to patch the 50 The Cost of Failure - Why Mission-Critical Should Be Your Mission vulnerability and distribute the solution before the vulnerability (and hopefully the fix)  Back to TOC is made public. This graph, published by Trend Micro, shows the number of days for common SCADA software vendors to patch security issues once notified by ICS-CERT. Note that many of these vulnerabilities are publicly announced before the respective vendors have provided a fix. Dependency on third-party components increases the length of time to find a solution to these issues. Pick a software company with a record of responding to threats in a timely fashion. See the full report here: https://documents.trendmicro.com/assets/wp/wp-hacker-machine-interface.pdf VTScada – Built to Power the Largest Critical Systems in the World In the graph above, the software vendor with the fastest patch time Trihedral Engineering, the makers of VTScada. This award-winning software has been making complex infrastructure easy to configure for over thirty-five years. Its unique unified design eliminates the gaps that plague other software platforms and allows systems to easily scale from two to over two million I/O. VTScada helps to eliminate downtime by supporting 51 The Cost of Failure - Why Mission-Critical Should Be Your Mission any number of redundant servers with automatic failover. The native Enterprise Histori-  Back to TOC an supports bi-directional synchronisation across all servers to keep your priceless data safe and available when you need it. Advanced Version Control is part of every application. VTScada is highly secure and has been used in some of the largest systems in North America for decades in industries such as power generation, broadcasting, water and wastewater, manufacturing, and oil & gas. VTScada continues to set the standard for mission critical software development. In 2022, VTScada’s Development Environment was certified to be compliant with IEC 62443-4-1 Maturity Level 2 Security for Industrial Automation and Control Systems. This defines secure development life cycle (SDL) requirements for products used in industrial automation and control systems. Trihedral was also awarded the 2022 Global Technology Innovation Leadership Award for SCADA and HMI systems from Frost & Sullivan, a research and consulting firm specializing in recognizing excellence in the field of industrial systems. Try the Free Industrial Version of VTScada VTScadaLIGHT is perfect for applications with up to 50 I/O. Individuals, businesses, and non-profits can install it on up to 10 PCs. There is even a step-by-step video tutorial. Download VTScadaLIGHT here: VTScada.com/light Contact Us North America (toll-free): 1.800.463.2783, Worldwide: 1.407.888.8203, info@trihedral.com, VTScada.com 52 Effective process control system migration, Part 3: Poll results, answers  Back to TOC Those who delay a DCS migration risk loss of uptime and reliability and have difficulty applying incremental improvements. Also, related controls and instrumentation upgrades incompatible with legacy equipment. D uring the Control Engineering webcast, “Effective Process Control Migration,” system integrators provided needed advice about distributed control system (DCS) migrations and the audience provided input in two poll questions often assist with effective process control system migrations. Many distributed control systems (DCS) and supervisory control and data acquisition (SCADA) systems have reached the end of useful life and are waiting for migrations. Expert presenters during the presentation were: • Don Bartusiak, president of Collaborative Systems Integration and co-chair of the Open Process Automation Forum. • Lynn Njaa, business development manager, DCSNext process automation consultant with Maverick Technologies. Poll questions: Audience provides input about DCS upgrades Participants listening live to the March webcast were asked to participate in the learning by answering two poll questions. The questions covered process and technology related opportunities missed by keeping a legacy-DCS rather than upgrading? 53 Effective process control system migration, Part 3: Poll results, answers Haven’t upgraded DCS? What process opportunities are missed?  Back to TOC During the webcast, audience was asked the following poll question: What process-related opportunities are you missing by keeping a legacy distributed control system rather than upgrading? Results follow: • 21.6% Uptime or reliability because of unplanned shut downs • 20.3% Difficulty applying incremental improvements • 17% Real-time transparency into processes and supply chain • 16.3% Modern alarm strategies • 16.3% Open systems standard-based architecture • 8.5% Sleep because of after-hours maintenance, repairs. Haven’t upgraded DCS? What technology opportunities are missed? The audience also was asked the following poll question: What technology-related opportunities are you missing by keeping a legacy distributed control system rather than upgrading? 54 Effective process control system migration, Part 3: Poll results, answers • 23% Related controls and instrumentation upgrades incompatible with legacy  Back to TOC equipment • 19.9% Modern cybersecurity tools • 18.3% Lack of remote monitoring and controls • 17.3% High-performance human-machine interface to attract younger engineers, get them up to speed faster and lower incident risk • 12% Antiquated training and simulation • 9.4% Inability to use mobility tools. More questions and answers about DCS migration After the Bartusiak and Njaa presented, there was time for a few questions to fill out the hour of instruction. Question: How do you integrate new process control systems with existing DCS or SCADA systems? Njaa: Planning is probably the best course of action for system integration. Having a SCADA system and mapping it over to a control system takes some time and documentation. I know that most of the projects that I’ve seen and done lag documentation, so mapping things out and planning things out is usually the best course of action. 55 Effective process control system migration, Part 3: Poll results, answers Bartusiak: Much of what requires an interface to is a package-mounted unit, or a  Back to TOC modular-unit, or an analyzer or something like that, and the workhorse means for the interface continues to be Modbus. When interfacing with a legacy system, a gateway product often is involved. The gateway works best when the supplier of the existing thing that you’re trying to interface to (not the new target system) really has to let you know what data structures and protocols are to build the gateway. There’s a technical and commercial aspect to whether or not it can even be done when systems are not built to industry standards. Question: How do you integrate different control systems of different brands from different vendors? Njaa: I have worked in many facilities that have a hodgepodge over the years, where the plant has grown by one separate unit at a time, or they have different legacy programmable logic controllers (PLCs) and then someone decides to put in a DCS and something else. They certainly can be all migrated and integrated into a common DCS. Or the data can be brought up into one operator interface where the view appears more seamless, if that’s what they choose to do. There are many ways to a approach system integration. Bartusiak: The short, provocative answer is that they cannot. [Or cannot easily be integrated.] Don Bartusiak and Lynn Njaa Don Bartusiak is president of Collaborative Systems Integration and co-chair of the Open Process Automation Forum. Lynn Njaa is business development manager, DCSNext process automation consultant with Maverick Technologies. 56 Integrating a DCS into an existing process cell  Back to TOC A distributed control system (DCS) needed to be installed in a process cell for a pharmaceutical company, but the process required an overhaul to the systems and standards in place to make it work. A pharmaceutical company was in the process of commissioning a control system for a process cell and asked ECS Solutions, a system integrator, to provide site support to complete the commissioning and qualification. The control system supplier had taken a simplistic approach to keep costs down. One controller with multiple human-machine interfaces (HMIs) provided the heart of the control system that also interfaced with some original equipment manufacturer (OEM) skids. The control hardware consisted of a controller with multiple HMIs. The controller code was all custom with little use of off-the-shelf products and was an assortment of custom blocks of code tied together, providing minimal functionality. Project preparation: Operators, sequence, procedures, documentation Little thought had been put into operators and how they would run the system to create quality product. Standard operating procedures had not yet been developed. Producing a quality product was dependent upon the operator selecting the correct sequence of functions from operator stations at multiple vessels. This approach exposed the company to risk in producing a quality product and controlling the rate of production. 57 Integrating a DCS into an existing process cell Since procedures had not been considered to this point, the system was not well docu-  Back to TOC mented and inevitably in store for substantial changes through the commissioning and qualification phases of the project. As the owners started to think practically about how operators would make product, change requests started pouring in. Changes to the custom logic were challenging. It was difficult to determine what effect small changes would have in other areas of the program. The integrator evaluated the situation and recommended a migration to the control system, providing batch software for the batch automation with process objects in the controller replacing much of the custom code. Procedural control, parameter transfers, data gathering The probability of errors in production led the integrator to recommend many activities carried out by the operators be significantly reduced. This was achieved by automating the operator activities related to procedural control, parameter transfers, and data gathering of the system. The integrator proposed the software be integrated into the system to provide a layer of automation that would reliably set the parameters for the equipment module and sequences at the right time, every time. The company agreed to add this layer of automation but requested the implementation of the software not interfere with the ongoing activities and all existing code and functionality be preserved without any changes being made. With the proposed integration, the operator would be required to simply select a recipe (stored by the sequencing engine) and select which equipment was needed to 58 Integrating a DCS into an existing process cell make the batch. The sequencing engine coordinated all activities, including the trans-  Back to TOC fer of parameter values, and the capturing of reports information. Furthermore, the system prompted the operator when a task required operator interaction. This work was carried out in such a way as to have no impact on the functionality of the existing equipment modules or sequences. Equipment model, phase classes, phases To do this, the integrator created an area model in the software suite to represent the existing equipment in the process area, with phase classes and phases representing the individual PLC equipment modules and PLC sequences. A wrapper logic was created as needed in phase manager phases to allow the software suite to properly control the existing equipment models and sequences. This resulted in a system capable of executing recipes without parameters being entered manually or equipment being started by an operator while capturing all pertinent report values. This improved the reliability and repeatability of the existing system while reducing human error. At this point to make a batch recipe, the integrator created procedures to encompass everything that needed to be done with every piece of equipment. Phases were created that could talk directly to the coordination sequences or coordination equipment modules. A wrapper was created for all their equipment modules, which lacked some of the flexibility required by the company. It should be noted the control system was integrated with the batch software, including key procedures and direct signatures. Existing custom code was replaced with a process objects standard to provide a more reliable and sustainable solution. 59 Integrating a DCS into an existing process cell Global standard library, standardized objects  Back to TOC The absence of a standard process object was the major driver for the migration. The custom code in the existing system had some limitations and issues in terms of coding, and later the control module was migrated. However, the equipment module layer remained standard since it was clearly commissioned. Some problems with the control module layer were identified – inconsistent logic, absence of signal filtering, and software anomalies – so the entire standard was replaced by a global standard library custom code, with standardized objects in place of the custom code. A significant amount of input/output (I/O) was migrated, and at the same time, the integrator was commissioning other systems. It became critical to maintain the timeline as the code was replaced and ensure it was thoroughly tested prior to validation. This required a risk response plan, recognizing errors due to migration would occur. Each controller was tested in a similar environment to prevent issues in terms of coding and ensuring sufficient memory in the processor to perform the migration. Simulated automation system, testing before live In terms of architecture, the integrator initially created a simulation system at their facilities, which allowed for testing and building recipes, and a test could be performed before implementing them in the live system. The simulation system was identical to the production system, giving latitude to things that could be done, and implementation was less invasive in the current architecture. Three different workstations, two of which were automation workstations, provided 60 Integrating a DCS into an existing process cell access to batch software and how the system was configured. At the automation work  Back to TOC stations, recipes could be edited, changes made to the HMI, or the controllers could be configured. Thin manager clients are available, and it is only necessary to provide the IP address and credentials of the thin manager server to automatically withdraw information so deploying a new client or replacing an existing client is quite straightforward. Every PC is virtualized with VMware and there is an ESXi post maintaining all machines, including the HMIs. In addition to the batch reports, the integrator also collaborated with the pharmaceutical company to produce custom reports. The reports show triggered operations and whether the operation passed or failed. The reason for a failure is also recorded. The major components of the reports are a recipe information section and a unit procedure summary. These sections capture details of interest regarding specific operations. The reports are created in a format used by the company. Changing role for control system integrator The integrator’s role in the project changed as time progressed. Initially, the aim was to put a wrapper on the existing code and not modify it, that is, the original agreement with the company. Subsequently, it was decided to make changes to improve the system, due to the lack of required functionality, and today the company can make extended batches, considered to be active production batches while the product awaits FDA approval. The batch software also has enabled a wealth of historical data to be logged in the form of batch records and reports, including electronic signatures. The modification has allowed the company to maximize the utilization of its equipment. Using a feature 61 Integrating a DCS into an existing process cell in the batch software, electronic prompts allow operators to review and respond, in-  Back to TOC cluding provision of detailed photographs of equipment setup for operators’ absolute activity clarity. Using the software suite recipes to drive clean-in-place (CIP) processes simplified the use of the CIP system and established repeatable consistent cleaning cycles that provide as much insight as product batches. This accelerated the commissioning and validation of those systems. The migration provided the consistency and reliability needed to provide the plant with the necessary functionality and achieved qualification and validation. This has allowed the pharmaceutical company to author version-controlled, electronic recipes that execute consistently from batch to batch. Christopher Rudd and Randy Otto Christopher Rudd, CEO, Rayn Business Development; Randy Otto, CEO, ECS Solutions. 62 SCADA & HMI Content Archive 2023 Winter Edition 2022 Fall Edition Thank you for visiting the SCADA & HMI eBook! If you have any questions or feedback about the contents in this eBook, please contact CFE Media at customerservice@cfemedia.com 2022 Summer Edition 2022 Spring Edition We would love to hear from you!