Water/wastewater capital projects commonly involve many levels of contractors and suppliers. Whether the projects are new greenfield developments or retrofit brownfield undertakings, a significant amount of coordination among these parties must occur to achieve startup success.
Systems with any amount of electrical controls and automation — which includes most any modern project — usually involve a control system integrator (SI). The SI designs and implements the control and visualization elements so that all subsystems operate and integrate properly, resulting in a fully functional project.
The task is complicated not only by the scope of supply assigned to multiple parties, but also by the number of vendors and products involved. An SI must work closely with many entities to ensure the success of capital projects in process plants, yet the communication may be obfuscated within the design/bid/build (DBB) project organizational hierarchy (Figure 1).
Significant underlying systems are usually supplied by original equipment manufacturers (OEMs), adding to the challenges. These include three main types of OEMs: suppliers of electrical equipment, suppliers of process skids and suppliers of automation components.
This article will focus on the relationships between an SI’s commissioning and startup team and each of these three types of OEMs for water/wastewater capital projects, although much of the discussion is applicable to many types of process plants and facilities.
Theory versus reality
In theory, most of the issues discussed in this article should have been resolved prior to startup during the design, development, factory testing and construction phases of the project. But in practice, there are so many interfaces and opportunities for technical and scope discrepancies that many issues persist into, or become exposed at, startup. Even with best efforts in the design and construction phases, some items can only be completed during startup because this is when all elements are fully installed, interfaced and in position for integrated testing under actual operating conditions.
Complications discovered late in a project are always more expensive and difficult to solve. Some startup crews are resigned to following the motto "last in, first out," which refers to the fact that they can’t perform their work until the last construction trade is done, yet they must be the first to fully complete their work so the process can run. For the SI, many detailed tasks must be performed during commissioning and startup, and any remaining issues require speedy resolution among the SI, the OEMs and other project entities.
Figure 1. The system integrator team must work closely with multiple entities to ensure successful completion of water/wastewater capital projects, including OEMs of packaged systems.
Automation commissioning and startup basics
Regardless of whether the SI is integrating OEM electrical equipment or process skids, there are some common activities. Projects use varying terminology, but for the purposes of this article, commissioning refers to basic control signal verifications to ensure equipment and systems can be monitored and commanded. Startup refers to functionally operating equipment and systems, first independently and then as part of a coordinated whole.
SIs accomplish their work by integrating a main automation system control platform, such as one or more programmable logic controller (PLC) or a distributed control system (DCS), with field devices and OEM equipment. Other common components include human-machine interface (HMI) computers in the control room and operator interface terminals in the field. Ideally, all communication, indication and control interfaces would be tested from end-to-end during a factory acceptance test (FAT) for the electrical OEM equipment, the control platform or both. But practically, these connections can sometimes only be implemented in the field.
Classic hardwired input/output (I/O) connections are simplistic and more burdensome to install than digital networks, but they are well understood and easy to verify. More complex fieldbus digital communication links might only require a single cable and are very capable, but these are more difficult to configure and troubleshoot.
Best practice for commissioning any automation platform with any other equipment is to first perform a complete I/O check documented on pregenerated forms to ensure all criteria are met. For hardwired I/O, this establishes that all discrete on/off signals and analog 4-20 mA signals are connected to the right place and operate with the right polarity and scaling. For fieldbus I/O, the process is similar, but one communication link can encompass many signals. The SI usually plays a central role in this effort, but the electrical contractor and OEM vendors must assist by verifying their portions of the system.
After commissioning is completed, the team can move to startup. At this point the SI and the OEM vendors remain involved, but the process design engineers and site operations personnel are now also required. These team members know the right order to operate the equipment and how to introduce water and other process streams. They understand how the main automation system needs to interact with field devices, process skids and OEM equipment.
For automated processes, if every contractor and vendor on a project remains in their own silo without effective communication and interaction with the others, the likelihood of mismatches and errors at the boundary interfaces increases dramatically. Even if every party internally tests their own area, integrated testing is the only way to prove full functionality.
On many projects, one of the first OEMs engaged by the SI’s commissioning and startup team is the supplier of electrical equipment because ensuring an adequate and safe supply of power is fundamental to a successful startup.
Figure 2. Power supply and distribution equipment, like this power lineup, must be carefully checked for correct installation, wiring and component sizing.
Engaging electrical OEMs
Electrical OEMs supply equipment to one or more contractors on a project. Examples of electrical generation and distribution equipment include switchgear, transformers, generators and panelboards. Electrical utilization equipment may include motor control centers, variable frequency drives and uninterruptible power supplies (Figure 2). Every project is unique, but in general terms the SI scope sometimes includes monitoring generation and distribution equipment, and more rarely automation of these items. However, these systems may have significant controls and interlocking systems in place when received at the site, with verification of proper operation required, usually by the OEM.
The SI is far more likely to get involved with automating motor control centers and variable frequency drives as these operate process motors. Furthermore, this equipment and the uninterruptible power systems are often intelligent devices, sometimes with onboard controls and network communications.
Therefore, the SI must place an early focus on reviewing submittals for the electrical OEM equipment and ensuring proper connections are available. For example, a variable frequency drive may support the Modbus TCP protocol, so the SI must ensure this protocol can be received and recognized by the main control system.
The SI may also need to send personnel in advance of commissioning and startup to work directly with electrical OEM personnel and installers to confirm the installation is coordinated with the SI automation detailed design.
Power distribution design is usually straightforward, but the scope can become less clear for automation. For instance, basic motor automation may involve interrelated detailed design, equipment, sensors and controls provided by a process equipment vendor, the electrical contractor and the SI. The SI has the specialized knowledge to evaluate all these items, and it is best positioned to identify and solve any gaps and overlaps.
Moving into detailed I/O checkout and commissioning of the electrical OEM equipment with the SI’s main automation system, digital network interfaces to intelligent power devices must be tested. Not only must basic signals be scaled properly, but intelligent devices have many configuration settings and modes of operation that also must be carefully selected and proven to ensure proper application. Just as I/O signals are documented on checkout sheets, the electrical OEM equipment configurations must be similarly documented.
Integration of process skids with the main automation system poses many of the same challenges as those found with electrical OEM equipment.
Figure 3. Process skids, such as this spiral screen, come in many configurations and sizes.
Integrating process skids
Process skids found in water/wastewater plants take many forms (Figure 3). Some are smaller and self-contained for shipment, while others must be broken down and assembled on site. Many come with complete onboard controls, while others may be procured as bare equipment to be automated in the field. Some are utility and support systems, while others perform a key process role. Here is a sampling of typical water/wastewater process skid types:
- Filtering
- Solids handling
- Reverse osmosis units
- Membrane bioreactors
- Ozone generators
- Ultraviolet systems
- On-site chemical production
- Chemical injection
- Odor control
- Fire suppression systems
- Engine drives
- Blower packages
Electrical OEM equipment as discussed in the previous section may involve some complications, but is mostly evaluated for basic functionality such as on/off, open/close, start/stop and run at a commanded speed. Process skids, however, are more likely to have extensive automation requirements, each of which affect overall performance. This includes sequencing for events like backwashes, cycling of equipment like ultraviolet systems based on demand, timing of trains within a unit, tuning of continuous regulatory control such as for chemical injection, and other process operations.
When process skids arrive with their own control package there may be a simple enable/disable interface exposed to the main automation system via a hardwired connection. But even in this case, a skid may also require operational parameters to be determined, set and tested. Therefore, just as when electrical equipment is commissioned and started up, the process skid vendors must be present and involved in the process because they are the specialists on how to operate their own equipment.
Furthermore, the situation scales up when multiple process skids must interact with each other. For instance, a chemical production system may supply a chemical injection system, or a filtering system might feed a reverse osmosis unit that feeds an ultraviolet system. The SI may be responsible for the main automation system controlling each of these process skids, and they may have programmed it to meet the design documents. But the process engineers and owner’s operators must work with the SI to ensure the implementation matches the design and the design functions as intended, with the process skids responding as specified.
Beyond the electrical OEM equipment and the process skids, the SI may need to solve problems central to the automation design or control platform.
Figure 4. Personnel starting up OEM equipment must be skilled working with automation platform hardware and software.
Automation OEM issues
The SI must be adept when working with automation platform elements such as the control hardware, software and field instrumentation (Figure 4). Here are some common areas that commissioning and startup crews must inspect and verify:
- Proper installation and wiring of instruments and devices (good grounding, signal isolation)
- Configuration of instruments, devices and OEM equipment
- Interface between each instrument and device, and the automation system (loop check)
- Proper installation and wiring of automation system components
- Controller software installation
- HMI software installation
- Basic closed-loop functionality
- Complete automation system functionality
- Sufficient operational personnel training
Each of these areas calls for formal test procedures to make sure all team members follow a consistent methodology, along with checkout forms to provide documented results. The SI is typically responsible for certain control platform hardware/software selection, device and instrument selection, control panel design and wiring design. The electrical contractor is responsible for field wiring. Various OEM vendors are responsible for their equipment, skids and devices. The process engineer is responsible for functional design, the SI is responsible for the programming and everyone is responsible for training.
So, if any one thing, small or large, fails to work as planned, it may be difficult to identify the root cause. The best run projects avoid assigning blame and instead employ a team-based problem-solving approach. Establishing this type of positive culture for executing the work and resolving the inevitable challenges must be led by the owner and general contractor, with the proper degree of autonomy granted to the SI.
Conclusion
Process automation capital projects, including many water/wastewater projects, involve many parts and players. With so many interrelated elements, it is easy to see how problems can persist into the commissioning and startup phase, even with predesigned and prepackaged OEM equipment. As the contractor most closely involved with the brain and nervous system of an automation project, the SI is uniquely positioned to discover and help solve any problems when integrating multiple subsystems. If the owner and general contractor empower this approach and facilitate early involvement of the SI, common problems can be quickly and efficiently resolved to achieve startup success.
Alan Horikawa, national service manager for Tesco Controls Inc., has more than 40 years of involvement with a wide range of industrial control systems. As a communications specialist, he has developed hands-on experience with all major instrumentation, control and telemetry systems used in water and wastewater. Horikawa led the teams commissioning one of the largest wastewater projects (more than 500 sites) in the southeastern U.S.
Scott Okasaki, Northern California field services manager, has more than 13 years’ experience with Tesco Controls Inc. in the modification, startup and support of industrial control system projects for the water and wastewater industries. He has led teams in the commissioning of Sacramento Regional County Sanitation District’s EchoWater Project, a $2 billion upgrade to produce cleaner water for discharge back into the Sacramento River.
