Experience from Implementing SCADA Alarm Alerting System in TNB Transmission System Azlan Muhamad Sufian Telecontrol Section, Engineering Department, TNB Transmission Level 4, Crystal Plaza, Jalan 51A/223, 46100 Petaling Jaya, Selangor, Malaysia
[email protected] Abstract— Tenaga Nasional Berhad (TNB) has been operating the Westinghouse SCADA (Supervisory Control and Data Acquisition) system to manage its power transmission network in Peninsular Malaysia since 1992. In 1999, the SCADA system was enhanced with an alarm alerting facility called the Substation Alarm Monitoring System (SAMS) [1], which was fully developed in-house and maintained by the Telecontrol Section, Engineering Department, Transmission Division, TNB. The system comprises several modular applications for logging SCADA alarm and sequence-of-event (SOE) information into a centralized database [2], providing remote access to the information and disseminating the alarms via e-mail and SMS (short messageing service) to the relevant maintenance personnel for alerting power system breakdown. For the past nearly ten years of its operation, SAMS has proven to be cost effective and beneficial solution to the organization. This paper focuses mainly on the evolution of the system. It describes how the system is used to assist in operation and maintenance of power system transmission, the problems identified during its operation and the solutions provided to improve the effectiveness of the system.
I. INTRODUCTION The SCADA system has become a vital source of information not only for operational purposes but also for maintenance and other purposes. Alarms and SOEs are the output produced in a SCADA system resulting from processing the data reported by the RTUs (remote terminal units) or SCS’s (substation control systems) at substations. Traditionally, the operators at the control centre are the main consumer of this information. As the SCADA system grows in I/O (input/output) counts, the rate of alarm arrival may increase proportionally and more personnel are needed to handle the alarms. Guidelines such as the EEMUA 191 [3] outline certain alarm rates should be achievable at steady state and during system disturbance by the correct configuration of a particular alarm system. As the operators may not be able to cope with abundant low-priority alarms mainly related to maintenance, the responsibility to handle them needs to be delegated to other parties by disseminating the alarms to the relevant personnel. The need to retrieve the SCADA alarms quickly and accurately by those interested parties had become the motivation of the development of Substation Alarm Monitoring System (SAMS). II. OVERVIEW OF TNB SCADA SYSTEM A SCADA system consists of the master station, the outstations and the communication subsystem as illustrated in
Fig. 1. A specific communication protocol is utilized to govern the message structure, its semantics and the way messages are exchanged.
Human-Machine Interface
Logger
RTU
Engineering Workstation
Communication Subsystem SCS Back-End Processor
Front-End Processor
Master Station
RTU
Outstations
Fig. 1 Main components of SCADA system
The master station comprises a network of servers and workstations, namely the front-end and the back-end processors, the human-machine interface (HMI) and the engineering workstation. The front-end processor is the gateway for the master station in which the protocol is implemented. Its main function is to pre-process the incoming data from outstations and outgoing data from the back-end processor. The back-end processor is the main processor that performs further processing of the incoming and outgoing data. It also contains the main database for storing outstation devices and I/O configuration as well as the real-time data from outstations. The engineering workstation is used to update the SCADA database and HMI display. The HMI is the operator console used for displaying the information from the back-end processor textually and graphically; and for activating control commands to the outstations. As end products of SCADA processing, alarms and SOEs are logged for hardcopy records by dedicated loggers. Alarm structure and its examples are shown in Table 1. SOEs are similar to alarms except that they are time-tagged by the outstations with 1ms accuracy. The outstations consist of devices installed at the remote substations for interfacing to the plants. These devices report plant equipment digital statuses and analogue values to the master station. The communication subsystem connects the master station and all outstations. In TNB, it is made up of power line carrier (PLC), fibre-optic and leased line networks. The TNB SCADA system called WESDAC32 is supplied by Westinghouse System Limited (WSL, now Schneider). It runs on VAX/VMS servers in DECnet network environment from Digital Equipment Corp. (acquired by Compaq, later merged with HP).
TABLE 1 ALARM STRUCTURE AND EXAMPLES
Date & Time (DD/MM/YYYY HH:MM:SS) 06/05/2008 11:36:21 06/05/2008 11:36:19 06/05/2008 11:13:46 06/05/2008 11:13:15 06/05/2008 11:06:25
Station Mnemonic TBIN BMKA TMID BMRH PGGS
Device Description STATION FIRE ALARM MMKA CB 205 SF6 LOW ST 1 132 KV T1 CB 110 JJNG1 CB L15 M2 DIST PROT OPERATED 275 KV BC W20 LOW AIR PRESSURE
Currently the master station is connected to over 400 outstations. From this figure, the majority of 82.5% of them are RTU based while the remaining 17.5% are SCS based stations. The RTUs and SCS’s are supplied by various vendors. The SCADA system utilizes a proprietary protocol called WISP+ (Extended Westinghouse Integrated System Protocol). III. OVERVIEW OF SUBSTATION ALARM MONITORING SYSTEM Many commercial alarm management packages are available in the market but the development team decided to build the system in-house due to economic reason. An overview of the developed system is presented in this section. A brief description for each of the four modular applications in SAMS with reference to Fig. 2 follows. SAMS Server ODBC WOMS Server File Sharing
CB Status File
SAMS E-mail
SAMS Engine
SCADA Server
SCADA Alarm Logger
ODBC PC running CAMS HMI
POP3
E-mail Server/ SMS Gateway
PC running IDNC RS-232
SMTP
Database (SQL Server)
SAMS Web
E-mail Client SMS
ODBC Mobile Phone
HTTP Client PC with Web Browser
Client PC with SAMS Access
Fig. 2 SAMS system configuration showing its modules, interfaces and external systems
A. System Configuration SAMS consists of four modular applications for specific roles, namely SAMS Engine, SAMS E-mail, SAMS Web and SAMS Access as shown in Fig. 2. The server running Windows 2000 operating system is located in the SCADA server room at TNB’s National Load Despatch Centre (NLDC). The applications were developed using commercial off-theshelf (COTS) tools, mainly MS Visual Basic and MS Visual C++ as well as open source package, namely PHP (for web application). The software was designed using object-oriented design methodology. As a client-server application, SAMS resides on the Intranet domain and is accessible by local and remote users via TNB’s Wide Area Network (WAN). SAMS takes advantage of existing information and communication technology (ICT) infrastructure that is readily available in the utility. So, installation of additional equipment apart from the server is not necessary.
Status OPERATED ALARM OPEN ALARM NORMAL
B. Module Description SAMS Engine intercepts alarms and SOEs between SCADA server and SCADA loggers via RS-232 serial communication ports and stores them into the centralized database. The MS SQL Server relational database is used to log the data. Fig. 3 illustrates the module user interface. SAMS E-mail filters out critical alarms i.e. circuit breaker trip and sends them to e-mail or SMS gateway using SMTP (simple mail transfer protocol) to alert the relevant maintenance personnel on standby. Other maintenance related alarms despatched by this module include: • Low battery charger • Low SF6 gas • Transformer guard • Station fire The module user interface is shown in Fig. 4. SAMS Web provides web interface to remote clients for querying SAMS database by HTTP (Hypertext Transfer Protocol) request using readily available web browser on their PCs. The MS IIS (Internet Information Server) is used for the web server with PHP plug-in. Fig. 5 shows a snapshot of the website viewed from a web browser. The website provides facility to search the alarms and SOEs: • For a specified date (and time) • From a specified regional area • From a specified station • For a specified description keyword SAMS Access, an alternative to the web interface, is a client program that needs to be installed at the remote client PCs for more flexible browsing and searching of the alarms via ODBC (open database connectivity) [2]. The module user interface is shown in Fig. 6. Alarm and SOE lists retrieved from SAMS website and SAMS Access are downloadable by the client’s PC and they can be manipulated (sorting, filtering etc.) further using suitable application such as MS Excel. The information can then be included in technical reports or records. SAMS website and SAMS Access modules have become the core applications for alarm surveillance at 24-hour manned substations. These strategic stations act as regional monitoring centres for smaller groups of other stations.
Fig. 3 SAMS Engine user interface
Fig. 4 SAMS E-mail user interface
Fig. 5 SAMS website
Fig. 6 SAMS Access user interface
IV. SYSTEM ENHANCEMENTS AND EXPANSIONS From the beginning of its operation to the present time, the system has been enhanced and expanded for improvement.
B. Expansion to External Systems In 2002, under a separate project, the Island Detection and Network Colouring (IDNC) application [5] which consumed the circuit breaker status alarms from SAMS was developed in-house by System Operation Department, TNB Transmission Division (see top-left side of Fig. 2). This application was used to assist the control centre operators to identify network island formation during power system split resulting from transmission line faults and a series of protection malfunctions. Unfortunately the IDNC application
A. Enhancement of Server and Software Modules SAMS was initially developed as a pilot project using a refurbished server back in 1999. However, due to increasing demand from the users, it was replaced with a higherperformance server in 2001. Enhancement was done to the original applications without major change in the system design [4].
was decommissioned a year later due to lack of software support and maintenance. Under another initiative from the Telecommunication Department of TNB ICT Division, the Communication Alarm Monitoring System (CAMS) was established in 2004. In this system (Fig. 2, bottom-left side), telecommunication equipment alarms are channelled from SAMS to CAMS and a third party HMI package is used to display the equipment statuses graphically. The department has also set up a dedicated control centre for managing the telecommunication system network using CAMS. Between 2007 and 2008, SAMS E-mail module was further enhanced by linking it to another external system called WebBased Outage Management System (WOMS). WOMS (shown in top-left side of Fig. 2), operated and maintained by the System Planning Department of TNB Transmission Division, provides information on substation outage scheduling. By linking to WOMS, alarms generated from outage activities can be identified and blocked from being despatched by SAMS E-mail. C. Extending Connectivity In mid-2008, another effort to enable SAMS web access via VPN (Virtual Private Network) was made with assistance from the IT Department, TNB ICT Division. By having this connectivity, it allows maintenance personnel to monitor the alarms virtually from anywhere outside the office using Internet connection, possibly in a Wi-Fi environment from their PDA devices. V. SYSTEM APPLICATIONS AND USER DOMAIN The main applications of the system and its user domain are described in this section. A. Decision Making Support Tool SAMS is used primarily as a decision-making support tool by the various departments. System Operation Department benefited from it through the use of IDNC application in detecting power system islanding. The Maintenance Department from the four regional areas utilise it for alerting power system breakdown. As a result, unexpected equipment failure can be avoided and fast power restoration can be done. The existence of SAMS has also led to the establishment of CAMS, which is used to manage telecommunication network in TNB by the ICT Division. B. Historical Data Source As secondary use, SAMS becomes a centralized archiving database for historical alarm analysis. The database is often being accessed by Engineering Department personnel and other parties during fault investigations. Performance analysis can also be done to produce statistical information for certain equipment. For example, a short-term internal study in 2005 revealed that the success rate of control operation (opening or closing circuit breakers) by NLDC is about 85% [6].
C. System Users The number of users from the various departments accessing the system at a particular time is estimated around 50 people. A study conducted in 2004 indicated that a higher number of people would access the system during tripping events or major system breakdown [7]. The number of users is expected to grow as new application is introduced. VI. IDENTIFIED PROBLEMS AND SOLUTIONS Some of the problems faced during the operation of SAMS and how they are managed are highlighted here. A. Ineffective Alarm Filtering The main problem faced by the system during the initial stage of its operation was the ineffective alarm filtering capability of SAMS E-mail module. Nuisance alarms due to intermittent states or relay chattering were falsely being alerted to the maintenance personnel, thus causing unnecessary troubleshooting actions. As a consequence, the users’ confident level on this service had decreased. To address this issue, SAMS E-mail module was reviewed and enhancement was made on the alarm filtering algorithm. To further improve it, the module has also been linked to the WOMS database to block alarms from outage activities. B. Unavailability of Dedicated Messaging Facility In the original SAMS configuration, the filtered alarms were being notified via e-mail paging service since it was not able to provide reliable SMS notification due to unavailability of dedicated messaging facility. Alarm notifications were often delayed or worse, totally failed. The maintenance personnel later had abandoned the pager, as it was inconvenience to have as a communication device in addition to the mobile phone. Only recently, the Telecommunication Department of TNB ICT Division has set up a dedicated SMS gateway to overcome this problem. At the point of writing, tests are being carried out and the SMS facility is scheduled for reactivation. C. Passive User Interface From feedback given by the users in 2006, SAMS Access, the client user interface module lacked the ability to automatically alert the users on the arrival of the most recent alarms. Since the module provided a static alarm view at a particular moment, the user had to manually refresh it regularly to obtain the latest alarm view. Provision for an automated function has been made after reviewing and enhancing the module. SAMS Access now is able to refresh the alarm view automatically at a preconfigured rate with audible alarm. A summary page for displaying persistent alarm list has also been provided for the user to track alarms that have not been normalised yet. D. Software Bugs SAMS also experienced several software problems due to buggy code and memory leak. Maintenance on the system had to be done by periodic restarting of the server that caused
disruption to the users. Nevertheless, after the bugs were identified and removed, the server now requires minimum maintenance and system availability has increased.
operation, SAMS has proven to be a cost effective solution to common problems faced by a power utility in alarm management.
VII. FUTURE PLANS The existing SCADA master station is due for replacement by the third quarter of 2008. The new system is currently under acceptance testing and is expected to be operational by 2009. Provision of facilities similar to SAMS has been incorporated in the specification of the new SCADA system. Existing SAMS modules could also be reused, hence reducing development time on the new system. A study is also to be conducted to explore the possibility to integrate SAMS with IEC 61850-based devices using suitable data exchange interfaces such as OPC (Object Linking and Embedding/OLE for Process Control) or CIM (Common Information Model). Its main purpose is for becoming a hub for substation information, commonly known as the enterprise gateway that can be shared across the organization.
ACKNOWLEDGMENT The author wishes to thank all relevant personnel who have contributed to the implementation of this project, particularly the development team members and the users across all TNB regional areas. He would also like to acknowledge the departmental top management for their support.
VIII. CONCLUSIONS SAMS provides a bridge from a legacy SCADA system to the outside world through the universal TCP/IP interface. As a result it has improved the effectiveness of communications between the control centre and other external parties in retrieving SCADA alarms. SAMS has successfully enabled SCADA information sharing among TNB personnel. With almost ten years of
[5]
REFERENCES [1] [2] [3] [4]
[6] [7]
W. A. W. Kamarulzaman et. al., “Substation Alarm Monitoring System”, in Proc. CEPSI’00, 2000. D. Proudfoot, D. Taylor, “How to Turn a Substation into a Database Server”, IEEE Computer Applications in Power, April 1999. Engineering Equipment and Materials User Association, “Alarm Systems: A guide to design, management and procurement”, Pub. No. 191, London: EEMUA 1999. A. M. Sufian, M. S. M. Taha, “Enhancement of Substation Alarm Monitoring System”, in Proc. TNB Technical Conference’01, Malaysia, 2001. T. A. Sulaiman, “Using the Real-Time Island Detection and Network Colouring Application in the National Load Despatch Centre”, in Proc. TNB Technical Conference’01, Malaysia, 2001. R. M. Yusof, “SAMS: Despatched, Success and Failed”, TNB, Industrial Training Report, 2005. “Telecontrol Section Internal Report on SAMS Users for Financial Year 2003/2004”, TNB, Internal Study Report, Sept. 14, 2004.