In order to ensure efficient testing we need to identify the efficiency criteria, i.e. which resource should be minimized. The key parameter that we can use is the time that it takes to prepare, execute, analyze and document the results of the tests.
Functional testing methods can be divided into several categories. They are related to the complexity of the functionality of the individual devices being used in the different levels of the hierarchical system, as well as the types of distributed functions implemented in it. This requires the selection of the right testing method for the specific type of test, as well is the use of testing tools that can automate the testing process.
From this point of view the following are the more commonly used testing methods:
A function in this case can be considered as a sub-system with different level of complexity, for example a system monitoring (SM) function, while the system is the complete redundant protection system.
Regardless of what is being tested, the test object needs to meet the requirement for testability. This is a design characteristic which allows the status (operable, inoperable, or degrade) of a system or any of its sub-systems to be confidently determined in a timely fashion. Testability attempts to qualify those attributes of system design which facilitate detection and isolation of faults that affect system performance. From the point of view of testability a functional element in a protection system is the unit that can be tested, because it is the smallest element that can exist by itself and exchange information with its peers in the protection system.
Another consideration is the purpose of the test and needs to clarify if the tests are performed in relation to acceptance of a new product or function to be used as a system monitor or process controller (or both), the engineering and commissioning of a substation component or the complete protection system or its maintenance. From that perspective different testing methods can be implemented even in the testing of the same functional element or function.
For example the testing of a system monitoring function during the user acceptance phase may focus on the testing of the measuring element characteristic using search test methods, while during the commissioning the operating times for different system conditions be the important ones achieved through transient simulation methods.
The knowledge of the internal behavior of the test object or more specifically the logic or algorithms implemented determine how the tests are being executed. The most commonly used test methods from this point of view are:
Black box testing
White box testing
An important aspect that needs to be considered during the testing is the availability of redundant devices performing the different protection system functions.
The following sections discuss in more detail the different testing methods listed above.
5.1 Black box testing
Black Box Testing is a very commonly used test method where the tester views the test object as a black box. This means that we are not interested in the internal behavior and structure of the tested function. Test data are derived solely from the specifications without taking advantage of knowledge of the internal structure of the function.
Black box testing is typically used for:
functional elements testing
protection system factory testing
protection system site acceptance testing
Since functional elements are defined as units that are the smallest that can exist independently and are testable, it is clear that black box testing is the only method that can be used for their testing.
The response of the test object to the stimuli can be monitored by the test system using the operation of physical outputs, communications messages or reports.
5.2 White box testing
White box testing is a method where the test system is not only concerned with the operation of the test object under the test conditions, but also views its internal behavior and structure. In the case of protection system it means that it will not only monitor the operation of the system at its function boundary, but also monitor the exchange of signals between different components of the system.
The testing strategy allows us to examine the internal structure of the test object and is useful in the case of analysis of its behavior, especially when the test failed.
In using this strategy, the test system derives test data from examination of the test object’s logic without neglecting the requirements in the specification. The goal of this test method is to achieve high test coverage through examination of the operation of different components of a complex function and the exchange of signals or messages between them under the test conditions.
This method is especially useful when we are testing distributed functions based on different logical interfaces. The observation of the behavior of the sub-functions or functional elements is achieved by through monitoring of the exchange of messages between the components of the test object.
The test scenarios however do not have to be different from the ones used under black box testing.
In IEC 61850 based systems white box testing is fairly easy to achieve based on the subscription to GOOSE messages whose data sets contain data attributes representing the status of all function elements that are used in the implementation of the tested function (for example SFM on Fig. 13).
5.3 Top-down testing
Top-down testing is a method that can be widely used for protection system, especially during site acceptance testing, when we can assume that all the components of the system have already been configured and tested.
Top-down testing can be performed using both a black box and a white box testing method.
The testing starts with the complete system, followed by function or sub-function testing and if necessary functional element testing.
In the case of factory acceptance testing, when not all components of a system or sub-system are available, it is necessary for the test system to be able to simulate their operation as expected under the test scenario conditions. In this case the test system creates the so called Stubs for functions or functional elements that are not yet available.
Each functional element is tested according to a functional element test plan, with a top-down strategy.
If we consider a protection system implementation in IEC 61850 for testing using a top-down approach, we will start with the definition of the function boundary.
The testing of the individual components of a system function might be required in the case of failure of a specific test, which is shown in Fig. 7. The function boundary for each of these tests is different and will require a different set of stimuli from the test system, as well as monitoring of the behavior of functional elements using different signals or communications messages.
5.4 Bottom-up testing
Bottom-up testing is a method that starts with lower level functions – typically with the functional elements used in the system – for example PTOC.
This method is more suitable for type testing by a manufacturer or acceptance testing by the user.
When testing complex multilevel functions or systems, driver functional elements must be created for the ones not available. The test system must be able to simulate any missing component of the system when performing for example factory acceptance testing.
There are many similarities in the test scenarios used in the bottom-up, compared to the top-down method. The main difference between the two methods is the order that the tests are performed and the number of tests required.