A systematic review of real-time detection and classification of power quality disturbances

Abstract

This paper offers a systematic literature review of real-time detection and classification of Power Quality Disturbances (PQDs). A particular focus is given to voltage sags and notches, as voltage sags cause huge economic losses while research on voltage notches is still very incipient. A systematic method based on scientometrics, text similarity and the analytic hierarchy process is proposed to structure the review and select the most relevant literature. A bibliometric analysis is then performed on the bibliographic data of the literature to identify relevant statistics such as the evolution of publications over time, top publishing countries, and the distribution by relevant topics. A set of articles is subsequently selected to be critically analyzed. The critical review is structured in steps for real-time detection and classification of PQDs, namely, input data preparation, preprocessing, transformation, feature extraction, feature selection, detection, classification, and characterization. Aspects associated with the type of disturbance(s) addressed in the literature are also explored throughout the review, including the perspectives of those studies aimed at multiple PQDs, or specifically focused on voltage sags or voltage notches. The real-time performance of the reviewed tools is also examined. Finally, unsolved issues are discussed, and prospects are highlighted.

1 Introduction

Power Quality (PQ) is defined as the set of characteristics of electricity at a given point in an electrical system [1], and these characteristics are evaluated against a set of agreed reference parameters. Thus, PQ Disturbances (PQDs) are deviations in these characteristics from the reference parameters, deviations which can be perceptible to the users of the electrical power grid (producers and consumers) [2]. Therefore, PQDs are classified and studied as the combination of voltage quality and current quality [2]. The impact of PQDs will depend on the severity of these deviations interfering with the expected operation of the electrical systems. Noticeable PQDs will directly affect the interaction between consumers and producers of electricity, leading to energy inefficiency, limited generation/consumption of electricity, malfunction and damage of sensitive equipment, maloperation of control-based industrial processes, etc. [3].

The detection and classification of PQDs are among the main components of PQ monitoring systems in the smart grid paradigm [3, 4]. The detection process helps to indicate the time and location of deviations in voltage and current, whereas the classification process contributes to the identification of disturbances and sources of disturbance, and the selection of adequate mitigation techniques to overcome current and/or future equipment malfunction. The real-time approach of PQ monitoring systems (synchronized, continuous, single, or multi-point measurements) can help in the understanding of PQD propagation in the power grid and allow accurate and faster decisions regarding the mitigation of PQ issues.

Some PQDs are more likely to appear and cause equipment malfunctioning than others. Voltage sag, also referred to as voltage dip, is the decrease of the RMS voltage between 0.1 and 0.9 pu lasting from 0.5 cycle to 1 min [5, 6]. Voltage sags are mainly caused by faults in the power system, transformer energizing, motor starting, and switching of heavy loads [5, 6]. The frequency of occurrence of voltage sags is between a few tens and one thousand events per year, with typical durations of less than 1 s, and voltage drops above 40% [6]. It is widely known as one of the PQDs that produce the highest economic losses in industrial, commercial, and microgrids/isolated networks [4, 7,8,9,10,11]. Other widely studied PQDs are swells, interruptions, imbalance, flicker, and harmonics [4, 9, 11], disturbances for which specialized algorithms have been proposed in the literature. In contrast, there are PQDs that have not yet been widely explored, especially in the scenario of high penetration of renewable energy resources and their interfacing power electronics circuits.

Voltage notches are steady-state sub-cycle waveform distortions produced by power electronic converters when current is commutated from one phase to another leading to short-duration overcurrent [5, 9, 12, 13]. A voltage notch is characterized by its depth, i.e., the average distance of the line voltage to the ideal sine wave during notch; width, i.e., the notch duration with values of less than half cycle; and area, i.e., the product of the notch depth times the notch width [12]. There is still a lack of comprehensive assessment and accurate techniques for real-time detection, classification, characterization, and aggregation of cycle-based voltage notches when considering the operation of power grids. Thus, notches have been mainly assessed with non-specific, multi-disturbance detection and classification techniques (“multiple PQDs” techniques).

The detection and classification of PQDs in both offline and real-time applications have been addressed from different perspectives. Several methods for the classification of three-phase unbalanced voltage sags due to faults are described in [14]. Signal processing and Artificial Intelligence (AI) techniques for classification of several PQ events are reviewed in [13], where Fourier Transform (FT), Short-Time FT (STFT), and Wavelet Transform (WT) are considered as the main signal processing techniques, and expert systems, fuzzy systems, Artificial Neural Networks (ANNs), and Genetic Algorithms (GA) as the main AI techniques. In [13], the authors also propose an initial structure for the detection and classification of PQDs, including feature extraction and classification (decision making). Some of the techniques used for signal analysis are described in [15]: FT, STFT, WT, Gabor Transform (GT), Stockwell Transform (ST), Kalman Filter (KF), etc., as well as some automatic classification techniques such as ANNs, fuzzy logic, Support Vector Machines (SVM), and Bayesian classifiers. The detection and analysis of voltage events are reviewed in [9], which focuses on offline and real-time techniques using a combination of WT and ANNs, SVM, and Fuzzy Expert Systems (FES). Reference [16] introduces the category of optimization techniques for the classification of PQ events and also presents a comparative chart taking into account input data (synthetic/practical) and input noise. In [17], WT is compared with other techniques for the detection of transient disturbances in voltage supply systems. Methods for the identification of voltage sag sources are explored in [18], which classifies them into single and multi-monitor (multi-point) based measurements. This first set of works (2003–2013) are focused on PQD detection and classification techniques, and most do not include pre-or post-processing techniques.

Comprehensive reviews on signal processing, AI, and optimization techniques applied in the detection and classification of PQDs are developed in [19, 20]. These reviews present two complementary structured methodologies for PQD detection and classification, i.e., input data space, feature extraction, feature selection, classification, and decision space [19], and pre-processing and post-processing stages [20]. A holistic taxonomy for signal processing, AI, and optimization techniques are also presented, highlighting the need for more methods in the detection and classification of real-time, noisy, three-phase, single and multiple PQDs.

The RMS method, WT, ST, ANNs, SVM, and some indices for voltage sag disturbance recognition are reviewed in [21]. In [4], WT and SVM are reviewed and applied for the detection and classification of sags, swells, and harmonics, including a table that relates PQDs and corresponding standards. In [22], a review of PQDs measurement and analysis on shipboard power systems is presented, which mainly refers to voltage and frequency fluctuations, fault detection and classification, voltage sags and swells, transients and voltage notching, harmonic distortion, and voltage imbalance. In addition to signal processing techniques, reference [10] classifies fuzzy logic, ANNs, SVM, particle swarm optimization, and GA as soft-computing techniques for feature extraction and classification of PQDs.

A comprehensive review and comparison of PQD detection and classification are presented in [23], which describes the advantages and disadvantages of different techniques. Signal processing techniques are classified into eight categories: FT, WT, ST, GT, KF, Hilbert-Huang Transform (HHT), Mathematical Morphology (MM), and others including Variational Mode Decomposition (VMD). For the classification of PQDs, seven categories are presented: ANNs, SVM, FES, neuro-fuzzy system, Extreme Learning Machine (ELM), deep learning, and miscellaneous pattern recognition techniques. In [24], a taxonomy is proposed for digital signal processing techniques, one which presents the categories of non-parametric and parametric techniques including in the latter KF, rotational invariance techniques, multiple signal classification, and autoregressive-moving-average, etc. These works contribute to widening the state-of-the-art techniques used for the detection and classification of PQDs, while outlining the advantages and disadvantages for the techniques. However, search rules used in the literature reviews are not presented.

Reference [25] presents a detailed assessment of the theory and application of ST for the detection and classification of PQDs. A mitigation stage is proposed in [3], where the input data is from the stages of detection and classification of PQDs in the context of renewable energy resources.

Real-time techniques are explicitly addressed for the detection and classification of PQDs in [11], which presents the search rules, the evolution of PQDs publications, the internal structure of a typical embedded system for PQD classification, and other comparative analyses. Moreover, the need for testing the detection and classification algorithms with no synthetic (measured) PQDs is highlighted in [26], and it concludes that such an aspect will be crucial for applications in smart grids. In this same approach, references [27] and [28] present a review of the potential applications of deep learning in smart grids, where the structured scheme of consecutive steps for detection and classification of PQDs is blurred into a compact, general, black-box approach. Finally, in [29], the main advantages and disadvantages of several techniques for the detection and classification of PQDs are summarized, including an accuracy assessment of the methods proposed in different publications.

Based on the above synthesis, there is extensive literature on detection and classification of PQDs but some relevant topics still require further research and analysis from different perspectives, e.g., algorithms for real-time applications, the need of applications in the field, etc. Other topics have been marginally explored such as characterization and classification of voltage notches. In this context, the proper selection of relevant literature to analyze the current state of the art is challenging. Therefore, this paper proposes a systematic and reproducible approach to identify relevant literature in both extensively and marginally explored topics. Moreover, the extensive literature is worth studying from the quantitative perspective with a bibliometric analysis to identify research trends. The bibliometric-based methodology proposed in this paper is then applied to carry out a systematic literature review of the real-time detection and classification of PQDs. The main contributions of this paper are:

• The proposal and application of a novel methodology for reproducible and systematic literature reviews.

  • The application of a bibliometric analysis to the metadata retrieved from bibliographic databases.

  • The description of techniques for real-time detection and classification of PQDs.

  • The categorization of techniques used in the stages of PQD detection and classification, and their corresponding advantages and disadvantages in the context of electrical engineering.

  • The identification and emphasis on marginally explored PQDs (voltage notches).

Figure 1 illustrates the structure and logical framework of the paper. In line with Fig. 1, the remainder of the paper is structured as follows. Section 2 shows the concepts and theoretical background for the understanding of PQDs and their classification, while Sect. 3 presents the stages of the systematic methodology for the literature review. Sections 4 and 5 develop quantitative and qualitative analyses of the literature in the bibliometric analysis and the literature review, respectively. Section 5 describes and analyzes the stages for PQD detection and classification. The discussion of the main findings related to the literature review is summarized in Sect. 6, and Sect. 7 proposes some perspectives for future research. Finally, Sect. 8 presents the conclusions.

Fig. 1

Structure and logical framework of the paper

2 Background on power quality disturbances (PQDs)

The characteristics of electricity at a given point on an electrical system, compared to a set of reference technical parameters are widely known as PQ [1]. The reference (ideal) signals in three-phase systems are three pure sinusoid waveforms with constant phase angles and amplitudes, and a 120° phase shift between them. A PQD therefore refers to any deviations from the reference voltages and currents. In this context, PQ is assessed as the combination of voltage quality and current quality [2].

Table 1 summarizes the features of typical PQDs in the power grid. The types of PQDs are grouped into variations (continuous deviations) and events (finite deviations).

Table 1 Classification of typical PQDs

Variations are usually produced in a typical operation of a power system, while events are generally unpredictable phenomena. Given the dynamic nature of the power system (power frequency changes, slow voltage changes, imbalance), the main sources of common variations (waveform distortion, notches, fluctuations) are the power electronics interfaces, e.g., for renewable energy sources, energy-saving equipment, electric vehicle chargers, and mains communication systems. Switching maneuvers, faults, and lightning strikes are typical sources of events. By using these two categories, the taxonomy of typical PQDs (continuous and finite deviations) can be structured for detection, processing, classification, and further processing.

According to Table 1, voltage notches are classified as variations (continuous deviations), while other disturbances in this category are power frequency variations which refer to deviations of the power system fundamental frequency from the nominal value (50 Hz or 60 Hz). Also, among variations, slow voltage changes refer to the increase (overvoltage) or decrease (undervoltage) of about 20% of the rated RMS voltage lasting more than 1 min. Waveform distortions including harmonics, interharmonics, and supraharmonics, are periodic deviations from the ideal power frequency sine wave, characterized by the spectral content of the deviation. Fluctuations are systematic changes of the voltage envelope and can be perceived as flicker. In addition, imbalance refers to differences between voltage and current magnitudes of phases in a three-phase system, and/or a deviation from the ideal 120° phase shift between phases [5].

Voltage sags are events (finite deviations) according to Table 1. Other events include swells, which refer to the increase of the RMS voltage above 1.1 pu lasting from half cycle to 1 min. Interruptions are characterized by the complete loss of voltage, i.e., less than 0.1 pu. Finally, transients are sudden changes in voltage or current that occur over a short period of time [5].

3 Methodology for the literature review

The general schema to conduct the literature review is based on the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) [30]. According to Fig. 2, the process is performed through four stages, namely, identification, screening, eligibility, and decision. Details on the stages are given in the following subsections.

Fig. 2

Stages of the systematic literature review

3.1 Identification and screening

In the identification stage, an advanced search rule is formulated in the Scopus database to obtain bibliographic metadata of publications related to the detection and classification of PQDs. The search rule is shown in Fig. 3 and is formed by logical operators and sets of terms to be searched only in titles or titles, abstracts, and keywords. Set 1 has general terms such as detection and classification, while terms with a broader meaning are listed in Set 2. To avoid misleading results with these broader terms, the constraints including terms of Set 1 in title, abstract, or keywords are merged in Gate A. Set 3 contains terms related to PQ with a focus on sags and notches, while Set 4 also refers to PQ with more vague terms. Thus, to avoid misleading results, the constraints including terms of Set 5 are merged in Gate C. The search merged in Gate E must include terms of general topics (B) and PQ (D). Finally, the terms of Set 6 are merged in Gate F to include more publications related to voltage notches because the results in this topic are very limited.

Fig. 3

Advanced search rule applied in the Scopus database

The search rule was applied on March 2nd, 2022, and limited to publications up to 2021. The search yielded 4068 records. Then, in the screening stage, duplicates were removed. The resulting 4059 records, including 2140 conference papers (53%), 1841 journal articles (45%), 61 reviews (2%), and 17 other types of publications, e.g., books, book chapters, etc. (< 1%), are studied quantitatively in the bibliometric analysis of Sect. 4.

3.2 Eligibility and decision

The eligibility and decision stages present the selection procedure to identify a reduced set of relevant and diverse research papers to be analyzed qualitatively in the critical review. Only journal and conference papers are considered because they summarize very well the state of research. The selection procedure is carried out in four steps as follows.

• 1. Categorization Two dimensions are defined to classify papers, namely, period according to date of publication (i.e., ≤ 2005, 2006–2010, 2011–2015, and 2016–2021) and type of PQD (i.e., multiple PQDs, voltage sag, and voltage notch). All papers are classified according to both dimensions to obtain a final selection of papers distributed in time and type of disturbance. The classification in period is straightforward, and the classification according to the type of disturbance is made by searching the terms sag (or dip) and notch in titles. Papers not classified as sag or notch are included in the category of multiple PQDs. The resulting distributions of papers in both dimensions are shown in the light-blue rectangles in Fig. 4.

• 2. Scoring equation An equation is formulated to measure the relevance of papers from different perspectives, with five proposed indices. The title similarity index is calculated as the ratio of word coincidences to the total of words in the paper title, while the coincidences are given by predefined terms related to the topic, e.g., real-time, sag, detection. Likewise, an abstract similarity index is calculated in abstracts, whereas the cross-citation index is defined as the ratio of cross-citations (i.e., the number of citations by papers in the search list) to the number of citations of the most cited paper of the same year or later. Similarly, the review cross-citation index is calculated considering only citations by review papers in the list. Finally, the journal index is defined as 1 for journals and 0 for conference papers.

Fig. 4

Number of papers through the process of selection for the critical review, organized by period and type of PQD

Weights are estimated for each of the five indices using the Analytic Hierarchy Process (AHP) [31]. AHP is a multicriteria decision-making approach in which factors are arranged in a hierarchic structure. Thus, weights are obtained by estimating the relative magnitude of pair-wise comparisons and further computations as described in [31]. The resulting weights for the proposed five indices are reported in Table 2. The total score is calculated for each paper by summing the indices multiplied by the corresponding weights. A total of 404 top papers (10% of the papers in the search list) are selected according to scores considering a larger proportion of papers for periods between 2011 and 2021 (i.e., 2011–2015, and 2016–2021) than for periods before 2011 (i.e., ≤ 2005, 2006–2010). This prioritization is done to consider more papers from recent periods. Also, the number of papers selected for types of disturbances is distributed as evenly as possible. The results are shown in the sky-blue rectangles in Fig. 4.

• 3. Quality assessment An assessment is conducted on the top 404 full-text papers to reduce the selection for the critical review, while the five questions in Table 3 are formulated. A set of three qualifications is defined for the questions: {0 (not at all), 0.5 (moderately), 1 (absolutely)}. AHP is then applied to estimate the weight for each question, and the results are reported in Table 3. The questions are evaluated for each paper, and the total score is calculated as the sum of the results of each question multiplied by the corresponding weights. Finally, 167 papers with a score higher than 0.8 are selected for the critical review, as reported by the dark-blue rectangles in Fig. 4.

• 4. Extension method Twelve additional papers related to voltage notches are identified and included because very few (only 13) are initially detected for this disturbance. The extension method applied to identify additional papers is based on the concepts of bibliographic coupling and co-citations as explained in [32]. The resulting number of papers is reported in the yellow rectangles in Fig. 4.

Table 2 Weights for indices of the scoring equation

Table 3 Questions for the quality assessment and their weights

The selection procedure conducted using the above four steps results in the papers reported in Table 4, where they are organized according to the proposed categories, i.e., period, and type of PQD.

Table 4 References identified for disturbance types and periods

4 Bibliometric analysis

The number of publications related to the detection and classification of PQDs over time is shown in Fig. 5, which also shows the number of those publications that include the term “real-time” (or “online” or “on-line”) in the title, abstract, or keywords.

Fig. 5

Evolution over time of the number of publications per type related to the detection and classification of PQDs

The increasing interest in PQ detection and classification is observed especially from 2000 to 2010 (see Fig. 5). An apparent decrease occurred in 2014, but then the trend became positive again up to 2019. While the trend of the total number of publications decreased in 2020 and 2021, those related to real-time show an apparent increase. Moreover, Fig. 5 shows that a small number of publications include the term “real-time” in title, abstract, or keywords, reaching significant appearances only in the last decade. This situation highlights the need for further studies in the real-time detection and classification of PQDs.

To observe the state of research by country, Table 5 reports the number of publications from different countries, according to the affiliation of the first author. The top seven contributors, i.e., those countries with more than 100 publications, are shown. China has the largest contribution with a significant lead over the rest, and is followed by India, USA, Brazil, and Spain. Table 5 also shows the number of those publications with the term “real-time”. The corresponding ranking is similar to the ranking for the total number of publications, with the exception of Brazil which appears in sixth place. Table 5 also reports the percentages of contribution of the top seven countries to the total number of publications, with the top three, i.e., China, USA, and India, contributing more than 50% of the total number of publications and more than 60% of the publications with the term “real-time”.

Table 5 Top seven countries according to the total number of publications on the topic

Regarding the original language of publications, English is dominant with 85%. Moreover, a significant 13% were written in Chinese, while the remaining 2% correspond to other languages such as Spanish, Polish, and Portuguese.

A set of categories from the search are also identified from titles and the results for the number of publications in each category are presented in Fig. 6. The number of publications with or without the term “real-time” in the title, abstract, or keywords, are also identified for each category.

Fig. 6

Number of publications by a topic and b type of PQD, with or without the term “real-time”, and respective percentages

Figure 6a reports the results for general topics and shows that the largest number of publications is related to detection. The term detection usually refers to algorithms for identifying the occurrence of a PQD. The second category according to the number of publications is monitoring, which refers to the tracking of power signals, including the PQDs. In this case, hardware implementation emerges as an important aspect. The third category is classification which refers to the recognition of the type of PQD, e.g., sag, notch, harmonics, transient, flicker. The category analysis is broader in meaning and comprises several aspects described in the rest of the categories. The category assessment includes methods for impact analysis of PQDs. Characterization refers to the quantification of parameters that define a PQD, e.g., depth and duration of voltage sags.

The corresponding percentages of publications with the term “real-time” in general topics are also shown in Fig. 6a. Monitoring has a substantial percentage because real-time implementation, i.e., algorithms and hardware, is usually discussed in this category. Detection is also important in real-time applications, e.g., for the operation of Dynamic Voltage Restorers (DVR) and protection systems. As the term “analysis” is a broader concept, it also includes some aspects of real-time applications. Classification, assessment, and characterization are more time-consuming tasks and are generally used in offline applications.

Figure 6b presents the results for types of PQDs. The large number of publications related to voltage sags demonstrates that research on the topic is already mature. On the other hand, only a few studies are related to notches, highlighting that research on the topic is incipient and requires further investigation given its relevance to the industrial sector. In the case of studies considering sags and notches at the same time, the terms are searched not only in titles but also in abstracts and keywords. However, few publications are identified. In this context, tools to analyze simultaneously sags and notches can improve the state of research and be useful for the industry.

Regarding the corresponding percentage of publications with the term “real-time”, Fig. 6b shows that more real-time applications have been implemented for voltage sags, e.g., for the operation of fault protection systems. However, the percentage is still low in this category. In the case of the simultaneous analysis of sags and notches, very few applications in real-time have been developed.

5 Literature review

A comprehensive critical literature review of real-time detection and classification of PQDs is developed in this section, based on 179 selected papers (see Sect. 3.2). The categorization of the type of PQDs according to Sect. 3.2 is used along with the development of the literature review to analyze the articles from the perspective of multiple PQDs, voltage sags, and voltage notches.

The category of multiple PQDs includes studies oriented towards the classification of the disturbances. Thus, most articles in this category present methods to distinguish between a set of PQDs identified from voltage waveforms. At the beginning of the research on the detection and classification of PQDs, articles were focused on tools to extract the features to distinguish the types of PQDs. For instance, reference [33] proposes WT as the means to transform and extract the features, while [40] utilizes ST to extract distinctive features. Thereby, in this first period, references [33] and [40], and similar studies in [34,35,36,37], laid the foundations for the classification of multiple PQDs. Afterward, studies were focused on the automatic classification of PQDs, mostly using AI techniques such as ANN [38, 39], and SVM [50]. In this context, other studies included the classification of combined PQDs, e.g., sag with harmonics, flicker with sag, [61]. Recently, real-time performance in the detection and classification of PQDs has also been a concern [11], because of the applications in fault protection systems and root-cause detection and mitigation. Another real-time application is the analysis of the propagation of PQDs using measurements in various nodes of the system.

The second category according to the type of PQDs refers to voltage sags. This set of articles is generally focused on the classification of voltage sags into different categories. For instance, some studies use three-phase classifications of sags [140, 158], which are strongly related to the type of faults that cause the voltage sags. Another aspect analyzed in this category is the classification of sags according to the root causes [133, 147, 149, 150, 180], e.g., faults, the starting of induction motors and heavy loads, and transformer energizing. The characterization of voltage sags, i.e., the quantification of the parameters such as duration, magnitude, starting and ending phase angle, etc., is also performed in some studies [132, 173, 174]. Finally, this set of articles also includes research on the detection of sags for applications such as DVR and protection systems operation [134, 136, 153]. In these applications, real-time detection is of utmost importance.

The third category according to the type of PQD is associated with voltage notches. These studies mainly deal with the detection of notches in voltage signals [189, 190, 194, 205, 206, 209]. The characterization of voltage notches is also performed in some cases. However, this characterization is usually incipient, ambiguous, and misses important features of voltage notches. Moreover, the classification according to types of voltage notches has not been addressed in the literature.

The categorization of articles in the type of disturbances aims to address different topics of interest: the classification of PQD types; the classification, characterization, and detection of voltage sags; and the classification, characterization, and detection of voltage notches. However, studies in the category of multiple PQDs include voltage sags [34,35,36, 38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58, 60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75, 77,78,79,80, 82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119, 121,122,123,124,125,126,127,128,129,130,131], and/or notches [33, 37, 40, 43, 49, 52, 54,55,56, 61, 62, 65,66,67,68, 74, 75, 77, 79, 82, 84, 85, 87, 91, 94,95,96,97,98,99,100, 102, 103, 105,106,107,108,109,110, 112,113,114,115,116, 118, 121, 123, 124, 130], though in such cases, there is no particular focus on those disturbances.

From a different perspective, the process for real-time detection and classification of PQDs can be analyzed from the stages and steps, as shown in the proposed schema in Fig. 7. In this literature review, four major stages are identified, namely: (i) input space, (ii) preprocessing, (iii) feature engineering, (iv) decision space. Moreover, each of the major stages includes the steps to achieve their goals. Eight steps are identified in the comprehensive process. However, not all the steps are always necessary, and hence the proposed schema should be considered flexible. In the context of this review, the steps are defined as follows:

• 1. Input data preparation (i) The step for obtaining data from different sources for designing and training algorithms. Sources of data include laboratory experiments, field measurements, simulations, and synthetic data generated with equations representing the diverse PQDs. This step constitutes the input space stage.

• 2. Data preprocessing (ii) The step for preparing input data to improve efficiency in the subsequent stages. Preprocessing has tasks such as segmentation, normalization, and denoising. Data validation to ensure reliability and correctness of data, i.e., data quality, is also part of the data preprocessing step. However, no extensive details have been described in the reviewed literature regarding data validation. This step accounts for the preprocessing stage.

• 3. Transformation (iii) The process of converting raw data from one domain, e.g., time, frequency, time–frequency, to another. Transformations used in PQD detection and classification include FT, WT, and ST, among others. The transformation step is the first of the feature engineering stage.

• 4. Feature extraction (iii) The computation of numerical indices (from the transformation outputs/coefficients) that are usable for tools in the decision space. These indices usually comprise statistical variables. The feature extraction step is part of the feature engineering stage.

• 5. Feature selection (iii) The process of selecting features and reducing dimensionality to enhance the efficiency in the decision space. Selection can be manual or automatic using optimization methods and dimensionality reduction techniques. The feature selection step provides the output of the feature engineering stage.

• 6. Detection (iv) The identification of states different from the normal operation through thresholds and triggers. Detection is a step in the decision space.

• 7. Classification (iv) The distinction of the type of PQD. Classification is mostly conducted automatically using AI techniques. Classification is a step in the decision space.

• 8. Characterization (of single PQDs) (iv) The calculation of the parameters that characterize a single PQD (e.g., for voltage sags: Point-On-Wave (POW) of initiation/ending, duration, magnitude, phase angle jump). Characterization is a step in the decision space.

Fig. 7

General scheme for real-time detection and classification of PQDs and transversal topics to be dealt with in the review

The literature review is structured and developed according to the eight steps to conduct the real-time detection and classification of PQDs. The categories according to the type of disturbance, i.e., multiple PQDs, voltage sags, and voltage notches are analyzed throughout the comprehensive review. Aspects of the real-time operation are also considered.

5.1 Input data preparation

The process of detection and classification of PQDs starts with the acquisition of signals. This process can be classified into two general groups according to the literature: measurement-based (laboratory, field) and model-based (equations, simulations) techniques. In most cases, the model-based techniques are used for algorithm training and measurement-based techniques for the deployment of the final detector/classifier.

Table 6 summarizes the advantages and drawbacks of each approach in the input data preparation step. The table also gives the references for where the techniques are applied. These references are classified according to the “type of disturbance” categories, i.e., multiple PQDs, sags, and notches. To give an idea of the importance and usage of the techniques, the percentage of the total number of references per category is also presented (see the total number of references per category in Table 4).

Table 6 Summary of input data sources

5.1.1 Field measurements

The acquisition of real signals under real operating conditions is the main goal of this approach. The same specialized equipment used in laboratory measurements is also required in field measurements, but the signals are directly provided by the power grid. The availability of events is therefore limited because of their unpredictability. Disturbances having noise, simultaneity between deviations and variations, and the evolution and propagation of PQDs affected by power demand variations characterize field measurements. These realistic factors require the development of robust algorithms for PQD detection and classification. Usually, measurements aim at catching the highest amount of information but sometimes they are focused on specific PQDs such as voltage sags and notches.

5.1.2 Equations

Instead of specialized equipment for the generation and measurement of PQDs, some can be abstracted into mathematical models that emulate real waveform distortions. In this waveform-level approach, the signals are initially built as ideal sinusoidal stored in computational structures (i.e., arrays and matrices) and then the different kinds of variations and events are superimposed. Although it is highly flexible and easy to change any parameters of these equations, it is rather challenging to exactly replicate a real disturbance and its evolution in time without knowledge of the system features. These equations can be extended to represent different kinds of disturbances, or specific disturbances such as voltage sags and notches.

5.1.3 Simulations

Simulations in specialized software also rely on mathematical models. However, this system-level approach allows not only the reproduction of different PQDs from different operating conditions of power systems, but also the assessment of the interactions and evolution of disturbances in both time and space. Therefore, simulations can be used to replicate operational conditions that produce a certain type of disturbance, modify those conditions to stress the system response, combine different disturbances in a controlled environment, including power system variations to forecast the future behavior of PQDs, etc. The simulation results are strongly dependent on how closely the models represent the real system, so a validation stage between real measurements and simulated results is necessary before forecasting results for other operating scenarios.

5.2 Data preprocessing

After the acquisition of PQDs, some techniques are used to handle the data before further processing. These techniques are optional and should not modify the useful information contained in the acquired signals. The additional hardware and/or software required at this stage may improve the performance of detection and classification algorithms in subsequent stages. Table 7 summarizes the advantages and drawbacks of these steps, indicates references where applications are observed, and reports usage percentages according to the “type of disturbance” categories.

Table 7 Summary of data preprocessing techniques

5.2.1 Segmentation

This technique is used for the isolation in the time domain of the most relevant sections of the acquired signals, especially when events (see Table 1) are to be analyzed. The computational burden of subsequent processing stages may be reduced when segmentation is correctly applied since non-relevant data is discarded. However, the decision on relevance can vary when different disturbances coexist at the same measured signals. Therefore, a set of thresholds are generally used when dealing with the detection and classification of multiple PQDs. Moreover, segmentation is less frequently used for specific disturbances, as can be seen in the literature for voltage sags and notches in Table 7.

5.2.2 Denoising/filtering

Similar to the segmentation technique, the filtering or denoising technique aims at discarding the non-relevant information in the acquired signals. In this case, the isolation is performed over the spectral components of the signals. For this reason, additional hardware (analog filters) and/or processing steps (digital filters) may be required. This technique is most useful when the bandwidth of the PQD of interest is previously known or when only emissions at certain frequencies are of interest (e.g., voltage and currents at the fundamental frequency only). Compared to segmentation and normalization techniques, denoising has been less used in the context of PQD detection and classification.

5.2.3 Normalization

Normalization is the most frequently used data preprocessing technique for multiple PQDs and voltage sags but is used less frequently in specific detection and classification of notches (see Table 7). Similar to segmentation and denoising, the normalization step reduces computational burden by avoiding the calculation of large numbers. This is achieved by the division of all quantities against a numerical base, i.e., the rated or peak value of signals. However, the scale is lost and the distinction between relevant and non-relevant amplitudes will depend on the relative choice of the numerical base.

5.3 Transformation

Following the acquisition of data and the eventual application of some preprocessing techniques, transformation is traditionally the most used stage for signal processing purposes. This stage aims at using a different representation of original data, usually by changing the analysis domain, to unveil hidden features and patterns.

As shown in Fig. 8, most of the transformations used in PQD detection and classification can be grouped into time, frequency, and time–frequency domains. Other miscellaneous transformation techniques are widely used in other domains (speech recognition, arrhythmia classification, denoising, image compression, etc.) and have been adopted by researchers to decompose and/or cluster voltage and current data from power systems for detection and classification of PQDs. The groups of transformation techniques applied to the detection and classification of PQDs are depicted in Fig. 8, where time-domain transformations are further divided into parametric and non-parametric techniques. Further descriptions of each of the time-, frequency-, and time–frequency-domain techniques are provided in the following sections.

Fig. 8

Taxonomy of transformation techniques

The most relevant advantages and disadvantages of the transformation techniques are presented in Table 8. The table also reports references of applications and the respective percentages per category according to the type of PQD.

Table 8 Summary of transformation techniques

5.3.1 Time domain

Time-domain transformations are widely used to track the evolution of monitored signal features through time. They are commonly used to analyze PQ events like interruptions, transients, rapid voltage changes, swells, and sags because they are in principle unpredictable and non-periodic phenomena (see Table 1). Nevertheless, PQ deviations like notches are also processed in the time domain since their spectral signature is spread over a wide frequency range. Time-domain transformations shown in Fig. 8 can be subdivided into non-parametric, e.g., tracking of symmetrical components, DQ0, Time-time Transform (TT), Clarke transform in complex domain Space Phasor Model (SPM), Phase Space Reconstruction (PSR), MM, etc., and parametric, e.g., KF, Phase-Locked Loop (PLL), Adaptive Filters (AF), etc. Non-parametric transformations decompose the original measured signals into components that clearly show how the PQDs are progressing in time, whilst parametric transformations use assumptions about the statistical distribution of the population from which the samples (signals) were taken. Parametric transformations are mostly used to track and statistically estimate magnitudes, phase angles, frequency, etc. Although transformations relying on phasor theory, like symmetrical components, belong in principle to frequency domain abstraction, the evolutions of the computed amplitudes and phase angles are usually analyzed in the time domain.

5.3.2 Frequency domain

The frequency-domain transformations are essentially applied to steady-state signals using FT. This transformation decomposes distorted signals into a summation of pure sinusoids having different frequencies. The Fast FT (FFT) is a widely known technique for Discrete FT (DFT) computation. The computation of DFT is carried out as [66]:

$$V^{n} \left[ k \right] = \sum\limits_{i = 0}^{N - 1} {v\left[ {i + \left( {n - 1} \right) \cdot N} \right]} \cdot \exp \left[ {{{ - j\left( {2\pi ki} \right)} \mathord{\left/ {\vphantom {{ - j\left( {2\pi ki} \right)} N}} \right. \kern-0pt} N}} \right]$$

(1)

where N is the number of samples in one cycle, n is the order number of the signal cycles, and n = 1, 2, …, 10, which gives the absolute value of the argument. Vⁿ[k] is the DFT for the samples contained in the nth cycle, v[i] represents the sampled input signal, i = 0, 1, 2, …, L–1, with L being the length of the signal.

For the detection of multiple PQDs, FFT has been used mostly for the computation of fundamental or specific harmonic amplitude, phase angle shift between harmonic components, RMS values, and total harmonic distortion [66, 89, 90, 97, 102]. Although FFT and DFT may yield inaccurate results for non-stationary signals, some studies have adopted this transformation for voltage sag detection, while some have used FFT for notch analysis [188] because the spectral components of this disturbance are rather spread over the whole frequency range. However, FFT has also been used in combination with WT [61, 80, 189], 192, 196] and Mode Decomposition (MD) [99] techniques to cope with non-stationary PQDs.

5.3.3 Time–frequency domain

Transformations classified in the time–frequency domain keep most of the advantages of time and frequency techniques. These techniques can provide time and frequency information at the same time, which helps improve the accuracy of PQD detection and classification algorithms.

STFT is the computation of FT over a section of the signal assumed to be stationary. STFT has been used for detection of multiple PQDs [35, 102] and voltage sags [137, 160].

WT, also known as a multiresolution analysis, decomposes the original signal into various scales of a short-term waveform called the “mother wavelet”. Discrete WT (DWT) is the discrete realization of WT and has been widely used for detection and classification of multiple PDQs as the main transformation technique [33, 34, 37,38,39, 42, 44,45,46, 48, 51, 53, 55, 59, 60, 62,63,64, 69, 76, 83, 95, 101, 110, 123] or combined with other techniques [35, 36, 41, 61, 73, 80, 102, 117, 126]. It has also been used as the main transformation technique for the detection and classification of voltage sags [133, 142, 146, 154, 168, 169] and notches [187, 190, 191, 193, 194, 197,198,199, 204, 208], or combined with other techniques for voltage sags [160, 181, 186] and notches [189, 192, 196, 202]. DWT is given by [33]:

$$DWT_{\psi } x\left( {m,n} \right) = \int_{ - \infty }^{\infty } {x\left( t \right)\psi_{m,n}^{*} \left( t \right)dt}$$

(2)

where ψ_m,n(t) is the mother wavelet.

ST is a hybrid technique that includes a phase correction to the WT and a variable Gaussian window to STFT to get a combination of these two techniques. ST can be calculated by multiplying the continuous WT with a phase factor as [40]:

$$S\left( {\tau ,f} \right) = \exp \left( {i2\pi f\tau } \right) \cdot W\left( {\tau a} \right)$$

(3)

ST has been used for the detection and classification of multiple PQDs as the main transformation technique [40, 43, 49, 52, 54, 57, 68, 70,71,72, 75, 77, 78, 81, 85, 92, 94, 96, 104, 105, 118, 120, 125] or in combination with a spline wavelet [41], TT [65, 109, 114], VMD [88], WT [117], and others [102]. Similarly, it has been used for the assessment of sags as the main technique [143, 149, 178] or combined with VMD [176] and FT [184].

Finally, GT, Hilbert Transform (HT), and Chirplet Transform (CT) are other time–frequency techniques used for the detection and classification of PQDs. GT is mostly known as an accurate tool for phasor estimation and has been used as a measurement tool for the analysis of events (short-term deviations). It has been mainly applied in combination with the Wigner distribution function, resulting in the so-called Gabor-Wigner Transform (GWT) [58], and with time–frequency representation. HT of a real-valued time-domain signal produces an orthogonal, real-valued time-domain signal that is 90° (π/2 radians) shifted from the original. This technique has been widely applied alongside Empirical Mode Decomposition (EMD), resulting in the so-called HHT. Researchers have applied HHT for the detection and classification of multiple PQDs [56, 73, 79, 84, 102, 107, 113], and specifically for voltage sags [147, 150, 173], but not yet to the specific assessments of voltage notches. CT can be seen as the generalization of FT, STFT, and WT. Analogous to the mother wavelets in WT, chirplets are usually generated from a single mother chirplet which is a windowing function. It has been applied for the assessment of multiple PQDs [98] and voltage notches [202]. Voltage sags have not yet been specifically addressed nor characterized with CT.

5.3.4 Miscellaneous

Transformation techniques widely used in other domains have been adopted by researchers for the detection and classification of PQDs. The MD technique is mainly composed of EMD [86, 93, 129] and VMD [88, 99, 100, 124, 130] for the assessment of multiple PQDs, and also for the specific assessment of voltage sags [176]. EMD takes the linear or non-linear input signal and iteratively decomposes it into a series of smaller components known as Intrinsic Mode Functions (IMF). VMD is based on a constrained variational optimization problem and is a non-recursively adaptive technique that decomposes a linear or non-linear input signal into a finite number of sub-signals or modes having specific sparsity properties (compactly band-limited IMF). There are also other techniques adapted for PQD detection and classification such as Sparse Signal Decomposition (SSD) on an overcomplete hybrid dictionary matrix [87], Singular Spectrum Analysis (SSA) along with Curvelet technique [103], 2D image techniques (gray-scale images) [112, 179, 181], numerical pencil [134], matrix pencil [153], and Goertzel method [166].

5.3.5 Quantitative analysis of transformation techniques

Figure 9 depicts how transformation techniques are distributed according to the analysis domain, how they are used (main technique or in combination with other techniques), type of disturbance, and real-time applications. Specifically, Figs. 9a and b rank respectively the most popular transformation techniques for detection and classification of PQDs, according to the absolute and relative number of articles found in this literature review. Techniques classified into the time–frequency (52%) and time domains (19.7%) represent 71.7% of the total number of transformation techniques, whereas WT and ST, which are time–frequency domain techniques, represent 44.1% of the total transformation techniques. WT is used either as the main technique or in combination with other techniques, such as FFT, ST, and other time-domain techniques.

Fig. 9

Distribution of papers in the identified transformation techniques. a The number of papers, b percentages and number of papers that use a combination of techniques, c percentages and number of papers from the time perspective, d the type of disturbance perspective, and e the real-time perspective

The evolution of transformation techniques is described in Fig. 9c. WT has been intensively used as the main time–frequency technique for the detection and classification of PQDs since 1996 [33], and from 2011 onwards it has been used in combination with other techniques. ST and other time–frequency techniques have increased their participation in PQDs detection and classification, as well as time-domain techniques such as SPM, PSR, and (extended) KF/AF. Nevertheless, the combination of several techniques has seen increased interest from researchers, especially in the last six years.

In terms of the PQDs to be detected and classified, Fig. 9d shows that WT, time-domain techniques, and the combination of several techniques have been used for the assessment of multiple PQDs as well as voltage sags and notches. Although WT has been the preferred technique for the assessment of voltage notches, time-domain techniques are becoming relevant for assessing this type of disturbance. The trend regarding transformation techniques is to develop one technique, or a combination of several, that can be used for the accurate detection and classification of the highest number of PQDs (variations and events, see Table 1).

Figure 9e shows that WT, ST, and time-domain techniques (SPM, PSR, and KF/AF) are used for real-time detection. Real-time detection and classification are also performed by the combination of transformation techniques such as WT and FFT, WT and ST, etc.

5.4 Feature extraction

Feature extraction aims to reduce the amount of data from the transformation stage that will be processed for the detection and classification of PQDs. A set of statistical, time series, spectral and image features can be used for this purpose. A set of features describing one PQD, e.g., variations, may not be suitable for describing another type of PQD, e.g., events. Therefore, establishing a set of comprehensive, robust, and accurate features that allow the detection of different PQDs is one of the most challenging tasks in the process.

Figure 10 depicts the categories into which the feature extraction stage can be further divided. The main advantages and disadvantages are listed in Table 9, which also includes references with applications and the respective percentage of usage according to the type of disturbance.

Fig. 10

Taxonomy of features for detection and classification

Table 9 Summary of feature extraction techniques

5.4.1 Statistical features

Mean and median are the set of most used statistical features for central tendency. Arithmetic mean (also known as arithmetic average) is a central tendency measure for a finite number of values from an observation process (sampling). This is calculated as the sum of all values divided into the amount of data and is a relatively simple way of computation (low computational cost) but sensitive to outliers (data with atypical values). This metric is widely used for feature extraction in multiple PQDs, and specifically for voltage sags and notches (see Table 9). Less sensitive to outliers but with a compulsory “ordering” process, the median is the other central tendency widely used for feature extraction. When dealing with a large enough dataset, the underlying population distribution may be assumed as normal (Gaussian). Therefore, similar values for both mean and median are obtained. However, many processes do not follow a normal distribution, and thus median may be a more accurate measure of central tendency than the mean.

In contrast to central tendency, a measure of dispersion is achieved by many different indices. Maximum and minimum values are easy-to-compute metrics that give general information about the analyzed dataset. The interquartile range is a descriptive metric defined as the difference between the 75th and 25th percentiles and needs an ordering process that may be challenging for online applications in embedded hardware. Deviation metrics in the form of maximum deviation, standard deviation, mean absolute deviation and median absolute deviation compute the distance between the observed value of a variable and a central tendency metric. Variance and Higher-Order Statistics (HOS) such as skewness and kurtosis aim at describing the shape of the underlying probability distribution function. These metrics are widely used for detection and classification of multiple PDQs, voltage sags and voltage notches (see Table 9).

5.4.2 Time series features

Time series are sequences of data points ordered in the time domain. The sampling process is usually performed at a fixed frequency, and therefore the time between successive samples is theoretically the same. In the context of electrical engineering, time-varying electrical variables such as voltages and currents are sampled through analog–digital converters, which convert the real-life analog signals into discrete signals. The cyclic nature of AC systems is defined by a cycle, which is the time that a signal repeats its values in the time domain. Therefore, it is possible to characterize a signal by extracting sample-based or cycle-based features.

Sample-based features take advantage of the evolution of discretized signals in the time domain. The features are the extracted samples and therefore very detailed information on signal evolution can be retrieved. However, a large amount of data can result from this stage if a high sampling frequency is used. Sample-based feature extraction techniques such as instantaneous values, phase angle and momentary deviation (Euclidean distance) are used for multiple PQDs, voltage sags and notches according to the references indicated in Table 9.

Cycle-based features are focused on the evolution of periodic signals over time. In this sense, the features are extracted in multiples of one cycle of the main signal and therefore the amount of processed data is much less than that using sampled-based features. However, information on sub-cycle PQDs like notches, spikes and transients is no longer available in this approach. RMS value, crest and form factors, energy, entropy, correlation, and signal-to-noise ratio are usual metrics computed from a cycle-based approach. These metrics are used for feature extraction of multiple PQDs, voltage sags and notches (see Table 9).

5.4.3 Spectral features

Spectral features are a set of indices that naturally result after the use of frequency or time–frequency domain transformation techniques. In the context of power systems, the fundamental frequency is the nominal frequency at which most of the electric power is generated and transmitted (theoretically 50 Hz or 60 Hz). In contrast, spectral distortion is the result of the nonlinear, nonconstant behavior of electrical equipment that indicates a deviation from the ideal (reference) pure sinusoidal signals. In the context of power systems, spectral distortion can be generally classified into the harmonic range (below 2 kHz), the so-called supraharmonic range (between 2 and 150 kHz) or the high-frequency range (above 150 kHz). Fundamental frequency and harmonic distortion are used for the detection and classification of multiple PQDs, voltage sags and notches (see Table 9). There is a special case of harmonic distortion, called Distortion Bands, where the distortion is computed in other ranges different from harmonic, supraharmonic or high-frequency ranges [197, 202].

5.4.4 Image features

Image features are mostly related to 2D functions and/or representations of PQDs. Taking advantage of the steady-state cyclic variation of voltage and current in power systems, some transformation techniques (SPM, PSR, instantaneous symmetrical components, etc.) describe these signals as phase vectors in the complex plane (phasors). From this, the most popular 2D feature extraction technique relies on Ellipse features since the resultant circumferences can give useful information about the features of the variations and events listed in Table 1. Ellipse features have been used for specific detection and classification of voltage sags [151,152,153, 157, 158, 163, 172]. Other features taken from the 2D representation of PQDs are shape features and factors (center of mass, eccentricity, convexity, centroid distance, chord length, etc.) [140, 169], binary image [50, 112], and image matrix [111], among others.

5.5 Feature selection

The step of feature selection involves identifying as few characteristics as possible to obtain enough information that can yield suitable results in the decision space stages (detection, classification, and/or characterization of PQDs). Hence, the ways of selection and the selected features depend on what suitable results mean in the context of each study. The feature selection also reduces the computational burden in the decision space and usually leads to more accurate results.

Different categories have been proposed for feature selection techniques in the literature. For instance, reference [76] proposes three main categories, namely, filtering, wrapper, and embedded, which are related to the level of dependence of the feature selection on the decision (learning) algorithms. Thus, filtering approaches are very independent, and embedded approaches intertwine the selection and decision algorithms. In this literature review, three groups of methods are identified, including handcrafted, optimization methods, and dimensionality reduction algorithms.

Table 10 summarizes the advantages and drawbacks of the most relevant feature selection techniques identified in the literature review according to the proposed classification. The table also reports the references where the methods are applied according to the type of disturbance categories and presents the usage percentages.

Table 10 Summary of feature selection techniques

It is worth mentioning that deep learning techniques perform an automatic process of feature selection. It is conducted in the first layers of the classifiers where the best features for the decision space are automatically selected. Therefore, no category is included in this section for this type of tool because no external intervention is required.

5.5.1 Handcrafted/empirical

In handcrafted feature selection, a detailed manual analysis is performed on the extracted features to observe the differences according to the type of PQD. For instance, this approach is observed in [33] where coefficients resulting from the WT and multiresolution analysis are analyzed for sample signals with different PQDs. The study in [33] and many others, e.g., [34,35,36,37,38,39,40,41], show how the analysis can be mostly supported by visual inspection of signals (time-domain waveforms) and extracted features (time-domain indices). Most of the handcrafted feature selections observed in the literature apply a contextual approach where a physical meaning is given to the extracted features. Some examples of contextual feature selection are [33,34,35,36,37,38,39,40,41]. The literature also shows that other empirical approaches such as sequential forward selection, sequential backward selection, and random mutation have been mainly used for the detection and classification of multiple PQDs. In these empirical approaches, a set of features are obtained beforehand based on expert knowledge or using similar features to previous studies. Then, in forward selection methods, e.g., [60, 88, 89, 91, 96, 102, 112], features from the established set are included one at a time, and the performance of the classifier is verified. The process is repeated until the performance has no apparent improvement. Conversely, backward selection, e.g., [88], uses the complete set of features and removes one at a time until the desired trade-off between accuracy and computational performance is reached. On the other hand, random mutation tests random subsets of features and selects the one with the best performance [57, 62, 64, 71, 117].

Those studies from the literature focused on voltage sags mainly use contextual feature selection. For instance, reference [132] extracts directly the features from the voltage waveforms according to standard definitions, i.e., initial phase angle shift, recovery period, voltage change. In other cases, coefficients from transforms, e.g., WT [133] and ST [143], are analyzed contextually according to their correspondence to the standard parameters that characterize voltage sags, i.e., magnitude, duration, etc. A combination of forward and backward selections with GA to select the best features to identify voltage sag source location (upstream or downstream) is proposed in [185].

In the studies focused on voltage notches, contextual feature selection is dominant for handcrafted methods [187,188,189,190,191,192,193,194,195,196,197,198,199,200,201,202,203,204,205,206,207,208,209,210]. In this case, selected contextual features are analyzed visually from the coefficients of WT.

5.5.2 Optimization methods

Optimization methods have also been used for feature selection purposes in PQD detection and classification. According to [212], a mathematical optimization method which consists of finding the best possible solution by changing variables that can be controlled, is often subject to constraints. Optimization methods can be classified as deterministic (exact) or stochastic (approximate), and stochastic methods can be further divided into heuristic and metaheuristic. Most methods used for PQD detection and classification are metaheuristic. Metaheuristic methods include exploratory search methods such as GA, and swarm optimization algorithms such as particle swarm optimization. For instance, GA is used in [55, 102, 109] to find the optimal set of features to classify multiple PQDs with high accuracy. This is based on the mechanics of selection and survival of the fittest, and consists of three operations including reproduction, crossover, and mutation [55]. The studies in [55, 102, 109, 127] show that GA maintains or improves the classification accuracy while it decreases the computational burden (reducing the number of features). A variation of GA is presented in [109], where a fast and elitist nondominated sorting is used to generate Pareto-optimal solutions. This modification offers better speed, solution spread, and convergence. The artificial bee colony optimization algorithm is used to improve the performance of a multiple PQDs classifier in [95], while the artificial bee colony is used in [123] in combination with particle swarm optimization to improve the accuracy of a PNN in PQD classification. The artificial bee colony algorithm is a swarm intelligence optimization technique where different types of bees apply strategies for finding the best sources of food (solutions). This algorithm uses three groups of bees, namely, employed bees, onlooker bees, and scout bees. Employed bees search for food in specific sources and share the information to onlooker bees, whereas scout bees search for new sources of food. This optimization method provides an optimal subset of features with fast convergence [95]. Other swarm colony algorithms have also been used for feature selection. For instance, ant colony optimization [114] is inspired by the foraging behavior of ants, and offers high accuracy in the classification of multiple PQDs and a faster solution to an optimal feature subset than other studies.

Feature selection based on optimization techniques is also used in studies focused on voltage sags. The study in [168] uses ant lion optimization to improve performance in classifying the underlying causes of voltage sags. This optimization technique mimics how ant lions hunt and consume their prey and provides advantages such as good population diversity, storing good solutions, exploitation, exploration, and flexibility [168]. Similarly, a teaching–learning-based optimization technique is used in [177]. The teaching–learning-based method is a population algorithm that mimics the influence of a teacher on the output of learners in a class. Results in [177] demonstrate good accuracy in classifying causes of voltage sags in noisy signals with a reduced subset of optimal features. GA is used in [185] for voltage sag source location classification.

5.5.3 Dimensionality reduction

In machine learning, dimensionality reduction refers to decreasing the number of input variables in a model, i.e., selecting a subset of the original variables (features) and converting the data to a lower-dimensional space [213]. Dimensionality reduction may be useful for efficiently storing and processing data. Extensive techniques are used in machine learning for dimensionality reduction. Principal Component Analysis (PCA) is one of the most used techniques. PCA is a multivariate technique that analyzes a data table with several variables and extracts the important information to represent the table as a reduced set of new orthogonal variables called principal components [214]. PCA is used for feature selection in multiple PQD classification in [62], while multiway PCA is used for the classification of voltage sags in [141]. Likewise, Independent Component Analysis (ICA) has been also used in applications of voltage sag classification [162, 177]. Other dimensionality reduction techniques observed in the literature for applications of multiple PQD classification include information gain measurement [42], Fischer linear discriminant analysis [100], k-means based apriori [76], Gini index-based threshold for selection of features [75, 98, 110, 112], and maximum relevance minimum redundancy [102]. In the case of voltage sag classification, k-means-singular value decomposition is also used [170], whereas decision trees and the Gini index are used in [211] for binary classification of voltage notches (non-notch/notch).

5.6 Detection

In this review, detection refers to the identification of states different from the ideal conditions of voltage and current waveforms (signals with no disturbance) through thresholds, triggers, and other techniques.

According to the above definition, detection may overlap in some cases with the step of classification because the classification process usually includes a category for ideal conditions of voltage and current waveforms. This situation mainly occurs in the classification of multiple PQDs, e.g., [118, 121]. In such cases, the detection should be considered as part of the classification algorithm and is mostly used in offline applications, where techniques are applied to stored signals. For real-time applications, given that no threshold or trigger occurs, the complete algorithms are constantly executed with a certain periodicity and the process requires high computational performance and storage capacity. There are exceptions in the classification of multiple PQDs, where detection is implemented as a previous and independent step of the process using techniques such as AF [36, 115], sine wave inference [64], and Euclidean square distance [106].

In the case of voltage sag classification, detection is mostly achieved as a previous independent step using techniques such as empirical/handcrafted definition of thresholds [132, 133, 137, 139,140,141, 148, 151,152,153, 163, 172], statistical-based sequential method [135], and ICA [177].

Detection has been described as an important step in combination with classification in the decision space. However, detection as the only aim of the decision space is also of interest for some applications such as the operation of DVR [134, 184], voltage sag compensators [136, 145], and protection systems [146]. In these applications, the process includes some characterization of single PQDs. For voltage sags, techniques used for detection include adaptive notch filter [136], KF [138, 156, 161], integrator model [155], harmonic footprint [160], ICA [162], and Goertzel algorithm [166]. When the interest of the study is the characterization of voltage notches, detection has been conducted using the Teager energy operator and threshold algorithm [201], and the Euclidean norm [206, 209].

5.7 Classification

The main purpose of classification is to categorize the PQDs observed in voltage and/or current signals according to the types of deviations from the ideal waveforms. For instance, Table 1 presents some of the deviation types that identify the categories of PQDs. Among the categories of PQDs there are voltage sags and swells, harmonic distortion, transients and voltage notches, flicker, imbalance, etc. Classification is mostly conducted according to those categories. However, some approaches also categorize the phenomena according to the root causes of the PQD. For instance, reference [81] presents a categorization according to different causes, namely, fault, self-extinguishing fault, line energizing, non-fault interruption, and transformer energizing. Similarly, voltage sags are usually classified according to the main underlying causes, i.e., faults, motor starting, and transformer energizing [175,176,177,178,179,180].

Several techniques for the classification of PQDs have been identified from the analysis of the selected literature. A taxonomy of these techniques is proposed in Fig. 11, which shows three major categories, namely, handcrafted, probabilistic, and AI-based methods. The latter category is divided into various subcategories as it is the most widely used in the literature for the detection and classification of PQDs. Especially, machine learning tools have been widely used for the task of classification. Machine learning is a field of AI that focuses on the development of algorithms to make computer systems able to learn from data. Machine learning can be further divided into supervised learning, i.e., when algorithms need labeled data for training, and unsupervised learning, when no labels are required but are automatically identified by algorithms. In this literature review, only supervised learning algorithms are analyzed because they are the most used ones.

Fig. 11

Taxonomy of methods for classification of PQDs

Table 11 reports the advantages and drawbacks of classification techniques, references of applications and percentage of usage according to the type of disturbance.

Table 11 Summary of classification techniques

5.7.1 Handcrafted/empirical

In handcrafted classification, thresholds to identify the categories of PQDs are defined by the observation of the extracted features during diverse experiments or by using expert knowledge on the physical interpretation of the phenomena. For instance, in [33] the coefficients of the WT are used to detect a variety of PQDs obtained from field measurements. Other approaches use handcrafted classification of voltage sags, e.g., reference [137] presents a method to classify sags according to their root causes including faults, motor starting, and transformer energizing based on thresholds for the STFT. Reference [148] classifies voltage sags according to the type of fault, based on defined ranges for indices calculated with symmetrical components. Other methods [153, 163, 172], analyze visually the ellipses generated through the SPM in the complex plane to define manually the ranges of the ellipse parameters for the types of sags.

5.7.2 Probabilistic

The probability of a signal containing a certain PQD is determined from the probability density functions of the extracted features associated with the disturbance. Examples of probabilistic methods include Parseval's theorem [34], the maximum likelihood [36], and the definition of ranges for statistical variables [40, 41]. Probabilistic methods are also used to identify the categories for classifying sags according to root causes and location (upstream or downstream). These methods include the singularity detection theory [133], the statistical-based sequential method [135], and the energy-based method [139].

5.7.3 Shallow artificial neural networks (ANNs)

ANNs are computational models of reasoning inspired by the human brain [215], and comprise a set of processors (neurons) interconnected through weights passing signals from one neuron to another. An ANN can model complex nonlinear functions using extensive simple operations. Shallow ANNs are formed by an input layer, one or two hidden layers, and an output layer. ANNs are typically used in classification problems where each neuron of the output layer represents a category and is activated according to the respective inputs. In the problem of multiple PQDs and voltage sag classification, several types of shallow ANNs have been used taking advantage of their flexibility and adaptability to problems where labels are well identified, for instance, the learning vector quantization [37,38,39], probabilistic neural network [44, 52, 56, 65, 71, 84, 95, 102, 123, 147, 168], self-organizing learning array [46], radial basis function [47, 83], multilayer perceptron [48, 53, 57, 65, 89, 93, 102, 168, 177, 178], adaptive linear network [82], feedforward [82, 85, 102], backpropagation [90], random vector functional link [113], and modular ANN [143]. Most recently, a learning algorithm known as ELM has been gaining popularity because of its remarkable efficiency. ELM randomly chooses hidden nodes and analytically determines the output weights of a single-layer feedforward network [216]. Examples of ELM applications are [81, 94, 100, 107, 108, 169, 176, 178].

5.7.4 Deep artificial neural networks (ANNs)

Deep learning has emerged as a new machine learning paradigm where deep ANNs are composed of multiple processing layers to learn representations of data with multiple levels of abstraction [217]. This paradigm can dramatically improve the automatic classification abilities in diverse areas such as speech recognition, image processing, and detection and classification of PQDs. A remarkable improvement provided by deep learning, and the main motivation to apply these types of algorithms to PQD detection and classification, is the ability of models to automatically extract the best set of features from raw data to conduct classification. Convolutional Neural Networks (CNN) have been widely used in multiple PQDs and voltage sag classification [103, 111, 116, 119, 121, 125, 130, 131, 158, 164, 167, 171, 179, 183]. Other deep learning models have been particularly used in the classification of voltage sags according to the root causes, e.g., deep feedforward ANNs [124], Long Short-Term Memory (LSTM) [129, 159, 180], Deep Belief Networks (DBN) [165, 175], and independently recurrent neural networks [182].

5.7.5 Decision trees

Decision trees are knowledge-based systems obtained by inductive inference from examples [218]. Then, these systems are driven by the explicit representation of knowledge. Simple in application, decision trees allow for high efficiency which is essential for real-time applications. Moreover, these models provide good physical interpretation of the phenomena. These advantages have motivated widespread use of decision trees, especially in the classification of multiple PQDs, though with lower usage in the classification of voltage sags and notches. Simple decision trees are used in the classification of multiple PQDs [42, 49, 66, 70, 75, 77, 85,86,87, 89, 96,97,98,99, 101, 102, 109, 114, 117, 118, 128]. Some variants such as random forest [102, 105, 112, 126] and bagging predictor [97, 98] and rule-based classifiers [72, 80, 91, 122], have been also used in the literature for multiple PQD classification. Rule-based classifiers are also used in [140, 151, 152] for voltage sag classification and in [200] for voltage notches, whereas decision trees are used in ensemble models as weak classifiers in combination with other methods [185] for voltage sag classification.

5.7.6 Support vector machines (SVM)

SVM are robust supervised learning models applied in classification and regression problems. The basic idea behind SVM is to maximize the gap between different classes [219]. Based on this feature, SVM can be trained using a reduced number of examples. This makes this tool promising in cases where extensive training data is not available as in some applications of PQD detection and classification. SVM have been widely used for detection and classification of PQDs and voltage sags, while having lower usage for voltage notches (see Table 11). Variants of SVM such as multiclass SVM [51, 110, 115], least square SVM [76, 101, 170], rank SVM [86], and directed acyclic graph SVM [92, 127] have been also used.

SVM are used for notch identification in [211], where a classifier, i.e., SVM, is used to obtain a binary categorization of voltage signals, namely, non-notch or notch.

5.7.7 k-nearest neighbor (k-NN)

The k-Nearest Neighbor (k-NN) algorithm finds a group of k objects in the training set that are closest to the test object, and bases the assignment of a label on the predominance of a particular class in this neighborhood [220]. Given the ease of implementation of the model, it has been applied in the classification of multiple PQDs and voltage sags [55, 65, 102, 114, 171]. Simplicity of the method has been the main motivation for application of k-NN in PQD detection and classification. However, it has been mainly used in combination with other methods because of its reported drawbacks such as the reduced accuracy and high sensitivity to the constant k. For instance, in [185], k-NN is used as a weak classifier in an ensemble model in combination with decision trees.

5.7.8 Fuzzy logic

Fuzzy logic approaches are based on the sound theoretical foundations of fuzzy sets [221]. The basic idea of fuzzy logic lies in the definition of true fuzzy logic values of variables between 0 and 1. Fuzzy logic allows a closer representation of human reasoning where the information usually has a level of uncertainty. In some applications of PQD detection and classification, data for training have high uncertainties and thus, in those cases, fuzzy logic represents an alternative approach. For automatic classification of PQDs, fuzzy logic has been used from the beginning of research on the topic. Some examples include [43, 48, 61, 132]. FES have also been implemented [67, 189], as well as extended fuzzy reasoning [45], and fuzzy C-means [54, 68, 96, 120].

5.7.9 Quantitative analysis of classification techniques

Figure 12 presents a quantitative analysis of the selected literature according to the classification techniques described above. In Fig. 12a, the absolute number of individual appearances of each technique is shown, i.e., the number of papers where only one technique (or set of techniques in the same category) is used for classification. It also presents the number of appearances of the techniques in combination with others and the total number of papers where classification is achieved with a combination of techniques from different categories. As seen, shallow ANNs are the most popular models in the reviewed literature, followed by decision trees, SVM, deep ANN, fuzzy logic, probabilistic and handcrafted method, and k-NN. Also, a significant number of publications (19) implement combined methods for classification. The number of papers that do not perform classification is 46, and hence, a total of 133 papers are categorized in techniques for the classification of PQDs.

Fig. 12

Distribution of papers in the identified classification techniques. a The number of papers, b percentages and number of papers that use a combination of techniques, c percentages and number of papers from the time perspective, d the type of disturbance perspective, and e the real-time perspective

Figure 12b shows the corresponding percentages of classification techniques according to the individual appearances. The distribution of combined techniques is also reported in absolute values. In this case, shallow ANNs are the most popular in combinations. For instance, shallow ANNs are used in combination with decision trees, SVM, fuzzy logic, probabilistic methods, and k-NN. Moreover, some methods include the combination of three and four types of classification approaches.

A report on the distribution of classification techniques from different perspectives is included in Fig. 12. Figure 12c shows the distribution of techniques in time divided into the periods defined for the categorization of papers, i.e., ≤ 2005, 2006–2010, 2011–2015, and 2016–2021. This indicates the absolute value of individual appearances of techniques in each period and the corresponding percentage. For instance, shallow ANNs present an increasing trend, where most of the papers were published in the period from 2016 to 2021. Similar behavior is also observed in decision trees with a more pronounced trend. An apparent observation is that all deep ANN approaches had been published in the last period, in agreement with the proliferation of deep learning. The combination of techniques exhibits a constantly increasing trend. Conversely, the use of probabilistic and fuzzy logic methods has been decreasing.

Figure 12d shows the distribution of papers according to the type of PQD, i.e., multiple PQDs, voltage sags, and voltage notches. Most papers performing classification are in the category of multiple PQDs as expected because the types of PQDs are distinguished in these cases. Classification is also used for voltage sags as they can be categorized according to the root causes. No categories are identified for voltage notches; therefore, classification is incipient in this case. It is also observed that deep ANN models are evenly used for multiple PQDs and voltage sag classification.

In Fig. 12e, the distribution of techniques according to the real-time application is presented. For instance, decision trees are more used in real-time approaches, which is attributable to the simple operation of algorithms. Similarly, the combination of techniques is mostly used for real-time applications, where the methods take advantage of each technique in terms of efficiency. Conversely, deep ANNs, fuzzy logic, and probabilistic methods are less used for real-time applications because of the cost of computation during the operation of algorithms.

5.8 Characterization

The characterization of single PQDs refers to the quantification of features that distinguish the disturbance. The single-PQD characterization referred to in this section usually differs from the characterization conducted in the stages described previously. The feature extraction stage presented in Sect. 5.4 explains methods to obtain statistical, time-series, spectral, and image-based characteristics that are useful as inputs for the classification techniques to identify the category of PQD. However, these characteristics are not necessarily representative of the physical features that describe the phenomena according to deviations and standard limits. For instance, in [168], the extracted features include the mean, variance, kurtosis, skewness, entropy, etc. of the coefficients from the WT to classify voltage sags according to their causes. Thereby, these features are not explicitly related to magnitude, duration, POW characteristics, and phase angle jump, which are the physical features that identify voltage sags. The abovementioned difference occurs because the purpose of feature extraction is to obtain characteristics that allow the most accurate and efficient results in the classification stage. Conversely, in the single-PQD characterization referred to in this section, which is mainly useful for assessing the severity and impact of the disturbance, the physical representation and interpretation of the phenomena is the most important aspect.

This review gives a particular focus to voltage sags and notches. Hence, only the aspects observed in the selected literature related to their characterization are briefly described in the following sections.

Table 12 summarizes the parameters used to characterize voltage sags and notches. A brief definition of each parameter is provided, and its relevance is highlighted. References of the selected literature that analyze each of the parameters are also indicated, as well as the usage percentage.

Table 12 Summary of parameters to characterize voltage sags and notches

5.8.1 Voltage sag characterization

A voltage sag is a decrease in the RMS voltage to between 0.1 and 0.9 pu lasting from 0.5 cycle to 1 min [5]. Voltage sag characterization is useful in assessing equipment sensitivity. For instance, sag magnitude is the main factor that determines if a piece of equipment will malfunction or stop working. Moreover, sag duration is important to establish the impact on industrial processes. Thus, for voltage sags lasting longer, the probability that industrial processes stop is higher. Other voltage sag characteristics including POW of initiation/ending and phase angle jump may affect the performance of power electronic equipment that uses phase angle information [6]. In the literature, single-event characteristics including magnitude and duration of sags have been widely addressed. However, POW characteristics and phase angle jump still require further research to be more accurately computed and to better determine the impact on equipment.

Approaches to characterizing voltage sags usually include the analysis of time domain waveforms and profiles (RMS value), or the analysis of ellipse parameters obtained by using the SPM, e.g., [140, 151,152,153, 157, 163, 172]. Interest in multistage characterization is emerging [157, 174], as the usual single-event characteristics may not be enough to describe voltage sag events in real conditions.

5.8.2 Voltage notch characterization

Voltage notches are sub-cycle waveform distortions characterized by a periodic voltage reduction lasting less than half a cycle. Notches may be caused by the normal operation of power electronic converters when current is commutated from one phase to another leading to short-duration overcurrent [5]. A voltage notch is defined by its depth, width, and area [12]. Moreover, the number of voltage notches occurring per cycle or half-cycle is an important feature in suitably assessing the aggregated impact of the disturbance. The number of notches also allows for identification of the type of converter causing the disturbance.

According to the analyzed literature, width is usually obtained in the process of voltage notch characterization. However, depth and area have been less studied. The number of notches per cycle can be extracted from various methodologies in the literature, but it is not explicitly performed. In general, the literature related to voltage notches is still incipient and there is a lack of research on the characterization of the phenomena to analyze, for instance, severity and impact on end-user equipment.

6 Discussion

The discussion is organized into remarks on the methodology for the review and the bibliometric analysis, and the development of the literature review throughout the stages for detection and classification of PQDs described in Fig. 7 i.e., input space, preprocessing, feature engineering, and decision space. Detailed technical aspects related to these stages are also presented, analyzed, and discussed.

6.1 Methodology for the review and bibliometric analysis

6.1.1 Methodology for the literature review

The methodology proposed in Sect. 3 to find the relevant literature is reproducible and scalable. However, a significant effort is required to retrieve and structure the bibliographic metadata to conduct the process. In this context, a gap in the process is observed and a possible improvement of the methodology lies in a higher level of automatization in data retrieval and cleaning. In addition, the indices formulated in the scoring equation (see Sect. 3.2—Scoring equation) to facilitate the selection of the most relevant literature are subject to improvement. Especially, the title and abstract similarity indices may be enhanced using more advanced language processing techniques.

6.1.2 Bibliometric analysis

The bibliometric analysis has revealed the increasing trend in publications related to the detection and classification of PQDs. Likewise, publications including real-time aspects are also increasing, but with a more conservative trend. Thereby, the need for more research on real-time aspects is highlighted. Data of origin of publications indicated that most of the research on the topic (more than 50% of publications) is performed in China, India, and the USA. This situation suggests that higher efforts in researching the topic should be performed worldwide because the context of power systems varies from region to region. Distribution on general topics indicates that real-time aspects are more related to monitoring and detection, and offline applications are more common in the characterization, assessment, and classification of PQDs. Looking at the type of disturbance, general-purpose approaches (multiple PQDs) and tools focused on voltage sags have extensive literature and are very mature. By contrast, research on voltage notches is still incipient. Additional quantitative analyses have been presented for the main steps in the process of real-time detection and classification of PQDs, i.e., transformation and classification. The results of these analyses indicate the most widely used techniques.

6.2 Input space and preprocessing

6.2.1 Input data preparation

The most popular techniques for the input space are the equation-and simulation-based signals. In the case of laboratory and field measurements, the signal acquisition is mainly carried out using DSP, FPGA, microcontrollers, and computers. Real-time digital simulators are a new trend in PQDs simulation. More research should be performed on the use of laboratory and field measurements for the assessment of tools for real-time detection and classification of PQDs, to deal with the uncertainties associated with real conditions.

6.2.2 Data preprocessing

Segmentation and normalization are usually applied for preprocessing of multiple PQDs and voltage sags. In addition, denoising/filtering is mostly applied to specific disturbances such as voltage notches. No extensive details regarding data validation to ensure reliability of data from field measurements have been found in the literature on PQD detection and classification. Therefore, further analysis of data validation in the context of PQD detection and classification may be useful.

6.3 Feature engineering

6.3.1 Transformation

Time–frequency and time-domain techniques are the two most popular for the transformation of PQDs in general, and voltage sags and notches. The techniques widely used in time–frequency domain are WT, ST, and HHT, or their combination with other techniques. The most used non-parametric transformation techniques in the time domain are SPM, PSR, and TT, whilst the most used parametric transformation techniques are KF (especially for voltage sags) and AF (e.g., adaptive linear network). There are still opportunities to apply real-time oriented transformation techniques to perform detection, classification, and characterization of specific disturbances such as voltage sags and notches using non-parametric techniques such as SPM, PSR, and MM, as well as parametric techniques such as Extended KF (non-linear filtering) and AF.

6.3.2 Feature extraction

The feature extraction stage is mostly performed using statistical and time series techniques. The mean value is the clear dominant statistical metric for central tendency, whilst statistical dispersion is usually measured through standard deviation, maximum/minimum value, and statistical values such as variance and HOS (skewness, kurtosis, etc.). On time series techniques, cycle-based indices as coefficients of energy (especially for voltage notches), RMS values, and Shannon entropy are the most common, whereas the most popular sample-based indices are absolute/maximum and event duration for multiple PQDs and voltage sags, and derivative as well as Euclidean distance specifically for voltage notches. The latter might be potentially useful for the assessment of sub-cycle disturbances (e.g., notches, transients).

6.3.3 Feature selection

The step of feature selection is only involved in designing the tools for detection and classification of PQDs but not in the operation. Thereby, the computation times for selecting the optimal set of features for specific applications is a process before the real-time operation. An adequate selection of features allows an enhanced operational efficiency and accuracy. Most approaches in the literature for feature selection are handcrafted based on expert and widely accepted knowledge, with the advantage of obtaining a better physical interpretation of the phenomena. However, handcrafted feature selection can be a time-consuming process. Optimization methods have also been used to provide enhanced results but usually become complicated when selecting a suitable set of features. Finally, dimensionality reduction approaches have shown promising results in the feature selection process providing a good compromise between simplicity of implementation and accuracy and efficiency of results. More effort should be expended on the automatization of dimensionality reduction techniques and the proper physical interpretation of the selected features.

6.4 Decision space

6.4.1 Detection

As a prior step in the process of classifying PQDs, detection may be essential in real-time operation because its independent implementation allows for improving efficiency and general performance. To that end, the algorithms for the detection of states different from the ideal conditions of voltage waveforms are performed constantly in a simple process in real-time. If a PQD is subsequently detected, a more complex process of classification is activated, thus reducing the computational burden of embedded systems. The abovementioned approach has not been extensively implemented and analyzed in the literature while such approach is likely to provide promising results for the implementation of PQD monitoring systems. Furthermore, in the case of voltage sags, detection is useful for applications such as the suitable operation of DVR and protection systems. In those cases, some characterization of the disturbance is necessary.

6.4.2 Classification

Techniques for automatic classification of PQDs have been widely studied in the literature, mainly focused on the categorization of multiple PQDs (sags, swells, notches, transients, harmonics, flicker, etc.). Other advances are in the classification of complex (combined) PQDs, the classification according to root causes, and real-time applications. According to the review, the latter two topics still require further research. The most popular technique for classification is shallow ANNs because of their flexibility in learning any pattern from any set of features. However, ANNs require extensive data and computational effort for training. Decision trees have also been extensively used for classification because of their simplicity, being proper for real-time applications. Recently, deep learning techniques are gaining interest because of the high level of automatization (feature extraction is performed automatically as a process within the technique). However, in these approaches, the physical interpretation of features is lost. Moreover, great potential is observed in the use of unsupervised learning techniques because they have not been yet extensively studied in the context of PQD classification.

6.4.3 Characterization

Single PQD characterization results in the physical interpretation of the phenomena and provides relevant information for the assessment of severity and impact on end-user equipment. Therefore, characterization is useful for analyzing disturbances from the electromagnetic compatibility perspective. Characterization of voltage sags has been widely addressed concerning magnitude and duration. However, POW characteristics and phase angle jump that impact on power electronic equipment still require further research. Multistage sag characterization is of emerging interest because of its occurrence in real conditions and needs further work. Voltage notch characterization is still incipient, especially regarding severity assessment. For instance, the definition of notch depth is ambiguous in the literature. Moreover, the analysis of notching ringing as described in [5] has not yet been addressed in the literature and the characterization of voltage notches may be much more challenging in this context.

6.5 Discussion of technical aspects

Technical issues associated with the steps for PQD detection and classification are discussed. The main technical issue related to input space and data preprocessing is associated to modeling of uncertainties occurring in field measurements. This requires applying data analysis techniques including data plausibility, data cleansing, statistical inference, etc. In the feature engineering stage, the real-time application of processing techniques is still challenging because of computation times in the most widely applied time–frequency-domain transforms such as WT and ST. Although promising results are obtained in the real-time application of time-domain transforms such as SPM, the challenge is the accurate representation of the phenomena and the analysis of single-phase voltages and currents. Feature selection using optimization methods is still challenging while the improvement in accuracy is limited. Therefore, a proper trade-off between complexity and accuracy should be considered in such cases. In the decision space, a challenging technical issue is the formulation of a comprehensive method considering all steps in the decision space, i.e., detection, classification, and characterization that may be useful for real-time applications, e.g., the analysis of PQD propagation to identify and localize root causes, the operation of protection systems, and the automatic implementation of mitigation measures.

7 Perspectives for future research

On transformation techniques, the research trend shows that time-domain and combinations of techniques from different domains are becoming relevant in PQD detection and classification in general. The trend also shows that research is towards the application of one transformation technique, or a combination of several, that can accurately detect the highest number of PQ variations and events rather than specific methodologies for specific disturbances.

WT, ST, and time-domain techniques (non-parametric techniques such as SPM, SPR and MM, as well as parametric techniques such as extended KF and AF) seem to have potential for real-time detection and classification of either PQDs in general (multiple) or the specific (e.g., voltage sags and notches). These techniques, among other characteristics, are flexible in detecting different PQDs and can be used in devices with restricted computational resources. Nevertheless, it is also acknowledged that the combination of different state-of-the-art techniques can also be of benefit for the detection and classification of PQDs. Opportunities for future research also exist in real-time detection, classification, characterization, and possibly in feature aggregation of sub-cycle disturbances, such as voltage notches and transients.

The use of powerful image classification techniques after the 2D transformation of signals is also a promising field of research. This approach allows the use of tools from the ever-increasing potential of the image processing and classification field, e.g., the attention mechanism to improve classification accuracy and transfer learning to reuse pre-trained models [167]. Also, a 2D transformation of signals allows the use of deep learning tools such as CNN. Alternatively, the use of simpler and robust techniques to analyze 2D figures in the complex plane such as Fourier descriptors may provide satisfactory results in classification accuracy and efficiency. Among the 2D transformations, SPM and PSR have shown good characterization capabilities and performance for real-time applications.

In the classification of multiple PQDs and voltage sags, CNN have shown promising results in recent studies [121, 158, 179]. The potential of CNN lies in its ability to automatically extract the best set of features to obtain very accurate results in the classification of PQDs, even in noisy environments. However, CNN have some drawbacks such as the high computational requirement in training and large number of model parameters that hinder real-time application and implementation in embedded systems because of the required additional storage. Moreover, the physical interpretation of features is lost because the automatic extraction is performed within the CNN. A combination of CNN with other techniques may help to overcome the drawbacks. For instance, the few-shot learning technique [171], can be used to reduce the high computational and large dataset requirements for training. Alternatively, ELM is gaining popularity [108, 176], because of its simple operation, sufficiently accurate results, and requirement of fewer training data. The efficiency of ELM also facilitates real-time application.

In the context of the ongoing digitalization of the power system and smart grid paradigm, real-time detection and classification of PQDs play an important role. By online identification of PQDs, a rapid pinpoint of the root-causes can be achieved, and prompt automatic mitigation measures can be implemented to reduce negative technical and economic impacts, e.g., fault location and clearance, flicker source location and mitigation, harmonic resonance source location and mitigation, etc. For this purpose, large amounts of data provided by advanced metering infrastructures, i.e., smart meters, PQ monitors, phasor measurement units, etc., would require advanced algorithms to perform real-time detection and classification of PQDs.

Specific research on voltage sags also offers areas for contribution. For instance, methodologies for voltage sag classification and characterization have focused on single-or three-phase voltages but, to the extent of this review, there is no comprehensive method to automatically classify and characterize single-and three-phase voltage sags. Some approaches have addressed voltage sag root cause location (upstream or downstream), but a more precise location (pinpoint) of sag origin should be achieved. This could be useful for the operation of protection systems and mitigation measures. Also, a more accurate real-time characterization of POW features and phase angle jump can be achieved. The characterization of multistage voltage sags is also of emerging interest because of their common occurrence in real conditions and the limitations of single event voltage sag characterization. Progress in this direction has been made in [157] and [174] using SPM and the multidimensional characterization, respectively, but more studies are still required to improve real-time performance.

Regarding the lack of research in voltage notches, there are several opportunities for further work. For instance, more accurate tools for detecting and characterizing notches are required, including the calculation of single PQD features such as depth, width, area, number of notches, and other indices to assess severity and impact on end-user equipment. Furthermore, automatic classification of voltage notches has not been addressed in the literature. Some categories can be identified such as normal notching and notching ringing [5], while the categorization would allow a better understanding of the notch PQD and an improved severity and impact assessment.

Only supervised learning techniques have been analyzed in this review for the AI-based classification of PQDs. However, unsupervised techniques could be useful in PQD detection and classification because no labeled data is required. This can facilitate the process of training algorithms. Furthermore, unsupervised and supervised learning algorithms can be used together to exploit the potential of both approaches. For instance, simple unsupervised learning algorithms can be used to detect states different from the ideal voltage signal, and then more complex supervised learning algorithms can be used for classification. This approach can enhance efficiency for real-time application.

8 Conclusion

The comprehensive and systematic review conducted in this paper initially develops a methodology to identify the most relevant articles in the detection and classification of PQDs. This methodology results in a scalable and reproducible process that contributed to proposed indices to assess publications in terms of topic similarity and quality of research. The narrow set of publications selected through the systematic process allows a comprehensive overview of the real-time detection and classification of PQDs.

The bibliometric analysis of the literature metadata demonstrates the increasing interest in PQD detection and classification. It also presents top publishing countries and researchers, and first quantitative insight into the relation of general topics, e.g., monitoring, detection, classification, and characterization with real-time applications. The need for further research in real-time approaches for PQD detection and classification is highlighted.

A comprehensive descriptive, qualitative, and quantitative review is performed throughout the stages for real-time detection and classification of PQDs, where techniques dealing with PQDs in general (multiple PQDs) or with specific (e.g., voltage sags and notches) are identified and described. The most relevant findings are summarized in taxonomy figures and tables. Also, more detailed quantitative analyses are provided for the most widely explored stages in the literature, i.e., transformation and classification.

The main remarks arising from the literature review are that transformation and classification techniques have been widely addressed and are very mature for offline applications. However, real-time applications still require more research to find efficient and accurate tools for real conditions in actual power systems. The computational burden is an essential aspect in this context, where embedded systems have limited resources. The proper integration of stages, e.g., preprocessing and feature engineering, and the development of new techniques can facilitate real-time applications.

Research gaps in voltage sags are addressed, including combined single-and three-phase analysis, sag root cause location, accurate and multistage sag characterization. Similarly, research gaps in voltage notches include accurate and unambiguous characterization and classification for severity and impact assessment.