1. SISN
  2. All Volumes and Issues
  3. Volume 15, 2024
  4. Analysis of Real-time Processing Approaches for Large Data Volumes in Metering Infrastructure

Analysis of Real-time Processing Approaches for Large Data Volumes in Metering Infrastructure

SISN.
2024;
Volume 15
: pp. 169 - 183
https://doi.org/10.23939/sisn2024.15.169
Authors:
  1. Roman Moravskyi
  2. Yevheniya Levus
1
Lviv Polytechnic National University, Lviv, Ukraine
2
Lviv Polytechnic National University, Lviv, Ukraine

Smart grid systems and communication technologies, such as Advanced Metering Infrastructure (AMI), have revolutionized utility service management and monitoring. AMI leverages smart meters equipped with advanced communication capabilities, facilitating bidirectional communication between utilities and consumers. The increasing deployment of smart meters and the adoption of sub-hourly data collection requirements by utility companies highlight significant data volume growth. Thus, there is a need for efficient real-time data processing solutions as existing approaches may not meet previously established Service-Level Agreements (SLAs) concerning performance, accuracy, and scalability metrics. This research aims to comprehensively review the latest publications relevant to distributed real-time data processing methods for smart grid applications and outline problems for further research. Specifically, the study delves into the effectiveness and application of reviewed approaches in managing the constant stream of data from smart meters and IoT devices within the smart grid context. By analysing existing methodologies and advancements, this study seeks to identify challenges and opportunities in real-time data processing for smart grid infrastructures, focusing on addressing the complexities of processing, managing, and storing large volumes of real-time data. The literature review revealed two primary applications of real-time data processing: optimization of data streaming performance and data analysis. The review encompasses various studies, each presenting distinct methodologies and technologies applied to address the challenges of processing large volumes of real- time data from smart meters and IoT devices. Future research should address the challenges and limitations discovered in this study.

distributed processing
data streaming
smart grid (SG)
smart meter (SM)
Advanced Metering Infrastructure (AMI)
  1. Hassan, A., Afrouzi, H. N., Siang, C. H., Ahmed, J., Mehranzamir, K., & Wooi, C.-L. (2022). A survey and bibliometric analysis of different communication technologies available for smart meters. Cleaner Engineering and Technology, 7, 100424. https://doi.org/10.1016/j.clet.2022.100424
  2. Tightiz, L., & Yang, H. (2020). A comprehensive review on IOT protocols' features in Smart Grid Communication. Energies, 13(11), 2762. https://doi.org/10.3390/en13112762
  3. Sikic, L., Jankovic, J., Afric, P., Silic, M., Ilic, Z., Pandzic, H., Zivic, M., & Dzanko, M. (2020). A comparison of application layer communication protocols in IOT-enabled Smart Grid. 2020 International Symposium ELMAR.     https://doi.org/10.1109/elmar49956.2020.9219030
  4. Lombardi, M., Pascale, F., & Santaniello, D. (2021). Internet of things: A general overview between architectures, protocols and applications. Information, 12(2), 87. https://doi.org/10.3390/info12020087
  5. Khan, B., & Pirak, C. (2021). Experimental Performance Analysis of MQTT and CoAP protocol usage for Nb-IOT Smart meter. 2021 9th International Electrical Engineering Congress (iEECON). https://doi.org/10.1109/ieecon51072.2021.9440273
  6. Anani, W., & Ouda, A. (2022). Wireless meter bus: Secure remote metering within the IOT smart grid. 2022 International Symposium on Networks, Computers and Communications (ISNCC). https://doi.org/10.1109/isncc55209.2022.9851807
  7. Office of Electricity Delivery and Energy Reliability. Advanced Metering Infrastructure and Customer Systems – Results from the Smart Grid Investment Grant Program (2016, September). https://www.energy.gov/sites/prod/files/2016/12/f34/AMI%20Summary%20Report_09-26-16.pdf
  8. Dey, A., Chakraborty, B., Dalai, S., & Bhattacharya, K. (2022). Insights and new practices for advanced metering infrastructure and smart energy metering framework in smart grid- A case study. 2022 IEEE Calcutta Conference (CALCON). https://doi.org/10.1109/calcon56258.2022.10060514
  9. Definition of Advanced Metering Infrastructure (AMI) - Gartner Information Technology Glossary. (n. d.).    Gartner.    https://www.gartner.com/en/information-technology/glossary/advanced-metering-infrastructure-ami
  10. Carou Álvarez, J. M., & Ramón, L. S. (2023). Smart meters. Encyclopedia of Electrical and Electronic Power Engineering, 441–447. https://doi.org/10.1016/b978-0-12-821204-2.00067-2
  11. Kumar, A., Thakur, S., & Bhattacharjee, P. (2018b). Real time monitoring of AMR enabled energy meter for AMI in Smart City – an IOT application. 2018 IEEE International Symposium on Smart Electronic Systems (iSES) (Formerly iNiS). https://doi.org/10.1109/ises.2018.00055
  12. Huang, J.-F., Zhang, G.-H., & Hsieh, S.-Y. (2021). Real-time energy data compression strategy for reducing data traffic based on Smart Grid AMI Networks. The Journal of Supercomputing, 77(9), 10097–10116. https://doi.org/10.1007/s11227-020-03557-8
  13. Abdulaal, M. J., Ibrahem, M. I., Mahmoud, M. M., Khalid, J., Aljohani, A. J., Milyani, A. H., & Abusorrah, A. M. (2022). Real-time detection of false readings in Smart Grid AMI using deep and ensemble learning. IEEE Access, 10, 47541–47556. https://doi.org/10.1109/access.2022.3171262
  14. Zhou, S. (2021). The effect of smart meter penetration on dynamic electricity pricing: Evidence from the United States. The Electricity Journal, 34(3), 106919. https://doi.org/10.1016/j.tej.2021.106919
  15. Mansoor, H., Ali, S., Khan, I. U., Arshad, N., Khan, M. A., & Faizullah, S. (2023). Short-term load forecasting using AMI data. IEEE Internet of Things Journal, 1–1. https://doi.org/10.1109/jiot.2023.3295617
  16. Huang, C.-M., Huang, Y.-C., Chen, S.-J., Yang, S.-P., & Huang, K.-Y. (2021). Ami load forecasting and interval forecasting using a hybrid intelligent method. 2021 IEEE 30th International Symposium on Industrial Electronics (ISIE). https://doi.org/10.1109/isie45552.2021.9576343
  17. Gold, R., Waters, C., & York, D. (2020). Leveraging advanced metering infrastructure to save energy. Washington DC: American Council for an Energy-Efficient Economy (ACEEE).
  18. Frequently Asked Questions (FAQs) – U.S. Energy Information Administration (EIA) (2023, October 20). Homepage – U.S. Energy Information Administration (EIA). https://www.eia.gov/tools/faqs/faq.php?id=108&t=3
  19. Kotsar, O. (2018). The development of Smart Systems for control, metering and energy management in liberalized electricity market of Ukraine. Tekhnichna Elektrodynamika, 2018(4), 110–117. https://doi.org/10.15407/techned2018.04.110
  20. Chukut, S., & Shumska, L. (2022). Introducing smart grid as part of a smart city using big data: current challenges and trends. Investytsiyi: Praktyka Ta Dosvid, (3), 88–95. https://doi.org/10.32702/2306-6814.2022.3.88
  21. Petko, S. (2023). The Republic of Korea experience of “smart-grid” implementing in the post-war recovery of Ukrainian Energy System. Marketing and Digital Technologies, 7(2), 8–18. https://doi.org/10.15276/mdt.7.2.2023.1
  22. 15 min imbalance settlement period in Norway and Sweden. eSett. (2023, May 5). https://www.esett.com/news/15-min-imbalance-settlement-period-in-norway-and-sweden/
  23. Rendroyoko, I., Setiawan, A. D., & Suhardi, S. (2021). Development of meter data management system based-on event-driven streaming architecture for IOT-based AMI implementation. 2021 3rd International Conference on High Voltage Engineering and Power Systems (ICHVEPS). https://doi.org/10.1109/ichveps53178.2021.9601104
  24. Dutta, S., Miranda, A., & Arboleya, P. (2023). Real-time data extraction, transformation and loading process for LV Advanced Distribution Management Systems. 2023 IEEE Belgrade PowerTech. https://doi.org/10.1109/powertech55446.2023.10202807
  25. Pau, M., Patti, E., Barbierato, L., Estebsari, A., Pons, E., Ponci, F., & Monti, A. (2018). A cloud-based smart metering infrastructure for distribution grid services and automation. Sustainable Energy, Grids and Networks, 15, 14–25. https://doi.org/10.1016/j.segan.2017.08.001
  26. Pires, F. M., Leon Quinonez, L., & de Souza Mendes, L. (2019). A cloud-based system architecture for advanced metering in smart cities. 2019 IEEE 10th Annual Information Technology, Electronics and Mobile Communication Conference (IEMCON). https://doi.org/10.1109/iemcon.2019.8936283
  27. Prakoso, M. H., Irawan, F., Sufianto, A. M., & Rediansyah, D. (2023). Comprehensive assessment of small batch advanced metering infrastructure utilisation on Java region to support Indonesian smart grid systems. 2023 4th International Conference on High Voltage Engineering and Power Systems (ICHVEPS). https://doi.org/10.1109/ichveps58902.2023.10257557
  28. Anugrahany, E., Supriyadi, G., Nugraha, D. A., W, O. P., & Mafruddin, M. M. (2021). Assessment procedure for advanced metering infrastructure implementation in Indonesia. 2021 3rd International Conference on High Voltage Engineering and Power Systems (ICHVEPS). https://doi.org/10.1109/ichveps53178.2021.9601053
  29. Moravskyi, R., Pustelnyk, P., Morozov, M., & Levus, Y. (2023). Cloud-based distributed approach for procedural terrain generation with enhanced performance. 2023 IEEE 18th International Conference on Computer Science and Information Technologies (CSIT). https://doi.org/10.1109/csit61576.2023.10324223
  30. Mohamed, Z. (2023, November 24). (thesis). Data streaming provenance in advanced metering infrastructures. Retrieved February 21, 2024, from https://hdl.handle.net/2077/79292.
  31. Taube, J., & Johnsson, W. (2022). (thesis). Streaming Analytics with Provenance in the Advanced Metering Infrastructure. Retrieved March 3, 2024, from https://odr.chalmers.se/handle/20.500.12380/305852
  32. Popović, I., Rakić, A., & Petruševski, I. D. (2022). Multi-agent real-time advanced metering infrastructure based on Fog Computing. Energies, 15(1), 373. https://doi.org/10.3390/en15010373
  33. Gupta, A., & Jain, S. (2022). Optimising performance of real-time big data stateful streaming applications on cloud. 2022 IEEE International Conference on Big Data and Smart Computing (BigComp). https://doi.org/10.1109/bigcomp54360.2022.00010
  34. Tantalaki, N., Souravlas, S., & Roumeliotis, M. (2019). A review on big data real-time stream processing and its scheduling techniques. International Journal of Parallel, Emergent and Distributed Systems, 35(5), 571–601. https://doi.org/10.1080/17445760.2019.1585848
  35. Zainab, A., Refaat, S. S., Abu-Rub, H., & Bouhali, O. (2020). Distributed computing for smart meter data management for electrical utility applications. 2020 Cybernetics & Informatics (K&I). https://doi.org/10.1109/ki48306.2020.9039899
  36. Zainab, A., Ghrayeb, A., Syed, D., Abu-Rub, H., Refaat, S. S., & Bouhali, O. (2021). Big Data Management in smart grids: Technologies and challenges. IEEE Access, 9, 73046–73059. https://doi.org/10.1109/access.2021.3080433
  37. Fu, Y., & Soman, C. (2021). Real-time data infrastructure at Uber. Proceedings of the 2021 International Conference on Management of Data. https://doi.org/10.1145/3448016.3457552
  38. Pandit, R., Astolfi, D., Hong, J., Infield, D., & Santos, M. (2022). SCADA data for wind turbine data- driven condition/performance monitoring: A review on state-of-art, challenges and future trends. Wind Engineering, 47(2), 422–441. https://doi.org/10.1177/0309524x221124031
  39. Cheng, G., Lin, Y., Abur, A., Gómez-Expósito, A., & Wu, W. (2024). A survey of power system state estimation using multiple data sources: Pmus, SCADA, Ami, and beyond. IEEE Transactions on Smart Grid, 15(1), 1129–1151.    https://doi.org/10.1109/tsg.2023.3286401
  40. Shrestha, M., Johansen, C., Noll, J., & Roverso, D. (2020). A methodology for security classification applied to Smart Grid Infrastructures. International Journal of Critical Infrastructure Protection, 28,  100342. https://doi.org/10.1016/j.ijcip.2020.100342
  41. Amović, M., Govedarica, M., Radulović, A., & Janković, I. (2021). Big Data in smart city: Management challenges. Applied Sciences, 11(10), 4557. https://doi.org/10.3390/app11104557
  42. Güçyetmez, M., & Farhan, H. S. (2023). Enhancing smart grids with a new IOT and cloud-based smart meter to predict the energy consumption with time series. Alexandria Engineering Journal, 79, 44–55. https://doi.org/10.1016/j.aej.2023.07.071
  43. Schulz, D., Lawanson, T., Ravikumar, K., & Cecchi, V. (2020). Loss estimation and visualization in distribution systems using AMI and Recloser Data. 2020 IEEE/PES Transmission and Distribution Conference and Exposition (T&D). https://doi.org/10.1109/td39804.2020.9299891
  44. Deng, B., Wen, Y., & Yuan, P. (2020). Hybrid short-term load forecasting using the Hadoop MapReduce framework. 2020 IEEE Power & Energy Society General Meeting (PESGM). https://doi.org/10.1109/pesgm41954.2020.9282094
  45. Hu, L., Zhang, L., Wang, T., & Li, K. (2020). Short-term load forecasting based on support vector regression considering cooling load in summer. 2020 Chinese Control And Decision Conference (CCDC). https://doi.org/10.1109/ccdc49329.2020.9164387
  46. Banik, S., Saha, S. K., Banik, T., & Hossain, S. M. (2023). Anomaly detection techniques in Smart Grid Systems: A Review. 2023 IEEE World AI IoT Congress (AIIoT). https://doi.org/10.1109/aiiot58121.2023.10174485
  47. Ibrahem, M. I., Abdelfattah, S., Mahmoud, M., & Alasmary, W. (2021). Detecting electricity theft cyber- attacks in Cat Ami System using machine learning. 2021 International Symposium on Networks, Computers and Communications (ISNCC). https://doi.org/10.1109/isncc52172.2021.9615629
  48. Takiddin, A., Ismail, M., Zafar, U., & Serpedin, E. (2021). Variational auto-encoder-based detection of electricity stealth cyber-attacks in Ami Networks. 2020 28th European Signal Processing Conference (EUSIPCO). https://doi.org/10.23919/eusipco47968.2020.9287764
  49. Qi, R., Zheng, J., Luo, Z., & Li, Q. (2022). A novel unsupervised data-driven method for electricity theft detection in Ami using observer meters. IEEE Transactions on Instrumentation and Measurement, 71, 1–10. https://doi.org/10.1109/tim.2022.3189748
  50. M. Jahid Hasan, A. S., Rahman, M. S., Islam, M. S., & Yusuf, J. (2023). Data Driven Energy theft localization in a distribution network. 2023 International Conference on Information and Communication Technology for Sustainable Development (ICICT4SD). https://doi.org/10.1109/icict4sd59951.2023.10303520