Deep learning for photovoltaic panels segmentation

Due to advanced sensor technology, satellites and unmanned aerial vehicles (UAV) are producing a huge amount of data allowing advancement in all different kinds of earth observation applications.  Thanks to this source of information, and driven by climate change concerns, renewable energy assessment became an increasing necessity among researchers and companies.  Solar power, going from household rooftops to utility-scale farms, is reshaping the energy markets around the globe.  However, the automatic identification of photovoltaic (PV) panels and solar farms' status is still an open question that, if answered properly, will help gauge solar power development and fulfill energy demands.  Recently deep learning (DL) methods proved to be suitable to deal with remotely sensed data, hence allowing many opportunities to push further research regarding solar energy assessment.  The coordination between the availability of remotely sensed data and the computer vision capabilities of deep learning has enabled researchers to provide possible solutions to the global mapping of solar farms and residential photovoltaic panels.  However, the scores obtained by previous studies are questionable when it comes to dealing with the scarcity of photovoltaic systems.  In this paper, we closely highlight and investigate the potential of remote sensing-driven DL approaches to cope with solar energy assessment.  Given that many works have been recently released addressing such a challenge, reviewing and discussing them, it is highly motivated to keep its sustainable progress in future contributions.  Then, we present a quick study highlighting how semantic segmentation models can be biased and yield significantly higher scores when inference is not sufficient.  We provide a simulation of a leading semantic segmentation architecture U-Net and achieve performance scores as high as 99.78%.  Nevertheless, further improvements should be made to increase the model's capability to achieve real photovoltaic units.

дистанційне зондування
сонячна енергія
photovoltaic systems
глибинне навчання
segmentation architecture U-Net
  1. National Renewable Energy Laboratory The Open PV Project, https://www.nrel.gov/docs/fy12osti/52971.pdf (2012).
  2. Helber P., Bischke B., Dengel A., Borth D.  EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification.  IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.  12 (7), 2217–2226 (2019).
  3. Qiu C., Mou L., Schmitt M., Zhu X. X.  Local climate zone-based urban land cover classification from multi-seasonal Sentinel-2 images with a recurrent residual network.  ISPRS Journal of Photogrammetry and Remote Sensing.  154, 151–162 (2019).
  4. Xiang Z., Tailai H., Aminjon G., Yu S., Xihui G., Jiangyuan Z., Shuzhe H., WonHo N., Nengcheng C., Dev N.  Deep Learning-Based 500\,m Spatio-Temporally Continuous Air Temperature Generation by Fusing Multi-Source Data.  Remote Sensing.  14 (15), (2022).
  5. Kerkech M., Hafiane A., Canals R.  Vine disease detection in UAV multispectral images using optimized image registration and deep learning segmentation approach.  Computers and Electronics in Agriculture.  174, 105–446 (2020).
  6. Osorio K., Puerto A., Pedraza C., Jamaica D., Rodriguez L.  A Deep Learning Approach for Weed Detection in Lettuce Crops Using Multispectral Images.  AgriEngineering.  2 (3), 471–488 (2020).
  7. Schut A. G. T., Traore P. C. S., Blaes X., de By R. A.  Assessing yield and fertilizer response in heterogeneous smallholder fields with UAVs and satellites.  Field Crops Research.  221, 98–107 (2018).
  8. Yanhao Z., Ning G., Chen S., Yang Y.  Impact of Rapid Urban Sprawl on the Local Meteorological Observational Environment Based on Remote Sensing Images and GIS Technology.  Remote Sensing. 13 (13), 2624 (2021).
  9. Yao Y., Yee L., Tung F., Zhenfeng S., Jie L., Deyu M., Hanchi Y., Yu Z.  Continuous Multi-Angle Remote Sensing and Its Application in Urban Land Cover Classification.  Remote Sensing.  13 (3), 413 (2021).
  10. Yu J., Wang Z., Majumdar A., Rajagopal R.  DeepSolar: A Machine Learning Framework to Efficiently Construct a Solar Deployment Database in the United States.  Joule.  2 (12), 2605–2617 (2018).
  11. Yongshi J., Xianhua J., Anzhi Y., Jingbo C., Yupeng D., Jing C., Yi Z.  Combined Multi-Layer Feature Fusion and Edge Detection Method for Distributed Photovoltaic Power Station Identification.  Energies. 13 (24), 6742 (2020).
  12. Da Costa M. V. C. V., de Carvalho O. L. F., Orlandi A. G., Hirata I., de Albuquerque A. O., e Silva F. V., Guimarães R. F., Gomes R. A. T., de C. Júnior O. A.  Remote Sensing for Monitoring Photovoltaic Solar Plants in Brazil Using Deep Semantic Segmentation.  Energies.  14 (10), 2960 (2021).
  13. Sadeghi M., Asanjan A. A., Faridzad M., Nguyen P., Hsu K., Sorooshian S., Braithwaite D.  PERSIANN-CNN: Precipitation Estimation from Remotely Sensed Information Using Artificial Neural Networks–Convolutional Neural Networks.  Journal of Hydrometeorology.  20 (12), 2273–2289 (2019).
  14. Park Y., Kwon B., Heo J., Hu X., Liu Y., Moon T.  Estimating PM2.5 concentration of the conterminous United States via interpretable convolutional neural networks.  Environmental Pollution.  256, 113395 (2020).
  15. Shen H., Jiang Y., Li T., Cheng Q., Zeng C., Zhang L.  Deep learning-based air temperature mapping by fusing remote sensing, station, simulation and socioeconomic data.  Remote Sensing of Environment.  240, 111692 (2020).
  16. Sun X., Wang P., Yan Z., Xu F., Wang R., Diao W., Chen J., Li J., Feng Y., Xu T., Weinmann M., Hinz S., Wang C., Fu K.  FAIR1M: A benchmark dataset for fine-grained object recognition in high-resolution remote sensing imagery.  ISPRS Journal of Photogrammetry and Remote Sensing. 116–130 (2022).
  17. Yating G., Wang Y., Li Y.  A Survey on Deep Learning-Driven Remote Sensing Image Scene Understanding: Scene Classification, Scene Retrieval and Scene-Guided Object Detection.  Applied Sciences.  9 (10), 2110 (2019).
  18. Cheng G., Han J., Lu X.  Remote Sensing Image Scene Classification: Benchmark and State of the Art.  Proceedings of the IEEE.  105 (10), 1865–1883 (2017).
  19. Xia G.-S., Tong X.-Y., Hu F., Zhong Y., Datcu M., Zhang L.  Exploiting Deep Features for Remote Sensing Image Retrieval: A Systematic Investigation.  IEEE Transactions on Big Data.  6 (3), 507–52 (2019).
  20. Shen C., Laloy E., Elshorbagy A., Albert A., Bales J., Chang F.-J., Ganguly S., Hsu K.-L., Kifer D., Fang Z., Fang K., Li D., Li X., Tsai W.-P.  HESS Opinions: Incubating deep-learning-powered hydrologic science advances as a community.  Hydrology and Earth System Sciences.  22 (11), 5639–5656 (2018).
  21. Su P., Liu Y., Tarkoma S., Rebeiro–Hargrave A., Petaja T., Kulmala M., Pellikka P.  Retrieval of Multiple Atmospheric Environmental Parameters From Images With Deep Learning.  IEEE Geoscience and Remote Sensing Letters.  19, 1005005 (2022).
  22. Lillesand T., Kiefer R., Chipman J.  Remote sensing and image interpretation. Wiley (2015).
  23. Kaku K.  Satellite remote sensing for disaster management support: A holistic and staged approach based on case studies in Sentinel Asia.  International Journal of Disaster Risk Reduction.  33, 417–432 (2019).
  24. Curran P. J.  Principles of remote sensing.  New York, Longman Scientific and Technical (1985).
  25. Xia G.-S., Hu J., Hu F., Shi B., Bai X., Zhong Y., Lu X., Zhang L.  AID: A Benchmark Data Set for Performance Evaluation of Aerial Scene Classification.  IEEE Transactions on Geoscience and Remote Sensing.  55 (7), 3965–3981 (2017).
  26. Xia G.-S., Bai X., Ding J., Zhu Z., Belongie S., Luo J., Datcu M., Pelillo M., Zhang L.  DOTA: A Large-scale Dataset for Object Detection in Aerial Images.  IEEE/CVF Conference on Computer Vision and Pattern Recognition (2018).
  27. Hu Q., Wu W., Xia T., Yu Q., Yang P., Li Z.-G., Song Q.  Exploring the Use of Google Earth Imagery and Object-Based Methods in Land Use/Cover Mapping.  Remote Sensing.  5 (11), 6026–6042 (2013).
  28. Verde N., Mallinis G., Tsakiri–Strati M., Georgiadis C., Patias P.  Assessment of Radiometric Resolution Impact on Remote Sensing Data Classification Accuracy.  Remote Sensing.  10 (8), 1267 (2018).
  29. Zhou K., Cheng T., Zhu Y., Cao W., Ustin S. L., Zheng H., Yao X., Tian Y.  Assessing the Impact of Spatial Resolution on the Estimation of Leaf Nitrogen Concentration Over the Full Season of Paddy Rice Using Near-Surface Imaging Spectroscopy Data.  Frontiers in Plant Science.  9, (2018).
  30. Lawrence R., Labus M.  Early Detection of Douglas-Fir Beetle Infestation with Subcanopy Resolution Hyperspectral Imagery.  Western Journal of Applied Forestry.  18 (3), 202–206 (2003).
  31. Etchanchu J., Rivalland V., Gascoin S., Cros J., Brut A., Boulet G.  Effects of multi-temporal high-resolution remote sensing products onsimulated hydrometeorological variables in a cultivated area (southwestern France).  Hydrology and Earth System Sciences Discussions. 1–23 (2017).
  32. Pique G., Fieuza R., Debaeke P., Al Bitar A., Tallec T., Ceschia E.  Combining High-Resolution Remote Sensing Products with a Crop Model to Estimate Carbon and Water Budget Components: Application to Sunflower.  Remote Sensing.  12 (18), 2967 (2020).
  33. Li Y., Chunyan C., Zhuoran W., Guanghui Q., Chao D., Gengxing Z.  Upscaling Remote Sensing Inversion Model of Wheat Field Cultivated Land Quality in the Huang-Huai-Hai Agricultural Region, China.  Remote Sensing.  13 (24), 5095 (2021).
  34. Xie X., Xu J. W., Zhao J. F., Liu S., Wang P.  Soil Moisture Inversion Using AMSR-E Remote Sensing Data: An Artificial Neural Network Approach.  Applied Mechanics and Materials.  501–504, 2073–2076 (2014).
  35. Said S., Khan S. A.  Remote sensing-based water quality index estimation using data-driven approaches: a case study of the Kali River in Uttar Pradesh, India.  Environment, Development and Sustainability.  23, 18252–18277 (2021).
  36. Dai X., He X., Guo S., Liu S., Ji F., Ruan H.  Research on hyper-spectral remote sensing image classification by applying stacked de-noising auto-encoders neural network.  Multimedia Tools and Applications.  80, 21219–21239 (2021).
  37. Protopapadakis E., Doulamis A., Doulamis N., Maltezos E.  Stacked Autoencoders Driven by Semi-Supervised Learning for Building Extraction from near Infrared Remote Sensing Imagery.  Remote Senisng.  13 (3), 371 (2021).
  38. Wang T., Crawford M. M.  Multi-Year Sorghum Biomass Prediction with UAV-Based Remote Sensing Data.  IEEE International Geoscience and Remote Sensing Symposium IGARSS (2021).
  39. Khusn U., Dewangkoro H. I., Arymurthy A. M.  Urban Area Change Detection with Combining CNN and RNN from Sentinel-2 Multispectral Remote Sensing Data.  International Conference of Computer and Informatics Engineering (IC2IE), Yogyakarta, Indonesia (2020).
  40. Yang Y., Newsam S. D.  Bag-of-visual-words and spatial extensions for land-use classification.  The 18th SIGSPATIAL International Conference on Advances in Geographic Information Systems (GIS'10), New York, NY, USA (2010).
  41. Zhou W., Newsam S., Li C., Shao Z.  PatternNet: A benchmark dataset for performance evaluation of remote sensing image retrieval.  ISPRS Journal of Photogrammetry and Remote Sensing. 197–209 (2018).
  42. Sumbul G., Charfuelan M., Demir B., Markl V.  Bigearthnet: A Large-Scale Benchmark Archive for Remote Sensing Image Understanding.  IGARSS 2019 – 2019 IEEE International Geoscience and Remote Sensing Symposium, Yokohama, Japan (2019).
  43. Zhang W., Liljedahl A. k., Kanevskiy M., Epstein H. E., Jones B. M., Jorgenson M. T., Kent K.  Transferability of the Deep Learning Mask R-CNN Model for Automated Mapping of Ice-Wedge Polygons in High-Resolution Satellite and UAV Images.  Remote Sensing.  12 (7), 1085 (2020).
  44. Balsi M., Moroni M., Chiarabini V., Tanda G.  High-Resolution Aerial Detection of Marine Plastic Litter by Hyperspectral Sensing.  Remote Sensing.  13 (8), (2021).
  45. Seydi S. T., Hasanlou M., Amani M., Huang W.  Oil Spill Detection Based on Multiscale Multidimensional Residual CNN for Optical Remote Sensing Imagery.  Journal of Selected Topics in Applied Earth Observations and Remote Sensing.  14, 10941–10952 (2021).
  46. Mohanty S. P., Czakon J., Kaczmarek K. A., Tarasiewicz P., Kunwar S., Rohrbach J., Luo D., Prasad M., Fleer S., Göpfert J. P., Tandon A., Mollard G., Rayaprolu N., Salathe M., Malte S.  Deep Learning for Understanding Satellite Imagery: An Experimental Survey.  Frontiers in artificial intelligence. Vol. 3 (2020).
  47. Haroon M., Shahzad M., Fraz M. M.  Multisized Object Detection Using Spaceborne Optical Imagery IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.  IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.  13, 3032–3046 (2020).
  48. Kranjčić N., Bek A., Durin B., Singh S. K., Kanga S.  Analysis of solar energy potential by remote sensing techniques in Varaždinska county, Croatia.  The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Vol. XLVI-4/W5-2021, 343–347 (2021).
  49. Singh S. K., Mishra S., Kashyap A. K., Jain S. K.  Mapping Solar Energy Potential Zones using Remote Sensing based Solar Data.  4th International Conference on Recent Developments in Control, Automation and amp; Power Engineering (RDCAPE), (2021).
  50. Mwanza M., Koray U.  GIS-Based Assessment of Solar Energy Harvesting Sites and Electricity Generation Potential in Zambia.  African Handbook of Climate Change Adaptation. Springer, Cham. 899–946 (2021).
  51. Sunter D. A., Dees J., Castellanos S., Callaway D., Kammen D. M.  Political Affiliation and Rooftop Solar Adoption in New York and Texas.  IEEE 7th World Conference on Photovoltaic Energy Conversion (WCPEC), (2018).
  52. Yu J., Zhecheng W., Arun M., Ram R.  DeepSolar: A Machine Learning Framework to Efficiently Construct a Solar Deployment Database in the United States.  Joule.  2 (12), 2605–2617 (2018).
  53. Zhou B., Khosla A., Lapedriza A., Oliva A., Torralba A.  Learning Deep Features for Discriminative Localization.  IEEE Conference on Computer Vision and Pattern Recognition (CVPR), (2016).
  54. Hou X., Wang B., Hu W., Yin L., Wu H.  SolarNet: A Deep Learning Framework to Map Solar Plants In China From Satellite Imagery.  ICLR 2020 Workshop on Tackling Climate Change with Machine Learning, Climate Change AI (2020).
  55. Dempster A. P., Laird N. M., Rubin D. B.  Maximum likelihood from incomplete data via the em algorithm.  Journal of the Royal Statistical Society.  39 (1), 1–38 (1977).
  56. Li X., Zhong Z., Wu J., Yang Y., Lin Z., Liu H.  Expectation-Maximization Attention Networks for Semantic Segmentation.  arXiv:1907.13426 (2019).
  57. Mayer K., Wang Z., Arlt M.-L., Neumann D., Rajagopal R.  DeepSolar for Germany: A deep learning framework for PV system mapping from aerial imagery.  International Conference on Smart Energy Systems and Technologies (SEST), (2020).
  58. Bonabeau E., Dorigo M., Theraulaz G.  Swarm Intelligence: From Natural to Artificial Systems. USA, Oxford University Press (1999).
  59. Zhuang L., Zhang Z., Wang L.  The automatic segmentation of residential solar panels based on satellite images: A cross learning driven U-Net method.  Applied Soft Computing.  92, 106283 (2020).
  60. Jie Y., Ji X., Yue A., Chen J., Deng Y., Chen J., Zhang Y.  Combined Multi-Layer Feature Fusion and Edge Detection Method for Distributed Photovoltaic Power Station Identification.  Energies.  13 (24), 6742 (2020).
  61. Ioannou K., Myronidis D.  Automatic Detection of Photovoltaic Farms Using Satellite Imagery and Convolutional Neural Networks.  Sustainability.  13 (9), 5323 (2021).
  62. Bradbury K., Saboo R., Johnson T. L., Malof J. M., Devarajan A., Zhang W., Collins L. M., Newell R. G.  Distributed solar photovoltaic array location and extent dataset for remote sensing object identification.  Scientific Data.  3 (1), 160106 (2016).
  63. Jiang H., Yao L., Lu N., Qin J., Liu T., Liu Y., Zhou C.  Multi-resolution dataset for photovoltaic panel segmentation from satellite and aerial imagery.  Earth System Science Data.  13 (11), 5389–5401 (2021).
  64. Masoom A., Kosmopoulos P., Bansal S.  Solar Energy Estimations in India Using Remote Sensing Technologies and Validation with Sun Photometers in Urban Areas.  Remote Sensing.  12 (2), 254 (2020).
  65. Zhang Y., Dai Z., Wang W., Li X., Chen S., Chen L.  Estimation of the Potential Achievable Solar Energy of the Buildings Using Photogrammetric Mesh Models.  Remote Sensing.  13 (13), 2484 (2021).
  66. Alhammad A., Sun Q. C., Tao Y.  Optimal Solar Plant Site Identification Using GIS and Remote Sensing: Framework and Case Study.  Energies.  15 (1), 312 (2022).
  67. Ronneberger O., Fischer P., Brox T.  U-Net: Convolutional Networks for Biomedical Image Segmentation.  Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015. 234–241 (2015).
  68. Prasanna M. S., Jakub C., Kamil A. K., Andrzej P., Piotr T., Saket K., Janick R., Dave L., Manjunath P., Sascha F., Philip G. J., Akshat T., Guillaume M., Nikhil R., Marcel S., Malte S.  Deep Learning for Understanding Satellite Imagery: An Experimental Survey.  Frontiers in Artificial Intelligence. Vol. 3 (2020).