Enhancing image inpainting through image decomposition and deep neural networks

A new approach to inpainting problems that combines domain decomposition methods (DDM) with deep neural networks (DNN) to solve partial differential equations (PDE) is presented.  First, this article examines different
existing and emerging approaches to inpainting while emphasizing their advantages and disadvantages in a unified framework.  After that, we introduce an algorithm that highlights the combination of DDM and DNN techniques for solving PDEs of a proposed mathematical inpainting model.  For this model, the modified approach that has been adopted uses the DNN method which is based on convolutional neural networks (CNN) to reduce the computational cost in our algorithm while maintaining accuracy.  Finally, the experimental results show that our method significantly outperforms existing ones for high-resolution images in paint stains.

deep neural
domain decomposition methods
inpainting
машинне навчання
partial differential equations
  1. Elharrouss O., Almaadeed N., Al-Maadeed S., Akbari Y.  Image inpainting: A review.  Neural Processing Letters.  51 (2), 2007–2028 (2020).
  2. Bertalmio M., Sapiro G., Caselles V., Ballester C.  Image inpainting.  Proceedings of the 27th annual conference on Computer graphics and interactive techniques. 417–424 (2000).
  3. Criminisi A., Shotton J., Konukoglu E.  Decision forests: A unified framework for classification, regression, density estimation, manifold learning and semi-supervised learning.  Foundations and trends® in computer graphics and vision.  7 (2–3), 81–227 (2012).
  4. Boujena S., Bellaj K., El Guarmah E. M., Gouasnouane O.  An improved nonlinear model for image inpainting.  Applied Mathematical Sciences.  9 (124), 6189–6205 (2015).
  5. Ben-Loghfyry A., Hakim A.  Time-fractional diffusion equation for signal and image smoothing.  Mathematical Modeling and Computing.  9 (2),  342–350 (2022).
  6. Gouasnouane O., Moussaid N., Boujena S., Kabli K.  A nonlinear fractional partial differentiation equation for image inpainting.  Mathematical Modeling and Computing.  9 (3), 536–546 (2022).
  7. Kichenassamy S.  The Perona–Malik paradox.  SIAM Journal on Applied Mathematics.  57 (5), 1328–1342 (1997).
  8. Voci F., Eiho S., Sugimoto N., Sekibuchi H.  Estimating the gradient in the Perona–Malik equation.  IEEE Signal Processing Magazine.  21 (3), 39–65 (2004).
  9. Bellaj K., Boujena S., El Guarmah E. M., Gouasnouane O.  One approach for image denoising based on finite element method and domain decomposition technique.  International Journal of Applied Physics and Mathematics.  7 (2), 141–147 (2017).
  10. Boujena S., Pousin J., El Guarmah E. M., Gouasnouane O.  An improved nonlinear model for image restoration.  Pure and Applied Functional Analysis.  2 (4), 599–623 (2017).
  11. Kharazmi E., Zhang Z., Karniadakis G. E.  hp-VPINNs: Variational physics-informed neural networks with domain decomposition.  Computer Methods in Applied Mechanics and Engineering.  374, 113547 (2021).
  12. Firsov D., Lui S. H.  Domain decomposition methods in image denoising using Gaussian curvature.  Journal of Computational and Applied Mathematics.  193 (2), 460–473 (2006).
  13. Smith B. F.  Domain decomposition methods for partial differential equations.  Parallel Numerical Algorithms. 225–243 (1997).
  14. Chan T. F., Mathew T. P.  Domain decomposition algorithms.  Acta Numerica.  3, 61–143 (1994).
  15. Van Eck D., McAdams D. A., Vermaas P. E.  Functional decomposition in engineering: a survey.  International Design Engineering Technical Conferences and Computers and Information in Engineering Conference.  227–236 (2007).
  16. Mahoney M. W., Drineas P.  CUR matrix decompositions for improved data analysis.  Proceedings of the National Academy of Sciences.  106 (3), 697–702 (2009).
  17. Wang Z., He G., Du W., Zhou J., Han X., Wang J., He H., Guo X., Wang J., Kou Y.  Application of parameter optimized variational mode decomposition method in fault diagnosis of gearbox.  IEEE Access.  7, 44871–44882 (2019).
  18. Han D.-R.  A survey on some recent developments of alternating direction method of multipliers.  Journal of the Operations Research Society of China.  10 (1), 1–52 (2022).
  19. Kelleher J. D.  Deep learning. MIT Press (2019).
  20. Alaa K., Atountiand M., Zirhem M.  Image restoration and contrast enhancement based on a nonlinear reaction-diffusion mathematical model and divide & conquer technique.  Mathematical Modeling and Computing.  8 (3), 549–559 (2021).
  21. Alaa H., Alaa N., Aqel F., Lefraich H.  A new Lattice Boltzmann method for a Gray–Scott based model applied to image restoration and contrast enhancement.  Mathematical Modeling and Computing.  9 (2), 187–202 (2022).
  22. Alaa N., Alaa K., Atounti M., Aqel F.  A new mathematical model for contrast enhancement in digital images.  Mathematical Modeling and Computing.  9 (2), 342–350 (2022).
  23. Pintor M., Angioni D., Sotgiu A., Demetrio L., Demontis A., Biggio B., Roli F.  ImageNet-Patch: A dataset for benchmarking machine learning robustness against adversarial patches.  Pattern Recognition.  134, 109064 (2023).
  24. Li D., Ling H., Kim S. W., Kreis K., Fidler S., Torralba A.  BigDatasetGAN: Synthesizing ImageNet with Pixel-wise Annotations.  Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 21330–21340 (2022).
  25. Prabhu V. U., Yap D. A., Wang A., Whaley J.  Covering up bias in CelebA-like datasets with Markov blankets: A post-hoc cure for attribute prior avoidance.  ArXiv preprint arXiv:1907.12917 (2019).
  26. Zhu H., Wu W., Zhu W., Jiang L., Tang S., Zhang L., Liu Z., Loy C. C.  CelebV-HQ: A large-scale video facial attributes dataset.  European Conference on Computer Vision. 650–667 (2022).
  27. Xie K., Gao L., Lu Z., Li C., Xi Q., Zhang F., Sun J., Lin T., Sui J., Ni X.  Inpainting the metal artifact region in MRI images by using generative adversarial networks with gated convolution.  Medical Physics.  49 (10), 6424–6438 (2022).
  28. Yi Z., Tang Q., Azizi S., Jang D., Xu Z.  Contextual residual aggregation for ultra high-resolution image inpainting.  Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 7508–7517 (2020).
  29. Li J., Wang N., Zhang L., Du B., Tao D.  Recurrent feature reasoning for image inpainting.  Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 7760–7768 (2020).
  30. Zhao H., Kong X., He J., Qiao Y., Dong C.  Efficient image super-resolution using pixel attention.  European Conference on Computer Vision. 56–72 (2020).
  31. Zhu M., He D., Li X., Li C., Li F., Liu X., Ding E., Zhang Z.  Image inpainting by end-to-end cascaded refinement with mask awareness.  IEEE Transactions on Image Processing.  30, 4855–4866 (2021).
  32. Wang N., Zhang Y., Zhang L.  Dynamic selection network for image inpainting.  IEEE Transactions on Image Processing.  30, 1784–1798 (2021).