ENHANCEMENT OF TRAFFIC IMAGES UNDER DIFFERENT WEATHER CONDITIONS USING PYNET
DOI:
https://doi.org/10.11113/aej.v14.20665Keywords:
Image enhancement, deep learning, semantic segmentation, traffic imagesAbstract
Convolutional neural networks have real practical application potentials, such as for autonomous driving and semantic segmentation, but they are known to be sensitive to image degradations and corruption. Over the years, image enhancement in deep learning has shown drastic improvement. However, the clarity and quality of images still badly affect the robustness of semantic segmentation, especially for traffic images under different weather conditions. This paper proposes image enhancement performance using PyNET deep learning methods for more robust semantic segmentation of traffic images under different conditions. This work also proposes a new metric to objectively estimate the performance known as Image Similarity Metrics (ISM). Modification to PyNET is made in this work to allow this deep learning model to be used to enhance traffic images under various weather conditions such as fog, night, and rain. We compared PyNET performance against the Deep Convolutional Networks (DPED) and the Cycle Generative Adversarial Networks (CycleGAN) to evaluate the improvement gained by these image enhancement methods. Based on our experiment, PyNET gives the best image enhancement performance among those three methods in all weather conditions according to our proposed ISM. To support the validity of the ISM result, we performed tests by semantic segmentation of traffic images using ResNet-18 on the PyNET-enhanced images. Based on semantic segmentation results, PyNET improves the semantic segmentation of traffic images under different weather conditions by as much as 17% accuracy and delivers performance that directly validates the ISM scores, by showing that PyNET delivers the best semantic segmentation improvement in fog and night images
References
P. P. Shinde and S. Shah, 2018. “A Review of Machine Learning and Deep Learning Applications,” International Conference on Computing Communication Control and Automation. 1–6, DOI: 10.1109/ICCUBEA.2018.8697857.
A. A. Khan, A. A. Laghari, and S. A. Awan, 2018., “Machine Learning in Computer Vision: A Review,” EAI Endorsed Transactions on Scalable Information Systems, 8: 1–11. DOI: 10.4108/eai.21-4-2021.169418.
F. Zantalis, G. Koulouras, S. Karabetsos, and D. Kandris, 2019., “A review of machine learning and IoT in smart transporttion,” Future Internet, 11: 1–23. DOI: 10.3390/FI11040094.
Y. Xin 2018. “Machine Learning and Deep Learning Methods for Cybersecurity,” IEEE Access, 6: 1–23. DOI: 10.1109/ACCESS.2018.2836950.
P. Gogas and T. Papadimitriou, 2021. “Machine Learning in Economics and Finance,” Computational Economics, 57: 1–4. DOI: 10.1007/s10614-021-10094-w.
F. Femling, A. Olsson, and F. Alonso-Fernandez 2018., “Fruit and Vegetable Identification Using Machine Learning for Retail Applications,” International Conference Signal Image Technology and Internet Based System (SITIS), 9–15. DOI: 10.1109/SITIS.2018.00013.
B. T.k., C. S. R. Annavarapu, and A. Bablani 2021. “Machine learning algorithms for social media analysis: A survey,” Computer Science Review 40: 1–32. DOI: 10.1016/j.cosrev.2021.100395.
H. I. Suk, 2017., An Introduction to Neural Networks and Deep Learning, 1st ed. Elsevier Inc.
Q. Zhang, M. Zhang, T. Chen, Z. Sun, Y. Ma, and B. Yu, 2018 “Recent advances in convolutional neural network acceleration,” Neurocomputing, 1–27. DOI: 10.1016/j.neucom.2018.09.038.
K. Li, Y. G., Q. Y., C. X., and D. 2020, “Low-Light Image Enhancement for Autonomous Driving System using DriveRetinex-Net,” 1: 1–5. Elsevier B.V. DOI: doi.org/10.1016/j.knosys.2020.106617.
K. Zhang, W. Zuo, S. Gu, and L. Zhang 2017 “Learning deep CNN denoiser prior for image restoration,” IEEE Conference on Computer Vision and Pattern Recognition.. 1–10. DOI: 10.1109/CVPR.2017.300.
S. Ghosh, N. Das, I. Das, and U. Maulik 2019. Understanding deep learning techniques for image segmentation,” ACM Computer Surveys 52: 1–35. DOI: 10.1145/3329784.
C. Wang, Y. Han, and W. Wang 2019. “An end-to-end deep learning image compression framework based on semantic analysis,” Applied Sciences, 9: DOI: 10.3390/app9173580.
D. H. Kim and H. Y. Lee 2017. “Image manipulation detection using convolutional neural network,” International Journal Applied Engineering Research. 12: 1–7,.
A. Noguchi and T. Harada, “Image generation from small datasets via batch statistics adaptation,” International Conference on Computer Vision (ICCV), 1–9. DOI: 10.1109/ICCV.2019.00284.
A. R. Pathak, M. Pandey, and S. Rautaray. 2018. “Application of Deep Learning for Object Detection,” Procedia Computer Science, 132: 1–12. DOI: 10.1016/j.procs.2018.05.144.
B. Chitradevi and P. Srimanthi. 2014., “An Overview on Image Processing Techniques,” International Journal of Innovative Research in Computer and Communication Engineering 2: 1–7. [Online]Available:http://www.hindawi.com/isrn/sp/2013/496701/.
S. Gollapudi 2018., “Learning For Computer Vision Tasks: A review,” International Conference on Intelligent Computing and Control, 1–5. DOI: 10.1007/978-1-4842-4261-2_3.
Y. Qi 2021., “A Comprehensive Overview of Image Enhancement Techniques,” Archives of Computational Methods in Engineering. 29: 583–607. DOI: 10.1007/s11831-021-09587-6.
S. Rahman, M. M. Rahman, K. Hussain, S. M. Khaled, and M. Shoyaib. 2014. “Image enhancement in spatial domain: A comprehensive study,” Conf. Comput. Inf. Technol. ICCIT, 1–6. DOI: 10.1109/ICCITechn.2014.7073123.
G. Singh and A. Mittal 2014. “Various Image Enhancement Techniques- A Critical Review,” International journal of innovation and Scientific Research, 10: 1–8
Z. Shi, Y. Feng, M. Zhao, and L. He 2019, “A joint deep neural networks-based method for single nighttime rainy image enhancement,” Neural Computing and Applications. 1–14. DOI: 10.1007/s00521-019-04501-5.
S. Joshi, R. Arindom, T. Dikshit, B. Anish, A. G. Deep, and P. Pallav. 2015. “Image Enhancement Techniques: A Study,” Indian Journal of Science and Technology . 8: 1–12. DOI: 10.17485/ijst/2015/v8i.
S. A. Nossier, J. Wall, M. Moniri, C. Glackin, and N. Cannings 2014. “An experimental analysis of deep learning architectures for supervised speech enhancement,” Electron, 10: 1–32. DOI: 10.3390/electronics10010017.
A. Ignatov, L. Gool, and R. Timofte 2020. “Replacing mobile camera isp with a single deep learning model,” IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops Work, 1–11. DOI: 10.1109/CVPRW50498.2020.00276.
A. Ignatov, N. Kobyshev, R. Timofte, K. Vanhoey, and L. Gool 2017., “DSLR-Quality Photos on Mobile Devices with Deep Convolutional Networks,” IEEE International Conference on Computer Vision (ICCV), 1–16. DOI: 10.1109/ICCV.2017.355.
J. Y. Zhu, T. Park, P. Isola, and A. A. Efros. 2017. “Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks,” IEEE International Conference on Computer Vision 2242–225. DOI: 10.1109/ICCV.2017.244.
D. Ngo, S. Lee, Q. H. Nguyen, T. M. Ngo, G. D. Lee, and B. Kang 2020 “Single image haze removal from image enhancement perspective for real-time vision-based systems,” Sensors (switzerl.,. 1–21. DOI: 10.3390/s20185170.
F. Hussain and J. Jeong 2016. “Visibility Enhancement of Scene Images Degraded by Foggy Weather Conditions with Deep Neural Networks,” Journal of Sensor. 2016: 1–10. DOI: 10.1155/2016/3894832.
H. Wang, Y. Li, and H. Moon 2021 “Robust Korean License Plate Recognition Based on Deep Neural Networks,” Sensors (swit-zerl., 1–18. DOI: 10.3390/s21124140.
C. Li, J. Zhu, L. Bi, W. Zhang, and Y. Liu 2022. “A low-light image enhancement method with brightness balance and detail preservation,” PLoS ONE, 17.
L. H. Pham and ; Duong Nguyen-Ngoc Tran; Jae Wook Jeon 2020 “Low-Light Image Enhancement for Autonomous Driving Systems using DriveRetinex-Net,” IEEE Int. Conf. Consum. Electron. - Asia, pp. 1–5. DOI: 10.1109/ICCE-Asia49877.2020.9277442.
S. Akintayo, G. L. Adedotun, and S. Kin 2016. “LLNet: A Deep Autoencoder approach to Natural Low-light Image Enhancement,” arXiv. 1–11,. [Online]. Available:https://arxiv.org/pdf/1511.03995.pdf
L. Ma, T. Ma, R. Liu, X. Fan, and Z. Luo 2022 “Toward Fast, Flexible, and Robust Low-Light Image Enhancement,” IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 1–10,.
W. Ren 2019. “Low-Light Image Enhancement via a Deep Hybrid Network,” IEEE Transactions on Image Processing. 28: 1–12. DOI: 10.1109/TIP.2019.2910412.
J. Mukhtarjee, K. Praveen, and V. Madumbu 2018. “Visual Quality Enhancement of Images under Adverse Weather Conditions,” 21st International Conference on Intelligent Transportation Systems (ITSC), ITSC, 1–8. DOI: 10.1109/ITSC.2018.8569536.
J. Younkwan, J. Lee, Y. Jeon, B. Ko, and M. Jeon 2021., “Task-Driven Deep Image Enhancement Network for Autonomous Driving in Bad Weather,” IEEE International Conference on Robotics and Automation (ICRA), 1–8. DOI: 10.1109/ICRA48506.2021.9561076.
T. Matsui, T. Fujisawa, T. Yamaguchi, and M. Ikehara 2018. “Single-Image Rain Removal Using Residual Deep Learning,” 25th IEEE International Conference on Image Processing (ICIP). 1–5. DOI: 10.1109/ICIP.2018.8451612.
Z. Wang, A. C. Bovik, H. R. Sheikh, S. Member, E. P. Simoncelli, and S. Member 2004., “Image Quality Assessment : From Error Visibility to Structural Similarity,” IEEE Transactions on Image Processing, 13: 1–13. DOI: 10.1109/TIP.2003.819861.
U. Sara, M. Akter, and M. S. Uddin. 2019, “Image Quality Assessment through FSIM , SSIM , MSE and PSNR — A Comparative Study,” Journal of Computer and Communications, 9: 1–11. DOI: 10.4236/jcc.2019.73002.
Z. Wang and A. C. Bovik 2002 “A universal image quality index,” IEEE Signal Process, 9: 1–4. DOI: 10.1109/97.995823.
D. Asamoah, E. Ofori, S. Opoku, and J. Danso 2018. “Measuring the Performance of Image Contrast Enhancement Technique,” International Journal of Computer Applications,. 181: 1–8,.
S. Minaee, Y. Boykov, and F. Porikli 2022., “Image Segmentation Using Deep Learning : A Survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence 44, DOI: 10.1109/TPAMI.2021.3059968.
Z. Zhou. 2019. “Attention based Stack ResNet for Citywide Traffic Accident Prediction,” IEEE International Conference on Mobile Data Management (MDM), 0–2. DOI: 10.1109/MDM.2019.00-27.
A. K. Agrawal, K. Agarwal, J. Choudhary, A. Bhattacharya, and S. Tangudu 2020. “Automatic Traffic Accident Detection System Using ResNet and SVM,” Fifth International Conference on Research in Computational Intelligence and Communication Networks (ICRCICN) pp. 1–6. DOI: 10.1109/ICRCICN50933.2020.9296156.
Z. Zhang 2021. “ResNet-Based Model for Autonomous Vehicles Trajectory Prediction,” IEEE International Conference on Consumer Electronics and Computer Engineering (ICCECE), pp. 1–4. DOI: vv10.1109/ICCECE51280.2021.9342418.
G. J. Brostow, J. Fauqueur, and R. Cipolla 2009, “Semantic object classes in video: A high-definition ground truth database,” Pattern Recognition Letters,. 1–10. DOI: 10.1016/j.patrec.2008.04.005.
M. R. Shortis, J. W. Seager, E. S. Harvey, and S. Robson 2005 “Influence of Bayer filters on the quality of photogrammetric measurement,” VideometricsVII: 1–9. DOI: 10.1117/12.588217.
R. M. Vogel 2020., “The geometric mean?,” Communication in Statistics- Theory and Methods, 1–14. DO: 10.1080/03610926.2020.1743313.
G. G. S. 2023. “Relationship Between Arithmetic Mean and Geometric Mean: Types, Differences, and Solved Examples,” EMBIBE.
