A High-Precision Method for Segmentation and Recognition of Shopping Mall Plans
<p>The entire procedure for segmenting and recognizing a shopping mall plan is presented. The Directory Text Matching System enters directory pictures to generate matching text, and the shopping mall plan is entered into the Shop Mall Plan Matching System to segment the region. Finally, the matched text is employed to aid recognition in order to acquire the information for each region.</p> "> Figure 2
<p>The procedure for the Directory Text Matching System is presented: (<bold>a</bold>) represents the original image, (<bold>b</bold>) represents the perspective correction results, (<bold>c</bold>) represents the unprocessed recognition results, and (<bold>d</bold>) represents the processed recognition results.</p> "> Figure 3
<p>The two boxes represent the detected characters’ smallest rectangular box. Using the position of the left character as a guide, when the right character box confirms that <inline-formula><mml:math display="block" id="mm146"><mml:semantics><mml:mrow><mml:msub><mml:mi>y</mml:mi><mml:mi>r</mml:mi></mml:msub><mml:mo>∈</mml:mo><mml:mi>δ</mml:mi></mml:mrow></mml:semantics></mml:math></inline-formula>, it is assumed that the two characters are roughly in the same horizontal position and can be matched.</p> "> Figure 4
<p>The entire procedure for the Shopping Mall Plan Matching System is presented. The preprocessing module uses the shopping mall plan as an input to generate a binary picture. In the segmentation module, the binary image is segmented to obtain each region to be identified. Each region is passed to the recognition module, in turn, to identify the room number, and the matching text is then traversed to acquire the corresponding region’s room name. After a successful match, the region is marked and the region’s information (room name and location) is returned.</p> "> Figure 5
<p>The preprocessing result is presented: (<bold>a</bold>,<bold>b</bold>) represent a partial enlarged view taken from <inline-formula><mml:math display="block" id="mm147"><mml:semantics><mml:mrow><mml:msub><mml:mi>F</mml:mi><mml:mn>0</mml:mn></mml:msub></mml:mrow></mml:semantics></mml:math></inline-formula>, (<bold>c</bold>) denotes the shopping mall plan <inline-formula><mml:math display="block" id="mm148"><mml:semantics><mml:mrow><mml:msub><mml:mi>F</mml:mi><mml:mn>0</mml:mn></mml:msub></mml:mrow></mml:semantics></mml:math></inline-formula>, (<bold>d</bold>) denotes the preprocessing result <inline-formula><mml:math id="mm149"><mml:semantics><mml:mrow><mml:msub><mml:mi>F</mml:mi><mml:mn>1</mml:mn></mml:msub></mml:mrow></mml:semantics></mml:math></inline-formula>, (<bold>e</bold>) denotes the result of exposure point detection <inline-formula><mml:math id="mm150"><mml:semantics><mml:mrow><mml:msub><mml:mi>F</mml:mi><mml:mrow><mml:mi>b</mml:mi><mml:mn>2</mml:mn></mml:mrow></mml:msub></mml:mrow></mml:semantics></mml:math></inline-formula>, (<bold>f</bold>) denotes the result of canny edge detection <inline-formula><mml:math display="block" id="mm151"><mml:semantics><mml:mrow><mml:msub><mml:mi>F</mml:mi><mml:mrow><mml:mi>b</mml:mi><mml:mn>3</mml:mn></mml:mrow></mml:msub></mml:mrow></mml:semantics></mml:math></inline-formula>, and (<bold>g</bold>) denotes the result of adaptive threshold segmentation <inline-formula><mml:math display="block" id="mm152"><mml:semantics><mml:mrow><mml:msub><mml:mi>F</mml:mi><mml:mrow><mml:mi>b</mml:mi><mml:mn>1</mml:mn></mml:mrow></mml:msub></mml:mrow></mml:semantics></mml:math></inline-formula>.</p> "> Figure 6
<p>Schematic diagram of two-stage region growing algorithm, taking the “C37” region as an example. (<bold>a</bold>) represents the traditional region growing process, and (<bold>b</bold>) represents the two-stage region growing process.</p> "> Figure 7
<p>The white region denotes the corridor after region growing; <italic>M</italic>, <italic>N</italic> denote the size of <inline-formula><mml:math display="block" id="mm153"><mml:semantics><mml:mrow><mml:msub><mml:mi>F</mml:mi><mml:mn>1</mml:mn></mml:msub></mml:mrow></mml:semantics></mml:math></inline-formula>, and <italic>m</italic>, <italic>n</italic> denote the corridor’s minimum rectangle size.</p> "> Figure 8
<p>(<bold>a</bold>) represents Epoch One’s detected results <inline-formula><mml:math display="block" id="mm154"><mml:semantics><mml:mrow><mml:msub><mml:mi>F</mml:mi><mml:mrow><mml:mrow><mml:mi>detected</mml:mi><mml:mtext>_</mml:mtext><mml:mi>mask</mml:mi></mml:mrow></mml:mrow></mml:msub></mml:mrow></mml:semantics></mml:math></inline-formula>, (<bold>b</bold>) represents Epoch Two’s preprocessing results <inline-formula><mml:math id="mm155"><mml:semantics><mml:mrow><mml:msubsup><mml:mi>F</mml:mi><mml:mn>2</mml:mn><mml:mo>′</mml:mo></mml:msubsup></mml:mrow></mml:semantics></mml:math></inline-formula>, and (<bold>c</bold>) represents the segmentation module’s input <inline-formula><mml:math id="mm156"><mml:semantics><mml:mrow><mml:msub><mml:mi>F</mml:mi><mml:mn>2</mml:mn></mml:msub></mml:mrow></mml:semantics></mml:math></inline-formula> of Epoch Two.</p> "> Figure 9
<p>The flow chart of multi-epoch segmentation and recognition is shown.</p> "> Figure 10
<p>(<bold>a</bold>) representation of the raw picture, and (<bold>b</bold>) representation of the outcomes of segmentation and recognition, following two epochs of segmentation and recognition. The regions of successful segmentation and recognition is filled with purple and labeled with the associated room number, whereas the successful segmentation (incomplete segmentation) but incorrect recognition area is labeled with red.</p> "> Figure 11
<p>Shopping mall plan. This can be downloaded at <uri>https://www.carrefourdunord.com/en/informations/sitemap</uri> (accessed on 10 October 2021).</p> "> Figure 12
<p>(<bold>a</bold>) represents the one-epoch result with traditional region growing; (<bold>b</bold>) represents the two-epoch result with traditional region growing; (<bold>c</bold>) represents the one-epoch result with two-stage region growing; (<bold>d</bold>) represents the two-epoch result with two-stage region growing. The purple-filled region denotes a room that has been successfully segmented and identified, whereas the red-filled area denotes a room that has been successfully segmented but incorrectly recognized. Rooms that are not successfully segmented still retain the colors in the original picture (without red and purple filling).</p> "> Figure 13
<p>Special case analysis. (<bold>a</bold>) represents the room number outside the room. (<bold>b</bold>) represents that the room number that is connected with irrelevant characters. (<bold>c</bold>) represents the plan with the lowest accuracy in the dataset, and the purple area represents the rooms successfully segmented and identified.</p> ">
Abstract
:1. Introduction
2. Methods
2.1. The Directory Text Matching System
Algorithm 1 Directory text matching algorithm |
Input: |
Output: |
1: |
2: |
3: |
4: |
5: foreach do |
6: |
7: for do |
8: |
9: |
10: |
11: |
2.1.1. Preprocessing
2.1.2. OCR Recognition and Matching
2.2. The Shopping Mall Plan Matching System
2.2.1. Preprocessing Module
2.2.2. Segmentation Module
Algorithm 2 Two-stage region growing algorithm |
Input: |
Output: |
1: |
2: |
3: |
4: |
5: |
6: |
7: |
8: |
2.2.3. Recognition Module
2.2.4. Multi-Epoch Segmentation and Recognition Algorithm
3. Experimental Process and Results
3.1. Introduction of Experimental Evaluation Criteria and Experimental Data
3.2. Experiment with Different OCR Systems
3.3. Experiment of Two-Stage Region Growing
3.4. Experiment of Multi-Epoch Segmentation and Recognition
4. Discussion
5. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
References
- de las Heras, L.-P.; Mas, J.; Sanchez, G.; Valveny, E. Wall Patch-Based Segmentation in Architectural Floorplans. In Proceedings of the 2011 11th International Conference on Document Analysis and Recognition (ICDAR 2011), Beijing, China, 18–21 September 2011; pp. 1270–1274, ISBN 978-1-4577-1350-7. [Google Scholar]
- Wijmans, E.; Furukawa, Y. Exploiting 2D Floorplan for Building-Scale Panorama RGBD Alignment. In Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, 21–26 July 2017; pp. 1427–1435, ISBN 978-1-5386-0457-1. [Google Scholar]
- Lladós, J.; López-Krahe, J.; Martí, E. A system to understand hand-drawn floor plans using subgraph isomorphism and Hough transform. Mach. Vis. Appl. 1997, 10, 150–158. [Google Scholar] [CrossRef]
- Bormann, R.; Jordan, F.; Li, W.; Hampp, J.; Hägele, M. Room segmentation: Survey, implementation, and analysis. In Proceedings of the 2016 IEEE International Conference on Robotics and Automation (ICRA), Stockholm, Sweden, 16–21 May 2016; pp. 1019–1026. [Google Scholar]
- Mewada, H.K.; Patel, A.V.; Chaudhari, J.; Mahant, K.; Vala, A. Automatic room information retrieval and classification from floor plan using linear regression model. Int. J. Doc. Anal. Recognit. 2020, 23, 253–266. [Google Scholar] [CrossRef]
- Ahmed, S.; Weber, M.; Liwicki, M.; Langenhan, C.; Dengel, A.; Petzold, F. Automatic analysis and sketch-based retrieval of architectural floor plans. Pattern Recognit. Lett. 2014, 35, 91–100. [Google Scholar] [CrossRef]
- Wang, W.; Dong, S.; Zou, K.; LI, W. Room Classification in Floor Plan Recognition. In Proceedings of the 2020 4th International Conference on Advances in Image Processing, Chengdu China, 13–15 November 2020; ACM: New York, NY, USA, 2020; pp. 48–54, ISBN 9781450388368. [Google Scholar]
- de las Heras, L.-P.; Ahmed, S.; Liwicki, M.; Valveny, E.; Sánchez, G. Statistical segmentation and structural recognition for floor plan interpretation. Int. J. Doc. Anal. Recognit. 2014, 17, 221–237. [Google Scholar] [CrossRef]
- Macé, S.; Locteau, H.; Valveny, E.; Tabbone, S. A system to detect rooms in architectural floor plan images. In Proceedings of the 8th IAPR International Workshop on Document Analysis Systems—DAS ′10, Boston, MA, USA, 9–11 June 2010; Doermann, D., Govindaraju, V., Lopresti, D., Natarajan, P., Eds.; ACM Press: New York, NY, USA, 2010; pp. 167–174, ISBN 9781605587738. [Google Scholar]
- Ahmed, S.; Liwicki, M.; Weber, M.; Dengel, A. Automatic Room Detection and Room Labeling from Architectural Floor Plans. In Proceedings of the 2012 10th IAPR International Workshop on Document Analysis Systems (DAS 2012), Gold Coast, QLD, Australia, 27–29 March 2012; IEEE: Piscataway, NJ, USA, 2012; pp. 339–343, ISBN 978-0-7695-4661-2. [Google Scholar]
- Ahmed, S.; Liwicki, M.; Weber, M.; Dengel, A. Improved Automatic Analysis of Architectural Floor Plans. In Proceedings of the 2011 International Conference on Document Analysis and Recognition, Beijing, China, 18–21 September 2011; pp. 864–869, ISBN 2379-2140. [Google Scholar]
- Ahmed, S.; Liwicki, M.; Dengel, A. Extraction of Text Touching Graphics Using SURF. In Proceedings of the 2012 10th IAPR International Workshop on Document Analysis Systems, Gold Coast, QLD, Australia, 27–29 March 2012; pp. 349–353. [Google Scholar]
- Rosin, P.L.; West, G.A.W. Segmentation of edges into lines and arcs. Image Vis. Comput. 1989, 7, 109–114. [Google Scholar] [CrossRef]
- Wang, L.; Chen, X.; Hu, L.; Li, H. Overview of Image Semantic Segmentation Technology. In Proceedings of the 2020 IEEE 9th Joint International Information Technology and Artificial Intelligence Conference (ITAIC), Chongqing, China, 11–13 December 2020; pp. 19–26, ISBN 978-1-7281-5244-8. [Google Scholar]
- Zhang, Y.J. Image Segmentation in the Last 40 Years. In Encyclopedia of Information Science and Technology, 2nd ed.; Khosrow-Pour, D., Ed.; IGI Global: Beijing, China, 2009; pp. 1818–1823. ISBN 9781605660264. [Google Scholar]
- He, H.; Qingwu, L.; Xijian, F. Adaptive local threshold image sefmentation algorithm. Optoelectron. Technol. 2011, 31, 10–13. [Google Scholar]
- Tremeau, A.; Borel, N. A region growing and merging algorithm to color segmentation. Pattern Recognit. 1997, 30, 1191–1203. [Google Scholar] [CrossRef]
- Han, J.; Duan, X.; Chang, Z. Target Segmentation Algorithm Based on SLIC and Region Growing. Comput. Eng. Appl. 2021, 57, 213–218. [Google Scholar]
- Khan, J.F.; Bhuiyan, S.M.A.; Adhami, R.R. Image Segmentation and Shape Analysis for Road-Sign Detection. IEEE Trans. Intell. Transp. Syst. 2011, 12, 83–96. [Google Scholar] [CrossRef]
- Rosenfeld, A. The Max Roberts Operator is a Hueckel-Type Edge Detector. IEEE Trans. Pattern Anal. Mach. Intell. 1981, 3, 101–103. [Google Scholar] [CrossRef] [PubMed]
- Lang, Y.; Zheng, D. An Improved Sobel Edge Detection Operator. In Proceedings of the 2016 6th International Conference on Mechatronics, Computer and Education Informationization (MCEI 2016), Shenyang, China, 11–13 November 2016; Atlantis Press: Paris, France, 2016. [Google Scholar]
- Liu, L.X.; Li, B.W.; Wang, Y.P.; Yang, J.Y. Remote sensing image segmentation based on improved Canny edge detection. Comput. Eng. Appl. 2019, 55, 54–58. [Google Scholar]
- Ronneberger, O.; Fischer, P.; Brox, T. U-Net: Convolutional Networks for Biomedical Image Segmentation. In Medical Image Computing and Computer-Assisted Intervention—MICCAI 2015: 18th International Conference, Munich, Germany, 5–9 October 2015; Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F., Eds.; Springer: Cham, Switzerland, 2015; pp. 234–241. ISBN 978-3-319-24573-7. [Google Scholar]
- Shelhamer, E.; Long, J.; Darrell, T. Fully Convolutional Networks for Semantic Segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 2017, 39, 640–651. [Google Scholar] [CrossRef] [PubMed]
- Memon, J.; Sami, M.; Khan, R.A.; Uddin, M. Handwritten Optical Character Recognition (OCR): A Comprehensive Systematic Literature Review (SLR). IEEE Access 2020, 8, 142642–142668. [Google Scholar] [CrossRef]
- Mittal, R.; Garg, A. Text extraction using OCR: A Systematic Review. In Proceedings of the 2020 Second International Conference on Inventive Research in Computing Applications (ICIRCA), Coimbatore, India, 15–17 July 2020; pp. 357–362, ISBN 978-1-7281-5374-2. [Google Scholar]
- LeCun, Y.; Bengio, Y.; Hinton, G. Deep learning. Nature 2015, 521, 436–444. [Google Scholar] [CrossRef] [PubMed]
- Breuel, T.M.; Ul-Hasan, A.; Al-Azawi, M.A.; Shafait, F. High-Performance OCR for Printed English and Fraktur Using LSTM Networks. In Proceedings of the 2013 12th International Conference on Document Analysis and Recognition (ICDAR), Washington, DC, USA, 25–28 August 2013; pp. 683–687, ISBN 978-0-7695-4999-6. [Google Scholar]
- Tencent Cloud General Optical Character Recognition. Available online: https://cloud.tencent.com/product/generalocr (accessed on 10 October 2021).
- Baidu Cloud General Optical Character Recognition. Available online: https://cloud.baidu.com/product/ocr/general (accessed on 10 October 2021).
- AdvancedEAST. Available online: https://github.com/huoyijie/AdvancedEAST (accessed on 10 October 2021).
- Deng, D.; Liu, H.; Li, X.; Cai, D. PixelLink: Detecting Scene Text via Instance Segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, New Orleans, LA, USA, 2–7 February 2018; Volume 32. [Google Scholar]
- Pixellink. Available online: https://github.com/ZJULearning/pixellink (accessed on 10 October 2021).
- CTPN. Available online: https://github.com/tianzhi0549/CTPN (accessed on 10 October 2021).
- Tian, Z.; Huang, W.; He, T.; He, P.; Qiao, Y. Detecting Text in Natural Image with Connectionist Text Proposal Network. In Computer Vision—ECCV 2016; Leibe, B., Matas, J., Sebe, N., Welling, M., Eds.; Springer International Publishing: Cham, Switzerland, 2016; pp. 56–72. ISBN 978-3-319-46483-1. [Google Scholar]
- EasyOCR. Available online: https://github.com/JaidedAI/EasyOCR (accessed on 10 October 2021).
- Baek, Y.; Lee, B.; Han, D.; Yun, S.; Lee, H. Character Region Awareness for Text Detection. In Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, 15–20 June 2019; pp. 9357–9366, ISBN 978-1-7281-3293-8. [Google Scholar]
- Shi, B.; Bai, X.; Yao, C. An End-to-End Trainable Neural Network for Image-Based Sequence Recognition and Its Application to Scene Text Recognition. IEEE Trans. Pattern Anal. Mach. Intell. 2017, 39, 2298–2304. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Lu, S.; Chen, B.M.; Ko, C.C. Perspective rectification of document images using fuzzy set and morphological operations. Sci. Image Vis. Comput. 2005, 23, 541–553. [Google Scholar] [CrossRef]
Target | Segmentation | Recognition | Recognition | ||||||
---|---|---|---|---|---|---|---|---|---|
1 | | | | √ | √ | C37: cafes | None | None | |
2 | | | | √ | Partly | C1: × | | C17: dress shops | |
3 | | | | √ | Partly | × | | CK02: EBGames | |
4 | | | | × | √ | × | | None | None |
Test samples | | | | |
Test results | | | | |
Image | EasyOCR | Baidu OCR | BOTH | ||||||
---|---|---|---|---|---|---|---|---|---|
DSR | ISR | DSR | ISR | DSR | ISR | ||||
1 | 0.613636 | 0.627907 | 72 s | 0.795455 | 0.813953 | 124 s | 0.818182 | 0.837209 | 114 s |
2 | 0.675676 | 0.937500 | 111 s | 0.675676 | 0.937500 | 188 s | 0.675676 | 0.937500 | 179 s |
Dataset | 0.494776 | 0.619626 | 92 s | 0.574627 | 0.719626 | 127 s | 0.667910 | 0.836449 | 132 s |
Image | Region Growing | Two-Stage Region Growing | ||||||
---|---|---|---|---|---|---|---|---|
DSR | SSR | ISR | DSR | SSR | ISR | |||
1 | 0.000000 | 0.954545 | 0.000000 | 106 s | 0.818182 | 0.977273 | 0.837209 | 114 s |
2 | 0.000000 | 0.801802 | 0.000000 | 174 s | 0.675676 | 0.720721 | 0.937500 | 179 s |
Dataset | 0.041045 | 0.759701 | 0.054028 | 125 s | 0.667910 | 0.798507 | 0.836449 | 132 s |
Image | Epoch One | Epoch Two | ||||||
---|---|---|---|---|---|---|---|---|
DSR | SSR | ISR | DSR | SSR | ISR | |||
1 | 0.000000 | 0.954545 | 0.000000 | 106 s | 0.568182 | 0.977273 | 0.581395 | 105 s |
2 | 0.000000 | 0.801802 | 0.000000 | 174 s | 0.036036 | 0.981982 | 0.036697 | 221 s |
Dataset | 0.041045 | 0.759701 | 0.054028 | 125 s | 0.220149 | 0.907463 | 0.242599 | 140 s |
Image | Epoch One | Epoch Two | ||||||
---|---|---|---|---|---|---|---|---|
DSR | SSR | ISR | DSR | SSR | ISR | |||
1 | 0.818182 | 0.977273 | 0.837209 | 114 s | 0.954545 | 0.977273 | 0.976744 | 61 s |
2 | 0.675676 | 0.720721 | 0.937500 | 179 s | 0.936973 | 0.936973 | 1.000000 | 87 s |
Dataset | 0.667910 | 0.798507 | 0.836449 | 132 s | 0.838060 | 0.925373 | 0.905645 | 121 s |
Image | |||||
---|---|---|---|---|---|
1 | 42 | 0 | 1 | 1 | 44 |
2 | 104 | 0 | 0 | 7 | 111 |
Dataset | 1123 | 18 | 99 | 100 | 1340 |
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Su, M.; Shi, W.; Zhao, D.; Cheng, D.; Zhang, J. A High-Precision Method for Segmentation and Recognition of Shopping Mall Plans. Sensors 2022, 22, 2510. https://doi.org/10.3390/s22072510
Su M, Shi W, Zhao D, Cheng D, Zhang J. A High-Precision Method for Segmentation and Recognition of Shopping Mall Plans. Sensors. 2022; 22(7):2510. https://doi.org/10.3390/s22072510
Chicago/Turabian StyleSu, Ming, Wei Shi, Dangjun Zhao, Dongyang Cheng, and Junchao Zhang. 2022. "A High-Precision Method for Segmentation and Recognition of Shopping Mall Plans" Sensors 22, no. 7: 2510. https://doi.org/10.3390/s22072510