MASSACHUSETTS INSTITUTE OF TECHNOLOGY ARTIFICIAL INTELLIGENCE LABORATORY Artificial Intelligence Memo Number 285 July 1971 Revised December 1973 The Binford-Horn LINE-FINDER * Berthold K. P. Horn ABSTRACT This paper briefly describes the processing performed in the course of producing a line drawing from an image obtained through an image dissector camera. The edge-marking phase uses a non-linear parallel line-follower. Complicated statistical measures are not used. The line and vertex generating phases use a number of heuristics to guide the transition from edge-fragments to cleanedup line-drawing. Higher-level understanding of the blocks-world is not used. Sample line-drawings produced by the program are included. . Work reported herein was conducted at the Artificial Intelligence Laboratory, a Massachusetts Institute of Technology research program supported in part by the Advanced Research Projects Agency of the Department of Defense and monitored by the Office of Naval Research under Contract Number N00014-70-A-0362-0005. Reproduction of this document, in whole or in part, is permitted for any purpose of the United States Government. II t - ,- i 1 I mo..~ ,,-... -- -(DM r- I maws a - -- -I I na1a4 (*.re* - * r'l.rU1m Q) C d-' d-' cd ' d Q) s 4 + s X O C , C, -,-I m 0 C Q C ) z s : C , & W a , . r & O E o I w ) C Q G + ) .. C n c d 8 a t n u 3 4 f - 4 + C I ) a c , PAGE 6 1 1 Line-finders can also be classified according to whether or "* not the filter function (also called the local predicate) applied to the image is linear. If visual noise were spatially independent, a case could be made for an optimal linear predicate. This however is not the case, since smudges, for example, have a distinctly non-random spatial distribution. Non-linear methods, while more complex, have a clear advantage on real images. Line-finders can be further classified according to whether they use a fixed raster scan or a search pattern which follows lines. The first method is convenient from the point of view of reading intensity values asynchronously ahead of time and also for program simplicity. In some cases the imaging device forces the use of a raster scan, particularly if it is an integrating device. Line-followers on the other hand can be made to be more sensitive and accurate at the price of program complexity. 3. - EDGE-MARKING. Input is obtained from a random-access image-dissector camera or a picture stored as a disk-file. linear parallel line-follower. The edge-marker is a non- That is, several lines are followed simultaneously, while the image is being scanned in a raster-like fashion, thus combining the good features of raster-scan with those of line-following. The separation between scan-lines is larger than the spacing of image points sampled along a line. Three scan- U1 Q) a, k r) 2 0 Q) C fJ I-. 3 w cd Q) w Q) C 0 M C d 0) P " 0) 3 I-. .-I cd Q) C cd C d C w 0 C w d a o + ' a C (d X 4 R d .r: - . % d .-I & 0 d V) r 0 U w d cd Q) 4 X O a C O N 3C d .,-I m C, C a o U C a O C Q) C d u U Q & C , R ) a 4 V) X u C o C , w C , W d C -rc w d x 4 0 X 2 0 U d w 0 m C d a 3 M M C w d P 3 , c d I-. 0 ) m C c d Q ) w C , O & a C, 4 V C I-4 s w w x o Q ) C , & O W -d C, C O C a C at , W C C Q ) d 3 0 w a z W D W C C, 5 t ' C, V) Q) 2 k 2 C0 0 a r n a Q) d . c % w rc w I-4 a , a o h cd w PAGE 8 -* We end up with a number of lists of tentative edge-points. Some of these lists will represent more than one edge (an L-joint for example), while some edges will be represented by more than one. list (a fragmented edge for example). These lists will often not include points very close to vertices, except at L-joints. This is because lines are prevented from encroaching too closely on each other by the nature of the correlation process and some of the tests described. Each list of tentative edge-points has associated with it a number of attributes such as type, direction and size of the intensity transition and how sure the program is that the edgefragments are not merely due to noise. Up to this stage not very many heuristics have to be invoked and consequently the possibility of corrupting the.data is small. -'% Numerous line-finders have been developed to this stage, few however produce as clean a set of tentative edge-points as T. Binfordts program. Very few line-finders have proceeded beyond this level to actually create the kind of useful cleaned-up linedrawing required by current higher-level scene-analysis programs. In part this is often due to inadequate edge-marking, but more often it is because unexpected difficulties are met when dealing with what at first sight seems a simple process: forcing the data into the form of a line-drawing with well-defined vertices. Q) C C, 4 d C, 5 cn a Q) Q) r) 0 k a C 0 d C, cd C, C 0 a M Q) LA w C, C d 0 a 1 0 00 a Q) ccc 0 C, V) d 4 Q) C C, V.r 0 (A C, C d 0 a PAGE 10 - -. stringent tests. yet to come. The less conservative and more difficult part is The data at this stage is quite compact already apd might profitably be handed to a program with some understanding of i blocks-world scenes instead of the somewhat arbitrary heuristic program described next. 5. - GENERATING VERTICES. Some of the vertices are clearly indicated by the close convergence of lines. A first estimate of the location of such a vertex is made by considering the center of gravity of the endpoints of the lines (if there are only two lines, their intersection is used instead). - A search is then made for all lines which end near this point and whose extension would pass very close to it. The vertex is finally declared at the point of least- squares perpendicular distance from these lines [Horn 19721. The appropriate lines are then connected to the vertex. Next, the program tries to establish T-joints. That is, it searches for lines that have an end-point close to another line and checks if it would we reasonable to extend the line to form a new vertex. line. - A K-joint is similarly made where a vertex is close to a Finally, crossing lines give rise to X-joints. Numerous heuristics inform this process; for example, lines already connected at one end are treated preferentially to those still free a t both ends. All vertices have now been found and an attempt is PAGE 11 made to extend unattached lines to nearby vertices. __I* The extension must be short and pass very close to the vertex to be accepted. Following this, vertices which are close together are conglomerated. discarded. Any lines not properly attached at this stage are The progress of these steps in the processing can be slowed down and viewed on a display, before the final product is disgorged in a convenient LISP-readable format. The heuristics, which developed empirically, depend on certain - tolerances which are initially calculated in terms of the line-scan interval, the known resolution of the imaging device, and the signal-to-noise ratio. These factors could be "tunedw to improve performance and accuracy, but this is probably not worth the effort (the time would be better spent on designing new ways of directing the vertex creation phase). A large portion of this assembly language program is concerned with debugging, displaying and performing the required list-processing. maintained in two forms: The data-base is As assertions about lines (and what vertices they connect to) and as assertions about vertices (and what lines impinge on them). These two forms are kept consistent by demons invoked vhen assertions are added, removed or changed in either data-base. The line and vertex generating phases of the program frequently need to settle questions of proximity for both points and lines. A set of four superimposed rectangular grids covering / PAGE 12 A* the image is used as a set of buckets for this purpose. This method is sometimes referred to as multi-entry coding [Horn 19721. Each point is entered into four buckets, and each line will appear in many. This makes for high speed, despite the need for a number of iterative applications of various heuristics to the whole datastructure. 6. - PERFORMANCE. An idea about how to produce line-drawings from images is of little use until it has been demonstrated by a working program. The program discussed here has processed over a hundred scenes and produces excellent line-drawings of simple blocks-world scenes. In more complicated cases, a number of shortcomings can be observed. The simplest and easiest to deal with is the absence or incompleteness of some of the lines, usually due to a lack of contrast between adjacent faces of an object. At times extra lines are introduced by shadows, smudges and noise. There is a trade-off between these two effects, and since present scene-analysis programs can handle missing lines better than extra lines, the threshold is set to favour the former, Occasionally a section of a line-drawing will be garbled, usually due to the combination of two unrelated vertices. This causes some distortion of the lines and may make the line-drawing locally uninterpretable. This last effect in particular PAGE 13 function of the amount of detail and can be avoided by using a finer resolution at the cost of an increase in scan and computation time. This of course is only possible if the imaging device has sufficient resolution. 7. COMMENTS 9EJ THIS APPROACH. Some of the ways in which images of sets of polyhedra differ from our intuitive model of equal intensity polygonal areas have important implications in other areas. Mutual illumination for example will prove to be a problem when one is developing a program that exhibits color constancy. Further, it should be noted that some of the edges missed by this program will also be missed by line-verifiers: sensitive. when following lines one can afford to be quite Fortunately the better line-proposers are very conservative and hardly ever propose a line were there shouldn't be one. One could perhaps accept their proposals without attempting verification. The only features that can be reliably determined from a corrupted image are those with significant spatial extension. Without such extension we cannot apply the integrative processes necessary to collect evidence for the existence of the feature. Vertices for this reason are perhaps not primitive elements of an image, but exist only as the intersection of lines. Letting the line-drawing program establish vertices introduces inaccuracies, PAGE 1 4 because it may join unrelated lines. 8. HOWTODO - BETTER. This program has no idea what a reasonable line-drawing should look like when it represents an image of polyhedra. Instead it is very general and will find arbitrary line-drawings. Observing the particular way in which things sometimes go wrong, one quickly comes to the conclusion that higher-level understanding of the scene being analysed could greatly improve the line and vertex creating phase of this program. As things stand now this understanding comes only after the line-finder has done its work. It would be hard to embed this kind of knowledge into such a large assembly language program. At the same time trying to implement the wlow-levelw routines in a higher level language would cause great inefficiencies. 9. - HISTORICAL NOTE. The line-finder described here consists of an edge-marker due to T. Binford and a line-drawer due to B. K. P, Horn. It was used as part of the system of vision and manipulation programs developed for the copy-demo by P. H. Winston, E, Freuder and B. K. P. Horn in the fall of 1 9 7 0 [Winston 1 9 7 1 6 19721. It is also the "hierarchicalu program referred to by Shirai [Shirai 19731. I i PAGE 15 10. REFERENCES. - Herskovits A. and Binford T. (1970) "On Boundary Detectionw A. I. Memo 183 (Cambridge, Mass. : A. I. Lab, M. I. T. 1 Horn B. K. P. (1969) "The Image Dissector Eyeswt A. I. Memo 178 (Cambridge, Mass.: A. I, Lab, M. I.T.) Horn B. K. P. (1972) VISMEM: A bag of 'roboticst formulaet1Vision Flash 34 (Cambridge, Mass.: A. I. Lab, M. I. T.) Shirai Y. (1973) "A Context Sensitive Line Finder for Recognition of Polyhedrav Artificial Intelligence, Vol 4, No 2. Winston P. H. (1971, 1972) Vision Flashes 7, 8, 9, 15 and 30 (Cambridge, Mass.: A. I. Lab, M. I.T. Winston P. H. (1972) "The M, I. T. Robot", Machine Intelligence 7 (Edinburgh: Edinburgh University Press)