Transportation Research Part C 21 (2012) 306–320 Contents lists available at SciVerse ScienceDirect Transportation Research Part C journal homepage: www.elsevier.com/locate/trc Online license plate matching procedures using license-plate recognition machines and new weighted edit distance Francisco Moraes Oliveira-Neto a, Lee D. Han a,⇑, Myong K. Jeong b a b Department of Civil and Environmental Engineering, The University of Tennessee, Perkins Hall 223, Knoxville, TN 37996, USA Department of Industrial and Systems Engineering and RUTCOR, Rutgers University, Piscataway, NJ 08854, USA a r t i c l e i n f o Article history: Received 15 November 2010 Received in revised form 14 November 2011 Accepted 14 November 2011 Keywords: Vehicle tracking License-plate recognition Edit distance Text mining Online plate matching Two-point plate survey a b s t r a c t License-plate recognition (LPR) technology has been widely applied in many different transportation applications such as enforcement, vehicle monitoring, and access control. Recently, there has been effort to exploit an LPR database for vehicle tracking using popular template matching procedures. Existing template matching procedures assume that the true reference string is always available. However, under a two-point LPR survey, a vehicle could have its plate misread at both locations generating a pair of misread strings (or templates) with no reference for matching. To compensate for LPR misreading problem, we propose a new weight function based on a probability model to match the observed outcomes of a dual LPR setup. Also, considering that reversal errors are never made in LPR machines, new editing constraints as a function of the string lengths are proposed to avoid compensation for reversal errors. These editing constraints are incorporated into the constraint edit distance formulation to improve the performance of the matching procedure. Finally, considering that previous template matching procedures do not take advantage of passage time information available in LPR databases, we present an online tracking procedure that considers the properties of probability distribution of vehicle journey times in order to increase the probability of correct matches. Experimental results show that our proposed procedure can improve the accuracy of LPR systems and achieve up to 97% of positive matches with no false matches. Further research is needed to extend the ideas proposed herein to plate-matching with multiple, i.e., more than two, LPR units. Published by Elsevier Ltd. 1. Introduction Among intelligent transportation systems, automated vehicle identification (AVI) is a powerful tool for electronic toll, traffic management, commercial vehicle operations, motor vehicle law enforcement, origin–destination (O–D) survey, and access control, among other applications (Nelson, 2000a). All these applications require a unique identification of a vehicle at a checkpoint; some of them require the vehicle to be tracked at multiple points, e.g., O–D survey. To this end, it would be desirable if each vehicle is required to carry a radio-frequency identification (RFID) tag, or some hardware of similar mechanism, to provide uniquely identifiable and useful vehicle and, perhaps even, driver information. This idea, however, has experienced much resistance and is unlikely to be implemented in the short term due to a number of reasons with privacy concerns being the most prohibitive. A more traditional and feasible approach is to identify vehicles by their license plates, which all vehicles are required to have, using automated license-plate recognition (LPR) systems. These systems were developed with the main objective of ⇑ Corresponding author. E-mail address: lhan@utk.edu (L.D. Han). 0968-090X/$ - see front matter Published by Elsevier Ltd. doi:10.1016/j.trc.2011.11.003 F.M. Oliveira-Neto et al. / Transportation Research Part C 21 (2012) 306–320 307 interpreting the alphanumeric characters on vehicle plates without human intervention. They typically rely on four main components: an imaging acquisition processor, a license plate detection system, a character segmentation and recognition engine and a computer to store the data (Rossetti and Baker, 2001; Wiggins, 2006). In other words, LPR operation consists in capturing images of moving vehicles, locating license plates, segmenting and recognizing plate characters, and storing information (e.g. plate images, plate numbers, time of passage, location) in a database for online verification or posterior analyses. Extensive literature is available on methods of license plate detection and character segmentation and recognition. A detailed survey can be found in Anagnostopoulos et al. (2008), and recent advances have been proposed by Chang et al. (2004), Anagnostopoulos et al. (2006), and Giannoukos et al. (2010). LPR systems have been employed in different transport applications since its launch into the commercial world in the 1980s (Nelson, 2000b). Such applications involve enforcement, vehicle monitoring, and access control (Nakanishi and Western, 2005; Rossetti and Baker, 2001). LPR can operate as a background system for electronic toll collection to indentify violators, or used to enforce speed limit over a segment of a road. In access control, vehicle plates are recognized and verified against a database to allow or deny access into a facility. In traffic monitoring, vehicles are detected at multiple points. Information such as travel time, O–D demand, and route choice, can be estimated subsequently for various planning and operational purposes. Automated LPR systems have the advantage of requiring no new devices onboard. The main disadvantage, though, is their limited accuracy in reading license plates, which predate the modern computer and sensor systems and were not designed for automated identification by machines. Some developers claim that character recognition engine is able to achieve near 100% of accuracy. However, such claims often hide important assumptions because many of tests are not conducted under realistic operational conditions found in practical applications (Nelson, 2003). The performance of LPR systems should be assessed through field testing under varying conditions of illumination, speed, camera offset angle, precipitation and so on. Usually limited resources prevent developers to perform such rigorous tests, but they should detail the conditions under which their systems achieved the stated accuracy. The truth is the potentialities of LPR are not quite realizable. Depending on the type of internal technology, installation, on-site calibration, weather, lighting, plate configuration, and a host of other conditions, LPR rarely recognizes more than 80% of the plates and often does worse than 60% (Nakanishi and Western, 2005). Fortunately, all is not lost; even when LPR fails to read a plate correctly, meaning at least one character is recognized erroneously, the system usually returns very valuable and mostly correct individual character information. By comparing the imperfectly read plate against another plate with comparable accuracy, or against a given database, one may still be able to render reasonable judgment in terms of whether there is actually a match. For instance, if two strings (sequences of characters) differ from each other by only one character, they may well have originated from the same plate. A measure of similarity between two strings can be so established as to indicate the likelihood of a match. The problem of tracking vehicles using LPR systems becomes non-trivial in applications involving vehicle tracking at two or multiple checkpoints, e.g. O–D estimation and travel time studies. Considering that the universe of possible vehicle plates is immense or unavailable, the matching mechanism must then match observations of undetermined accuracy without any reference. Such problem is similar to the partial plate survey problem (for a detailed review of the partial plate survey see for example Watling and Maher (1988, 1992), and Watling (1994)), if the LPR machine always misreads one character at the same position of each plate. But for our case in hand any character on a plate can be independently, and arguably equally, misinterpreted by the machine. Therefore a single reading could have come from multiple vehicles, making the task of plate matching in this universe of uncertainty and possibilities quite remarkable. This research focuses on the problem of matching plate readings made by a dual setup (one single O–D pair) of LPR equipments. We propose a compensation methodology based on a technique of text mining named edit distance (ED). This technique aims to measure how close two strings (sequences of characters) are from each other based on weight functions (which can be unitary values or based on statistical data) to compare each individual pair of characters. By using unitary cost functions, we are only counting the least number of distinct pair of characters when comparing two strings. Whereas, if the cost function is somewhat based on error probabilities, we actually estimate the likelihood that two pair of strings are the same. Oliveira Neto et al. (2009) have employed the edit distance to compensate for errors made by the LPR machines. They applied this technique to compare two sets of strings generated from two LPR machines located about 1.4 miles apart to recognize plate numbers of trucks traveling westbound on Interstate 40 in Knoxville, Tennessee. Their approach was able to match accurately about 80% of plates with 3% of false matches. Their results are promising as there is room for further refining their ED formulation to account for some character misinterpretations made by LPR equipments. For the purpose of LPR matching application, existing template matching procedures can be improved in three aspects. First, it is assumed that the truth, i.e., the accurate string, is always available in some database ready to be referenced. However, for multi-point LPR surveys, a vehicle plate could be misread at both locations generating a misread and erroneous reference string (or template). Thus, the true reference is unknown and the only information that can be estimated are the probabilities that certain plate characters are read or misread given by the confusion matrix of each machine (see Section 3.1 for its definition). In order to overcome this drawback, a new weight function is proposed based on the probability of finding a genuine match for a certain sequence of editing operations when comparing a pair of strings read by a dual LPR setup. To this end, an unlikely misreading receives greater penalty than a likely misreading would in the calculation of weight functions. 308 F.M. Oliveira-Neto et al. / Transportation Research Part C 21 (2012) 306–320 The second aspect of the three is related to reversal errors. LPR machines never make reversal errors, which involve the swapping of two adjacent characters. Therefore, the order in which the recognized characters appear in the observation strings should be kept throughout the edit operations when comparing pair of strings. In order to avoid reversal errors during the matching process, new editing constraints are proposed as a function of the string lengths. These editing constraints can be combined with the constraint edit distance (CED), as defined in Section 2.1, to improve the performance of matching procedure. Finally, traditional template matching procedures do not consider travel time information or the passage time information available from LPR databases. We present the vehicle tracking procedures that considers the probability distribution of travel times of a vehicle in order to increase the likelihood of correct matches and reduce survey errors. The paper is organized into five sections including this introduction. Section 2 presents a review of some popular similarity (distance) measures between two strings and some formulations for the weight (or cost) functions for the similarity measures. In Section 3 we present the proposed weight functions for the edit distance and the new matching methodology considering the travel time of vehicles. In Section 4, we compare the performance of our proposed procedures with some popular approaches using the real-life case study. Finally, Section 5 concludes the paper with future research directions. 2. Background 2.1. Similarity measures between two strings The process of matching two strings involves a sequence of comparisons of individual characters to determine the degree of similarity between two strings. In the literature of text mining, edit distance is one of the popular measures for gauging the similarity between two strings. Given two strings x and y, edit distance calculates the number of fundamental operations required to transform x into y. These fundamental operations are termed as substitutions when a character in x is replaced by the corresponding character in y; insertions when a character in y is inserted into x, thereby increasing the length of x by one character; or deletions when character in x is deleted, thereby decreasing the length of x by one character (Duda et al., 2000). The edit distance d(x ? y) between two strings x and y, can be calculated based on the following recurrent equation, as proposed by Wagner and Fischer (1974): dðx ! yÞ ¼ minfdði  1; j  1Þ þ cðxi ! yj Þ; dði  1; jÞ þ cðxi ! eÞ; dði; j  1Þ þ cðe ! yj Þg ð1Þ where d(i, j) is the edit distance between the substrings x[1, . . . , i] and y[1, . . . , j], of x and y, respectively, and d(0, 0) = 0. The c‘s are the cost functions. For example, c(xi ? yj) is the cost for the change (substitution) from xi to yj. The cost of c(xi ? e), where e represents the empty character, is incurred by a deletion of xi, and the cost of c(e ? yj) is the incurred from an insertion of yj. Therefore, the edit distance d(x ? y) would be given by d(|x|, |y|), where the notation || corresponds to the length of a string. Various extensions of the original edit distance measure have been proposed to account for different situations. The original assignment for the cost functions as proposed by Levenshtein (1966) was to set c(xi ? yj) = 0 if xi = yj, or c(xi ? yj) = 1 otherwise (xi and yj cannot be e at the same time). Ocuda et al. (1976) proposed the generalized edit distance (GED) to assign different weights to the edit operations as a function of the character or the characters involved. For example, a cost associated with the edit substitution ‘‘U’’ ? ‘‘V’’ could be smaller than the edit substitution ‘‘Q’’ ? ‘‘V’’. The error rates can be reduced by adjusting the values of the weight for each fundamental edit operation in accordance with the kinds and occurrence probabilities. In addition to weight assignments, Oommen (1986) also proposed to constraint ED by the number and types of edit operations to be included in the optimal edit transformation. He named this new approached as constraint edit distance (CED). The main idea of the CED is to search for the optimal ED subject to a certain number of substitutions, insertions, and deletions. The most recent major advance in ED calculation is Markov edit distance (MED) as proposed by Wei (2004). The main idea is to calculate ED according to lengths of sub-patterns and a simple measure that compares how close the histograms of the two sub-patterns are. The cost function in MED is defined as c(p1 ? p2), where p1 and p2 are two sub-patterns where at least one of them is not a single symbol of the alphabet. Wei pointed out that in working with sub-patterns the statistical dependencies among the values assumed by adjacent positions in patterns can be better exploited in such a way that a variety of string operations are incorporated, in addition to all operations already defined in previous literatures. The weight (or cost) functions can play an important role in the calculation of the ED measure. Several authors presented different ideas for considering the types of errors that may be present in a given application domain. In an application of handwritten text recognition, Seni et al. (1996) introduced additional operations (merge, split and pair-substitution), refined these set of operations as unlikely, likely, and very likely, and established the order of importance of the new classification of operations relative to each other. They subsequently assigned proper cost to each of the classes of operation, e.g., an unlikely deletion is more penalized than a likely deletion. Marzal and Vidal (1993) computed the weight function using the estimated probability matrix for substitutions, insertions and deletions of any pair of symbols of the alphabet for the application of hand written digit recognition. They transformed the probability matrix into weight function by computing the negative logarithm of each probability value. F.M. Oliveira-Neto et al. / Transportation Research Part C 21 (2012) 306–320 309 MED, as proposed by Wei (2004), defines the probability of a certain sequence of operations to convert a x into y as a Gibbsian probability distribution function, which in turn is defined as P(x ? y) = exp(U(x ? y)/T)/Z, where T and Z are constant parameters to be calibrated, and U(x ? y) is the energy involving in any of the sequence of edit operations to transform x into y. The most desirable configuration for transforming x into y would be the one that maximizes P(x ? y), which, in turn, is also the configuration that minimizes U(x ? y). It is easy to see that ED and GED are special cases of MED. 2.2. License plate matching application Based on the literature review on edit distance this section presents some initial considerations to the problem of matching templates read by LPR machines. Consider, for example, that a license plate whose tag is ‘‘4455HZ’’ will be read by two LPR machines located in two different locations. Suppose the LPR system reads the plate as x = ‘‘44S5H2’’ and y = ‘‘4455HZ’’ at the two respective locations. Visually these two outcomes look very similar and would not be hard to conclude that they may belong to the same plate. However, suppose that among other possible plates, another plate was observed earlier at the first station generating the following reading z = ‘‘4415HZ’’. This new outcome also looks very similar to the outcome denoted by y, adding some complication to problem. Under uncertainty about the ground truth value, we should discern which match is more likely. In the following discussion, suppose that there are only two possible matches in this example: either (x, y) or (z, y). Applying the ED’s original formulation (with zero or one cost assignments) for the two above matches, two edit operations (the substitutions ‘‘S’’ ? ‘‘5’’ and ‘‘2’’ ? ‘‘Z’’) are required to convert x into y, and only one (the substitution ‘‘1’’ ? ‘‘5’’) to convert z into y. Based on this result we would mistakenly claim that ‘‘4455HZ’’ and ‘‘4415HZ’’ were generated from the same plate, since d(x ? z) < d(x ? y). Alternatively suppose that the LPR character-by-character reading uncertainty is known. In this case it is possible to estimate the likelihood associated with each edit operation to convert either x into y or z into y. With this additional information, we would choose the right match if it were more likely to convert x into y than z into y, or at least we would not make a mistake if it were very unlikely to convert z into y. The LPR interpretation uncertainty can indeed be translated into a matrix of reading probabilities where each cell represents the likelihood of certain pair-wise character symbol occurrence. Typically, there is a relatively high chance of certain characters (e.g. ‘‘1,’’ ‘‘0,’’ and ‘‘B’’) being misread (e.g. ‘‘I,’’ ‘‘O,’’ and ‘‘8,’’ respectively) by the LPR machine. These LPR errorreading likelihoods can then be used to estimate probabilities of pair-wise character association between the outcomes of two consecutive LPR stations (as discussed in the previous example), which in turn serve as inputs for the weights (or the cost functions) to be used in the edit distance calculation. Details on this are presented in Section 3.1. In designing the weight function, however, one should keep in mind that LPR-based plate-matching task at hand is different from common text-mining applications in the sense that there is no reference or list of true values to match the target value. For each recognized string from one location there is a set of other recognized strings for matching at another location, but the true plate number is unknown. That is the reason why the weight function should be resultant from the association of each LPR unit’s reading probability. Such a cost function is expected to increase the number of positive matches and reduce false-positive matches. With respect to the ED framework, we will assess the GED and CED’s formulations in this study. By comparison to GED, CED’s formulation may be a more suitable measure for matching pairs of strings from LPR machines as it may adequately reflect the most common reading errors made by the LPR machines. Considering the possible errors that may happen when recognizing characters of USA plates, the most common errors are as follows: (1) one or more characters are missed by the LPR machine resulting in a decrease in the length of the original string; (2) graphics and markings on and around the plate frame are recognized as additional characters resulting in an increase of string length; and (3) one or more of the characters are misread as other characters resulting in a wrong string of correct length. One key observation here is that reversal errors, where two sequential characters are each read correctly but in reverse order, are never expected. As such, any compensation technique we use should not reflect such errors. The GED framework on the other hand do not completely account for the actual errors encountered in LPR applications and may sometime compensate for reversal errors. For example, given two strings x = ‘‘ABC123’’ and y = ‘‘BCA123’’, which certainly come from two different vehicles, under GED framework, d(x ? y) computation would consist of a deletion of the first character ‘‘A’’ from x and a insertion of character ‘‘A’’ after character ‘‘C’’ in x yielding a total of two editing operations. Whereas, if compensation for reversal errors were prohibited, the number of operations in this example would be three substitutions, ‘‘A’’ ? ‘‘B’’, ‘‘B’’ ? ‘‘C’’, ‘‘C’’ ? ‘‘A’’, instead. The latter result could be achieved only if ED were constrained by the number of editing operations, such as allowing only substitutions without deletions and insertions in this case. 3. Proposed matching procedure and assessment methodology In this study we deal with the problem of matching vehicle plates from a single LPR origin–destination survey, or twopoint setup, or dual LPR setup. In such case, two LPR units are installed on the road to capture plate numbers of moving vehicles. Depending on the objective, the system can operate temporary or permanently. Each unit is setup in a different location or station. In this paper, we denote the two stations as station g and station h. Station h is located downstream of station g. 310 F.M. Oliveira-Neto et al. / Transportation Research Part C 21 (2012) 306–320 In regards to the proposed matching procedure, we define the second station as reference point for matching. Therefore, every time a plate is captured at station h, there will be a number of candidate plates already captured at station g for matching purposes. Our problem is first to decide whichever candidate is the most suitable to match the reference plate and second to infer whether the resulting match is true or false. In this decision process, the main cost measure to compare two LPR outcomes is given by their corresponding ED measure. In the subsequent subsections we formalize the proposed matching methodology. The main contribution of the method is in the way that two LPR imperfect readings are compared, which is reflected by the editing weight function proposed in Section 3.1. In the matching framework, the edit distance is the primary measure to compare pair of LPR outcomes. In Section 3.2 we propose to find the most likely set of matches based only on ED measure, i.e. a set that minimizes an overall ED cost assignment. The idea was to assess if the ED measure is capable of matching LPR outcomes without the help of the time stamps provided by the machines. In some applications such as vehicle speeding and travel time monitoring, the LPR units are usually setup near each other. In this case, the information of time stamps can be suitable in the matching procedure, when discerning if a match is true or false. In Section 3.3, we propose a simple method to incorporate this information into the matching procedure. In an attempt to adequately backtrack the possible mistakes made by LPR equipment, in Section 3.4, a new set of editing constraints is proposed to be employed in the CED formulation. 3.1. Weight scheme proposed In designing the new weight functions for the application of vehicle tracking, we use the fact that the LPR recognition engine segments the plate string and thus recognizes each individual character independently of its location and of its adjacent character(s). Two import implications can be drawn from this fact, as follows: Remarks: 1. The ability of the LPR machine in reading plate characters can be translated into a matrix of probabilities, as defined in Section 3.1.1, where each individual character symbol is associated with its corresponding possible LPR readings. 2. When comparing two LPR outcomes using the ED measure, the sequence of edit operations to convert a string x into a string y is independent of each other, i.e. any pair-wise character symbol in the sequence does not affect subsequent pair-wise symbol comparisons. 3.1.1. Confusion matrix Before formally describing the weight function formulation, we first define the probability matrix of LPR character interpretation. This matrix is a key element of the matching procedure, and thus it is assumed that such matrix containing the likelihoods of character interpretations by each machine is available or can be estimated from a dataset containing both readings and true values of the license plate numbers. We name such matrices as ‘‘confusion matrices’’. Remark: The confusion matrix is a transition matrix, whose contents are the conditional probabilities of character interpretation. Lets denote this matrix as Cl where the element C lij , for i = 1, . . . , N and j = 1, . . . , N, can represent either the conditional probability p(ri|tj) of a given true character tj being recognized as ri by an LPR machine located at station l, or the inverse conditional probability p(tj|ri) of a recognized character ri having as its ground truth the character tj. A row-sum of the entries of Cl is equal to one if C lij ¼ pðt j jri Þ, otherwise a column-sum of the elements of Cl is one if C lij ¼ pðr i jtj Þ. The diagonal elements of the matrix represent the probabilities that characters are correctly recognized by a LPR machine. The offdiagonal elements represent the misreading probabilities. In our problem of vehicle tracking, each matrix Cl is an N by N square matrix where N is the total number of possible alpha-numeric characters. In our application, N is 37, which includes 36 alphanumeric characters plus a null or missing character. The intention of estimating conditional probabilities for the extra empty or null character was first to capture the likelihood of missing certain characters by the LPR unit, and second to strive in representing the case where additional characters are added to any LPR outcome due to the presence of other symbols, not necessarily license plate characters, in the vehicle plate. Ultimately, the additional row and column for the null character in the confusion matrix makes deletion and insertion operations possible. 3.1.2. Weigh function Let x = x1x2, . . . , xi, . . . , xlx and y = y1y2, . . . , yi, . . . , yly be any two sequence of characters read at stations g and h with string lengths equal to lx and ly, respectively. Suppose that the two strings are disposed along the axes of a grid, as illustrated in Fig. 1, with editing operations represented as the following moves on the grid: downward along the diagonal for substitution, eastward for deletion, and vertical downward for insertion. There are a multitude of editing operation combinations to F.M. Oliveira-Neto et al. / Transportation Research Part C 21 (2012) 306–320 311 Fig. 1. Path on a grid for a general comparison of two strings. convert x into y, which can be adequately represented by all possible directed paths from (0, 0) to (lx, ly) on the grid. The probability of a given sequence of editing operations to compare x and y is given by the following formulation: pðx ! yÞ ¼ n Y pðik ; jk Þ ð2Þ k¼0 where, n is the number of nodes on a path between (i0, j0) = (0, 0) and (in, jn) = (lx, ly). The term p(ik, jk) is the probability of the corresponding editing operation associated with the point (ik, jk) on the grid. Later on, we interpret p(ik, jk) as the likelihood of observing a character outcome of yjk at station h, for a given character outcome of xik at station g, namely, pðyjk jxik Þ. On the grid, the moves (ik  1, jk  1) ? (ik, jk), (ik  1, jk) ? (ik, jk), and (ik, jk  1) ? (ik, jk) represent substitution, deletion and insertion, respectively. The most likely match or sequence of editing operations is given by the sequence with the highest probability. Naturally, this implies the minimization of the negative natural logarithm of Eq. (2), as follows: dðx ! yÞ ¼ min ( n X k¼0 1 log pðik ; jk Þ  ) ð3Þ Notice that Eq. (3) is similar to the ED formulation of Eq. (1). In this case, the proposed weight function can be calculated by cðik ; jk Þ ¼ logðpði 1;j ÞÞ. This cost function can be directly implemented in existing edit distance measures such as GED and CED. k k The probability p(ik, jk) can be estimated from sample data. Remind that there is no true reference string (plate) to match the LPR outcomes x and y. However, the characters associated with a point (ik, jk) on the grid of Fig. 1 are both bounded to a known list of alpha-numeric characters. Based on this assertion, in the next Section 3.1.3 we propose to apply conditional probability theory to associate the character interpretation probabilities given by the confusion matrices Cg and Ch for stations g and h, respectively, and obtain estimates of p(ik, jk) for any possible character transition. 3.1.3. Computation of the conditional probability of association character outcomes To estimate the key probability p(ik, jk) in the weight function of Eq. (3), we propose to estimate a association probability between the outcomes xik and yjk observed at stations g and h, respectively, over the space of alpha-numeric ground truth values. We define such character association likelihood as the conditional probability pðyjk jxik Þ of observing yjk at h given that xik has being observed at g. To simplify the subsequent descriptions in this subsection, lets drop the subscription of xik and yjk . Therefore x and y represent any character outcome at station g and h, respectively. Furthermore, let t be the actual character, or the ‘‘truth’’. Assuming that any character is equally likely to appear anywhere in the sequence of a plate string and that the LPR machines work independently, we can estimate the conditional probability of observing the character outcome y at h, given a character outcome x at g, for a given list containing all possible t characters, as the following expression: pðyjxÞ ¼ P pðx; yÞ pðx; yjtÞpðtÞ ¼Pt pðxÞ y;t pðx; yjtÞpðtÞ ð4Þ which can be further simplified as: pðyjxÞ ¼ X pðyjtÞpðtjxÞ ð5Þ t where pðtjxÞ ¼ pðxjtÞpðtÞ and pðxÞ ¼ pðxÞ P t pðxjtÞpðtÞ. Notice that Eq. (5) is composed by a summation of products with two factors p(y|t) and p(t|x) each, which can be viewed as entries of two confusion matrices. Let Cg be the confusion matrix whose contents are the values p(t|x) and Ch the confusion 312 F.M. Oliveira-Neto et al. / Transportation Research Part C 21 (2012) 306–320 matrix containing the entries p(y|t). Remind that by convention the sum of all entries in a row of Cg equals to one and the sum of all entries in a column of Ch equals to one. An estimate of all 37  37 character reading transitions, or conditional probabilities p(y|x), is given by the resulting matrix of Eq. (6), as follows: C ¼ Cg  ðCh ÞT ð6Þ where (Ch)T denotes the transpose of Ch. We name the matrix C as the ‘‘character-transition matrix’’, whose elements Cij’s are given by the transition probabilities Cij = p(yj|xi), where i = 1, . . . , N; and j = 1, . . . , N. Furthermore it is easy to see that the sum of all entries in a row of C equals to one. According to Eq. (6) one way of obtaining C is to estimate each of the confusion matrices Cg and Ch separately. The data needed to estimate a confusion matrix, Cl, for a LPR machine, denoted by l, is a list, or data vector, containing pairs (r, t) of character readings (e.g. r = ‘‘A’’) and ground truth (e.g. t = ‘‘4’’) characters for a given survey period. To this end, we collect a sample of plate readings and images over a period of LPR operation. With this data, we manually extract the ground truths of the plate numbers from the actual plate images. Finally, we can either estimate the conditional probability p(r|t) or the reverse probability p(t|r) by computing the relative occurrence of r for a given t, or of t for a given r, observed in the sample. It is worth noting that the case where there exists a reference list of characters for matching becomes a particular case of the proposed formulation in Eq. (6). As such if one of the machines worked perfectly, meaning that either Cg = I or Ch = I (where I is a 37  37 identity matrix), C would reduce to the confusion matrix of the other machine, either (Ch)T or Cg, respectively. Finally, the probability p(ik, jk) in Eq. (3), approximated by p(yj|xi), can be obtained directly by simply searching for the cell in matrix C in which the associated characters correspond to those involved in the editing operation at node (ik, jk) on the grid in Fig. 1. 3.2. Matching methodology without passage time information In this section we describe the challenge of matching plates observed at two survey locations, i.e., a dual-LPR setup, without using passage time information. At this first matching algorithm the goal was to find the most likely set of matches from a LPR dataset using only the ED measure to infer about the probability of a genuine match. The motivation of finding this most likely set was to assess the discriminative power of various formulations of similarity measures. Since the number of pair-wise combinations is expected to be large (number of outcomes at station g multiplied by the number of outcomes at station h), if any similarity measure is capable of discriminating genuine matches from false ones in this worst case scenario (without using passage time information), we may claim that it is a good similarity measure for our LPR application. Our proposed vehicle tracking based on LPR technology can be summarized as follows. First, for each outcome (plate number as reported by the LPR machine) at station h, a vector with length equal to the number of pair-wise combinations formed with all outcomes at station g is constructed, where each entry is a corresponding value of the similarity measure (edit distance); second, the assignment with the least ED cost value is selected as a potential match; finally, a threshold on the ED values is used to discriminate the resulting pair-wise matches between potential positive and false matches. Fig. 2 shows a flowchart of this procedure. Notice that the number of observations in the two LPR stations can differ as some vehicles either do not pass through both stations or their plates may not be captured at both stations. This may result in an increased chance of false matches. 3.3. Vehicle tracking with passage time information For the same two-point plate-matching problem we propose a procedure that incorporates the passage time information, if available via mechanisms such as time stamping. This procedure is to decide whether a plate currently read at the downstream station h can be matched to a plate belonging to the subset of the most recent plates read at the upstream station g. Applications of such a procedure include automated speed enforcement, real-time travel time monitoring, and vehicle tracking. In essence, the proposed matching procedure is to match any current outcome yj, where j is the jth record in the list of outcomes observed at station h, to a subset of the earliest observations at station g. The subset of candidates at station g is formed by those outcomes whose corresponding passage time falls within a time window, which is bounded by the upper and lower limits of the expected travel times between the two stations. Furthermore, the width of a time window is adaptively adjusted depending on the edit distance. When the edit distance is large, we make the width of a time window smaller to reduce the probability of a false-positive match. Let the pair-wise string (xi, yj) be a potential match, where xi is the ith outcome observed at station g and yj is the jth outcome read at station h. That means that xi is the best (i.e. with the least ED value) earliest previous observation at station g to match yj. Let tij = vj  ui be the time-difference between the synchronized passage times, where ui and vj are the corresponding time stamps at stations g and h, respectively. Furthermore, let smin and smax be a minimum and maximum threshold values, F.M. Oliveira-Neto et al. / Transportation Research Part C 21 (2012) 306–320 313 Fig. 2. The proposed matching procedure. Fig. 3. Matching procedure considering passage time information. respectively, for the similarity measure denoted by dij = d(xi ? yj). As such, if dij > smax, (xi, yj) is very unlikely to be genuine. Conversely, if dij 6 smin, (xi, yj) is very likely to be genuine. In the following decision process of classifying a match in false or genuine, we based our beliefs in two attributes: the ED value and the time-difference tij. The magnitude of the edit distance is the primary evidence and the value tij is used as secondary evidence. 314 F.M. Oliveira-Neto et al. / Transportation Research Part C 21 (2012) 306–320 Fig. 4. Time–space diagram for the proposed matching procedure with passage time information. The matching procedure, as shown in Figs. 3 and 4, can be described by the following steps: (1) Match any current observation yj at station h to a subset of the earliest previous observations at station g bounded by an time-frame period constraint denoted by tfg, and search among the candidates for the best string xi with the least edit distance; (2) If dij 6 smin, we declare the match (xi, yj) as genuine; (3) Otherwise, if smin < dij 6 smax, we declare (xi, yj) a valid match only if the time-difference tij falls within a time window constraint denoted by tw = (l  d, l + d), where l is the mean journey for vehicles observed at station h within the earliest time-frame tfh, and d is the amplitude of tw for the corresponding period. The time-frame tfg to establish the subset of outcome candidates at station g is given by tfg = (vj  jtu, vj  jtl), where jtu and jtl are predefined upper and lower bounds of the expected journey time. Therefore, if vj  jtu 6 tij 6 vj  jtl, the corresponding pair (xi, yj) is classified as a potential match, though unnecessarily genuine. The limit bounds of the time window constraints are determined based on the journey time variation and the edit distance magnitude. We assume that the journey times of vehicles detected at a short time period tfh belong to a symmetrical density function, such as the normal distribution. In this case, the time difference tij is probably a genuine journey time if it is closer to the expected mean journey time. However, if dij is larger, there is a lower confidence that tij is a genuine journey. Therefore the amplitude of tw should decrease for larger values of dij. In view of this, we propose to calculate the amplitude d of the time window constraint by the following expression: d ¼ zðdij Þr; smin < dij  smax ð7Þ h where r is standard deviation of the mean journey time of vehicles observed at the corresponding time-frame tf ; and z(dij) is the number of standard deviations that define the interval limits, which is a monotonically decreasing function of the similarity measure value dij. A quadratic functional form, Eq. (8), is proposed for z(dij). rffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi smax  dij zðdij Þ ¼ 9  max s  smin ð8Þ Eq. (8) guarantees that at the largest time window, when z = 3, dij is equal to its lowest value, i.e. dij = smin. Whereas when dij = smax the time window constraint vanishes, meaning that when the similarity measure approaches its highest possible value the analyzed match will be considered genuine only if the time-difference tij is exactly equal to the mean journey time. Parameters l and r can be estimated from a sample of passage times from matches classified as genuine during the earliest survey period tfh at h, or from matches obtained in previous days during the same survey period. Outliers can be eliminated from the sample if they lie outside a constraint interval calculated around the sample median. 3.4. New editing constraints LPR machines usually do not reverse the sequence of characters on the plates. For this reason it is very likely that any pair of recognized strings can have their characters lined up if they come from the same vehicle. Therefore in this research, it is worth noting that, besides analyzing the GED formulation, the CED with editing constraints defined based on the length differences of the strings being compared is also assessed. For any pair of recognized strings x and y, with lengths lx and ly, we propose the following constraint sets (i, e, s) of insertions, deletions, and substitutions to transform x to y. (a) (i, e, s) = (|lx  ly|, 0, lx), if ly > lx; F.M. Oliveira-Neto et al. / Transportation Research Part C 21 (2012) 306–320 315 (b) (i, e, s) = (0, |lx  ly|, ly), if ly < lx; (c) (i, e, s) = (0, 0, lx), if ly = lx. The three restrictions above state that insertions or deletions will be allowed only if the lengths of two strings are different, otherwise only substitutions will be allowed. Thus, the CED with such editing constraints defined as a function of the string lengths may potentially eliminate false-positive matches that otherwise would be obtained if a GED formulation were used. 4. Case study and experimental results The proposed matching methods were applied to match plates from a real dataset. The case study involves tracking large trucks (with gross weights over 80,000 lb) using a two-point configuration on the Interstate highways in Knoxville’s metropolitan area. 4.1. Experiment design Two sets of LPR equipment were used to capture license plates of westbound trucks on I-40, one at the Campbell Station Road (LPR Station 1) and the other downstream at the weigh station (LPR Station 2). Both units used internal detection software to trigger the camera and an infra-red-based illuminator when a vehicle is present in the camera’s field of view. Due to the experimental nature of the study and the limited number of LPR cameras could be afforded at the time, only plate data for the outside lane (next to the shoulder) were collected on 5 weekdays for roughly 3 h each day. The distance between two stations was about 1.4 miles. A total of 2671 plates were captured at the first station and 1530 were captured at the second "0""1""2""3""4""5""6""7""8""9""A""B""C""D""E""F""G""H""I" "J""K""L""M""N""O""P""Q""R""S""T""U""V""W""X""Y""Z" "" "0" 98 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 "1" 0 89 0 1 0 0 0 2 0 0 0 0 0 0 0 0 0 0 2 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 3 "2" 0 0 94 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 5 0 "3" 0 1 0 97 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 "4" 1 0 0 0 95 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 "5" 0 0 0 0 0 98 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 "6" 0 0 0 1 0 0 91 0 2 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 "7" 0 0 0 1 0 0 0 96 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 "8" 1 0 0 0 0 0 0 0 95 1 0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 "9" 0 0 0 0 0 0 0 0 0 97 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 "A" 0 3 1 0 2 1 3 0 3 1 78 0 0 0 0 0 0 0 0 0 1 1 1 1 0 0 0 1 0 1 1 0 0 1 0 1 1 "B" 2 1 0 0 0 0 0 0 29 1 0 55 1 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 9 "C" 0 0 0 0 0 0 0 0 1 0 0 0 99 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 "D"19 1 0 1 0 0 0 0 0 0 0 0 0 74 0 0 0 0 0 0 0 0 1 0 1 1 0 0 0 0 0 0 0 0 0 0 0 "E" 3 0 0 0 0 0 0 0 0 0 0 0 0 0 87 0 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 0 5 "F" 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 89 0 2 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 2 0 "G" 0 0 0 0 0 0 22 2 5 0 0 2 0 0 0 0 63 2 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0 2 "H" 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 98 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 "I" 1 32 0 1 0 0 0 1 1 2 0 0 0 1 0 1 0 1 36 0 0 2 1 0 0 1 0 0 0 3 2 1 0 0 3 1 9 "J" 0 11 1 2 0 1 0 0 0 1 0 0 0 0 0 0 0 1 0 72 0 1 0 1 0 0 0 0 0 0 1 0 0 0 0 0 6 "K" 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 92 0 2 0 0 0 0 2 0 0 0 0 2 3 0 0 0 "L" 0 7 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 85 0 0 0 0 0 0 0 2 0 0 0 0 0 0 2 "M" 0 0 0 0 0 0 0 0 0 0 0 0 0 3 0 0 0 6 0 0 6 0 65 10 0 0 0 0 0 0 0 0 6 0 0 0 3 "N" 0 0 0 3 0 0 0 3 0 0 0 0 0 0 0 0 0 6 0 0 0 0 0 89 0 0 0 0 0 0 0 0 0 0 0 0 0 "O"77 2 0 0 0 0 0 0 1 0 0 0 0 2 0 0 0 0 0 0 0 0 0 0 13 0 4 0 0 0 0 0 0 0 0 0 1 "P" 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 99 0 0 0 0 0 0 0 0 0 0 0 "Q"44 0 0 0 0 11 0 0 0 11 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 22 0 0 0 0 0 0 0 0 0 11 "R" 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 84 0 0 0 0 0 2 0 0 2 "S" 0 3 1 3 0 25 1 0 3 3 0 0 1 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 57 0 0 0 0 0 0 0 3 "T" 0 5 0 2 0 2 0 5 0 2 0 0 0 0 2 0 2 0 2 0 0 2 0 0 0 0 0 0 0 64 0 0 0 0 2 0 9 "U" 7 0 0 0 0 0 0 0 0 0 0 0 0 1 0 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 89 0 0 0 0 0 0 "V" 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 3 85 0 0 6 0 1 "W" 0 0 0 2 0 0 0 0 0 0 2 0 0 2 0 0 0 7 0 0 0 2 7 7 0 0 0 0 0 0 0 0 70 0 0 0 2 "X" 0 0 0 2 0 0 2 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 93 0 0 0 "Y" 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 1 1 95 0 0 "Z" 0 0 3 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 94 0 "" 15 3 5 4 3 8 1 3 9 8 1 0 3 0 1 1 0 5 1 0 1 2 1 0 0 6 0 1 1 2 0 1 1 1 7 3 0 Fig. 5. Confusion matrix Cg estimated from the first set of calibration data. 316 F.M. Oliveira-Neto et al. / Transportation Research Part C 21 (2012) 306–320 station. Among these, a total of 787 were manually verified as identical. In addition to reading plates, the equipment also ‘‘stamped’’ each plate image with time information. The raw images stored in the LPR system database were viewed manually to compare with the detection reports. The results showed an average accuracy of 61% for LPR station 1 and 63% for LPR station 2. 4.2. Calibration of the character-transition matrix and edit distance calculation example The dataset was divided into two parts: one for calibration of the model parameters and the other for comparison of the performances of the similarity measures. Since there were 5 days of data, all combinations of three datasets out of the total five were used as calibration data, with the remaining two datasets as validation data. Thus, each of the ten (5!/3!2!) combinations with 3 days of data was used to estimate twenty ‘‘confusion matrices,’’ i.e., 10 matrices of type Cg and 10 matrices of type Ch. As an example, Figs. 5 and 6 present the estimated two matrices, Cg and Ch, resulting from the first set of calibration data (the first 3 days of data). In this example, a total of 8908 and 6222 character pairs was used to obtain Cg and Ch, respectively. The character-transition matrix (Fig. 7) from the first set of calibration data was then estimated by Eq. (6). The following expression demonstrates how the cell C29,6 corresponding to the pair (‘‘S’’, ‘‘5’’) is calculated. Prð\5"j\S"Þ ¼ Prðy ¼ \5"jt ¼ \0"ÞPrðt ¼ \0"jx ¼ \S"Þ þ    þ Prðy ¼ \5"jt ¼ zÞPrðt ¼ zjx ¼ \S"Þ þ    þ Prðy ¼ \5"jt ¼ \"ÞPrðt ¼ \"jx ¼ \S"Þ ¼ 28% ð9Þ where z is any ground truth character within the ordered sequence of alphanumeric characters from ‘‘0’’ to the null character ‘‘’’. According to the above result there is a probability of 28% that a character ‘‘5’’ is observed at station h given that a character ‘‘S’’ is observed at station g. "0""1""2""3""4""5""6""7""8""9""A""B""C""D""E""F""G""H""I" "J""K""L""M""N""O""P""Q""R""S""T""U""V""W""X""Y""Z" "" "0" 80 0 0 0 0 0 0 0 1 1 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 1 "1" 0 92 0 0 1 0 0 0 0 0 0 0 1 2 0 0 0 0 2 6 2 0 0 0 7 1 0 0 0 10 0 2 0 0 1 0 26 "2" 0 0 96 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 7 2 "3" 0 0 0 93 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 5 "4" 0 0 0 0 93 1 1 0 0 0 1 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 4 "5" 0 0 0 1 0 93 0 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 "6" 0 0 0 1 1 2 91 0 1 0 0 0 0 0 3 2 3 0 0 0 2 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 2 "7" 0 0 0 0 0 0 0 97 0 1 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 2 2 "8" 0 0 0 0 0 0 1 0 87 0 0 7 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3 "9" 0 0 0 0 0 0 0 0 0 93 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 6 "A" 0 0 0 0 3 0 1 0 0 0 95 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 3 2 0 0 0 0 0 0 0 2 "B" 0 0 0 0 0 0 0 0 4 1 0 93 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 4 "C" 0 0 0 0 0 0 0 0 0 0 0 0 93 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 "D" 3 0 0 0 0 0 0 0 0 0 0 0 0 82 0 0 0 0 0 0 0 0 0 0 0 0 8 0 0 0 2 0 0 0 0 0 3 "E" 0 0 0 0 0 0 0 0 0 0 0 0 0 0 87 0 0 0 0 0 0 4 0 0 0 1 0 0 0 0 0 0 0 0 0 0 1 "F" 0 0 0 0 0 0 0 0 0 0 0 0 1 0 3 93 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 "G" 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 91 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 "H" 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 95 0 0 0 0 3 0 0 0 0 0 0 0 0 0 4 0 0 0 0 "I" 0 6 0 0 0 0 0 0 0 0 0 0 0 2 3 2 0 2 90 0 0 17 0 0 0 1 0 0 0 10 22 0 0 3 0 1 27 "J" 0 1 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 90 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 5 "K" 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 93 0 0 0 0 0 0 3 0 0 0 0 0 3 0 0 0 "L" 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 79 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3 "M" 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 79 4 0 0 0 0 0 0 0 0 0 0 0 0 0 "N" 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3 89 0 0 0 3 0 0 0 0 0 0 0 0 0 "O"11 0 0 0 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 0 79 0 33 0 0 0 2 0 0 0 0 0 0 "P" 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 93 0 0 0 0 0 0 0 0 1 0 0 "Q" 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 7 0 50 0 0 0 0 0 0 0 0 0 0 "R" 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 2 0 3 0 0 0 0 89 0 0 0 0 0 0 0 0 0 "S" 0 0 0 0 0 3 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 86 0 0 0 0 0 0 0 1 "T" 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 0 76 0 2 0 0 1 0 1 "U" 1 0 0 0 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 4 7 0 8 0 0 0 73 0 0 0 0 0 1 "V" 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 92 0 0 7 0 2 "W" 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 7 4 0 0 0 0 0 0 0 2 96 3 0 0 0 "X" 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 92 0 1 0 "Y" 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 88 0 0 "Z" 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 88 0 "" 3 0 1 1 1 1 1 1 1 0 3 0 1 0 3 0 0 0 0 2 0 0 0 0 0 2 0 3 2 5 0 2 0 0 0 0 1 Fig. 6. Confusion matrix Ch estimated from the first set of calibration data. 317 F.M. Oliveira-Neto et al. / Transportation Research Part C 21 (2012) 306–320 "0""1""2""3""4""5""6""7""8""9""A""B""C""D""E""F""G""H""I" "J""K""L""M""N""O""P""Q""R""S""T""U""V""W""X""Y""Z" "" "0" 78 0 0 0 0 0 0 0 1 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0 11 0 0 0 0 0 2 0 0 0 0 0 2 "1" 0 82 0 1 1 0 0 2 0 0 0 0 0 0 0 0 0 0 8 2 0 0 0 0 0 0 0 0 0 1 0 0 0 0 1 0 0 "2" 0 0 91 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 5 1 "3" 0 1 0 91 0 1 1 1 0 1 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 "4" 0 1 0 0 88 1 2 0 0 1 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 "5" 0 0 0 0 1 91 2 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 1 "6" 0 1 0 1 1 1 82 0 3 0 1 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 "7" 0 1 1 1 0 0 0 93 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 1 "8" 2 1 0 0 0 0 1 0 82 1 0 5 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 1 "9" 1 1 0 1 1 0 0 1 1 91 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 1 0 0 0 0 0 "A" 0 3 1 0 3 1 2 0 3 1 74 0 0 0 0 0 0 0 1 0 1 1 1 1 0 0 0 1 0 1 1 0 0 2 0 1 2 "B" 2 3 0 0 0 0 0 0 29 2 0 53 1 0 0 0 0 1 3 1 0 0 0 0 0 1 0 1 0 0 0 0 0 0 0 0 0 "C" 0 1 0 0 3 0 0 0 1 0 0 0 92 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 "D"17 3 0 1 0 0 0 0 0 0 0 0 0 61 0 0 0 0 1 0 0 0 1 0 10 1 0 0 0 0 3 0 0 0 0 0 1 "E" 3 1 0 0 0 0 2 0 0 0 2 0 0 0 76 2 0 0 4 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 0 2 "F" 0 0 0 0 0 0 2 2 0 0 0 0 0 0 2 83 0 2 2 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 2 0 "G" 0 1 0 0 0 2 22 2 5 0 0 2 2 0 0 0 59 2 1 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 "H" 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 94 2 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 "I" 1 33 0 2 0 0 0 2 1 2 0 0 0 2 0 1 0 1 38 2 0 2 1 0 0 1 0 0 0 4 2 1 0 0 2 1 0 "J" 0 16 1 4 0 1 0 0 0 2 0 0 0 0 0 0 0 1 3 65 0 1 0 1 0 0 0 0 0 0 1 0 0 0 0 0 2 "K" 0 2 0 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 86 0 1 0 0 0 0 3 0 0 0 0 2 4 0 0 0 "L" 0 8 0 0 2 0 0 0 0 0 0 0 0 0 4 0 0 0 16 0 0 68 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 "M" 0 1 0 0 0 0 2 0 0 0 0 0 0 3 0 0 0 9 1 0 6 0 52 11 0 0 0 2 0 0 0 0 11 0 0 0 0 "N" 0 0 0 3 0 0 0 3 0 0 0 0 0 0 0 0 0 5 0 0 0 0 3 79 0 0 0 0 0 0 3 0 3 0 0 0 0 "O"62 3 0 0 0 0 0 0 1 0 0 0 0 4 0 0 0 0 1 0 0 0 0 0 21 0 3 0 0 0 3 0 0 0 0 0 2 "P" 0 1 1 0 0 0 0 0 0 1 0 0 0 0 1 0 0 0 1 0 0 0 0 0 0 92 0 0 0 0 0 0 0 0 1 0 2 "Q"36 3 0 1 1 10 0 0 0 11 0 0 0 3 0 0 0 0 3 1 0 0 0 0 13 0 11 0 1 0 3 0 0 0 0 0 1 "R" 0 1 0 0 0 0 0 0 8 0 2 0 0 0 0 0 0 0 1 0 5 0 0 2 0 0 0 75 0 0 0 0 0 2 0 0 3 "S" 0 4 1 3 0 28 1 0 3 3 1 0 1 0 0 0 1 0 1 0 0 0 0 0 0 0 0 0 50 0 0 0 0 0 0 0 2 "T" 0 13 0 3 0 2 0 5 0 3 0 0 0 0 2 0 2 0 11 1 0 2 0 0 0 0 0 0 0 49 0 0 0 0 2 0 3 "U" 7 0 0 0 0 0 0 0 0 0 0 0 0 3 0 1 0 1 19 0 0 0 0 0 3 0 0 0 0 0 65 0 0 0 0 0 0 "V" 0 2 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 0 0 2 2 79 2 0 5 0 2 "W" 0 1 0 2 0 0 0 0 0 0 2 0 0 2 0 0 0 9 1 0 0 2 5 6 0 0 0 0 0 0 0 0 68 0 0 0 0 "X" 0 0 0 2 0 0 2 2 0 0 0 0 0 0 0 0 0 0 3 0 3 0 0 0 0 0 0 0 0 0 0 0 3 86 0 0 0 "Y" 0 1 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 2 0 7 1 1 83 0 0 "Z" 0 0 10 0 2 0 0 4 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 83 0 "" 12 4 6 4 3 8 2 4 8 7 1 0 3 0 1 1 0 5 2 0 1 2 1 0 2 6 0 1 1 2 0 2 2 1 6 2 1 Fig. 7. Character-transition matrix C estimated from the first set of calibration data. Table 1 Examples of generalized edit distance calculation. xi yj p(yj|xi) log ‘‘4’’ ‘‘4’’ ‘‘S’’ ‘‘5’’ ‘‘H’’ ‘‘2’’ ‘‘4’’ ‘‘4’’ ‘‘5’’ ‘‘5’’ ‘‘H’’ ‘‘Z’’ 0.885 0.885 0.280 0.914 0.937 0.055 0.122 0.122 1.273 0.090 0.065 2.906 GEDðx ! yÞ ¼ P log   1 pðyj jxi Þ  1 pðyj jxi Þ 4.579 ¼ zi yj p(yj|zi) log ‘‘4’’ ‘‘4’’ ‘‘1’’ ‘‘5’’ ‘‘H’’ ‘‘Z’’ ‘‘4’’ ‘‘4’’ ‘‘5’’ ‘‘5’’ ‘‘H’’ ‘‘Z’’ 0.885 0.885 0.001 0.914 0.937 0.829 0.122 0.122 6.535 0.090 0.065 0.188 GEDðz ! yÞ ¼ P log   1 pðyj jzi Þ ¼    1 pðyj jzi Þ 7.122 318 F.M. Oliveira-Neto et al. / Transportation Research Part C 21 (2012) 306–320 To illustrate the calculation of the edit distance using the estimated character-transition matrix of Fig. 7, we calculated the GED values for those matches presented in the example of Section 2.2, where we had to decide whether the outcome x = ‘‘44S5H2’’ or z = ‘‘4415HZ’’ match y = ‘‘4455HZ’’. Table 1 presents, in columns 1 and 2, the best GED’s alignments of the matches (x, y) and (z, y), and, in columns 3 and 4, the character-by-character conditional probabilities and the resulting editing weight values, respectively. The final GED value is also presented for each of the two matches. Notice that GED(x ? y) < GED(z ? y). Therefore, based only on the similarity measure we would choose the outcome x over z to match y. 4.3. Performance of the vehicle tracking procedures Considering the possible ways of defining the editing weights into the recurrent calculation of ED (see the grid of Fig. 1), the following four procedures were indentified to calculate the ED between pair of strings:     D1: Edit distance with zero or one cost assignments, which corresponds to the original idea of Levenshtein. D2: GED using weight function as in Eq. (3), with p(ik, jk) estimated by p(y|x), as defined in Eq. (5). D3: Original CED with zero or one cost assignments and constrained by the editing sets defined in Section 3.4. D4: CED using weight function as in Eq. (3) and p(ik, jk) as in Eq. (5), and constrained by the editing sets defined in Section 3.4. The performance of the proposed procedures, D2 and D4, were compared to the popular ED (D1) and CED (D3) methods. The four procedures were applied to all ten combinations of the two remaining (unused) days of data for validation. As mentioned earlier, the performance of the similarity measures was investigated under the worst case scenario that consisted of matching up the two sets of plates for each remaining day, without using passage time information or the recorded time stamps. The main premise there was that under this worst-case scenario the most suitable measure for LPR application should be able to accurately match every two set of plates with fewer false matches. Regarding the measures of performance, the percentage of positive matches and the percentage of false-positive matches were calculated for a range ED-thresholds covering the domain of all possible ED values, ranging from 0 to 20. In order to evaluate the performance measures, we used the ground truth of the plate numbers manually extracted from the actual plate images. The accuracy of each similarity measure was then established by drawing curves relating the percentage of correct matches to the percent of false-positive matches over the domain of ED values. To this end, ten such charts were constructed and all of them presented similar results as the one shown in Fig. 8 for one of the possible combination of datasets. The first thing to notice in Fig. 8 is that existing ED or CED combined with our proposed weight functions yielded considerably improved performance for vehicle tracking. Second, with respect to two measure frameworks, GED and CED, there was little evidence of difference in performance between these two measures over all 10 analyses performed. That is, empirical evidence cannot support that CED equipped with the proposed editing operation constraints, as defined in Section 3.4, is a better, or worse, procedure to compensate for the mostly common errors made by LPR machines. In general, either D2 or D4 measures were able to achieve around 90% of positive matches with about 5–8% of false matches. In addition, it is worth noting that these measures achieved almost 80% of positive matches with approximately Fig. 8. The performance of different vehicle tracking procedures. F.M. Oliveira-Neto et al. / Transportation Research Part C 21 (2012) 306–320 319 1–2% of false matches. Thus, it seems that any ED formulation equipped with the proposed weight functions has the most discriminative power to match data from LPR systems, when the target or reference values for matching are unknown. 4.4. Performance of the online vehicle tracking procedure In this section the accuracy of the proposed online matching procedure with the passage time information incorporated was assessed. This procedure was combined with the most suitable similarity measure chosen previously and was calibrated and evaluated using the 10 sets of data combinations. The time-frame tfg was defined assuming an upper bound for the vehicle speed of 90 mph and a lower bound of 35 mph. The parameters l and r were estimated by the moving averages and standard deviations of every early time block tfh containing at least 5 matches classified as genuine. From the calibration datasets, was established that tfh = 10 min was a suitable survey period for this purpose. Outliers were removed based on the procedure proposed by Clark et al. (2002) where any journey time value lying outside the interval determined by Eq. (10) was classified as an outlier. Me  3  Pn i¼1 jJT i n  Me j ð10Þ where JTi denotes journey time for vehicle i; Me is the median for each 10-min block of journey times; and n is the number of observations inside each block. The ED thresholds, smin and smax, were calibrated using the same sets of calibration data that were used to estimate the character-transition matrices. To this end, the matching procedure combined with measure D2 was then applied to each of the 10 calibration sets. In the calibration process smin was chosen to be a fixed value and we made smaxsx varies until a desired performance was achieved. A value of 5 was chosen for the parameter smin, which corresponded to a limit below which no false matches was observed in all 10 calibration sets. The maximum threshold that provided a performance of false-positive matching rate below 2% for all calibration sets was selected for the parameter smax. This value came out to be approximately equal to 20. After applying the matching procedure combined with measure D2 to all 10 combinations of 2 days of data used for validation, it was in general achieved between 95% and 98% of genuine matches and about 0.5–1.5% of false matches. As a base scenario for comparison, we calibrated and applied the same procedure using measure D1. As a result, for the calibrated parameters smin = 0 and smax = 5, we achieved a positive matching rate between 93.5% and 96.5% but with false-positive matching rate of about 2.2–3.6%. 5. Conclusions This paper assesses the performance of different vehicle tracking procedures using a dual setup (two-point tracking) of LPR units. We proposed a new procedure to compute the weight functions that can be incorporated with known similarity distance between outcomes of the two units when the confusion matrix of the LPR machines are available or can be estimated. As pointed out before, the formulation proposed for the weight function is more general than the ones used in existing text-mining applications, where a reference list is always available for matching. Therefore, the situation in which one of the LPR units works perfectly is a particular case of the situation analyzed in this paper. This is the major contribution of this research effort, not only for the transportation research community but also for the pattern recognition community. A field study using two LPR units was conducted on I-40 in the vicinity of Knoxville, Tennessee. The experimental results and analyses showed that the most suitable procedures for vehicle tracking based on the plates recognized by a dual LPR setup are either GED or CED formulation combined with the weight function and editing constraints proposed in this paper. These procedures can attain a positive matching rate of about 90% with only 5–8% of false matches. This represents a promising result for transportation planning purposes, or even for automatic traffic speed monitoring. When the travel time information is incorporated as second level constraint into the matching procedure the performance increases in terms of a higher positive matching rate of about 95–98% and a lower false-positive matching rate of about 0.5–1.5%. Although the positive matching rate increased significantly when travel time information was used, further work is needed to validate this procedure for other situations of traffic conditions and setups of the LPR machine. The procedure was applied during periods with minor traffic variation resulting in limited dispersion of travel times, which might have contributed to the superior performance. In addition, the two stations were relatively closed to each other without major sources of traffic disturbance to disperse the travel times as most vehicles (trucks in this case) travelled in direct fashion through two station sections. Further studies are needed to extend this research for the case of multiple sequential and parallel LPR units, such as in a large highway network, where truck/freight origin–destination information might be extracted. Another issue of interest is the sample size (number of outcomes) needed to estimate the character misreading matrices Cg and Ch. These estimates are site-dependent as they reflect the characteristic (mix of various plate designs, vehicular speed and viewing angle, lighting and weather, etc.) of the locations where the LPR machines are installed. If there are many sources of variations and noises in the process of estimating the misreading probabilities, a larger amount of data would be needed to achieve a required precision. Such estimation would require also more human intervention since the ground truth values of the plates were extracted manually. 320 F.M. Oliveira-Neto et al. / Transportation Research Part C 21 (2012) 306–320 In situations where LPR cameras are installed permanently, human effort may be reduced through a self-learning mechanism of estimating the conditional probabilities of the character-transition matrix C. The sample size is still an issue in this case. Although, the system could potentially learn in a continuous fashion until a desirable or acceptable level of precision is attained, or until the conditional probabilities converge to certain values within a tolerable threshold. Details of this will be presented in an upcoming paper. Another research direction that the authors should be continuing on is the use of image processing techniques in order to eliminate potential false-positive matches. The idea is to include into the decision process a third measure, besides the ED and the time-difference, based on image comparisons. It is believed that this additional information would eliminate any possibility of false-positive match. Further studies can also undertake the problem of matching vehicle plates when multiple images are captured for each vehicle. Specifically, consecutive frames of the same vehicle are the result of license plate detection using video sequence. In this case, the most likely plate number should be determined, among different records, to represent the targeted vehicle. To deal with this, Giannoukos et al. (2010) have proposed a simple procedure to group consecutive plate records and select the most probably combination of characters within the given group. With respect to vehicle plate matching, the interest is in the most probably pair of records within all pair-wise combinations, when comparing two sets of multiple records. This deserves further investigation. Acknowledgments This research is partially sponsored by National Transportation Research Center, Inc. under a USDOT RITA Grant (DTRT06-0043-U09) and partially sponsored by a USDOT FHWA Dwight D. Eisenhower Graduate Fellowship award. References Anagnostopoulos, C.N.E., Anagnostopoulos, I.E., Loumos, V., Kayafas, E., 2006. A license plate-recognition algorithm for intelligent transportation system applications. IEEE Transactions on Intelligent Transportation Systems 7 (3), 377–392. Anagnostopoulos, C.N.E., Anagnostopoulos, I.E., Psoroulas, I.D., Loumos, V., Kayafas, E., 2008. License plate recognition from still images and video sequences: a survey. IEEE Transactions on Intelligent Transportation Systems 9 (3), 377–391. Chang, S.L., Chen, L.S., Chung, Y.C., Chen, S.W., 2004. Automatic license plate recognition. IEEE Transactions on Intelligent Transportation Systems 5 (1), 42– 53. Clark, S.D., Grant-Muller, S., Chen, H., 2002. Cleaning of matched license plate data. Transportation Research Record 1804, 1–7. Duda, R.O., Hart, P.E., Stork, D.G., 2000. Recognition with Strings. Pattern Classification,, Wiley Inter Science, pp. 413–420 (Chapter 8). Giannoukos, I., Anagnostopoulos, C.N., Loumos, V., Kayafas, E., 2010. Operator context scanning to support high segmentation rates for real time license plate recognition. Pattern Recognition 43, 3866–3878. Levenshtein, V.I., 1966. Binary codes capable of correcting deletions, insertions and reversals. Soviet Physics Doklady 10 (8), 707–710. Marzal, A., Vidal, E., 1993. Computation of normalized edit distance and applications. IEEE Transactions on Pattern Analysis and Machine Intelligence 15 (9), 926–932. Nakanishi, Y.J., Western, J., 2005. Ensuring the security of transportation facilities: evaluation of advanced vehicle identification technologies. Transportation Research Record 1938, 9–16. Nelson, L.J., 2000a. Identification/Recognition: A Change of Order. ITS World, pp. 32–33. Nelson, L.J., 2000b. Snap decisions. Traffic Technology International, 50–52. Nelson, L.J., 2003. An avid reader. Traffic Technology International, 72–74. Ocuda, T., Tanaka, E., Kasai, T., 1976. A method for correction of garbled words based on the Levenstein metric. IEEE Transactions on Computers 25 (C-2), 172–177. Oliveira Neto, F.M., Han, L.D., Jeong, M.K., 2009. Tracking large trucks in real-time with license plate recognition and text-mining techniques. Transportation Research Record 2121, 121–127. Oommen, B.J., 1986. Constrained string editing. Information Sciences 40 (3), 267–284. Rossetti, M.D., Baker, J., 2001. Applications and evaluation of automated license plate reading systems. In: 11th ITS America Meeting. CD-ROM. Conference Proceedings, Miami Beach, FL. Seni, G., Kripasundar, V., Srihari, R., 1996. Generalizing edit distance to incorporate domain information: handwritten text recognition as a case study. Pattern Recognition 29 (3), 405–414. Wagner, R.A., Fischer, M.J., 1974. The string-to-string correction problem. Journal of the Association Computer Machinery 21 (1), 168–173. Watling, D.P., 1994. Maximum likelihood estimation of an origin-destination matrix from a partial registration plate survey. Transportation Research Part B 28B (4), 289–314. Watling, D.P., Maher, M.J., 1988. A graphical procedure for analyzing partial registration plate data. Traffic Engineering & Control 29, 515–519. Watling, D.P., Maher, M.J., 1992. A statistical procedure for estimating a mean origin-destination matrix from partial registration plate survey. Transportation Research Part B 26B (3), 171–193. Wei, J., 2004. Markov edit distance. IEEE Transactions on Pattern Analysis and Machine Intelligence 6 (3), 311–320. Wiggins, A., 2006. ANPR Technology and Applications in ITS. 2006. In: Research into Practice: 22nd ARRB Conference Proceedings. CD-ROM. Australia Road Research Board, ARRB, Canberra, Australia, October of 2006.