Pattern Recognition 33 (2000) 351}373
Knowledge-based ECG interpretation: a critical review Mahantapas Kundu, Mita Nasipuri*, Dipak Kumar Basu Department of Computer Science & Engineering, Jadavpur University, Calcutta 700 032, India Received 16 June 1998; accepted 10 March 1999
Abstract This work presents a brief review of some selected knowledge-based approaches to electrocardiographic (ECG) pattern interpretation for diagnosing various malfunctions of the human heart. The knowledge-based approaches discussed here include modeling an ECG pattern through an AND/OR graph, a rule-based approach and a procedural semantic network (PSN) based approach for ECG interpretation. However, certain syntactic approaches to ECG interpretation are also covered, considering their precursory roles to knowledge-based ECG interpretation. A fuzzy-logic-based approach is included in the discussion to show how imprecision can be dealt with in modeling cardiological knowledge. A domain-dependent control algorithm is discussed to show how the production level parallelism can be exploited to reduce the length of the match}resolve}act cycle of a rule based ECG interpretation system. The review also contains a brief description of some recent applications of connectionist approaches to ECG interpretation. This discussion "nally ends with a comparative assessment of performances of all the above-mentioned knowledge-based approaches to ECG interpretation and some hints about the future directions of work in this "eld. ( 2000 Pattern Recognition Society. Published by Elsevier Science Ltd. All rights reserved. Keywords: Automated ECG interpretation; Knowledge-based system; AND/OR graph; Procedural semantic network; Fuzzy logic; Connectionist approach; Arti"cial neural network
1. Introduction The Electrocardiogram (ECG) [1}3], as shown in Fig. 1(a) and (b), is the record of variation of bio-electric potential with respect to time as the human heart beats. Interpretation of ECG patterns is needed for diagnosing malfunctions of the human heart. Ventricular heart rate is the most common item of information among those which can be extracted from the ECG tracing of the human heart by measuring the time distance between two successive R peaks shown in Fig. 1(a). It is required for detection of ventricular "brillation and other lifethreatening arrhythmias or abnormal cardiac rhythms. The need for automated ECG interpretation was primarily felt to ensure continuous and constant observation of the patients' ECG tracings on the oscilloscope at the
* Corresponding author. E-mail address:
[email protected] (M. Nasipuri)
Intensive Coronary Care Unit (ICCU). For the patients, mostly su!ering from myocardial infarction, at the ICCU, the sooner is the detection of arrhythmias, the greater is the chance of recovery [4]. This is so because it was observed that life-threatening arrhythmias were usually preceded by less-severe premonitory arrhythmias [5] and by timely detection and proper therapeutic treatment of the latter, the occurrences of the former might be avoided. The technique of ECG interpretation followed at the ICCU in early 1960s was semiautomatic. It was performed through continuous visual observation of the ECG tracing on the oscilloscope alongwith an automatic heart rate alarm [6]. The importance of automated ECG interpretation was also felt in analyzing the prerecorded 24 h ECG tracing from the ambulatory patient for investigation of possible cardiac events, related to various physical activities and life situations [7]. Usually, computer-based ECG interpretation performs classi"cation of ECG patterns after extracting necessary wave features from the preprocessed digitized
0031-3203/00/$20.00 ( 2000 Pattern Recognition Society. Published by Elsevier Science Ltd. All rights reserved. PII: S 0 0 3 1 - 3 2 0 3 ( 9 9 ) 0 0 0 6 5 - 5
352
M. Kundu et al. / Pattern Recognition 33 (2000) 351}373
Fig. 1. (a) The pattern of a typical ECG segment. (b) Four di!erent chambers of the human heart. (c) Necessary wave features for classi"cation of arrhythmia patterns.
ECG patterns. Since 1960s a great deal of research has so far been done in this "eld [6,8}63,86}106] and knowledge-based ECG interpretation [64}71] has evolved as an o!shoot of it. The major objective of knowledge-based ECG interpretation is to achieve precise and accurate diagnosis through closely modeling the physician's ability to diagnose ECGs by application of various deductive approaches from the "eld of knowledge based computing. A typical block diagram of a knowledge-based ECG interpretation system is shown in Fig. 2. In this system, all the domain knowledge, acquired in consultation with the experts in the "eld of cardiology, is logically encapsulated in a separate module, called the knowledge base (KB). The KB is full of knowledge pieces which are selectively applied to interpret ECG patterns. How best this interpretation can be made with selection and subsequent application of knowledge pieces is determined by
a control program, also known as an inference engine. The task of the inference engine or control program is viewed as a search process seeking a proper sequence of knowledge pieces whose application can interpret the input ECG pattern or classify it into one of the normal and the abnormal ECG pattern classes. So, with the development of the knowledge-based ECG interpretation system, two things become possible. Firstly, addition of any knowledge, that may be acquired for better diagnosis of ECGs in future, can be performed without recompiling the control program, which greatly simpli"es the construction and maintenance of the system. Secondly, due to seggregation of the search process in the form of a control program, heuristics or domain dependent task speci"c information can easily be applied with the control program to make the search for wave features and pattern classes more e$cient.
M. Kundu et al. / Pattern Recognition 33 (2000) 351}373
353
Fig. 2. The block diagram of a typical knowledge-based system for ECG interpretation.
At this point, one should not presume that the other approaches to computerized ECG interpretation, which do not fall in the category of knowledge-based ones, do not require the knowledge about the structural description of the ECG pattern. They do require the knowledge about various medically meaningful geometric features of the ECG pattern. But, unlike the knowledge-based approaches, that knowledge is not easily separable from the description of the task-dependent ad hoc procedures, which constitute the other category of methods for computerized ECG interpretation, called procedural approaches [72]. Under the procedural approaches to ECG interpretation, searches for automatic extraction and interpretation of useful wave features become sometimes exhaustive in nature [8] and there is no provision for application of an heuristic that may reduce the search for "ts. In late 1970s, several research attempts [9}11,17, 29,42,43,46,47,61] were made to encode ECG patterns as strings of symbols, representing various primitive geometric patterns, and to develop syntaxes of normal and abnormal ECGs with formal grammars. Because of easy availability of several algorithms for piecewise linear approximation of discrete functions [9,11,16], the straight line becomes a natural choice for the primitive geometric patterns that constitute complex ECG patterns. Depending on line lengths and slope values, straight lines are encoded into di!erent symbols. Context-free grammars [11,61] and regular grammars [29] are used for ECG pattern classi"cation. Extraction of geometric features, such as straight lines and parabolas, from ECG patterns require "nding the best-"t curve for a large portion of an ECG pattern by running a number of curve-"tting routines. It involves high cost of extensive search. The search for a possible primitive can be focussed to a suitable portion of an ECG pattern by utilizing the structural knowledge of the ECG pattern, encoded in the production rule which has already been selected for derivation. This idea was "rst
introduced for structural pattern recognition of carotid pulse waves by developing a general Waveform Parsing System (WAPSYS) [64], and then extended with the best "rst state space search algorithm for the linguistic analysis of waveforms [73] in general and "nally applied for recognition of ECG patterns [65]. The restriction, imposed by traditional parsing method for strict left to right processing of the input string, was also relaxed in the WAPSYS to allow the detection of more readily recognizable portions of the wave features earlier. Since such features are widely separated in most of the waves, any attempt for early detection of these features may make the task of identi"cation of ambiguous parts of the waveform easy. The general waveform parsing system followed a top-down approach. One major di$culty of applying traditional parsing methods for ECG interpretation is that the parsing methods follow either a top-down or a bottom-up approach strictly, but one such approach alone cannot perform uniformly well over all portions of the ECG pattern. The data directed bottom-up approach works well over those portions of the ECG pattern where concentration of di!erent geometric structures occurs, whereas the model-directed top-down approach works well over the other portions where possibilities for hypothesizing the structures are very few. In order to get the bene"ts of these two approaches together, the CFG, representing an ECG pattern class, is converted into an AND/OR graph and the problem of ECG pattern interpretation is thereby reduced to an AND/OR graph search problem. Kanal applied a best "rst state-space search method, known as SSSH [65,73], from the "eld of Arti"cial Intelligence (AI) for ECG pattern interpretation through an AND/OR graph. The method introduced by Kanal is based on the morphological waveform knowledge of ECG tracings only. But more accurate interpretation of the ECG pattern is possible by applying the physiological event knowledge of the cardiac conduction system together with the morphological waveform knowledge. This is
354
M. Kundu et al. / Pattern Recognition 33 (2000) 351}373
achieved in the Causal Arrhythmia Analysis (CAA) System [66}68], implemented with the knowledge representation language called Procedural Semantic Networks (PSN). In the CAA system a frame-type representation of semantic network is used to maintain a strati"ed knowledge base that contains knowledge about the waveform of ECG signals, knowledge about the physiology of rhythm disorders, and their interrelationships. Projection links are introduced to describe the concept-to-concept relationships accross di!erent KBs. Fast detection of arrhythmias is necessary for the patients in the ICCU where round-the-clock monitoring of ECGs is performed. In view of this, a rule-based approach [70,71] is developed for real-time interpretation of ECG pattern classes by using &if-then' form of rules and facts are prepared with extracted features from the input ECG pattern. It uses modus ponens (MP) as rule of inference from two valued "rst order predicate logic (FOPL). A control program is also there to use the rules and facts to determine the class membership of the ECG pattern. A block diagram of the system has already been shown in Fig. 2. The above methods assume that knowledge of ECG patterns is fully precise and that variation in the input pattern always follows the pattern description stored in the KB. But, in practice, some pattern may slightly di!er from what is described in the KB or the knowledge that is acquired from the physicans may not be quite precise. In such situations, conventionally followed two valued logic, used in all the previously described methods, cannot provide satisfactory solution, i.e. it cannot lead to rational decisions in an environment of uncertainty and imprecision unlike what a human physician can do. Generalized Modus Ponens (GMP) [74], a rule of inference taken from fuzzy logic, is shown to have the capability of detecting "ner variations in ECG patterns without making any augmentation of the existing KB. The previously described rule-based system has been modi"ed for this purpose in Refs. [75}78]. In another separate attempt [69] the control algorithm for the rule-based system is modi"ed to take care of production level parallelism to reduce the length of the match}resolve}act cycle of the system. This opens up possibilities for the development of cost e!ective multipatient arrhythmia recognition systems. All the knowledge based approaches to ECG interpretation, discussed so far, are in general featured by a high level of abstraction and macroscopic view of human reasoning. As an alternative to that, low level, microscopic models of biological networks, controlling human reasoning, are also recently being employed for closer modeling of the cardiologists' diagnostic ability through connectionist or arti"cial neural networks (ANNs) based approaches [79]. Connectionist approaches are mainly featured by learning ability, massive parallelism and fault tolerance. Adaptive resonance the-
ory (ART) networks and multi layer perceptrons (MLPs), two ANN models, have been successfully applied for extraction of wave features [80,81] and recognition of fatal arrhythmias (ventricular tachycardia and ventricular "brillation) [62,81], respectively, from noisy time varying ECG patterns. The major objectives of this review work are "rstly to present a brief account of the state-of-the-art of knowledge based ECG interpretation techniques with a comparative assessment of their performances, secondly to identify the outstanding problems that still creating hinderance to the more satisfactory functioning of computer based ECG interpretation techniques, reported in the recent time, and "nally to indicate possible scope of further work for improvement of their performances.
2. ECG patterns and cardiac disorders The electrocardiogram (ECG) is the record of variation of the bioelectric potential with respect to time as the human heart beats. Fig. 1(a) shows the pattern of a typical ECG segment, recorded under the normal condition of the human heart. In this "gure, various points of de#exions, which are marked as P, Q, R, S and T depending on the curvature around each point, are called waves and correspond to actions of di!erent muscular chambers of the human heart, shown in Fig. 1(b). The P wave corresponds to the contraction of the atria, i.e. the upper heart chambers. The forces of contraction in the atria are generated due to an excitation initiated at the SA node located high at the right atria. This excitation gradually spreads out from the atria to ventricles through the A-V node, the his-bundle and purkinje "bers respectively, causing contraction of the ventricles. This contraction and the relaxation of atria correspond to the R wave alongwith the Q and S waves that form the QRS complex, appearing as a spike in the diagram. Occurring after the QRS complex, the T wave corresponds to relaxation in the ventricles. A single beat of the human heart starts with an atrial contraction and ends with a ventricular relaxation. The pumping action of the human heart is maintained with continuous generation of such heart beats one after another till one's death. Thus the ECG pattern under the normal condition of the human heart shows a periodic nature with R waves occurring at regular intervals of time. Since ECG patterns get distorted with rhythm disorders in the human heart, computerized ECG interpretation requires extraction and analysis of di!erent wave features, like the shape and size of the QRS complex, the number of P waves in an ECG cycle and some other features, shown in Fig. 1(c) for classi"cation of ECG patterns.
M. Kundu et al. / Pattern Recognition 33 (2000) 351}373
3. Representation of ECG patterns and cardiological knowledge For medical diagnosis of ECG patterns, they are usually represented in terms of various medically de"ned wave features, like P wave, P-R duration, QRS complex, S-T segment etc., which in turn are represented with simple geometric structures like straight line, parabolas and peaks [9,11,16]. The cardiological knowledge that is needed for interpreting ECG patterns can be categorized as morphological waveform knowledge of ECG tracings and the Physiological event knowledge of the cardiac conduction system. For usual purposes, the morphological waveform knowledge of ECG tracings is su$cient for quick detection of arrhythmias. Various knowledge-based ECG interpretation approaches mainly di!er from each other in the way the cardiological knowledge is represented. Depending on this their performances also vary. 3.1. AND/OR graph representation of ECG patterns In this approach, an ECG pattern with all its hierarchically decomposed components is represented through an
355
AND/OR graph [65,73], in which nodes are suitably labeled with names of various pattern components. Each node in this graph represents a component pattern except the start node, a specially designated node that represents the entire pattern. It is like that each symbol, nonterminal or terminal, in a context free grammar (CFG) representing a class of ECG patterns, stands for a component pattern except the start symbol which represents the entire pattern. As a production rule in the CFG indicates how a pattern or subpattern, represented by the nonterminal symbol in the left-hand side of the rule, can be decomposed into component patterns, a node in the AND/OR graph does the same, being connected to its successors in the graph through directed arcs. Fig. 3 shows an AND/OR graph that corresponds to a CFG representing a class of typical ECG patterns. Some nodes which are pointed to by hyper-arcs in the AND/OR graph are called AND successors, and the others which are pointed to by normal arcs are called OR successors. The AND successors of a node are labeled with the names of the structural components of a pattern or subpattern denoted by the node. The OR successors of a node are labeled with the names of various alternative
Fig. 3. The AND/OR graph representing a typical ECG pattern.
356
M. Kundu et al. / Pattern Recognition 33 (2000) 351}373
structural descriptions of a pattern or subpattern, like various alternative production rules of a CFG, which involve identical nonterminal symbols to their left hand sides. In Fig. 3, node 1 represents an entire ECG pattern which is constituted with a R wave denoted by node 2 and a train of ECG patterns denoted by node 3. A train of ECG patterns is in turn constituted with an ECG cycle (node 7) starting from a R wave and ending at a Q peak of the next cycle, and the end portion of the train (node 8). The end portion of the train may either end at a sample point denoted by node 10 or continue with the rest portion of the said train, denoted by node 51, depending on the length of the ECG pattern. The tip nodes in this graph represent various pattern primitives and act as the terminal symbols of the pattern grammar. The pattern primitives for the ECG pattern are de"ned as line segments or parabolic segments "tted under certain constraints, such as slope limits, curvature limits, noise limits and segment length limits, which are stored as attributes of the nodes in the graph. The numeric labels on the edges, entering into the node of the AND/OR graph in Fig. 3, indicate the order in which the geometric structures represented by the nodes are to be searched in the input pattern. Thus, the numeric labels help in making non-left-to-right processing of the input pattern possible. 3.2. Rule-based representation of cardiological knowledge For interpretation of arrhythmic ECG patterns, the cardiological knowledge based on the extracted wave features from the input ECGs is to be stored in the computer. The extracted wave features which form the basis of the caridological knowledge are each described through a simple sentence or proposition which can be either true or false depending on the input ECG pattern. For example, an ECG instance with an average heart rate of 100 beats per minute or above is referred as high rate in cardiology. This wave feature can be symbolically denoted by a predicate, HIGH-HEART-RATE, which can be either true or false depending on the time durations of the extracted R-R intervals. This is all about simple propositions. For representation of compound propositions, logical connectives like &and', &or', ¬', are used with such simple propositions. Each unit of the cardiological knowledge, used for analysing arrhythmia patterns on the basis of extracted wave features, is represented in the form of an &if-then rule' or implication as described in [70]. For example, Sinus tachycardia, a class of arrhythmia, occurs if the average heart rate in the input ECG is 100 beats per minute or more } this knowledge is represented as
or HIGH-HEART-RATENSINUS-TACHYCARDIA In this rule or implicational formula, the if-part is called antecedent or precondition and the then-part consequent or conclusion. The above implicational formula is equivalent to the following: &(HIGH-HEART-RATE) s(SINUS-TACHYCARDIA). 3.3. Representation of domain knowledge by PSN A procedural semantic network (PSN) knowledge base consists of objects, which can be tokens, classes, links, relations, and programs. A token represents a particular entity like a QRS pattern, a T wave, a R-R interval, 1 second of time etc. Tokens are interrelated through links representing binary relationships between entities. A class represents certain generic concepts like the class of numbers, the class of QRS patterns, the class of R-R intervals, etc. Each token is an instance of at least one class and each link an instance of at least one relation. Fig. 4 shows two tokens as 1 and 60, classes as R-Rduration, Seconds, and beats-per-minute, a relation as Heart-rate-of and a link relating 1 and 60. It also shows INSTANCE-OF relationship between 1 and R-R duration, 1 and second, 60 and Beats-per-minute, link 1-60 and Heart-rate-of. The semantics or the meaning of a class is de"ned by attaching four programs to each class, which specify respectively how to insert, remove and fetch instances of the class, and how to test whether an object is an instance of the class. Likewise, each relation has four associated programs that specify how to insert or remove instance links, how to fetch all the objects in the range of the relation associated with a particular instance of the domain, and how to test if two objects are interrelated through an instance of the relation. This approach is more modular than what was generally followed by treating the network as a data structure and providing a general global interpreter to query, modify and search the structure.
if (the average heart rate is high) then (the diagnosis is sinus-tachycardia)
Fig. 4. Representation of the relation Heart-rate-of.
M. Kundu et al. / Pattern Recognition 33 (2000) 351}373
In order to design a structured representation of a large knowledge base, PSN o!ers three primitive relations viz. INSTANCE-OF, PART-OF, and IS-A. It has been already shown that INSTANCE-OF links between tokens and classes that constitute their types. In PSN, the INSTANCE-OF relation is used to relate all objects, including tokens, classes, links, relations and programs, to their respective types. The PART-OF relation is based on the organizational principle of Aggregation /Decomposition for structuring a PSN knowledge base. Each concept, a class or relation, can be viewed as an aggregate of some simpler concepts. For example, how a QRST-composite shape can be conceptualized as a combination of its di!erent structural
357
components, is illustrated by the di!erent slots, labeled with the names of corresponding (structural) components in its frame description, shown in Fig. 5(a). The IS-A relation is based on the organizational principle of Specialization/Generalization for structuring a PSN knowledge base. On this principle, classes can be split into subclasses to generate a taxonomy of classes, known as an IS-A hierarchy. The IS-A relation relates a subclass to its class and is graphically represented by a thick arrow with double edges. For example, the class QRST-composite-shape can have "ve subclasses, as shown in Fig. 5(b). A sub-class, which is related to a class by an IS-A relation, can inherit all the properties or slots from the class due to property of inheritance. This is
Fig. 5. (a) Representation of a QRST-COMPOSITE-SHAPE. (b) IS-A hierarchy of QRST-COMPOSITE-SHAPE. (c) The strati"ed knowledge base of the CAA system.
358
M. Kundu et al. / Pattern Recognition 33 (2000) 351}373
why a subclass is de"ned by imposing certain constraints on the ranges of values of the inherited slots and/or de"ning certain additional slots for it only. A class can at a time be related with more than one class by IS-A and INSTANCE-OF relations. The CAA system [66}68] using PSN knowledge bases has to work for recognition purposes. In order to guide the search process, when certain input pattern fails to match with a given class in a PSN knowledge base, various exception conditions, depending on the nature of failure, are associated with a class de"nition, and similarity links are used to associate the other classes to be tried next after an exception occurs. The physiological knowledge base, in the CAA system, represents the knowledge to explain abnormalities in the cardiac conduction system and interpret rhythm disorder as a causal phenomenon in this subdomain. Four basic phases or events of the cardiac conduction system, which are considered as the basic building blocks of the physiological knowledge base, are identi"ed as depolarization, under-repolarization, partial-repolarization, and fullrepolarization. By using PART-OF relation, the basic phase events are aggregated to form di!erent cardiac cycles, as shown in Fig. 5(c). Likewise the cardiac cycles are aggregated to form di!erent cardiac activities, activities to di!erent beats and beats to di!erent beat patterns. Thus a complete beat is conceptualized to consist of a SA-node cycle, an atrium activity, an AV-node activity, a ventricular activity and "nally, a beat pattern is conceptualized to consist of several such beats. In order to represent the causal relationships among the basic events of a single cycle and among the events of di!erent cycles, causal links, as shown in Fig. 5(c), are used. Arrowheaded dotted lines in the "gure denote transfer type links. An event, directed by a transfer type link, occurs when the causative event, shown at the arrow tale, completes in normal course. Initiation type links are shown with arrow-headed double lines in the Fig. 5(c). Each such link is used when the causative event, due to a given subject, triggers a new event of another subject. 3.4. Representation of cardiological knowledge using fuzzy logic Fuzzy logic is based on the concept of fuzzy sets [74]. Fuzzy sets, in contrast to classical sets, have greater #exibility to capture faithfully various aspects of incompleteness and imperfection in a situation. For example, a classical set A, de"ning an arrhythmia class Sinus tachycardia, includes all ECG instances with the average heart rate 100 beats per minute or above by assigning a membership value 1 to each, and excludes all other with the average heart rate below 100 beats per minute by assigning a membership value 0 to each. So the membership function, denoted by k , for a classical set A can A have either 0 or 1 value for any element, u, of its do-
main of discourse denoted by ;. This is expressed as k (u)3M0,1N. The de"nition of Sinus tachycardia as a A classical set is very strict in the sense that it cannot express the impreciseness of the concept that is felt if some one hesitates to classify an instance of ECG pattern with the average heart rate not exactly but nearing 100 beats per minutes as a very clear case of Sinus tachycardia. In order to accommodate this "ne imprecision associated with the de"nition of Sinus tachycardia, the membership value should be allowed to vary smoothly in [0, 1] through a suitably drawn membership function. As a result, the said instance of ECG pattern can be "tted into the de"nition of Sinus tachycardia but its membership value will be something below unity. This is possible by de"ning the concept Sinus tachycardia as a fuzzy set A where kA(u) is the degree of belongingness of u to A or the degree of possessing some imprecise property represented by u and kA(u)3[0, 1]. kA(u) may also be viewed as the extent to which the concept associated with A must be stretched to "t u into it. In fuzzy logic, a fuzzy relation is also de"ned in the same way as the fuzzy set is de"ned. If A represents an n-ary fuzzy relation in ;";1];2]2]; , then the membership function n kA is expressed as kA (u)3 [0, 1], where u3;. 3.4.1. Cardiological propositions in fuzzy logic The meaning of a proposition or simple sentence in fuzzy logic, in contrast to that of a proposition in twovalued logic*which can be either true or false*is expressed by equating the value of a linguistic variable to a fuzzy relation, both of which are implicit in the proposition. For example, in the proposition `average heart rate of ECG cycle is higha, `average heart rate of ECG cyclea is a linguistic variable and `higha is a fuzzy relation. In canonical form, this can be expressed as average-heart-rate (ECG-cycle) is HIGH Like a fuzzy set or relation, which can have a membership function, a linguistic variable can have a possibility distribution de"ned as the set of possible values of the variable with the assumption that possibility is a matter of degree. The values of a linguistic variable in a proposition are constrained by equating its individual possibility distribution to the fuzzy relation in the proposition. Thus the fuzzy relation in a proposition plays the role of an elastic constraint on the variable that is implicit in the natural language proposition. The possibility distribution of the linguistic variable &average-heart-rate(ECGcycle)' can be expressed as follows: PossMaverage-heart-rate(ECG-cycle)N "kHIGH(u), u3;, where ; is the domain of discourse, kHIGH is the membership function of HIGH and kHIGH(u) is the grade of
M. Kundu et al. / Pattern Recognition 33 (2000) 351}373
359
to cardiology by heuristically estimating possibility distributions for linguistic values of the variables concerning di!erent concepts of caridology. Sentences like `if average heart rate of ECG cycle is high then diagnosis is Sinus tachycardiaa, which are used to express various rules for cardiological diagnosis, can each be viewed as a combination of two simple propositions connected through an implication. Such propositions expressing rules are called conditional propositions. The said conditional proposition, in canonical form, is expressed in either of the following forms: (diagnosis (ECG-cycle) is SINUS-TACHYCARDIA) Fig. 6. Membership functions of di!erent linguistic values of average-heart-rate.
If (average-heart-rate (ECG-cycle) is HIGH) or
membership of u in HIGH or equivalently the degree to which u, a numerical value of average-heart-rate, satis"es the constraint induced by the relation HIGH. In order to indicate the fuzzy relation in the above proposition, boldfaced capital letters are used. Fig. 6 depicts the function kHIGH which is plotted on the basis of clinical experiences. The lingusitic variable average-heart-rate(ECG-cycle) may also have other values like not-HIGH, more-or-lessHIGH and very-HIGH. Out of all possible values of a linguistic variable, one like HIGH is called a primary term since the other values of the linguistic variable can be generated from it as follows, when nothing is speci"cally known about them.
(average-heart-rate (ECG-cycle) is HIGH) N(diagnosis (ECG-cycle) is SINUS-TACHYCARDIA)
k "Jk , .03%v03v-%44 13*.!3:v5%3.
In general, a conditional proposition in fuzzy logic, is expressed as (X is A) N (> is B) where X and > represent two linguistic variables, A and B represent two fuzzy sets. Each such implication in fuzzy logic induces a conditional possibility distribution of the linguistic variable > given the possibility distribution of the linguistic variable X, written as % . The fuzzy relation, which (Y@X) corresponds to the conditional possibility distribution function % , is computed as follows (Y@X)
k "(k )2, 7%3: 13*.!3:v5%3.
% (u, v)"kA B(u, v)"minimum(1,1!kA(u)#kB(v)) ? (Y@X)
k "1!k . /05 13*.!3:v5%3.
where u3;, v3<, ; is the universe of discourse of A and < the universe of discourse of B. The rules used in the rulebase of the fuzzy-logic-based system for ECG interpretation are of three types, the purely fuzzy rules with fuzzy propositions in both if-part and then-part, the non-fuzzy rules with non-fuzzy propositions in both if-part and then-part, and mixed category rules with fuzzy and non-fuzzy propositions. Three practical rules from the three respective categories are shown below:
Terms like &more-or-less', &very', ¬' are called modi"ers of the primary term. The fuzzy relation more-or-lessHIGH is called a dilation of HIGH since the dilation tends to increase the degree of membership of all partial members of HIGH by spreading out the characteristic membership function curve of more-or-less-HIGH. The opposite of dilation is concentration and it is, in this case, very-HIGH. It tends to decrease the degree of membership of all partial members of HIGH by concentrating the characteristic membership function curve of veryHIGH. The fuzzy relation not-HIGH is called antonym of HIGH. Fuzzi"cation of facts in the system [71] is performed by comparing numeric values of the extracted ECG features against certain thresholds and assigning appropriate linguistic values to them. For example, a P-R duration having a value 0.185 to 0.199 s is fuzzi"ed as (P-R duration is more-or-less-HIGH). Details of these thresholds for fuzzi"cation can be found in [71,75]. The example, discussed above, shows how the work [71,75}78] attempts to model human intuition relating
(i) If (QRS duration is ABNORMAL) then (QRS complex is ABNORMAL). (ii) If (R-R duration is 70% or more premature) then (diagnosis is R-on-T). (iii) If (current ECG cycle is VPB) ' (last ECG cycle is VPB) ' (R-R duration is nearly equal to P-P duration) ' (average heart rate is 150 beats/min or more) then (diagnosis is ventricular-tachycardia). Out of these rules of three categories, only those of purely fuzzy category generate fuzzy inferences.
360
M. Kundu et al. / Pattern Recognition 33 (2000) 351}373
4. Diagnosis of ECG patterns Diagnosis of ECG patterns, under knowledge-based approaches, is performed by application of the stored knowledge in interpretation of the input ECG patterns. This is called inference process. A control program that guides how the stored pieces of the knowledge can be selectively applied for a step-by-step interpretation of an input ECG pattern is called Control system. The logical connections of a control system to other components of a knowledge based ECG interpretation system are shown in Fig. 2. Keeping in view of one speci"c type of knowledge representation technique, one speci"c inference method is designed. 4.1. Control strategy for the WAPSYS In order to determine the curve-"t constraints required for completing the de"nitions of various pattern primitives, Waveform Parsing System (WAPSYS) [64] "rst allows the user to try various curve "tters on some sample ECGs. A testing module of the system is then used to automatically identify the same curve "t constraints on other samples to check the tentative grammer. In order to recognize an input ECG, probable candidates for a prominent waveform feature are "rst located in it by WAPSYS. Then the parsing process starts, being guided by the knowledge of the ECG pattern stored through an AND/OR graph. Smaller the number labeling an edge, higher the prominence of the node at the edge-tip. As the AND/OR graph of Fig. 3 shows, the upslope (UPSLOP) of a R wave (SRT) in an ECG pattern is the most prominent waveform feature. When the existence of a waveform feature, primitive or compound, is ensured in an interval of the input pattern, the corresponding node in the AND/OR graph is con"rmed. Expansion of a node is done in view of con"rming its descendants according to their priorities. In order to accomplish the task of expanding or con"rming a node, WAPSYS calls a user supplied semantic routine, meant for purposes as calling curve "t constraints, recording peak measurements or noting intervals. The routine is also meant for making further measurements such as adding intervals together until the interval for a complete ECG cycle is obtained. The WAPSYS, after automatically locating the steepest upslope in the input ECG pattern, con"rms node 11 and expands node 12 to locate the large negative down slope of the R wave in a speci"ed interval adjacent to right of the upslope and on recognition of the large negative down slope, node 12 is con"rmed. With this, node 2 is also con"rmed and a R wave in the input ECG is located. The WAPSYS then expands nodes 3, 7 and 2 in order. With the con"rmation of node 2, the next R wave in the wave train is located, and the period, rate and baseline drift of the "rst cycle between R waves are computed. In order to parse the wave seg-
ment between the two R waves in the input pattern, the WAPSYS then expands node 9. Con"rmation of node 9 requires con"rmation of all waves and intervals in the cycle. Nodes 8 and 51 are expanded to locate the next R wave in the input wave train. When no part of the input wave train is left to be parsed, node 10 is con"rmed. 4.2. Control strategy for the rule based system A rule-based expert system is formed by representing the domain knowledge in the way described in Section 3.2. The expert system [70] that interprets ECG patterns applies the domain knowledge on the extracted wave features to draw an inference. The rule of inference, followed by the expert system, is Modus Ponens [82]. It works as follows: Fact: Rule:
HIGH-HEART-RATE HIGH-HEART-RATENSINUSTACHYCARDIA Inference: SINUS-TACHYCARDIA The above example shows that under Modus Ponens [70], a rule can be applied only when its precondition can be satis"ed with any of the facts about the input ECG pattern. With a rule application, the consequent of the rule is considered as the inference. Classi"cation of an input ECG pattern into one of the arrhythmia classes requires several such rule applications, which form a chain of inferences. All facts prepared from the input ECG pattern are conjunctively connected to form the fact expression that is stored in the fact base shown in Fig. 2. An inference, drawn through a rule application, is also stored in the fact base to be used as a new fact in subsequent rule applications, resulting into a chain of inferences necessary for class"cation of the input ECG pattern. The rule, cited in the above example, is of simplest type. Let us consider the following three practical rules to observe how these rules can be applied by the control system to form a chain of inferences with an input ECG pattern. 1. (i) If (QRS amplitude criteria is not satis"ed) then (QRS complex is abnormal). (ii) If (QRS duration criteria is not satis"ed) then (QRS complex is abnormal). 2. If (R-R duration is 20% or more premature) ' (QRS complex is abnormal) then (diagnosis is VPB) 3. If (a VPB occurs in current cycle) ' (another VPB occurred in last cycle) ' (average heart rate is as high as 150 beats/min) then (diagnosis is ventriculartachycardia).
As illustrated in Fig. 7, the rule number 1(i) or 1(ii) can be directly applied with the extracted fact that &QRS
M. Kundu et al. / Pattern Recognition 33 (2000) 351}373
361
Fig. 7. The chain of inferences using Rule nos. 1 and 2.
duration or amplitude criteria is not satis"ed' to assert a new fact that &QRS complex is abnormal'. With this asserted fact and another extracted fact that &R-R duration is premature', rule number 2 can be applied to assert another new fact that ¤t ECG cycle is VPB'. With this and two more facts that &another VPB occurred in last cycle' and &average heart rate is 150 beats per minute', the rule number 3 can be applied to complete the chain of inferences that results into the "nal conclusion that &diagnosis is ventricular-tachycardia'. A selected rule is applied over the factbase under the supervision of the control system by asserting its conclusion. The control system completes inference about the current ECG cycle after classifying it into one of the disjunctive ECG pattern classes in the goal expression. Each time all the facts about an ECG cycle arrive at the factbase the control system is activated for drawing inferences. Also the diagnosis of one cycle is completed before all the facts about the next cycle arrive at the factbase. This makes it feasible to utilize the result of diagnosis of an earlier cycle in inferring about the next cycle. 4.3. Control strategy for arrhythmia recognition under the CAA system The control strategy for arrhythmia recognition depends on three major activites of the control system, viz., peak detection, waveform analysis and event analysis [66}68]. First, peaks in the input ECG are detected by application of a syntactic approach introduced by Horowitz [9]. It requires piecewise linearization of the ECG pattern by the split-and-merge technique. The piecewise linearized pattern is encoded into a string of symbols to be parsed by a grammar, specially designed to recognize and measure the attributes of all peak patterns in the input ECG. After all peaks are detected and measured, some consecutive peaks, with high amplitudes and steep slopes, are chosen as the candidate wave pattern for analysis. Recognition of wave shapes from the candidate pattern starts by hypothesizing the most generic shape class and grad-
ually specializing it into more speci"c classes along the IS-A class hierarchy described in the morphological KB. Con"rmation of a class hypothesis requires satisfying all slot constraints in the class de"nition by the morphological features of the candidate wave patterns. Thus the wave shape recognition strategy, which works through gradually specializing class hypotheses, also requires aggregation of features of the candidate pattern for hypothesis con"rmation. If an attempt to con"rm a hypothesis fails, an alternative hypothesis is chosen with the help of similarity links. Recognition of a candidate wave pattern is completed with the con"rmation of the hypothesized classes for it. The recognized wave classes are projected into the physiological event domain to form their corresponding event hypothesis. For example, a candidate wave pattern recognised as STANDARD-QRST-COMPOSITE-SHAPE with STANDARD-QRS-COMPLEX-SHAPE and NORMAL-T-WAVE-SHAPE, can be projected into the event domain as illustrated earlier in Fig. 5(c). This generates the set of basic event hypotheses, which consists of the depolarization and the partial repolarization phase events of the left ventricle. In order to perform event analysis with the basic events hypotheses, the causal links which are associated with the basic events are used to generate expectations for other events. All these events are gradually aggregated to hypothesize beat event class depending on the projected and expected events. Thus, from each candidate wave pattern (also known as a set of signal tokens), a beat event class is hypothesized. Since several beat patterns constitute a periodic arrhythmia pattern, its repetitive behaviour is de"ned by the recursive de"nition of beat-pattern frames as shown in Fig. 8. In order to recognize an arrhythmia in the input ECG pattern, global hypothesis about an arrhythmia class is to be generated "rst. The hypothesis projection method described above is used to examine each projected class and decide whether the class must be included in the current global hypothesis as a component hypothesis. In the process of forming beat patterns, causal links between adjacent beats allow
362
M. Kundu et al. / Pattern Recognition 33 (2000) 351}373
Fig. 8. Various specializations of the class REPETITIVE- RHYTHM-PATTERN.
the system to verify the causal relationship that governs the pace-making mechanism on a beat-to-beat basis. While recursively hypothesizing consecutive beat events, the system can rate the degree of consistency by testing them against the corresponding wave sequence. Salient features of the control strategy can be summarized as follows: (i) Provision for means of hypothesis projection accross distinct KBs. (ii) Exploitation of causal knowledge about cardiological events. (iii) Recognition of repetitive event sequences and detection of beat-to-beat relationship.
patterns of normal pacing beats and Sinus tachycardia. There, a heart rate of 105 beats per minute is interpreted as a case of Sinus tachycardia, whereas, in reality, it is closer to a case of Sinus tachycardia rather than a clear one. In order to take care of such variations in the ECG patterns, the present system, equipped with Generalized Modus Ponens (GMP) [74], can perform quali"ed diagnosis. Here an average heart rate of 105 beats per minute is expressed in the form of a fuzzi"ed fact as &averageheart-rate(ECG-cycle) is more-or-less-HIGH' and this cannot exactly satisfy the precondition in the above conditional proposition. In such cases, the inference, under GMP, is drawn by correlating the existing fact with the rule's precondition as follows:
4.4. Control strategy for the fuzzy rule based system
Fact:
Fig. 9 shows the block diagram of the fuzzy-rule-based system [71,75}78] designed for interpretation of ECG patterns. The strategy followed for forming the necessary sequence of inferences by the system is in principle same as that illustrated in Section 3.4.1 for the rule-based system. But the rule of inference that guides a single rule application under fuzzy logic is not same as MP (Modus Ponens) of two-valued logic. Since fuzzy logic allows imprecise mode of reasoning, a multitude of separate rules, accounting for each possible variation of pattern primitives of the ECG cycle, need not be introduced as would have otherwise been necessary for accommodating these variations within the system based on two-valued logic. Consider an example. In the rule-based arrhythmia recognition system that was designed to work under two-valued logic, an average heart rate of 100 beats per minute was used as a line of demarcation between ECG
(average-heart-rate (ECG-cycle) is more-orless-HIGH) Rule: (diagnosis (ECG-cycle) is SINUS-TACHYCARDIA) If (average-heart-rate(ECG-cycle) is HIGH) Inference: (diagnosis(ECG cycle) is more-or-less-SINUSTACHYCARDIA) The above example illustrates that a composite constraint, from the given propositions, is induced into the inference, as an inferred fuzzy relation more-or-lessSINUS-TACHYCARDIA, through a process of constraint propagation. The inferred fuzzy relation is obtained by Zadeh's compositional inference rule [74] as follows: Fact: (X is AH) Rule: (X is A)N(> is B) Inference: (> is BH)
M. Kundu et al. / Pattern Recognition 33 (2000) 351}373
363
Fig. 9. The system for fuzzy logic-based ECG interpretation.
where, BH"AH " (ANB), with " being the symbol for Zadeh's compositional inference rule, kHB (v)"max Mmin(kA*(u), kB A(u, v))N, u3;, v3<, u @ where ; is the universe of discourse for A and AH and < is the universe of discourse for B and BH. The above computation can be viewed as a vector}matrix product with multiply and add operations replaced by min and max [74,83], respectively. Rules with more than one conjunctively related fuzzy preconditions of the form ((X is A ) ' (X is A )N(> is B)) 1 1 2 2 are to be expressed as (((X is A )N(> is B)) s ((X is A )N> is B))) 1 1 2 2 to allow to draw inferences as follows: Fact 1: Fact 2: Rule:
(X is AH) 1 1 (X is AH) 2 2 (((X is A ) N (> is B)) s ((X is A ) 1 1 2 2 N (> is B))) Inference: (> is (BH ; BH)) 1 2
In order to compute the union of two fuzzy relations, the following expression [74] can be used: kBH u BH(v)"max(kH1(v), kH2(v)) B B 1 2 where v3< and < is the universe of discourse for B, BH and BH. 1 2 The rules, which have nonfuzzy relations in their conclusions, give nonfuzzy inferences when applied over the facts. 4.4.1. Rule selection mechanism Matching the fuzzy preconditions of rules with existing facts necessitates matching of the corresponding fuzzy relations. For this, a normalized distance measure d ben tween fuzzy relations A and AH, both of which are de"ned in the same universe of discourse ;, is selected as a measure of dissimilarity. It is computed as follows: 1 n d (A,AH)" + DkA(u )!kAH(u )D n i i n i/1 where ; is the universe of discourse for A and AH, u 3; i and n is the cardinality of ;. The said measure is experimentally found best [71,75] for the fuzzy relations, involved with the present application. In order to match conjunctively related preconditions of a rule against the existing facts, the maximum of
364
M. Kundu et al. / Pattern Recognition 33 (2000) 351}373
all dissimilarities arising out of the said preconditions is to be considered. The block for &Rule Selection Mechanism' in Fig. 9 works for computing the dissimilarity measures for all applicable rules. Selection of a rule, which has the minimum dissimilarity measure with respect to the facts about the current ECG cycle, is also the task of this block. 4.4.2. Linguistic approximation function The block &Linguistic Approximation Function', shown in Fig. 9, works to "nd an equivalent linguistic expression for the inferred fuzzy relation, delivered from the control system. It checks for dilation or concentration in the inferred fuzzy relation by comparing the sum of its membership values with that of its normal variant. A dilated relation is comprehended by putting the modi"er &more-or-less', before the relation name, whereas a concentrated one is comprehended by putting the modi"er &very' before the same. 4.5. Rule-based ECG interpretation with a distributed control A distributed reasoning approach is followed in Refs. [69,71] for rule-based ECG interpretation. The task of the control system, described in Section 3.2, is subdivided here into two subtasks. The subtasks are performed by two cooperating submodules which are capable of communicating with each other through speci"ed communication paths. Such parallelism in the functions of the control system is utilized to enhance the speed of ECG interpretation process. The approach, followed here for modularization of the control system, also makes the conditions for development and maintenance of the system more favourable. Distributed reasoning systems are very suitable for exploiting the inherent parallelism that often exists in a rule-based system. Such parallelism may broadly be classi"ed as (1) Match level parallelism. (2) Task level parallelism. For the ECG interpretation problem, studied here, the match level parallelism is more relevant. The two main aspects of match level parallelism are (i) Production level parallelism. (ii) Condition level parallelism. Production level parallelism refers to the situation where a fact, when available, may be made to match with the preconditions of all the rules of the rule base simultaneously instead of sequentially. Condition level parallelism refers to the situation when evaluating the antecedent of a rule requires the testing of more than one precondi-
tions and all these preconditions may be tested concurrently instead of one after another sequentially. The objective of the distributed reasoning system that performs ECG interpretation is to speed up the inference system by reducing the length of match}resolve}act cycle for rule "ring. It tries to exploit the production level parallelism present in the rule base. In the proposed system, the tasks of matching and rule selection are carried out concurrently. Another important objective of this system is to incrementally process the ECG waveform. With incremental processing, it is possible to process each component fact about an ECG cycle as soon as it is available instead of waiting for the remaining facts about the cycle to be computed. Thus, the aim of the incorporation of the capability of incremental processing is to make the proposed system more suitable for on-line processing of ECG signals as required in the ICCU. Fig. 10 shows the block diagram of the system. At the core of the system there is a rule-based expert which works over the facts, prepared in succession by the preprocessing and wavefeature extraction modules. The rule-based expert has all its usual components, as mentioned before. Every entry of the fact base corresponds to a predicate and links all rules which use the predicate as precondition. The di!erent arrhythmia classes alongwith a class of normal ECGs constitute the goal expression as disjunctions of literals and correspond to di!erent entries of the goal base. There are certain subgoals, like &(QRS complex is abnormal)', as shown in Fig. 7, which require assertion after they are proved. This is to facilitate further inferences. For example, once a ventricular premature beat (VPB) is detected with an abnormal QRS complex occuring in a premature R-R interval, the next question arises whether it is a case of Ventricular tachycardia with more than two VPBs occurring successively at a heart rate of 150 beats/min or more. So every entry of the goal base, which corresponds to a particular goal or subgoal, stores certain actions to be followed after the goal or subgoal is proved. Entries of the rule base correspond to di!erent rules which are same as those described in Section 3.2. The consequent part of every rule bears a reference to a goal or subgoal. This is denoted through a link from the corresponding entry of the rule base to the appropriate entry of the goal base. The inferences drawn by the control module through a sequence of rule applications are stored in the archival storage of information outside the system. The control system, which always maintains a list of partially applicable rules on the basis of the latest available facts, attempts to "re an appropriate one from the list after the arrival of every piece of fact. All functions of the control system can be better understood if it is considered to be consisting of two concurrent processes capable of communicating with each other. The process(d1) always checks if a fact arrives and, on receiving
M. Kundu et al. / Pattern Recognition 33 (2000) 351}373
365
Fig. 10. Rule based ECG interpretation system with distributed control.
a fact, it sends a copy of the list of applicable rules corresponding to the fact from the fact base to the process(d2). The process(d2) always maintains a list of partially applicable rules on the basis of the most recently available facts by taking union of the existing list and the list received from the process(d1). If all preconditions for any of the rules from the union are found satis"ed, then the process(d2) immediately "res the rule after making the current list null. Firing of a rule is followed by creation of diagnositic instances by process(d2) if an abnormality is detected in the input signal. Process(d2), guided by the actions enlisted in the Goal table, also sends the proven subgoal to process(d1) if it requires assertion, after being proved. It may also be noted that the implementation here is fast enough so that all facts of an ECG cycle are completely processed before the arrival of facts about the next cycle.
5. Connectionist approaches Connectionist approaches [79], also known as arti"cial neural networks (ANNs) based approaches, seek to
develop simpli"ed cell (neuron) level models of the biological neural networks for drawing inferences. Information in ANNs are encoded in the connection weights and learning is achieved by suitably adjusting the strengths of connections between neurons, i.e., by weight updation. The features of an ANN, which make it attractive for ECG interpretation, are its adaptivity to frequently changing scenario, robustness to imprecision and uncertainty, massive parallelism and high degree of interconnections, and "nally its fault tolerant capability. A fuzzy rule-based system has already been discussed in respect to its e!ectiveness to deal with imprecision and uncertainty, prevailing in the ECG patterns. However, it requires quite an intensive amount of computation which can otherwise be well managed by an ANN. The problem of detecting biologically signi"cant points in noisy and time variant ECG patterns has also been discussed previously. An ANN, due to its learning ability, may be a suitable choice for this. Unlike the rule based system, which relies on the expert's input, the ANN models, used for ECG interpretation, o!er the advantage of knowledge base, derived directly from the set of training examples. Such ANN-based systems with generalization
366
M. Kundu et al. / Pattern Recognition 33 (2000) 351}373
capability are adaptive to the variations of the input ECGs and once trained are capable of providing diagnosis speedily. Multi layer perceptron (MLP) and adaptive resonance theory (ART) networks are the important models of ANNs, which in the recent time have been successfully applied for solving various subproblems of ECG interpretation. Applications of MLPs in removal of noise from the ECG data [81], and for recognition of fatal arrhythmias like VF and VT [62,81] have shown encouraging results. Some ART networks have also been applied for extracting wave features from the ECG patterns. A Fuzzy ARTMAP network [81] has been successfully used for R wave detection. Two ART networks [80,81] have been applied for establishing the search regions for Q and S points. The performance of the system for detection of Q and S points with ART2 networks has been experimentally found to be more robust than the conventional slope detection method [84]. The average recognition errors for Q and S points are found to be 0.59 and 0.60 ms, respectively, under the said system. Compared to the 4 ms sampling interval of AHA database, the rates of occurrences of an error more than 4 ms are only 1 and 2% for Q and S points, respectively.
6. Discussion It is clear from the previous sections that the primary objective of all the knowledge based ECG interpretation approaches is closer modeling of cardiological domain knowledge for better diagnosis of ECG patterns. In ful"lling this objective, each approach is devised with one particular knowledge representation technique. AND/OR graph representation, "rst order predicate logic (FOPL), procedural semantic network and fuzzy
logic are the knowledge representation techniques followed there. The ECG interpretation approaches which are based on AND/OR graph, FOPL and fuzzy logic use the morphological domain knowledge of the ECG pattern only, whereas the CAA system which is based on PSN uses the knowledge of cardiac conduction system in addition to that of the ECG pattern. The CAA system can be taken as a model to show how the physiological knowledge of the individual electrical discharges can be correlated with the morphological knowledge of the ECG pattern to achieve a very high degree of reliability (over 80%) in diagnosing abnormal ECGs, compared to certain traditional ECG analysis programs [66}68]. But due to non availability of information about its response time, the question about its suitability of use in the ICCU remains unanswered here. The rule-based system that was experimentally found to have taken on the average 0.5158 ms, on an 80286 based PC/AT with 12 MHz clock rate, is suitable for ICCU application. However, the rule based system, working under two valued logic, lacks the ability of dealing with incomplete and imprecise knowledge, which is very common in medical diagnosis of ECGs. The fuzzy logic based system is experimentally found to have improved performances in this regard. Fig. 1 shows two outputs obtained from the same lead II ECG data by rule based system and the fuzzy logic based system respectively. Diagnoses, obtained through two valued logic and fuzzy logic, are shown on the same diagram of Fig. 11. The diagnoses with two valued logic appear at the top and that with fuzzy logic appears at the bottom of the diagram. The diagnoses for cycle(d1) of data set(d4) are obtained as &(QRS complex is more-or-less-NORMAL) and (diagnosis is more-orless-SINUS-BRADYCARDIA)' from the fuzzi"ed facts like &(QRS duration is NORMAL), (QRS amplitude is
Fig. 11. Comparison of diagnoses performed through two valued logic and fuzzy logic on data set (d4).
M. Kundu et al. / Pattern Recognition 33 (2000) 351}373
367
Fig. 12. (a) The result of noise removal from ECG data with the MLP network. (b}c) Diagnoses performed through the MLP network for VF and VT detection. (d) The results of "ducial point detection in ECG data with the system of two fuzzy ARTMAP networks.
more-or-less-NORMAL) and (average heart rate is more-or-less-LOW)'. When the result of this diagnosis is compared with that obtained through two valued logic, it is observed that
fuzzy logic-based system takes care of "ner variations in the input ECG patterns and quali"es the diagnosis with the modi"er &more-or-less', which was in previous case a plain one. This is achieved through fuzzi"cation of facts
368
M. Kundu et al. / Pattern Recognition 33 (2000) 351}373
Fig. 12. (Continued.)
M. Kundu et al. / Pattern Recognition 33 (2000) 351}373
and approximation of diagnoses and, of course, without the need of adding any new rule to the existing rule base, used previously for two valued logic [70,71]. The average heart rate in data set(d4) is 57 beats/min. Assuming 60 beats/min to be the normal heart rate, the "ner variations of the heart rate, slightly di!erent from the normal, is well detected by the fuzzy method without calling for addition of any new rule. This is made possible by enabling the rule based system to deal with the inexact and incomplete knowledge. The average execution time required for interpretation of an ECG cycle on an 80286 based PC/AT with 12 MHz clock rate was observed as 153 ms Even this time of 153 ms is much less than the average time period (1000 ms) of a normal ECG cycle. Thus, the requirement for diagnosing ECG signals from multiple patients is amply satis"ed with this method and it also provides a more appropriate approach to better diagnosis of multiple patients in real time. As the fuzzy logic based approach shows one direction of work, branching from the rule based approach in view of improving the quality of ECG diagnosis, the distributed control algorithm shows another direction of work, branching from the same in view of enhancing the speed of ECG interpretation. The distributed control algorithm for the rule-based arrhythmia recognition system was tested with real ECG data, under a simulated environment. The computer system, used for the work, was 80286 based PC/AT with 12 MHz clock rate. With the said experimental setup, the average processing time for arrhythmia recognition was observed as 0.3968 ms per ECG cycle. The system speedup as compared with the serial version of the control algorithm is calculated as 1.3("0.5158 ms/0.3968 ms) from the experimental results. The gain may be higher if the size of the rule base used for the prototype system is extended. Some of the results, obtained through the above mentioned connectionist approaches on ECG data, are shown in Figs. 12(a)}(d). Fig. 12(a) shows the e!ect of using the MLP network on ECG data for noise removal. Figs. 12(b) and (c) show accurate detection of two fatal arrhythmias, requiring special attention, with the use of another MLP network, mentioned previously in Section 5. Fig. 12(d) shows accurate detection of "ducial points in ECG data by application of the fuzzy ARTMAP networks, also referred to in Section 5. All these results are encouraging ones.
7. Future directions of the work One major di$culty in replicating the physician's knowledge through predicate calculus expressions is that sometime physicians prefer to express their experiences with dispositions like `Usually such thing happensa or
369
`Quiet often such thing happensa. Modeling of the dispositional knowledge [74] is necessary for betterment of the ECG diagnosis. Application of fuzzy logic can be an appropriate "eld of study in resolving this problem. For knowledge based interpretation of other bio-electic signals like Carotid-Pulse wave, Electro-encephalogram (EEG), Blood pressure wave [8], etc., which follow a regular pattern under the normal condition of the human body, the present technique can provide some useful hints to the researchers. The ECG interpretation system, for which the average time needed for diagnosing each ECG cycle is much less than the time period of an ECG cycle, can be further studied for the development of a real-time multipatient cost e!ective ECG interpretation system. In order to integrate the power of fuzzy logic with that of the neural networks, investigations are going on for exploring the scope of application of certain neuro-fuzzy models [85] for ECG interpretation under the CMATER Project at Jadavpur University.
Acknowledgements The work is partly supported by the `Development of Fuzzy Logic Technology for interpretation of Bioelectric Signalsa project of the Centre for Microprocessor Application for Training Education and Research, Department of Computer Science and Engineering, Jadavpur Unviersity. The project was funded by the AICTE, Government of India.
References [1] M.J. Goldman, Principles of Clinical Electrocardiography, 11th Edition, Lange Medical Publications, Drawer L, Los Altos, California 94022. [2] D. Rowlands, Understanding the Electrocardiogram: A New Approach, Churchill Livingstone, Medical Division of Longman Group Ltd., ISBN 0443025061, 1980. [3] M.B. Conover, Pocket Guide to Electrocardiography, Galgotia Pub. Pvt. Ltd., New Delhi, 1989. [4] H.W. Day, An intensive coronary care area, Dis. Chest, 44 (1963) 423}427. [5] B. Lown, M.D. Klein, P.I. Hershberg, Coronary and precoronary care, Amer. J. Med. 46 (1969) 705}724. [6] L.J. Thomas Jr., K.W. Clark, C.N. Mead, K.L. Ripley, B.F. Spenner, G.C. Oliver Jr., Automated Cardiac disrhythmia analysis, Proc. IEEE 67 (9) (1979) 1322}1399. [7] N.J. Holter, New method for heart studies, Science 134 (1961) 1214}1220. [8] J.R. Cox Jr., F.M. Nolle, R.M. Arthur, Digital analysis of the electroencephalogram, the blood pressure wave, and the electrocardiogram, Proc IEEE 60 (1962) 1137}1164.
370
M. Kundu et al. / Pattern Recognition 33 (2000) 351}373
[9] S.L. Horowitz, A syntactic algorithm for peak detection in waveforms with applications to cardiography, CACM 18 (5) (1975) 281}285. [10] E. Skordalakis, Syntactic ECG processing: a review, Pattern Recognition 19 (4) (1986) 305}313. [11] K. Udupa, I.S.N. Murthy, Syntactic approach to ECG rhythm analysis, IEEE Trans. Biomed. Eng. BME-27 (7) (1980). [12] M.L. Ahlstrom, W.J. Tompkins, Automated high speed analysis of Holter tapes with microcomputers, IEEE Trans. Biomed. Eng. BME-30 (10) (1983) 651}657. [13] P.W. Macfarlane, ECG waveform identi"cation by digital computer, Cardiovasc. Res. 5 (1971) 141}146. [14] M. Kunt, H. Rey, A. Lightenberg, Preprocessing of electrocardiograms by digital techniques, Signal Processing 4 (2&3) (1982) 215}222. [15] C.H.H. Chu, E.J. Delp, Impulse noise suppression and normalization of electrocardiogram signals, IEEE Trans. Biomed. Eng. 36 (2) (1989). [16] T. Pavilidis, S.L. Horowitz, Segmentation of plane curves, IEEE Trans. Comput. C-23 (8) (1974) 860}870. [17] G. Papakonstantinou, F. Gritzali, Syntactic "ltering of ECG waveforms, Comput. Biomed. Res. 14 (1981) 158}167. [18] W.C. Mueller, Arrhythmia detection program for an ambulatory ECG monitor, Biomed. Sci. Instrum. 14 (1978) 81}85. [19] J.P. Abenstein, W.J. Tompkins, A new data reduction algorithm for real time ECG analysis, IEEE Trans. Biomed. Eng. BME-29 (1) (1982) 43}48. [20] J.R. Cox et al., Compact digital coding of electrocardiographic data, in Proceedings of the sixth Hawai International Conference of System Science (Honolulu, Hawaii), 1973, pp. 333}336. [21] M.E. Womble et al., Data compression for storing and transmission of ECGs and VCGs, Proc. IEEE. 65 (5) (1977) 702}706. [22] P.O. Borjesson, O. Pahlm, Predictive coding for ECG data compression, Tech. Rep. TR-87, Dept. of Telecommunication Theory, University of Lund, Sweden, 1977. [23] U.E. Ruttimann, H.V. Pipberger, Compression of the ECG by prediction of interpolation and entropy encoding, IEEE Trans. Biomed. Eng. BME-26 (11) (1979) 613}622. [24] N. Ahmed, P.J. Milne, S.G. Harris, Electrocardiographic data compression via orthogonal transforms, IEEE Trans. Bomed. Eng. BME-22 (6) (1975) 484}487. [25] A. Sandaman, B. Sapir, Third order polynomial-its use in data compression, Signal Processing 15 (4) (1988) 405}418. [26] C.N. Mead, T. Ferriero, K.W. Clark, L.J. Thomas, Jr., J.R. Cox, Jr., G.C. Oliver, An improved Argus/H system for high-speed ECG analysis, Comp. Cardiol. (IEEE Cat. No. 75CH1018-C), 1975, pp. 7}13. [27] R. Dillman, N. Judell, S. Kuo, Replacement of AZTEC by correlation for more accurate VPB detection, Comp. Cardiol, (IEEE Cat. No. 78CH1391-2C), 1978, pp. 29}32. [28] W. Rey, J.D. Laird, P.G. Hugenholtz, P-wave detection by digital computer, Comput. Biomed. Res. 4 (1971) 509}522.
[29] G. Belforte, R. De Mori, F. Ferraris, A contribution to the automatic processing of electrocardiograms using syntactic methods, IEEE Trans. Biomed. Eng. BME-26 (1979) 125}136. [30] F. Gritzali, Towards a generalized scheme for QRS detection in ECG waveforms, Signal Processing 15 (2) (1988) 183}192. [31] W.P. Holsinger, K.M. Kempner, M.H. Miller, A QRS preprocessor based on digital di!erentiation, IEEE Trans. Biomed. Eng. BME-18 (1971) 212}217. [32] A. Ligtenberg, M. Kunt, A robust digital QRS detection algorithm for arrhythmia monitoring, Comput. Biomed. Res. 16 (1983) 273}286. [33] G.M. Friesen et al., A comparison of the noise sensitivity of nine QRS detection algorithms, IEEE Trans. Biomed. Eng. 37 (1) (1990) 85}98. [34] W.A.H. Engelse, C. Zeelenberg, A single scan algorithm for QRS-detection and feature extraction, IEEE Comput. Card., IEEE Computer Society, Long Beach, 1979, pp. 37}42. [35] P.M. Mahoudeaux et al., Simple microprocessor based system for on-line ECG analysis, Med. Bio. Eng. Comput. 19 (1981) 497}500. [36] J. Fraden, M.R. Neuman, QRS wave detection, Med. Biol. Eng. Comput. 18 (1980) 125}132. [37] C.L. Feldman, M. Hubelbank, Cardiovascular monitoring in the coronary care unit, Med. Instrum. 11 (5) (1977). [38] J.M. Arnold, P.M. Shah, W.B. Clarke, Arti"cial rejection in a computer system for the monitoring of arrhythmias, Comput. Cardiol. (IEEE Cat. no. 75CH1018-1C), 1975, pp. 163}167. [39] R. Dillman, N. Judell, S. Kuo, Replacement of AZTEC by correlation for more accurate VPB detection, Comput. Cardiol. (IEEE Cat. No 78CH1391-2C), 1978, pp. 29}32. [40] Quin-Lan Cheng, Ho Soo Lee, N.V. Thakor, ECG waveform analysis by signi"cant point extraction - II, Comput. Biomed. Res. 20 (1987) 428}442. [41] K.P. Lin, W.H. Chang, QRS feature extraction using linear prediction, IEEE Trans. Biomed. Eng. 36 (10) (1989) 1050}1055. [42] P. Trahanias, E. Skordalakis, G. Papakonstantinou, A synatactic method for the classi"cation of the QRS patterns, Pattern Recognition Lett. 9 (1989) 13}18. [43] E. Skordalakis, P. Trahanias, Primitive pattern selection and extraction in ECG waveforms, in Proceedings of eighth International Conference on Pattern Recognition, 1986, pp. 380}382. [44] S.H. Rappaport, L. Gillick, G.B. Moody, R.G. Mark, QRS morphology classi"cation: quantitative evaluation of di!erent strategies, Comput. Cardiol. 1982, pp. 33}38. [45] E. Kimura, Y. Mibikura, S. Miura, Statistical diagnosis of electrocardiogram by theorem of Bayes, Jpn. Heart J. 4 (1963) 469}488. [46] G. Papakonstantinou, E. Skordalakis, F. Gritzali, A attribute grammar for QRS detection, Pattern Recognition 19 (4) (1986) 297}303. [47] G. Papakonstantinou, An interpreter of attribute grammars and its application to waveform analysis, IEEE Trans. Software Eng. SE-7 (3) (1981) 279}283.
M. Kundu et al. / Pattern Recognition 33 (2000) 351}373 [48] R.E. Bonner et al., A new computer program for analysis of scalar electrocardiograms, Comput. Biomed. Res. 5 (1972) 629}653. [49] D.A. Coast, R.M. Stern, G.G. Cano, S.A. Briller, An approach to cardiac arrhythmia analysis using hidden Markov models, IEEE Trans. Biomed. Eng. 37 (9) (1990) 826}836. [50] W. Gersch, P. Lilly, E. Dong Jr., PVC detection by the heart-beat interval data-Markov chain approach, Comput. Biomed. Res. 8 (1975) 370}378. [51] R.V.H. Vinke et al., Classi"cation of cardiac rhythms using theory of Markov chains, in: Comput. Cardiol. Vol. 6, IEEE, New York, 1979, pp. 255}258. [52] P. Schluter, D. Clapham et al., The design and evaluation of a computer based system for holter tape analysis, in: Comput. Cardiol., IEEE, New York, 1984, pp. 193}196. [53] C.L. Feldman, Computer Based Holter Scanners } Current Status, Cardiodata, Northboro, MA, 1984. [54] M. Kundu et al., Microprocessor based on-line feature extraction technique for ECG waveform, in: Proceedings of Third Syposium International de lngineria Biomedica, Madrid, Spain, 7}9 Oct. 1987, pp. 385}389. [55] M. Nasipuri, D.K. Basu, R. Dattagupta, M. Kundu, S. Banerjee, A new microprocessor based on-line arrhythmia diagnostic system, in: Proceedings of IEEE Region Ten Conference (TENCON '87), Seoul, Korea, 26}28 Aug. 1987, pp. 1187}1191. [56] M. Nasipuri, D.K. Basu, M. Kundu, R. Dattagupta, A new microprocessor based system for on-line data reduction of ECG signal and arrhythmia detection, in: Proceedings of Biomedizinische Technik 88, Universitat Stuttgart, 7}10 Sep. 1980, pp. 75}76. [57] M. Nasipuri, D.K. Basu, R. Dattagupta, M. Kundu, S. Banerjee, Microprocessor based real-time monitoring of multiple ECG signals, in Proceedings of Eleventh Annual Symposium on Computer Applications in Medical Care (SCAMC), Washington D.C., U.S.A., 1}4 Nov. 1987, pp. 822}827. [58] M. Nasipuri, M. Kundu, D.K. Basu, Microprocessor based simultaneous processing of multiple holter tapes, in: Proceedings IEEE Region Ten Conference (TENCON '89), Bombay, 22}24 Nov. 1989, pp. 500}502. [59] M. Kundu et al., A parallel architecture for real time recognition of ECG waveforms, J. Inst. Eng. (India) 73 (Part ET4) (1993) 158}161. [60] N.V. Thakor, Y.S. Zhu, K.V. Pan, Ventricular tachycardia and "brillation detection by a sequential hypothesis testing algorithm, IEEE Trans. Biomed. Eng. 37 (9) (1990) 837}843. [61] P. Trahanias, E. Skordalakis, Syntatic pattern recognition of the ECG, IEEE Trans. Pattern Anal. Mach. Intell. PAMI-12 (7) (1990) 648}657. [62] Z. Yi-Sherg, N.V. Thakor, Ventricular tachycardia and "brillation detection by a sequential hypothesis testing algorithm, IEEE Trans. Biomed. Eng. 37 (9) (1990) 837}843. [63] P.M. Shah et al., Automatic real-time arrhythmia monitoring in the intensive coronary care unit, Am. J. Cardiol. 39 (4) (1977).
371
[64] G. Stockman, L. Kanal, M.C. Kyle, Structural pattern recognition of carotid pulse waves using a general waveform parsing system, CACM 19 (12) (1976) 688}695. [65] F.L. Xiong, B.A. Lambird, L.N. Kanal, An experiment in recognition of electrocardiogram using a structural analysis algorithm, in Proceedings of IEEE International Conference on Systems, Man and Cybernetics, Bombay and New Delhi, Dec. 1983 } Jan. 1984, pp. 382}387. [66] T. Shibahara et al., CAA: a knowledge based system using causal knowledge to diagnose cardiac rhythm disorders, Tech. Reps. on Research at the Laboratory for Computational Medicine at the University of Toronto, LCMTR84-1, May 1984. [67] J.K. Tsotsos, T. Shibahara, Knowledge organization and its role in temporal and causal signal understanding: the ALVEN and CAA projects, in: N. Ccercone, G. McCalla (Eds.), The Knowledge Frontier, Springer, Berlin, 1987. [68] J. Mylopoulos, T. Shibahara, J.K. Tsotsos, Building knowledge-based systems: the PSN experience, IEEE Comput. 16 (10) (1983) 83}88. [69] M. Kundu et al., A reasoning system for on-line interpretation of ECG signal, accepted in IEEE Region Ten Conference (TENCON '93), Beijing, 19}21 October 1993. [70] M. Kundu, M. Nasipuri, D.K. Basu, A rule-based expert system for real-time interpretation of ECG patterns, in: Proceedings of International Seminar held on occasion of Ninth Indian Engineering Congress at Calcutta, Vol. 2, 15}20 Dec. 1994, pp. 827}838. [71] M. Kundu, Studies on some AI techniques for interpretation of ECG waveforms, Doctoral Dissertation, Computer Sc. & Engg. Dept., Jadavpur University, India, 1994. [72] C.A. Steinberg, S. Abraham, C.A. Caceres, Pattern recognition in the clinical electrocardiogram, IRE Trans. Biomed. Electron. BME-9 (1962) 23}30. [73] G.C. Stockman, L.N. Kanal, Problem reduction representation for the lingusitic analysis of waveforms, IEEE Trans. Pattern Anal. Mach. Intell. PAMI-5 (3) (1983) 287}298. [74] L.A. Zadeh, Fuzzy logic, IEEE Comput. (1988) 83}89. [75] M. Kundu, M. Nasipuri, D.K. Basu, A knowledge based approach to ECG interpretation using fuzzy logic, IEEE Trans. Systems Man Cybernet}Part B 28 (2) (1998) 237}243. [76] M. Kundu et al., A fuzzy expert system for interpretation of arrhythmia in ECG signal, in: Proceedings of Korea/Japan Joint Conference on Expert Systems with the Third World Congress on Expert Systems and others, Seoul, Korea, 2}5 Feb. 1993, pp. 567}579. [77] M. Kundu et al., An approach for reasoning with uncertainties in interpretation of bio-electrical signals, in: Proceedings of Ninth International Congress of Cybernetics and Systems, World Organization of Systems and Cybernetics, New Delhi, India, 18}23 Jan. 1993, pp. 372}376. [78] M. Kundu et al., An expert system for qualitative decision making in interpretation of ECG signals, XVIII Annual Convention and Exhibition (ACE '92) organized by IEEE Calcutta Section and BITM Calcutta, 21}23 Nov. 1992, pp. 204}207.
372
M. Kundu et al. / Pattern Recognition 33 (2000) 351}373
[79] A.K. Jain, J. Mao, K.M. Mohiuddin, Arti"cial neural networks: a tutorial, IEEE Comput. (1996) 31}44. [80] Suzuki, Self-organizing QRS wave recognition in ECG using neural networks, IEEE Trans. Neural Networks 6 (6) (1995) 1469}1477. [81] K. Paul, A connectionist approach to ECG interpretation, M.C.S.E. Thesis, Comp. Sc. & Eng. Dept., Jadavpur University, Calcutta, India, 1997. [82] N.J. Nilson, Principles of Arti"cial Intelligence, Springer, Berlin, 1982, pp. 193}273. [83] Meng-Hiot Lim, Yoshiyasu Takefuji, Implementing fuzzy rule-based systems on silicon chips, IEEE Expert 5 (1) (1990) 31}45. [84] M. Okajima, Computer Diagnosis of Electrocardiogram, Nakayama Shoten, Tokyo, 1989 (in Japanese). [85] S.K. Pal, S. Mitra, Multi layer perceptron, fuzzy sets and classi"cations, IEEE Trans. Neural Networks 3 (5) (1992) 683}697. [86] H.V. Pipberger et al., Preparation of electrocardiographic data for analysis by digital electronic computer, Circulation 21 (1960) 413}418. [87] H.V. Pipberger et al., Automatic screening of normal and abnormal electrocardiograms by means of a digital electronic computer, Proc. Soc. Exp. Biol. Med. 106 (1961) 130}132. [88] L.D. Cady, M.A. Woodbury, L.J. Tick, M.M. Gertler, A method for electrocardiogram wave pattern estimation, Example left ventricular hypertrophy, Circ. Res. 9 (1961) 1078. [89] H.V. Pipberger, Use of computers in interpretation of electrocardiograms, Circ. Res. 11 (1962) 555. [90] L. Stark, M. Okajima, G.H. Whipple, Computer pattern recognition techniques: electrocardiographic diagnosis, CACM (1962) 527}531. [91] T.Y. Young, W.H. Huggins, Computer analysis of electrocardiograms using a linear regression techniques, IEEE Trans. Biomed. Eng. BME-11 (1964) 60}67. [92] M.D. Feezor, A.G. Wallace, R.W. Stacy, A real time uniform analyser for detection of ventricular premature beats, J. Appl. Physiol. 29 (4) (1969) 541}545. [93] C.L. Feldman, M. Hubelbank, P.G. Amazeen, Real-time cardiac arrhythmia monitoring with the PDP-7, DECUS Proceedings, 1969, pp. 345}353.
[94] L.J. Haywood, V.K. Murthy, G.A. Harvey, S. Saltzberg, On-line real-time computer algorithm for monitoring the ECG waveform, Comput. Biomed. Res. 3 (1970) 15}25. [95] J.R. Cox, Jr., H.A. Fozzard, F.M. Nolle, G.C. Oliver, Some data transformations useful in electrocardiography, Computers in Biomedical Research, Vol. 3, Academic Press, New York, 1969, pp. 181}206. [96] Y. Nomura, Y. Takaki, S. Toyama, Automatic measurement of the electrocardiogram by digital computer, Jpn. Circ. J. 30 (1966) 21}28. [97] E. Kimura, Automatic interpretation of electrocardiogram by electronic computer, Jpn. J. Med. 8 (1969) 182}184. [98] F.W. Stallmann, H.V. Pipberger, Automatic recognition of electrocardiographic waves by digital computer, Circ. Res. IX (1961) 1138}1143. [99] L. Stark, J.F. Dickson, G.H. Whipple, H. Horibe, Remote real-time diagnosis of clinical electrocardiograms by a digital computer system, Ann. N.Y. Acad. Sci. 126 (1967) 851}872. [100] J. Wartak, J.A. Milliken, J. Karchmar, Computer program for pattern recognition of electrocardiograms, Comput. Biomed. Res. 4 (1970) 344}374. [101] D. Wortzman, B. Gilmore, H.D. Schwetman, J.I. Hirsch, A hybrid computer system for the measurement and interpretation of electrocardiograms, Ann. N.Y. Acad. Sci. 128 (1966) 876}899. [102] J. Wartak, Computer-aided recognition of electrocardiograms, Acta Cardiol. 22 (1967) 350}361. [103] S. Yasui et al., Computer diagnosis of electrocardiograms by means of a joint probability, Israel J. Med. Sci. 5 (1969) 913}916. [104] J.A. Milliken et al., Use of computers in the interpretation of electrocardiograms, Can. Med. Assoc. J. 101 (1969) 39}43. [105] J.M. Neilson, High speed analysis of ventricular arrhythmias from 24 hour recordings, in: Computers in Cardiology (IEEE Cat. No. 74CH0879-7C), 1974, pp. 55}59. [106] F.M. Nolle, G.C. Oliver, R.E. Kleiger, J.R. Cox, Jr., K.W. Clark, H.D. Ambos, The Argus/H system for rapid analysis of ventricular arrhythmias, in: Computers in Cardiology (IEEE Cat. No. 74CH0879-7C), 1974, pp. 37}42.
About the Author*MAHANTAPAS KUNDU received his B.E.E, M.E. Tel. E and Ph.D. (Eng.) degrees from the Jadavpur University (J.U.), Calcutta, India in 1983, 1985 and 1995, respectively. He worked as a Research Engineer in the Centre for Microprocessor Application for Training Education & Research, J.U., during 1985}88. He joined the J.U. as a faculty member in 1988. He is currently working as a Reader in the Computer Sc. and Eng Dept., J.U. He has co-authored a text book on computer fundamentals and a number of research papers in the areas of pattern recognition, image processing, multimedia databases, arti"cial intelligence and bio-medical signal processing.
About the Author*MITA NASIPURI received her B.E.Tel.E., M.E.Tel.E. and Ph.D. (Eng.) degrees from Jadavpur University, Calcutta, India in 1979, 1981 and 1990, respectively. She is currently a Professor in the Computer Science and Engineering Department of Jadavpur University. Her current research interest includes computer architecture, image processing, multimedia systems, bio-medical signal processing etc. She has about 30 research publications in International/National Journals and International/National conferences. She is Senior Member of the IEEE, U.S.A. and Fellow of The Institution of Engineers (India).
M. Kundu et al. / Pattern Recognition 33 (2000) 351}373
373
About the Author*DIPAK KUMAR BASU received his B.Tel.E, M.E.Tel.E. and Ph.D. (Eng.) degrees from Jadavpur University, Calcutta, India, in 1964, 1966 and 1969, respectively. He joined the Jadavpur University, Calcutta, India, in 1968, as a faculty member, in the Electronics and Tele-communication Engineering Department. He is currently a Professor in the Computer Science and Engineering Department of the same University. His "eld of interest includes digital electronics, microprocessor applications, bio-medical signal processing, knowledge based systems, image processing and multimedia systems. He has more than 50 research publications. He is a Fellow of the Institution of Engineers (India) and West Bengal Academy of Science and Technology and a Senior Member of the IEEE.
Pattern Recognition 33 (2000) 375}384
An e$cient syntactic approach to structural analysis of on-line handwritten mathematical expressions Kam-Fai Chan, Dit-Yan Yeung* Department of Computer Science, Hong Kong University of Science and Technology, Clear Water Bay, Kowloon, Hong Kong Received 29 September 1998; accepted 2 March 1999
Abstract Machine recognition of mathematical expressions is not trivial even when all the individual characters and symbols in an expression can be recognized correctly. In this paper, we propose to use de"nite clause grammar (DCG) as a formalism to de"ne a set of replacement rules for parsing mathematical expressions. With DCG, we are not only able to de"ne the replacement rules concisely, but their de"nitions are also in a readily executable form. However, a DCG parser is potentially ine$cient due to its frequent use of backtracking. Thus, we propose some methods here to increase the e$ciency of the parsing process. Experiments done on some commonly seen mathematical expressions show that our proposed methods can achieve quite satisfactory speedup, making mathematical expression recognition more feasible for real-world applications. ( 2000 Pattern Recognition Society. Published by Elsevier Science Ltd. All rights reserved. Keywords: De"nite clause grammar; Document processing; Mathematical expression recognition; Structural analysis
1. Introduction Many documents in scienti"c and engineering disciplines contain mathematical expressions. The input of mathematical expressions into computers is often more di$cult than the input of plain text, because mathematical expressions typically consist of special symbols and Greek letters in addition to English letters and digits. With such a large number of characters and symbols, the commonly used type of keyboard has to be specially modi"ed in order to accommodate all the keys needed, as done in Ref. [1]. Another method is to de"ne a set of keywords to represent special characters, as in LATEX [2]. However, working with specially designed keyboards or keywords requires intensive training. Alternatively, by taking advantage of pen-based computing technologies, one can simply write mathematical expressions on an electronic tablet for the computer to recognize them.
* Corresponding author. Tel.: #852-2358-6977; fax: #8522358-1477 E-mail address:
[email protected] (D-Y. Yeung)
Mathematical expression recognition consists of two major stages: symbol recognition and structural analysis. Character recognition, as the most common type of symbol recognition problems, has been an active research area for more than three decades [3]. Structural analysis of two-dimensional patterns also has a long history [4]. However, as emphasized in Refs. [5}7], very few papers have addressed speci"c problems related to mathematical expression recognition. In a mathematical expression, characters and symbols are typically arranged as a complex two-dimensional structure, possibly of di!erent character and symbol sizes. This makes the recognition process more complicated even when all the individual characters and symbols can be recognized correctly. Moreover, to ensure that a mathematical expression recognition system is useful in practice, its recognition speed is also an important factor to consider. It is well known that parsing can be done in polynomial time with Earley's algorithm [8] while most of the other types of parsers take exponential time. However, as Covington [9] tried to argue, exponential parsers can be fast when the length of the sentence to parse is short. Also, a long sentence can usually be broken up into shorter sentences that can be parsed separately.
0031-3203/00/$20.00 ( 2000 Pattern Recognition Society. Published by Elsevier Science Ltd. All rights reserved. PII: S 0 0 3 1 - 3 2 0 3 ( 9 9 ) 0 0 0 6 7 - 9
376
K-F. Chan, D-Y. Yeung / Pattern Recognition 33 (2000) 375}384
In this paper, we will mainly focus on the structural analysis aspect of mathematical expression recognition. First of all, we will review some related work. Then, we will discuss some problems that have to be overcome during the structural analysis stage. Afterwards, we will propose to use de"nite clause grammar (DCG) as a formalism to de"ne a set of replacement rules for parsing mathematical expressions. Unlike some parsers which may take quite a long time to construct even when all the grammar rules are available, DCG rules are already in a readily executable form. However, a DCG parser is potentially ine$cient due to its frequent use of backtracking. Thus, we will propose some methods for increasing the e$ciency of the parsing process. In addition, we will explain how our proposed approach works through use of an illustrative example. Finally, we will present and discuss some experimental results which are then followed by some concluding remarks.
2. Related work One of the earliest papers on mathematical expression recognition was presented by Anderson [10] in 1968. He used a purely top-down approach for parsing mathematical expressions. The algorithm starts with one ultimate syntactic goal and tries to partition the problem (i.e. goal) into sub-goals, until either all sub-goals have been satis"ed or all possibilities have been exhausted in vain. The algorithm is syntax-directed since it is guided by some grammar rules. However, experiments showed that the algorithm is not very e$cient due to the partitioning strategy used for the rules which involve two nonterminal symbols on the right-hand side. As a result, up to n!1 partitions can be generated by a set of n characters, and each of these partitions may further generate more partitions. In 1970, Chang [4] proposed a method for the structural analysis of two-dimensional mathematical expressions. The algorithm mainly makes use of the ideas of operator precedence and operator dominance. It consists of two major steps, grouping operator sequences and building a structure tree. E$ciency was taken into consideration in the proposed algorithm. However, the methods described are quite tedious. It is not straight forward to understand how they actually work in practical examples. In 1971, Martin [11] discussed some issues relating to both computer input and output of mathematical expressions. For the input case, however, not enough technical details about the replacement rules used were provided in the paper, but it raised the question of ambiguities found in mathematical expressions though with no solutions provided. In addition, it also proposed some methods to make the parsing process more e$cient, but again without real implementation.
Some papers related to this topic only dealt with some speci"c parts of the recognition process. For example, Wang and Faure [12] applied a statistical approach for determining some relationships among symbols in mathematical expressions, such as on the same line, exponent and subscript. Pfei!er [13] designed a parser for contextfree languages in order to parse two-dimensional structures like mathematical expressions. However, all the discussions in that paper are limited to parsing in a theoretical sense with no real examples shown. Grbavec and Blostein [14] used a graph rewriting approach for the understanding of mathematical expressions. Their system made use of knowledge about notational conventions to avoid the need for backtracking. Other papers in the 1980s and 1990s investigated both the character recognition and structural analysis stages with emphasis on some speci"c themes. BelaH id and Haton [5] worked on some simple mathematical expressions and elaborated more on solving the ambiguity problem by taking advantage of contextual information. Lee and Lee [7,15] proposed a method for recognizing symbols in mathematical expressions. Their aim was to translate the expressions from two-dimensional structures into one-dimensional character strings. Dimitriadis and Coronado [6], instead, put emphasis on the detection and correction of errors. Chou [16] proposed to use a two-dimensional stochastic context-free grammar for the recognition of printed mathematical expressions. His approach was designed for handling noise and random variations. In the grammar, each production rule has an associated probability. The main task of the process is to "nd the most probable parse tree for the input expression. The overall probability of a parse tree is computed by multiplying together the probabilities for all production rules used in a successful parse. As a result, the process is computationally quite expensive. Okamoto and Miao [17] took advantage of some speci"c knowledge of notational conventions of mathematics. Their method can "nd the structures of expressions without the need for parsing. Twaakyondo and Okamoto [18] extended the work of Okamoto and Miao [17] and Okamoto and Miyazawa [19] by using two strategies, namely, top-down and bottom-up structure processing methods. Again, with their approach, structures can be obtained without parsing. On the other hand, Lee and Wang [20] built a symbol relation tree for an expression and used some heuristics to correct recognition errors. Like the previous two, this method also does not require parsing. Ha et al. [21] de"ned an expression tree as an abstraction of a mathematical expression. The construction of such an expression tree can be done through top-down ("nding all the primitive objects) and bottom-up (resolving spatial relationships among objects) processes.
K-F. Chan, D-Y. Yeung / Pattern Recognition 33 (2000) 375}384
Recently, an approach based on hidden Markov models for character recognition was proposed [22]. The resulting mathematical expressions are recognized using a soft-decision approach [23]. Such an approach can ensure that alternative solutions are generated and explored under ambiguous cases.
3. Problems in structural analysis of mathematical expressions Mathematical expressions are two-dimensional structures. This nature and some other properties make their recognition non-trivial in many ways. Here are two examples: 1. The relationships among symbols in a mathematical expression sometimes depend on their relative positions. For example, in the expression `a2a, 2 is the superscript of a representing the square of a. However, in `a a, 2 is the subscript of a denoting only 2 a variable name. Although it is somewhat unusual, `a2a may be used to represent the multiplication of a by 2. 2. The same group of characters can have di!erent meanings under di!erent contexts. For example, `dxa has di!erent meanings in `:x2 dxa and in `cy#dxa. In the "rst expression, `dxa is part of the integral. However, in the second one, the same two letters become the multiplication of two variables. These problems have to be taken into consideration when we process mathematical expressions in the following steps. 3.1. Grouping symbols Before we can interpret the symbols, we must "rst group them properly into units. This can be done by using as heuristics some conventions in writing mathematical expressions. Some of these conventions are as follows: 1. Digits which together form a unit should be of the same size and be written on the same horizontal line. For example, 210 is only one unit but 210 consists of two units, i.e., 2 and 10. 2. Some letters together may form a unit, like some trigonometric functions such as tan, sin and cos. Before considering a group of letters as a concatenation of variables, we have to "rst check whether they are in fact some prede"ned function names. 3. Symbols other than letters and digits should be considered as separate units.
377
3.2. Determining relationships among symbols Determining the relationships among symbols, to some extent, can be viewed as grouping several smaller units into one larger unit. Again, some conventions can be used as heuristics: 1. Some fence symbols, such as parentheses, group the enclosed units into one single unit. For example, (a#b) is a unit which holds the sum of a and b. 2. Some binding symbols, like fraction line, J and +, dominate their neighboring expressions. For example, in +10 i, three units, i.e., 10, i"1, and i are bound to i/1 the symbol + which together give meaning to the expression as the sum of 1, 2,2, 10. 3. The ideas of operator precedence and operator dominance [4] can also be used for grouping units. For example, in a#b/c, the meaning becomes a#(b/c) due to the fact that `/a has higher precedence than `#a. The operator `#a is said to dominate `/a. However, in (a#b)/c, the meaning becomes (a#b)/c since `/a dominates `#a in this case. 4. Parsing with binding symbol preprocessing and hierarchical decomposition Most previous works in mathematical expression recognition did not put much emphasis on explaining how the replacement rules are used for structural analysis, or the explanations are too tedious and sometimes too ad hoc [4,10,13]. To remedy such weaknesses, we propose to use de"nite clause grammar (DCG) [24] as a formalism to concisely and precisely describe our set of replacement rules for parsing mathematical expressions. Note that a grammar expressed in DCG is highly declarative and can be directly executed by a Prolog interpreter. However, DCG parsers are known to be potentially ine$cient due to backtracking. In this section, we will propose some methods for increasing the e$ciency of the parsing process. 4.1. Basic notations for DCG DCG is similar to BNF, with some minor notational di!erences summarized as follows: 1. `::"a is replaced by `- -'a. 2. Non-terminals are not put inside brackets any more. Instead, terminals are now in square brackets. 3. Symbols are separated by commas and each rule is terminated by a full stop. There are some major di!erences between DCG and BNF though. In DCG, some Prolog predicates (enclosed inside M N) can be put in the body of any rule so that the semantics of a rule can be incorporated into its syntax. In
378
K-F. Chan, D-Y. Yeung / Pattern Recognition 33 (2000) 375}384
addition, arguments can be added to non-terminal symbols of the grammars. 4.2. Conventional backtracking parsing in DCG The simplest way of parsing a two-dimensional expression is to "rst translate it into its equivalent one-dimensional representation and then parse it with an existing parser. Since there already exist many compilers or interpreters for parsing string-based mathematical expressions, some extra work can be saved by taking this approach. Fig. 1 shows an example of such translation. Now, suppose that the parser we are going to use is a DCG parser and we need to create it from scratch. How many rules do we need? In general, the simplest expressions are the ones that involve arithmetic operations. As we know, all the binary arithmetic operators are left-associative. However, topdown parsers, such as a DCG parser, cannot handle left-recursive grammars. This problem can be solved easily by transforming those left-recursive grammars to right-recursive ones. However, although the strings generated by any left-recursive grammar and its corresponding right-recursive grammar can be the same, their internal structures may be di!erent. Hence, some "xing e!orts may be required subsequently. Anyhow, the grammar rules for arithmetic operations are extremely simple. They are as follows:
tiple levels in the grammar rules. In general, the operators at a level always have higher precedence than the ones above them. Similar techniques can also be applied to the unary operator, as well as spatial operators like implicit multiplication, subscr ipt, exponent and parentheses. Here are the grammar rules: factor([neg, A]) - -' [-], sub}expr(A). factor(A) - -' sub}expr(A). sub}expr(MEName) - -' sub}term(MEName). sub}term(MEName) - -' sub}factor(A), sub}expr(B), Mis}adjacent(A, B, MEName)N sub}term(A) - -' sub}factor(A). sub}factor(MEName) - -' expr}unit(A), sub}expr(B), Mis}sub}exp(A, B, MEName)N. sub}factor(A) - -' expr}unit(A). expr}unit(MEName) - -' [’(’], expr(S1), [’)’], Madd}expr}unit(S1, MEName)N. expr}unit(A) - -' [A], Mis}expr(A)N.
In order to handle functions, inde"nite integral, fraction and square root, the following rules are needed:
parse}equation(A) - -' equation(A). equation([", A, B]) - -'expr(A), ["], expr(B). expr([Op, A, B]) - -' term(A), [Op], Mis}add}sub(Op)N, expr(B). expr(A) - -' term(A). term([Op, A, B]) - -' factor(A), [Op], Mis}mul}div(Op)N, term(B). term(A) - -' factor(A).
Note that multiplication and division have higher precedence than addition and subtraction. Such precedence relationships can be implemented easily by having mul-
Fig. 1. Translating an expression from its two-dimensional form into a one-dimensional representation.
Notice that it usually takes comparatively longer time for a DCG parser with the above grammar rules to return the tree structure of an expression, because some sub-structures may be re-generated again and again during the backtracking steps. Therefore, the bigger the
K-F. Chan, D-Y. Yeung / Pattern Recognition 33 (2000) 375}384
379
structure is, the longer the time it takes. Fig. 2 depicts the tree structure for the expression shown before in Fig. 1. 4.3. Parsing with left-factored rules Although the grammar rules in the previous section are highly comprehensible, they are not very e$cient from the implementation point of view. For example, in the following two grammar rules,
we must "rst "nd term(A). If the next symbol Op is neither an addition operator nor a subtraction operator, we then backtrack to the second rule. However, in the second rule, the same step of "nding term(A) is repeated again. To tackle this problem, we can perform left factoring on the same rule to give the following result:
In the above left-factored grammar rule, the result of term(A) is passed into the next sub-goal more}term(A, B). If the respective operator is found, we then continue to process more terms. Otherwise, the input structure is returned as output. The main idea of left factoring is to rewrite some grammar rules so that decisions can be deferred until enough input tokens have been seen in order to make the right choice [25]. The following is the set of grammar rules corresponding to the rule set in the previous section, with some of the rules replaced by left-factored ones as shown below:
4.4. Parsing with binding symbol preprocessing As mentioned in Section 2, binding symbols always dominate their neighbors. For example, in the expression shown in Fig. 1, the fraction line in (6x#4y)/2 dominates the sub-expressions 6x#4y and 2. Instead of putting them in a one-dimensional form for further parsing, we can directly parse the two expressions "rst and then construct the "nal structure of the fraction from the intermediate results. The resulting structure will be stored in memory, with a name introduced to denote the fraction that the structure represents. There is no need to generate the structure for this fraction again during the subsequent processing. The resulting tree structures are shown in Fig. 3. As shown, the original tree structure is now partitioned into two sub-structures. This eliminates some repeated generation steps, and therefore can lead to signi"cant speedup. The grammar rules corresponding to binding symbol preprocessing are as follows:
4.5. Parsing with hierarchical decomposition
Fig. 2. Tree structure of the mathematical expression in Fig. 1.
The above idea can be extended to further partition the sub-structures into even smaller structures. Instead of
380
K-F. Chan, D-Y. Yeung / Pattern Recognition 33 (2000) 375}384
parsing the entire expression, we will parse all the subexpressions "rst and then parse the resulting expression. This idea is similar to hierarchical decomposition in AI planning [26]. Sub-expressions are detected using the following rules: 1. Parentheses have higher precedence than the other operators. Whatever enclosed inside a pair of parentheses should form an expression. 2. Some symbols in an expression, for example, : and dx in an inde"nite integral expression, enclose a sub-expression in between. With these, we can perform some preprocessing steps for "nding sub-expressions. Each sub-expression is then parsed separately. Afterwards, we can compose the "nal tree structure from a set of sub-structures. Here is the list of relevant DCG rules for parsing with hierarchical decomposition:
As mentioned before, although the strings generated can be the same, the internal structures may be di!erent if we rewrite some left-associative grammar rules into right-associative ones. Hence, we need a procedure for "xing the resulting structure to re#ect the correct associativity between operators and their operands. The following is such procedure written in Prolog, which is self-explanatory:
5. Experimental results and discussions
Fig. 3. Tree structures generated as a result of parsing with binding symbol preprocessing.
In this experiment, we perform tests on a number of di!erent expressions which were extracted from Ref. [27]. Expressions are grouped into four domains, namely, elementary algebra, trigonometric functions, geometry and inde"nite integrals. In each domain, there are three sizes of expressions, i.e, small, medium and large. Each size consists of "ve di!erent expressions. Totally, there are 60 expressions. Initially, the input is simply a sequence of points. After some segmentation steps, we then use the character recognition method proposed in Ref. [28]. Due to the high accuracy achieved by the method and the fact that those 0 expressions are neatly written, all the characters and symbols in the expressions can be recognized without errors. The recognized characters and symbols are then converted to objects with associated attributes, including location, size, and identity. Note that the objects can be put in an arbitrary order for our subsequent processing. The next step is to group the objects. Here we use a method similar to the one used in Ref. [19]. Afterwards, we perform parsing using di!erent techniques as described above and then compare their e$ciency.
K-F. Chan, D-Y. Yeung / Pattern Recognition 33 (2000) 375}384
Note that time may not be a very good measure of e$ciency since it may di!er from machines to machines. Hence, instead we use the number of logical inferences as a machine-independent performance measure. Table 1 shows the di!erences between conventional backtracking parsers with and without the use of left factoring. The result shows that the set of grammar rules we used plays an important role in terms of e$ciency. Parsing with rules which are not left-factored gives us an exponential running time with respect to the size of the expressions. However, with the left-factored version, the time taken is greatly reduced since all the intermediate results are fully utilized and there is much less repetitive construction of intermediate structures. For binding symbol preprocessing, saving is possible only when such symbols appear in the expressions.
381
Table 2 shows the di!erences between hierarchical decomposition parsing that uses left factoring only and that uses binding and fence symbol preprocessing as well. Our results show that speedup can be achieved for those expressions that contain some binding and fence symbols. In order to show the potential for practical use with hierarchical decomposition parsing, we also tabulate the time taken for parsing di!erent sizes of expressions in di!erent domains. Our recognition system implemented in Prolog runs on a Sun SPARC 10 workstation. The timer starts when the list of objects is passed to the parsing procedure and ends when the "nal structure is returned. Table 3 summarizes the result. Notice that the time required for recognizing the structures of some mathematical expressions of typical sizes
Table 1 Di!erences between conventional backtracking parsers with and without the use of left factoring Expressions
Number of logical inferences required for conventional backtracking parsing
Elementary algebra
Without left factoring
y"x#b/4a a2!b2"(a!b)(a#b) 16ab2c#256a3e!3b4!64a2bd r" 256a4 Trigonometric functions 1 cos a" sec a 1!cos 2a tan2 a" 1#cos 2a tan
(B!C) b!c (B#C) " tan 2 b#c 2
Geometry r"Jx2#y2 (x!a)y2"!x2(x#a)
S
p"
(ac#bd)(ab#cd) (ad#bc)
Inde"nite integrals
With left factoring
89 014 258 420
1435 2979
19 161 397
14 158
Without left factoring
With left factoring
56 307
1422
215 795
3459
5 356 302
5390
Without left factoring
With left factoring
164 576 411 192
1646 3745
374 511 627
7144
Without left factoring
With left factoring
:ex dx"ex
108 026
2682
P P
183 078
4280
52 307 610
10 314
u bx k dx" # log v v d d2 Ja2!x2 J(a2!x2)3 dx"! x4 3a2x3
382
K-F. Chan, D-Y. Yeung / Pattern Recognition 33 (2000) 375}384
Table 2 Di!erences between hierarchical decomposition parsing with and without the use of binding and fence symbol preprocessing Number of logical inferences required for hierarchical decomposition parsing Expressions
With left factoring only
A
B
f u y2$Ju!p y! # "0 2(u!p) 2 2 sin A" Js(s!a)(s!b)(s!c) bc
SA
2ab 1 cos C" a#b 2
P P
B
c2 ab 1! (a#b)2
1 xe~x2 dx"! e~x2 2
1 xJ(a2!x2)3 d x"! J(a2!x2)5 5
With left factoring, binding and fence symbol preprocessing
6242
3772
30 271
5173
11 905
6317
16 000
4307
10 084
4919
Table 3 Time required for recognizing the structures of di!erent expressions with hierarchical decomposition parsing Time required for hierarchical decomposition parsing (in seconds) Small size
Median size
Large size
Expression domain
Min.
Median
Max.
Min.
Median
Max.
Min.
Median
Max.
Elementary algebra Trigonometric functions Geometry Inde"nite integrals
0.02 0.02 0.02 0.02
0.03 0.02 0.03 0.05
0.05 0.05 0.05 0.05
0.05 0.05 0.05 0.07
0.07 0.07 0.08 0.08
0.08 0.07 0.10 0.08
0.08 0.08 0.10 0.10
0.15 0.10 0.12 0.15
0.25 0.15 0.15 0.17
ranges from 0.02 to 0.25 s. Nevertheless, the parser used is relatively simple. In fact, the whole parser has been listed in the previous section.
6. Conclusion Pen-based computing o!ers us a natural human-computer interface, such as an on-line mathematical expression editor. Such an editor, however, cannot be put into practical use without a sophisticated mathematical expression recognition subsystem. In this paper, we have proposed and demonstrated some methods for de"ning replacement rules in a clear and concise manner for parsing mathematical expressions. More importantly, it manages to o!er the much needed speed for practical use. In addition, the replacement rules are already in their executable form so that no exact programming is needed for implementing the rules.
Since our methods do not make use of stroke order information, they may also be used for o!-line mathematical expression recognition. However, some problems in mathematical expression recognition have not been addressed in this paper, including ambiguity resolution, error detection, and error correction. With a clear and concise formalism in the parsing phase, these issues will be relatively easy to tackle, using, for example, some error-correcting parsing techniques. Detail investigation of these issues will be provided in a separate paper.
7. Summary In a mathematical expression, characters and symbols are typically arranged as a complex two-dimensional structure, possibly of di!erent character and symbol sizes. This makes the recognition process more complicated even when all the individual characters and symbols can be recognized correctly. Moreover, to ensure that
K-F. Chan, D-Y. Yeung / Pattern Recognition 33 (2000) 375}384
a mathematical expression recognition system is useful in practice, its recognition speed is also an important factor to consider. In this paper, we propose to use de"nite clause grammar (DCG) as a formalism to de"ne a set of replacement rules for parsing mathematical expressions. With DCG, we are not only able to de"ne the replacement rules concisely, but their de"nitions are also in a readily executable form. However, a DCG parser is potentially ine$cient due to its frequent use of backtracking. Thus we propose some methods here to increase the e$ciency of the parsing process. Some experiments are done on 60 commonly seen mathematical expressions that are in four domains, namely, elementary algebra, trigonometric functions, geometry and inde"nite integrals. The results show that the set of grammar rules we used plays an important role in terms of e$ciency. Parsing with rules which are not left-factored gives us an exponential running time with respect to the size of the expressions. However, with the left-factored version, the time taken is greatly reduced since all the intermediate results are fully utilized and there is much less repetitive construction of intermediate structures. In addition, we also show that our proposed methods can achieve quite satisfactory speedup, making mathematical expression recognition more feasible for real-world applications. Since our methods do not make use of stroke order information, they may also be used for o!-line mathematical expression recognition. However, some problems in mathematical expression recognition have not been addressed in this paper, including ambiguity resolution, error detection, and error correction. With a clear and concise formalism in the parsing phase, these issues will be relatively easy to tackle and will be addressed in our future research.
Acknowledgements This research work is supported in part by the Hong Kong Research Grants Council (RGC) under Competitive Earmarked Research Grants HKUST 746/96E and HKUST 6081/97E awarded to the second author.
References [1] F. Grossman, R.J. Klerer, M. Klerer, A language for highlevel programming of mathematical applications, in Proceedings of the International Conference on Computer Languages, Miami Beach, FL, 1988, pp. 31}40. [2] L. Lamport, Latex } A Document Preparation System } User's Guide and Reference Manual, Addison-Wesley, Reading, MA, 1985.
383
[3] C.C. Tappert, C.Y. Suen, T. Wakahara, The state of the art in on-line handwriting recognition, IEEE Trans. Pattern Anal. Mach. Intell. 12 (8) (1990) 787}808. [4] S.K. Chang, A method for the structural analysis of 2-D mathematical expressions, Information Sciences 2 (3) (1970) 253}272. [5] A. BelaH id, J.-P. Haton, A syntactic approach for handwritten mathematical formula recognition, IEEE Trans. Pattern Anal. Mach. Intell. 6 (1) (1984) 105}111. [6] Y.A. Dimitriadis, J.L. Coronado, Towards an ART based mathematical editor, that uses on-line handwritten symbol recognition, Pattern Recognition 28 (6) (1995) 807}822. [7] H.-J. Lee, M.-C. Lee, Understanding mathematical expressions using procedure-oriented transformation, Pattern Recognition 27 (3) (1994) 447}457. [8] J. Earley, An e$cient context-free parsing algorithm, Comm. ACM 13 (1970) 94}102. [9] M.A. Covington, Natural Language Processing for Prolog Programmers, Prentice-Hall, Englewood Cli!s, NJ, 1994. [10] R.H. Anderson, Syntax-directed recognition of handprinted 2-D mathematics, in: M. Klerer, J. Reinfelds (Eds.), Interactive Systems for Experimental Applied Mathematics, Academic Press, New York, 1968, pp. 436}459. [11] W.A. Martin, Computer input/output of mathematical expressions, in Proceedings of the Second Symposium on Symbolic Algebraic Manipulation, Los Angeles, CA, 1971, pp. 78}89. [12] Z.X. Wang, C. Faure, Structural analysis of handwritten mathematical expressions, in Proceedings of the 9th International Conference on Pattern Recognition, Rome, Italy, 1988, pp. 32}34. [13] J.J. Pfei!er, Jr., Parsing graphs representing two dimensional "gures, in Proceedings of the IEEE Workshop on Visual Languages, Seattle, WA, pp. 200}206 (1992). [14] A. Grbavec, D. Blostein, Mathematics recognition using graph rewriting, in Proceedings of the Third International Conference on Document Analysis and Recognition, Montreal, Canada, 1995, pp. 417}421. [15] H.-J. Lee, M.-C. Lee, Understanding mathematical expressions in a printed document, in Proceedings of the Second International Conference on Document Analysis and Recognition, Tsukuba Science City, Japan, 1993, pp. 502}505. [16] P.A. Chou, Recognition of equations using a two-dimensional stochastic context-free grammar, in Proceedings of the SPIE Visual Communications and Image Processing IV, Philadelphia, PA, vol. 1199, 1989, pp. 852}863. [17] M. Okamoto, B. Miao, Recognition of mathematical expressions by using the layout structures of symbols, in Proceedings of the First International Conference on Document Analysis and Recognition, Saint-Malo, France, 1991, pp. 242}250. [18] H.M. Twaakyondo, M. Okamoto, Structure analysis and recognition of mathematical expressions, in Proceedings of the Third International Conference on Document Analysis and Recognition, Montreal, Canada, 1995, pp. 430}437. [19] M. Okamoto, A. Miyazawa, An experimental implementation of a document recognition system for papers containing mathematical expressions, in: H.S. Baird, H. Bunke, K. Yamamoto (Eds.), Structured Document Image Analysis, Springer, Berlin, 1992, pp. 36}53.
384
K-F. Chan, D-Y. Yeung / Pattern Recognition 33 (2000) 375}384
[20] H.-J. Lee, J.-S. Wang, Design of a mathematical expression recognition system, in Proceedings of the Third International Conference on Document Analysis and Recognition, Montreal, Canada, 1995, pp. 1084}1087. [21] J. Ha, R.M. Haralick, I.T. Phillips, Understanding mathematical expressions from document images, in Proceedings of the Third International Conference on Document Analysis and Recognition, Montreal, Canada, 1995, pp. 956}959. [22] M. Koschinski, H.-J. Winkler, M. Lang, Segmentation and recognition of symbols within handwritten mathematical expressions, in Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing, Detroit, MI, Vol. 4, 1995, pp. 2439}2442. [23] H.-J. Winkler, H. Fahrner, M. Lang, A soft-decision approach for structural analysis of handwritten mathematical expressions, in Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing, Detroit, MI, Vol. 4, 1995, 2459}2462.
[24] F. Pereira, D. Warren, De"nite clause grammars for language analysis } a survey of the formalism and comparison with augmented transition networks, Artif. Intell. 13 (1980) 231}278. [25] A.V. Aho, R. Sethi, J.D. Ullman, Compilers: Principles, Techniques, and Tools, Addison-Wesley, Reading, MA, 1986. [26] S.J. Russell, P. Norvig (Eds.), Arti"cial Intelligence: A Modern Approach, Prentice-Hall, Englewood Cli!s, NJ, 1995. [27] D. Zwillinger (Ed.), CRC Standard Mathematical Tables and Formulae, 30th ed., CRC Press, Boca Raton, 1996. [28] K.F. Chan, D.Y. Yeung, Elastic structural matching for on-line handwritten alphanumeric character recognition, in Proceedings of the 14th International Conference on Pattern Recognition, Brisbane, Australia, 1998, pp. 1508}1511.
About the Author*KAM-FAI CHAN received his B.Sc. degree from Radford University, M.Sc. degree from the University of South Carolina, and Ph.D degree from the Hong Kong University of Science and Technology, all in computer science. He is currently a postdoctoral research associate in the Department of Computer Science at the Hong Kong University of Science and Technology. His major research interests include pattern recognition, logic programming and Chinese computing. About the Author*DIT-YAN YEUNG received his B.Sc.(Eng.) degree in electrical engineering and M.Phil. degree in computer science from the University of Hong Kong, and his Ph.D. degree in computer science from the University of Southern California in Los Angeles. From 1989 to 1990, he was an assistant professor at the Illinois Institute of Technology in Chicago. He is currently an associate professor in the Department of Computer Science at the Hong Kong University of Science and Technology. His current research interests are in the theory and applications of pattern recognition, machine learning, and neural networks. He frequently serves as a paper reviewer for a number of international journals and conferences, including Pattern Recognition, Pattern Recognition Letters, IEEE Transactions on Pattern Analysis and Machine Intelligence, IEEE Transactions on Image Processing, and IEEE Transactions on Neural Networks.
Pattern Recognition 33 (2000) 385}398
Morphological waveform coding for writer identi"cation E.N. Zois, V. Anastassopoulos* Electronics Laboratory, Physics Department, University of Patras, Patras 26500, Greece Received 23 April 1997; received in revised form 25 August 1998; accepted 16 February 1999
Abstract Writer identi"cation is carried out using handwritten text. The feature vector is derived by means of morphologically processing the horizontal pro"les (projection functions) of the words. The projections are derived and processed in segments in order to increase the discrimination e$ciency of the feature vector. Extensive study of the statistical properties of the feature space is provided. Both Bayesian classi"ers and neural networks are employed to test the e$ciency of the proposed feature. The achieved identi"cation success using a long word exceeds 95%. ( 2000 Pattern Recognition Society. Published by Elsevier Science Ltd. All rights reserved. Keywords: Writer identi"cation; Person veri"cation; Morphological features; Waveform coding
1. Introduction Handwritten patterns constitute the behavioral part of biometrics approach towards person veri"cation which is not invasive in contrast to that of physiological biometrics ("ngerprints or iris characteristics). O!-line writer veri"cation systems based on signatures have been studied extensively in the past [1]. A writer veri"cation system based on handwritten text is expected to provide discrimination results equivalent to those obtained from signatures, since text has been reported to comprise rich and stable information [2]. Furthermore, a handwritten sentence can be determined and changed by the writer at will. In high security data systems like those involved in "nancial transactions, the "rst step towards reaching a speci"c person's data is usually carried out by means of the personal identi"cation number (PIN) number. However, handwritten patterns such as the signature or a word can be used on a complementary basis to improve system reliability. In order to increase further the reliabil-
* Corresponding author. Tel.: #30-61-996-147; fax: #3061-997-456. E-mail address:
[email protected] (V. Anastassopoulos)
ity of the veri"cation system, many handwritten words can be used by means of fusion techniques [3]. In general, writer discrimination and veri"cation approaches based on handwritten text are hardly found in the literature [4,5]. Security reasons or speci"c law restrictions have prevented serious results of signi"cant importance on the topic from publicity [6]. To the knowledge of the authors few publications are related to writer discrimination and especially to feature extraction [7,8]. Feature extraction from handwritten text can be carried out using approaches that resemble those of signature veri"cation. However, features which contain information of the trace of each word are usually preferable. In this work a writer identi"cation method is proposed, which is based on the use of a single word. The image of the word is properly preprocessed and projected onto the horizontal direction. Projection functions have been used widely in the literature for contour feature extraction [9,10], signature analysis [11,12] and recognition of handwritten characters (Latin, Chinese, etc.) and numerals [13,14]. A projection is a global shape descriptor which provides a kind of line image coding [10]. The obtained projections are segmented in parts which are morphologically [15,16] processed in order to obtain the required feature vector. The morphological processing is a type of granulometry, i.e. the measure of area reduction through successive openings [16,17]. Two di!erent types
0031-3203/00/$20.00 ( 2000 Pattern Recognition Society. Published by Elsevier Science Ltd. All rights reserved. PII: S 0 0 3 1 - 3 2 0 3 ( 9 9 ) 0 0 0 6 3 - 1
386
E.N. Zois, V. Anastassopoulos / Pattern Recognition 33 (2000) 385}398
of windows are applied on the segments of the projection functions to control the #ow of information from one part to the other. The blanks between the letters are also considered in the formation of the feature vector. Both the statistical properties of the feature space and the capability of the speci"c features for writer identi"cation are extensively studied. This study includes the underlying pdf for the feature vector components as well as the separability of the clusters in the feature space. For this purpose a cluster separability measure is proposed and analyzed. Next, two di!erent classi"cation schemes are tested. Namely, the Bayesian classi"er and the neural networks. In the classi"cation procedure, the binary decision problem (writer veri"cation: is this person he who claims to be?) and the general classi"cation problem (writer identi"cation: identify a writer among many others) are studied. A writer veri"cation error smaller than 5% is achieved. The error becomes smaller while increasing feature dimensionality. A database [18] was created employing 50 writers, while an English and the equivalent Greek word were used to demonstrate that the method is language independent. The paper is organized as follows. In Section 2 the deeveloped database is presented. In Section 3 the procedure used for feature extraction is analyzed. In Section 4 the formation of the feature space is explained and criteria for measuring cluster intra-distance and inter-distance are presented. An extensive study on the statistics of the feature space is also carried out. Section 5 deals with the experimental performance of two classi"cation schemes in the multicategory and the two-category case. The conclusions are drawn in Section 6.
2. The database and data preprocessing Data acquisition and preprocessing constitute an essential step towards feature extraction and writer discrimination. Speci"cally, the acquisition stage a!ects the quality of the image, which in turn determines the reliability of the feature vector and the recognition procedure. The o!-line procedures dealt in this work give full discretion to obtain good quality images. 2.1. The database The database employed for writer identi"cation was created so that two di!erent issues are appropriately addressed. Firstly, a considerable number of samples was recorded to ensure the validity of the experimental results. Secondly, the database was constructed using both an English and a Greek word in order to show that the applicability of the feature vector is independent of the language used. Accordingly, a blank page of size A4 is divided in 45 shells (15 lines]three columns). Each writer had to "ll in all the shells of the page with the word
Fig. 1. Scanned word samples from the database: (a) English; (b) Greek word.
&characteristic' and the shells of another page with the equivalent Greek word (Fig. 1). The only constraint was that the writer should write down the words inside the shells. Fig. 1 shows a word sample by a speci"c writer in a speci"c shell after the scanning process. For each shell an image "le with dimensions 230 by 70 pixels is created, with 256 gray levels. A total of 50 writers has been recorded in the database which is available to the research community through Internet [18].
2.2. Preprocessing The database is "rstly preprocessed so that the derived features are as far as possible independent of the writing conditions. Preprocessing is employed to eliminate redundant information caused by the randomness of scanning, the di!erence between the pens used by di!erent writers as well as the capability of the paper used to soak up ink. In all the above cases the use of the most appropriate image enhancement techniques depend on operators experience. The preprocessing algorithms applied in this work are image thresholding and curve thinning. Due to satisfactory image acquisition conditions, these two algorithms are regarded adequate to reveal the special characteristics of each writers handwriting. Firstly, thresholding was applied to both English and Greek words of the database. Histogram thresholding is used in order to separate gray (word) and white pixels. The threshold used was between 170 and 180 gray value for all images. A thresholded result (black and white image) is shown in Fig. 2a accompanied by its thinned version (Fig. 2b). The thinning process produces the trace of the thresholded images with only one pixel width. The algorithm realizes simple morphological transformations (openings) with four structuring elements of di!erent orientations applied successively on the image only once.
E.N. Zois, V. Anastassopoulos / Pattern Recognition 33 (2000) 385}398
Fig. 2. Preprocessing stage: (a) thresholded image; (b) thinned image.
387
Fig. 3. The projection function corresponding to the image of the word &characteristic'.
3. Feature extraction The feature extraction procedure is described in this section. The proposed feature vector is obtained by means of morphologically transforming the projection functions of the thinned images. The length of the projections is "rstly normalized. Afterwards, morphological openings are applied to the segments of the projection for feature extraction. 3.1. The normalized projection function The projection function is derived by mapping the two-dimensional thinned image to a one-dimensional function. This function contains information about the spatial pixel distribution of the word trace along the horizontal direction. More speci"cally, it is formed by measuring the black pixels contained in each column of the thinned image [9}14]. In Fig. 3 the thinned image as well as the corresponding projection function f (x) are shown. The zeros of the function which correspond to the blanks between the letters contain signi"cant information about the speci"c handwriting. Accordingly, two versions of the projection function are created as shown in Fig. 4. Both functions have been shifted to the origin, whereas in the second function (Fig. 4b) the zero bins are eliminated. Furthermore, the length of the function is not constant even for those samples written by the same writer. In order to make the feature independent of the word length the functions are resampled so that the total number of bins is 100, as shown in Fig. 5. Resampling to a constant length of 100 points incorporates antialising procedures and is carried out using a special MATLAB routine. Both functions are invariant under translation since they are shifted to the origin. Rotation invariance is assumed to exist since each person writes along the horizontal line. Hereafter, the normalized in length function containing the blanks will be addressed as the
Fig. 4. Raw projection functions: (a) function containing blanks between letters; (b) function without blanks.
Fig. 5. Final projection function resampled to 100 bins: (a) projection function (PF); (b) compressed projection function (CPF).
388
E.N. Zois, V. Anastassopoulos / Pattern Recognition 33 (2000) 385}398
&projection function (PF)' while the normalized in length function without blanks as &compressed projection function (CPF)'. The resulting projection functions are of too large dimensionality to be considered as features for discrimination. It is well known that large dimensionality is the curse of every pattern recognition technique. In this work, the information content of both the PF and CPF is further compressed by means of morphological transformations. 3.2. Morphological transformation and the feature vector Mathematical morphology [15] is based on set theory and deals with the interaction of one set called the structuring element (SE) with the set to be transformed. Information on the characteristics of a binary object can be obtained by various transformations with di!erent SE [16,17]. The two basic operations in morphology are erosion and dilation. Erosion (>) of a set X by a SE B is a shrinking operation de"ned as follows [15]: X>B"Y X , ~b b|B
Fig. 6. The morphological opening on the projection function: (a) initial projection function; (b) opening with line SE of length 3; (c) opening with line SE of length 7.
(1)
while dilation (=) is an expanding operation X=B"Z X , `b b|B
(2)
where the subscript denotes geometric translation. On the other hand, the morphological opening (") is de"ned as an erosion followed by a dilation with the same SE X"B"(X>B)=B.
(3)
After an opening operation the original object X has been smoothed from those details which the structuring element B does not "t in [17]. This fact can be used to measure the loss of information when gradually increasing the size of the structuring element. In case of onedimensional function, morphological transformations operate on the umbra [15] of the function f (x) regarding it as a set. Assuming that the SE g(x) is a line segment with length ¸ and zero valued in the domain it is de"ned, erosion and dilation are expressed respectively as ( f>g)(x)" min M f (z)!g(z!x)N" min ( f (z)) z|D z|D z~x|G z~x|G
(4)
and ( f=g)(x)" max M f (z)#g(z!x)N" max ( f (z)) z|D z|D z~x|G z~x|G
Fig. 7. Partition of the region of the projection function f (x) for feature vector extraction. Rectangular windows. The number of segments is 5.
(5)
where D is the domain of f (x) and G is the domain of g(x). The morphological opening f " g of the function f by the SE g is de"ned according to (3). The measurement of the gradual reduction in the area using openings is called granulometry or pattern spectrum [16,17] and is used, in this work, to give the feature vector for writer discrimination. The di!erences between successive openings denote the amount of information that is removed by the increasing in size structuring element. In Fig. 6 are shown graphically the results when successive openings are applied to the PF f (x) with line SE g (x) of length ¸"3,7, respectively. The "nal feature vector is created merely by partitioning the projections into a number of segments and measuring the relative amount of area that the two SE (with lengths 3 and 7) reject in each block. Fig. 7 shows the case of segmenting the region of f (x) into "ve subblocks, thus extracting a 10-dimensional feature vector.
E.N. Zois, V. Anastassopoulos / Pattern Recognition 33 (2000) 385}398
389
Generally, the components of the feature vector p are de"ned as follows
C
p" i
Mes( f )!Mes( f " g ) 3 Mes( f )
D
for i"1 ,2, m
(6)
i
and
C
p " i`m
Mes( f " g )!Mes( f " g ) 3 7 Mes( f )
D
for i"1 ,2, m(7) i where g and g denote the line SE with length ¸"3 and 3 7 7 respectively, Mes(.) is the area under the function in the argument, i stands for the ith segment of the region of the function f (x) and m is the partition cardinality. The feature vector p is similar to the pattern spectrum described in the literature [16,17]. Each component p of i the feature vector p describes the pixels allocation along the trace of the word in each segment. Actually, p dei scribes the "ne details of the trace while p describes i`m the coarse distribution of the pixels in each segment. Speci"cally, if the "rst m p 's are large enough they reveal i a tendency of the writer to persist in vertical lines. Whereas, if the rest p 's are quite large smoother distribution of i the lines into both directions is expected. The feature vector components corresponding to the same segment (i.e. p and p ) are somewhat correlated. i i`m This will be examined in the next section where covariance matrix properties are analyzed. Furthermore, some kind of correlation is expected between neighboring components p and p . This is due to the fact that i i`1 information near the edge of the segment is prone to moving towards the next segment owing to both handwriting variations and the resampling procedure. Partitioning the functions with overlapping trapezoidal segments a new feature vector q is derived which is more stable to information shift. Fig. 8 shows the way the above idea is implemented in case of partitioning into "ve segments. Speci"cally, for the evaluation of the area of a function (Mes(.)) in a speci"c segment, part of the area of the adjacent segment is considered multiplying by a trapezoidal instead of a rectangular window. The feature vectors p and q have been extensively tested as far as the achieved separability in the feature space is concerned.
4. Feature space statistics and properties The statistical characteristics of the derived four types of features are exploited in this section and conclusions are drawn about their classi"cation capabilities. The extent of the clusters into the feature space is examined by means of the eigenvalues of the cluster covariance matrices. Information on the correlation of the features can also be obtained from these covariance matrices. Next, the pdf of the features is exploited using the K}S "t test. The Gaussian pdf is found to be a good candidate for
Fig. 8. Partition of the region of the projection function f (x) for feature vector extraction. Five trapezoidal windows are used.
one of the types of the features employed. For the same feature, maximum cluster separability in the feature space is observed. A cluster separability measure is introduced and analyzed theoretically. This measure is used to assess writer separability. 4.1. Class covariance matrices The covariance matrix of a population is a means of measuring the variance of each component in the feature space as well as the correlation between the feature components. It also provides, in case we are confronted with high dimensionality data sets, a measure for the intrinsic dimensionality. Thus, diagonalization procedures provide the principal components of the orthogonal features which lie along the eigenvectors of the covariance matrix. The ability of the above procedure to make apparent the most dominant components, leads to reduction of the original feature space. Evaluating the covariances for every feature type and for a partition range of 5}10 segments, the maximum number of dominant eigenvalues is found to be smaller than eight. The small intrinsic dimensionality of the feature space results in the following. Firstly, the number of samples per writer is considered adequate for feature vector mean and covariance estimation with reduced bias, thus giving consistent error probabilities. Secondly, the use of distance as a similarity measure, which is a mapping to a one-dimensional space, causes small distortion to the classi"cation information since the intrinsic dimensionality of the original space is small. The eigenvalues of the covariance matrix corresponding to the "rst writer and for a speci"c partition level are given in Table 1. Additionally, working out the covariances of all data sets it was found that the eigenvectors corresponding to the minimum eigenvalues di!er from writer to writer. Consequently, there is not a common base in the feature space which could be used for simultaneously rotating all clusters in order to reduce the dimensionality of the feature space and decorrelate the features.
390
E.N. Zois, V. Anastassopoulos / Pattern Recognition 33 (2000) 385}398
Table 1 Eigenvalues of the covariance matrix for writer No.1. The features were extracted using "ve trapezoidal windows 0.0003
0.0005
0.0006
0.0018
0.0032
0.0036
0.0048
0.0069
0.0091
0.0134
Table 2 Correlation coe$cients for features from trapezoidal windows without blanks and compressed projection function 1.0000 0.1078 0.3882 0.3409 !0.3434 0.1286 !0.3043 0.0492 0.1060 0.2510
0.1078 1.0000 !0.0049 0.5087 !0.1226 0.0481 !0.3393 0.0912 !0.0493 !0.0959
0.3882 !0.0049 1.0000 0.1224 0.2385 0.1026 0.0099 !0.1267 !0.1192 0.0631
0.3409 0.5087 0.1224 1.0000 !0.0427 0.6019 !0.0695 0.2438 0.2664 !0.0951
!0.3434 !0.1226 0.2385 !0.0427 1.0000 !0.0825 0.7678 !0.2553 0.2318 !0.1416
A measure of feature correlation can be obtained by examining the non-diagonal elements of the covariance matrix. If we de"ne the covariance matrix of a data cluster C and its elements c as i ij c "EM(x !m)(x !m)tN (8) ij i j where x and m are the feature vector and the sample mean, respectively, for a speci"c writer, then the components r (correlation coe$cients) of the correlation ij matrix R emanate from the relation c ij . r " (9) ij Jc c ii jj The above correlation coe$cients were evaluated for the features extracted using rectangular and trapezoidal windows. Next, the same calculations were carried out for both the PF and the CPF, respectively. It is concluded that features are less correlated when rectangular windows are employed. This result was expected since using trapezoidal windows, neighboring feature components (e.g. p and p ) share a common amount of information. 3 4 However, even in the case of trapezoidal windows the correlation of the features seldom exceeds 0.3, a value which can be considered to represent weak correlation. Finally, correlation coe$cients remain almost the same when the features are coming from functions with or without blanks between letters. Tables 2 and 3 show the correlation coe$cients between rectangular and trapezoidal windows for one writer. 4.2. Statistical behavior of the proposed features The statistics of the feature components and their agreement with the normal density is examined by means of the statistical "t tests [20], and especially the Kol-
0.1286 0.0481 0.1026 0.6019 !0.0825 1.0000 0.0570 0.4989 0.1640 0.0408
!0.3043 !0.3393 0.0099 !0.0695 0.7678 0.0570 1.0000 !0.3006 0.3297 !0.0867
0.0492 0.0912 !0.1267 0.2438 !0.2553 0.4989 !0.3006 1.0000 0.0556 0.4705
0.1060 !0.0493 !0.1192 0.2664 0.2318 0.1640 0.3297 0.0556 1.0000 !0.2697
0.2510 !0.0959 0.0631 !0.0951 !0.1416 0.0408 !0.0867 0.4705 !0.2697 1.0000
mogorov}Smirnov (K}S) test [21]. Speci"cally, we have to disprove, to a certain required level of signi"cance, the null hypothesis H that a data set follows a predeter0 mined distribution function. Disproving the null hypothesis in e!ect we prove that the data set comes from a di!erent distribution. On the other hand, proving the null hypothesis shows that the data set is consistent with the considered distribution function. Other methods found in the literature for inspecting the form of a distribution are the Parzen windows and the K-nearest neighbors [22]. In order to examine the similarity between two cumulative functions we de"ne D as the maximum observed value of their absolute di!erence as shown in Fig. 9: D" max DS (x)!P(x)D N ~=:x:=
(10)
where S (x) is the cumulative function of the sample data N and P(x) is a known distribution function. Under certain conditions and given that hypothesis H is true, the 0 Kolmogorov}Smirnov statistic D follows the cumulative distribution [21] = F (D)"1!2 + (!1) j~1e~2j2j2, D j/1
(11)
where j"(JN#0.12#0.11/JN)D and N is the number of data samples used. We must reject H if D is larger 0 than a constant c. This constant is determined in terms of the signixcance level a [20]: a(c)"PMD'cDH N 0 "1!PMD(cDH N 0 "1!F (c) D
(12)
E.N. Zois, V. Anastassopoulos / Pattern Recognition 33 (2000) 385}398
391
Table 3 Correlation coe$cients for features emerging from rectangular segments without blanks and compressed projection function 1.0000 0.1178 0.1286 0.2756 !0.2234 0.2001 !0.3361 !0.0701 0.1524 0.2739
0.1178 1.0000 !0.1017 0.2624 !0.0314 0.0387 !0.3671 0.1033 0.0414 !0.1651
0.1286 !0.1017 1.0000 !0.0616 !0.0765 0.1204 !0.0593 !0.0772 !0.1302 0.0528
0.2756 0.2624 !0.0616 1.0000 0.0510 0.2293 !0.0198 0.0905 0.5343 !0.3166
!0.2234 !0.0314 !0.0765 0.0510 1.0000 !0.1891 0.4613 0.1166 0.3905 !0.1579
0.2001 0.0387 0.1204 0.2293 !0.1891 1.0000 0.1450 0.1656 0.0887 0.1128
!0.3361 !0.3671 !0.0593 !0.0198 0.4613 0.1450 1.0000 !0.3532 !0.0082 0.0637
!0.0701 0.1033 !0.0772 0.0905 0.1166 0.1656 !0.3532 1.0000 0.2982 0.0383
0.1524 0.0414 !0.1302 0.5343 0.3905 0.0887 !0.0082 0.2982 1.0000 !0.3551
0.2739 !0.1651 0.0528 !0.3166 !0.1579 0.1128 0.0637 0.0383 !0.3551 1.0000
each writer the normal density can be considered as good approximation to the data. It is worth mentioning that the best approximation to Gaussian statistics was achieved by the features derived using the compressed projection functions and trapezoidal windows in the feature extraction procedure. Similar experimental procedure was carried out for features obtained using rectangular windows and/or the simple projection function. Normal pdf hypothesis was not found strong enough in these cases. Hence, the trapezoidal windows are proved to be a natural process which strengthens the validity of Gaussian pdf for the derived features. 4.3. Cluster separability measure
Fig. 9. Kolmogorov}Smirnov statistic D.
From Eq. (12) we can determine c for a speci"c signi"cance level a. Accordingly, F (c)"1!a veri"es our D con"dence about the validity of H . In practice, for an 0 observed value D this con"dence is expressed as the 1 probability D can be of the smallest values of D, i.e. 1 P(D (D)"1!P(D(D )"1!F (D ) 1 1 D 1
(13)
In our experiment a data set of n"45 points (number of words) was available for each writer and each feature component for the K}S test to be applied. The value of D as well as the con"dence about H were evaluated for 0 500 data sets (50 writers]10 feature vectors) and for a low-level partition ("ve segments). Fig. 10a provides the histogram of the measured values of D, while in Fig. 10b the distribution of our con"dence about H is shown. 0 The majority of the D's is around 0.1 which corresponds to a degree of con"dence larger than 75%. Similar results were obtained using the Greek word. This supports our claim that for each individual feature component and for
A person's handwriting is not precisely repeatable since it changes with physical and mental state, as well as with the age [1]. Generally, we can distinguish between two kinds of handwriting variability. The intraclass variability which describes the variations within a class (same writer), and the interclass variability which describes the di!erences between writers. Ideally, intraclass variability should be as low as possible, while interclass variability should be as large as possible. In practice, classes are not well separated. A quality factor which indicates the separability between two classes is introduced here. This quality factor expresses the maximum theoretical error in classifying the samples of two clusters when these two clusters are normally distributed and intermixed across the line of their larger variances. Let us consider two normally distributed populations u and u with two dimensional pdf's p " a b a N(k , p , k , p ) and p "N(k , p , k , p ), respecax ax ay ay b bx bx by by tively, and the same a priori probabilities P(u )"P(u ). a b The above example can easily be generalized for higher dimensions where real problems are met. Fig. 11 shows the contour plot of two normal distributions having variances in x and y axes equal to unity, while their means are selected so that their Euclidean distance
392
E.N. Zois, V. Anastassopoulos / Pattern Recognition 33 (2000) 385}398
Fig. 10. (a) Histogram of D for 500 data sets (50 writers and a 10-dimensional feature vector) using the compressed projection function and trapezoidal windows. Partition level is 5 and the word used is the English one. (b) The corresponding histogram for the degree of con"dence (1!F (D)) about the hypothesis H for 500 data sets. D 0
Fig. 11. Contours corresponding to p "1, for two normal i distributions with equal variances and mean distance equal two.
Fig. 12. Decision boundary for the two class problem.
follows: equals two. Inside each individual circle lies 60% of the samples. We seek a quantity which would express the separation between these two populations. This separation is easier when the populations are quite distant and the dispersion of each one is small. In order to measure the distance between two populations, we use the Euclidean distance of their means, whereas the dispersion of each population is measured using the largest eigenvalue j of its covariance matrix. This eigenvalue imax is related to the length of the largest semiaxis of the cluster hyperellipsoid and the corresponding standard deviation Jj "p . Thus, we introduce the ratio imax imax R in order to express the separability of two classes as
D(m , m ) a b R(a, b)" , Max(Jj )#Max(Jj ) a b
(14)
where D(m , m ) is the Euclidean distance of the populaa b tion means and Max(j) is the maximum eigenvalue for each class. In the worst-case scenario (maximum classi"cation error), the eigenvectors of the covariance matrix which correspond to the greatest eigenvalues lie in the direction Mm6 !m6 N as shown in Fig. 12. The separability a b ratio R in this case is theoretically calculated as D(m , m ) m !m a b a R " " b T Max(Jj )#Max(Jj ) p #p a b a b
(15)
E.N. Zois, V. Anastassopoulos / Pattern Recognition 33 (2000) 385}398
393
and the maximum theoretical error when the above situation holds is taken from
(17) becomes after some mathematical manipulations [24]
P(error)"P(x, y3R Du )#P (x, y3R Du ) b a a b
P(error)"0.5!0.5erf
P
p(x, yDu ) P (u ) dx dy a a
"
Rb
P
p(x, yDu ) P (u ) dx dy b b Ra = = " p (x, y D u ) P (u ) dx dy a a a y/~= x/s = s # p (x, yDu ) P (u ) dx dy, (16) b b b y/~= x/~= where the a priori probabilities P(u ) and P(u ) are a b considered equal to 0.5 and the bivariate densities p and a p are decorrelated. This way p and p are separable with b a b respect to x and y so that Eq. (16) becomes #
P P P P C
1 1 P(error)" 2 J2pp
P P
=
A
B
(x!m )2 ax exp ! dx 2p2 ax ax x/s 1 (x!m )2 s bx # dx . exp ! 2p2 J2pp x/~= bx bx (17)
A
B D
This integral becomes minimum for p "p when ax bx s is placed in the middle of m !m [22]. In this bx ax case, s equals (m #m )/2 and using Eqs. (15) and ax bx
A B R
J2
.
(18)
Using simple MATLAB routines the theoretical maximum classi"cation error is found to be 16.5% for R"1 which is the case described in Fig. 11. For R'1 or in case that the clusters higher dispersion is not in the m6 !m6 direction, the success is expected higher. a b Experimentally, the proposed quality factor R was evaluated for each feature type, considering u as the 1 writer under examination and u as the set of all the 2 other writers (totally 49). The u class can be viewed as 2 noise which must be rejected from the genuine classes u . 1 A low-level partition was used ("ve segments) for the projection functions. The results are shown in Fig. 13. From this "gure it is obvious that the highest values of R (higher separability) are obtained when trapezoidal windows and compressed projection functions are employed (Fig. 13a). Therefore, the experimental classi"cation procedure in the next section is carried out using only the corresponding type of features.
5. Classi5cation approaches and discrimination results In order to evaluate the performance of the proposed features for writer discrimination, a comparative study is
Fig. 13. Histogram of R for all pairs u and u obtained from 50 writers and the English word. Partition level is 5 (10-dimensional 1 2 feature vector). (a) Features from compressed projection functions and trapezoidal windows. (b) Features from compressed projection functions and rectangular windows. (c) Features from simple projection functions and rectangular windows. (d) Features from simple projection functions and trapezoidal windows.
394
E.N. Zois, V. Anastassopoulos / Pattern Recognition 33 (2000) 385}398
carried out by means of two well-established classi"cation schemes. The conventional Bayesian approach using weighted distance measures is examined "rst. The simple multilayer perceptron is tested, next. In biometrics the most common issues concerning the e!ectiveness of features, which potentially describe the behavior of a person, are identi"cation and veri"cation. In the "rst case, an unknown sample is classi"ed among a number of writers thus answering the question `who does this sample belong to?a. The latter case deals with the problem of deciding whether a word or text belongs to a speci"c writer or not. So, the question arising here is `does that sample belong to that specixc person?a. In both cases, the classi"cation approach followed was to employ the database described in Section 2 and form the feature space and the corresponding covariance matrices. The word under test is assigned to a speci"c writer based on the distance of the feature vector from the corresponding cluster center. 5.1. Classixcation using Bayesian approach According to the material exposed in the previous section, the Gaussian pdf is a satisfactory approximation regarding one set of the features. For this same feature, the maximum separability in the feature space is achieved based on the ratio R. In this subsection, the identi"cation problem is considered using this particular feature. The classi"cation criterion is based on the weighted distance from the center of each cluster d "(x!m )t C~1(x!m ), (19) i i i i where m is the center of each cluster, and C is the i i corresponding covariance matrix. In case that each C i equals I, the Euclidean distance is obtained. The estimation of m and C constitutes the training procedure of i i the conventional Bayesian classi"er. For the 50-writer group, the classi"cation procedure was based on the leave-one-out-method [22]. According to this method the covariance matrix C as well as the i center m of each cluster in the feature space are found i using 2249 points. Then the remaining point is assigned to the writer with the minimum d . The procedure is i repeated for all 2250 points of the feature space. Both the mean and the covariance must be determined every single time. However, methods have been developed and used here that overcome the problem of designing N"2250 classi"ers [23]. This is done by properly weighing the mean and the covariance of each class using the following relations: 1 M K "M K ! (X(i)!M K ), ik i N !1 k i i 1 RK "RK # RK ik i N !2 i i
(20)
N i (X(i)!M K )(X(i)!M K )t, (21) ! i k i (N !1)(N !2) k i i where M K , RK are the mean and covariance estimates of ik ik the i-class without the X(i) sample, N is the class populak i tion, and MK , RK are the estimates when all the samples i i are used. The method was applied to both the English and the Greek word. The partitions (number of segments) employed in order to examine the e$ciency of the feature vector were 5, 6, 7, 8, 10, 12 and 15 leading the dimensionality of the feature space to 10, 12, 14, 16, 20, 24, and 30, respectively. The experimental results of the above 50-writer identi"cation procedure are presented in Table 4a and b. The total identi"cation success was found 92.48% for the English word and 92.63% for the Greek one. This is considered satisfactory for simultaneously discriminating 50 writers and using only one word. It has been observed that beyond a partition level (which in our case is 10) the feature e$ciency drops drastically. This is due to the fact that there exist highly correlated feature components, a large number of which is zero. This makes the calculation of the weighted distances impossible. Therefore, only the Euclidean distance can be used which, however, results in a poor success rate. The veri"cation problem was experimentally studied in the following way. For each writer, two di!erent classes were de"ned. The "rst class (u ) contained the 1 genuine samples of the speci"c writer, whereas the other (u ) contained the samples of the remaining 49 writers. 2 Thus, a total of 50 pairs was formed and evaluated. For each pair the individual cluster centers m (i"1, 2) i and covariance matrices C (de"ned as previously) were i evaluated using 2249 out of the 2250 points in the feature space. After that, the remaining point was classi"ed into one of the two classes (writer i or not) based on the minimum weighted distance as (19) indicates. The leaveone-out-method was repeated 2250 times. The type of feature achieving the maximum identi"cation rate (Table 4, 10 segments partition) was used in this experiment, as well. The results are presented in Table 5. The veri"cation error is of the order of 5% for both words when the weighted distance is used. The mean value of R was evaluated for the 50 writers in order to have an approximate measure for the veri"cation rate. However, it is noted that the classi"cation error is much smaller than that determined by the separability measure R, since the orientation of the cluster hyperellipsoids in the feature space, in general, is di!erent from the direction de"ned by the line m6 !m6 . i j 5.2. Classixcation using neural networks The performance of the neural network classi"cation scheme is investigated in case of the general identi"cation problem using 50 writers. So, the capability of the
E.N. Zois, V. Anastassopoulos / Pattern Recognition 33 (2000) 385}398
network to separate the feature space and correctly classify the majority of the samples is tested. The classi"er employed was a three-layer neural network with 20 neurons for the input and the hidden layer. The 20-dimensional feature vector (10 level partition was used) was inserted into the input layer. The output layer consists of six neurons. Consequently, the six-bit binary number at the output of the network will point to one of the "fty clusters. The network was trained using half of the samples (1150). The rest samples (1100) were used to test the network in the discrimination procedure. The method was applied to both the English and the Greek word. The classi"cation results obtained by means of this procedure are given in Table 6. The identi"cation error is in the order of 3.5%. It is noted that the identi"cation success is higher than that obtained using the classical Bayesian approach followed previously. This is due to the fact that employing neural networks the feature space is divided into decision regions independently of the underlying cluster statistics. The training procedure terminates when the sum of squared errors becomes quite small as it is shown in Fig. 14. Actually, this error may be relatively large when it is used as a criterion for stopping the
Table 4 (a) Identi"cation rate in 50 writers case: English word Partition level Euclidean distance Weighted distance
5
6
7
8
10
12
15
48.18 46.70 54.85 56.14 56.74 51.22 44.29 69.51 71.59 80.77 87.59 92.48
X
X
(b) Identi"cation rate in 50 writers case: Greek word Partition level Euclidean distance Weighted distance
5
6
7
8
10
23
25
395
training of the network, especially when the number of training samples is small and the clusters are not well separated in the feature space [19]. Therefore, it is preferable to train the network so that the best performance from the available samples is achieved by allowing the error to become as small as possible through several epochs. The veri"cation issue, which means separate one cluster from the rest 49, can be solved much easier using a simpler neural structure. This happens because the feature space is to be separated into two di!erent regions only. The experimental results acquired were better than those taken when solving the general identi"cation problem. However, for each writer a di!erent network is required. Cumulative experimental results are shown in Table 7. The veri"cation error is of the order of 2%.
6. Conclusions and discussion A new feature vector is proposed for writer discrimination by means of morphologically transforming the projection function of a word. This waveform is a description of the way the pixels of the word are distributed along the direction of projection. The feature vector is formed ignoring the blanks between the letters since in this case the separability of the clusters is better. Furthermore, the use of trapezoidal windows (segments) for the formation of the feature vector results in Gaussian statistics and higher separability in the feature space. An extensive study for the statistics (pdf ) of the feature components was provided using the Kolmogorov}Smirnov "t test. The dimensionality of the feature vector is determined by the length of the word, the number of SEs used to process
Table 6 Classi"cation results for the general identi"cation problem using the neural network approach: partition level is 10
50.85 55.30 58.56 59.41 63.74 65.22 60.57
Language
Classi"cation success
70.85 77.37 83.89 86.11 92.63
English Greek
1061/1100 (96.5%) 1069/1100 (97.0%)
X
X
Table 5 Classi"cation results for binary decision problem (veri"cation): partition level is 10 Language and similarity measure used
Mean separability measure R
Maximum expected classi"cation error based on R (%)
Experimental classi"cation error (%)
English Euclidean English weighted Greek Euclidean Greek weighted
0.75 0.75 0.79 0.79
22.30 22.30 21.01 21.01
12.25 6.35 14.73 5.57
396
E.N. Zois, V. Anastassopoulos / Pattern Recognition 33 (2000) 385}398
Fig. 14. Sum of squared errors in the identi"cation problem: English word. Partition level is 10.
Table 7 Cumulative experimental results for the veri"cation problem using the neural network approach: partition level is 10 Language
Classi"cation success
English Greek
1075/1100 (97.7%) 1085/1100 (98.6%)
the obtained waveform as well as by the number of segments the original image is divided into (partition level). A database was built to test the discrimination capabilities of the proposed feature. Fifty writers were employed for this purpose and two di!erent words of the same length were used, an English word and the corresponding (same meaning) Greek one. It is shown throughout the paper that the identi"cation results obtained using these two words are equivalent, which means that the proposed feature is independent of the language used. The database can be accessed by any researcher though the Internet. The classi"cation results obtained using the proposed feature can be considered satisfactory given that only one word is employed for writer discrimination. Two di!erent classi"cation schemes were tested, namely, the conventional Bayesian classi"er and the neural networks. The veri"cation problem was solved considering the writer to be veri"ed as belonging to class u , while the 1 rest of the writers form the class u . The veri"cation error 2 using the Bayesian approach is in the order of 5% as shown in Table 5. It is shown in Table 7 that using the neural nets this error becomes quite smaller (2%). In the general identi"cation case, where each class corresponds to a di!erent writer, the classi"cation error is in the order of 7% as shown in Table 4. The corresponding error when neural networks are employed is found to be 3.5% (Table 6). All experimental results were obtained using a 20-dimensional feature vector, which corresponds to
a 10-level partitioning. For this partitioning the highest identi"cation rate was acquired for the speci"c length of the words in our data base (Table 4). Furthermore, it was proved that the components of the proposed feature vector are correlated (Tables 2 and 3). Thus, the intrinsic dimensionality of the feature space which can be obtained using the eigenvalues of the covariance matrices is small (Table 1). In order to avoid a complicated theoretical analysis to test the separability of the clusters in the feature space employing the covariances and the correlation of the features, a separability measure was proposed described by Ref. [14]. This measure was statistically analyzed and used successfully throughout the experiments (Table 5). The computational load required to carry out the whole procedure can be divided into two parts. The feature extraction step and the classi"cation process. Feature extraction, which is the main objective of this paper, is performed in milliseconds for each of the words in our database, using MATLAB routines (for thresholding, thinning, projecting and resampling) and simple min}max "lters for morphological transformations. The computer system used was a Pentium-133 running Windows-NT and MATLAB 5. The time required to perform classi"cation depends on the method employed (Bayesian or Neural) and the training procedure (leave-one-out-method, back-propagation, etc.). The training procedures in our experiments were of the order of minutes. For further classi"cation improvement more than one word can be used by means of fusion techniques [3].
7. Summary In this work a writer identi"cation method is proposed, which is based on the use of a single word. A new feature vector is employed by means of morphologically transforming the projection function of the word. First, the image of the word is properly preprocessed (thresh-
E.N. Zois, V. Anastassopoulos / Pattern Recognition 33 (2000) 385}398
olded thinned) and then projected onto the horizontal direction. The obtained projections are segmented in parts which are morphologically processed in order to obtain the required feature vector. The morphological processing is a type of granulometry. Two di!erent types of windows are applied to the segments of the projection to control the #ow of information from one part to the other. The blanks between the letters are also taken into consideration in the formation of the feature vector. An extensive study for the statistics (pdf) of the feature components is provided employing the Kolmogorov} Smirnov "t-test. The identi"cation success depends on the dimensionality of the feature vector which in turn depends on the length of the word, the length of the SEs and the partition level. Furthermore, it is proved that the components of the proposed feature vector are correlated. In order to avoid a complicated theoretical analysis to test the separability of the clusters in the feature space, a separability measure is proposed. This measure is statistically analyzed and used successfully throughout the experiments. A database was built to test the discrimination capabilities of the proposed feature. Fifty writers were employed for this purpose and two di!erent words of the same length were used, an English word and the corresponding (same meaning) Greek one. It is shown throughout the paper that the identi"cation results obtained using these two words are equivalent, which means that the proposed feature is independent of the language used. The database can be accessed by any researcher through the Internet. The capability of the proposed feature for writer discrimination is extensively studied. Two di!erent class"cation schemes are tested, namely, the Bayesian classi"er and the neural networks. In the classi"cation procedure the binary decision problem and the general classi"cation problem are studied. The classi"cation results obtained using the proposed feature can be considered satisfactory given that only one word is used for writer discrimination. The veri"cation error using the Bayesian approach is of the order of 5%, while for the neural nets this error becomes quite smaller (2%). In the general identi"cation case the classi"cation error is of the order of 7%, whereas the corresponding error when neural networks are used is found to be 3.5%.
[4]
[5] [6] [7]
[8]
[9] [10]
[11]
[12] [13]
[14]
[15] [16]
[17]
[18]
[19]
[20]
References [21] [1] R. Plamondon, G. Lorette, Automatic signature veri"cation and writer identi"cation } The state of the art, Pattern Recognition 22 (2) (1989) 107}131. [2] M. Yoshimura, F. Kimura, On a formal measurement of individual variability appeared in handwritten characters, Trans. IECE Japan 9 (1980) 795}802. [3] E.N. Zois, V. Anastassopoulos, Fusion of correlated decisions for writer identi"cation, Proceedings of the IEE
[22] [23] [24]
397
Third European Conference on Handwriting Analysis and Recognition, Brussels, 14}15 July 1998. V. Clement, K. Steinke, R. Naske, The application of image proccesing and writer recognition techniques to the forensic analysis of handwriting, Proceedings of the International Conference Secur., West Berlin, 1980, pp. 5}11. V. Clement, Forensic writer recognition, Proceedings of the NATO Advanced Studies Institute, 1981, pp. 519}524. K. Steinke, Recognition of writer by handwriting images, Pattern Recognition 14 (1981) 357}364. E.N. Zois, V. Anastassopoulos, Methods for writer identi"cation, Proceedings of the International Conference on Circuits and Systems, Rodos Island, Greece, 13}16 October 1996, pp. 740}743. F. Mihelic, N. Pavesic, L. Gyergyek, Recognition of writer of handwritten texts, Proceedings of the International Conference on Crime Countermeasures, Kentucky University, Lexington, 1977, pp. 237}240. M. Levine, Vision in Man and Machine, McGraw-Hill, New York, 1985. T. Pavlidis, Algorithms for shape analysis of contours and waveforms, IEEE Trans. Pattern Anal. Machine Intell. PAMI-2 (1980) (4) 301}312. Y. Qi, B.R. Hunt, Signature veri"cation using global and grid features, Pattern Recognition 27 (12) (1994) 1621}1629. R. Nagel, A. Rosenfeld, Computer detection of freehand forgeries, IEEE Trans. Comput. C-26 (9) (1977) 895}904. S.W. Lee, J.S. Park, Nonlinear shape normalization methods for the recognition of large set handwritten characters, Pattern Recognition 27 (7) (1994) 902}985. T. Hilderbrand, W. Liu, Optical recognition of handwritten Chinese characters: advances since 1980, Pattern Recognition 26 (2) (1993) 205}225. S.R. Sternberg, Grayscale morphology, Comput. Vision Graphics Image Process. 35 (1986) 333}355. P. Maragos, Pattern spectrum and multiscale shape representation, IEEE Trans. Pattern Anal. Machine Intell. PAMI-11 (7) (1989) 701}716. V. Anastassopoulos, A.N. Venetsanopoulos, The classi"cation properties of the spectrum and its use for pattern identi"cation, Circuits Systems Signal Process. 10 (3) (1991) 117}143. E.N. Zois, V. Anastassopoulos, &writer identi"cation data base II', ftp://anemos.physics.upatras.gr/pub/handwriting/HIFCD2, Electronics Laboratory, Physics Dept., University of Patras, Patras, Greece, 1998. J.P. Drouhard, R. Sabourin, M. Godbout, A neural network approach to o!-line signature veri"cation using directional pdf, Pattern Recognition 29 (3) (1996) 415}424. A. Papoulis, Probability, Random Variables and Stochastic Processes, McGraw-Hill, New York, 1994. W.H. Press et al., Numerical Recipes, Cambridge University Press, Cambridge, 1992. R.O. Duda, P.E. Hart, Pattern Classi"cation and Scene Analysis, Wiley, New York, 1970. K. Fukunaga, Statistical Pattern Recognition, Academic Press, San Diego, 1990. M. Abramowitz, I.E. Stegun, Handbook of Mathematical Functions, U.S Dept. of Commerce, National Bureau of Standards, 1972.
398
E.N. Zois, V. Anastassopoulos / Pattern Recognition 33 (2000) 385}398
About the Author*ELIAS ZOIS was born in Athens, Greece in 1971. In 1994 he received his B.Sc. degree in Physics from Physics Department, University of Patras, Greece. In 1997 he received his M.Sc. in Electronics from Electronics Laboratory, University of Patras, Greece. His Master Thesis was on handwritten text analysis and writer identi"cation. He is now a research fellow in the same laboratory working towards his Ph.D. His main interests are analog circuits design, digital image processing and pattern recognition and classi"cation with emphasis on handwritten text analysis and discrimination. About the Author*VASSILIS ANASTASSOPOULOS was born in Patras, Greece, in 1958. He received the B.Sc. degree in Physics in 1980 from the University of Patras, Greece and the Ph.D. in Electronics in 1986 from the same University. His Ph.D. Thesis was on Digital Signal Processing and in particular, on Delta Modulation Filters. From 1980 to 1985 he was employed as a research assistant in the Electronics Laboratory, University of Patras. From 1985 to 1989 he was a lecturer in the same Laboratory. From 1989 to 1990 he worked as a research associate in the Department of Electrical Engineering, University of Toronto, on Nonlinear Filters and Pattern Recognition and Classi"cation Techniques. From 1992 he is an Assistant Professor in Electronics Laboratory, University of Patras, Greece. Since 1990, he has been in close cooperation with AUG Signals Ltd, a Canadian company, working on Radar Signal Detection, IR Image Processing and Data Fusion. He worked with this company during his sabbatical in Canada (1994}1995). His research interests are within the scope of Digital Signal Processing, Image Processing, Radar Signal Processing and Pattern Recognition and Classi"cation. He is a member of the IEEE.
Pattern Recognition 33 (2000) 399} 412
On-line recognition of cursive Korean characters using graph representation Keechul Jung*, Hang Joon Kim Department of Computer Engineering, Kyungpook National University, Sangyuk-dong, Pookgu, Taegu, 702-701 South Korea Received 31 March 1997; accepted 4 March 1999
Abstract The automatic recognition of cursive Korean characters is a di$cult problem, not only due to the multiple possible variations involved in the shapes of characters, but also because of the interconnections of neighboring graphemes within an individual character. This paper proposes a recognition method for Korean characters using graph representation. This method uses a time-delay neural network (TDNN) and graph-algorithmic post-processor for grapheme recognition and character composition, respectively. The proposed method was evaluated using multi-writer cursive characters in a boxed input mode. For a test data set containing 26,500 hand-written cursive characters, a 92.3% recognition rate was obtained. ( 2000 Pattern Recognition Society. Published by Elsevier Science Ltd. All rights reserved. Keywords: Cursive character; Graph representation; TDNN; Grapheme interpretation graph; Viterbi; Grapheme
1. Introduction The automatic recognition of cursive characters has recently enjoyed an increased interest from researchers, especially in view of the development of pen-based notebook computers for a more natural and e$cient man}machine interface. To develop a recognition method for a set of characters, we need to understand the characteristics of the character set. Korean, like Chinese and Japanese, is a large-alphabet language and it has many similar characters constructed with graphemes. Moreover, on-line Korean characters have wide variations in their number of strokes and shapes. These characteristics make it di$cult to recognize on-line Korean characters. Korean character recognition methods can be classi"ed according to the recognition units of character [1], stroke [2] or stroke segment [3], and grapheme [4,5]: (1) The use of character as a unit requires excessive
* Corresponding author. Tel.: #82-53-940-8694; fax: #8253-957-4846 E-mail address:
[email protected] (K. Jung)
memory space and processing time. Accordingly, a large-classi"cation technique is often used, however, this results in an incomplete algorithm with many errors; (2) Since there can be many variations of stroke segments for a given stroke, a cursive stroke will necessarily include many di!erent representations. Whole cursive strokes have been used as recognition units in previous research, yet this has a drawback as it restricts the addition of new strokes; and (3) As Korean characters are composed of two or three graphemes, some researchers have used the grapheme as a recognition unit. For hand-printed Korean characters, this approach is e$cient since the character recognition problem is reduced to a grapheme recognition problem, which is simpler and appropriate for the composition rules of Korean characters. However, segmenting a hand-written character into graphemes without contextual knowledge is a very di$cult task. In this paper, we propose a grapheme-based recognition method of Korean characters using a graph representation [6,7]. The graph representation of a character consists of a set of nodes and edges: nodes include the grapheme recognition results and edges include the transition probabilities between two graphemes. After all the graphemes of a character are identi"ed (a grapheme
0031-3203/00/$20.00 ( 2000 Pattern Recognition Society. Published by Elsevier Science Ltd. All rights reserved. PII: S 0 0 3 1 - 3 2 0 3 ( 9 9 ) 0 0 0 6 2 - X
400
K. Jung, H.J. Kim / Pattern Recognition 33 (2000) 399}412
Fig. 1. Block diagram of proposed system.
interpretation graph is constructed), a character interpretation score can be computed using the Viterbi algorithm. This character interpretation score will ultimately determine the stroke segmentation position. Such a recognition-based segmentation results in many `tentative graphemes (candidate cut)a, because without contextual knowledge it is di$cult to segment graphemes from a hand-written character. The character recognition scores determine the stroke segmentation (de"nite cut) into graphemes, and once the graphemes of a character are identi"ed, the character can be easily recognized. Most Korean character recognition systems try to segment graphemes from a character, however, this is not easy as stroke contacts frequently occur due to various writing styles. Such a recognition-based segmentation method is frequently used for o!-line character recognition and on-line English recognition [7}10]. In the proposed method, the boundary of tentative graphemes is de"ned using heuristic segmentation points. Corner points are assumed to be most appropriate for heuristic segmentation points since Korean characters are com). A TDNN posed of mostly straight lines (except is used to identify the graphemes. This network is a multilayer feed-forward network, the layers of which perform successively higher-level feature extractions to produce a "nal classi"cation. TDNNs have also been very successful for speech and character recognition [6,11,12]. In the character composition stage, we identify the most appropriate path (character) in the graph produced by the TDNN. Fig. 1 shows a diagram of the processes and information available at the various stages of our system. This paper is organized as following. A detailed explanation of each step of the proposed system is included in Section 2. Experimental results are outlined in Section 3, and the "nal conclusions are summarized in Section 4. 2. Methodology The proposed recognition system consists of six steps: input data, preprocessing, candidate cut generation,
feature extraction, grapheme recognition, and character recognition. The block diagram above outlines the proposed recognition system, and a detailed explanation of each step follows: In Fig. 1, we illustrate the various steps of the proposed system. The rounded rectangles include the names of each stage (denoting the activity in that step), and the rectangles beneath indicate the method or tool used in that stage. The input to the system is a set of handwritten components sampled by a digitizer. A component consists of a pair of vectors x(t), y(t), t3M0, 1,2, i,2, nN that de"ne the tablet's pen position along the two orthogonal axes of its writing surface, sampled at "xed time intervals. Because the raw data contain numerous variations in writing speed, in stage (b) the data are resampled in order to obtain a constant number of regularly spaced points on the trajectory. Stage (c) de"nes `tentative graphemesa delimited by `corner pointsa. These tentative graphemes are then passed to stage (d) which performs the feature extraction of each tentative grapheme. Stage (e) recognizes the graphemes using a TDNN. The recognition results are then gathered into a grapheme interpretation graph and "nally in stage ( f ), the graph of grapheme recognition scores is used to identify the best path. This approach is introduced in Ref. [13]. 2.1. Input data and preprocessing An input character is a sequence of absolute (x, y) positions on a tablet. The tablet dimension used is 5]5 inch and the resolution is 500 LPI. In this study, no constraints are placed on the writers other than a boxed input mode. Writers are not given any guidelines about writing speed or style, only instructions about operating the tablet. To reduce geometric distortion, the input sequence of positions is resampled to points regularly spaced in an arc length using linear interpolation (shown in Fig. 2b). The pen trajectory is slightly smoothed using following
K. Jung, H.J. Kim / Pattern Recognition 33 (2000) 399}412
401
Fig. 2. Examples of preprocessing.
Fig. 3. Examples of candidate cut generation.
Eq. (1). The symbol x means the x-position and y does i i y-position in x}y coordination. x "(!3x #12x #17x #12x !3x )/35. i i~2 i~1 i i`1 i`2 (1) The arrows of Fig. 2a denote the writing direction in a stroke which is the locus from the pen-down to the pen-up position. As shown in Fig. 2a, the strokes of a Korean character are usually directed from top to bottom and left to right. 2.2. Candidate cut generation Corner points [14] are used for candidate cut generation. This method detects the local maximum convex or concave curvatures of a stroke using a chain code representation. An example of detected corner points is shown in Fig. 3a. Intuitively, corner points mean the pen-start, pen-end, and abrupt direction change positions. In Fig. 3b, all strokes are numbered to show the order of the tentative graphemes.
2.3. Feature extraction At this stage, the feature vectors of all possible tentative graphemes are extracted for use as input data for the TDNN (grapheme recognizer). One or more segments are combined to form a tentative grapheme. Accordingly, a tentative grapheme represents a combined segment that links one or more segments. These tentative graphemes are normalized into an appropriate input size before being extracted as feature vectors in order to reduce the time and scale distortion in the hand-written data. Each tentative grapheme is normalized into seven feature vectors with 54 length points and is bounded and vary between 0 and #1. Some examples of tentative graphemes are shown in Fig. 4. An example of feature vectors is illustrated in Fig. 6. In this "gure, time increases along the horizontal axis from left to right, and each line of boxes corresponds to the following feature vector. A feature includes pen up/down, pen coordinates, direction and curvature [11] (Fig. 5 shows the meaning of direction (h) and curvature (/)). The direction of the pen movement is represented into two features. The sine and cosine values of direction for
402
K. Jung, H.J. Kim / Pattern Recognition 33 (2000) 399}412
Fig. 4. Examples of tentative graphemes.
seen in the intermediate representation. All these position correspondences between an input character and intermediate representation can be detected with careful observance. 2.4. Grapheme recognition
Fig. 5. Meaning of curvature (/) and direction (h).
each position are calculated. The curvature of the pen movement also has two features. The sine and cosine values of the angle between two consecutive points are calculated. The curvature and direction of the x}y coordination are normalized between 0 and #1. These kinds of feature vectors include local information along the pen trajectory. They are then passed to a TDNN in order to extract higher order features and "nally classify the grapheme. All these components are bounded and vary between 0 and #1 as follows:
G
H
1 if pen is up, x!x .*/ , , f " f " 1 x !x 0 0 otherwise, .!9 .*/ f " 2 y
y!y .*/ , f "(cos h#1)/2, 3 !y .!9 .*/
f "(sin h#1)/2, f "(cos /#1)/2, 4 5 f "(sin /#1)/2. 6 In Fig. 6, position A indicates a drastic directional change. Therefore, the darkness of the feature vectors representing direction and curvature also changes suddenly. Position B is in the middle of a gentle curve, and the intermediate representation around point B shows a non-changing curvature (feature f and f do not vary 5 6 much) and smooth curve. Point C is a straight line: indicated by the constant direction and zero curvature, as
A TDNN is very useful for representing the time relationships between events [12], thus it was used as the grapheme recognition engine. Extracted feature vectors were used as the input to the TDNN in order to obtain recognition scores. These results were then accumulated into a grapheme interpretation graph that was used to identify the best path for deciding on a result character. 2.4.1. Time-delay neural network architecture For the recognition of graphemes, a three-layer timedelay neural network was constructed. Its overall structure is shown in Fig. 7. At the lowest level, the number of input nodes is equal to the dimensions of the input vector (each tentative grapheme is normalized into seven feature vectors with 54 length points). A neuron has an input "eld that is time restricted. The output layer shows the grapheme recognition scores. There is one output per grapheme class (three outputs per nil grapheme), thereby, providing scores for all the graphemes of the Korean characters. 2.4.2. Training and recognition A TDNN was trained with graphemes from handprinted and cursive character data (Fig. 14). At the start of the experiments, all the segmentations into graphemes were completed automatically as only hand-printed data were used since it can be easily divided into segments. Accordingly, the grapheme recognition rate for the cursive data, as shown in Table 3, was relatively lower. The TDNN was trained with a back-propagation algorithm. The purpose of algorithm was to minimize the sum of the squared errors between the TDNN outputs and target values during training. The target value was set at #1 for a correct interpretation and at 0 for all other classes.
K. Jung, H.J. Kim / Pattern Recognition 33 (2000) 399}412
403
Fig. 6. Example of feature vectors from the data in Fig. 4. The tentative graphemes of the character shown in Fig. 4 have been resampled with 54 points. The color of each box indicates the value of the component and all these components are bounded between 0 and #1 [6].
Fig. 7. Time-delay neural network: (a) what is done at a neuron used in (b); (b) architecture of the network; (c) parameters of the network architecture [11].
The training of the TDNN was achieved with several sets of 1183 grapheme examples (corresponding to 500 selected characters) produced by eight writers. Some characters consisted of two graphemes, that is, a "rst consonant and vowel, while other characters had three graphemes including a last consonant. The actual distribution of all the training sets of graphemes extracted from the hand-printed data is shown in Fig. 8. The horizontal axis denotes 67 Korean graphemes. The distribution of samples among the di!erent grapheme classes in the training set was non-uniform and some of them were omitted. However, the omitted graphemes are of no concern, as since they were not tested, they had nothing to do with the recognition rate. These missing graphemes are also rarely used in daily life. As shown in Fig. 15, the recognition rates of the
grapheme classes that had a relatively small class distribution were somewhat lower than the other classes. The recognizer was trained with both valid graphemes and `counter-examplesa. Counter-examples include meaningless graphemes (nil graphemes), which should receive a nil interpretation. These include stroke segments that connect two graphemes or stroke segments within a grapheme and have three outputs in the TDNN's output node as they have many variations in their feature vectors (Fig. 14c). Even though only hand-printed data were used to train the grapheme recognizer, a reasonable recognition rate for cursive data could still be acquired. When the recognition rates for cursive characters exceed about 90%, training data from cursive characters can be automatically collected using the proposed recognition
404
K. Jung, H.J. Kim / Pattern Recognition 33 (2000) 399}412
Fig. 8. Distribution of the graphemes used in training and testing.
Fig. 9. All graphemes derived from Korean characters.
system. Accordingly, this saves on the labor required to manually segment cursive characters, and slightly enhances the grapheme recognizer. Mis-recognized characters must still be manually segmented for use as training data. 2.5. Character recognition: searching for the best path on the graph Korean has di!erent characteristics from English or Chinese. To make a good recognition system, it is important to know the characteristics of the target characters. The following are the basic structure and composition rules of Korean characters: Ten of the simple graphemes are vowels and the rest are consonants. The characters are composed of 1}4 simple consonants and 1}3 simple vowels. As this composition rule is too complicated, most researchers de"ne and use complex graphemes. If complex graphemes are adopted, a single character includes 1 consonant, 1 vowel, and 1 last consonant (optional). Complex vowels and consonants are made by combining simple vowels and simple consonants, respectively. Fig. 9 shows the complete variation of graphemes derived from Korean characters. Korean characters have a two-dimensional structure. Fig. 10 shows each type, where VV denotes a vertical vowel, HV a horizontal vowel, C1 the "rst consonant, and C2 the last consonant. For every character, there must be one "rst consonant and at least one vowel. If it
Fig. 10. The structure of Korean characters: (a) general structure; (b) six types of Korean characters.
exists, the "rst consonant should be on the left of the vertical vowel and on top of the horizontal vowel. The optional last consonant is below the "rst consonant and vowel. Except for the vertical vowel, all graphemes have vertical relationships in two dimensions. There are a possible 11,172 characters with simple grapheme combination rules, and about 3000 of them are used daily. The frequency of each character, grapheme, and stroke is a very important factor for developing a character recognition system. The frequency of each character was checked from newspaper articles. From newspaper articles over three months, 175,955 characters were selected and their frequency tested. From the results, it was clear
K. Jung, H.J. Kim / Pattern Recognition 33 (2000) 399}412
405
Fig. 11. Grapheme interpretation graph produced by the TDNN. (a), (b) and (c) represent the recognized graphemes, thereafter, these graphemes are combined into one character.
that only a small portion of the possible 11,172 characters was used frequently. Therefore, it could be determined that only 500 characters could cover 99% of Korean text. Fig. 8 also demonstrates a huge di!erence in usage among the various graphemes. In this research, character recognition is associated with a graph where the nodes contain grapheme recognition scores. A one-to-one correspondence exists whereby every path through the graph branches corresponds to a particular legal segmentation of the input character, and, conversely, every possible legal segmentation of the input character corresponds to a particular path through the graph. In Fig. 11, the vertical and horizontal lines indicate the start and end segment numbers of tentative graphemes, respectively. The various shaded blocks in Fig. 11 signify the recognized results of each sequence of a tentative grapheme: the darker the block, the higher the recognition score. Each tentative grapheme has a double index shape like the i}j grapheme in Fig. 11, which connects the ith segment of the input character at the grapheme starting point to the jth segment at the grapheme ending
the best-scoring path (corresponding to an interpretation for a character). There are many possible character interpretations in the graph, extending from the starting node of the "rst column to the terminal node. The probability of a given path is the sum of the product of all its node and edge values. Each node in the graph is assigned a value, which is derived from the recognizer score for the tentative grapheme corresponding to that node. An edge value signi"es the transition probability between two graphemes. The probabilities needed to build a conventional HMM-type network are denoted as Mn, A, BN in HMM notation. These statistics can be calculated by examining the characters within the database and tabulating the frequency with which each grapheme follows every other graphemes. All of these statistical counts are then normalized to yield the "nal probabilities. The following is the notation used in Fig. 12. In the implementation, the outputs of the TDNN were considered as symbol probabilities (B matrix of HMM; Eq. (2)). X(i, j) represents the node in the grapheme interpretation graph for all tentative graphemes.
the value of output node of a TDNN correspondingto a grapheme x the value of x(i, j)" sum of the values of all output nodes of a TDNN
point. For example, Fig. 11a represents the recognition score for the combined tentative grapheme of the input character inclusive of segments one through six. In this graph representation, the Viterbi algorithm provides a convenient method for rapidly determining
(2)
P(>(i, j)D X(a, b)) is the transition probability from node X(a, b) to node >(i, j) on the grapheme interpretation graph. Table 1a presents the initial state probability distributions of the "rst consonant from the grapheme interpretation graph (Eq. (3)). This can
406
K. Jung, H.J. Kim / Pattern Recognition 33 (2000) 399}412
Fig. 12. Search algorithm.
be denoted as (x(1, m)Dx(0, 0)) in the algorithm of Fig. 12. As shown in Table 1, graphemes such as ( ) are frequently used as the "rst consonant.
Table 1b presents the transition probabilities from "rst consonants to vowels, whereas Table 1c presents from vowels to last consonants. These are shown in the following equation:
the number of characters beginning with x n(x)" total number of characters
the number of transitions from x to y p(yDx)" the number of transitions from x
(3)
(4)
K. Jung, H.J. Kim / Pattern Recognition 33 (2000) 399}412
407
Table 1 Probabilities used in composing graphemes
Character interpretation is achieved by searching for the best path using the Viterbi algorithm. In Fig. 12, a formal statement of the algorithm is outlined, which is actually just a simple version of forward dynamic programming.
When investigating the best path using the Viterbi algorithm, some improper (uncomposable) character interpretations can occur. These search paths are pruned, whereby any path that corresponds to an improper
408
K. Jung, H.J. Kim / Pattern Recognition 33 (2000) 399}412
Fig. 13. Korean grapheme composition rule.
Fig. 14. Examples of the training and test data (a) hand-printed data; (b) cursive data; (c) ligature (dotted line indicates nil graphemes).
character interpretation can be removed from the graph. These are implemented in the Viterbi algorithm (the algorithm tests whether it would keep the Korean grapheme composition rule: such as a Korean character is written structurally with a head consonant, a vowel, and an optional bottom consonant). For example, among Korean characters, either a ligature or a vowel can follow a "rst consonant, and either a ligature or a last consonant can follow a vowel. Fig. 13 illustrates the basic Korean composition rule. Each node of Fig. 13 indicates a possible state in composing a character, and the double-lined circle signi"es a terminal node where the character composition is completed. According to the algorithm in Fig. 12, the transition probability between graphemes, denoted as p(x(m, i!1)Dx(i, j)), is only multiplied in the case of a solid line transition.
3. Experiment results In order to verify the proposed recognition method, further training and test data was used, as shown in Fig. 14, experiments were performed in several di!erent environments, and ACECAT was used as an alternative input device. Experiments were conducted using a database containing a total of 60,079 unconstrained on-line handwritten Korean characters. This database
consisted of four subsets. Database 1 was generated by eight di!erent writers with a few constraints (e.g. consecutive graphemes could not be connected i.e. noncursive). Database 2 was from KAIST (Korea Advanced Institute of Science & Technology) written with no constraints. Databases 3 and 4 were written by 30 individuals with no constraints. All characters were written according to a boxed input mode. Table 2 shows the number of characters in the training and test sets. A TDNN was simulated in a C-language program on an IBM Pentium compatible machine. Character recognition speed varied according to the character ranging from 0.12 s to 2.27 s. Accordingly, an additional feature that can cope with this recognition speed variation needs to be identi"ed. Fig. 15 presents the grapheme recognition results for each grapheme. For example the distributions of and are smaller than the others, accordingly, the recognition rates of these graphemes are lower. The grapheme recognition rates are presented in Table 3. We have this result after training the TDNN using only the hand-printed data. Fig. 16 shows an example of character composition. The nodes marked with a circle are de"nite segmented graphemes based on a character interpretation result. Table 4 below presents the recognition results. An evaluation of the proposed recognition system was completed in three steps. Test 1 only used hand-printed data
K. Jung, H.J. Kim / Pattern Recognition 33 (2000) 399}412
409
Table 2 Database used in experiments Database
Characteristics
The number of data For training
DB1 DB2 DB3 DB4
Hand-printed From KAIST Cursive Cursive
For testing
500 (characters)]8 (persons) 19579 (characters) 500 (characters)]8 (persons) 500 (characters)]20 (persons)
17,500 (characters) 500 (characters)]10 (persons)
Fig. 15. Grapheme recognition result for each grapheme.
Table 3 Experimental grapheme recognition results
No. of grapheme Correct recognition Recognition rate(%)
Hand-printed data
Cursive data
5372 5116 95.2
10744 9242 86.0
as training data, whereas tests 2 and 3 included both hand-printed and cursive data. The average recognition rate for cursive characters was about 92.3% (Test number 3). When the TDNN was trained with more data, slightly higher rates were achieved. Also the data from Table 4 indicates a better performance using the proposed recognition system compared with that from Ref. [8]. Accordingly, the proposed
410
K. Jung, H.J. Kim / Pattern Recognition 33 (2000) 399}412
Fig. 16. An example of character composition: (a) input character; (b) grapheme interpretation graph. Table 4 Recognition results Test No.
Training data
Testing data
Recognition rates of Ref. [8] (%)
Recognition rates of proposed method (%)
1
DB1
DB1 DB2 DB3 DB4
100 78.7 83.2 79.3
100 84.3 91.9 80.2
2
DB1#DB2
DB1 DB2 DB3 DB4
100 90.8 82.1 82.7
100 93.2 92.1 86.4
3
DB1#DB2 #DB3#DB4
DB1 DB2 DB3 DB4
100 89.2 91.2 90.7
100 91.4 94.8 93.5
Fig. 17. Examples of mis-recognition.
method can produce a reasonable recognition rate and can be applied in a variety of practical environments. Fig. 17 shows examples of mis-recognition. The main recognition error reasons include: (1) Di$culty in seg-
menting a cursive character stroke into tentative graphemes; (2) As shown in the second of the mis-recognition examples, sometimes the recognizer cannot discriminate between a ligature and a grapheme segment.
K. Jung, H.J. Kim / Pattern Recognition 33 (2000) 399}412
411
References
Table 5 Error ratio according to error type Mis-recognition types
Rates (%)
Grapheme mis-recognition of TDNN Character composition error Candidate cut generation error Ambiguous character Excessive hook Etc.
21.9 10.7 22.4 20.2 15.4 9.4
Table 5, illustrates various error types produced by the proposed recognition system.
4. Conclusion This paper has presented a grapheme-based Korean character recognition method using graph representation. Grapheme recognition results were used to determine de"nite stroke segmentation points, and a TDNN was utilized as the grapheme recognizer. A TDNN has desirable properties related to the dynamic structure of an on-line character. It can detect the temporal structure of characters and their temporal relationships, plus these features learned by the network are insensitive to time shifts. At the character composition stage, the combination of the Viterbi search along with a Korean grapheme composition rule produces the capability to cope with non-ideal cases including the addition of redundant strokes between well-separated ones written in a handprinted letter, the deletion of short strokes between graphemes, abrupt change between curves, or smooth curves between abrupt strokes. Complex graphemes and graphemes with many corner points, like , require a longer time for recognition. The development of a more complex composition rule for graphemes will be attempted to minimize these longer recognition times. The "fth and sixth mis-recognition examples indicate the drawback of using corner points as segmentation points. Accordingly, alternative additional information needs to be identi"ed to compensate for this shortcoming. Future research will investigate the application of the proposed method to character segmentation in an online run-on mode.
[1] H.D. Lee, T.K. Kim, T. Agui, M. Nakajima, On-line recognition of cursive Hangul by extended matching method, J. KITE 26 (1) (1989) 29}37. [2] P.K. Kim, J.Y. Yang, H.J. Kim, On-line cursive Korean character recognition using extended primitive strokes. The third PRICAI (Paci"c Rim International Conference on A.I.), August 1994, pp. 816}821. [3] O.S. Kwon, Y.B. Kwon, An on-line recognition of Hangul handwriting using dynamic generation of line segments, Proceeding of The Twentieth KISS Spring Conference, 1993, pp. 151}154. [4] T.K. Kim, E.J. Rhee, On-line recognition of successively written Korean characters by suitable structure analysis for Korean characters, J. Korea Inform. Sci. Soc. (KISS) 20 (6) (1988) 171}181. [5] D.G. Sim, Y.K. Ham, R.H. Park, Online recognition of cursive Korean characters using DP matching and fuzzy concept, Pattern Recognition 27 (12) (1994) 1605}1620. [6] H. Weissman, M. Schenkel, I. Guyon, C. Nohl, D. Henderson, Recognition-based segmentation of on-line run-on handprinted words: input vs. output segmentation, Pattern Recognition 27 (3) (1994) 405}420. [7] K.C. Jung, S.K. Kim, H.J. Kim, Recognition-based segmentation of on-line cursive Korean characters, Proc. ICNN 6 (1996) 3101}3106. [8] K.C. Jung, S.K. Kim, H.J. Kim, Grapheme-based on-line recognition of cursive Korean characters, J. KITE 33-B (9) (1996) 124}134. [9] D.J. Lee, S.W. Lee, A new methodology for gray-scale character segmentation and recognition, Proceedings of third International Conference On Document Analysis and Recognition, Montreal, Canada, August, 1995, pp. 524}527. [10] J.H. Bae, S.H. Park, H.J. Kim, Character segmentation and recognition in Hangul document with alphanumeric characters, J. KISS 23 (9) (1996) 941}949. [11] I. Guyon, P. Albrecht, Y. Le cun, J. Denker, W. Hubbard, Design of a neural network character recognizer for a touch terminal, Pattern Recognition 24 (2) (1991) 105}119. [12] A. Waibel, T. Hanazawa, G. Hinton, K. Shikano, K. Lang, Phoneme recognition using time delay neural networks, IEEE Trans. Acoust. Speech Signal Process 37 (1989) 328}339. [13] C.J.C. Burges, O. Matan, Y. Le Cun, D. Denkerm, L.D. Jackel, C.E. Stenard, C.R. Nohl, J.I. Ben, Shortest path segmentation: a method for training neural networks to recognize character string, IJCNN'92, Baltimore, vol. 3. IEEE press, New York, 1992. [14] X. Li, N.S. Hall, Corner detection and shape classi"cation of on-line handprinted Kanji strokes, Pattern recognition 26 (9) (1993) 1315}1334.
About the Author*KEECHUL JUNG has been a Ph.D. student in the Arti"cial Intelligence Lab. of the Computer Engineering Department at the Kyungpook National University since March, 1996. He had been awarded the degree of Master of Science in Engineering in Computer Engineering. His research interests include image processing, pattern recognition.
412
K. Jung, H.J. Kim / Pattern Recognition 33 (2000) 399}412
About the Author*HANG JOON KIM received the B.S. degree in Electrical Engineering from the Seoul National University, Seoul, S. Korea in 1977, the M.S. degree in Electrical Engineering from the Korea Advanced Institute of Science and Technology in 1979 and the Ph.D. degree in Electronic Science and Technology from Shizuoka University, Japan in 1997. From 1979 to 1983, he was a full-time Lecturer at the Department of Computer Engineering, Kyungpook National University, Taegu, South Korea, and from 1983 to 1994 an Assistant and Associate Professor at the same department. Since October 1994, he has been with the Kyungpook National University as a Professor. His research interests include image processing, pattern recognition and arti"cial intelligence.
Pattern Recognition 33 (2000) 413}425
A target recognition technique employing geometric invariants Bong Seop Song!, Il Dong Yun", Sang Uk Lee!,* !School of Electrical Engineering, Seoul National University, Seoul, South Korea "Department of Control and Instrumentation Engineering, Hankuk University of F.S. Yongin, South Korea Received 26 March 1998; received in revised form 9 December 1998; accepted 2 March 1999
Abstract In this paper, we propose an e$cient target recognition technique in the complex scenes. For the viewpoint invariant matching, the well-known geometric invariants, the invariants of "ve coplanar points and the invariants of "ve coplanar lines, which are invariant under the projective transform, are employed. To alleviate the computational complexity, the circular ordered matching technique and compatibility test are proposed, utilizing the geometric properties. Moreover, stability of the invariants for "ve coplanar points to the input noise is analyzed, and a method to select the stable invariants is also suggested. The performance of the proposed algorithm is evaluated on both various synthetic and real images. The results of the experiments show that the proposed matching technique is both e!ective and stable to the input noise for the target recognition in complex images. ( 2000 Pattern Recognition Society. Published by Elsevier Science Ltd. All rights reserved. Keywords: Target recognition; Geometric invariant; Circular ordered matching; Stability; Selection rule
1. Introduction It is one of the main goals of computer vision to recognize and understand the objects in the input image. In general, object recognition is carried out by extracting dominant features, such as points, lines, or curves from the input image, followed by matching the extracted features with those of the model. However, the projective transform, which occurs in the 2-D imaging process, causes unavoidable di$culty, since it produces di!erent images, according to the viewpoint of the camera [1]. To eliminate the e!ect of the projective transform, complicated high-level techniques are usually adopted [2,3]. However, the geometric invariants [4] are known to provide relatively easy solutions for the related problems, since the geometric invariants possess the properties which are invariant to the viewpoints. Thus, the geomet-
* Corresponding author. Realtime Vision Inspection Lab, Automation and Systems Research Institute, Seoul National University, San 56-1, Shimlim-Dong, Kwanak-Gu, Seoul, 151742, South Korea. Tel.: #822-887-2672; fax: #822-878-8198 E-mail addresses:
[email protected] (B.S. Song), yun@san. hufs.ac.kr (I.D. Yun),
[email protected] (S.U. Lee)
ric invariants have received considerable interests in computer vision research recently. Several geometric invariants have been suggested, including algebraic invariants [5,6] and di!erential invariants [7,8], which have been applied to simple planar object recognition. Among the well-known geometric invariants, the invariants of "ve coplanar points and "ve coplanar lines [5,9] are the most easily applicable ones. Several techniques to recognize a simple planar object, based on these invariants, have been proposed so far [5,10]. Target recognition is a part of object recognition, in which attempts are made to localize the object, corresponding to the desired target model, in the complex input image. In this paper, we employ the points and linear segments as the features, and determine acceptable group of features, by matching them with those of the model, based on the invariants of "ve coplanar points and "ve coplanar lines. Thus, in this paper, it is assumed that all of the features in the target are extracted from the input image. However, in this approach, one would expect two major problems. First, since all combinatorial candidates among the input features should be compared with the model, the search space becomes enormous, requiring serious computational load. In the conventional invariants-based recognition, hash table [6,10]
0031-3203/00/$20.00 ( 2000 Pattern Recognition Society. Published by Elsevier Science Ltd. All rights reserved. PII: S 0 0 3 1 - 3 2 0 3 ( 9 9 ) 0 0 0 8 5 - 0
414
B.S. Song et al. / Pattern Recognition 33 (2000) 413}425
has been used for fast indexing of the model database. However, it is di$cult to adapt the hash table for the target recognition, since the input image for target recognition normally contains many objects in background. Next, the invariants are known to be sensitive to the input noise, which may result in false match in noisy environment. In fact, in the feature extraction, the input noise inevitably causes the change in the position of the feature, yielding erroneous invariants. Thus, in this paper, attempts are made to develop an e$cient target recognition technique, which is fast in target matching and stable to the input noise [4]. To improve the matching e$ciency, we propose the circular ordered matching algorithm, utilizing the geometric information of the model which is preserved under the projective transform. The proposed algorithm is composed of three steps; compatibility test, geometry analysis, and circular matching. In the compatibility test, relations among the input features are examined, forming the compatibility table, which is then used for pruning the search space. For an example, if any two points are too far to be parts of the desired target, all candidate sets, including both points, can be excluded from the search space, using the compatibility test. The geometry analysis attempts to "nd the preserved structural properties of the features; Any set of points are uniquely divided into two groups, outer points forming the convex polygon and inner points. Under the projective transform, these structures are preserved for the same points. Moreover, for the convex polygon, the circulation order of the points is also preserved under the transform. Thus, the features are re-ordered circularly, and circular matching is equivalent to full comparison. Thus, the searching space can be amazingly diminished, alleviating the computational load signi"cantly. On the other hand, the stability of the invariants to the input noise is of importance in real-world environment. To achieve this goal, we investigate the sensibility of the invariants for "ve coplanar points to the input noise, which is shown to be dependent on the geometric structure of "ve points. Thus, in this paper, the selection rule for the stable invariants is also discussed. This paper is composed of "ve sections. In Section 2, the proposed matching algorithm is described, and the stability of the invariants for "ve coplanar points is discussed in Section 3. Experimental results on both synthetic and real images are presented in Section 4, and "nally the conclusions are given in Section 5.
made object, such as building or vehicle, and the target model is provided as a set of feature points or linear segments. For the given model, the invariants are calculated in advance, forming the invariant vector. From the input image, "rst corner points or linear segments are extracted from the input image as the features of the target. Corners are characterized by the local maxima of the curvature, which can be obtained using various corner operators [11]. Linear segments are higher level features. The edges are extracted from the input image by masking, followed by "tting to linear segments. Then, each combinatorial candidate set among the input features is compared with the model, based on the di!erence of the invariant vector. However, as the number of extracted feature increases, the number of candidates grows rapidly, requiring enormous computational loads. Thus, for practical use, a scheme to alleviate the computational load is preferred, which will be focused in this section. 2.1. Invariant vectors and xtting measure It is well-known that there are two independent geometric invariants, I and I for any "ve coplanar points, 1 2 de"ned as S S I " 431 521, 1 S S 421 531
(1)
where S is the determinant of 3]3 matrix, which is ijk composed of three points P , P , and P , given by i j k S "DP P P D. ijk i j k
(2)
Since points and lines are dual in the projective plane, the invariants of "ve coplanar lines are also described as Eq. (1). Instead of the coordinates of the points, the coe$cients of the lines are used [5]. But, it is worthy to mention that if three or more points among the "ve points are collinear, or three or more lines are concurrent or parallel, the invariants in Eq. (1) could be zero or unde"ned. Thus, such degenerate con"gurations should be avoided by carefully selecting the features constituting the model. Notice that the degenerate con"gurations can be easily found by examining the geometric structure of the model. Assume that the model is composed of M features (corners or linear segments), we de"ne the invariant vectors g and g representing the model, given by 61 62
2. Proposed target recognition technique In this section, we shall describe the proposed target recognition technique, employing the invariants of "ve coplanar points and "ve coplanar lines. In this paper, however, it is assumed that the desired target is a man-
S S I " 421 532, 2 S S 432 521
g" 6i
C D I (1, 2, 3, 4, 5) i I (2, 3, 4, 5, 6) i F
I (M!1, M, 1, 2, 3) i I (M, 1, 2, 3, 4) i
, i"1, 2,
(3)
B.S. Song et al. / Pattern Recognition 33 (2000) 413}425
415
Fig. 1. Example of geometric con"guration of the model.
where I is the invariants for the corner points or the i invariants for the linear segments, given in Eq. (1). Then, target matching is carried out, based on the sum of the normalized error rate, given by
AK
K K
KB
M g !g( g !g( 1k 1k # 2k 2k , E" + (4) g g 1k 2k k/1 where g is the kth element of the vector g , and g( is the ik ik 6i invariant corresponding to g for a candidate in the ik image, respectively.
equivalent to full match. However, in the re-ordered set, the outer points can be compared fully by circulating the points, instead of permuting.
2.2. Circular ordered matching technique To match N features from the input image with M features of the model, a total of P ("N!/(N!M)!) N M candidate sets should be compared with the model, which requires too exhaustive search for practical implementation. To alleviate the computational complexity, we propose the circular ordered matching technique, by taking account of the geometric con"guration of the model. Fig. 1 is an example of the geometric con"guration of the model composed of seven feature points. Any set of coplanar points can be uniquely divided into two groups, M outer points which constitute the convex hull encloso ing all points, and M inner points. Under the projective i transform, the number of outer points and inner points are preserved, since, in spite of the di!erent viewpoint, the outer point is remained to be the outer and vice versa. So, let us de"ne (M , M ) as the geometric type of the model, o i which possesses also invariant property. Moreover, the circulation order of the outer points is also invariant under the transform. Thus, for the given set of features, we rearrange them circularly as [P 2 P iDO O 2 O o], M 1 M 1 2
Fig. 2. The block diagram for the proposed algorithm.
(5)
where P indicates the ith inner point, and O indicates i j the jth outer point, respectively. Since the order of the points can be changed, all permutation of the points is
2.3. Matching strategy and implementation The overall block diagram for the proposed algorithm is shown in Fig. 2. For the given target model, the invariant vector table can be generated in o!-line processing. First, the model is analyzed to understand the geometric con"guration of the model. Then, we obtain the geometric type of the model, which is then rearranged circularly. The stable invariants of "ve coplanar points are chosen, based on the selection rule, which will be discussed in the next section, and the invariant vectors are prepared. To enhance the matching speed, circulated version of the invariant vectors are also obtained to form the invariant vector table. For the input image, "rst, the corner points or linear segments are extracted. Then, C ("N!/(N!M)! M!) N M sets of the features in the input images are tested sequentially. Among the candidates, there exist incompatible pairs of the features which can not be parts of one object. For an example, if we know approximate dimension of the target in the input image, two points whose distance exceeds the dimension can be considered as incompatible. A candidate set including these incompatible pairs
416
B.S. Song et al. / Pattern Recognition 33 (2000) 413}425
need not be matched with the model. Moreover, a triple of collinear points, a triple of concurrent lines, and a triple of parallel lines are considered to be incompatible, since these may generate the degenerate con"gurations. For this purpose, the compatibility table is prepared, by examining the compatibilities among the input features. Only the candidate sets satisfying the compatibility test are forwarded to the next stage. Then, the candidate set is examined, yielding the geometric type. The geometric type of the desired target is same as that of the model. Thus, the candidates yielding di!erent geometric type should be rejected. Finally, the features in the candidate set are ordered circularly and the invariant vectors are calculated. Matching is carried out with the invariant vector table of the model, and the candidate set, yielding the minimum error rate, is selected for the target. For the linear segments, the matching algorithm is similar to that for the points. For simplicity, we restrict the model to be a polygon, for which the circulation order of lines can be speci"ed. The candidate set for the linear segments is tested on the geometric structure, followed by the circular ordered matching. In summary, the proposed algorithm consists of three steps; compatibility test, geometry test, and circular ordered matching. In our approach, by exploiting an additional geometric information of the model, the computational loads are signi"cantly alleviated, which will be described in more detail in Section 4.
3. Stability of the invariants for 5ve coplanar points In the feature extraction, it is di$cult to localize the features correctly, due to the input noise and the problem, related to employing the threshold. It is a common practice to employ the threshold in extracting the features [11,12]. Thus, the feature extraction could be a!ected by the choice of the threshold. In other words, many false features could be extracted, or the desired features could be missed, if we employ inappropriate threshold. Also, the position of the extracted feature cannot be correctly identi"ed, due to the input noise. In this case, erroneous invariants could be extracted, yielding false matches. Thus, we shall examine the stability of the invariants to the input noise in this section. The invariants of "ve coplanar points, de"ned in Eq. (1), is the cross ratio of the areas of four triangles having one point in common. In fact, we can de"ne 30 di!erent invariants in total for "ve points, which are dependent to each other, since there are "ve choices for the common point and six possible combinations of triangles for each common point. Ideally, these are all invariant for the same points in di!erent views. However, the input noise inevitably disturbs the position of the feature points, yielding incorrect invariants. Thus, the matching perfor-
mance is signi"cantly a!ected by the error rate ER of an invariant for "ve coplanar points I, de"ned as
K K
ER"
*I , I
(6)
where *I is the error of I, caused by the input noise. In experiments, it is observed that for relatively small input noise, ER in Eq. (6) exhibits speci"c statistics, according to the geometric con"guration of the "ve points. More speci"cally, the error rates of the 30 invariants for "ve coplanar points are quite di!erent, depending on the structure of the points. The observation implies that the error rates of the invariants depend not only on the input noise, but also the geometric con"guration of the points. Thus, by investigating the relationship between the geometric con"guration of the "ve points and the stability of the invariants, we are able to choose the stable one. From the de"nition in Eq. (1), I in Eq. (6) can be expressed, in terms of the areas of four triangles, given by S S I" 1 2. S S 3 4 By di!erentiating Eq. (7), we obtain
(7)
S S S S S S LI" 2 LS # 1 LS ! 1 2 LS ! 1 2 LS . (8) 1 S S 2 S2S 3 S S2 4 S S 3 4 3 4 3 4 3 4 Thus, the error rate of the invariant I is approximated to
K
K
*S *S *S *S 3# 4! 1! 2 , (9) S S S S 3 4 1 2 indicating that the error rate ER is determined by each area error rate *S /S of four triangles. However, the area i i of the triangle is given by ER +
S "1 a b sin h , (10) i 2 i i i where a and b are the lengths of two borders joined with i i the angle h . Di!erentiation of S yields i i LS "1 (b sin h La #a sin h Lb #a b cos h Lh ). (11) i 2 i i i i i i i i i i Thus, the area error rate can be approximated to *S *a *b *h i + i# i# i . (12) S a b tan h i i i i Note that the variation of the angle h , *h , is related with i i the length of the borders as shown in Fig. 3, yielding the following relation: *a *b *h + i# i. (13) i a b i i From Eqs. (12) and (13), the area error rate is rewritten as
A
BA
*S *b *a i+ i# i S a b i i i
B
1 1# . tan h i
(14)
B.S. Song et al. / Pattern Recognition 33 (2000) 413}425
417
Fig. 3. Variation of border length and angle in triangle by noise.
Table 1 Comparison of error rates to the noise Noise (p2)
Average error rate
Minimum error rate
Error rate for the selected invariants
0.5 1.0 2.0 3.0 4.0
0.086 0.144 0.339 0.519 0.796
0.015 0.022 0.031 0.039 0.045
0.017 0.025 0.035 0.045 0.053
Thus, Eq. (14) provides an important information to choose the stable one among the possible 30 invariants. More speci"cally, we can choose the stable invariant, by comparing the following measure for the four triangles which compose each invariant:
A
BA
B
4 1 1 1 + # 1# . (15) a b tan h i i i/1 i In other words, for "ve coplanar points, we select two invariants which yield minimum value of Eq. (15) among 30 invariants. Table 1 shows the results of computer simulation. In experiments, "ve points are randomly generated in the X> plane in the range from (0, 0) to (100, 100). For one set of points, a Gaussian noise is added to each pixel, and the error rate is calculated for all 30 invariants of "ve coplanar points. But, notice that the results in the table are the averages of error rates for 10,000 sets of points. Without employing the proposed selection rule, probably one would expect to obtain the average error rate. However, even for small noise, it is observed that the average error rate is too high, compared to the minimum error rate. But, the error rate for the selected invariant is much lower than the average error rate, which is in fact very
Fig. 4. Comparison of the the matching speed (}e}: full search, }#}: circular ordered matching, }h}: proposed). (a) Number of the model points"6. (b) Number of the model points"7.
close to the minimum error rate. Thus, it is concluded that the stability of the invariants for "ve coplanar points to the input noise can be signi"cantly enhanced, by the proposed selection rule for the invariants.
4. Experiments To evaluate the performance of the proposed target recognition technique, various experiments are carried out both on the synthetic and real images. 4.1. Matching speed First, the improvement in the matching speed is examined. In our approach, additional geometric information, which is invariant under the projective transform, is utilized to reduce the search space, as described in
418
B.S. Song et al. / Pattern Recognition 33 (2000) 413}425
Fig. 5. Simulations on synthetic image. (a) Original Image. (b) Transformed Image. (c) Model 1. (d) Model 2. (e) Matched output.
B.S. Song et al. / Pattern Recognition 33 (2000) 413}425
419
Section 2. Three matching algorithms are considered: full search, match using the circular ordered matching, and the proposed technique employing both the circular ordered matching and compatibility test. The computational time required for each algorithm is compared as increasing the number of the input points. The simulation is performed on SUN Spark 20 workstation. The results are shown in Fig. 4, in which (a) represents the matching time when the number of model points is 6, and (b) shows the result when the number of model points is increased to 7. Notice that the X-axis represents the number of the input features, and the >-axis represents required matching time in log scale, respectively. As the number of the input features increases, matching time increases rapidly. However, the time required for the circular ordered matching technique is shown to be only 10% of that required for full search. The compatibility test depends on the speci"c information of the target. In the experiments, the size of the target
is restricted to half of full image. By combining the compatibility test, the computational time is further decreased to 1 & 1 , of those of the full search. 100 1000
Fig. 6. Comparison of the recognition rate with Gaussian noise (}e}: "xed invariants, }#}: selected invariants). (a) Model 1. (b) Model 2.
Fig. 7. Comparison of the recognition rate with uniform noise (}e}: "xed invariants, }#}: selected invariants). (a) Model 1. (b) Model 2.
4.2. Stability test on the synthetic image Next, how the proposed selection rule for the invariant a!ects the overall performance is examined on the synthetic image. As shown in Section 3, by employing the selection rule, we can choose the stable invariant of "ve coplanar points, which yields low error rate to the input noise. Fig. 5 shows the synthetic image used in the experiment. The original image, shown in Fig. 5(a), is transformed to other viewpoint, and the noise is added to each point, which is shown in Fig. 5(b). However, it is assumed that we already have the target model as shown in Figs. (c) and (d), respectively. Notice that model 1 in Fig. 5(c) has six points, and model 2 in Fig. 5(d) has seven points, respectively. The targets, corresponding to the models,
420
B.S. Song et al. / Pattern Recognition 33 (2000) 413}425
are correctly recognized as shown in Fig. 5(e). The matching is performed with the model, employing two invariant vectors, i.e., "xed invariants and selected invariants. Then, the results of best match, which yields the lowest error rate, are compared. Through the repeated 1000 trials, the recognition rates for two invariant vectors are obtained, which are presented in Figs. 6 and 7, respectively. Fig. 6 shows the result when the Gaussian noise is
added to the feature positions. Fig. 6(a) is the result for model 1, and Fig. 6(b) is the result for model 2, respectively. As the variance of the input noise increases, the recognition rate decreases. However, by employing the selected invariants, the recognition rate is improved signi"cantly. Comparing the results in Fig. 6(a) with (b), the improvement in the recognition rate is more pronounced for the case of 7 points, demonstrating the merit of the
Fig. 8. Target recognition in noisy environment. (a) Noisy input image. (b) Target model. (c) False match by "xed invariants. (d) Match by selected invariants.
B.S. Song et al. / Pattern Recognition 33 (2000) 413}425
Fig. 9. Comparison of recognition rate with increasing false points (}e}: selected invariants, }#}: "xed invariants).
421
proposed selection rule. That is, with more points, the geometry of the model becomes more complex, such as including more inner points, and an inadequate selection of the invariants may critically a!ect to the overall performance. With an adequate selection of the invariants, we can properly take advantage of the information provided by more points. When the uniform noise is added, as is expected, the results shown in Fig. 7 are similar to those for the Gaussian noise. In the feature extraction, many false features could be extracted or missed by inadequate threshold. The false features extracted by low threshold could be mismatched with the model. Fig. 8 shows an example. Fig. 8(a) is an input image, which is obtained by projecting the original image shown in Fig. 5(a) and disturbing by small positional noise and several false corner points. The target model is shown in Fig. 8(b). When the "xed invariants are used, it fails to recognize the correct target as shown in
Fig. 10. Target recognition employing the invariants of "ve coplanar points. (a) Input image. (b) Target model. (c) Corner detection. (d) Result.
422
B.S. Song et al. / Pattern Recognition 33 (2000) 413}425
Fig. 8(c). However, by employing the selected invariants, desirable result shown in Fig. 8(d) is obtained. Next, we attempt to evaluate the performance by adding more false point features to the input shown in Fig. 8. Through repeated 1000 trials with random false points, the recognition rate by employing the selected invariants is compared with that by using "xed invariants, as shown in Fig. 9. Notice that the positions of original points are also disturbed by uniform noise. When the "xed invariants are used, the performance is seriously degraded as the number of false points increases. But, the selected invariants yield high recognition rate, even for many false points. 4.3. Experiments on real images The proposed target recognition technique is also evaluated on real images. Figs. 10 and 11 show examples
of the target recognition for the complex image. For the extraction of corners, Susan corner "nder [11] is used, which employs nonlinear "ltering for noise resistant fast extraction of corners. To extract the linear segments, Nevatia's linear feature extraction method [12] is used in this paper. Fig. 10(a) is the input image, and Fig. 10(b) is the target model. The model is composed of the corner points of the target seen from the top. From the input image, the corner points are extracted as shown in Fig. 10(c). Finally, it is matched with the target model, employing the invariant vector for "ve coplanar points, and the correct target is determined as shown in Fig. 10(d). Note that the view-points are di!erent, and we can observe many false corners as shown in Fig. 10(c). Moreover, the positions of corners are incorrectly extracted, due to the input noise. However, it is seen that the desired target is correctly recognized by the proposed technique.
Fig. 11. Target recognition employing the invariants of "ve coplanar lines. (a) Input image. (b) Target model. (c) Line extraction. (d) Result.
B.S. Song et al. / Pattern Recognition 33 (2000) 413}425
An example of the target recognition, based on the invariants of "ve coplanar lines, is also shown in Fig. 11. The target model is composed of linear segments as shown in Fig. 11(b), which is observed from the top of the target. From the input image, the linear segments are
423
extracted as shown in Fig. 11(c), in which many false linear segments are observed. But, the target is correctly recognized as shown in Fig. 11(d). Finally, Fig. 12 demonstrates the limitation of the proposed technique, by showing the incorrect match.
Fig. 12. Example of target mismatch. (a) Input image (b) Target model. (c) Corner detection. (d) Magni"ed image of (c). (e) Result.
424
B.S. Song et al. / Pattern Recognition 33 (2000) 413}425
Fig. 12(a) is the input image, where the desired target, corresponding to the target model in Fig. 12(b), is indicated by an ellipse. From the input image, the corner points are extracted as shown in Fig. 12(c). As you can see in the magni"ed image of Fig. 12(d), one of the corner points in the target is missed, unfortunately. Thus, the proposed technique fails to provide the correct match as shown in Fig. 12(e). Though the stability of the invariants are improved by the proposed technique, it cannot provide desirable target matching, if the position of any feature is severely disturbed or missed in the feature extraction. Thus, for proper processing of the proposed technique, it is required that the features composing the target should be correctly extracted from the input image.
5. Conclusion In this paper, we have investigated the problems related to the target recognition, based on the geometric invariants of "ve coplanar points and "ve coplanar lines. To alleviate the enormous computational burden, e$cient matching technique using the compatibility test and the circular ordered matching was proposed. In addition, the behavior of the invariants for "ve coplanar points to the small disturbance of the feature points was investigated, and the selection rule to choose the stable invariant was also suggested. Experimental results showed the superb performance of the proposed technique. The circular ordered matching, combined with the compatibility test, was shown to enhance the matching speed considerably. And, by using the selection rule for the invariants of "ve coplanar points, very high recognition rates are achieved, even if the input noise is added to the feature position and many false points are included in the set of the input features. However, the proposed algorithm has a few limitations. First, only coplanar features can be utilized for the recognition, which is inadequate for the recognition of general 3D objects. Second, the target would be recognized incorrectly, if any of the features composing the target are missed, due to imperfect feature extraction. Thus, for more #exible object recognition, it is necessary to study the invariant-based matching technique for both 3D features and insu$cient set of features, which is our future research problems.
6. Summary In this paper, we propose an e$cient target recognition technique in the complex scenes. For the viewpoint invariant matching, the well-known geometric invariants, the invariants of "ve coplanar points and the invariants of "ve coplanar lines, which are invariant under the projective transform, are employed.
Target recognition is a part of object recognition, in which attempts are made to localize the object, corresponding to the desired target model, in the complex input image. To solve the problem, we extract the feature points or line segments from the input image and determine acceptable group of features, by matching them with those of the model, employing the invariants of "ve coplanar points and "ve coplanar lines. However, in this approach, one would experience two major problems. First, since all combinatorial candidates among the input features should be compared with the model, the search space becomes enormous, requiring serious computational load. Next, the employed invariants are known to be sensitive to the input noise, which may result in false match in noisy environment. Thus, in this paper, attempts are made to develop an e$cient target recognition technique, which is fast in target matching and stable to the input noise. To improve the matching e$ciency, we propose the circular ordered matching algorithm, utilizing the geometric information of the model which is preserved under the projective transform. The proposed algorithm is composed of three steps; compatibility test, geometry analysis, and circular matching. In the compatibility test, relations among the input features are examined, forming the compatibility table, which is then used for pruning the search space. For an example, if any two points are too far to be parts of the desired target, all candidate sets including both points can be excluded from the search space using the compatibility test. The geometry analysis attempts to "nd the preserved structural properties of the features; Any set of points are uniquely divided into two groups, outer points forming the convex polygon and inner points. Under the projective transform, these structures are preserved for the same points. Moreover, for the convex polygon, the circulation order of the points is also preserved under the transform. Thus, the features are re-ordered circularly, and circular matching is equivalent to full comparison. Thus, the searching space can be amazingly diminished, requiring acceptable computational load. On the other hand, the stability of the invariants to the input noise is essential to employ the algorithm in real environment. To achieve this goal, we investigate the sensibility of the invariants for "ve coplanar points to the input noise, which is shown to be dependent on the geometric structure of "ve points. Through the investigation, the selection rule for the stable invariant is suggested. The performance of the proposed algorithm is evaluated on both various synthetic and real images. The results show that using the proposed techniques, we can alleviate the computational complexity of the matching process signi"cantly, compared to the conventional approaches. Moreover, by employing the selected invariants, the improvement in the recognition rate is shown to be pronounced in the noisy environment.
B.S. Song et al. / Pattern Recognition 33 (2000) 413}425
References [1] R.O. Duda, P.E. Hart, Pattern Classi"cation and Scene Analysis, Wiley, New York, 1973. [2] R.M. Haralick, Using perspective transform in scene analysis, Comput. Graphics, Image Process. 13 (1980) 191}221. [3] E. Lutton, H. Maitre, J. Lopez-Krahe, Contribution to the determination of vanishing points using Hough transform, IEEE Trans. Pattern Anal. Mach. Intell. 16 (4) (1994) 430}483. [4] J.L. Mundy, A. Zisserman, Geometric Invariance in Computer Vision, The MIT Press, Cambridge, 1992. [5] D.A. Forsyth, J.L. Mundy, A.P. Zisserman, C. Coelho, A. Heller, C.A. Rothwell, Invariant descriptors for 3-D object recognition and pose, IEEE Trans. Pattern Anal. Mach. Intell. 13 (10) (1991) 971}991. [6] S. Carlsson, Projectively invariant decomposition and recognition of planar shapes, Int. J. Comput. Vision 17 (2) (1996) 193}209.
425
[7] I. Weiss, Noise-resistant invariants of curves, IEEE Trans. Pattern Anal. Mach. Intell. 15 (9) (1993) 943}948. [8] I. Weiss, Geometric invariants and object recognition, Int. J. Comput. Vision 10 (3) (1993) 207}231. [9] E.B. Barrett, P.M. Payton, N.N. Haag, M.H. Brill, General methods for determining projective invariants in imagery, CVGIP: Image Understanding 53 (1) (1991) 46}65. [10] C.A. Rothwell, A. Zisserman, J.L. Mundy, D.A. Forsyth, E$cient model library access by projective invariant indexing function, Proceedings of IEEE Conference on computer Vision and Pattern Recognition 1992, pp. 109}114. [11] S.M. Smith, M. Brady, Susan } a new approach to low level image processing, Int. J. Comput. Vision 23 (1) (1997) 45}78. [12] R. Nevatia, K.R. Babu, Linear feature extraction and description, Comput. Graphics Image Process. 13 (1980) 257}269.
About the Author*BONG SEOP SONG received the B.S. and M.S. degrees in electrical engineering from Seoul National University, Seoul, Korea, in 1992 and 1994, respectively. He is currently working toward the Ph.D. degree in computer vision with the School of Electrical Engineering, Seoul National University. His research interests are on the model-based object recognition and the applications of invariance in computer vision. About the Author*IL DONG YUN received the B.S., M.S., and Ph.D. degrees from Seoul National University, Seoul, Korea, in 1989, 1991, and 1996, respectively. He is currently a associate professor of the School of Electronics and Control Engineering, Hankuk University of F.S., Yongin, Korea. In 1996}1997, he was a senior researcher in the Daewoo Electronics. His current research interests are content based image retrieving, multi-resolution surface modeling and 3-D computer vision, especially in surface re#ection mechanism, photometric stereo. About the Author*SANG UK LEE received the B.S. degree from Seoul National University, Seoul, Korea, in 1973, the M.S. degree from Iowa State University, Ames, in 1976, and Ph.D. degree from the University of Southern California, Los Angeles, in 1980, all in electrical engineering. In 1980}1981, he was with the General Electric Company, Lynchburg, VA, and worked on the development of digital mobile radio. In 1981}1983, he was a Member of Technical Sta!, M/A-COM Research Center, Rockvill, MD. In 1983, he joined the Department of Control and Instrumentation Engineering at Seoul National University as an Assistant Professor, where he is now a Professor of School of Electrical Engineering. Currently, he is also a$liated with the Institute of New Media and Communications in Seoul National University. His current research interests are in the areas of the image and video signal processing, digital communication, and computer vision. He served as an Editor-in-Chief for the Transaction of the Korean Institute of Communication Science from 1994 to 1996. Currently, he is a member of the editorial board of the Journal of Visual Communication and Image Representation and an Associate Editor for IEEE Transaction on Circuit and Systems for Video Technology. He is a member of Phi Kappa Phi.
Pattern Recognition 33 (2000) 427}444
A model-based neural network for edge characterization Hau-San Wong!, Terry Caelli", Ling Guan!,* !School of Electrical and Information Engineering, The University of Sydney, Sydney, NSW 2006, Australia "Center for Mapping, The Ohio State University, Columbus, OH 43212, USA Received 26 August 1998; accepted 11 March 1999
Abstract In this paper, we investigate the feasibility of characterizing signi"cant image features using model-based neural network with modular architecture. Instead of employing traditional mathematical models for characterization, we ask human users to select what they regard as signi"cant features on an image, and then incorporate these selected features directly as training examples for the network. As a "rst step, we consider the problem of the characterization of edges, which are usually regarded as signi"cant image features by humans. Unlike conventional edge detection schemes where decision thresholds have to be speci"ed, the current NN-based edge characterization scheme implicitly represents these decision parameters in the form of network weights which are updated during the training process, and which thus allow automatic generation of the "nal binary edge map without further parameter adjustments. Experiments have con"rmed that the resulting network is capable of generalizing this previously acquired knowledge to identify important edges in images not included in the training set. In particular, one of the important attributes characterizing the current approach is its robustness against noise contaminations: the network can be directly applied to noisy images without any re-training and alteration of architecture, as opposed to conventional edge detection algorithms where re-adjustment of the threshold parameters are usually required. ( 2000 Pattern Recognition Society. Published by Elsevier Science Ltd. All rights reserved. Keywords: Image processing; Computer vision; Edge detection; Neural networks; Unsupervised learning
1. Introduction In this paper, we investigate the feasibility of employing human-de"ned features as training inputs for neural networks specially designed for feature detection. Speci"cally, we have investigated the e$ciency of a model-based network [1}4] in acquiring accurate characterization of what humans regard as features through a human}computer interaction (HCI) approach, and in generalizing this acquired knowledge to novel features which are equally signi"cant but have not been explicitly speci"ed in the training data.
* Corresponding author. Tel.: #61-2-9351-3131; fax: #612-9351-3847. E-mail addresses:
[email protected] (H-S. Wong),
[email protected] (T. Caelli),
[email protected] (L. Guan)
Edge characterization constitutes an important subbranch of feature extraction where the primary aim is to detect those pixels in an image with speci"c types of changes in intensities [5]. In view of this, the process is usually divided into two stages: one, the detection of speci"c types of change; two, a decision about whether they are to be accepted as relevant features to a given problem domain. Typical examples of di!erence measure used in the "rst stage include the Roberts operator, the Prewitt operator and the Sobel operator [5]. The second stage is de"ned by a decision process on the resulting edge magnitudes. This work is particularly concerned with this latter aspect of edge detection. In simple edge detection, a global threshold on the edge magnitudes is usually interactively chosen to produce a binary image specifying the edge locations. This result is usually not satisfactory due to the overlapping of the edge magnitude ranges of important features and strong background noise. More sophisticated
0031-3203/00/$20.00 ( 2000 Pattern Recognition Society. Published by Elsevier Science Ltd. All rights reserved. PII: S 0 0 3 1 - 3 2 0 3 ( 9 9 ) 0 0 0 8 8 - 6
428
H-S. Wong et al. / Pattern Recognition 33 (2000) 427}444
approaches, including the Canny edge detector [6] and the Shen}Castan edge detector [7], employ an expanded threshold set in the so-called hysteresis thresholding operation, where a signi"cant edge is de"ned as a connected series of pixels with the edge magnitude of at least one member exceeding an upper threshold, and with the magnitudes of the other members exceeding a lower threshold. In addition, the requirement for exact edge locations usually requires some form of Laplacian of Gaussian (LoG) "ltering [8] to detect the zero crossings, or analogous "ltering operations in which more parameters have to be speci"ed to determine the proper image resolution at which the edge detection operation is to take place. This, together with the previous thresholding parameters, gives rise to a large variety of possible parameter combinations, each of which will result in a very di!erent appearance for the "nal edge map. In this paper we have explored a neural computing approach to estimating such parameters to "t human performance. It is our anticipation that still more parameters will be required for future edge-detection operations. In view of this, a logical choice for a proper representation of these parameters would be in the form of connection weights for a neural network. To do this, we have developed a model-based neural network for edge characterization which is based on the modular architecture proposed by Kung and Taur [2]. In this case the connection weights of the network encode both the edge-modeling parameters in the high-pass "ltering stage and the thresholding parameters in the decision stage. In other words, the input to the network is the original gray-level image, and the output is a binary image which speci"es the location of the detected edges. This is in contrast with previous uses of NN for edge detection [9}11], where the primary motivation is to generalize the traditional di!erencing operators by replacing them with NN. As a result, the emphasis of their approaches is on the learning of the "lter coe$cients in the "rst "ltering stage, and the thresholds for the "nal decision stage are usually chosen heuristically.
2. Network architecture The proposed architecture is composed of a hierarchical structure consisting of clusters of neurons forming sub-networks, with each neuron of a sub-network encoding a particular subset of the training samples. In the training stage, each sub-network encodes di!erent aspects of the training set, and in the recognition stage an arbitration process is applied to the outputs of the various sub-networks to produce a "nal decision. The motivation for using this architecture is that, in edge detection, it would be more natural to adopt multiple sets of thresholding decision parameters and apply the appropriate set of parameters as a function of the
Fig. 1. The architecture of the model-based neural network edge detector.
local context, instead of just adopting a single set of parameters across the whole image as in traditional approaches. The modular decision-based architecture thus constitutes a natural representation of the above adaptive decision process if we designate each sub-network to represent a di!erent background illumination level, and each unit in the sub-network to represent di!erent prototypes of edge-like features under the corresponding illumination level. The architecture of the feature detection network is shown in Fig. 1. The internal representation scheme for edge information at the various hierarchical levels are explained below. 2.1. Characterization of edge information For illustrative purposes we consider a 3]3 moving window over the image. Representing the gray-level values in the window W as a vector x"[x 2x ]T3R9, 1 9 and de"ning the mean of these values as follows: 1 9 (1) x6 " + x , i 9 i/1 we summarize the gray-level information in the window in terms of a vector m"[m , m ]T3R2, which character1 2 izes the two dominant gray-level values within the window and is de"ned as follows: M "Mx : x (x6 N, 1 i i M "Mx : x *x6 N, 2 i i 1 m " + x, 1 DM D M i 1 xi| 1 1 m " + x, 2 DM D M i 2 xi| 2 m #m 2. m6 " 1 2
(2) (3) (4) (5) (6)
H-S. Wong et al. / Pattern Recognition 33 (2000) 427}444
2.2. Sub-network G
429
r
We associate each sub-network G , r"1,2, k with r a prototype background illumination gray-level value g r (the choice of the appropriate number of sub-networks k will be discussed in Section 4.3). We then assign all those 3]3 windows in the image with their background gray-level values closest to g to the sub-network G . r r Speci"cally, a particular window W in the image, with its associated parameters m6 , m , m de"ned in Eqs. (2)}(6), is 1 2 assigned to the sub-network G H if the following condir tions are satis"ed: g H3[m , m ], (7) r 1 2 Dm6 !g HD(Dm6 !g D, r"1,2, k, rOrH, (8) r r where [m , m ] is the closed interval with m , m as its 1 2 1 2 endpoints (this is in contrast with the column vector representation of m, which we denote as [m m ]T). In 1 2 this way, we partition the set of all 3]3 windows in the image into clusters, with all members in a single cluster exhibiting similar levels of background illumination. Hereafter, we denote the conditions in Eqs. (7) and (8) collectively as xPG H. The operation of this sub-network r assignment process is illustrated in Fig. 1.
Fig. 2. The architecture of a single sub-network G . r
2.3. Neuron N in sub-network G rs r Each sub-network G contains n neurons N , s" r rs 1,2, n, with each neuron encoding the various di!erent edge prototypes which can exist under the general illumination level g . We associate each neuron with a r weight vector w "[w w ]T3R2, which serves as rs rs,1 rs,2 a prototype for the vector m characterizing the two dominant gray-level values in each 3]3 window W. We assign a certain window with corresponding vector m to the neuron N H if the following condition is satis"ed: rs Em!w HE(Em!w E, s"1,2, n, sOsH. (9) rs rs In this work, we have chosen n"2 in order that one of the neurons encodes the prototype for weak edges and the other encodes the prototype for strong edges. We designate one of the weight vectors w , s"1, 2 as the rs weak-edge prototype wl and the other one as the strongr edge prototype wu according to the following criteria: r f wl "w l, where s "arg min (w , !w ). r rs l s rs,2 rs,1 f wu"w u, where s "arg max (w !w ). r rs u s rs,2 rs,1 Given the weak-edge prototype wl "[wl wl ]T, the r r,1 r,2 measure (wl !wl ) plays a similar role as the threshold r,2 r,1 parameter in conventional edge-detection algorithms in specifying the lower limit of visibility for edges, and is useful for identifying potential starting points in the image for edge tracing. The operation of the neuron assignment process is illustrated in Fig. 2.
Fig. 3. The structure of the dynamic edge-tracking neuron.
2.4. Dynamic tracking neuron N d Independent of the neuron clusters G and N , we r rs associated a dynamic tracking neuron N with the netd work. This neuron is global in nature in that it does not belong to any of the sub-networks G , and that all of the r sub-networks can interact with this particular neuron. Similar to the neurons N in each sub-network G , the rs r dynamic neuron consists of a dynamic weight vector w "[w w ]T3R2. In addition, it consists of a scalar d d,1 d,2 parameter g which is analogous to the illumination level d indicator g of each sub-network. Thus, this particular r neuron takes on both the characteristics of a subnetwork and a neuron. The structure of the dynamic tracking neuron is shown in Fig. 3. The primary purpose of this neuron is to accommodate the continuously varying characteristics of an edge as it is being traced out in the recognition phase. As opposed to the usual practice of varying the neuronal weights during the training phase, this neuron is inactive during
430
H-S. Wong et al. / Pattern Recognition 33 (2000) 427}444
the acquisition of the edge con"gurations. It is only during the recognition phase and upon the detection of a primary edge point, which is a comparatively prominent image feature location (the exact de"nition is given in a later section), that we dynamically vary the weight vector w and illumination level indicator g of this neurd d on to trace out all those secondary edge points connected to the primary edge point. The detailed operation of this neuron is described in a later section. 2.5. Binary edge conxguration Suppose that the vector m for the current window W is assigned to neuron N with weight vector w . We can rs rs de"ne the function Q : R9]R2PB9, where B"M0, 1N, which maps the real vector x3R9 representing the graylevel values of the current window to a binary vector b"Q(x, w )"[q(x , w )2q(x , w )]T3B9, in terms of rs 1 rs 9 rs the component mapping q : R]R2PB as follows:
G
0 if Dx !w D(Dx !w D, i rs,1 i rs,2 q(x , W )" i rs 1 if Dx !w D*Dx !w D. i rs,1 i rs,2
(10)
The binary vector b assumes a special form for valid edge con"gurations. Some of the possible valid edge con"gurations are shown in Fig. 4. During the network training phase, we acquire all the human-speci"ed valid edge con"guration patterns in a process to be described in a later section. We will store all these patterns in an edge conxguration set C which forms part of the overall network. A distinguishing feature of the set C is that it is closed under an operation R , p@4 which permutes the entries of a vector b3B9 in such a way that, when interpreted as the entries in a 3]3 window, R (b) is the 453 clockwise rotated version of b. p@4 This operation is illustrated in Fig. 5. The purpose of imposing this structure on C is that the edge con"guration in the training image may not exhibit all of its possible rotated versions, so that the above constraint can facilitate the detection of those rotated edge con"gurations not present in the training set.
3. Network training The training of the network proceeds in three stages: in the "rst pass, we determine the prototype illumination
Fig. 5. Illustrating the operation R . p@4
level g , r"1,2, k for each sub-network G by competir r tive learning [12,13]. In the second stage, we assign each window W to its corresponding subnetwork and then determine the weight vectors w , again using competitive rs learning. In the third stage, we assign each window to its corresponding neuron in the correct sub-network, and extract the corresponding binary edge con"guration pattern b as a function of the winning weight vector w . We rs apply the operation R successively to b to obtain the p@4 eight rotated versions of this pattern, and insert these patterns into the edge con"guration memory C. 3.1. Determination of g for sub-network G r r Assuming that the current window with associated parameter m6 is assigned to the sub-network G according r to the conditions in Eqs. (7) and (8). Using competitive learning, we update the value of g using the following r equation: g (t#1)"g (t)#a(t) (m6 !g (t)). r r r
(11)
The learning stepsize a(t) is successively decreased according to the following linear schedule
A B
t a(t#1)"a(0) 1! , t f
(12)
where t is the total number of iterations. f 3.2. Determination of w for neuron N rs rs In the second stage, we assume that the current window with associated feature vector m has been assigned to the neuron N within the sub-network G . Again, rs r employing competitive learning, we can update the current weight vector w using the following equation rs w (t#1)"w (t)#a(t)(m!w (t)), rs rs rs
(13)
where the stepsize a(t) is successively decreased according to Eq. (12). 3.3. Acquisition of valid edge conxgurations
Fig. 4. Examples of valid edge con"gurations.
In the third stage, after determining the background illumination level g prototypes for the sub-networks and r
H-S. Wong et al. / Pattern Recognition 33 (2000) 427}444
the weight vectors w for all the neurons, we once again rs assign all the 3]3 windows to their corresponding sub-networks and the correct neurons within the subnetworks. Assuming that the current window is assigned to neuron N within sub-network G . We can transform rs r the current window W with gray-level vector x3R9 into a binary edge con"guration vector b3B9 according to Eq. (10): b"Q(x, w ), rs
(14)
where w is the associated weight vector of N . The rs rs binary vector b is thus a human-specixed valid edge conxguration and is inserted into the valid edge con"guration set C. To satisfy the requirement that the set C be closed under the operation R , we generate eight edge p@4 con"gurations b , j"0,2, 7 using R as follows: j p@4
(p1) (p2) (p3)
431
xPG refer to conditions (7) and (8) for some r. r m !m *wl !wl , where wl is the weak-edge 2 1 r,2 r,1 r prototype vector of G . r b"Q(x, w )3C, where w is the weight vector rs rs associated with the assigned neuron for x.
Condition (p1) speci"es that the background illumination level value of the current window should match one of those levels associated with a sub-network. Condition (p2) ensures that the magnitude of gray-level variation, which is characterized by the di!erence m !m , is 2 1 greater than the corresponding quantity of the weakedge prototype of the assigned subnetwork. Condition (p3) ensures that the binary edge con"guration corresponding to the current window is included in the edge con"guration memory, C, of the network. 4.2. Location of secondary edge points
b "b, 0
(15)
b "R (b ), j"0,2, 6 j`1 p@4 j
(16)
and insert all eight patterns into the con"guration set C.
4. Recognition phase In the recognition phase, the network is required to locate all those pixels in a test image with similar properties as the acquired feature prototypes } thus testing the generalization capability of the network. This recognition phase proceeds in two stages. In the "rst stage. all pixels in the test image with high degree of conformance with the acquired edge prototypes are designated as primary edge points. This is analogous to the stage of hysteresis thresholding in conventional edge-detection algorithms where we locate those edge points with their edge magnitudes greater than an upper threshold. In the second stage, starting from the high conformance edge points, we apply a recursive edgetracing algorithm to locate all those pixels connected to these edge points satisfying a relaxed conformance criterion, which are the secondary edge points. This is analogous to the hysteresis thresholding operation where we specify a lower threshold for all those edge points connected to those points with edge magnitudes above the upper threshold. 4.1. Location of primary edge points In this "rst stage, we inspect all 3]3 windows in the test image and designate all those windows with corresponding parameters m6 , m , m as primary edge points, if 1 2 the following conditions are satis"ed:
At this stage, we utilize the dynamic tracking neuron N associated with the network to trace the secondary d edge points connected to a primary edge point. The weights of the tracking neuron are initialized using the relevant parameters mp , mp and m6 p of the current window 1 2 W containing the detected primary edge point: w (0)"mp , d,1 1
(17)
w (0)"mp , d,2 2
(18)
g (0)"m6 p, d
(19)
where w "[w w ]T is the weight vector of the dyd d,1 d,2 namic neuron and g is a local illumination level indid cator which keeps track of the varying gray-level values as the current edge is being traced. As mentioned above, both of them are dynamic quantities such that their values are continually updated during the recognition phase rather than the training phase. We have adopted this approach since during the tracing of an edge, although the primary edge point and its associated secondary edge points may reasonably conform to a particular edge prototype in the trained network, there will inevitably be deviations, especially when the network is applied to a test image which it has not been trained on before. In these cases, the weights of the dynamic neuron will serve as a specialized edge prototype as distinguished from the general edge prototypes represented by the sub-networks and neurons of the original network. This is especially important for the accurate conversion of the gray-level values x of the current window W into a binary edge con"guration b to be matched against the valid con"gurations in the edge pattern memory C. After the proper initialization of the dynamic neuron, we apply a recursive edge-tracing algorithm to locate all
432
H-S. Wong et al. / Pattern Recognition 33 (2000) 427}444
those pixels connected to the primary edge points by recursively checking a speci"c set of secondary edge validation criteria at each 8-neighbor of the current primary edge point, and designating those satisfying the criteria as secondary edge points. For a particular primary edge point, we will locate all its associated secondary edge points using the dynamic neuron. The following set of conditions are speci"ed for the validation of the secondary edge points. (s1) (s2)
b"Q(x, w )3C, where w is the weight vector asd d sociated with the dynamic neuron. g 3[m , m ], where g is the local illumination d 1 2 d level indicator associated with the dynamic neuron.
Condition (s1) is similar to the case for primary edgepoint detection, where we match the corresponding binary edge con"guration of the current window with those in the edge con"guration memory C. Condition (s2) is a modi"ed version of condition (7) in the requirements for xPG , which ensures that a potential secondary edge r point should be under similar levels of background illumination levels (as summarized by the parameters of the dynamic neuron) as those points in the portion of edge already traced out. Notice that we have allowed the possibility of weak secondary edge points by omitting the constraints on edge magnitude (Condition (p2) for primary edge detection), as long as the points are connected to a primary edge point. For each validated secondary edge point, we update the local illumination level indicator g of the dynamic d neuron using the mean value m6 s of the corresponding window W: g (t#1)"g (t)#a(0)(m6 s!g (t)). (20) d d d In addition, if the secondary edge point satis"es condition (p2) for primary edge point detection } implying that the edge strength of the current point is comparable with that of a primary edge point, we utilize the vector ms of the corresponding window to update the weight vector of the dynamic neuron w (t#1)"w (t)#a(t)(ms!w (t)). (21) d d d The purpose of using validated secondary edge points to update g is to allow the dynamic neuron to closely d track the local gray-level values in order for condition (s2) (for secondary edge point detection) to be accurately detected. The decision to use a xxed learning step size a(0) as opposed to the usual decreasing stepsize sequence also re#ects this purpose. That is, a "xed stepsize allows the gradual decaying of the initial conditions during edge tracing such that the value of g will re#ect the most d current average gray-level values in the local neighborhood. On the other hand, the decision to use only those validated points which satisfy the edge magnitude
criterion (p2) for updating w is that, during the d edge-tracing process, we may encounter very weak edge points which are, nevertheless, validated through their satisfaction of the secondary edge point detection criteria (s1) and (s2). However, the vector ms extracted from the current window is unlikely to be representative of the characteristics of the current edge being traced, and thus should not be included in the updating of the local edge prototype encoded in the dynamic weight vector w . The learning stepsize a(t) in this d case is thus allowed to decrease gradually such that the weight vector still partially retains the characteristics of the initial primary edge point from which we start the tracing. 4.3. Determination of the network size We recall that each sub-network G is characterized r by a weak edge prototype vector wl , and a strong edge r prototype vector wu. The corresponding component difr ferences wl !wl and wu !wu thus re#ect the range r,2 r,1 r,2 r,1 of edge magnitudes covered by the current sub-network. We de"ne the following average prototype vector wN r"[w6 w6 ]T3R2 as follows: r,1 r,2 wl #wu r. wN r, r (22) 2 The interval [w6 , w6 ] (as opposed to the vector notar,1 r,2 tion [w6 , w6 ]T) thus represents the average gray-level r,1 r,2 range covered by the current sub-network. For a parsimonious representation of the edges, we usually avoid excessive overlapping of these associated intervals of the sub-networks. According to the weight update Eq. (13), if we initialize w (0) such that g 3[w (0), w (0)], s" rs r rs,1 rs,2 1, 2 and if we update w (t) with those vectors m such that rs g 3[m , m ] as speci"ed in criterion xPG , then it is r 1 2 r obvious that g 3[w , w ], s"1, 2 for the "nal weight r rs,1 rs,2 vector w , and therefore g 3[w6 , w6 ] for the average rs r r,1 r,2 prototype vector wN r. In view of this, we have de"ned the notation of substantial overlap between intervals of the sub-networks as follows: for sub-network G , if there r exists one or more sub-networks G with r@Or, such that r{ the condition g 3[w6 , w6 ] is satis"ed as well, then we r r{,1 r{,2 say that there is substantial overlapping between the associated intervals of sub-networks G and G . r r{ Adopting the usual de"nition of the membership function for set A
G
1 if a 3 A, I (a)" A 0 if a N A,
(23)
we de"ne the interval overlap measure Ok for sub-network r G as follows: r k Ok" + I 6 r{,1 6 r{,2 (g ) (24) r *w ,w + r r{/1
H-S. Wong et al. / Pattern Recognition 33 (2000) 427}444
Fig. 6. (a) Eagle image. (b)}(e) Edge examples supplied by human user.
433
434
H-S. Wong et al. / Pattern Recognition 33 (2000) 427}444
Fig. 7. (a) Flower image. (b)}(e) Edge examples supplied by human user.
H-S. Wong et al. / Pattern Recognition 33 (2000) 427}444
Fig. 8. Edge tracings by human users in independent trials (solid curve: real edge, dashed curves: tracings by human users).
which essentially counts the number of sub-networks substantially overlapping with sub-network r (including itself ). For example, apart from satisfying the condition g 3[w6 , w6 ], if there is one other sub-network r@, r@Or r r,1 r,2 such that the condition g 3[w6 , w6 ] is satis"ed as r r{,1 r{,2 well, then Ok"2. r The mean interval overlap measure OM k is then de"ned as follows: 1 k OM k" + Ok. (25) r k r/1 In general, the values of these two measures depend on the number of sub-networks k in the network. According to these de"nitions, a natural criterion to adopt for the choice of k to avoid substantial overlapping between sub-networks, while ensuring adequate representation of the edges in the training images, is to select the maximum possible number of sub-networks k while subjecting to the contraint OM k(2 to avoid substantial interval overlapping. More precisely, we select kH such that kH"maxMk : OM k(2N.
(26)
5. Experimental results For network training, we have selected various edge examples from two images, a #ower and an eagle, and incorporated them into the training set. To increase the size of the training set, we have prepared multiple edge tracings by a single user for each of the two images. The eagle image is shown in Fig. 6(a) and the four independent tracings by a single user for this image are shown in Figs. 6(b)}(e). The corresponding image and tracings for the #ower image are shown in Figs. 7(a)}(e). In all cases the task was to `trace outa signi"cant edges in the image. From Figs. 6 and 7, it is evident that the preference of a single user is highly correlated: most of the preferred edge points are re-selected in all the four tracings, although there are variations between trials. Although it is not immediately obvious how these highly correlated
435
tracings can increase the training set size, this can be explained by referring to Fig. 8. In the "gure, the solid curve represents the `truea edge and the dashed curves represent the various di!erent tracings in independent trials. In most of the cases, we only expect the human user to trace the edge approximately, and so each of the independent trials produces tracings which intersect different portions of the true edge. As a result, each independent trial increases the number of training examples corresponding to locations on the true edge. One may argue that the multiple tracings also increase the number of non-edge points close to the `truea edge due to the inexact tracings, but the very nature of the current network in acquiring new edge con"gurations will ensure that most of these non-edge points will be automatically rejected: in order for a training edge point with its associated neighboring gray levels x to modify any weight vector under a certain subnetwork G , it has to satisfy the condition xPG , which r r requires the checking of the membership condition g 3[m , m ]. For a non-edge point, the length of the r 1 2 interval [m , m ] will be negligibly small, and it highly 1 2 improbable that any gray-level indicator g associated r with the sub-networks will be included in that interval, thus preventing this point from modifying any weight vector in the network. We "rst applied the NN-based edge characterization scheme to the images employed in the training set. This was done to test the performance of the network in generalizing the sparse tracings of the human users to identify the other edge-like features on the same image. We have applied condition (26) in selecting the number of sub-networks, which results in kH"5. The detected edges for the eagle image using the current method are shown in Fig. 9(a). A comparison of Fig. 9(a) with Figs. 6(b)}(e) demonstrates the generalization capability of the NN-based edge detector: starting just from the acquired features in Figs. 6(b)}(e) and Figs. 7(b)}(e), the network was able to locate other important edges in the eagle's image as perceived by humans. More importantly, the training process only requires a small number of images and a small number of edge examples within these images, as shown in Figs. 6 and 7. Although Fig. 9(a) seems to be a satisfactory caricature of the original image, the performance of the edge characterizer has to be validated further by comparing the result with that of standard edge detectors. The performance of current edge detectors is usually controlled by a set of tunable parameters, and we would expect that the result of the NN edge detector should correspond to the performance of those edge detectors under near-optimal settings of parameters. We have compared the performance of the NN edge detector with that of the Shen}Castan edge detector [7] in Figs. 9 and 10: the Shen}Castan edge detector can be
436
H-S. Wong et al. / Pattern Recognition 33 (2000) 427}444
Fig. 9. Edge-detection results for Eagle. (a) Detected edges using NN. (b)}(e) Detected edges using Shen}Castan edge detector with di!erent hysteresis thresholds t , t ("lter parameter a"0.3). (b) t "10, t "15. (c) t "20, t "25. (d) t "30, t "35. (e) t "40, 1 2 1 2 1 2 1 2 1 t "45. 2
H-S. Wong et al. / Pattern Recognition 33 (2000) 427}444
considered an improved version of the Canny edge detector [6], which is widely used as a comparison standard in edge-detection research. This edge detector employs an exponential "lter in the smoothing operation prior to the di!erencing operation, which, as claimed by the authors, provides a better degree of edge localization than the Gaussian "lter used in the Canny edge detector. The associated parameters of the Shen}Castan edge detector include the exponential "lter parameter a, which determines the degree of smoothing prior to edge detection, and the hysteresis thresholds t , t . There are a large 1 2 number of possible combinations between these parameters, with each corresponding to a very di!erent resulting edge pro"le. Fig. 9 compares the NN feature detector result with the Shen}Castan edge detector under various settings of the hysteresis thresholds, while "xing the "lter parameter a at 0.3. The result of the NN edge detector
437
is shown in Fig. 9(a) and the results for the Shen} Castan edge detector are shown in Figs. 9(b)}(e). The lower hysteresis threshold ranges from t "10}40, and 1 we have set the upper threshold t "t #5. In general, 2 1 the choice of this upper threshold t is not as critical as 2 the choice of the lower threshold t , as long as the 1 condition t 't is satis"ed. Therefore, one can also 2 1 replace the increment 5 with other convenient values, as long as it is not excessively large, without a!ecting the results greatly. From Figs. 9(b)}(e), we can observe that the detected edge pro"le is sensitive to the choice of t and t . In 1 2 general, lower values of t and t will reveal more details 1 2 but at the same time cause more false positive detections, as can be seen in Fig. 9(b). On the other hand, higher values of thresholds will lead to missed features as in Fig. 9(e). In our opinion, Fig. 9(c), with t "20 and 25, 1 constitutes an adequate representation of the underlying
Fig. 10. Edge-detection results for Eagle. (a)}(d) Detected edges using Shen}Castan edge detector with di!erent "lter parameters a (hysteresis thresholds t "20, t "25). (a) a"0.4. (b) a"0.5. (c) a"0.6. (d) a"0.7. 1 2
438
H-S. Wong et al. / Pattern Recognition 33 (2000) 427}444
Fig. 11. Edge-detection results for Flower. (a) Detected edges using NN. (b)}(e) Detected edges using Shen}Castan edge detector with di!erent hysteresis thresholds t , t ("lter parameter a"0.3). (b) t "10, t "15. (c) t "20, t "25. (d) t "30, t "35. (e) t "40, 1 2 1 2 1 2 1 2 1 t "45. 2
H-S. Wong et al. / Pattern Recognition 33 (2000) 427}444
features of the image. This can be compared with the result of the NN edge detector in Fig. 9(a). We can see that the features detected by the current approach are similar to those under the near optimal settings of the threshold parameters of a conventional edge detector. The important point is that the current approach directly acquires the appropriate parameter settings through the human-speci"ed features and no trial and error is required. In Fig. 10, we compare the performance of the NN feature detector with the Shen}Castan edge detector under di!erent settings of the "lter parameter a with "xed t and t . The results for the Shen}Castan edge detector 1 2 are shown in Figs. 10(a)}(d) and we can compare them with the previous NN result in Fig. 9(a). The settings for the "lter width a range from a"0.4}0.7. Although there is no corresponding concept of smoothing in the case of the NN detector, we can regard the setting of a as an
439
alternative means to control the detection rate of false positives: in general, decreasing a will lead to more false positives, and increasing a will remove those false positives but lead to missed features. In addition, varying a will lead to the gradual shifting of the edges due to the smoothing action of the "lter. We can observe all these e!ects as we increase a from Figs. 10(a)}(d). In particular, the shifting of the edges lend an unnatural appearance to the corresponding edge pro"les which is especially noticeable in Figs. 10(c) and (d). The adjustment of this parameter is again avoided in the NN edge detector which directly assimilates the characteristics of humande"ned features through the learning process. Comparison results for the #ower image are shown in Figs. 11 and 12, where Fig. 11 shows the e!ect of varying the thresholds t , t , and Fig. 12 shows the e!ect of 1 2 varying the "lter width a. The detected edges using the NN approach are shown in Fig. 11(a). For the
Fig. 12. Edge-detection results for Flower. (a)}(d) Detected edges using Shen}Castan edge detector with di!erent "lter parameters a (hysteresis thresholds t "20, t "25). (a) a"0.4. (b) a"0.5. (c) a"0.6. (d) a"0.7. 1 2
440
H-S. Wong et al. / Pattern Recognition 33 (2000) 427}444
Shen}Castan edge detection results, we may consider Fig. 11(c) (with thresholds t "20 and 25) as a near 1 optimal representation and Fig. 11(d) as an adequate representation. Comparing these results with Fig. 11(a), we notice that the quality of the NN detection results lies in between these two and thus can certainly be considered a close approximation to the optimal result. However, comparing Figs. 11(a) with (c), we see that the detected textures for NN are not as prominent as those in the Shen}Castan detection result. This can partly be attributed to the fact that no training examples have been taken from this area in Figs. 7(b)}(c) and no explicit texture modeling has been implemented in the current model, and partly due to the particular value of threshold used in Fig. 11(c) (we can compare this with the detected textures in Figs. 11(d) and (e) under di!erent thresholds). In general, we can expect the texture detection result to improve if we explicitly model the edges and textures as separate entities in the network [14].
On the other hand, we may consider Fig. 11(b) as over-cluttered with non-essential details, while missed features are clearly noticeable in Fig. 11(e). In Fig. 12, we notice the same e!ect of edge dislocation due to changes in the width parameter a. Again we emphasize that no process of trail and error in selecting parameters is required for the NN edge detector as compared with the Shen}Castan edge detector. We further evaluated the generalization performance of the NN edge detector by applying the network trained on the eagle and #ower features in Figs. 6 and 7 to some other images. In Fig. 13(a), image of a clock and other objects are shown, and Fig. 13(c) shows an image with natural scenery. The corresponding detection results are shown in Figs. 13(b) and (d), respectively. The results are very satisfactory, considering that the network is trained only on the eagle and #ower images. In addition, the preference of the current user in selecting more prominent edges as training examples, as shown in Figs. 6 and 7,
Fig. 13. (a) Clock image. (b) Detected edges using NN. (c) Natural scenery image. (d) Detected edges using NN.
H-S. Wong et al. / Pattern Recognition 33 (2000) 427}444
441
Fig. 14. (a) Eagle image with additive Gaussian noise (p "10). (b) Detected edges using NN. (c)}(d) Detected edges using Shen}Castan n edge detector with di!erent hysteresis thresholds t , t ("lter parameter a"0.3). (c) t "20, t "25, (d) t "25, t "30. 1 2 1 2 1 2
is also re#ected in some of the results. For example, in Fig. 13(d), the detection results mainly consist of the more prominent features such as the sailboat and the broad outline of trees, while less conspicuous features, such as the clouds and the textural patterns within the tree outlines, are omitted. We have also tested the robustness of the current approach against noise contaminations. Fig. 14(a) shows the addition of zero-mean Gaussian noise with a standard deviation of p "10 to the eagle image. We have n applied the same NN as in the previous noiseless case to this image without any re-training and alteration of architecture. The result is shown in Fig. 14(b) showing that although some false alarms occurred, the overall e!ect is not serious and the result is reasonably similar to the noiseless case. On the other hand, for the Shen} Castan edge detector, if we choose the previous optimal threshold of t "20 and t "25 (Fig. 14 (c)), the e!ect of 1 2
noise is clearly noticeable, and we will have to re-adjust the thresholds to t "25 and t "30 to eliminate its 1 2 e!ect (Fig. 14(d)). For the other three images, the results for the noisy case are shown in Figs. 15(b), (d) and (f). Again, notice that, due to the edge-modeling capability of the neural network, the e!ect of noise contamination is not signi"cant.
6. Conclusion We have developed a model-based feature detection neural network with modular architecture which directly assimilates the essential characteristics of human-speci"ed features through a learning process. The speci"c architecture of the network divides the training features into sub-classes in such a way as to re#ect the di!erent preferences of human beings in regarding intensity
442
H-S. Wong et al. / Pattern Recognition 33 (2000) 427}444
Fig. 15. (a) Flower image with additive Gaussian noise (p "10). (b) Detected edges using NN. (c) Clock image with additive Gaussian n noise (p "10). (d) Detected edges using NN. (e) Natural scenery image with additive Gaussian noise (p "10). (f ) Detected edges n n using NN.
H-S. Wong et al. / Pattern Recognition 33 (2000) 427}444
discontinuities as features under di!erent illumination conditions. Conventional edge-detection algorithms using NN mainly focus on generalizing the front-end "ltering operation while continuing to select the detection thresholds explicitly. The current NN-based edge detector implicitly represents these decision parameters in the form of network weights which are updated during the training process, and which thus allow automatic generation of the "nal binary edge map without further parameter adjustments. In addition, due to the speci"c architecture of the network, only positive training examples in the form of edge tracings on the image are required, unlike previous NN-based edge-detection schemes where both positive and negative examples are required. It is also important to note that, in contrast to current edge-extraction methods, this NN encapsulates the nonstationary nature of edges in so far as di!erent criteria apply to di!erent image regions as a function of the local intensity distributions. The di!erent rules which capture such di!erent criteria are learnt by the proposed NN. This NN edge detector has been successfully applied to both the image from which its training data originate and to novel images with promising results. In particular, no re-training of the network and no alteration of architecture are required for applying the network to noisy images.
References [1] T. Caelli, D. Squire, T. Wild, Model-based neural network, Neural Networks 6 (1993) 613}625.
443
[2] S.Y. Kung, J.S. Taur, Decision-based neural networks with signal/image classi"cation applications, IEEE Trans. Neural Networks 6 (1) (1995) 170}181. [3] R. Anand, K. Mehrotra, C.K. Mohan, S. Ranka, E$cient classi"cation for multiclass problems using modular neural networks, IEEE Trans. Neural Networks 6 (1) (1995) 117}124. [4] S.H. Lin, S.Y. Kung, L.J. Lin, Face recognition/detection by probabilistic decision-based neural networks, IEEE Trans. Neural Networks 8 (1) (1997) 114}132. [5] R.C. Gonzalez, R. Woods, Digital Image Processing, Addison-Wesley, Reading, MA, 1992. [6] J. Canny, A computational approach to edge detection, IEEE Trans. Pattern Anal. Mach. Intell. 8 (6) (1986) 679}698. [7] J.J. Shen, S.S. Castan, An optimal linear operator for step edge detection, CVGIP: Graphical Models and Image Processing 54 (2) (1992) 112}133. [8] D. Marr, E. Hildreth, Theory of edge detection, Proc. Royal Soc. B-207 (1980) 187}217. [9] J.C. Bezdek, D. Kerr, Training edge detecting neural networks with model-based examples, Proceedings of the Third IEEE International Conference Fuzzy Systems, Piscataway, NJ, 1994, pp. 894}901. [10] J.C. Bezdek, R. Chandrasekhar, Y. Attikiouzel, A geometric approach to edge detection, IEEE Trans. Fuzzy Systems 6 (1) (1998) 52}75. [11] V. Srinivasan, Edge detection using neural networks, Pattern Recognition 27 (12) (1994) 1653}1662. [12] S. Haykin, Neural Networks: A Comprehensive Foundation, Macmillan, New York, 1994. [13] T. Kohonen, Self-Organizing Maps, 2nd Edition, Springer, Berlin, 1997. [14] H.S. Wong, L. Guan, Adaptive regularization in image restoration using evolutionary programming, Proceedings of the IEEE International Conference on Evolutionary Computation, 1998, pp. 159}164.
About the Author*HAU-SAN WONG received the B.Sc. degree and the M. Phil. degree in Electronic Engineering from the Chinese University of Hong Kong, in 1991 and 1993, respectively. He was a research assistant in the Department of Mechanical and Automation Engineering of the same university in 1994. Since 1995, he has been a postgraduate student in the Department of Electrical Engineering, the University of Sydney, Australia, where he is completing his Ph.D. degree. His research interests include neural networks, computer vision and image restoration. About the Author*TERRY CAELLI received his B.A. (Hons.) in Joint Mathematics and Psychology, and Ph.D. in Visual Pattern Recognition from University of Newcastle, Australia in 1972 and 1975, respectively. He is Professor and Deputy Director of Center of Mapping at Ohio State University. He previously held the positions of Killam Professor in Science at the University of Alberta, Canada (1982}1988), Professor at Queen's University, Canada (1988}1989), Professor of Cognitive Science at University of Melbourne (1989}1994), and Professor and Head of School of Computing at Curtin University of Technology, Australia. Prof. Caelli was a DFG Guest Professor at University of Munich (1986}1987). Prof. Caelli's research interests are in two areas. The "rst is concerned with investigating the processes involved in image interpretation: the ability of machines to recognize and automatically describe structures in images in terms of the world properties they represent. Application domains include manufacturing, environment, GIS and resources. This involves speci"c applications of systems analysis, di!erential geometry, signal processing, decision theory and multivariate statistical methods. The second is in general pattern recognition technologies and their integration with neural network, machine learning and expert systems. These technologies play a critical role, for us, in developing image interpretation systems and in our development of general intelligent information processing systems. Prof. Caelli was associate editor of IEEE Trans. on Pattern Analysis and Machine Intelligence from 1992}1994, and is the associate editor of Pattern Recognition, Neural Networks, Spatial Vision and Int. J. of Pattern Recognition and Artixcial Intelligence. Prof. Caelli is a senior member of IEEE and a member of Int. Pattern Recognition Society. He has published six Books, 18 book chapters, over 120 international referred journal articles, and around 60 referred conference proceedings articles. He was awarded the Convocation Medal for Professional Excellence, University of New-castle in 1994, and the Best Paper Awards from Pattern Recognition Society in 1991 and 1994.
444
H-S. Wong et al. / Pattern Recognition 33 (2000) 427}444
About the Author*LING GUAN received his Ph.D. degree in Electrical Engineering from University of British Columbia in Canada in 1989. He was a research engineer at Array Systems Computing Inc, Toronto, Canada in machine vision and image processing. In 1994, he was a visiting fellow at British Telecom, working on low bit-rate video coding using wavelet transforms. Since October 1992, he has been with the University of Sydney, Australia, where he is currently a Senior Lecturer in Electrical Engineering, and the Director of Signal and Multimedia Processing Laboratory. His research interests include multimedia signal processing (retrieval, coding, human computer communications), machine learning, adaptive image and signal processing, computational intelligence, and multiresolution signal processing. He is an associate editor of IS&T/SPIE Journal of Electronic Imaging and Journal of Real-Time Imaging. He has guest-edited/is guest-editing a number of special issues for international journals, including Proceedings of the IEEE Special Issue on Computational Intelligence. He also serves on the editorial board of CRC Press' Book Series on Image Processing. He has been involved in organizing many international conferences, including Program Co-Chair of IEEE Int. Conf. on Eng. of Complex Computer Systems, California, 1998, and General Co-Chair of IEEE Workshop on Neural Networks for Signal Processing, Sydney, 2000. Dr. Guan is a Senior Member of IEEE, and a Member of SPIE. He is currently serving on IEEE Signal Processing Society Technical Committee on Neural Networks for Signal Processing. He has published about 100 technical papers, and is co-editing a book titled Multimedia Image and Video Processing to be published by CRC Press.
Pattern Recognition 33 (2000) 445}463
Extracting color halftones from printed documents using texture analysis Dennis F. Dunn!,*, Niloufer E. Mathew" !Department of Computer Science, University of Wyoming, Laramie, WY 82071-3682, USA "23287 Sagebrush, Novi, MI 48375, USA Received 28 August 1998; accepted 25 February 1999
Abstract Printed documents often contain text and halftones on the same page. When processing these documents electronically, it is important to separate halftones from text and other illustrations, since their processing requirements di!er. We present an algorithm that extracts both grayscale and color halftones from other information on a page. Halftones are commonly produced as a pattern of dots on a uniform lattice. The halftone dot pattern forms an `invisible texturea, producing a spectral component distinct from other information on the page. We show that a single circularly symmetric bandpass "lter, tuned to this spectral component, can be used to segment halftones having arbitrary shape and size, even in the presence of image rotations. ( 2000 Pattern Recognition Society. Published by Elsevier Science Ltd. All rights reserved. Keywords: Texture; Document layout analysis; Bandpass "lters; Color halftones; Page segmentation
1. Introduction Printed documents often contain text, halftones, and other illustrations on the same page. When processing documents electronically, it is important to separate these di!erent data types, because, typically, their processing requirements di!er. For example, text might be sent to an OCR system for interpretation, whereas illustrations might be compressed for e$cient storage. The process of segmenting a page into homogeneous regions and labeling the regions by data type is called document layout analysis (DLA). A variety of DLA methods appear in the literature and can be divided into three fundamental approaches: topdown, bottom-up, and texture-based. When analyzing a document top-down, pages are split into blocks, checked for homogeneity, and further divided, if neces-
* Corresponding author. Tel.: #307-766-4914; fax: #307766-4036. E-mail address:
[email protected] (D.F. Dunn)
sary. Most of these methods are based on run length smoothing [1,2] or projection cuts [3]. Bottom-up methods group pixels into connected components, which are then merged into progressively larger regions [4}7] Methods combining top-down and bottom-up concepts have also been proposed [8]. Texture-based methods treat the di!erent data types as di!erent textures, where each texture is characterized by some distinctive repeating pattern. Some methods, attempt to detect these patterns directly using spatial masks. Examples include the Texture Co-occurrence Spectrum method of Payne [9] and the neural network-trained masks used by Jain [10]. Other methods exploit the fact that repeating patterns produce distinct spectral signatures [11}14]. Bandpass "lters are then employed to isolate regions associated with those signatures. Although the methods mentioned above are all e!ective in certain domains, robust DLA has not yet been achieved. For example, some methods are intolerant [1}3] or have limited tolerance [8] to document skew. Thus, a page that is rotated signi"cantly when scanned would be problematic. Also, several approaches are restricted to documents having a Manhattan layout; i.e.,
0031-3203/00/$20.00 ( 2000 Pattern Recognition Society. Published by Elsevier Science Ltd. All rights reserved. PII: S 0 0 3 1 - 3 2 0 3 ( 9 9 ) 0 0 0 8 9 - 8
446
D.F. Dunn, N.E. Mathew / Pattern Recognition 33 (2000) 445}463
clear horizontal and vertical white gaps between and within regions [1}4,7,8]. While this is often the case, it is not true in general. What makes DLA so di$cult is that no single approach seems to be suitable for all the data types encountered on a page; e.g., e!ective methods for isolating text may require assumptions (Manhattan layout, for example) that are not always valid for halftones. The approach advocated here is to simplify DLA by eliminating halftones from the page, using an algorithm customized for that purpose. The idea is to preprocess the digitized page to identify regions containing halftones and replace them with the background color. The extracted halftones can be discarded or processed separately. In this way, existing DLA techniques can be applied to a more homogeneous input. Implementing this strategy requires a reliable and e$cient algorithm for segmenting halftones from other information on the page. This paper presents such an algorithm. We show that a single circularly symmetric bandpass "lter can robustly segment both grayscale and color halftones from a digitized page, independently of other information on the page. It can do so regardless of the number, size, shape, or orientation of the halftone region(s). We further show how this "lter can be easily designed, using a sample page from the document. Although other methods have been proposed for segmenting halftones, the ability to accurately segment color and grayscale halftones of arbitrary shape is novel to this approach. It is also important to note that only a single "lter is required for all halftones in the entire document. In contrast, a bank of "lters was required by others [11,12]. The remaining sections provide more detail. Section 2 describes grayscale and color halftones and their spectral characteristics. Section 3 describes the segmentation algorithm. Section 4 presents experimental results, and Section 5 is the conclusion.
2. Halftones and their spectra Commercial printing processes produce documents by depositing ink on paper. Only a few colors of ink are used, however, so shades of gray or color cannot be generated directly. Consequently, continuous-tone images (color or grayscale) are typically printed as a halftone. A halftone consists of an array of closely spaced microdots, all produced with the same color of ink. At normal viewing distances, the individual dots are imperceptible to the naked eye. Instead, the eye integrates the dots and surrounding white space (within a local neighborhood), producing the perception of continuous shades of color. Di!erent shades can be simulated by locally varying dot density. When clusters of micro-dots are centered on a uniform grid, the halftoning method is called clustered-dot ordered dither. If the position of the micro-dots
is unrestricted, aliasing and other visual artifacts can be reduced by employing various optimizing techniques. Dispersed-dot ordered dither and error di!usion are examples of this approach. Because of its simplicity, commercial printers typically use a version of clustered-dot ordered dither halftoning called the `classical screena method. To produce a halftone for a grayscale image, a glass plate etched with a grid of "ne lines (the `screena) is placed between the camera lens and the image to be reproduced. The presence of the screen transforms continuous tones in the input to a pattern of macro-dots in the output. The macro-dots are uniformly spaced and their size is proportional to the input grayscale. The screening process can be simulated digitally by using clusters of micro-dots to form macro-dots of varying sizes. Although macro-dots are larger than micro-dots, they are still visually imperceptible. A halftone for a color image can also be produced using a variation of the classical screen method (see Fig. 1). First, the input image is "ltered to decompose it into four-color components: cyan, magenta, yellow, and black. This four-color system is referred to as CMYK. A continuous-tone image is generated for each of the four-color components.1 Next, each component image is converted to a halftone, using the classical screen method. The four halftones are then overlaid to form a single composite halftone. The overlays are rotated relative to each other to minimize dot overlap, which reduces inter-color MoireH patterns and chromatic errors (Fig. 2). When the composite halftone is observed from normal viewing distances, the eye integrates the colored dots, producing the illusion of continuous shades of color. Fig. 3 shows a color halftone and an enlargement revealing the halftone dots. Clustered-dot ordered dither halftones have an interesting property. Because the macro-dots (or clusters of micro-dots) occur on a closely spaced uniform grid, the dot pattern forms a texture that is imperceptible to the naked eye. As Dunn et al. [15] have shown, this `invisiblea texture produces high-frequency spectral energy that is distinct from the visible information in the halftone and from other information on the page. More importantly, halftones derived from di!erent images (but produced by the same screen) are e!ectively instances of the same texture, even though the halftones may visually appear quite di!erent. This is because the halftone dot spacing is the same. Fig. 4 shows a grayscale halftone and its associated spectrum. Note the four bright spots
1 In principle, only the three subtractive primary colors (cyan, magenta, yellow) are required, since black can be produced by mixing the three colors; however, black is used seperately to improve image quality and to conserve ink.
D.F. Dunn, N.E. Mathew / Pattern Recognition 33 (2000) 445}463
447
Fig. 1. Color halftoning process.
Fig. 2. Traditional screen angles for color halftones.
(circled in black) that are produced by the uniformly spaced halftone dots. By exploiting this property, Dunn et al. [15] were able to extract grayscale halftones from printed documents, using a single bandpass "lter (a Gabor "lter). The "lter transforms halftones into highcontrast regions that, when segmented by thresholding, distinguish them from the other regions in the document. Color halftones also produce concentrations of spectral energy at high frequencies [16]. Because a color halftone is a composite of four overlaid halftones, though, the spectrum of a color halftone is more complex. To better understand the nature of this spectrum, consider what occurs when a color halftone is scanned. As mentioned earlier, color halftones are produced using the
CMYK color scheme. Color scanners, though, use photodetectors that are sensitive to red, green, and blue (RGB). Conversion between the two-color schemes is as follows: Cyan absorbs red, so it stimulates the green and blue photodetectors (Cyan" White!Red); Magenta stimulates the red and blue photodetectors (Magenta" White!Green); Yellow stimulates the red and green photodetectors (Yellow" White!Blue); Black absorbs all the color and therefore stimulates none of the photodetectors. When a color halftone is scanned, the scanner produces three grayscale images, each representing the color intensity in one of the RGB color bands. Because each CMYK color stimulates multiple RGB photodetectors, the spectrum of each color band is a composite.
448
D.F. Dunn, N.E. Mathew / Pattern Recognition 33 (2000) 445}463
Fig. 3. (a) Color halftone; (b) Enlargement of a portion of halftone in (a) illustrating the halftone dots.
Fig. 4. (a) Halftone of a grayscale image digitized at 300 dpi; (b) Log-magnitude spectrum of the halftone illustrating the concentration of spectral energy (circled in black) at high frequencies induced by the halftone dots.
Each color band is in#uenced by the dots contained in two halftones from the set of three CMY component halftones and the white background surrounding the dots from all four halftones. Since each component halftone is produced using the same halftone screen but at di!erent angles of rotation, the high-frequency spectral components of the color halftone lie on a circle centered at DC (see Fig. 5). This suggests that a single circularly symmetric "lter with an annular passband might be a good choice to extract both color and grayscale half-
tones. Section 3 describes a halftone-segmentation algorithm based on this concept.
3. Halftone segmentation algorithm This section presents an algorithm for segmenting regions containing grayscale or color halftones from other information on a printed page. The algorithm handles rotated images and regions having arbitrary shape. Fig. 6
D.F. Dunn, N.E. Mathew / Pattern Recognition 33 (2000) 445}463
449
Fig. 5. Spectra of the three color bands (RGB) from a digitized halftone. Bright spots just outside black circles illustrate the circular distribution of spectral energy induced by the rotated halftone screens: (a) Spectrum of red band; (b) Spectrum of green band; (c) Spectrum of blue band.
Fig. 6. Schematic of halftone segmentation algorithm.
450
D.F. Dunn, N.E. Mathew / Pattern Recognition 33 (2000) 445}463
provides a schematic overview of the algorithm. An input page is digitized by a scanner. The scanner produces three grayscale images, which represent the input in terms of the color bands (RGB). Each image then undergoes a "ltering operation. First, the image is "ltered with a circularly symmetric bandpass "lter in the frequency domain. The absolute value of the "lter output is then computed and smoothed with a lowpass "lter. This "ltering process transforms the halftones within each image into high-contrast regions suitable for segmentation. The three images are then passed to the segmentation stage, where the halftones are extracted from the page. A detailed description of the algorithm follows.
frequency domain, is equivalent to taking the product of their Hankel transforms, in the spatial domain. Hence, to obtain an expression for the "lter impulse response (spatial domain), the Hankel transforms of the two circularly symmetric functions, de"ned in Eq. (2), is computed and their product taken. The Hankel transform of d(q!q ) 0 is 2pq J(2pq r), where J is the zero-order Bessel func0 0 2 2 2 tion. The Hankel transform of e~2p p q is given by (1/2np2) exp !(r)2/2p2. The "lter impulse response, h(r, h), can be mathematically approximated as
3.1. Choosing the bandpass xlter
where r, h represent polar coordinates in the spatial domain.
The use of a single bandpass "lter for extracting halftones is a key feature of this paper. This section provides motivation for the choice of "lter and describes how it is designed. The "lter was chosen to match the spectral characteristics of halftones. As mentioned earlier, due to the halftone dot spacing, a high-frequency `imperceptiblea texture is present in halftones. This texture produces distinct high-frequency spectral components distinct from other information in a document. As illustrated in Fig. 5, the Fourier spectra of the three color bands exhibit high-frequency spectral peaks that lie on a circle centered at DC. This suggests using a circularly symmetric "lter with an annular passband to extract halftones. Although a variety of "lters could be used, we have chosen a "lter having a Gaussian-like radial cross section. This decision was motivated by the good space/frequency localization of the Gaussian. An expression for the "lter in the frequency domain is given by H(q, /)"e~2p2p2(q~q0)2, q*0,
(1)
where q and / are the frequency-domain polar coordinates. The passband of the "lter is determined by two parameters; the radial center frequency q and band0 width p. To obtain an expression for the "lter in the spatial domain, we "rst form an approximation to Eq. (1) by convolving a circular impulse sheet d(q!q ), positioned 0 at a radial distance q from DC, with a circularly sym02 2 2 metric Gaussian e~2p p q centered at DC. Or expressed mathematically, H(q, /)+d(q!q )*e~2n2p2q2, 0
(2)
where * refers to the convolution operator. To express Eq. (2) in the frequency domain, we observe that the Fourier transform of a circularly symmetric function is circularly symmetric. [17]. Thus, Eq. (2) can be expressed in terms of the Hankel transform. The convolution of two circularly symmetric functions, in the
q !(r)2 h(r, h)+ 0 J(2pq r) exp , 0 p2 2p2
(3)
3.2. Determining the xlter parameters This section describes how the "lter parameters q and 0 p are determined. The radial frequency of the "lter q is 0 directly related to the halftone dot spacing. Although the value of q is critical to "lter performance, it can easily be 0 determined by searching the Fourier spectrum of a prototypical digitized page. Since the spectral energy due to the halftone is signi"cantly higher in frequency than other information on the page (if this were not true, the halftone dots would be visible), the search can be limited to frequencies above some cuto! frequency, as de"ned by the range of possible halftone screens. Halftones in printed documents, like newspapers and magazines for example, are produced using standard halftone screens. Magazines use halftone screens in the range 133}150 lpi (grid lines per inch), whereas newspapers use screens in the range 85}100 lpi. For a color halftone, the radial frequency is obtained by searching the Fourier spectrum of any one of the color bands. Since within a document, halftones are typically produced by the same halftone screen, once the radial frequency of the "lter is determined, it is used for the entire document. The "lter bandwidth p, unlike q , is not critical and can 0 be determined heuristically. Dunn et al. [15] suggest using a large bandwidth, because it o!ers more spatial resolution, which improves the localization of the halftone boundaries. In this paper, p"4 pixels for most examples. Evidence con"rming the algorithm's insensitivity to "lter bandwidth has been presented by Mathew [18]. 3.3. Applying the xlter The "lter parameters are determined from a prototypical page of the document. Once designed, the same "lter can be used for the entire document. Applying the
D.F. Dunn, N.E. Mathew / Pattern Recognition 33 (2000) 445}463
bandpass "lter to a digitized page is performed in the frequency domain. The "lter, de"ned by H(q, /) in Eq. (1), is multiplied by the Fourier transforms of three colorband images, as depicted in Fig. 6. Because the bandpass "lter is real-valued, the "lter response (in the area of the image occupied by the halftone) is sinusoidal-like. To recover the amplitude envelope, we compute the absolute value of the "lter response and smooth the output with a Gaussian lowpass "lter. The result of the "ltering operations produces three grayscale images, called xltered images, representing the spectral energy in each of the color bands. Since the spectral energy can be distributed arbitrarily across the three bands, it is necessary to combine the three "ltered images, to obtain the spectral energy of the color halftone. We combine the spectral energy of the three color bands by summing the three "ltered images at each pixel, i.e., 3 F (x, y)" + F (x, y), s k k/1
∀x, y,
(4)
where F is the composite image, and F is a "ltered s k image corresponding to color band k. Fig. 7(a) shows a page containing text and a color halftone, and F (x, y) is s shown in Fig. 7(b). Observe that much of the information contained in the halftone is recovered by the "ltering operation. This is to be expected. As Dunn et al. [15] observed, the spectrum of a halftone can be approximated by the spectrum of the original image replicated at integer multiples of the halftone dot frequency. Since the bandpass "lter is tuned to this frequency, much of the image baseband signal is retained. Note, however, that information that is not part of the halftone is not retained. In this case, the band of text at the top of the image is now black in Fig. 7(b), signifying low "lter response. To ultimately segment the halftone from the image, the composite image F (x, y) is passed to the s segmentation stage, which is described in the next section. 3.4. The segmentation stage Because the "ltering operation responds much better to halftones than other information on the page, in principle, simple thresholding should be su$cient to perform segmentation. Thresholding is e!ective except when the halftone contains regions that are very dark or very light. As the color approaches black, the size of the macro-dots increases until the dots signi"cantly overlap. When this happens, the dot pattern no longer forms a texture, and the distinct spectral properties are lost. Similarly, as the color approaches white, the dots disappear, and little spectral energy above DC is produced. Fig. 7(c) illustrates this phenomena. Fig. 7(c) is a binary image, produced by thresholding the composite image in Fig. 7(b). Note that black and white regions in the halftone are misclassi"ed.
451
A solution to this problem was proposed by Dunn et al. [15]. Their approach assumed that all halftones occupy rectangular regions within a page. Misclassi"ed black and white regions were correctly included as part of the halftone by computing a minimum-area rectangular bounding box. While the rectangular-region assumption is valid for most halftones within a document, it is not always true. This section describes an alternative segmentation algorithm that does not restrict region shape. The segmentation scheme begins with simple thresholding. Thresholding transforms the composite "ltered image, F (x, y), representing the spectral energy of the s halftone (Fig. 7(b)), into a binary-valued image F (x, y) as b shown in Fig. 7(c). In F (x, y), white corresponds to b halftone regions (areas of signi"cant "lter response) and the remaining areas are set to black. The "lter response to halftones is signi"cantly greater than for other regions, so a wide range of thresholds may be used. Results demonstrating the insensitivity of the algorithm to threshold are presented in Section 4.6. As mentioned earlier, black and white regions within the halftone are misclassi"ed by thresholding. The next step is to reclassify the improperly classi"ed pixels. Misclassi"ed black areas within the halftone region can be identi"ed by locating corresponding black areas in the original halftone. To accomplish this, we "rst produce a grayscale version of the original input image by summing pixel values across the three color bands. Mathematically, 3 I (x, y)" + I (x, y), s k k/1
(5)
where I (x, y) is the grayscale version of the original input s image and I (x, y), k"1, 3 represent the scanner output k images corresponding to the three color bands (RGB). I (x, y) is then thresholded to produce a binary image s I (x, y). The threshold used is the same as that chosen for b F (x, y). Threshold selection is discussed in Section 4.6. s An example of I (x, y) is shown in Fig. 7(d). F (x, y), b b representing the thresholded "lter output, is then logically combined with I (x, y), according to the following b logic expression: O (x, y)"2I (x, y)sF (x, y). b b b
(6)
A tabular representation of Eq. (6) is provided in Table 1. As shown in Table 1, black pixels in F (x, y), having b a corresponding dark pixel in the original halftone, are set to white. Other combinations of pixel colors leave F (x, y) unchanged. In e!ect, the result of logically comb bining I (x, y) and F (x, y) is as follows: b b (1) Regions within a halftone characterized by high "lter response, which have been properly classi"ed (set to white) in the thresholded "lter output (Fig. 7(c)), are una!ected. (2) Properly classi"ed non-halftone regions
452
D.F. Dunn, N.E. Mathew / Pattern Recognition 33 (2000) 445}463
comprising the background are una!ected. (3) The problematic misclassi"ed black regions within the halftone, characterized by low "lter response, are properly classi"ed (set to white) and included as part of the halftone (Fig. 7(e)). (4) Text, which was black in the original image, is (incorrectly) set to white. (5) The problematic white regions within the halftone, are (incorrectly) set to black.
Unfortunately, the text and white regions within the halftone are misclassi"ed. If line drawings were present, they too would appear white, representing a misclassi"cation. We correct these errors using a form of pixel labeling. Although the text is misclassi"ed, it is almost always separated from the halftone by `white spacea. Thus, pixel
Fig. 7. (a) Digitized page with text and a color halftone; (b) F (x, y), the sum of "lter outputs; (c) F (x, y), produced by thresholding s b F (x, y) at 64 (out of 255); (d) I (x, y), sum of the three color bands, thresholded at 64. (e) O (x, y), the result of logically combining F (x, y) s b b b and I (x, y); (f ) Result after "rst pixel labeling operation to reclassify the text; (g) Result after second pixel labeling to reclassify the white b regions within the halftone.
D.F. Dunn, N.E. Mathew / Pattern Recognition 33 (2000) 445}463
453
Fig. 7. (Continued.)
labeling within the halftone region can be used to properly reclassify the text. First, a 3]3 median "lter is applied once to remove small connected components attributed to noise. Then, pixel labeling is initiated by choosing a white seed pixel from O (x, y). The seed pixel is selected b from the largest contiguous row of white pixels having "lter responses above threshold. This ensures that the seed will be taken from a region containing a
halftone. The seed pixel is labeled as part of the halftone, and all 8-connected white pixels are labeled recursively. To deal with multiple halftones, pixel labeling is applied multiple times, until the largest contiguous row of white pixels is small (say (20 pixels). After pixel labeling terminates, all non-labeled white pixels are changed to black. The result after pixel labeling is shown in Fig. 7(f).
454
D.F. Dunn, N.E. Mathew / Pattern Recognition 33 (2000) 445}463
Table 1 Logically combining I (x, y) and F (x, y) b b I (x, y) b
F (x, y) b
O (x, y) b
Black Black White White
Black White Black White
White White Black White
While, pixel labeling corrects the misclassi"ed text, it does not eliminate the problematic white regions within the input halftone, which still remain misclassi"ed (bottom of Fig. 7(f)). Recovering these misclassi"ed regions is easy, though, because they are surrounded by white. If this were not true, then the misclassi"ed regions would just be a continuation of the background and technically not part of the halftone. The approach is to perform a second pixel labeling operation on the output of the "rst pixel labeling operation } this time pixel labeling occurs within the non-halftone regions. A black seed pixel is selected from the largest contiguous row of black pixels, and the seed pixel is labeled as a non-halftone. Neighboring black pixels are labeled recursively. Pixel labeling is applied multiple times to deal with disconnected non-halftone regions. Fig. 7(g) shows the "nal result. Observe that the input halftone is e!ectively segmented from the rest of the information in the page.
4. Experimental results This section demonstrates the e!ectiveness of the algorithm for extracting color and grayscale halftones from the other information in a document. A variety of examples are provided. All input images were 512]512 pixels. 4.1. Example 1: extracting multiple halftones from a page This example demonstrates the algorithm's ability to extract multiple color halftones surrounded by text, a situation encountered in many printed documents. Within a given document, halftones are normally produced with the same halftone screen. As a result, they can be viewed as being instances of the same texture. Hence, despite their visual di!erences, a single "lter, tuned to the same halftone frequency, can be used to extract the multiple halftones. Fig. 8(a) displays a portion of a digitized page containing two-color halftones along with text. The three-color bands produced by the scanner are each bandpass "ltered. An automatic search of the spectrum of Fig. 8(a) produced a "lter center frequency of 0.496 cycles/pixel. The bandwidth was set to 4.0 pixels. The three "lter
outputs are summed and thresholded at 64 (out of 255), producing the binary image shown in Fig. 8(c). This image constitutes a "rst approximation to the desired segmentation, where white corresponds to the halftone and black represents everything else. Note, however, that dark areas in the halftone have been misclassi"ed (set to black). Using Eq. (6) followed by pixel labeling corrects these errors. First, the sum of the three color bands is computed, producing a grayscale version of the input. This grayscale image is then thresholded at 64 (out of 255), producing a binary version of the input (shown in Fig. 8(b)). The logical combination of Fig. 8(b) and (c) using Eq. (6) results in Fig. 8(d). Note that the halftone regions are now approximately correct, but that the text is now misclassi"ed (white). Three stages of pixel labeling (two passes with white seed pixels and one pass with a black seed) produces the "nal segmentation shown in Fig. 8(e). 4.2. Example 2: extracting halftones under rotation The circularly symmetry of the bandpass "lter allows for arbitrary image rotation. Fig. 9(a) shows a portion of a page that was rotated by about 453 during scanning. The "lter center frequency was estimated at 0.433 cycle/pixel, which is similar to Example 1. The center frequencies are similar because both Figs. 8(a) and 9(a) were extracted from magazines, which typically use the similar screen sizes. Fig. 9(b) shows the sum of the "lter outputs, and Fig. 9(c) shows the result of applying Eq. (6). The "nal segmentation, after pixel labeling, is shown in Fig. 9(d). 4.3. Example 3: extracting non-rectangular grayscale halftones A black and white photograph has varying shades of gray. Reproducing a black and white photograph in a newspaper or magazine by halftoning requires only black ink, unlike color halftones. This example demonstrates that the algorithm can successfully extract grayscale halftones from a newspaper. Also, one of the strong features of the algorithm is its applicability to non-rectangular regions, a drawback of many earlier page segmentation techniques. Fig. 10(a) is a digitized page containing text and a grayscale halftone, taken from a newspaper. Note that the halftone is enclosed by a circular boundary. Since a halftone from a newspaper is produced using a coarser halftone screen compared to magazines, the computed center frequency is lower. For this example, a center frequency of 0.289 cycles/pixel was computed. Grayscale halftones are processed in the same way as color halftone, but since the only color present in the halftone is black, the spectra of the three color bands are identical.
D.F. Dunn, N.E. Mathew / Pattern Recognition 33 (2000) 445}463
455
Fig. 8. (a) Digitized page containing two color halftones and text; (b) Sum of color bands, thresholded at 64 (out of 255); (c) Sum of "lter outputs, thresholded at 64 (q "0.496 cycles/pixel, p"4.0 pixels); (d) Logical combination of (b) and (c) using Eq. (6); (e) Final result 0 after the second pixel labeling operation.
456
D.F. Dunn, N.E. Mathew / Pattern Recognition 33 (2000) 445}463
Fig. 9. (a) Image rotated by about 453; (b) Sum of "ltered outputs, thresholded at 64 (q "0.433 cycles/pixel, p"4.0 pixels); (c) Logically 0 derived result using Eq. (6); (d) Final result after pixel labeling.
Figs. 10(b)}(d) shows the results from intermediate stages of the algorithm. Fig. 10(e) shows the "nal segmentation.
algorithm's insensitivity to bandwidth, p was set to 15 pixels. The outputs from the algorithm are shown in Figs. 11(b)}(e).
4.4. Example 4: extracting non-rectangular color halftones This section demonstrates the ability of the algorithm to extract a non-rectangular color halftone. The image (Fig. 11(a)), taken from a magazine, contains a halftone of arbitrary shape. The "lter center frequency was 0.441 cycles/pixel. Also, to demonstrate the
4.5. Example 5: extracting color halftones containing large regions of black Fig. 12 illustrates the ability of the algorithm to successfully extract halftones containing signi"cant regions of black. Observe that only a small portion of the halftone is
D.F. Dunn, N.E. Mathew / Pattern Recognition 33 (2000) 445}463
457
Fig. 10. (a) Digitized input page with text and grayscale halftone; (b) Sum of color bands thresholded at 64 (out of 255); (c) Sum of "lter outputs thresholded at 64 (q "0.289 cycles/pixel, p"4.0 pixels); (d) Logical combination of (b) and (c) using Eq. (6); (e) Final result 0 after pixel labeling.
458
D.F. Dunn, N.E. Mathew / Pattern Recognition 33 (2000) 445}463
Fig. 11. (a) Digitized input containing text and a color halftone. (b) Sum of color bands, thresholded at 64 (out of 255). (c) Sum of "ltered outputs, thresholded at 64 (q "0.441 cycles/pixel, p"15.0 pixels). (d) Logical combination of (b) and (c) using Eq. (6). (e) Final 0 segmentation result after pixel labeling.
D.F. Dunn, N.E. Mathew / Pattern Recognition 33 (2000) 445}463
459
Fig. 12. (a) Digitized input page containing text and a color halftone consisting of large regions of black; (b) Sum of color bands, thresholded at 64 (out of 255); (c) Sum of "lter outputs, thresholded at 64 (q "0.440 cycles/pixel, p"4.0 pixels); (d) Logical 0 combination of (b) and (c) using Eq. (6); (e) Final segmentation result after pixel labeling.
460
D.F. Dunn, N.E. Mathew / Pattern Recognition 33 (2000) 445}463
recovered by thresholding the "lter output (Fig. 12(c)); however, the subsequent segmentation steps are able to recover the entire halftone. 4.6. Example 6: insensitivity to threshold level In all previous examples, the threshold level for both F (x, y) and I (x, y) was 64 out of 255. Identical thresholds s s
were used for convenience and the value was determined experimentally. Figs. 13 and 14 show the e!ect of using other thresholds. Fig. 13(a) represents the same input image and "lter parameters as Fig. 7(a), only the threshold levels di!er. Segmentation results for threshold levels of 32 and 128 are shown in Figs. 13(c)}(g), respectively. Fig. 14 is a non-rotated version of Fig. 9. The same "lter parameters were used in both "gures but three
Fig. 13. (a) Digitized input with text and color halftone; (b) Sum of "lter outputs, thresholded at 32 (out of 255); (c) Final segmentation result after thresholding at 32 (out of 255) and pixel labeling; (d) Sum of "lter outputs, thresholded at 128 (out of 255); (e) Sum of color bands, thresholded at 128 (out of 255); (f ) Logical combination of (d) and (e) using Eq. (6); (g) Final segmentation result after thresholding at 128 (out of 255) and pixel labeling.
D.F. Dunn, N.E. Mathew / Pattern Recognition 33 (2000) 445}463
461
Fig. 13. (Continued.)
di!erent thresholds were used in Fig. 14. Final segmentation results are shown in Figs. 14(b)}(d). Note, that except for the small segmentation error in Fig. 14(d), "nal segmentation results were independent of threshold. Since experimental evidence suggests that our algorithm is insensitive to threshold level, no automated method for selecting threshold level was developed. A discussion of threshold selection is, however, provided below. Threshold for F (x, y). A low threshold for F (x, y) will s s tend to classify some non-halftone regions as being part
of the halftone. This will occur in certain areas of the text where character spacing produces a weak response from the bandpass "lter. The uniform background, though, will produce a response close to zero. So even a low threshold will not include the background. The smaller regions of misclassi"ed text will be eliminated by the median "lter. Larger regions will be properly reclassi"ed by pixel labeling. A high threshold, on the other hand, can cause problems as shown in Fig. 13(d). If a legitimate halftone region
462
D.F. Dunn, N.E. Mathew / Pattern Recognition 33 (2000) 445}463
Fig. 14. (a) Test image, non-rotated version of Fig. 9(a); (b) Final segmentation result, using a threshold of 32 (out of 255); (c) Final segmentation result, using a threshold of 64 (out of 255); (d) Final segmentation result, using a threshold of 128 (out of 255). Note the presence of some segmentation error at this threshold.
does not survive thresholding and is not black, it may never be recovered. White regions might also be misclassi"ed after thresholding, but since we assume that they are surrounded by the halftone (otherwise they would be indistinguishable from the white background), the subsequent pixel-labeling operation correctly reclassi"es these regions. Threshold for I (x, y). Unlike F (x, y), a high threshold s s is preferred for I (x, y). The goal here is to locate regions s of black in the input image so that they can be matched with black regions in the "lter-output image. It does not
matter if more of the halftone is included. A problem occurs only if the background does not survive thresholding and is made black (the text will be black regardless of threshold, so the text is not a!ected). Thus, the threshold level must be lower than the background.
5. Conclusion The advent of color in the printing industry provides new challenges in the area of document processing. Many
D.F. Dunn, N.E. Mathew / Pattern Recognition 33 (2000) 445}463
of the earlier document layout analysis techniques dealt only with grayscale halftones. The goal of this research was to develop methods to extract both color and grayscale halftones from other information in the document. The approach capitalizes on the spectral characteristics of halftones, which allows a single bandpass "lter with an annular passband to extract halftones. The circular symmetry of the "lter provides invariance to image rotation. This is an improvement over previous methods, which either employed orientation-speci"c "lters or made assumptions about region shape to deal with rotated images. While the algorithm presented in this paper is successful for extracting halftones, it has certain limitations. First, the algorithm cannot handle the unusual situation where halftone shading overlays text. In this case, the halftone and the text occupy the same region, and the text will be incorrectly interpreted as black areas within the halftone. Second, a large white region within the halftone could be problematic. In particular, if the halftone contains a long contiguous horizontal band of white that exceeds the longest contiguous string of background pixels, the second pixel-labeling operation would choose a seed pixel from within the white region, leading to the misclassi"cation of that region. Such cases are rare, however. Third, a high sampling rate (typ. 300 dpi) must be employed to capture the texture characteristics of the halftone. This sample rate is also used by other methods, however [5,7,8]. Finally, halftoning techniques that use dispersed dot patterns, such as dispersed-dot ordered dither or error di!usion, produce a broadband spectrum that is not suitable for this algorithm. These techniques, though, are much less common.
References [1] K.Y. Wong, R.G. Casey, F.M. Wahl, Document analysis system, IBM J. Res. 26 (6) (1982) 647}656. [2] F.M. Wahl, K.Y. Wong, R.G. Casey, Block segmentation and text extraction in mixed text/image documents, Comp. Vision Graphics Image Proc. 20 (1982) 375}390.
463
[3] D. Wang, S.N. Srihari, Classi"cations of newspaper image blocks using texture analysis, Comp. Vision Graphics Image Proc. 47 (1989) 327}352. [4] L.A. Fletcher, R. Kasturi, A robust algorithm for text string separation from mixed text/graphics images, IEEE Trans. Pattern Anal. Mach. Intell. 10 (1988) 910}918. [5] L. O'Gorman, The document spectrum for page layout analysis, IEEE Trans. Pattern Anal. Mach. Intell. 15 (11) (1993) 1162}1173. [6] A. Simon, J.-C. Pret, A.P. Johnson, A fast algorithm for bottom-up document layout analysis, IEEE Trans. Pattern Anal. Mach. Intell. 19 (3) (1997) 273}277. [7] A.K. Jain, B. Yu, Document representation and its application to page decomposition, IEEE Trans. Pattern Anal. Mach. Intell. 20 (3) (1998) 294}308. [8] T. Pavlidis, J. Zhou, Page segmentation and classi"cation, Comp. Vision Graphics Image Proc. 54 (6) (1992) 484}496. [9] J.S. Payne, T.J. Stonham, D. Patel, Document segmentation using texture analysis, Proceedings of the 12th IAPR, Jerusalem, Israel, October 1994, pp. 380}382. [10] A.K. Jain, Y. Zhong, Page segmentation using texture discrimination, Pattern Recognition 29 (5) (1996) 743}770. [11] A.K. Jain, S. Bhattacharjee, Text segmentation using Gabor "lters for automatic document processing, Mach. Vision Appl. 5 (1992) 169}184. [12] T. Randen, J.H. Hus+y, Segmentation of text/image documents using texture approaches, Proceedings of the NOBIM-konferansen-94, Asker, Norway, June 1994, pp. 60}67. [13] D. Dunn, W. Higgins, J. Wakeley, Texture segmentation using 2-D Gabor elementary functions, IEEE Trans. Pattern Anal. Mach. Intell. 16 (2) (1994) 130}149. [14] D. Dunn, W. Higgins, Optimal Gabor "lters for texture segmentation, IEEE Trans. Image Proc. 4 (7) (1995) 947}964. [15] D.F. Dunn, T.P. Weldon, W.E. Higgins, Extracting halftones from printed documents using texture analysis, Opt. Eng 36 (4) (1997) 1044}1052. [16] D.F. Dunn, N. Mathew, Extracting color halftones from printed documents using texture analysis, Proceeding of the IEEE International Conference Image Processing, vol. I, Santa Barbara, CA, October 1997, pp. 787}790. [17] R.N. Bracewell, in: The Fourier Transform and Its Applications, McGraw-Hill, New York, 1978. [18] N.E. Mathew, Extraction of color halftones using texture analysis, Master's thesis, The Pennsylvania State University, 1997.
About the Author*DENNIS F. DUNN received the B.S. degree in chemical engineering from Case Western Reserve University in 1969, the M.Eng. degree in engineering science from The Pennsylvania State University in 1981, and the Ph.D. degree in computer science from The Pennsylvania State University in 1992. He was an Assistant Professor with the Computer Science and Engineering Department, The Pennsylvania State University from 1992 to 1997. He is presently an Assistant Professor with the Computer Science Department, the University of Wyoming. His main research interests are computer vision, document analysis, image processing, texture analysis, and human-vision modeling. About the Author*NILOUFER MATHEW received the B.S and M.S degrees in Electrical Engineering in 1993 and 1997, respectively, from the National Institute of Engineering, Mysore, India and The Pennsylvania State University, University Park, PA. Her main research interest was in digital halftoning. She is currently a Software Engineer at Decision Consultants Incorporated in South"eld, MI.
Pattern Recognition 33 (2000) 465}481
Feature matching constrained by cross ratio invariance A. Branca*, E. Stella, A. Distante Istituto Elaborazione Segnali ed Immagini - C.N.R., Via Amendola 166/5, 70126 Bari, Italy Received 8 October 1997; received in revised form 29 May 1998; accepted 25 September 1998
Abstract The main aim of this work is to propose a new technique to solve the well-known feature correspondence problem for motion estimation. The problem is formulated as an optimization process whose energy function includes constraints based on projective invariance of cross-ratio of "ve coplanar points. Starting from some approximated correspondences, estimated by radiometric similarity, for features with high directional variance, optimal matches are obtained through an optimization technique. The new contribution of this work consists of a matching process, re"ning the raw measurements, based on an energy function minimization technique converging to an optimal solution for most of the features by taking advantage of some good initial guess, and in the use of cross ratio as geometrical invariant constraint to detect and correct the mismatches due to wrong radiometric similarity measures. Though the method is based on geometrical invariance of coplanar points, it is not required that all features have to be coplanar or to preprocess the images to detect the planar regions. Experimental results are presented for real and synthetic images, and the performance of the novel approach is evaluated on di!erent image sequences and compared to well-known techniques. ( 2000 Pattern Recognition Soceity. Published by Elsevier Science Ltd. All rights reserved. Keywords: Autonomous robots; Motion analysis; Feature matching; Geometrical invariants; Cross ratio
1. Introduction An important role in intelligent mobile robot navigation is played by arti"cial vision. In order to interact with its environment, a robot should be able to process dynamic images for recovering its motion and the structure of the environment. It is well known that a signi"cant amount of useful information about 3D motion and structure can be obtained from the 2D image motion produced by the projection, onto the image plane, of the movements of the camera and of the 3D objects. Though, theoretically, an accurate estimation of 2D image motion could be obtained employing spatial and temporal image derivatives (gradient-based methods) [1}5], in real contexts more robust 3D information can be recovered by
* Corresponding author. Tel.: #39-80-5481969; fax: #3980-5484311 E-mail addresses:
[email protected] (A. Branca), stella@ iesi.ba.cnr.it (E. Stella),
[email protected] (A. Distante)
few correspondences found among signi"cant features extracted from the temporal image sequence (featurebased methods). The main advantage of this kind of approach is to be una!ected by variations in the image intensities and to have a great e$ciency because only few distinct primitive features (edges, junctions, rectangles, curves, contours) need to be matched. The motion correspondence problem, for featurebased approaches, can be formulated as follows: given n frames acquired at successive times, and given m features in each frame, "nd a one-to-one mapping between features belonging to di!erent frames. This requires knowledge of frame-to-frame correspondences between measurements taken at di!erent times but originated from the same geometric feature [6}10]. Approaches involving local constraints on features [11}15], often based on correlation of grey levels of patches centered on features extracted from considered images, assuming that are radiometric similarities, are fast but more sensitive to noise. The main problem of these techniques is represented by false matches generated by some ambiguities arising when trying to "nd the best correlated region.
0031-3203/00/$20.00 ( 2000 Pattern Recognition Soceity. Published by Elsevier Science Ltd. All rights reserved. PII: S 0 0 3 1 - 3 2 0 3 ( 9 9 ) 0 0 0 8 4 - 9
466
A. Branca et al. / Pattern Recognition 33 (2000) 465}481
A more global constraint should be used in order to `disambiguatea matches [16}23]. Classically, standard methods for handling matching errors embody both local constraints (based on feature characteristics) and global constraints (based on a priori knowledge about scene structure and 3D motion) within some search algorithms oriented to explore the space of potential matches with the aim to extract a "nal correct set of matches. Some criteria are de"ned in order to select between correct and incorrect sets of matches, generally through a functional measuring the quality of alternative mappings. Consequently, a lot of time is consumed to solve a combinatorial problem. Moreover, due to unavoidable errors in feature extraction techniques, only a subset of the extracted features will be matched. In the paper, a more robust and general approach to solve the correspondence problem among 2D projected feature points is proposed. Our method is able to estimate optimal matches for most of the features extracted from a frame acquired at a given instant time t. In fact, features at time t#1 are selected as optimal measurements with respect to some imposed constraints, included into an appropriate functional. Extraction and matching of features at time t#1 are considered as a unique process consisting of the minimization of the de"ned error functional. Certainly, the robustness of the approach still depends on the reliability of the imposed constraints. Recent works focus on some results of the projective geometry in order to recover some interesting invariants to be used as constraints in feature matching [24}28]. Projective invariants, popular in object recognition as useful descriptors of objects [29], are properties of the scene that remain invariants under projective transformations, consequently, they represent useful tools to check feature correspondences with a low computational cost. Since most of the indoor scenes include several planar surfaces, most of the used invariants [29,30] have been de"ned for planar objects (using geometric entities such as points, lines and conic) since in this case, there exists a plane projective transformation between the object and the image space. The simplest numerical property of an object that is unchanged under projection to an image is the cross ratio of four collinear points or "ve coplanar points. The projective invariance of cross ratio of "ve coplanar points has been used in literature as constraint for optimal match selection in tracking algorithms (assuming that all considered features are coplanar [31]), planar region detection or object recognition (assuming to know a priori the correct correspondences [32]) using probabilistic analysis [5,33}37]. Actually, the performance of probabilistic approaches depends on the choice of the rule to employing when deciding whether "ve image points have a given cross ratio [38,39]. The de"nition of a robust decision rule, which takes into account the e!ects on the value of the cross ratio of small
errors in locating the image points, and thresholds on the probabilities in the decision rule is a di$cult task. An unexplored "eld is the consideration of projective geometrical invariance constraints into a general optimization process for computing correspondences, without any a priori knowledge about feature coplanarity. Our method estimates correspondences among arbitrary features, extracted from successive images, imposing the `cross ratio invariance of coplanar pointsa as constraint among all considered features. The correspondence problem is formulated as an optimization problem solved by minimizing an appropriate energy function where the constraints on radiometric similarity and projective invariance of coplanar points are de"ned. The method can be considered as a correlation-based approach which takes into account the projective invariance of coplanar points in computing the optimal matches. Cross-ratio invariance is imposed to correct (or to optimize) the matches computed using a correlationbased approach. Though the method is based on planarity constraints and no assumption about coplanarity of considered features is imposed, it is not limited to be used in contexts where the planarity is assumed in the whole image (i.e. the distance camera-scene is larger or in video-based motion analysis, e.g. group games image interpretation) but it works well also in more general contexts where a number of separate planar regions are present. The method we use takes advantage by considering several subsets of "ve points (features) obtained as combinations of available sparse features. If the features are on di!erent planes the minimization approach will constraint only coplanar features to in#uence each other, though subsets include also non-coplanar features. Through a cooperativity process, information is propagated among subsets and, because of the imposed radiometric similarity, matches among non-coplanar features are avoided. Summarizing, once high variance features are extracted in the "rst frame using the Moravec's interest operator [12] (Section 2), raw matches, estimated through radiometric similarity, are updated, iteratively, in order to estimate the correct corresponding features in the second frame (Section 3) by minimizing a cost function (Section 4). The performances (Section 5) of the method have been evaluated on a synthetical image sequence by comparing the error between the computed motion "eld and the actual one. Moreover, the method has been tested on real-image sequences acquired by a TV camera mounted on a mobile robot moving in a stationary environment.
2. Match initialization Our aim is to describe a general approach in order to estimate optimal matches for features extracted in the
A. Branca et al. / Pattern Recognition 33 (2000) 465}481
"rst frame of a temporal sequence through the minimization of an energy functional combining appropriate constraint functions. Displacement vectors are estimated only for `higha interest features (such as corners and edges) which are the projection onto image plane of conspicuous 3D points (X, >, Z). The corresponding projected points (x"f X/Z, y"f >/Z) (where f is the focal length of the camera) are salient points that can be more easily matched than other points. We use the interest operator introduced by Moravec to isolate points with high directional variance [11]. The variance between neighboring pixels in four directions (vertical, horizontal and two diagonal) is computed over a window (the smallest value is called the interest operator value). Features are chosen where the interest measure has local maxima in windows of variable size. Once high variance features p "(x , y ) are extracted i i i in the "rst frame I (acquired at time t) the best possible 1 candidate matches q "(x6 , y6 ) are selected in the second i i i frame I (acquired at time t#1) using a correlation2 based measure (radiometric similarity). Given a feature p "(x , y ) in the "rst image we select the feature i i i q "(x6 , y6 ) in the second image in order to maximize the i i i radiometric similarity (1) between the corresponding w]w centered windows = (p )"[x !w, x #w]] 1 i i i [y !w, y #w] and = (q )"[x6 !w, x6 #w]] i i 2 i i i [y6 !w, y6 #w]. i i m[= (p )= (q )]!m[= (p )]m[= (q )] 1 i 2 i 1 i 2 i , R(p , q )" i i Jv[= (p )]v[= (q )] 1 i 2 i
(1)
where m and v represent, respectively, the mean and variance functions. Actually, since we compute correspondences between regions, false matches are unavoidable: the correct match point can mismatch the center of the highest correlated window or several windows can have the same correlation value. Matches computed through correlation can represent only an initial guess that we propose to improve through an optimization approach.
3. Match optimization In order to verify the goodness of raw matches and to correct all mismatches, we propose to impose the projective invariance of cross ratio of any "ve coplanar point set Q"(p , p , p , p , p ). 1 2 3 4 5
467
Fig. 1. The pencil of four coplanar lines has a cross-ratio de"ned by the angles between lines. Any line intersecting the pencil has the same cross-ratio for the points of intersection of the line with the pencil.
on a straight line l projected from any vertex < (center of 1 upon another di!erprojection) onto points Ma N i i/1,2, 4 ent straight line l . Introducing an appropriate Cartesian 2 coordinate system (so that the four points Ma N lie i i/1,2,4 on the x-axis), the cross ratio is written as (x !x )(x !x ) 1 4 2. cr(a , a , a , a )" 3 1 2 3 4 (x !x )(x !x ) 3 2 4 1
(2)
Since points and lines are dual in the projective plane, there exists an equivalent cross-ratio for coplanar lines. The dual relation to collinearity of points on a line is the incidence of straight lines to a point. A cross-ratio is de"ned on four coplanar lines (u , u , u , u ), incident at 1 2 3 4 the same point (pencil of lines), in terms of the angles among them (Fig. 1) and is given by sin(a )*sin(a ) 13 24 , cr(a , a , a , a )" 13 24 23 14 sin(a )*sin(a ) 23 14
(3)
where a is the angle formed by the incident lines u and ij i u . For any line l which cuts the pencil, the four points j Ma N of intersection of the line l and the pencil i i/1,2,4 de"ne a cross-ratio cr(a , a , a , a ) on the line l equal to 1 2 3 4 cross ratio of angles cr(a , a , a , a ). 13 24 23 14 Any "ve coplanar point set Q"(p , p , p , p , p ) 1 2 3 4 5 (three of which nolt collinear) will be characterized by the cross-ratio invariance of the pencil of coplanar lines generated by joining a point of Q with the other four (Fig. 2): CR(Q)"cr(a , a , a , a ). 13 24 23 14
(4)
3.2. The constraint satisfaction model 3.1. Cross ratio dexnition The projective invariance of "ve coplanar points comes from the projective invariance of cross ratio of four separated points on a line. Let MA N be four points i i/1,2, 4
For each subset of "ve coplanar points P in the "rst n image I , the corresponding points Q in the second 1 n image I should have the same cross ratio. Actually, 2 we apologize that planar surfaces are present in the
468
A. Branca et al. / Pattern Recognition 33 (2000) 465}481
constraints to be satis"ed by all N-dimensional combinations of the unknown measurements. Combinatorial optimization problems have NP complexity, so the computational response time is exponential with respect to the number of variables. Combinatorial problems are solved de"ning an energy function as the sum of all constraint function values obtained for all possible combinations of the unknown measurements. In our context, unknown measurements are represented by features in the second image which result to be the best matches for features extracted in the "rst image, and the energy function, that will be minimized, consists of the sum of the quadratic norm of constraint functions estimated on feature subsets: Fig. 2. Any set of "ve coplanar points has the cross-ratio of the corresponding pencil.
NS E" + DDC(S )DD2, n n/0
(5)
environment, but we do not know anything about the coplanarity of the extracted features. In fact, given "ve arbitrary features, if the cross ratio is not preserved, we cannot decide if the features are on di!erent planes or they have been mismatched. In most of the previous works, using cross-ratio invariance, the assumptions about coplanarity or correctness of matches are a priori imposed. Our goal is to provide optimal matches without a priori knowledge about feature coplanarity. The idea is to overcome this twofold problem by imposing the cross-ratio invariance constraint to be satis"ed on a lot of combinations of available features. Assuming a number of coplanar correct matches are available, through a cooperativity process the propagation of information among subsets permits coplanar features to in#uence each other in order to obtain correct matches. Five correct coplanar features will in#uence other coplanar features to satisfy the imposed constraint, on the other hand, non-coplanar features will not in#uence each other because for them the constraint is not satis"ed. However, since neighboring points have a high probability to belong to the same plane, in our experiments we have applied the algorithm independently on distinct sets of neighboring points. This choice has the result to speedup the process, too. Moreover, to avoid errors in match estimates due to the presence of non-coplanar subsets, we still impose the radiometric similarity to be satis"ed.
where
4. A least-squares minimization approach
4.1. The cross-ratio constraint
Our problem becomes a constraint satisfaction problem to be solved through an optimization approach. The optimization approaches are based on the satisfaction of global constraints requiring given equations to be solved by N-dimensional variables representing subsets of N unknown measurements. Actually, we are dealing with a combinatorial optimization, requiring appropriate
In our context, the correspondence problem can be solved by minimizing the energy function (5) where the constraint function C is a cross-ratio-based function of coplanar points. The global cross-ratio constraint to be satis"ed in order to solve the correspondence problem consists of the sum of all di!erences between the crossratio computed for each subset of "ve features of the "rst
f C is the considered constraint function; f S is a subset of matches among features (p ) selected in n i the "rst image and features (q ) recovered in the second i image after the match initialization step; f N is the number of considered subsets of matches; S Energy minimization of Eq. (5) can be performed by relaxation using the well-known gradient steepest descent (GSD) method. Since the energy function E could be non-convex, depending on the constraint functions, the GSD could stop the minimization process in a local minimum. This problem could be resolved using an alternative minimization technique, such as the graduate non-convexity algorithm proposed in Ref. [40] to manage that kind of problem. However, we have veri"ed that the result of minimization strongly depends on the reliability of input data [41]. The practical di$culty in applying GSD may cause non-optimal converging to a local minimum that can sometimes be quite slow requiring many thousand of gradient descent steps. Better performances can be reached if some initial matches, estimated by correlation, can be considered correct enough, so, the GSD is able to determine the global minimum of E.
A. Branca et al. / Pattern Recognition 33 (2000) 465}481
image and the cross-ratio computed for the candidate matches in the second image. The energy function to be minimized in order to solve the correspondence problem is NS E" + DDCR(P )!CR(Q )DD2, n n n/1
(6)
where CR(P ) is the cross-ratio function de"ned in Eq. (3) n estimated in the "rst image, while CR(Q ) is the crossn ratio estimated for the candidate matches in the second image. The number of considered subsets N is S
A B
N" S
N feat 5
,
(7)
where N feat is the number of features extracted and candidated for the correspondence process. In our case, some correct initial matches, estimated through radiometric similarity in the match initialization step, satisfying the imposed constraint, will improve the optimization process. Once matches are initialized, they are re"ned by minimizing Eq. (6). Eq. (6) will be minimized only when its partial derivatives with respect to all features Mq N in the second image i I equal zero: 2 LE NS LCR(Q ) n "0, " + d ((CR(P )!CR(Q )) in n n Lq Lq i n/0 i
(8)
where d is a binary function assuming value 1 if the ith in feature q is in the nth subset Q . i n Satisfying the condition (8) for all q amounts to solve i a system on N feat simultaneous equation in N feat unknowns. Thus, the solution which minimizes the squared norm of the di!erence-vector (5) consists of "nding the set of points Mq N so that system (8) is solved. i i/1,2,N feat Since system (8) is nonlinear we must use an iterative approach to compute the optimal solution. Starting from some approximate matches, the algorithm improves the solution until a predetermined convergence criterion is satis"ed. Due to the nonlinearity of the system more than one solution can exist. The success to reach a global minimum, and not to be trapped in a local minimum, depends on having a good "rst-guess for the solution. The algorithm will converge to an optimal solution assuming that a number of input matches is correct enough. The goal is to reject the noise introduced by the correlation measurements. The proposed approach converges iteratively to the correct correspondence points Mq N by implementing i gradient descent along the E(q ) surface. i Optimal correspondences are estimated by updating iteratively the initial candidate matches using the follow-
469
ing updating rule: LE *q " (1!R ), i Lq i i
(9)
where the R 3[0, 1] term, representing the radiometric i similarity, has been introduced to avoid that, in the cooperative process, bad matches in#uence the correct ones. However, correct measurements are not in#uenced by erroneous estimates, because for them the updating (*q ) is zero, being satis"ed the imposed constraints. On i the other hand, mismatches are in#uenced by correct matches determining the noise rejection.
5. Experimental results The approach has been tested on a number of image sequences with the aim to verify the performances of the algorithm to detect bad matches generated by the match initialization step and at the same time to correct them. 5.1. Quantitative error analysis A "rst set of experiments has been performed using the synthetic sequence div-tree, a test sequence presented in Ref. [1] to evaluate the performances of several optic #ow estimation techniques. For this sequence the groundtruth 2D motion "eld is known. Thus a quantitative error analysis can be performed. In particular, based on Ref. [1], the angular error between the true 2D velocity vector v "(x , y ) and the estimated vector v "(x , y ) c c c e e e has been computed as t "arccos(vN ) vN ), (10) E c e where the two vectors, v "(x , y ) and v "(x , y ) are c c c e e e represented as 3D direction vectors: 1 , vN " c Jx2#y2#1 ) (x , y , 1)T c c c c 1 vN " . (11) e Jx2#y2#1 ) (x , y , 1)T e e e e The sequence has been generated by moving a synthetic camera along its line of sight towards a textured tilted plane. The focal length is 16 mm and the velocity is constant over the sequence and equal to 0.2 focal lengths per frame. The 2D motion "eld varies from 1.29 pixels/frame on the left side to 1.86 pixels/frame on the right. Experiments have been performed by considering the frames from the 20th to the 40th (Fig. 3). In Fig. 4(a) graphical representation of the average angular error, estimated as speci"ed in Eq. (10), and the standard deviation between the actual 2D motion "eld and the computed
470
A. Branca et al. / Pattern Recognition 33 (2000) 465}481
Fig. 3. Frame 20th(a) and 40th(b) of the div-tree sequence.
Fig. 4. Average error (a) and standard deviation (b) of #ow "elds computed varying the velocity (focal length per frame) of the camera moving towards a textured tilted plane (Div-Tree sequence).
#ow "eld is reported. The tests have been performed by varying the camera velocity from 0.2 focal lengths per frame (the original one) to 4.20, with the corresponding depth values Z for the middle varying from 12.6 (at velocity 0.2) to 4.6 (at velocity 4.2). Camera velocities have been obtained by extracting frames from the original sequence with steps varying from 1 (velocity 0.2) to 21 (velocity 4.2). In Table 1 the results of our method are compared with those of some 2D motion "eld estimation techniques tested in Ref. [1]. In particular, we have considered the two-#ow-based methods of Lucas [42] and Uras [43] and the two correspondence methods of Singh [44] and Anandan [9]. The motion "eld maps for these methods are estimated as described in Ref. [8], from each motion "eld the set of velocity vectors corresponding to the same feature points used in our method have been extracted for comparison. Though a feature-based method, like ours, is generally designed to work with large velocities and a gradientbased method is used to treat slow motions, the experimental results reported in Table 1 show that our method performs better than both gradient and correspondence methods when all velocities are greater approximately than 3 pixel/frame (i.e. 0.6 focal length per frame in our experiments). For smallest velocities, the correspondence methods of Singh and Anandan perform quite similarly to our method, while the gradient-based methods of Lucas and Uras perform better than our method, but require a too long temporal sequence (15 frames in our tests) to be applied in real experiments. In Fig. 5 the error estimates of Anandan method (whose performances seem to be most similar to ours for slow motions) are compared with those of our method.
A. Branca et al. / Pattern Recognition 33 (2000) 465}481
471
Table 1 Comparison of results. The average error and the standard deviation, in parentheses, are reported for di!erent optic #ow computation methods Velocity Lucas Uras Singh Anandan Our method
0.2 2.01(1.97) 4.87(3.03) 9.79(6.07) 15.49(11.52) 14.36(8.93)
0.4 1.82(1.35) 4.32(3.06) 4.81(4.22) 9.86(7.93) 12.36(12.25)
0.6 4.18(3.58) 7.34(8.22) 4.56(4.48) 7.05(6.34) 5.10(2.90)
1.2 64.72(74.98) 21.06(27.77) 18.36(28.91) 4.82(5.88) 3.40(2.16)
2.4
68.54(47.94) 47.79(49.92) 2.42(1.07)
deviation estimated for the initial motion "elds and the optimized ones (with respect to the correct motion "eld (Fig. 9(b)) are reported in Fig. 8. Some of the motion "elds analyzed in Fig. 8 are in Fig. 10. 5.2. Application to passive navigational motion
Fig. 5. Average error of #ow "elds computed varying the velocity (focal length per frame) of the camera moving towards a textured tilted plane (Div-Tree sequence). Comparison of our method with that of Anandan.
Finally, in Figs. 6 and 7 the reported experimental results } relative to two image sequences the divtree and the room-seq [45] (consisting of 11 images 120]120, the motion of the camera has a dominance of forward translation) * show the ability of the method to estimate new matches without initialization with correlation. The approach has been tested also with the aim to verify the ability of the algorithm to correct bad matches. In order to test the performance of the method in contexts where a high number of mismatches are provided by the correlation step, a number of experiments has been performed by adding noise with a normal distribution with mean 0 and standard deviation ranging from 0.5 to 3 to raw matches estimated by comparing two frames of the div-tree sequence acquired at velocity 2.4 (i.e. by considering the frames 19 and 31 (Fig. 9(a) of the sequence). The relative average error and standard
A set of experiments has been performed on images acquired in our laboratory with a TV camera mounted on a pan-tilt head installed on our vehicle SAURO. The focal length of the TV camera is 6 mm. The performances of the method have been evaluated by applying the approach in the particular context of Passive Navigational Motion. Passive navigation [46] is the ability of an autonomous agent to determine its motion with respect to the environment. In the context of vehicle navigation the most important goal is to recover the relative motion between the observer and the scene [47], in order to avoid collisions or to make on-line adjustments to the current navigational path. The two main egomotion parameters, that allow to perform the above tasks, are the Heading Direction and the Time To Collision (TTC). Our aim is to test our method by evaluating the error in egomotion parameters estimation from a 2D motion "eld computed as described above. In our experiments we use the heading direction of a vehicle moving in a stationary environment to make on-line adjustments to the current navigational path. The analysis of the error performed in this process will give an estimate of the good quality of our 2D motion computation approach. 5.2.1. Heading direction estimation Assuming a perspective projection model in which a world point P"(X, >, Z) projects on the image point (x, y)"f (X/Z, >/Z), where f is the focal length, Longuet-Higgins and Prazdny [48] derived the following equations to describe the general rigid motion of an
472
A. Branca et al. / Pattern Recognition 33 (2000) 465}481
Fig. 6. Experimental results relative to the div-tree sequence. Frames 1 and 10. (a) Flow estimated through correlation. Initial matched points: 13. (b) Flow estimated after minimization. Sparse points: 20. (c) Energy function minimization.
observer moving in a stationary world: ¹ !x¹ z!xyR #(1#x2)R !yR , u" x x y z Z(x, y) ¹ !y¹ z!(1#y2)R #xyR #xR v" y x y z Z(x, y)
(12)
with (¹ , ¹ , ¹ ) the 3D translational velocity compox y z nents, (R , R , R ) the 3D rotational velocity compox y z nents, (u, v) the projected velocity of a point (X, >, Z) on
the image plane. In the passive navigation context the viewer can translate on a #at ground and rotate only around an axis normal to the ground, the resulting 2D motion "eld has a radial topology: on the image plane all 2D velocity vectors radiate from a singular point (that is the point where the #ow vanishes) called focus of expansion (FOE). The FOE is the projection on the image plane of the direction along which the observer moves. A rotation, occurring while the observer translates, will cause a FOE location shifting by always preserving the
A. Branca et al. / Pattern Recognition 33 (2000) 465}481
473
Fig. 7. Experimental results relative to the room-seq sequence. Frames 1 and 10. (a) Frame 1 of the sequence with overimposed features. (b) Energy function minimization. (c) Flow estimated through correlation. Initial matched points: 18. (d) Flow obtained after minimization. Sparse points: 25.
radial shape. The FOE location can be correctly estimated as the point where the translational motion vanishes.
A
B
¹ ¹ x, y . (FOE , FOE )"f x y ¹ ¹ z z
(13)
In our experiments we estimate the heading direction of the vehicle moving in a stationary world by computing
the FOE location from the 3D translational motion parameters, estimated as described in Ref. [49] from the 2D displacement vector computed through our approach. In Fig. 11 two images acquired during the experiments and the computed displacement "eld with the estimated FOE location are shown. These two images have been acquired under translational motion on a rectilinear path of the vehicle with the TV camera optical axis aligned
474
A. Branca et al. / Pattern Recognition 33 (2000) 465}481
Fig. 8. Average error (a) and standard deviation (b) of #ow "elds obtained by adding noise (with a normal distribution with mean zero and standard deviation varying from 0 to 3) to all 2D displacement vectors estimated through correlation. The camera velocity is 2.4 focal length per frame.
with the direction of translation. While the images in Figs. 12 and 13 have been acquired under a curvilinear motion of the vehicle. Our aim is to use the estimated heading direction to correct the vehicle motion. The strategy consists of acquiring two images at times t and t#dt, to estimate the FOE location and performing a correction of the current path. This requires to known a priori the relation between the FOE and the motion of the vehicle. To do so, a set of tests have been performed in order to align the TV camera with the direction of translation of the vehicle and to relate the FOE location to the actual motion of the vehicle. In particular, "rst the vehicle has been constrained to translate along a rectilinear path, while the camera is rotating around the axis normal to the ground plane. For di!erent degrees of camera rotation the same path was repeated. The obtained results are reported in Fig. 14. We can observe that the FOE location changes linearly with the camera rotation angle and the ranges of variation of FOE location at di!erent angles are well separated. The results so obtained have been used to align the optical axis of the camera with the direction of translation of the vehicle. Then, through a number of experiments constraining the vehicle to move on di!erent curvilinear paths, a relation between FOE location and vehicle motion has been found. In Fig. 15 the results obtained for di!erent degrees of vehicle rotation are reported. We can observe that in this case a small rotation of the vehicle corresponds to the highest shift of the FOE location, because vehicle and camera rotate at the same time. b Fig. 9. (a) Frame 31 of div-tree sequence. (b) Actual motion "eld between the 19th and the 31st frames.
A. Branca et al. / Pattern Recognition 33 (2000) 465}481
475
Fig. 10. Flow "elds relative to div-tree sequence computed by considering the frames 19 and 31 (i.e. velocity 2.4). (a1) Raw #ow "eld and (b1) Raw #ow "eld with added random noise with normal distribution with mean 0 and standard deviation 3. (a2) Flow after the minimization of (a1). (b2) Flow after the minimization of (b1).
Once the camera has been aligned with the vehicle motion and a mapping between FOE location and vehicle motion has been found, a "nal set of experiments have been performed to evaluate the performance of the vehicle to make on-line adjustments to a current
known navigational path by using the FOE location estimates. This set of experiments have been performed by constraining the vehicle (a) to deviate from a known path by performing small rotations, (b) to estimate such rotation
476
A. Branca et al. / Pattern Recognition 33 (2000) 465}481
Fig. 11. The images have been acquired while the vehicle is translating on a rectilinear path, the distance between the two frames is 500 mm. Estimated FOE: x"!10, y"15.
and (c) to drive the vehicle in order to realign it with the prede"ned path. The average error in estimation of the deviation from the prede"ned path has been evaluated to be approximately of 0.33 with a variance of 0.1.
6. Conclusions The paper proposes a new technique to estimate a robust sparse correspondence map among the features
extracted from images acquired at di!erent times. The approach is based on iterative re"nements of matches initialized through correlation, by minimizing a cost function formulated by including constraints based on the projective invariance of cross-ratio of "ve coplanar points. Though the method is based on planarity constraints, no assumption about coplanarity of considered features is imposed and the application is not limited to contexts where the planarity is veri"ed in the whole image. As shown by the results of the experiments the system works well in di!erent contexts without any
A. Branca et al. / Pattern Recognition 33 (2000) 465}481
477
Fig. 12. The images have been acquired while the vehicle is moving on a curvilinear path, the distance between the two frames is 500 mm, the rotation angle is 13. Estimated FOE: x"126, y"!1.
knowledge about feature coplanarity. It only requires that a number of distinct planar regions to be presented in the scene. The future work will involve the use of functional value in di!erent subsets to perform a clustering of the features belonging to di!erent planar regions.
7. Summary In the context of intelligent mobile robot navigation an important role is played by arti"cial vision. In order to interact with its environment, a robot should be able to process dynamic images for recovering its motion and the structure of environment. It is well known that a signi"cant amount of useful information about 3D
motion and structure can be obtained from the 2D image motion resulting from the projection on the image plane of camera and 3D objects movements. Though a well approximation of 2D image motion can be obtained employing spatial and temporal image gradients (#owbased methods), often, in some contexts, more robust 3D informations are recovered from some correspondance established between signi"cant features extracted from the temporal sequence images (feature-based methods). In the present work a more simple and general featurebased approach to solve the correspondance problem between 2D projected feature points is proposed. The correspondance problem is formulated as an optimization problem constrained by cross-ration invarience of "ve arbitrary coplanar points. Our goal is to de"ne an
478
A. Branca et al. / Pattern Recognition 33 (2000) 465}481
Fig. 13. The images have been acquired while the vehicle is moving on a curvilinear path, the distance between the two frames is 500 mm, the rotation angle is 23. Estimated FOE: x"150, y"!2.
algorithm to estimate correspondances between arbitrary features extracted from successive images imposing the `cross-ratio invarience of coplanar pointsa as constraint between all considered features. The proposed feature-based approach solves the correspondance problem by minimizing an appropriate energy function where constraints on radiometry similarity and projective geometric invarience of coplanar points are de"ned. The method can be considered as a correlationbased approach which take into account the projective invariance of coplanr points in computing the optimal
matches. Cross-ratio invariance is imposed to correct (or to optimize) the matches computed using a correlationbased approach. Features in the second frame are selected as measurements satisfying some imposed constraints included in an appropriate energy functional. In other words, extraction of features in the second frame and match computation are considered a whole process consisting in an energy function minimization. Though the method is based on planarity constraints and no asssumption about coplanarity of considered features is made, it is not limited to the application in
A. Branca et al. / Pattern Recognition 33 (2000) 465}481
Fig. 14. The estimated FOE locations indicate the shift of the FOE when the vehicle translates on a rectilinear path with the optical axis of the camera rotated of di!erent degrees with respect to the direction of translation.
479
The method we use take advantage by considering many intersecting subsets of "ve points obtained as combinations of available sparse features. If the features are on di!erent planes the minimization approach constraints only coplanar features to in#uence each other, though subsets include also non-coplanar features. Through a coperativity process, information is propagated between subsets and, due to the imposed radiometric similarity, matches between not coplanar features are avoided. Once high variance features are extracted in the "rst frame using the Moravec's interest operator, raw matches, estimated through radiometric similarity, are updated iteratively in order to estimate the correct corresponding features in the second image that minimize a cost function. The new contribution of this work consists in the matching process based on re"nement of raw measurements, in the energy function minimization technique converging to an optimal solution by taking advantage from some good initial guess, and in the use of cross-ratio as geometrical invariant constraint to detect mismatches obtained through radiometric similarity measures and at same time to correct them. Though the method is based on geometrical invarience of coplanar points, it is not required all matching features to be coplanar or to prepocessing the images for segmentation in planar regions. Experimental results are presented for real and synthetic images, and the performance of the novel algorithm is evaluated on di!erent image sequences using a quantitative error estimation.
References
Fig. 15. Heading direction estimates of the vehicle moving on curvilinear paths with di!erent rotations. The optical axis of the camera is aligned with the direction of translation.
contexts where the planarity is veri"ed in the whole image } i.e. the distance camera-scene is lare or in videobased motion analysis * but it works well also in more general contexts where a number of distinct planar regins are presented.
[1] J.L. Barron, D.J. Fleet, S.S. Beauchemin, T.A. Burkitt, Performance of optical #ow techniques, Proceedings of the Conference on Computer Vision and Pattern Recognition Champaign, IL, 1992, pp. 236}242. [2] J.Y. Aloimonos, Z. Duric, C. Fermuller, L. Huang, E. Rivlin, R. Sharma, Behavioral visual motion analysis, Proceedings of DARPA Image Understanding Workshop, Jan. 1992, pp. 521}533. [3] J. Heel, Direct estimation of structure and motion from multiple frames, AI Memo 11190, MIT Arti"cial Intelligence Laboratory, March 1990. [4] S. Negahdaripour, B.K.P. Horn, Direct passive navigation, IEEE Trans. PAM 9 (1) (1987) 168}176. [5] D. Sinclair et al. Robust estimation of egomotion from normal #ow, Int. J. Comput. Vision 13 (1) (1994) 57}69. [6] L. Dreschler, H. Nagel, Volumetric model and 3D trajectory of a moving car derived from monocular TV frame sequences of a street scene, Comput. Graphics Image Process. 20 (1982) 199}228. [7] E.C. Hildreth, The Measurement of Visual Motion, MIT Press, Cambridge MA, 1983.
480
A. Branca et al. / Pattern Recognition 33 (2000) 465}481
[8] W. Ho!, N. Ahuja, Surfaces from stereo: integrating feature matching, disparity estimation, and contour detection, IEEE Trans. Pattern Anal. Mach. Intell. 11 (2) (1989) 121}136. [9] P. Anandan, A computational framework and an algorithm for the measurement of visual motion, Int. J. Comput. Vision 2 (3) (1989) 283}310. [10] J. Weng, N. Ahuja, T.S. Huang, Matching two perspective views, IEEE Trans. Pattern Anal. Mach. Intell. 14 (8) (1992) 806}825. [11] N. Ayache, in: Arti"cial Vision for Mobile Robots, MIT Press, Cambridge MA, 1991. [12] H.P. Moravec, The Stanford Cart and the CMU Rover, Proceedings of the IEEE, 1983. [13] D. Marr, T. Poggio, A computational theory of human stereo vision, Proc. Roy. Soc. London B 204 (1979) 301}328. [14] P. Fua, Combining stereo and monocular information to compute dense depth maps that preserve depth discontinuities, Proceedings IJCAI '91. [15] J. Martin, J.L. Crowley, Comparison of correlation techniques, in: U. Rembold, et al. (Eds.), Intelligent autonomous systems, IOS Press, Karlsruhe, 1995, pp. 86}93. [16] V. Venkateswar, R. Chellappa, Hierarchical stereo and motion correspondence using feature groupings, Int. J. Comput. Vision 15 (1995) 245}269. [17] S. Sull, N. Ahuja, Integrated matching and segmentation of multiple features in two views, Comput. Vision Image Understanding 62 (3) (1995) 279}297. [18] I.K. Sethi, R. Jain, Finding trajectories of feature points in a monocular image sequence, IEEE Trans. Pattern Anal. Mach. Intell. 9 (1987) 56}73. [19] M.J. Fletcher, K. Warwick, R.J. Mitchell, The application of a Hybrid tracking algorithm to motion analysis, IEEE Proceedings of the CVPR 1991, pp. 84}89. [20] K. Rangarajan, M. Shah, Establishing motion correspondence, IEEE Proceedings CVPR 1991, 103}108. [21] S.S. Vincent, X. Hwang, Tracking feature points in timevarying images using an opportunistic selection approach, Pattern Recognition 22 (1992) 247}256. [22] Q. Zheng, R. Chellappa, Point correspondence and motion detection in long image sequence, IEEE Proceedings of the CVPR '93. [23] C.L. Cheng, J.K. Aggarwal, A two stage approach to the correspondence problem via forward-searching and backward-correcting, Proceedings of the 10th International Conference on Pattern Recognition Atlantic City, NJ, 16}21 June 1990, pp. 173}177. [24] J.L. Mundy, A. Zisserman (Eds.), Geometric Invariance in Computer Vision MIT Press, Cambridge, MA, USA, 1992. [25] L. Quan, Invariants of six points and projective reconstruction from three uncalibrated images, IEEE Trans. Pattern Anal. Mach. Intell. 17 (1) (1995). [26] G. Sparr, An algebraic/analytic method for reconstruction from image correspondence, Proceedings of the seventh Scandinavian Conference on Image Analysis, 1991. [27] O. Faugeras, What can be seen in 3D with an uncalibrated stereo rig? Proceedings of ECCV '92. [28] R. Hartley, R. Gupta, T. Chang, Stereo from uncalibrated cameras, Proceedings of CVPR '92.
[29] I. Weiss, Projective invariants of shapes, Proceedings of DARPA Image understanding workshop, Pittsburgh, Pennsylvania, USA, 1990, pp. 650}659. [30] D. Forsyth, J.L. Mumndy, A. Zisserman, C. Coelho, A. Heller, C. Rothwell, Invariant descriptors for 3D object recognition and pose, IEEE Trans. Pattern Anal. Mach. Intell. 13 (10) (1991) 971}991. [31] U. Uenorhara, T. Kanade, Geometric invariants for veri"cation in 3D object tracking, Proceedings of IROS '96. [32] D. Sinclair, A. Blake, Qualitative planar region detection, Int. J. Comput. Vision 18 (1) (1996) 77}91. [33] H. Chabbi, M.O. Berger, Using projective geometry to recover planar surfaces in stereovision, Pattern Recognition 29 (4) (1996) 533}548. [34] S. Carlsson, Projectively invariant decomposition and recognition of planar shapes, Int. J. Comput. Vision 17 (2) (1996) 193}209. [35] C.A. Rothwell, A. Zisserman, D.A. Forsyth, J.L. Mundy, Planar object recognition using projective shape representation, Int. J. Comput. Vision 16 (1995) 57}99. [36] D. Oberkampf, D.F. DeMenthon, L.S. Davis, Iterative pose estimation using coplanar feature points, Comput. Vision Image Understanding 63 (3) (1996) 495}511. [37] K. Kanatani, Computational cross ratio for computer vision, CVGIP: Image Understanding 60 (3) (1994) 371}381. [38] S.J. Maybank, Probabilistic analysis of the application of cross ratio to model based vision: misclassi"cation, Int. J. Comput. Vision 14 (1995) 199}210. [39] S.J. Maybank, Probabilistic analysis of the application of cross ratio to model based vision, Int. J. Comput. Vision 16 (1995) 5}33. [40] A. Blake, A. Zisserman, Visual Reconstruction, MIT Press, Cambridge, MA, 1987. [41] O. Axelsson, V.A. Barker, Finite Element Solution of Boundary Value Problems: Theory and Computation, Academic Press, New York, 1984. [42] B.D. Lucas, T. Kanade, Optical navigation by the method of di!erences, International Joint Conference on Arti"cial Intelligence, 1985, pp. 981}984. [43] S. Uras, F. Girosi, A. Verri, V. Torre, A computational approach to motion perception, Biol. Cybernet. 60 (1989) 79}87. [44] A. Singh, Optic #ow computation: a uni"ed perspective, IEEE Press, 1990. [45] H.S. Sawhney, A.R. Hanson, Trackability as a cue for potential obstacle identi"cation and 3D description, Int. J. Comput. Vision 11 (3) (1993) 237}263. [46] C. Fermuller, Passive navigation, Int. J. Comput. Vision 14 (2) (1995) 147}158. [47] C. Fermuller, Qualitative egomotion, Int. J. Comput. Vision 15 (1/2) (1995) 7}29. [48] H.C. Longuet-Higgins, K. Prazdny, The interpretation of a moving retinal image, Proc. Roy. Soc. London Ser. B 208 (1980) 385}397. [49] A. Branca, G. Convertino, E. Stella, A. Distante, A neural network for egomotion estimation from optical #ow, British Machine Vision Conference, Birmingham, UK, September 1995.
A. Branca et al. / Pattern Recognition 33 (2000) 465}481
481
About the Author*ANTONELLA BRANCA was born in Poggiardo (Lecce-Italy) in 1968. She received the degree in Computer Science from the University of Bari in 1992. Since 1993 she has worked as a research associate at the Institute for Signal and Image Processing of Italian National Research Council. Her areas of interest include computer vision, arti"cial neural networks, and robotics. About the Author*ETTORE STELLA was born in Bari (Italy) in 1960 and received the degree in Computer Science from Bari University in 1984. From 1987 to 1990 he was a research scientist at the Italian Space Agency (ASI) at the Centro di Geodesia Spaziale (CGS) in Matera (Italy). Since 1990 he has been a research scientist at IESI-CNR. His current interests are in computer vision, planning, neural networks, control of a mobile robot operating in indoor environment and obstacle avoidance. About the Author*ARCANGELO DISTANTE was born in Francavilla Fontana (Brindisi-Italy) in 1945. He received the degree in Computer Science at the University of BARI, Italy in 1976. Until 1981 he worked for I.N.F.N. (National Nuclear Physics Institute) and subsequently for IESI-CNR. Currently, he is the coordinator of the Robot Vision Group at IESI-CNR and director of the IESI Institute. His research interests are focused on 3D object reconstruction, representation of visual information and generation of 3D modelling, shape representation for image understanding, vision for robotic navigation, and architectures for computer vision. Dr. A. Distante is a member of the IAPR and SPIE.
Pattern Recognition 33 (2000) 483}501
Localizing a polyhedral object in a robot hand by integrating visual and tactile dataq Michael Boshra*, Hong Zhang Department of Computing Science, University of Alberta, Edmonton, Alberta, Canada T6G 2H1 Received 28 September 1998; received in revised form 2 March 1999; accepted 2 March 1999
Abstract We present a novel technique for localizing a polyhedral object in a robot hand by integrating visual and tactile data. Localization is performed by matching a hybrid set of visual and tactile features with corresponding model features. The matching process "rst determines a subset of the object's six degrees of freedom (DOFs) using the tactile feature. The remaining DOFs, which cannot be determined from the tactile feature, are then obtained by matching the visual feature. A couple of touch and vision/touch-based "ltering techniques are developed to reduce the number of model feature sets that are actually matched with a given scene set. We demonstrate the performance of the technique using simulated and real data. In particular, we show its superiority over vision-based localization in the following aspects: (1) capability of determining the object pose under heavy occlusion, (2) number of generated pose hypotheses, and (3) accuracy of estimating the object depth. ( 2000 Pattern Recognition Society. Published by Elsevier Science Ltd. All rights reserved. Keywords: 3D object recognition; Pose estimation; Visual data; Tactile data; Sensor integration; Robot hand; Object manipulation
1. Introduction Determining the pose of an object in a robot hand is important in many robotic tasks that involve manipulation of objects (e.g., automated assembly). Such pose information is needed to close the feedback loop of the robot controller. This enables the controller to intelligently react to undesirable object movements, which can be due to slippage or collision, for example. There are two types of sensory data that can be utilized in the task of localizing an object in a robot hand: visual and tactile. The visual data can be provided by a visual sensor monitoring the robot workspace, while the tactile
* Corresponding author. B-232 College of Engineering, University of California, Riverside, CA 92521, USA. Tel.: #1-909787-3954; fax.: #1-909-787-3188 E-mail address:
[email protected] (M. Boshra) q This research was partially supported by the National Science and Engineering Research Council of Canada.
data can be obtained from tactile sensors mounted on the hand. These types of data have di!erent characteristics. Firstly, visual data are relatively global; i.e., they capture the visible part of the object as seen from the visualsensor viewpoint. Unfortunately, visual data provide only a 2D projection of the 3D world. Thus, there is a substantial loss of 3D information. Tactile data, on the other hand, provide 3D information about the object, but unfortunately these data are local and in many cases insu$cient to localize the object. Secondly, visual sensing is sensitive to occlusion of the object by visual obstacles (e.g., the robot hand), a problem that is not experienced by tactile sensing. In fact, tactile data often provide information about visually occluded parts of the object. Thirdly, it is generally not guaranteed whether a given visual feature (e.g., an edge or a junction) belongs to the model object. This is not the case with tactile features, since they come from direct contact with the object. From this comparison, summarized in Table 1, it is reasonable to expect that by integrating both types of sensory data, more robust object localization can be achieved.
0031-3203/00/$20.00 ( 2000 Pattern Recognition Society. Published by Elsevier Science Ltd. All rights reserved. PII: S 0 0 3 1 - 3 2 0 3 ( 9 9 ) 0 0 0 5 9 - X
484
M. Boshra, H. Zhang / Pattern Recognition 33 (2000) 483}501
Table 1 A comparison between visual and tactile data Aspect
Visual data
Tactile data
Scope Dimension Occlusion Pertinence
Global 2D Applicable Not guaranteed
Local 3D n/a Guaranteed
The problem of 3D object recognition1 has received signi"cant attention during the last two decades (e.g., see surveys [1}4]). Most 3D object recognition systems rely on a single type of sensory data such as vision [5}10], range [1,11}13], or touch [14}16]. Thus, they are unsuitable for utilizing various types of sensory data which may be readily available in some tasks, such as ours, in order to improve e$ciency and robustness. There have been few e!orts for integrating visual and tactile data in the context of 3D object recognition. Luo and Tsai [17] use a pre-compiled decision tree to recognize 3D objects on a plane. The "rst level of this tree utilizes moment invariants of the object silhouette, while the latter levels use tactile data to completely discriminate between model objects. The tactile data are provided by two tactile sensing arrays mounted on a parallel-jaw gripper that approaches the object at pre-determined directions. Allen [18] uses passive stereo vision to guide a tactile sensor, mounted on a robot arm, to explore and construct a partial 3D description of the scene object. This description is then matched with model objects to recognize the scene one. Ambiguity is resolved by further tactile exploration of the scene object. These two approaches assume that the object to be recognized is not grasped by a robot hand, and so they are unsuitable in our case. Perhaps the most relevant technique to ours is the one presented by Browse and Rodger [19]. They propose a system to recognize 3D objects with three discretized degrees of freedom (one rotational and two translational). For each visual or tactile feature, the discrete set of consistent object/pose hypotheses is generated. These sets are then intersected to identify the object and determine its pose. Limitations of this approach are recognition of objects with only three degrees of freedom, and limited accuracy due to discretization of the pose space. In this paper, we present a novel technique for localizing a polyhedral object in a robot hand by integrating visual and tactile data. These data are assumed to be provided by a monocular visual sensor and a square planar-array tactile sensor. Hypotheses about the object pose are generated by matching a hybrid set of visual
and tactile features with corresponding model ones. The scene feature set consists of a visual junction, of any number of edges, and a tactile feature. For the sake of presentation, we consider two types of tactile features resulting from the following cases of tactile contact: (1) The object surface in contact with the tactile sensor totally covers the sensor array (see Fig. 1a), and (2) the contact surface partially covers the sensor and only one edge appears in the sensor array (see Fig. 1b). We refer to the polygonal patches resulting from these two contacts as S-patch and SE-patch, respectively. These two types of tactile features correspond to model surface and surfaceedge pair, respectively. We refer to these model features as S-surface and SE-surface, respectively. The matching process "rst determines a subset of the object's six degrees of freedom (DOFs) using the tactile feature. The remaining DOFs, which cannot be determined from the tactile feature, are then obtained by matching the visual feature. In addition, we present a couple of "ltering techniques to reduce the number of model feature sets that are actually matched with a given scene feature set. One technique utilizes touch-based constraints on the 3D space occupied by the object. The other one uses vision/ touch-based constraints on a number of transformationinvariant attributes associated with the model set. The proposed vision/touch-based localization technique has several advantages over common vision-based localization techniques. Firstly, it is capable of determining object pose in heavily occluded visual images that are problematic to vision-based methods. Secondly, the average number of generated hypotheses, per scene feature set, is considerably less than the number of those generated visually. This reduces the computational load on subsequent processes that verify the generated hypotheses (e.g. [8,20}22]). Thirdly, the accuracy of estimating the object depth (with respect to the visual sensor) can be signi"cantly better when vision is integrated with touch. These advantages are demonstrated experimentally in Section 4. The rest of the paper is organized as follows. In the next section, we describe the technique used to generate pose hypotheses in detail. Touch- and vision/touch-based "ltering techniques are presented in Section 3. In Section 4, we present experimental results using simulated and real data, and "nally, in Section 5, we provide conclusions.
2. Pose estimation In this section, we present the technique used to generate pose hypotheses by integrating visual and tactile data. 2.1. Overview
1 We assume that object recognition involves determining both identity and pose of objects.
We represent the 3D pose of an object, O, with respect to a coordinate frame, =, by three rotations about the
M. Boshra, H. Zhang / Pattern Recognition 33 (2000) 483}501
485
Fig. 1. Cases of tactile contact and associated tactile-sensor frames: (a) surface contact (S-patch): (1) origin: center of the patch, (2) z-axis: outward normal of the model surface in contact with the patch, (3) x- and y-axes: placed arbitrarily in the tactile-patch plane, (b) surface-edge contact (SE-patch): (1) origin: mid point of the edge, (2) z-axis: outward normal of the model surface in contact with the patch, (3) y-axis: normal to the edge toward the interior of the patch, (4) x-axis: along the edge such that the right-hand rule is satis"ed.
principal axes of =, followed by three translations along the same axes. The transformation matrix, WT , correO sponding to the 3D pose of O with respect to = can be written as follows [23]:
Js, with a model junction, Jm, after transforming it by iT to form iJm. According to our choice of tactile-sensor t frame (Fig. 1), iT can be expressed as follows: t
WT "¹rans(x, t )¹rans(y, t )¹rans(z, t ) O x y z ]Rot(z, h )Rot(y, h )Rot(x, h ) z y x
i T "¹rans(z, t )Rot(y, h )Rot(x, h ) s iz iy ix for S-surface S i iT " t i T "¹rans(y, t )Rot(z, h )jT se iy iz s for SE-surface (S , E ).2 j i
A
B
ch ch ch sh sh !sh ch ch sh ch #sh sh t z y z y x z x z y x z x x sh ch sh sh sh #ch ch sh sh ch !ch sh t z y x z x z y x z x y , " z y !sh ch sh ch ch t y y x y x z 0 0 0 1 (1)
where Rot(u, h ) is the rotation matrix of angle h about u u the u-axis, ¹rans(u, t ) is the translation matrix of t along u u the u-axis (u"x, y or z), ch"cos(h), and sh"sin(h). We consider a tactile-sensor coordinate frame as the reference frame. As will be seen later, this choice simpli"es the processes of pose computation and "ltering. Furthermore, it enables the computation of the touchbased DOFs o!-line, thereby reducing the on-line computational requirements. Figs. 1a and b show the chosen tactile-sensor frames for S- and SE-patches, respectively. An object pose hypothesis consists of two components representing the DOFs determined by matching visual and tactile features. In particular, we have TT "T iT , (2) O v t where TT is a pose expressed with respect to a tactileO sensor frame, iT is a partial pose determined by matcht ing a tactile feature with the ith model feature, and T is v a partial pose determined by matching a visual junction,
G
(3)
Notice that matching a tactile S-patch with a model S-surface determines three DOFs (h , h , t ), while matchx y z ing an SE-patch with an SE-surface determines additional two DOFs (h , t ). From Eqs. (1)}(3), it is easy to z y see that T can be expressed as v
G
¹rans(x, t )¹rans(y, t )Rot(z, h ) x y z for S-patch/S-surface match
T" v ¹rans(x, t ) x for SE-patch/SE-surface match.
(4)
Thus, the vision-based matcher has to determine three DOFs in the S-patch case (h , t , t ), and only a single z x y DOF in the SE-patch case (t ). x The matching process consists of two o!- and on-line stages. These stages can be outlined as follows: f Ow-line stage: The set of touch-based partial pose hypotheses, T "MiT N, is computed for each type of t t tactile features (S- and SE-patches). Model junctions, Jm, are transformed by each partial hypothesis,
2 To simplify the notation, we assume that the edge index, i, uniquely identi"es SE-surface features.
486
M. Boshra, H. Zhang / Pattern Recognition 33 (2000) 483}501
iT 3T , to form set iJm. The set of all partially transt t formed junction sets is stored in a junction database, Jm "6i iJm. db f On-line stage: one or more scene feature sets are considered for pose generation. Each set consists of a tactile feature and a visual junction, Js, where Js is expressed with respect to the frame associated with the tactile feature. Pose hypotheses are generated by matching Js with partially transformed junctions in the junction database, Jm . A partially transformed juncdb tion, iJm3Jm , is matched with Js to determine fJm, db a hypothesis about the pose of junction Jm in the scene (given a match between the tactile feature and the ith model feature). In addition, this process determines T , v the vision-based partial pose that complements iT , as t
de"ned in Eq. (2). The number of model junctions actually matched with Js is signi"cantly reduced by using two "ltering techniques (Section 3). Figs. 2a and b illustrate the overall matching process for the S- and SE-patch cases, respectively. 2.2. Determination of touch-based DOFs In this section, we derive the DOFs that can be obtained by matching a tactile feature with a corresponding model one. First, we derive the three DOFs that can be determined in the S-patch case (h , h , t ). The equation of x y z model S-surface S , with respect to the object coordinate i
Fig. 2. An illustration of the matching process using visual and tactile features: (a) S-patch case, (b) SE-patch case. Note that c is the camera viewpoint.
M. Boshra, H. Zhang / Pattern Recognition 33 (2000) 483}501
Fig. 3. Transformation of S-surface S to the tactile S-patch: i (a) after rotation about the x- and y-axes, (b) after translation along the z-axis.
487
Fig. 4. Transformation of model SE-surface (S , E ) to the tactile i j SE-patch: (a) after rotation about the z-axis, (b) after translation along the y-axis.
frame, can be represented as (5) nix"d , i where ni is the outward normal of the surface, and d is i the distance from the origin to the surface in the direction of ni (see Fig. 3a). According to our choice of the tactilesensor frame (Fig. 1a), the equation of S after the "nal i transformation is zL x"0,
(6)
where zL , the direction vector along the positive direction of the z-axis, is the outward normal of the surface after transformation. From Eqs. (5) and (6), we observe that the outward normals of S before and after the "nal i transformation are ni and zL , respectively. From Eq. (1), these normals are related by zL "Rot(z, h )Rot(y, h )Rot(x, h )ni. (7) iz iy ix Pre-multiplying both sides of Eq. (7) by Rot~1(z, h ), we iz get zL "Rot(y, h )Rot(x, h )ni. iy ix Expanding Eq. (8), we get the following equation:
AB A
sh sh iy ix ch ix 1 !sh ch sh iy iy ix From Eq. (9), we obtain 0
0 "
ch iy 0
AB
BA B
sh ch iy ix !sh ix ch ch iy ix h and h ix iy
n ix n . iy n iz as follows:
(8)
(9)
n h "arctan iy (10) ix n iz h "!arctan2(n , sh n #ch n ). (11) iy ix ix iy ix iz If n "n "0, when ni is parallel to the x-axis, then there iy iz will be an in"nite number of solutions for h . In this case, ix we arbitrarily set h to zero. ix After rotating S by h and h about the x- and y-axis, i ix iy respectively, the surface is now parallel to the x}y plane,
with the outward normal in the direction of zL (see Fig. 3a). Since rotation of a plane does not change its distance from the origin, a translation of !d along the i z-axis will translate S to the x}y plane (see Fig. 3b). That i is, t "!d . (12) iz i Next, we determine the two edge-dependent DOFs in the SE-patch case, h and t , which are associated with jz jy SE-surface (S , E ). After transformation by the three i j DOFs determined above in Eqs. (10)}(12), edge E of j surface S lies in the x}y plane. Thus, it can be represented i by the following 2D equation: nj x"d (13) j where nj is the normal of edge E that points to the j interior of surface S , and d is the distance from the origin i j to the edge in the direction of nj (see Fig. 4a). According to our choice of the tactile-sensor frame (see Fig. 1b), the 2D equation of E after the "nal transformation is j yL x"0, (14) where yL , the direction vector along the positive direction of the y-axis, is the edge normal that points to the interior of the surface. From Eqs. (13) and (14), the normals of edge E before and after rotation about the z-axis are j nj and yL , respectively. That is,
AB A
BA B
ch !sh n jz jz jx . (15) 1 sh ch n jz jz jy From Eq. (15), h is obtained as follows: jz h "arctan2(n , n ). jz jx jy After rotating edge E by h about the z-axis, the edge is j jz now parallel to the x-axis at distance d from it (see j Fig. 4b). Thus, a translation of !d along the y-axis will j 0
"
488
M. Boshra, H. Zhang / Pattern Recognition 33 (2000) 483}501
align edge E with the tactile edge. That is, j t "!d . jy j 2.3. Determination of vision-based DOFs In this section, we derive the DOFs that can be obtained by matching visual and model junctions. Given a partially transformed model junction from the junction database, iJm3Jm , and a visual junction, Js, the db
objective of the vision-based matcher is to compute the undetermined DOFs that transform iJm so as to match Js. As mentioned in Section 2.1. these DOFs are h , t and z x t for the S-patch case, and t for the SE-patch case. y x We de"ne the following: 1. Functions: dir(u, v)"direction vector from u to v, angle(u, v)"angle between u and v.
Fig. 5. An illustration of the matching process: (a) partially transformed model junction iJm, (b) visual junction Js, (c) a match between iJm and Js to form fJm.
M. Boshra, H. Zhang / Pattern Recognition 33 (2000) 483}501
489
Fig. 6. Matching in the S-patch case: (a) computing rotation, (b) computing translation.
2. Camera: c"camera viewpoint, f"camera focal length. 3. Line of sight: ¸(v)"3D line from c through image point v, ¸(v, a)"point on ¸(v) at a distance a from c, ¸(v , a)"u component of ¸(v, a) (u"x, y or z). u 4. Partially transformed model junction iJm (see Fig. 5a): ivm"junction vertex, iem"direction of the kth junction edge, k DiJmD"number of junction edges. 5. Visual junction Js (see Fig. 5b): vs"junction vertex, us "end vertex of the kth junction edge, k es "direction of the kth edge (es "dir(vs, us )), k k k DJsD"number of junction edges, ws"direction of line of sight ¸(vs) (ws"dir(c, v4)), Gs "half-plane formed by ¸(vs) and es , k k Ps "half-plane Gs translated to the origin, k k hs "normal of half-plane Ps (hs "ws]e4 ). k k k , The equation of half-plane Ps is k hskx"0.
(16)
Line of sight ¸(vs) can be represented as x"c#aws,
(17)
where a is the distance from c to x in the direction of ws. Model junction iJm is said to match visual junction Js, if it can be transformed by the undetermined DOFs, to form fJm, so that the following necessary conditions are satis"ed (refer to Fig. 5c):
1. The direction of each model edge, fem, lies inside half k plane Ps .3 k 2. The model vertex fvm lies on line of sight ¸(vs), and the distance from the camera viewpoint c to fvm is greater than the camera focal length f. The matching steps for the S- and SE-patch cases are presented in the next two sections. 2.3.1. The S-patch case In this case, we need to determine h , t and t . z x y Since the only unknown rotation angle is h , fem is z k con"ned to a cone that embeds iem, and whose axis k coincides with the z-axis. This cone, Cm, is parameterized k by h as follows: z
A
B
ch !sh 0 z z x" sh ch 0 iem. z z k 0 0 1
(18)
Also, fem is con"ned to lie inside half-plane Ps . Thus, it k k can be determined by the intersection of Cm and Ps , as k k shown in Fig. 6a. Solving Eqs. (16) and (18), we obtain an equation of the form: a sin(h )#b cos(h )"c, (19) z z where a"!hs iem #hs iem , b"hs iem #hs iem and kx ky ky kx kx kx ky ky c"!hs iem . Solving Eq. (19), we obtain the following kz kz 3 For fem to lie inside Ps , hs ) fem should be approximately 0, k k k k and ws]fem should be in the same direction as hs . k k
490
M. Boshra, H. Zhang / Pattern Recognition 33 (2000) 483}501
expressions for h : z
G
A
arcsin
h" z
c Ja2#b2
n!arcsin
A
c
B
!arctan2(b, a),
B
Ja2#b2
(20) !arctan2(b, a).
Notice that there is a solution, only if Dc/J(a2#b2)D)1. In addition, notice that there is a singularity when a"b"0. This occurs when either hs or iem is parallel to k k the z-axis. To avoid this case, we select the visual edge with the largest (a2#b2) for computing h . Each generz ated h is then veri"ed by utilizing the remaining visual z edges as follows. The direction of each remaining edge, iem, is rotated by h about the z-axis to form fem. The "rst l z l matching condition, outlined above, is then applied to determine whether to accept h . z For an accepted h , matching continues by computing z the translation parameters t and t as follows: The x y position of model vertex ivm after the "nal transformation, fvm, is con"ned to a plane, Qm, that passes through ivm and is parallel to the x}y plane (see Fig. 6b). Plane Qm is represented by the following equation: zL x"ivm. (21) z It is easy to see that ivm is determined by the intersection of plane Qm with line of sight ¸(vs). Solving Eqs. (17) and (21), we obtain a , the distance from c to fvm: L ivm!c z . (22) a " z L ws z Substituting Eq. (22) into Eq. (17), we get fvm:
A
B
fvm"c#a ws. (23) L Notice that fvm has to satisfy the second matching condition, for matching to continue. The relationship between ivm and fvm is
A
B AB
ch !sh 0 t z z x ch 0 ivm# t . fvm" sh z z y 0 0 1 0
tion vectors do not change with translation, we have fem"iem, for all k3[1, DJmD]. If any fem is not accepted by k k k the "rst matching condition, then matching fails. Otherwise, it proceeds by examining model vertex ivm. It is easy to see that the position of ivm after the "nal transformation, fvm, is con"ned to a line, Mm, that passes through ivm and is parallel to the x-axis. This line can be represented as x"ivm#axL ,
where xL is the direction vector along the positive direction of the x-axis. Thus, fvm is determined by the intersection of lines Mm and ¸(vs), if such intersection exists. To test for intersection in the presence of uncertainty, we compute the distance between Mm and ¸(vs), i.e., length of the common normal. If this distance is less than some threshold, then intersection is assumed to exist; otherwise matching fails. If there is an intersection, then fvm is approximated as follows. Let the common normal of ¸(vs) and Mm intersect them at x and x , L M respectively, and let a and a be their corresponding a's L M (see Fig. 7). The position of fvm can be approximated by substituting a into Eq. (27). That is, M fvm"ivm#a xL . M
(28)
The relationship between ivm and fvm and can be represented as fvm"ivm#t xL . x
(29)
From Eqs. (28) and (29), we directly obtain t : x t "a . x M
(30)
Note that fvm has to be accepted by the second matching condition for matching to succeed. It should also be noted that there is a singularity if line ¸(vs) is parallel to the x-axis, since the pair (x , x ) will not be unique. In L M this case, another visual junction should be examined.
(24)
From Eq. (24), we obtain t and t as follows: x y t "fvm!(ch ivm!sh ivm), (25) x x z x z y t "fvm!(sh ivm#ch ivm). (26) y y z x z y Notice that there is a singularity if ws "0 (see Eq. (22)). z This occurs when ¸(vs) is parallel to the x}y plane. In this case, matching fails, and so another visual junction should be examined. 2.3.2. The SE-patch case In this case, the only DOF that needs to be determined on-line is the translation along the x-axis, t . Since direcx
(27)
Fig. 7. Matching in the SE-patch case.
M. Boshra, H. Zhang / Pattern Recognition 33 (2000) 483}501
491
Fig. 8. Application of a volumetric constraint, i<: (a) i< is inconsistent with junction Js, (b) i< is consistent with junction Js.
Fig. 9. Volumetric constraints: (a) S-patch case, (b) SE-patch case.
3. Filtering techniques
3.1. Volumetric constraints
In this section, we describe a couple of "ltering techniques for reducing the number of model junctions that are actually matched with a visual junction. These techniques are especially important since the size of the junction database can be large. For a polyhedron with N surfaces, and N edges per surface, the total number S E of junctions in the junction database is of order O(N2N ) S E and O(N2N2) for S- and SE-patch cases, respectively. S E Section 3.1 presents a "ltering technique that utilizes touch-based volumetric constraints on the 3D space occupied by the object. Section 3.2 describes the other technique, which uses vision/touch-based constraints on transformation-invariant model attributes.
For each possible match between a tactile feature and the ith corresponding model feature, F , a volumetric i constraint, i<, is constructed o!-line. This constraint provides a bound on the 3D space occupied by the object, given that model feature F corresponds to the i tactile feature. The set of all volumetric constraints is stored in a volume database, V "Mi
492
M. Boshra, H. Zhang / Pattern Recognition 33 (2000) 483}501
excluded from matching. On the other hand, if all lines of sight intersect i< (e.g., see Fig. 8b), then junctions in iJm are considered for matching. This "ltering technique is very useful for eliminating erroneous visual junctions that are distant from the tactile sensor. Let O be a model object, and let iO be the object after transformation by iT (refer to Eq. (3)). In the S-patch t case, the partially-transformed object iO is constrained to move in the x}y plane such that the S-surface assumed to be in contact with the tactile sensor, S , totally covers the i sensor. Since h is not known o!-line, we can approximz ate volumes in this case by cylinders along the z-axis, as illustrated in Fig. 9a. Dimensions of these cylinders are derived in Appendix A. In the SE-patch case, the partially transformed object iO is constrained to move along the x-axis such that the model edge assumed to match the tactile edge totally covers it. Hence, we approximate volumes in this case by cuboids parallel to the principal axes (see Fig. 9b). Derivation of the cuboid dimensions is presented in Appendix A.
that lies inside half plane Ps can be represented by an k anti-clockwise rotation of vector ws about axis hs , where k the rotation angle, h, is in the range [03, 1803] (see Fig. 10). The rotation matrix corresponding to an anticlockwise rotation of angle h about axis k is expressed as follows [23]: Rot(k, h)"
A
B
k k vh#ch k k vh!k sh k k vh#k sh x x y x z z x y k k vh#k sh k k vh#ch k k vh!k sh , x y z y y z y x k k vh!k sh k k vh#k sh k k vh#ch x z y y z x z z
(31)
where vh"1!ch. Thus, the relationship between ws (h) k and ws can be expressed as ws (h)"Rot(hs , h)ws. k k
(32)
3.2. Constraints on transformation-invariant model attributes It is a common practice in many 3D object recognition systems to utilize transformation-invariant attributes of model feature sets, in order to reduce the number of scene/model matching processes (e.g., [13,16,24}26]). Given a scene feature set, a number of bounds on invariant model attributes are computed. A model set can be consistent with the scene set, only if its attributes lie within the respective scene bounds. Thus, a large number of incompatible model sets can be eliminated from matching using very few comparison operations. An index or a hash table on invariant model attributes can also provide direct access to the compatible model sets, thereby signi"cantly improving speed. In our case, the scene feature, a visual junction, is used to derive bounds on a set of invariant attributes of the model junctions in the junction database Jm . Notice that the tactile feature is db implicitly included in the process by choosing a tactilesensor reference frame. Now, let us determine the invariant attributes of a model junction, iJm. For an S-patch, since only h , z t and t need to be determined on-line, it is easy to x y show that angle(zL , iem) and ivm are invariant attributes. k z For an SE-patch, since the only DOF that needs to be determined on-line is t , it can be easily shown that x angle(xL , iem), angle(yL , iem), angle(zL , iem), ivm and ivm are k k k y z invariant attributes. Thus, for a scene junction, Js, the number of corresponding attribute bounds is (1#DJsD) and (2#3DJsD) for S- and SE-patch cases, respectively. Scene bounds on the angular model attributes can be determined as follows. The direction of a vector, ws (h), k
Fig. 10. Representation of vectors that lie inside half plane Ps . k
Fig. 11. Computing bounds on h: Bounds [amin, amax] and [amin, amax] correspond to the sections of ¸(vs) and ¸(us ) that lie k k k inside volume i<. In such a case, we have h3[hmin, hmax].
M. Boshra, H. Zhang / Pattern Recognition 33 (2000) 483}501
A tighter bound on h can be obtained by utilizing the volumetric constraints. Let [amin, amax] and [amin, amax] k k correspond to the sections of ¸(vs) and ¸(us ) that lie inside k i< (e.g., see Fig. 8b). Given that model feature F corresi ponds to the tactile feature, it can be shown that h is bounded by (see Fig. 11) hmin"angle(ws, dir(¸(vs, amin), ¸(us , amax))), k k hmax"angle(ws, dir(¸(vs, amax), ¸(us , amin))). k k Notice that these bounds are re-computed for every i< (consistent with Js). Thus, bounds corresponding to i< are used when "ltering junctions in iJm only. It can be seen that the bounds on angle(pL , ws (h)), where pL is subk stituted by xL , yL or zL , provide a constraint on invariant attribute angle (pL , iem) of model junctions that can be k consistent with Js. These bounds are obtained by determining the minimum and maximum values of angle (pL , iws (h)), subject to the constraint h3[hmin, hmax]. Details k of such a process are presented in Appendix A. Scene bounds on the distance model attributes can be determined as follows. From Eq. (17), it is easy to obtain the following bounds on attribute ivm (where u is subu stituted by either y or z): [min(¸(vs , amin), ¸(vs , amax)), max(¸(vs , amin), ¸(vs , amax))]. u u u u As in the angular-attribute case, these bounds are recomputed for every i<. 4. Experimental results In this section, we present a number of simulation and real experiments to demonstrate the performance of the proposed vision/touch-based localization technique.
493
4.1. Uncertainty handling Discussions in the previous two sections assume perfect sensory data, mainly for the sake of presentation clarity. In practice, however, uncertainty has to be dealt with in a formal way. In our work, the uncertainty associated with visual and tactile features is modeled as follows: 1. Tactile-sensor frame: The actual origin of the frame is assumed to be within some distance, *r , of the estit mated one. The actual principal axes are assumed to be obtained from the estimated ones by rotating them about an arbitrary axis, where the rotation angle is between 0 and some threshold, *h . t 2. Tactile vertex: The actual tactile vertex, of a tactile edge, is assumed to be within some distance, *d , of the t estimated one. 3. Visual vertex: The actual visual vertex is assumed to be within some pixel distance, *r , of the estimated v one. Thresholds used throughout the system are automatically determined based on the above error model. In addition, the volumetric constraints are appropriately dilated to handle the uncertainty of the tactile features. 4.2. Simulation experiments Fig. 12a shows a synthetic scene of a three-"ngered robot hand inserting a peg to a slot. Fig. 12b shows results of the feature extraction process. Nine three-edge junctions and 21 two-edge junctions are extracted from such a scene. Since it is hard to determine whether a given junction belongs to the model object, we use all of these
Fig. 12. A synthetic scene of a peg being inserted to a slot: (a) scene, (b) extracted edges.
494
M. Boshra, H. Zhang / Pattern Recognition 33 (2000) 483}501
Fig. 13. Peg-to-a-slot scene: Positions of the tactile sensor: (a) surface contact, (b) surface-edge contact.
junctions for matching. A tactile sensor of dimensions 10 mm]10 mm is assumed to exist on the middle "nger of the robot hand. Fig. 13 shows the tactile images obtained by assuming that the tactile sensor is mounted at two di!erent positions on the middle "nger. To perform a quantitative comparison between our vision/touch-based localization technique and visionbased localization, we use a technique similar to that presented by Kanatani [7] and Wong and Kittler [10] to visually generate pose hypotheses. In these techniques, the visual feature used for matching is a three-edge junction, where one of the junction edges is completely visible. Since it is hard to guarantee that an edge is completely visible (due to occlusion, shadows and imperfections in the feature extraction process), each junction edge with a non-free end is considered to be completely visible, in turn. Thus, for n three-edge visual junctions, the total number of visual features used for matching can be up to 3n. For each type of scene feature sets (three-edge junction, S-patch#junction, SE-patch#junction), localization is performed 100 times. In each run, the sensory data are randomly perturbed, according to the bounded-error model outlined above, assuming *h "53, *r "5 mm, t t *d "0.75 mm, and *r "1 pixel. The results obtained t v in all runs are averaged and summarized in Table 2. Columns in this table are:
1. Scene feature set: The scene feature set used to generate hypotheses. 2. Scene sets: Number of scene feature sets. 3. Total model sets: Total number of model feature sets corresponding to the scene set. 4. Filtered model sets 1: Average number of model sets, per scene set, that are accepted by the volumetric constraints. 5. Filtered model sets 2: Average number of model sets, per scene set, that are accepted by both the volumetric constraints and those on transformation-invariant attributes. 6. Generated hypotheses: Average number of generated hypotheses, per scene set. From Table 2, we can observe the following: 1. The "ltering techniques are very e!ective in reducing the number of model junctions passed to the matcher. On the average, only 30% and 0.6% of the model junctions are passed to the matcher for S- and SEpatches, respectively. Clearly, the "ltering techniques are considerably more powerful in the SE-patch case, due to the presence of extra 3D information provided by the tactile edge. Notice that the number of model sets actually used in matching is comparable to the case when only visual features are used (124.1 and 10.6 model sets for S- and SE-patches, respectively, compared to 60 model sets in the vision-only case). This is despite the fact that the total number of model sets is much higher when a tactile feature is included in the scene set (420 and 1800 for S- and SE-patches, respectively, but only 60 for three-edge junctions). 2. Incorporating a tactile feature in the scene feature set results in a considerably less number of generated hypotheses, per scene set, than when only visual features are used. The extent of reduction is 57% (95%) for the S-patch (SE-patch) case. This is because a tactile feature provides more explicit (3D) information about the object pose. This enables the development of "ltering techniques that limit model sets passed to the matcher. Furthermore, the presence of 3D data in the scene feature set reduces the number of successful scene/model matches. Notice that 67% of the matches are successful when only visual features are used, compared to 14% and 21% success rates for S- and
Table 2 Peg-to-a-slot scene: generated hypotheses Scene feature set
Three-edge junction S-patch (Fig. 13a)#junction SE-patch (Fig. 13b)#junction
Scene sets
25 30 30
Model sets
Generated hypotheses
Total
Filtered 1
Filtered 2
60 420 1800
N/A 280.2 780.0
N/A 124.1 10.6
40.3 17.4 2.2
M. Boshra, H. Zhang / Pattern Recognition 33 (2000) 483}501
495
Fig. 14. First real scene: (a) scene, (b) extracted edges.
Fig. 15. First real scene: (a) original tactile image, (b) thresholded image.
Table 3 Peg-to-a-slot scene: Errors in estimating the object pose Scene feature set
Three-edge junction S-patch (Fig. 13a) #junction SE-patch (Fig. 13b) #junction
Error Rotational
Translational Depth
2.33 1.33
34.3 mm 4.5 mm
33.1 mm 3.0 mm
1.93
6.6 mm
4.1 mm
SE-patches, respectively (the success rates in the vision/touch case are expected to be even lower, if no "ltering is applied).
Next, we compare the accuracy of the poses generated visually with those generated by integrating visual and tactile data. Table 3 shows such a comparison, which is also obtained by running the vision- and vision/touchbased localization systems 100 times under di!erent perturbation values and averaging. Columns in this table are: 1. Rotational error: Average angle between the quaternion representing actual rotation and those of the estimated ones. 2. Translational error: Average distance between origin of the actual object frame and those of the estimated ones. 3. Depth error: Average absolute error in estimating the object depth with respect to the visual sensor (i.e., translation along the optical axis).
496
M. Boshra, H. Zhang / Pattern Recognition 33 (2000) 483}501
Fig. 16. First real scene: Valid hypotheses for: (a) vision-only case (six hypotheses), (b) S-patch case (four hypotheses), (c) SE-patch case (four hypotheses).
Table 4 First real scene: generated hypotheses Scene feature set
Three-edge junction S-patch#junction SE-patch#junction
Scene sets
7 17 17
Model sets
Generated hypothesis
Total
Filtered 1
Filtered 2
48 288 1152
N/A 271.1 768.0
N/A 73.4 1.65
48 24 1.65
M. Boshra, H. Zhang / Pattern Recognition 33 (2000) 483}501
497
Fig. 17. Second real scene: (a) scene, (b) extracted edges.
Fig. 18. Second real scene: (a) original tactile image, (b) thresholded image. Table 5 Second real scene: generated hypotheses (all visually generated hypotheses are erroneous) Scene feature set
Three-edge junction S-patch#junction SE-patch#junction
Scene sets
11 34 34
Model sets
Generated hypotheses
Total
Filtered 1
Filtered 2
48 288 1152
N/A 251.2 700.2
N/A 63.5 8.8
From Table 3, we observe that the error in rotation is quite similar in all cases. For the translational error, we observe that it is an order of magnitude worse when only visual features are used. Observing the depth-error column in Table 3, we can see that, in the vision-only case, most of the translational error is due to poor objectdepth estimation.
30.5 23.5 4.2
The above observation can be explained as follows. In the vision-only case, the object depth is estimated by matching a fully visible visual edge with a model edge [7,10]. It can be shown that the accuracy of such a process is proportional to the actual object depth, along with the relative error in determining the visual-edge length. Thus, we can expect poorer depth estimation for short
498
M. Boshra, H. Zhang / Pattern Recognition 33 (2000) 483}501
Fig. 19. Second real scene: valid hypothesis for: (a) S-patch case, (b) SE-patch case.
visual edges, and for objects that are not close to the image plane (in this experiment, the depth of the peg is 925.0 mm). On the other hand, in the vision/touch case, depth is estimated by determining the intersection of a visual-vertex line of sight with a plane (line) parallel to the tactile plane (tactile edge) in the S-patch (SE-patch) case, as explained in Section 2.3. The error in estimating the location of the intersection point depends on: (1) rotational and translational uncertainties of the tactile-sensor frame, (2) distance between the tactilesensor frame and the line of sight (due to the rotational uncertainty of the tactile-sensor frame), and (3) uncertainty of the line of sight. The accuracy of the intersection point is weakly dependent on the object depth because of the following reasons: (1) the uncertainty of the tactilesensor frame is independent of the object depth, (2) the distance between the tactile-sensor frame and the line of sight is bounded by the size of the object, and (3) the uncertainty of the line of sight is relatively very small, an observation veri"ed experimentally,4 which makes its error contribution insigni"cant. It should be noted that we are not trying to draw general conclusions about the accuracy of pose estimates generated visually compared to those generated by integrating visual and tactile data. Obviously, this depends on many factors including the amount of uncertainty in both visual and tactile data, geometric relationships between features, and object depth. Our objective is to highlight some common situations (in particular, when the object
is far from the image plane) in which vision/touch-based pose estimation can be much more accurate than visionbased estimation.
4 We have found that the `conea representing bounded lineof-sight uncertainty has a very small angle, which makes the divergence rate of the cone very slow.
5 In order to obtain tactile edges of good quality, we `fuseda several images obtained under additional manually exerted forces.
4.3. Real experiments The proposed technique has also been tested using real data. An NEC CCD camera is used to provide visual images of dimensions 480]512. Tactile data are provided by an Interlink piezo-resistive tactile sensor. This sensor is a 16]16 planar array of dimensions 25.4 mm] 25.4 mm. We have attached the tactile sensor to a jaw of a Schunk parallel-jaw gripper that is mounted on a PUMA 260 robot arm. A rectangular polyhedron is chosen as the model object. Fig. 14a shows a scene of the parallel-jaw gripper grasping the model object. Results of the feature extraction process are shown in Fig. 14b. Three three-edge junctions and 14 two-edge junctions are extracted from such a scene. Fig. 15a shows the original tactile image. The darkness of each tactile element (tactel) in this image is proportional to the pressure on it. The tactile image is "ltered and thresholded to form the binary image shown in Fig. 15b. This image is analyzed to detect an SEpatch.5 To test the system performance in the S-patch case, we also consider the center of the sensor as an Spatch of negligible dimensions (notice that the center lies inside the tactile polygon). Localization results are shown
M. Boshra, H. Zhang / Pattern Recognition 33 (2000) 483}501
in Table 4. From this table, we can draw conclusions similar to those reached in the simulation experiments. The generated hypotheses are veri"ed by comparing the visual and tactile features with corresponding model features. Touch-based veri"cation is "rst performed using a data-driven indexing scheme presented in Ref. [20]. Hypotheses consistent with the tactile feature are then visually veri"ed using a pixel-based approach presented in Ref. [21]. Fig. 16 shows wireframes of the valid hypotheses (i.e., those generated from valid scene sets), for each of the three scene feature sets. From this "gure, we observe that the visually generated poses have signi"cantly lower accuracy than those generated using vision and touch. As explained before, this problem is mainly due to poor depth estimation in vision-based localization. Our second real scene, Figs. 17 and 18, involves a heavily-occluded object. In such a scene, there is insu$cient visual information to determine the object pose; only one valid, two-edge, junction is extracted. As in the previous scene, the sensor center is selected as an S-patch of negligible dimensions to test the system performance in the S-patch case. The results obtained are summarized in Table 5. These results further reinforce the conclusions drawn earlier. Figs. 19a and b show the valid generated hypothesis for the S- and SE-patch cases, respectively. This experiment demonstrates the capability of the proposed vision/touch-based technique to localize objects under heavy occlusion.
5. Conclusions A novel technique has been presented for localizing a polyhedral object in a robot hand by integrating visual and tactile data. Localization is performed using a hybrid set of visual and tactile features. The matching process "rst determines a subset of the object's DOFs using the tactile feature. The remaining DOFs, which cannot be obtained from the tactile feature, are then determined by matching the visual feature. A couple of touch-based and vision/touch-based "ltering techniques are developed to reduce the number of model feature sets that are actually compared with a given scene set. Superiority of the proposed technique over vision-based localization has been experimentally demonstrated in the following aspects: (1) capability of determining object pose under heavy occlusion, (2) number of generated pose hypotheses, and (3) accuracy of object-depth estimation. Although only two types of tactile features are considered in this paper, other feature types can be handled in a similar fashion. For example, assume that we have a tactile edge contact. Four DOFs can be determined from the tactile edge (rotation about, and translation along the edge can not be determined). The remaining two DOFs can then be obtained by matching a visual feature (e.g., a junction).
499
The proposed approach can also be extended to handle non-polyhedral objects. For example, assume that we are given a cylindrical model object. A tactile edge generated from contact between the cylinder and the tactile sensor constrains all of the cylinder's DOFs except for translation along the edge. This parameter can then be determined from the visual data by matching the cylinder's circular ends with corresponding visual features. Appendix A A.1. Determination of the dimensions of cylinder i< Heights of cylinder i<, ih` and ih~ (superscripts# and!refer to measurements along the positive and negative directions of an axis, respectively) are obtained as follows: ih`"max Mio N, ih~"min Mio N, jz jz j j where model vertex io 3iO. j To compute the radius of cylinder i<, ir, we approximate the square tactile array by a circle of diameter d , t where d is the dimension of the array. Let iS be the t i model surface S after transformation by iT , and ; the i s polygon obtained by `shrinkinga iS by 0.5d . In this case, i t ir is the maximum distance between the vertices of iO and the z-axis such that the tactile circle is totally inside iS . It i can be shown that this maximum distance occurs when a vertex in ;, u , translates to the origin (notice that k u "0). Thus, ir is given by kz ir"max MJ(io !u )2#(io !u )2N. jx kx jy ky j, k A.2. Determination of the dimensions of cuboid i< It can be easily shown that the range of translations along the x-axis that keep the tactile edge inside model edge E , after transforming it by iT , is i se (l !l ) (l !l ) t , !p # i t , !p ! i x x 2 2
C
D
where l is the length of the tactile edge, l is the length of t i edge E , and p is the center of E after transforming it by i i iT (notice that p "p "0). Dimensions of cuboid se y z i< are determined as follows: (l !l ) t, id`"max Mio N!p # i x jx x 2 j (l !l ) t, id~"min Mio N!p ! i x jx x 2 j id`"max Mio N, id~"min Mio N, y jy y jy j j id`"max Mio N, id~"min Mio N. z jz z jz j j
500
M. Boshra, H. Zhang / Pattern Recognition 33 (2000) 483}501
It can be seen that one can compute all the dimensions of i< o!-line except for id` and id~, since they depend on x x l that is only known at run-time. These two dimensions t can be computed o!-line assuming l "0, and then upt dated on-line at a negligible cost by adding Glt to 2 id` and id~, respectively. x x A.3. Determination of the Bounds on angle(p; , ws (h)) k De"ne b(h)"angle(pL , ws (h))"arccos(pL ' ws (h)), k k
(A.1)
where h3[hmin, hmax], pL is a unit vector, and ws (h) is as k de"ned in Eq. (32). Substituting from Eqs. (31) and (32) into Eq. (A.1), we get b(h)"arccos(asin(h)#bcos(h)),
(A.2)
where a"pL ) (hs]ws) and b"pL ' ws. It can be easily shown that the extreme point of b(h) occurs at he"arctan a/b. We choose the solution of he that falls in the range [03, 1803]. The bounds on b(h) are
G
[bmin, bmax]"
[9] S. Ullman, R. Basri, Recognition by linear combinations of models, IEEE Trans. Pattern Anal. Mach. Intell. 13 (10) (1991) 992}1006. [10] K.C. Wong, J. Kittler, Recognizing polyhedral objects from a single perspective view, Image and Vision Comput. 11 (4) (1993) 211}220. [11] P. Flynn, A.K. Jain, BONSAI: 3D object recognition using constrained search, IEEE Trans. Pattern Anal. Mach. Intell. 13 (10) (1991) 1066}1075. [12] W. Kim, A.C. Kak, 3D object recognition using bipartite matching embedded in discrete relaxation, IEEE Trans. Pattern Anal. Mach. Intell. 13 (3) (1991) 224}251. [13] F. Stein, G. Medioni, Structural indexing: e$cient 3D object recognition, IEEE Trans. Pattern Anal. Mach. Intell. 14 (2) (1992) 125}145. [14] R.E. Ellis, Geometric uncertainties in polyhedral object recognition, IEEE Trans. Robotics Automat. 7 (3) (1991) 361}371. [15] W.E.L. Grimson, T. Lozano-Perez, Model-based recognition and localization from sparse range or tactile data, Int. J. of Robotic Res. 3 (3) (1984) 3}35. [16] W.E.L. Grimson, T. Lozano-Perez, Localizing overlapping parts by searching the interpretation tree, IEEE Trans. Pattern Anal. Mach. Intell. 9 (4) (1987) 469}482.
[min(b(hmin), b(he), b(hmax)), max(b(hmin), b(he), b(hmax))] if he3[hmin, hmax], [min(b(hmin), b(hmax)), max(b(hmin), b(hmax))]
Notice that the value of tan(he) is unde"ned if both a and b are zero. This occurs when pL is perpendicular to plane Ps . In this case, we have [ b.*/, b.!9]"[903, 903]. k References [1] F. Arman, J.K. Aggarwal, Model-based object recognition in dense-range images } a review, ACM Comput. Surveys 25 (1) (1993) 5}43. [2] P.J. Besl, R.C. Jain, Three-dimensional object recognition, ACM Comput. Surveys 17 (1) (1985) 75}145. [3] R.T. Chin, C.R. Dyer, Model-based recognition in robot vision, ACM Comput. Surveys 18 (1) (1986) 67}108. [4] P. Suetens, P. Fua, A.J. Hanson, Computational strategies for object recognition, ACM Comput. Surveys 24 (1) (1992) 5}61. [5] H.H. Chen, Pose determination from line-to-plane correspondences: existence condition and closed-form solutions, IEEE Trans. Pattern Anal. Mach. Intell. 13 (6) (1991) 530}541. [6] M. Dhome, M. Richetin, J. Lapreste, G. Rives, Determination of the attitude of 3D objects from a single perspective view, IEEE Trans. Pattern Anal. Mach. Intell. 11 (12) (1989) 1265}1278. [7] K. Kanatani, Constraints on length and angle, Comput. Vision Graphics Image Process. 41 (1) (1988) 28}42. [8] D.G. Lowe, Three-dimensional object recognition from single two-dimensional images, Arti"cial Intell. 31 (3) (1987) 355}395.
otherwise.
[17] R.C. Luo, W. Tsai, Object recognition using tactile-array sensors, Proc. IEEE Int. Conf. Rob. and Aut., San Francisco, CA, 1986, 1248}1253. [18] P.K. Allen, Integrating vision and touch for object recognition tasks, Int. J. of Robot. Res. 7 (6) (1988) 15}33. [19] R.A. Browse, J.C. Rodger, Techniques for combining tactile and visual perception for robotics, in: M.A. Goodale (Ed.), Vision and Action: The Control of Grasping, Ablex Publishing Corporation, 1990, pp. 295}320. [20] M. Boshra, H. Zhang, Use of tactile sensors in enhancing the e$ciency of vision-based object localization, Proc. IEEE Int. Conf. Multisensor Fusion and Integ. for Intell. Sys. Las Vegas, Nevada, 1994, pp. 243}250. [21] M. Boshra, H. Zhang, An e$cient pixel-based technique for visual veri"cation of 3D object hypotheses, Proc. IEEE Int. Conf. Robotics and Automation, Minneapolis, Minnesota, 1996, pp. 3472}3477. [22] W.E.L. Grimson, D.P. Huttenlocher, On the veri"cation of hypothesized matches in model-based recognition, IEEE Trans. Pattern Anal. Mach. Intell. 13 (12) (1991) 1201}1213. [23] R.P. Paul, Robot Manipulators: Mathematics, Programming, and Control, MIT Press Cambridge, MA, 1981. [24] R.C. Bolles, P. Horaud, 3DPO: A three-dimensional part orientation system, Int. J. Robotic Res. 5 (3) (1986) 3}26. [25] A. Califano, R. Mohan, Multidimensional indexing for recognizing visual shapes, IEEE Trans. Pattern Anal. Mach. Intell. 16 (4) (1994) 373}392. [26] O.D. Faugeras, M. Hebert, The representation, recognition and locating of 3D objects, Int. J. Robotic Res. 5 (3) (1986) 27}52.
M. Boshra, H. Zhang / Pattern Recognition 33 (2000) 483}501
501
About the Author*MICHAEL BOSHRA received B.Sc. and M.Sc. degrees in Computer Science from the University of Alexandria, Egypt, in 1988 and 1992, respectively, and a Ph.D. degree in Computing Science from the University of Alberta, Canada, in 1997. He is currently a post-doctoral fellow in the Center for Research in Intelligent Systems (CRIS) at the University of California, Riverside. From 1989 to 1992, he worked as a research assistant at the National Research Center, Giza, Egypt. His current research interests include object recognition, sensor fusion, performance prediction, and multi-dimensional indexing structures. About the Author*HONG ZHANG is an Associate Professor in the Department of Computing Science, University of Alberta, Edmonton, Alberta, Canada. He is director of the Robotics Research Laboratory and his current research interests include sensor data fusion, tactile sensing, dextrous manipulation, tele-robotics, and collective robotics. He received his B.S. degree from Northeastern University in 1982 and his Ph.D. degree from Purdue University in 1986, both in Electrical Engineering. He subsequently worked as a post-doctoral fellow in the GRASP Laboratory, University of Pennsylvania, for 18 months before joining University of Alberta. His other work experience includes Kajima visiting professorship at Kagoshima University in 1993 and an STA fellowship at the Mechanical Engineering Laboratory, Japan, in 1994}1995.
Pattern Recognition 33 (2000) 503}513
Image-based form document retrieval Jinhui Liu, A.K. Jain* Department of Computer Science and Engineering, Michigan State University, East Lansing, MI 48824-1226, USA Received 29 September 1998; accepted 22 February 1999
Abstract We address the problem of image-based form document retrieval. The essential element of this problem is the de"nition of a similarity measure that is applicable in real situations, where query images are allowed to di!er from the database images. Based on the de"nition of form signature, we have proposed a similarity measure that is insensitive to translation, scaling, moderate skew ((53) and variations in the geometrical proportion of the form layout. This similarity measure also has a good tolerance to line detection errors. We have developed a prototype form retrieval system based on the proposed similarity measure. Preliminary experimental results on a database containing 100 di!erent kinds of forms are encouraging. ( 2000 Pattern Recognition Society. Published by Elsevier Science Ltd. All rights reserved. Keywords: Document analysis; Form processing; Form recognition; Image database; Line detection; Similarity measure; Document retrieval
1. Introduction Form processing is an essential operation in business and government organizations. Although the percentage of electronic "lings is increasing, paper-based forms will continue to be widely used in the foreseeable future. Because of the many advantages o!ered by on-line processing, much e!ort has been devoted to image-based form analysis and recognition. This includes practical systems for machine reading of form data from scanned images [1}3], form dropout [4}7] and form layout description [8]. Other investigations on various aspects of form analysis and recognition can be found in Refs. [9}15]. In this paper we consider the following problem: Given a form image database and a query image, how do we retrieve form images in the database with the same or similar layout structure as the query? This problem is similar to the form recognition work investigated in Ref. [1]. However, the di!erence lies in that our query
* Corresponding author. Tel.: 001-517-353-6484; fax: 001-517432-1061 E-mail address:
[email protected] (A.K. Jain)
images are not prepared for data entry, but for providing su$cient clues for similarity-based retrieval. So, the query images can be at a di!erent scale (resolution) or have relatively poor quality compared to database images. The challenges in form image analysis arise from the large variety of form formats (layouts). Although form formats can be very #exible, we assume that the forms have the following layout characteristics: (i) a form has two horizontal frame lines serving as top and bottom boundaries. It is recommended that the forms have two vertical frame lines serving as left and right boundaries, and (ii) most of the useful data items in a form are enclosed by rectangles formed by frame lines. We address the problem of image-based retrieval for forms that have the above characteristics. Our goal is to propose a similarity measure for forms that is insensitive to translation, scaling, moderate skew ((53), and image quality #uctuations. Sometimes, forms of the same type that come from di!erent printing sources di!er slightly in their "nal layout, so we do not assume the invariance of the geometrical proportion of the frame structure. Although we only address the retrieval problem in this paper, it is obvious that this similarity measure can also be used to automatically determine form type for form
0031-3203/00/$20.00 ( 2000 Pattern Recognition Society. Published by Elsevier Science Ltd. All rights reserved. PII: S 0 0 3 1 - 3 2 0 3 ( 9 9 ) 0 0 0 6 6 - 7
504
J. Liu, A.K. Jain / Pattern Recognition 33 (2000) 503}513
data entry systems and automatically organize images in constructing form image databases.
2. Form signature Form signature is the feature used to evaluate the similarity between forms in our system. Among the many components contained in a form, such as lines, preprinted text "elds, "lled-in text "elds, check boxes and circles, form identi"cation number, logos, etc., frame lines re#ect the essential layout structure of the form and are su$cient to uniquely identify the form type in most cases. Further, frame lines can be detected more reliably than other components. So, we de"ne form signature based on frame lines.
to de"ne sequences H and <: H"Mh , h ,2, h ,2, h H N, y )y if i(j, 0 1 i n ~1 i j <"Mv , v ,2, v ,2, v V N, x )x if i(j. (1) 0 1 i n ~1 i j Based on sequences H and <, we de"ne form signature as follows: De5nition 2. The frame structure constructed by H and < is called a form signature. In this paper, a form signature constructed by H and < is denoted by F(H, <), where F means `forma. Fig. 1(a) illustrates an example of the form signature. Now we introduce the concept of a cell, which will be used in de"ning our similarity measure. De5nition 3. A cell is a rectangle in a form de"ned by two horizontal frame lines and two vertical frame lines and within which there is no other frame line.
2.1. Form signature based on frame lines Suppose that a form has a closed rectangular border. If not, two virtual vertical lines are added as left and right borders. De5nition 1. A frame line is a horizontal line with its endpoints on vertical lines or a vertical line with its endpoints on horizontal lines. According to De"nition 1, hanging lines that are usually used to indicate "lled-in "elds in forms are excluded from frame lines. A frame line is represented by the coordinates of its start point (xs, ys) and terminal point (xt, yt). We use h to denote a horizontal frame line i and v to denote a vertical frame line in a form: i h "M(xs, y )N, (xt, y )N, and v "M(x , ys), (x , yt)N. i i i i i i i i i i We can sort the horizontal frame lines in ascending y-order and the vertical frame lines in ascending x- order
Let C denote the set of all the cells in a form: C"Mc , c ,2, c C~1N, where n is the number of cells in 0 1 n C the form. Each c can be described by a 4-tuple: (top, left, k bottom, right), which describes the cell's scope. We use the item traversal algorithm [8] to obtain all the cells in a form. Fig. 1(b) shows all the cells in the form signature shown in Fig. 1(a). 2.2. Extracting form signature from form's image We scan form images in a binary mode. There are two main steps in deriving the form signature from the binary image: (i)
Line detection. We adopt the method proposed in Ref. [4], which detects horizontal and vertical lines
Fig. 1. Form signature and cells.
J. Liu, A.K. Jain / Pattern Recognition 33 (2000) 503}513
based on Block Adjacency Graph (BAG). This method easily adapts to the moderate skews in the image ((53), and its special feature in handling characters that touch form lines makes it applicable for both blank and "lled-in forms. (ii) Frame standardization. The detected lines can be regarded as a coarse form signature. Further processing is needed to standardize the frame. First, each double-line is merged into a single line. A double-line is de"ned as a pair of parallel lines very close to each other; second, we delete any hanging lines present in the form; third, the skew of the remaining lines is corrected by a linear transformation according to the estimated skew value [4]; "nally, we move the endpoints of each horizontal line onto the nearest vertical lines and vice versa to obtain the form signature. Fig. 2 illustrates these steps by an example. The form signature obtained by the above two steps may not be perfect when the image quality is poor. For example, long frame lines usually break up into several segments in this case. So, the similarity measure should be robust to line breaks.
505
3. Similarity measure for forms In order to de"ne the similarity measure, we "rst introduce the concept of a grid on the form image plane. 3.1. Grid on form plane We perform a simple operation, called line grouping on form signature as follows. Collinear horizontal lines form an H-group that is represented by the y-coordinates of these horizontal lines; collinear vertical lines form a V-group that is represented by x-coordinates of these vertical lines. If there are n H-groups and n V-groups in a form signaHG VG ture, then we obtain the following two sequences: H "Mp , p ,2, p ,2, p HG N, p (p if i(j, G 0 1 i n ~1 i j < "Mq , q ,2, q ,2, q VG N, q (q if i(j. (2) G 0 1 i n ~1 i j Suppose that A is a sub-sequence of H and B is G a sub-sequence of < : G A"Ma , a ,2, a N, a "p , n(i)(n( j) if i(j, 0 1 k~1 i n(i) B"Mb , b ,2, b N, b "q , m(i)(m( j) if i(j. 0 1 l~1 i m(i) (3)
Fig. 2. Extracting form signature.
506
J. Liu, A.K. Jain / Pattern Recognition 33 (2000) 503}513
Fig. 3. Grid on form plane.
Fig. 4. An example of the similarity measure between two forms.
Note that A and B form a grid on a form's plane, which is denoted by Grid(A, B). According to Eq. (3), Grid(A, B) has (l!1) mesh cells per row and (k!1) mesh cells per column. The mesh cell at row i and column j is represent-
ed by M (i, j), with its vertical scope from a to Grid(A, B) i a and horizontal scope from b to b . i`1 j j`1 Fig. 3(a) illustrates the H and < of the sample G G form signature shown in Fig. 1. Fig. 3(b) is an example
J. Liu, A.K. Jain / Pattern Recognition 33 (2000) 503}513
of Grid(A, B) for this form signature with A" Mp , p , p , p N and B"Mq , q , q , q N. 0 2 3 5 0 1 4 6 With the Grid(A, B) superimposed on the form signature, the following two functions can be de"ned:
In order to re#ect the number of cells present in a mesh cell of Grid(A, B), we de"ne N (i, j) as Grid(A, B) follows:
A
N (i, j)" + Intersect(c , M (i, j)) Grid(A, B) k Grid(A, B) ck|CGrid(A, B)
Intersect (c , M (i, j)) k Grid(A, B)
G
if c WM (i, j)OH, k Grid(A, B) 0 if c WM (i, j)"H. k Grid(A, B)
1
"
507
(4)
B
1 . Segments(c , Grid(A, B)) k
]
(7)
3.2. Similarity measure
Segments (c , Grid(A, B)) k @A@~2 @B@~2 " + + Intersect (c , M (i, j)), k Grid(A, B) i/0 j/0
(5)
where c 3C, k"0, 1,2, n !1; DAD denotes the number k C of elements in sequence A. We denote by C , the set of all the cells intersectGrid(A, B) ing Grid(A, B): C "Mc D Segments(c , Grid(A, B))'0N. Grid(A, B) k k
(6)
Basically, the proposed similarity measure is de"ned by comparing the number of cells in the corresponding mesh cells of two grids with the same dimension. According to Eq. (7), the number of cells in a mesh cell is determined by the relative position of form lines, so this similarity measure is insensitive to translation, rotation, scaling and variations of geometrical proportion. It is also insensitive to line breaking, which does not a!ect line grouping and only a!ects the cell counts of the mesh
Fig. 5. Block diagram of the retrieval system.
508
J. Liu, A.K. Jain / Pattern Recognition 33 (2000) 503}513
Fig. 6. Some query images used in our experiments.
cells at the location where the line break occurred. However, the line detection procedure may miss some lines or add a few spurious lines, which may result in incorrect line grouping.
Suppose that F(Hu,
J. Liu, A.K. Jain / Pattern Recognition 33 (2000) 503}513
509
Fig. 7. Query results.
group Hr and vertical line group
Hu satisfying G Hu "MAuDDAuD"k , au "pu , auh~ "puuHG N. S h 0 0 k 1 n ~1 Similarly,
(10)
510
J. Liu, A.K. Jain / Pattern Recognition 33 (2000) 503}513
Fig. 8. Retrieve speci"c forms (e.g., by name) in the same type of forms.
border of the input form as their borders. Elements in Hr and
Let us call (Grid(Au, Bu), Grid(Ar, Br)) a grid pair. Eq. (12) searches for the optimal grid pair which results in the minimal overall cell count di!erence. In other words, it tries to match the similar parts of two forms as much as possible in terms of the cell count di!erence. Therefore, the proposed similarity measure is also robust to incorrect line groupings for poor quality images. It is easy to deduce from Eq. (12) that S3(0, 1]. The larger the value of S, the more the two forms are similar to each other. According to Eqs. (9)}(12), the grids for the form that has fewer horizontal line groups will have "xed horizontal gridlines which correspond to all the horizontal line groups in the form; and the grids for the form that has more horizontal line groups will have u r C.*/(@HGuG@, @HGrG@)~2 di!erent horizontal gridlines combina.!9(@H @, @H @)~2
tions from which the optimal one should be selected. A similar analysis can be done for the vertical gridlines. So, the number of grid pairs for evaluating the cell count di!erence is C.*/(@HuGuG@, @HrGrG@)~2]C.*/(@VuGuG@, @VrGrG@)~2 . Fig. 4 .!9(@H @,@H @)~2 .!9(@V @, @V @)~2 gives an example of computing similarity between two forms. In this example, the input form has only one "xed grid because it has fewer horizontal and vertical line groups. The reference form has C2]C2"60 grids } only 4 5 two of them are illustrated in Fig. 4, in which Grid 2 is one of those grids that give the similarity value. Currently, we use an exhaustive search to "nd the grid pair which gives the minimal cell count di!erence. This is not computationally attractive if the number of line groups is large and the di!erence between the numbers of line groups in the two forms is large. In order to decrease the computational complexity of matching, we only calculate the similarity measure for reference form signatures satisfying DDHu D!DHr DD)2, G G Dnu !nr D)32. C C
DD
and (13)
4. Form retrieval experiments In designing our system, we assume that forms with the same layout structure in the database have already been
Fig. 9. Forms of di!erent types which have the same form signature.
J. Liu, A.K. Jain / Pattern Recognition 33 (2000) 503}513 511
512
J. Liu, A.K. Jain / Pattern Recognition 33 (2000) 503}513
organized into the same group. So, before the system accepts a query, the signature of every type of form in the database can be obtained and be used to represent the relevant group of form images. The retrieval procedure only needs to compute the signature of the query image and then compare it with all the form signatures in the database. The block diagram of the retrieval system is illustrated in Fig. 5 (The retrieval score reported by our system is 1000 times the similarity value S). The system was developed using Visual C##5.0 and runs under Windows NT 4.0. We have tested our system on a form image database containing 100 di!erent types of forms. The forms used in our experiments include both blank forms and "lled-in forms. These forms of di!erent sizes are typically printed on A4 size paper and were scanned at a resolution ranging from 200 to 400 dpi and a skew angle ranging from !5 to 53. The image size varied from 860]560 to 3400]4400. Also, noise was introduced in some of the images by erasing some frame lines. We tested the system with 200 query images on a Pentium 166 MHz PC. For complex forms, the average time for signature extraction is about 8 s per image, and for retrieval it is about 5 s. The retrieval result is regarded as correct if the correct form signature in the database gets the highest score. The retrieval results for 95% of the 200 images were correct. This retrieval error can be reduced further if we relax the constraints speci"ed in Eq. (13). But, this would lead to an increased retrieval time. Fig. 6 illustrates four of the query images. The retrieval results for these query images are shown as the "ve most similar forms in the database that were retrieved (see Fig. 7).
5. Conclusions and future work We have addressed the problem of image-based form document retrieval. The central issue of this problem is the de"nition of a similarity measure between a pair of forms. Based on the de"nition of form signature, we have proposed a similarity measure that is insensitive to translation, scaling, moderate skew ((53) and variations of the geometrical proportion of the form layout. A larger amount of skew can be handled by "rst applying a deskewing procedure [16]. This similarity measure also has good tolerance to line breaks due to line detection errors, which are almost inevitable when the quality of the query image is poor. A prototype form retrieval system has been developed based on this similarity measure. Preliminary experimental results on a database containing 100 di!erent kinds of forms are encouraging. The conditions speci"ed in Eq. (13) might be too restrictive for tolerating errors on line grouping, which usually occur due to missed lines or spurious lines for poor quality images. Therefore, it is desirable to relax the constraints speci"ed in Eq. (13). Improving the e$ciency
of computing the similarity measure will be one of our future work. Also, if the task is to retrieve speci"c images in a group of forms with the same format (Fig. 8) or discriminate di!erent form types with the same form signature (Fig. 9), then additional information needs to be extracted from the forms. Acknowledgements We would like to thank Dr. Jianchang Mao and Dr. Moidin Mohiuddin of IBM Almaden Research Center for their support of this work. References [1] S. Gopisetty, R. Lorie, J. Mao, M. Mohiuddin, A. Sorin, E. Yair, Automated forms-processing software and services, IBM J. Res. Develop. 40 (2) (1996) 211}230. [2] J. Mao, M. Abayan, K. Mohiuddin, A model-based form processing sub-system, Proceedings of ICPR'96, 1996, pp. 691}695. [3] R. Casey, D. Ferguson, K. Mohiuddin, E. Walach, Intelligent forms processing system, Mach. Vision Appl. 5 (1992) 142}155. [4] B. Yu, A.K. Jain, A generic system for form dropout, IEEE Trans. Pattern Anal. Mach. Intell. 18 (11) (1996) 1127}1134. [5] G. Maderlechner, Symbolic subtraction of "xed formatted graphics and text from "lled in forms, Mach. Vision Appl. 3 (1990) 457}459. [6] S. Taylor, R. Fritzson, J. Pastor, Extraction of data from preprinted forms, Mach. Vision Appl. 5 (1992) 211}222. [7] K. Fan, J. Lu, L. Wang, H. Liao, Extraction of characters from form documents by feature point clustering, Pattern Recognition Lett. 16 (1995) 963}970. [8] J. Liu, X. Ding, Y. Wu, Description and recognition of form and automated form data entry, Proceedings of ICDAR'95, 1995, pp. 579}582. [9] D. Doermann, A. Rosenfeld, The processing of form documents, Proceedings of ICDAR'93, 1993, pp. 497}501. [10] Y. Tang, C.Y. Suen, C. Yan, M. Cheriet, Financial document processing based on sta! line and description language, IEEE Trans. SMC 25 (5) (1995) 738}754. [11] A. Laurentini, P. Viada, Identifying and understanding tabular material in compound documents, Proceedings of ICPR'92, 1992, pp. 405}409. [12] D. Wang, S. Srihari, Analysis of form images, Proceedings of ICDAR'91, 1991, pp. 181}190. [13] T. Watanabe, Q. Luo, N. Sugie, Layout recognition of multi-kinds of table-form documents, IEEE Trans. Pattern Anal. Mach. Intell. 17 (4) (1995) 432}445. [14] J. Chen, H. Lee, An e$cient algorithm for form structure extraction using strip projection, Pattern Recognition 31 (9) (1998) 1353}1368. [15] L. Tseng, R. Chen, Recognition and data extraction of form documents based on three types of line segments, Pattern Recognition 31 (10) (1998) 1525}1540. [16] B. Yu, A.K. Jain, A robust and fast skew detection algorithm for generic document, Pattern Recognition 29 (10) (1996) 1599}1629.
J. Liu, A.K. Jain / Pattern Recognition 33 (2000) 503}513
513
About the Author*JINHUI LIU received the B.S., M.S. and Ph.D. degrees in electronic engineering from Tsinghua University, Beijing, China, in 1990, 1992 and 1997, respectively. He was a postdoctoral researcher in the Department of Computer Science and Engineering and Center for Microbial Ecology at Michigan State University from 1997 to 1999 and was a visiting scientist at the IBM Almaden Research Center from July to October 1998. He is now a postdoctoral researcher in the Department of Computer Engineering and Computer Science at the University of Missouri}Columbia. His research interests include document image understanding, pattern recognition, and machine intelligence. He has performed research on form processing, handprinted Chinese character recognition, handwritten numeral recognition, postal address recognition and bacterial image analysis. About the Author*ANIL JAIN is a University Distinguished Professor and Chair of the Department of Computer Science and Engineering at Michigan State University. His research interests include statistical pattern recognition, Markov random "elds, texture analysis, neural networks, documents image analysis, "ngerprint matching and 3D object recognition. He received the best paper awards in 1987 and 1991 and certi"cates for outstanding contributions in 1976, 1979, 1992, and 1997 from the Pattern recognition Society. He also received the 1996 IEEE Trans. Neural Networks Outstanding Paper Award. He was the Editor-in-Chief of the IEEE Trans. on Pattern Analysis and Machine Intelligence (1990}1994). He is the co-author of Alogorithms for clustering Data, Prentice-Hall, 1988, has edited the book Real-time Object Measurement and Classi"cation, Springer-Verlag, 1988, and co-edited the books, Analysis and Interpretation of Range Images, Springer-Verlag, 1989, Markov Random Fields, Academic Press, 1992, Arti"cial Neural Networks and Pattern Recognition, Elsevier, 1993, 3D Object Recognition, Elsevier, 1993, and BIOMETRICS: Personal Identi"cation in Networked Society, Kluwer in 1998. He is a Fellow of the IEEE and IAPR. He received a Fulbright research award in 1998.
Pattern Recognition 33 (2000) 515}519
A note on constrained k-means algorithms Michael K. Ng* Department of Mathematics, University of Hong Kong, Pokfulam Road, Hong Kong Received 26 August 1998; received in revised form 16 February 1999; accepted 16 February 1999
Abstract This paper describes extensions to the k-means algorithm for clustering data sets. By adding suitable constraints into the mathematical program formulation, an approach is developed, which allows the use of the k-means paradigm to e$ciently cluster data sets with the "xed number of objects in each cluster. The new algorithm is presented and the e!ectiveness of the algorithm is demonstrated with experimental results. ( 2000 Pattern Recognition Society. Published by Elsevier Science Ltd. All rights reserved. Keywords: Clustering; Constraints; k-means algorithm; PCB insertion
1. Introduction The k-means-type algorithm [1}5] is well known for its e$ciency in clustering large data sets. The main idea behind the k-means algorithm is the minimization of an objective function usually taken up as a function of the deviations between all patterns from their respective cluster centers. The minimization of such an objective function is sought employing an iterative scheme which starts with an arbitrary chosen initial cluster membership in an iterative manner to obtain a better clustering result. The sum of squared Euclidean distances measures has been adopted in most of the studies [1] related to these algorithms due to its computational simplicity. We remark that the cluster at each iteration can be calculated in a straightforward manner. The distribution of objects among clusters utilizing a speci"c cluster and the updating of cluster centers are two main steps of the k-means algorithm [3,5]. The algorithm alternatives between two steps until the value of the objective function cannot be reduced any more. In some applications, we are interested in partitioning a set of objects into clusters with the "xed number of objects. For instance, the surface mount technology
* Tel.: #852-28592252; fax: #852-25592225 E-mail address:
[email protected] (M.K. Ng)
(SMT) machine inserts electronic components into de"ned positions on a printed circuit board (PCB) and the components are supplied from a set of reels each containing a tape of identical electronic components [6,7]. Because of the limited capacity of feeder bank, the electronic assembly plant uses multiple assembly to produce PCBs with a number of types of components being greater than the maximum number of reels, i.e., more than one assembly process in the PCB insertion problem. To minimize the time used for placing a set of electronic components onto a PCB, a solution technique based on the clustering approach is proposed [8] to "nd a good assembly plan. In the clustering-based technique, we need to partition electronic components into some clusters but also require the number of electronic components in each cluster to be less than or equal to the limited capacity of feeder bank. Another application is mining association rules in large databases [9]. The mining problem can be reduced to "nding all large itemsets with respect to a given threshold. The development of parallel algorithms for mining association rules is an important problem [10,11]. The main activity in "nding large itemsets is the computing of support counts of candidate itemsets. We can use the distributed pruning technique [10,11] to prune and reduce the number of candidates in the parallel mining process. We "nd that the e!ect of pruning increases as the data between partitions becomes more skew, i.e., the similarity of item sets in each cluster is high. However, in
0031-3203/00/$20.00 ( 2000 Pattern Recognition Society. Published by Elsevier Science Ltd. All rights reserved. PII: S 0 0 3 1 - 3 2 0 3 ( 9 9 ) 0 0 0 5 7 - 6
516
M.K. Ng / Pattern Recognition 33 (2000) 515}519
order to have nice parallelism of the fast parallel and distributed mining algorithm, the number of item sets in each cluster should be almost the same. We remark that some k-means-type clustering algorithms [4] can be modi"ed in the splitting rule in order to control the number of cluster members. However, in this paper, we consider the k-means clustering algorithm that has been studied by Anderberg [1], and Jain and Dubes [3]. The new algorithm extends this k-means algorithm by adding suitable constraints into the mathematical program formulation to minimize the clustering objective function. These constraints allow the use of the k-means paradigm to e$ciently cluster data sets with the "xed number of objects in each cluster. The outline of the paper is as follows: In Section 2, the notation and mathematical formulation of the problem are established. The convergence of our new k-means algorithm will be shown. Finally, experimental results are reported to illustrate the e!ectiveness of the algorithm.
2. Mathematical formulation The k-means clustering algorithm can be stated as the algorithm [3] which attempts to minimize the cost function k n F(=, Z)" + + w DDZ !X DD2 li l i 2 l/1 i/1 subject to
(1)
w 3M0, 1N, 1)l)k, 1)i)n, (2) li k + w "1, 1)i)n, (3) li l/1 where n is the number of objects, k is a "xed and known number of clusters, X 3Rm(1)i)n) are objects in the i feature space Rm, w represents the association of object li X with cluster Z , ="[w ] is an k]n matrix, i l li Z"[Z , Z ,2, Z ] is an m]k matrix and E ) E is the 1 2 k 2 Euclidean distance measure. In order to cluster data sets such that each cluster contains the "xed number of objects, we add the following constraints into the mathematical programming: n + w "c , 1)l)k. (4) li l i/1 Here c is an integer and n"+k c . The extension of the l l/1 l k-means algorithm is to determine = and Z by minimization of F with the constraints in Eqs. (2)}(4). Minimization of F in Eq. (1) with the constraints in Eqs. (2)}(4) forms a class of constrained nonlinear optimization problems whose solution is unknown. The usual method towards optimization of F in Eq. (1) is to use partial optimization for Z and =, see Jain and Dubes
[3]. In this method we "rst "x Z and "nd necessary conditions on = to minimize F. Then we "x = and minimize F with respect to Z. This process is formalized in the k-means algorithm as follows. Algorithm 1. The k-means algorithm: 1. Choose an initial point Z(1)3Rmk. Determine =(1) such that F(=, Z(1)) is minimized. Set t"1. 2. Determine Z(t`1) such that F(=(t), Z(t`1)) is minimized. If F(=(t), Z(t`1))"F(=(t), Z(t)), then stop; otherwise goto step 3. 3. Determine =(t`1) such that F(=(t`1), Z(t`1)) is minimized. If F(=(t`1), Z(t`1))"F(=(t), Z(t`1)), then stop; otherwise set t"t#1 and goto Step 2. The matrices Z and = can be calculated according to the following two theorems. Theorem 1. ¸et = K be xxed and consider Problem (P1): min F(= K , Z). Z Then the minimizer ZK of Problem (P2) is given by +n w X ZK " i/1 li i, 1)l)k. l +n w i/1 li In Algorithm 1, if ZK is "xed, we need to solve the following Problem (P2): min F(=, ZK ) s.t. Eqs. (1)}(4). W It is easy to see that Problem (P2) can be viewed as the transportation problem which is one of determining a minimal cost shipping schedule of a commodity between sources and destinations. Speci"cally, suppose there are n objects (sources) and k clusters (destinations), and that a supply of 1 unit is available at the ith object and a demand of c objects is to be met at the lth cluster. l Clearly, this problem would have optimal solutions if the total supply (total number of objects) is equal to the total demand (the total number of objects in clusters). There are many e$cient algorithms [12,13] for solving the transportation problem (P2). Theorem 2. Problem (P2) can be solved by the transportation problem algorithm. According to Theorems 1 and 2, we can use the kmeans paradigm to cluster data sets and restrict that the number of objects in each cluster is a "xed number. The next theorem gives a proof of convergence for our proposed k-means algorithm. Theorem 3. The new k-means algorithm converges in a xnite number of iterations.
M.K. Ng / Pattern Recognition 33 (2000) 515}519
Proof. All the basic feasible solutions of the set
G
517
Table 1 Results for the original k-means algorithm and the constrained k-means algorithm
k n w D + w "1, 1)i)n, + w "c , 1)l)k, li li li l l/1 i/1
H
and w *0, 1)i)n, 1)l)k li
have integer values (0 or 1) since the constraint matrix is totally unimodular. We note that there are only a "nite number of basic feasible solutions. We then show that each possible feasible solution appears at most once by the k-means algorithm. Assume that =(t1)"=(t2) where t Ot . According to the k-means algorithm we can com1 2 pute the minimizers Z(t1) and Z(t2) of Problem (P1) for ="=(t1) and ="=(t2), respectively. Therefore, we have F(=(t1), Z(t1))"F(=(t2), Z(t1))"F(=(t2), Z(t2)).
Data set n
k
t (s) o
error
t (s) c
(t !t )/t % c o o
1 2 3 4 5 6 7 8 9 10 11 12
3 4 5 6 3 4 5 6 3 4 5 6
5.94 4.95 4.62 6.27 19.75 15.80 18.17 21.33 38.75 36.25 35.00 37.50
19.2 15.8 16.9 19.7 17.8 25.2 29.1 20.4 25.5 36.1 30.3 34.7
7.65 5.76 5.34 8.80 27.30 23.22 23.84 32.80 45.54 47.07 42.88 53.55
28.79 16.36 15.58 40.35 38.23 46.96 31.21 53.77 17.52 29.85 22.51 42.80
120 120 120 120 240 240 240 240 360 360 360 360
However, the sequence F( ) , ) ) generated by the k-means algorithm is strictly decreasing. Hence the result follows. h In the next section, we will demonstrate the e$ciency of our new k-means algorithms.
3. Computational results The algorithm presented in Section 2 has been tested on a Sun Sparcstation 2 machine running the Unix Operating System on several test problems. We have used the following set of test problems: generate each attribute (m"5) of X randomly in the range [!5, 5], i i"1,2, n. We have tested the e$ciency of the algorithm for various combinations of n and k. These combinations include values for n"120, 240 and 360, and values for k"3, 4 and 5. In the tests, we require the number of objects in each cluster is equal to n/k. Each data set contains 100 problems. Table 1 shows the average CPU times for each data set used by the original k-means algorithm (t ) and the constrained k-means algorithm (t ). o c We "nd that the CPU time used by the constrained k-means algorithm is in average one-third greater than that used by the original k-means algorithm. Thus, the computational time of the constrained k-means algorithm does not increase signi"cantly when we compare it with the original k-means algorithm. As for the performance of the clustering result, the proposed constrained k-means algorithm can produce clusters with the "xed number of objects. However, the corresponding error of the original k-means algorithm is about 24.2, where the error is de"ned by
K
K
n error"+ number of objects in the ith cluster! . k i In other words, there are about 24 objects more than that we required in the clustering result.
Fig. 1. SMT machine.
Finally, we apply the constrained k-means algorithm to the printed circuit board (PCB) insertion problem. The machine is shown schematically in Fig. 1. As production output, the machine produces PCB's with components installed into the appropriate places in a placement sequence determined by the program. Our approach is to partition the set of electronic components into some clusters but also require the number of electronic components in each cluster to be almost the same, and the number is less than or equal to the limited capacity of feeder bank. After the clusters are determined, an e$cient algorithm for "nding the insertion sequence can be employed. Thus, we need to use the constrained k-means algorithm to determine these clusters. In each cluster, we expect that the corresponding positions of electronic components of each objects are also close together and therefore the travel time of the X}> table for the insertion of these electronic components will be small.
518
M.K. Ng / Pattern Recognition 33 (2000) 515}519
Table 2 Results for a real data set using the constrained k-mean algorithm Number of electronic components Number of types of components
91 37
Existing solution by the plant (time unit"s) E The lower bound (time unit"s) ¸ 1 Clustering-based solution by the algorithm (time unit"s)A The lower bound (time unit"s) ¸ 2
40.62 (37 reels, 1 assembly) 30.72 (37 reels, 1 assembly) 39.11 (15 reels, 4 assemblies) 36.58 (15 reels, 4 assemblies)
We test a real data set from the plant to illustrate the result. Table 2 shows the existing solution (the insertion sequence is by human experience) used by the plant and the lower bound [8] for the total assembly time of this data set. In this case, there is only one assembly of the PCB production process. For this solution, we are not satis"ed with the existing solution given by the plant since (E!¸ )/¸ ]100% is about 32.23%. We remark 1 1 that the electronic assembly plant only uses the machine to produce PCBs with the number of types of components being less than or equal to the maximum number of reels in the current production environment. In order to compare the performance of our approach with the existing solution used by the plant, we assume that the maximum number of reels in the feeder carriage is 15. Under the assumption, we need to use multiple assembly to produce this PCB data set. In Table 2, we show the result by using the clustering-based algorithm together with a comparison of the corresponding lower bound ¸ . 2 This result shows that our algorithm is quite e!ective. The percentage ((A!¸ )/¸ ]100%) is about 6.92% 2 2 ((32.23% by the existing solution). It implies that the constrained k-means algorithm can provide an e!ective clustering result. Then a better assembly plan comprising of multiple assembly sequence and placement orders of reels in the feeder can be derived. In summary, we have introduced the constrained kmeans algorithm for clustering objects. The most important result of this work is the procedure in Theorem 2 that allows the k-means paradigm to be used in generating the clusters with the "xed number of objects. The other important result is the proof of convergence that demonstrates a nice property of the constrained k-modes algorithm. Moreover, the experimental results have shown that the constrained k-means algorithms are e!ective. 4. Summary The k-means-type algorithm is well known for its e$ciency in clustering large data sets. The main idea behind
the k-means algorithm is the minimization of a objective function usually taken up as a function of the deviations between all patterns from their respective cluster centers. The minimization of such an objective function is sought employing an iterative scheme which starts with an arbitrary chosen initial cluster membership in an iterative manner to obtain a better clustering result. In some applications, we are interested in partitioning a set of objects into clusters with the "xed number of objects. For instance, the Surface Mount Technology (SMT) machine inserts electronic components into de"ned positions on a Printed Circuit Board (PCB) and the components are supplied from a set of reels each containing a tape of indentical electronic components. Because of the limited capacity of feeder bank, the electronic assembly plant uses multiple assembly to produce PCBs with number of types of components being greater than the maximum number of reels, i.e., more than one assembly process in the PCB insertion problem. Another application is mining association rules in large databases. When we use the distributed pruning technique to prune and reduce the number of candidates in the parallel mining process. We "nd that the e!ect of pruning increases as the data between partitions becomes more skew, i.e., the similarity of item sets in each cluster is high. However, in order to have nice parallelism of the fast parallel and distributed mining algorithm, the number of item sets in each cluster should be almost the same. This paper describes extensions to the k-means algorithm for clustering data sets. By adding suitable constraints into the mathematical program formulation, an approach is developed, which allows the use of the kmeans paradigm to e$ciently cluster data sets with the "xed number of objects in each cluster. The new algorithm is presented and the convergence of our new k-means algorithm will be showed. Finally, numerical results are reported to demonstrate the e!ectiveness of the algorithm. We also apply the constrained k-means algorithm to the printed circuit board insertion problem.
Acknowledgements Research by Michael Ng was supported in part by HKU CRCG Grant nos. 10201824 and 10201939. References [1] M. Anderberg, Cluster Analysis for Applications, Academic Press, New York, 1973. [2] G. Ball, D. Hall, A clustering technique for summarizing multivariate data, Behavioral Sci. 12 (1967) 153}155. [3] A. Jain, R. Dubes, Algorithms for Clustering Data, Prentice-Hall, Englewood Cli!s, NJ, 1988. [4] Y. Linde, A. Buzo, R. Gray, An algorithm for vector quantizer design, IEEE Trans. Commun. 28 (1980) 84}95.
M.K. Ng / Pattern Recognition 33 (2000) 515}519 [5] J.B. MacQueen, Some methods for classi"cation and analysis of multivariate observations, in: Proceedings of the Fifth Symposium on Mathematical Statistics and Probability, vol. 1, University of California Press, Berkeley, CA, 1967, pp. 281}297. [6] M. Ball, M. Magazine, Sequencing of insertion in printed circuit board assembly, Oper. Res. 36 (1988) 192}201. [7] L. McGinnis, M. Ammons, L. Carlyle, G. Granmer, K. Depuy, C. Ellis, T. Xu, Automated process planning for printed circuit card assembly, IIE Trans. 24 (1992) 18}30. [8] M. Ng, Heuristics approach to printed circuit board insertion problem, J. Oper. Res. Soc. 49 (1998) 1051}1059. [9] R. Agrawal, T. Imielinski, A. Swami, Mining association rules between sets of items in large databases, in: Proceed-
[10]
[11]
[12]
[13]
519
ings of the 1993 ACM-SIGMOD International Conference on Management of Data, Washington, DC, 1993. R. Agrawal, J. Shafer, Parallel Mining of Association Rules: Design, Implementation and Experience, Technical Report TJ10004, IBM Research Division, Almaden Research Center, 1996. J. Park, M. Chen, P. Yu, E$cient parallel data mining for association rules, in: Proceedings of the International Conference on Information and Knowledge Management, Baltimore, MD, Nov. (1995). R. Ahuja, T. Magnanti, J. Orlin, Network Flows: Theory, Algorithms, and Applications, Englewood Cli!s, Prentice-Hall, NJ, 1993. J. Kennington, R. Helgason, Algorithms for Network Programming, Wiley, New York, 1980.
About the Author*MICHAEL K. NG was born in Hong Kong, China, in 1967. He received B.Sc. and M.Phil. degress in Mathematics from the University of Hong Kong, in 1990 and 1993, respectively, and Ph.D. degree in Mathematics from the Chinese University of Hong Kong, in 1995. From 1995 to 1997 he was a Research Fellow at the Australian National University. He is currently an Assistant Professor in the Department of Mathematics at the University of Hong Kong. Ng's research interests are in the areas of scienti"c computing, operations research and data mining.
Pattern Recognition 33 (2000) 521}528
A new edited k-nearest neighbor rule in the pattern classi"cation problem Kazuo Hattori*, Masahito Takahashi Department of Electrical Engineering and Electronics, Toyohashi University of Technology, Tempaku-cho, Toyohashi 441-8580, Japan Received 29 October 1998; accepted 16 February 1999
Abstract A new edited k-nearest neighbor (k}NN) rule is proposed. For every sample y in the edited reference set, all the k- or (k#l)-nearest neighbors of y must be in the class to which y belongs. Here l denotes the number of samples which tie with the kth nearest neighbor of y with respect to the distance from y. The performance of the rule proposed has been investigated using three classi"cation examples. As a result, it is shown that the rule proposed will yield good results in many pattern classi"cation problems. ( 2000 Pattern Recognition Society. Published by Elsevier Science Ltd. All rights reserved. Keywords: Pattern classi"cation problems; k-Nearest neighbor rule; Wilson's edited k-nearest neighbor rule; Edited reference set; Number of samples misclassi"ed
1. Introduction In pattern classi"cation problems, the k-nearest neighbor (k}NN) rule [1}3] is a simple nonparametric decision rule. In the rule, an input sample is assigned to the class to which the majority among its k-nearest neighboring labeled samples are assigned. The k}NN rule [1}3] has been widely used since it is e!ective, when probability distributions of the feature variables are not known, and therefore, Bayes decision rule [2] is not applicable. Wilson [4] proposed an edited k}NN rule to improve the 1}NN rule [1}3]. In his rule, editing the reference set is "rst performed: Each sample in the reference set is classi"ed using the k}NN rule [1}3] and the set formed by eliminating it from the reference set. All the samples misclassi"ed are then deleted from the reference set. Afterward, any input sample is classi"ed using the 1}NN rule [1}3] and the edited reference set. Wilson's edited k}NN rule [4] has yielded good results in many "nitesample-size problems, although its asymptotic optimality has been disproved [5,6].
* Corresponding author. Fax: #0532-44-6757
Dasarathy [7] has developed a condensing method to edit the reference set: His method provides the minimal consistent subset (MCS) which is used as the edited reference set. All the samples in the reference set can be correctly classi"ed using the 1}NN rule [1}3] and the MCS. Dasarathy [7] has shown that the performance of the 1}NN rule [1}3] su!ers little degradation when a given large reference set is replaced by its much smaller MCS. Kuncheva [5] has applied a genetic algorithm [8] to editing the reference set for the k}NN rule [1}3]. Some "tness functions for the genetic algorithm [8] have been proposed, and the "tness function including a penalizing term is found to be e!ective to edit the reference set. In the editing method using the MCS [7] and that using the genetic algorithm [5,8], the number of samples in the edited reference set is considerably small, and therefore, the computational burden to classify input samples is much reduced. However, classi"cation accuracy in these two editing methods [5,7] is usually found not to be higher than that in the k}NN rule [1}3]. In Wilson's edited k}NN rule [4], the number of samples in the edited reference set is usually not so small. With respect to classi"cation accuracy, however, his rule is usually better in practical classi"cation problems than
0031-3203/00/$20.00 ( 2000 Pattern Recognition Society. Published by Elsevier Science Ltd. All rights reserved. PII: S 0 0 3 1 - 3 2 0 3 ( 9 9 ) 0 0 0 6 8 - 0
522
K. Hattori, M. Takahashi / Pattern Recognition 33 (2000) 521}528
the editing method using the MCS [7] and that using the genetic algorithm [5,8]. The purpose of this work is to propose a new edited k}NN rule. In the rule proposed, high classi"cation accuracy is more weighted than decrease in the number of samples in the reference set. Three classi"cation examples are presented to test the rule proposed. The performance of the rule proposed has been compared with those of the k}NN rule [1}3] and Wilson's edited k}NN rule [4].
using the k@}NN rule [1}3] and = (k) for all these pairs 1 of k and k@. The input sample should be assigned to the majority class which is determined by the classi"cation results obtained for all these pairs of k and k@. If a tie is present among classes determined by the classi"cation results, the input sample cannot but be assigned to a class arbitrarily chosen from the classes which tie.
3. Experimental results and discussion 2. A new edited k}NN rule In this section, a new edited k}NN rule has been proposed. Let = "Mx , x ,2, x N be the set of n 0 1 2 n labeled samples, namely, a reference set. In the rule proposed, the reference set = is edited as follows: 0 E-1. For each labeled sample x , 1)i)n, "nd the ki nearest neighbors of x from the set =(x ) which is i i given by Mx , x ,2, x , x ,2, x N. Let d (x ) be 1 2 i~1 i`1 n j i the distance from x to its jth nearest neighbor, and i assume d (x ))d (x ))2)d (x ). Let <(k, x ) be 1 i 2 i k i i the set of the k-nearest neighbors of x . If l di!erent i labeled samples in =(x ) satisfy the condition that i the distance from x to each of them is equal to i d (x ), include these l labeled samples in <(k, x ), k i i i.e., obtain the (k#l)-nearest neighbors of x . i Check the condition that all the samples in <(k, x ) i should be assigned to the class to which x is i assigned. If the condition is satis"ed, take x as i a typical sample in the class to which x belongs; i otherwise, x is not any typical sample. i E-2. Obtain the set of all the typical samples as the edited reference set = (k). 1 Any input sample is classi"ed using the k@}NN rule [1}3] and = (k), where k@ is not restricted to 1. In the 1 practical use of the rule proposed, the values of k and k@ should be optimally determined as follows: K-1. Determine the range of k and that of k@, properly. Express any possible pair of k and k@ as (k, k@). K-2. For each pair (k, k@), obtain = (k) and classify each 1 sample x in = using the k@}NN rule [1}3] and the 0 set = (k, x). Here = (k, x) is the set formed by 1 1 eliminating x from = (k), if x is in = (k). If x is not 1 1 in = (k), the set = (k, x) is given by = (k). Evaluate 1 1 1 the number of samples misclassi"ed for each pair (k, k@). K-3. Find the pair (k , k@ ) which yields the minimum m m number of samples misclassi"ed, and take it as the optimal pair of k and k@. If more than one pair yield the minimum number of samples misclassi"ed, take all these pairs as the optimal pairs of k and k@. When more than one pair are obtained as optimal pairs of k and k@, any input sample should be classi"ed
Three classi"cation examples are presented to test the rule proposed. For the sake of comparison, the k}NN rule [1}3] and Wilson's edited k}NN rule [4] are also tested in the examples. The values of k and k@ are taken in the range of 1)k)15 and 1)k@)15. In Wilson's edited k}NN rule [4], the 1}NN rule [1}3] is usually used with the edited reference set to classify input samples, i.e., k@ is restricted to 1. In this work, however, the k@}NN rule [1}3] with k@*1 is used in Wilson's edited k}NN rule [4]. With respect to classi"cation accuracy, the rule proposed and Wilson's edited k}NN rule [4] are estimated as follows: Each sample x in the data set of the example is classi"ed using the k@}NN rule [1}3] and the edited reference set, if x is not in the edited reference set. If x is in the edited reference set, the set formed by eliminating x from the edited reference set is obtained as the reference set, and x is classi"ed using the k@}NN rule [1}3] and the reference set thus obtained. The number of samples misclassi"ed is then evaluated. Classi"cation accuracy of the k}NN rule [1}3] is estimated as follows: Each sample x in the data set of the example is classi"ed by the k}NN rule [1}3] in which the k-nearest neighbors of x are obtained from the set formed by eliminating x from the data set. The number of samples misclassi"ed is then evaluated. The "rst example is classi"cation of the Iris data set [9]. The data set consists of 150 #owers which are classi"ed into three classes, namely, Setosa, Versicolor and Virginica. Each class includes 50 #owers. Every #ower is characterized by four features, and therefore, it is expressed by a four-dimensional feature vector. Table 1 shows the number of samples taken from each class to form the edited reference set in the rule proposed and that in Wilson's edited k}NN rule [4]. Evaluation of the number of samples misclassi"ed has been systematically performed. The results of the rule proposed and those of Wilson's edited k}NN rule [4] are shown in Tables 2 and 3, respectively. The results of the k}NN rule [1}3] are shown in Table 4. The second example is classi"cation of the butter#ytype data set which was presented by Pham and Corrochano [10]. The data set consists of 51 two-dimensional points. It is classi"ed into two classes forming the `wingsa of the butter#y. Fig. 1 depicts the data set and
K. Hattori, M. Takahashi / Pattern Recognition 33 (2000) 521}528
523
Table 1 Number of samples taken from each class in editing the Iris data set Wilson's edited k-NN rule
Rule proposed k
Setosa
Versicolor
Virginica
Total
Setosa
Versicolor
Virginica
Total
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
50 50 50 50 50 50 50 50 50 50 50 50 50 50 50
47 47 45 44 43 43 42 40 39 36 35 33 31 31 30
47 45 43 41 36 36 34 34 33 32 31 29 29 28 26
144 142 138 135 129 129 126 124 122 118 116 112 110 109 106
50 50 50 50 50 50 50 50 50 50 50 50 50 50 50
47 47 47 47 47 47 46 47 47 46 48 47 47 47 47
47 47 47 47 48 47 49 48 48 48 48 48 48 49 49
144 144 144 144 145 144 145 145 145 144 146 145 145 146 146
Table 2 Number of samples misclassi"ed by the rule proposed in the Iris data set k k@
1
2
3
4
5, 6
7
8
9
10
11
12
13
14
15
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
5 5 4 4 4 4 3 3 3 3 3 3 3 3 4
5 5 4 4 4 4 2 2 3 2 3 3 3 3 3
4 4 4 4 3 3 4 2 2 2 2 3 3 2 3
5 5 4 4 4 3 3 3 2 3 4 3 4 5 8
6 6 8 7 6 8 9 8 8 8 9 9 10 9 11
6 6 8 7 9 8 9 8 9 9 9 9 9 9 9
5 5 9 8 9 8 8 8 9 9 9 9 8 9 8
5 5 9 8 9 8 8 8 9 9 9 9 9 9 8
7 7 9 9 9 8 7 7 7 9 8 9 8 8 8
9 9 9 9 9 9 8 9 9 9 8 9 9 10 9
11 11 9 9 10 9 9 9 9 9 10 10 10 10 13
10 10 7 8 9 10 9 10 10 9 11 10 10 11 11
12 12 10 10 10 10 10 10 10 10 11 11 13 11 14
8 8 9 10 8 9 10 10 10 10 13 11 13 11 10
Average
3.60
3.33
3.00
4.00
8.13
8.27
8.07
8.13
8.00
10.93
10.00
shows that there is much overlapping between two classes 1 and 2. The number of samples taken from each class to form the edited reference set is shown in Table 5. The numbers of samples misclassi"ed are shown in Tables 6}8. The third example is classi"cation of the data set of 162 two-dimensional points which are shown in Fig. 2. The data set was "rst used by Yan [11]. As shown in Fig. 2,
8.93
9.87
9.67
four classes 1}4 are present in the data set. Table 9 shows the number of samples taken from each class to form the edited reference set. Tables 10}12 show the numbers of samples misclassi"ed. As is clear from Tables 1, 5 and 9, the number of samples in the edited reference set in the rule proposed is equal to or less than that in Wilson's edited k}NN rule [4]. This is due to the fact that for any sample x in the
524
K. Hattori, M. Takahashi / Pattern Recognition 33 (2000) 521}528
Table 3 Number of samples misclassi"ed by Wilson's edited k}NN rule in the Iris data set k k@
1, 2, 3, 4, 6
5
7, 14
8
9
10
11
12
13
15
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
5 5 4 4 4 4 3 3 3 3 3 3 3 3 4
5 5 4 4 4 4 5 4 3 4 3 5 4 3 4
5 5 4 4 4 4 5 4 4 5 3 5 5 4 4
5 5 4 4 4 4 4 4 3 3 3 4 3 3 4
5 5 4 4 4 4 4 3 5 4 3 5 5 4 3
5 5 4 4 4 4 4 3 4 4 3 5 5 4 3
6 6 5 5 4 4 4 3 5 4 4 5 5 4 3
6 6 5 5 4 4 4 4 3 3 3 4 3 3 4
6 6 5 6 4 4 5 3 3 3 3 4 3 3 3
6 6 5 5 4 4 5 4 4 5 3 5 5 4 4
Average
3.60
4.07
4.33
3.80
4.13
4.07
4.47
4.07
4.07
4.60
Table 4 Number of samples misclassi"ed by the k}NN rule in the Iris data set k"k@ 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 6 6 6 6 5 6 5 5 5 6 4 5 5 4 4
Average 5.20
reference set, the condition for x to be included in the edited reference set in the rule proposed is severer than that in Wilson's edited k}NN rule [4]: In Wilson's edited k}NN rule [4], the majority among the k-nearest neighbors of x must be in the class to which x belongs for x to be included in the edited reference set. In the rule proposed, however, all the k- or (k#l)-nearest neighbors
Table 5 Number of samples taken from each class in editing the butter#y-type data set Wilson's edited k}NN rule
Rule proposed
Fig. 1. Butter#y-type data set.
k
Class 1
Class 2
Total
Class 1
Class 2
Total
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
27 25 25 24 22 21 21 21 21 19 18 17 16 16 16
22 20 18 17 17 15 14 14 14 13 13 11 11 10 10
49 45 43 41 39 36 35 35 35 32 31 28 27 26 26
27 27 25 26 26 26 27 26 25 26 25 25 25 26 26
23 23 24 24 24 24 24 24 23 23 23 23 23 23 23
50 50 49 50 50 50 51 50 48 49 48 48 48 49 49
K. Hattori, M. Takahashi / Pattern Recognition 33 (2000) 521}528
525
Table 6 Number of samples misclassi"ed by the rule proposed in the butter#y-type data set k k@
1
2
3, 4
5
6
7, 8, 9
10
11
12
13
14
15
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
1 1 1 0 0 0 0 0 1 1 2 1 2 1 1
1 1 0 0 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
0 0 0 0 1 1 1 0 1 1 1 1 1 1 1
0 0 1 1 1 1 1 1 1 1 1 1 1 1 2
0 0 1 1 1 1 1 1 1 1 1 1 2 2 2
0 0 1 0 1 1 1 1 1 1 1 1 2 1 2
0 0 1 0 1 1 1 1 1 1 1 1 1 1 2
1 1 1 1 2 1 1 1 2 1 3 2 2 2 2
1 1 1 1 2 1 1 1 1 1 3 2 2 2 2
1 1 1 1 2 1 2 1 2 2 3 2 2 2 3
1 1 1 1 2 1 1 1 2 2 3 2 2 1 3
Average
0.80
0.87
1.00
0.67
0.93
1.07
0.93
0.87
1.53
1.47
1.73
1.60
of x obtained from the procedure E-1 must be in the class to which x belongs. In the rule proposed, the condition for a sample to be included in the edited reference set becomes severer with
Table 8 Number of samples misclassi"ed by the k}NN rule in the butter#y-type data set
Table 7 Number of samples misclassi"ed by Wilson's edited k}NN rule in the butter#y-type data set
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 1 1 2 1 1 1 0 1 3 2 3 3 3 2 2
k"k@
k k@
1, 2
3
4, 5, 6, 8
7
9, 11, 12, 13
10, 14, 15
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
1 1 2 1 1 1 0 1 2 2 3 2 2 2 2
1 1 2 1 1 1 2 2 3 3 3 3 3 3 3
1 1 2 1 1 1 1 2 3 3 3 3 3 3 3
1 1 2 1 1 1 0 1 3 2 3 3 3 2 2
1 1 2 1 1 1 2 1 3 3 3 2 2 2 3
1 1 2 1 1 1 1 1 3 3 3 2 2 2 3
Average
1.53
2.13
2.07
1.73
1.87
1.80
Fig. 2. Data set of 162 points.
Average 1.73
526
K. Hattori, M. Takahashi / Pattern Recognition 33 (2000) 521}528
Table 9 Number of samples taken from each class in editing the data set of 162 points Wilson's edited k}NN rule
Rule proposed k
Class 1
Class 2
Class 3
Class 4
Total
Class 1
Class 2
Class 3
Class 4
Total
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
36 36 36 36 36 35 35 35 33 33 33 33 31 31 31
57 56 56 56 56 55 55 55 55 55 55 54 54 54 54
33 33 33 31 31 29 29 29 28 28 24 24 24 24 24
34 34 33 33 33 33 32 32 32 32 32 32 32 32 31
160 159 158 156 156 152 151 151 148 148 144 143 141 141 140
36 36 36 36 36 36 36 36 36 36 36 36 36 36 36
57 57 57 57 56 56 56 56 56 56 56 56 56 56 56
33 33 34 34 34 34 34 34 34 34 34 34 34 34 34
35 35 34 34 34 34 33 33 33 33 33 34 33 33 33
161 161 161 161 160 160 159 159 159 159 159 160 159 159 159
Table 10 Number of samples misclassi"ed by the rule proposed in the data set of 162 points k k@
1, 2
3
4, 5
6
7, 8
9, 10
11
12
13, 14
15
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
0 0 1 1 3 3 3 3 3 2 3 3 3 3 3
1 1 1 1 3 3 3 3 3 2 3 3 3 3 3
0 0 1 1 2 2 2 2 2 2 3 3 3 3 3
0 0 2 2 2 2 2 2 2 2 2 2 2 2 2
1 1 2 2 2 2 2 2 3 2 3 2 2 2 3
1 1 2 1 2 1 1 1 2 2 2 2 2 2 3
0 0 2 1 2 1 1 1 2 2 2 2 2 2 2
1 1 2 2 2 2 2 2 2 2 2 2 2 2 2
0 0 1 1 1 1 2 2 2 2 2 2 2 2 2
1 1 1 1 1 1 2 2 2 2 2 2 2 2 2
Average
2.27
2.40
1.93
1.73
2.07
1.67
1.47
1.87
1.47
1.60
increasing k. Therefore, the number of samples in the edited reference set becomes smaller with increasing k. This tendency is found in Tables 1, 5 and 9. In the tables, however, the number of samples in the edited reference set in Wilson's edited k}NN rule [4] is not so changed by k. In the special case of k"1 and l"0, the edited reference set in the rule proposed is identical with that in Wilson's edited k}NN rule [4]. This special case is found for k"1 in Table 1.
From Tables 2}4, the minimum number of samples misclassi"ed in the Iris data set [9] is estimated to be 2 in the rule proposed, 3 in Wilson's edited k}NN rule [4] and 4 in the k}NN rule [1}3]. Further, it is found from Tables 2}4 that the results of the rule proposed for k"2 and 3 are much better than those of the k}NN rule [1}3] and Wilson's edited k}NN rule [4] for 1)k)15: The average number of samples misclassi"ed in the range of 1)k@)15 is very small for k"2 and 3 in the rule
K. Hattori, M. Takahashi / Pattern Recognition 33 (2000) 521}528 Table 11 Number of samples misclassi"ed by Wilson's edited k}NN rule in the data set of 162 points k
527
better than those of the k}NN rule [1}3] and Wilson's edited k}NN rule [4] for 1)k)15.
4. Conclusions
k@
1, 2
3, 4, 5, 6, 12
7, 8, 9, 10, 11, 13, 14, 15
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
1 1 1 1 2 2 3 2 3 2 2 2 3 3 3
0 0 1 1 3 3 3 3 3 3 3 3 3 3 3
1 1 1 1 3 3 3 3 3 3 3 3 3 3 3
Average
2.07
2.33
2.47
Table 12 Number of samples misclassi"ed by the k}NN rule in the data set of 162 points
A new edited k}NN rule has been proposed, and its performance is investigated using three classi"cation examples. In the examples, the rule proposed has yielded better results than the k}NN rule [1}3] and Wilson's edited k}NN rule [4]. This results from the fact that good edited reference sets are obtained in the rule proposed. The condition for a sample to be included in the edited reference set in the rule proposed is severer than that in Wilson's edited k}NN rule [4]. In the rule proposed, every sample y in the edited reference set for any k must be surrounded by its k- or (k#l)-nearest neighbors which are all in the class to which y belongs. Accordingly, every sample y in the edited reference set is very proper as a typical sample in the class to which y belongs. The number of samples in the edited reference set in the rule proposed is shown to be equal to or less than that in Wilson's edited k}NN rule [4], and it decreases with increasing k. Therefore, editing the reference set using a too large value of k is not good, and k should be properly determined in some range of k. If the values of k and k@ are optimally determined by taking the procedures from k-1 to k-3, the rule proposed will yield good results in many pattern classi"cation problems.
k"k@ 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 1 1 1 1 2 2 3 3 3 3 3 2 3 3 3
Average 2.27
proposed. However, Table 2 shows that the results of the rule proposed for k*5 are not good. This will be due to serious decrease of samples in the edited reference set for k*5. From Tables 6}8, the minimum number of samples misclassi"ed in the butter#y-type data set [10] is obtained as zero in the rule proposed, Wilson's edited k}NN rule [4] and the k}NN rule [1}3]. Tables 6}8 show that in the evaluation of the average number of samples misclassi"ed in the range of 1)k@)15, the results of the rule proposed for 1)k)11 are much better than those of the k}NN rule [1}3] and Wilson's edited k}NN rule [4] for 1)k)15. From Tables 10}12, the minimum number of samples misclassi"ed in the data set of 162 points [11] is evaluated as zero in the rule proposed and Wilson's edited k}NN rule [4], and as 1 in the k}NN rule [1}3]. Tables 10}12 show that in the evaluation of the average number of samples misclassi"ed in the range of 1)k@)15, the results of the rule proposed for 9)k)15 are much
5. Summary A new edited k}NN rule has been proposed. In the rule, the condition for a sample x to be included in the edited reference set is that all the k- or (k#l)-nearest neighbors of x must be in the class to which x belongs. Here l denotes the number of samples which tie with the kth nearest neighbor of x with respect to the distance from x. Accordingly, this condition is severer than that in Wilson's edited k}NN rule, and every sample in the reference set in the rule proposed is very proper as a typical sample in the class to which it belongs. The number of samples in the edited reference set in the rule proposed is equal to or less than that in Wilson's edited k}NN rule, and it decreases with increasing k. Three classi"cation examples are presented to test the rule proposed. The k}NN rule and Wilson's edited k}NN rule are also performed in the examples. The rule proposed is found to yield better results than the k}NN rule and Wilson's edited k}NN rule. Any input sample can be classi"ed using the k@}NN rule and the edited reference set. If the values of k and k@ are optimally determined by taking the procedures presented in this work, the rule proposed will yield good results in many pattern classi"cation problems.
528
K. Hattori, M. Takahashi / Pattern Recognition 33 (2000) 521}528
References [1] E. Fix, J.L. Hodge Jr., Discriminatory analysis, nonparametric discrimination, consistency properties, U.S. Air Force Sch. Aviation Medicine, Randolf Field, Texas, Project 21-49-004, Contract AF 41 (128)-31, Rep. 4, 1951. [2] R.O. Duda, P.E. Hart, Pattern Classi"cation and Scene Analysis, Wiley, New York, 1973. [3] P.A. Devijver, J. Kittler, Pattern Recognition: A Statistical Approach, Prentice-Hall, London, 1982. [4] D.L. Wilson, Asymptotic properties of nearest neighbor rules using edited data, IEEE Trans. System Man Cybernet. 2 (1972) 408}421. [5] L.I. Kuncheva, Fitness functions in editing k}NN reference set by genetic algorithms, Pattern Recognition 30 (1997) 1041}1049.
[6] P.A. Devijver, J. Kittler, On the edited nearest neighbor rule, Proc. 5th Int. Conf. on Pattern Recognition, 1980, pp. 72}80. [7] B.V. Dasarathy, Minimal consistent set (MCS) identi"cation for optimal nearest neighbor decision systems design, IEEE Trans. System Man Cybernet. 24 (1994) 511}517. [8] D. Goldberg, Genetic Algorithms in Search, Optimization and Machine Learning, Addison-Wesley, Reading, MA, 1989. [9] E. Anderson, The irises of the Gaspe peninsula, Bull. Am. Iris Soc. 59 (1935) 2}5. [10] D.T. Pham, E.J.B. Corrochano, Self-organizing neuralnetwork-based pattern clustering method with fuzzy outputs, Pattern Recognition 27 (1994) 1103}1110. [11] H. Yan, Prototype optimization for nearest neighbor classi"ers using a two-layer perceptron, Pattern Recognition 26 (1993) 317}324.
About the Author*KAZUO HATTORI received his Ph.D. degree in Applied Physics from Tokyo University, Japan, in 1977. He is at present an Associate Professor in the Department of Electrical Engineering and Electronics at Toyohashi University of Technology. His research interests are in the areas of cluster analysis, fuzzy inference and decision theory. About the Author*MASAHITO TAKAHASHI received his M.E. degree in Electrical Engineering and Electronics from Toyohashi University of Technology, Japan, in 1997. He is currently engaged in Ph.D. research in the Department of Electrical Engineering and Electronics, Toyohashi University of Technology.
Pattern Recognition 33 (2000) 529}531
A new parallel feature-based stereo-matching algorithm with "gural continuity preservation, based on hybrid symbiotic genetic algorithms J.Y. Goulermas*, P. Liatsis Control Systems Centre, Department of EE&E, UMIST, P.O. Box 88, Manchester, M60 1QD, UK Received 19 May 1999
1. Introduction The calculation of shape from stereo requires matching of points from the left image with points from the right image. Many classes of stereo-matching algorithms have been proposed [1}3] with di!erences in the type of tokens they match and the optimisation method and the constraints they use. They are mainly categorised as area-based or feature-based. Feature-based methods can be either low level, which try to match individual edgels or high level, which try to match edge structures, such as lines or curves. While high-level methods enable comparisons of more complex features and incorporate naturally the "gural continuity (FC) [1,4] constraint, they can su!er from problems of imprecise disparity estimation as well as fragmentation and fragility [3]. In this work we present a novel low-level edge-based algorithm. Its main advantages are explicit optimisation of the FC across bands of epipolars, elimination of the ordering constraint (OC) and highly parallel distributed execution. Each epipolar is searched in parallel for a high con"dence intra-row matching decision (at the local level). Simultaneously, temporary matching information is propagated to the adjoining epipolars for minimisation of FC violations (at the global level).
2. Intra-row formulation and matching Using canonical camera con"guration we can restrict the search within a single epipolar. To match the ith
* Corresponding author. Tel.: #161-200-4651; fax: #161200-4647.
edgel l of the uth row in the left image, with the jth edgel ui r of the same row in the right image, we de"ne a match uj functional M(l , r ) which calculates the merit or con"ui uj dence of matching the two edgels. In general, the closer the attributes of two features are, the higher their matching potential. We construct a functional M( ) ) which takes into account the mean intensity similarity of the left and right edge-delimited intervals, the gradient orientation, the gradient magnitude and the local edge-pro"le resemblance estimated by a 3}4 chamfer distance transform. Each attribute similarity is estimated separately and their combination is achieved through fuzzy logic. We de"ne a single matching feasibility curve for each attribute, giving a response from 0 to 1 depending on whether the similarity is low or high. M( ) ) combines the "ve responses as a con#uence of the possibilities, via a product norm. We follow such a fuzzy approach because a weighted sum is more sensitive to violations of the geometric and photometric invariance between the two stereo images. To take into account occlusions, we also de"ne M(l , o) (where o is a special token assigned ui with replacement) to be the possibility of having the left edgel l unmatched. Its value is not "xed and depends on ui the possibility of ignoring the matching of l with its best ui right-epipolar candidate. M(o, r ) is de"ned similarly for uj occluded right edgels. Having de"ned M( ) ) for all possible cases, the search for the optimum intra-row matching is achieved via the following constrained maximisation:
A
B
max + + M(l , r )x #+ M(l , o)y #+ M(o, r )z ui uj ij ui i uj j i j i j (1) s.t: y #+ x "1, ∀i and z #+ x "1, ∀j, i ij j ij j i
0031-3203/00/$20.00 ( 2000 Pattern Recognition Society. Published by Elsevier Science Ltd. All rights reserved. PII: S 0 0 3 1 - 3 2 0 3 ( 9 9 ) 0 0 1 4 5 - 4
(2)
530
J.Y. Goulermas, P. Liatsis / Pattern Recognition 33 (2000) 529}531
where x , y and z are 1, i! l matches r , l is unij i j ui uj ui matched and r is unmatched, respectively, and 0 otheruj wise. This modelling of the problem is equivalent to a weighted bipartite graph and the optimum intra-row matching sequence can be obtained by "nding the maximally weighted bipartite matching of the graph using linear assignment problem (LAP) solvers. However, because this produces a one-to-one matching, we propose a graph transformation to convert the optimisation to a one-to-at-most-one matching, so that it accounts for occlusions. The transformation introduces special dummy nodes and arcs in order to keep the graph bipartite. Other low-level algorithms [1}3] de"ne a match functional and use dynamic programming (DP) to "nd the optimum decision per epipolar. Nevertheless, since DP requires the OC to work, when there are thin occluding objects and/or wide camera baselines (which give rise to matching reversals), the search falters [1]. Although, Eqs. (1) and (2) produce optimal matching decisions for each epipolar, the decisions are only locally (at the intra-row level) optimal. At the global (inter-row) level, where FC factors are to be considered, these decisions are suboptimal and give rise to Type-I [4] errors. The key idea of this work is to seek a particular set of locally suboptimal intra-row decisions, which become optimal at the global level.
3. Symbiotic genetic algorithms (SGAs) To produce optimality at both levels we make use of genetic algorithms (GAs). A newly introduced type of GAs, the coevolutionary GAs [5], is capable of deploying a set of populations that attempt to solve interdependent subproblems or subparts of a di$cult large-scale problem with greater e$ciency. In our approach we allocate a single population at each epipolar, whose members uniquely represent valid bipartite matchings for that epipolar. The populations evolve symbiotically in such a way, that search is driven by two objectives: the maximisation of Eq. (1) and the minimisation of FC violations. The latter objective can only be sustained through symbiosis, as it requires global information. Available information pertaining to the current best members/decisions from the neighbouring populations/ epipolars is propagated to allow a quantitative measurement of FC. Speci"cally, for the uth population P and its u kth chromosome c , we suggest the following symbiotic uk credit assignment function: M(c ) (1!w ) uk # s f (c )"w + w(Dv!uD) u uk s N + w(Dv!uD) u v|Vu v|Vu q(c , cH) M(cH) uk v v . j N uv v
]
(3)
The "rst part of the summation corresponds to the "rst objective, normalised by the number N of the total u number of features on both sides of the epipolar. The second part corresponds to the second objective. The con#ict of the two objectives is balanced with a "xed weight w 3(0, 1). < is the set of row indices which are s u connected with row u through at least one edge chain (at the preprocessing stage, Canny-based edge detection is followed by a contour tracking which groups all edgels into distinct chains). < corresponds to dependent popuu lations (symbionts) of P that in#uence and direct its u evolution. The term w(Dv!uD) represents a linear weighting of the contributions of the symbionts of P . The u greater the distance Dv!uD of row v3< from row u is, the u less critical the symbiosis between P and P are and thus, u v the weighting decreases. j is a measure of the structural uv dependency between rows u and v and is equal to the cardinality of the set of left-edgel pairs of the two rows that belong to the same left chain. q(c , cH) is a measure uk v of the matching compatibility between the two bipartite matching decisions represented by the chromosomes c and cH (the currently best member of population P ). uk v v It is equal to the number of left edgel pairs that belong to the same left chain and at the same time, they match right edgels which also belong to the same right chain. In this way, the ratio q( ) )/j is a value within [0, 1] that manifests the FC-compatibility between two matching sequences. The higher the ratio, the better way the FC is abided by. Using Eq. (3) all the populations coevolve and in#uence each other symbiotically. The "tness of a chromosome c is reinforced if it is FC-compatible with the best uk members of the adjacent populations, which, at the same time, have high intra-row scores M(cH). v We have developed a set of problem-speci"c genetic operators as well as a non-binary chromosome encoding scheme. Crossover and mutation are designed so that they never produce invalid solutions, that is decisions with disparities out of a prede"ned range [d , d ] .*/ .!9 or decisions which violate the uniqueness constraint (as we assume opaque objects only). Thus, the search is restricted within feasible regions. To increase e$ciency we propose a novel genetic operator, called Interbreeding. Like crossover it produces new o!spring, but it mates multiple parents from di!erent populations. Since the populations correspond to di!erent species (di!erent epipolars) with incompatible genetic material, we designed a probabilistic &projection' mechanism which uses the concept of edge and "gural continuity, to generate o!spring that have better chances in satisfying the FC constraint. Thus, the operator incorporates useful properties of crossover recombination as well as local search. Furthermore, by using our proposed graph transformation method (mentioned in Section 2), we apply a fast LAP algorithm to each epipolar prior to the SGA optimisation. Since, the use of the robust fuzzy functional produces locally optimal decisions which are expected to
J.Y. Goulermas, P. Liatsis / Pattern Recognition 33 (2000) 529}531
531
Fig. 1. Testing on the Pentagon images.
be in the vicinity of the globally optimal ones, we use the LAP generated solutions to hybridise the collective of the SGA populations. Speci"cally, we seed all the chromosomes of all the initial populations with LAP generated decision fragments. Although in both cases solutions are identical, the evolution time needed for the SGAs to converge is reduced dramatically.
4. Results Fig. 1 shows results for the Pentagon pair, where images (A&B) are produced by the LAP algorithm and (C&D) from the SGAs. (A&C) show the "nal interpolated disparity maps; the improvement of FC of the SGAs over the LAP is apparent from the higher smoothness and the reduced streaking. (B&D) are symmetric normalised intensity images representing the q(d , d )/j u v uv ratios de"ned for every row pair (u, v), with d and d u v being the "nal matching (disparity) sequences at those rows. The darker each pixel at coordinates (u, v) is, the
higher the FC is preserved between rows u and v in the "nal disparity map. (D) illustrates a higher degree of inter-row consistency than (B), as it appears darker, especially near the diagonal region.
References [1] U.R. Dhond, J.K. Aggarwal, Structure from stereo*a review, IEEE Trans. SMC 19 (6) (1989) 1489}1510. [2] G.A. Jones, Constraint, optimisation and hierarchy: reviewing stereoscopic correspondence of complex features, CVIU 65 (1) (1997) 57}78. [3] T. Ozanian, Approaches for stereo matching: a review, Model. Identi"cation Control 16 (2) (1995) 65}94. [4] R. Mohan, G. Medioni, R. Nevatia, Stereo error detection, correction and evaluation, IEEE Trans. Pattern Anal. Mach. Intell. 11 (2) (1989) 113}120. [5] M.A. Potter, K.A. DeJong, A cooperative coevolutionary approach to function optimisation, third PPSN Conference, Lecture Notes Computer Science, Vol. 886, Springer, Berlin, 1994, pp. 249}257.
