JP3903783B2 - Face metadata generation method and apparatus, and face similarity calculation method and apparatus - Google Patents

Description

ここで、特徴ベクトルｖ₀を２回観測するとすれば、二つの観測ベクトルｖ₁とｖ₂が得られる（数２）。
【００３１】
【数２】

いま、誤差ベクトルε₁、誤差ベクトルε₂に対して相関を持つ信頼性指標θ₁、信頼性指標θ₂がそれぞれ得られ、その事後分布がｐ(ε|θ₁)、ｐ(ε|θ₂)であるとする。
【００３２】
このような分布が得られたときのｖ₁とｖ₂の差ベクトルｓの事後分布がｐ(ｓ|θ₁,θ₂)と表されるとすれば、パターン間の類似度ｄ(ｖ₁,ｖ₂)として、次の対数尤度を用いることができる。
【００３３】
【数３】

事後分布ｐ(ε|θ₁)、ｐ(ε|θ₂)がそれぞれ正規分布であるとすると、差ベクトルｓの分布ｐ(ｓ|θ₁, θ₂)も正規分布となる。ここで、誤差ベクトルの事後分布ｐ(ε|θ_i) (i=1,2)として平均０で共分散行列Σε(θ_i)の正規分布を考えると、差ベクトルｓの分布は、平均０で、共分散行列Σ_s(θ₁, θ₂)は次式（数４）となる。
【００３４】
【数４】

つまり、事後分布ｐ(ｓ|θ₁, θ₂)は次式（数５）で表される。
【００３５】
【数５】

従って、（数３）は、次式（数６）に示すように、共分散行列Σε(θ₁)やΣε(θ₂)を用いて、信頼性指標θ₁,やθ₂に対して適応的なマハラノビス距離に書き表すことができる。
【００３６】
【数６】

誤差ベクトルの各要素間における独立性を仮定すれば、（数５）は次式（数７）となる。
【００３７】
【数７】

ここで、σ_s,k(θ₁, θ₂)²は共分散行列Σ_s(θ₁, θ₂)のｋ番目の対角要素であり、つまり、観測誤差の分散である。また、σε_,k(θ₁)²、σε_,k(θ₂)²は、それぞれ共分散行列Σε(θ₁)、Σε(θ₂)のｋ番目の対角要素である。ｓ_kは差ベクトルｓのｋ番目の要素である。
【００３８】
このように正規分布を仮定することで、（数３）は、次式（数８）のように各特徴ベクトルの要素毎の分散σε_,k(θ₁)、σε_,k(θ₂)を用いて、信頼性指標θ₁やθ₂に対して適応的なマハラノビス距離によって類似度を定義することができる。
【００３９】
【数８】

ここで、ｖ_1,k、ｖ_2,kはそれぞれ特徴ベクトルｖ₁、ｖ₂のｋ番目の要素である。
【００４０】
上記説明では事後分布ｐ(ｓ|θ₁, θ₂)として正規分布を仮定したが、以下、混合正規分布を仮定する。さて、事後分布ｐ(ｓ|θ₁, θ₂)として、次式（数９）によって示すように、正規分布ｐ(ｓ|θ₁, θ₂, j) (j=1,2,...,M)の和によって事後分布ｐ(ｓ|θ₁, θ₂)が表現できると仮定する。
【００４１】
【数９】

従って、次式（数１０）により適応的混合マハラノビス距離を定義することができる。
【００４２】
【数１０】

事後分布ｐ(ｓ|θ₁, θ₂, j)の共分散行列Σ_s(θ₁, θ₂, j)およびP(j)の推定については、一般的な推定方法である最尤推定法やＥＭアルゴリズムを用いて推定することができる (C.M. Bishop, "Neural Networks for Pattern Recognition", Oxford University Express, 1995)。
【００４３】
混合正規分布を仮定することで、分布をより正確に近似でき、マッチング性能が向上するが、大量の学習データが必要になるとともに演算量もまた大幅に増加する。
【００４４】
前述の誤差分布に基づくマハラノビス距離は、検索する顔が他の登録されている顔データの中でどれに最も近いかを調べる顔同定(face identification)のような問題に対して優れている距離である。
【００４５】
一方、顔識別(face verification)の問題では、入力された顔が登録された画像との同一性を判定する上で、受け入れるか、棄却するかが重要な問題である。以下で説明する「判別距離」と名付ける距離は、この顔識別問題に対しては 前述のマハラノビス距離を用いる場合よりも、優れた類似度尺度となっている。
【００４６】
いま、顔の二つの特徴ベクトルｖが、一致すると判定すべき、つまり、二つの特徴ベクトルが同一クラス内に属している(例えば、二つの特徴ベクトルが同じ人物の顔データである)ならば、それらの特徴ベクトルの組み合わせはクラスＷに属しているとする。また、二つの特徴ベクトルｖが不一致と判定すべき、つまり、クラス間の特徴ベクトルである(例えば、二つの特徴ベクトルが異なった人物の顔データである)ならば、それらの組み合わせはクラスＢに属しているとする。
【００４７】
二つの特徴ベクトルｖ₁、ｖ₂に対して信頼性指標θ₁、θ₂が得られるとする。差ベクトルｓと二つの信頼性指標θ₁、θ₂(以下、二つの信頼性指標のセットを[θ_i]と表記する。)が観測されたときに一致している と見倣すクラスＷと不一致と見倣すべきクラスＢとの判別問題を考えると、次式（数１１）の判別則(decision rule)が得られる。
【００４８】
【数１１】

上記（数１１）の左辺は、ベイズ定理(Bayes Theorem)によって次式（数１２）のように書き直すことができる。
【００４９】
【数１２】

ここで、Ｗ、Ｂと[θ_i]の生じる確率は独立であると仮定して、P(Ｗ,[θ_i])=P(Ｗ)P([θ_i])、P(Ｂ,[θi])=P(Ｂ)P([θ_i])としている。
【００５０】
パターン間の距離ｄ(ｖ₁,ｖ₂)として（数１２）の対数尤度を計算することで、次式（数１３）のように、顔識別問題に適した類似度を求めることができる。
【００５１】
【数１３】

個別のマッチング毎に事前確率Ｐ(Ｗ)、Ｐ(Ｂ)が異なり、且つ、それを知ることができるならば、（数１３）の第２項を計算することが望ましい。しかし、多くの場合、個別のマッチング毎に事前確率を知ることができないので、事前確率は一定と仮定することで、第２項を一定と見倣し、類似度の計算からは除外する。
【００５２】
事後確率Ｐ(ｓ|Ｗ,[θ_i])、Ｐ(ｓ|Ｂ,[θ_i])を、それぞれクラス内分布ｐ_W(ｓ|[θ_i])、クラス間分布ｐ_B(ｓ|[θ_i])と書き代えると、次式（数１４）となる。
【００５３】
【数１４】

次に、クラス内分布ｐ_W(ｓ|[θ_i])、クラス間分布ｐ_B(ｓ|[θ_i])がそれぞれ正規分布であると仮定し、その平均がそれぞれ０、共分散行列がそれぞれΣ_W([θ_i])、Σ_B([θ_i])とすれば、事後分布はそれぞれ次式（数１５）で書き表すことができる。
【００５４】
【数１５】

上式を（数１４）に代入すると(但し、（数１４）の第２項は省略)、次式（数１６）で示す距離を得ることができる。これを「適応的判別距離」と呼ぶことにする。
【００５５】
【数１６】

差ベクトルｓの各要素間における独立性を仮定すれば、（数１５）は、次式となる。
【００５６】
【数１７】

ここで、σ_W,k(θ_i)²、σ_B,k(θ_i)²は、それぞれ共分散行列Σ_W(θ_i)、Σ_B(θ_i)のｋ番目の対角要素であり、つまり、クラス内分散とクラス間分散に相当する。ｓ_kは差ベクトルｓのｋ番目の要素である。
【００５７】
このように正規分布を仮定することで（数１６）は、次式（数１８）のように、各特徴ベクトルの要素毎のクラス内分散σ_W,k(θ_i)²、クラス間分散σ_B,k(θ_i)²を用いて、信頼性指標[θ_i]に対して適応的な判別距離による類似度を定義することができる。
【００５８】
【数１８】

上記までの説明ではクラス内分散σ_W,k(θ_i)²、クラス間分散σ_B,k(θ_i)²として 正規分布を仮定したが、以下では混合分布を仮定する。
【００５９】
次式（数１９）のように、クラス内分布ｐ_W(ｓ|[θ_i])、クラス間分布ｐ_B(ｓ|[θ_i])として、それぞれ正規分布ｐ_W(ｓ|[θ_i], j_W) (j_W=1,2,...,M_W)、ｐ_B(ｓ|[θ_i], j_B) (j_B=1,2,...,M_B)の和によって事後分布が表現できると仮定する。
【００６０】
【数１９】

従って、この対数尤度を用いて次式（数２０）の適応的混合マハラノビス距離を導くことができる。
【００６１】
【数２０】

クラス内分布ｐ_W(ｓ|[θ_i], j_W)、クラス間分布ｐ_B(ｓ|[θ_i], j_B)の共分散行列Σ_W(s|[θ_i], j_W)、Σ_B(s|[θ_i], j_B)および P(j_W)、P(j_B)の推定については、最尤推定法やＥＭアルゴリズムを用いて推定することができる。
【００６２】
混合分布を仮定することで、分布をより正確に近似でき、マッチング性能を 向上させることができるが、同時に大量の学習データが必要になるとともに、演算量も大幅に増加する。
【００６３】
このように顔特徴量に対してさらに信頼性指標を抽出することによって、信頼性指標に対して適応的な距離規範を導くことができ、高精度な顔認識機構を構築することができる。なお、上記では、特徴ベクトルに対する信頼性指標をスカラー量(一つの成分のみ)か、ベクトル量(複数の成分をもつ)か特定していないが、どちらの場合でも成立する議論であり、複数の要素を用いることで、性能の向上が期待できる。
【００６４】
具体的な信頼性指標については、実験的に有効な信頼性指標を発見することが必要であり、顔認識の場合では、画像のコントラストを表すコントラスト指標や、正面顔の認識では、照明変動や姿勢変動によって生じる顔画像の左右の対称性の歪みからの量を表す非対称指標を用いると高い効果が得られ、それらの信頼性指標を組み合わせてベクトル量とすることで、より精度の向上が期待できる。
【００６５】
（実施の形態）
図１は、本発明の一実施形態による顔画像マッチングシステムを示すブロック図である。以下、顔画像マッチングシステムについて詳細に説明する。
【００６６】
図１に示すように、本発明による顔画像マッチングシステムには、顔画像を入力する顔画像入力部１１と、入力された顔画像からそれらの顔特徴量、信頼性指標の顔メタデータを生成する顔メタデータ生成部１２と、抽出された顔メタデータを蓄積する顔メタデータ蓄積部１３と、顔メタデータから顔の類似度を算出する顔類似度算出部１４と、顔画像を蓄積する顔画像データベース１５と、画像の登録要求・検索要求に応じて、画像の入力・メタデータの生成・メタデータの蓄積・顔類似度の算出の制御を行う制御部１６と、顔画像や他の情報を表示するディスプレイの表示部１７と、が設けられている。
【００６７】
また、顔メタデータ生成部１２は、入力された顔画像から顔特徴を抽出する顔特徴抽出部１２１と、信頼性指標を抽出する信頼性指標抽出部１２２とから構成され、顔類似度算出部１４は、信頼性指標から事後分布に関するパラメータ情報を推定する分布推定部１４１と顔特徴量と分布推定部１４１からの事後分布情報から顔特徴量間の距離を算出する距離算出部１４２によって構成される。
【００６８】
登録時には、画像入力部１１では、スキャナあるいはビデオカメラなどで顔写真等を顔の大きさや位置を合わせた上で入力する。あるいは、人物の顔を直接ビデオカメラなどから入力しても構わない。この場合には、前述のMoghaddamの文献に示されているような顔検出技術を用いて、入力された画像の顔位置を検出し、顔画像の大きさ等を自動的に正規化する方がよいであろう。
【００６９】
また、入力された顔画像は必要に応じて顔画像データベース１５に登録する。顔画像登録と同時に、顔メタデータ生成部１２によって顔メタデータを生成し、顔メタデータ蓄積部１３に蓄積する。
【００７０】
検索時には同様に顔画像入力部１１によって顔画像を入力し、顔メタデータ生成部１２にて顔メタデータを生成する。生成された顔メタデータは、一旦 顔メタデータ蓄積部１３に登録するか、または、直接に顔類似度算出部１４へ送られる。検索では、予め入力された顔画像がデータベース中にあるかどうかを確認する場合(顔同定)には、顔メタデータ蓄積部１３に登録されたデータの一つ一つとの類似度を算出する。最も類似度が高い(距離値が小さい)結果に基づいて制御部１６では、顔画像データベース１５から、顔画像を選び、表示部１７等に顔画像の表示を行い、検索画像と登録画像における顔の同一性を作業者が確認する。
【００７１】
一方、予めＩＤ番号等で特定された顔画像と検索の顔画像が一致するかどうかを確認する場合(顔識別)では、特定されたＩＤ番号の顔画像と一致するか否かを顔類似度算出部１４にて計算し、予め決められた類似度よりも類似度が低い(距離値が大きい)場合には一致しないと判定し、類似度が高い場合には一致すると判定し、その結果を表示部１７に表示する。このシステムを入室管理用に用いるならば、表示する代わりに、制御部１６から自動ドアに対して、その開閉制御信号を送ることで、自動ドアの制御によって入室管理を行うことができる。
【００７２】
上記のように、顔画像マッチングシステムは動作するが、このような動作はコンピュータシステム上で実現することもできる。たとえば、次に詳述するようなメタデータ生成を実行するメタデータ生成プログラムおよび類似度算出を実行する類似度算出プログラムをそれぞれメモリに格納しておき、これらをプログラム制御プロセッサによってそれぞれ実行することで顔画像マッチングを実現することができる。
【００７３】
次に、この顔画像マッチングシステムの動作、特に顔メタデータ生成部１２と顔類似度算出部１４について詳細に説明する。
【００７４】
（１）顔メタデータ生成
顔メタデータ生成部１２では、位置と大きさを正規化した画像I(x, y)を用いて、顔特徴量を抽出する。位置と大きさの正規化は、例えば、目位置が(16, 24)、(31, 24)、サイズが46×56画素となるように画像を正規化しておくとよい。以下では、このサイズに画像が正規化されている場合について説明する。
【００７５】
顔特徴量として、所謂、固有顔の手法(前述Moghaddamの論文)を用いて特徴抽出する。つまり、画像中の画素値を要素とする特徴ベクトルΛに対して、顔画像サンプル集合[Λ]の主成分分析によって得られた基底ベクトルの中から選択した部分基底ベクトルによって特定される基底行列Ｕと、顔画像サンプル集合[Λ]における平均ベクトルである平均顔Ψとを用いて、特徴ベクトルｖ＝Ｕ^T(Λ−Ψ)を算出し特徴ベクトルとする。このように、入力画像ではなく、部分基底ベクトルを用いることで、入力画像のデータ量を削減することができる。このようにデータ量を削減することは、メタデータベースにおける蓄積量を少なくするだけではなく、高速なマッチングを実現するためには重要な要素である。この特徴ベクトルの次元数としては、例えば、４８次元の特徴量を用いればよい。
【００７６】
このように一般的に用いられる固有顔による手法の他に、主成分分析に対して判別分析を組み合わせた手法(前述、W. Zhaoの論文)等を用いて部分基底ベクトルを規定しても構わない。
【００７７】
また、前述したような顔画像サンプルにおける特徴ベクトル集合[Λ]の主成分分析や判別分析によって得られる基底ベクトルの中から選択した部分基底ベクトルによって特定される基底行列Ｕnと、顔画像を左右反転される画素値の変換と対応するように基底ベクトルの要素を入 れ換えた反転部分基底ベクトルによって特定される基底行列Ｕmの線形和によって得られる行列Ｕ(＝ aＵn + bＵm)を基底行列として、顔特徴ベクトルを抽出してもよい。例えばa=b=1とすると、得られる顔特徴ベクトルは、入力画像空間で左右の変換に対して対称な成分のみを抽出 することができる。顔は本来左右対称であるので、照明の影響で非対称となった画像成分や顔の向きが正面に向いていないために生じる非対称成分が本来ノイズに相当するものであるので、これを除去し、対称な成分のみを抽出することで、照明や姿勢の変動に対して安定な顔特徴量を抽出することが可能となる。
【００７８】
また、画像をフーリエ変換し、得られた複素フーリエ成分の各成分の大きさを要素とするベクトルを特徴ベクトルΛとして算出し、上記のような主成分分析や判別分析によって次元圧縮を行い顔特徴量を抽出してもよい。このように画像をフーリエ変換することで、位置ずれに対して強い顔特徴量を抽出することができる。このようにして、顔特徴抽出部１２１では、顔特徴量ｖを抽出する。
【００７９】
信頼性指標抽出部１２２では、顔特徴量ｖの信頼性指標として有効なコントラスト指標θ_contrastと非対称性指標θ_asymmetricを抽出する。コントラスト指標θ_contrastとして、顔画像 I(x,y)の画素値の標準偏差を次式（数２１）によって計算する。
【００８０】
【数２１】

ここでround()は数値の丸め処理を意味する。このように抽出された信頼性指標θ_contrastを[0,1,2,...,15]の4 bitの範囲に収まるように、値域を越えるものは値を制限する。なお、上記ではコントラスト指標として、画像の標準偏差を計算したが、分散や画像中の画素値の最大値と最小値の差を抽出してもよい。画像中の画素値の最大値と最小値による コントラスト指標は、標準偏差や分散による場合よりも演算量が少ないが、効果は相対的に小さい。
【００８１】
非対称性指標θ_asymmetricとして、顔画像 I(x,y)とその反転画像との間の差分の絶対値(１乗)の平均を次式（数２２）により抽出する。
【００８２】
【数２２】

このように抽出された信頼性指標θ_asymmetricを[0,1,2,...,15]の4 bitの範囲に収まるように値域を越えるものは値を制限する。なお、上記では非対称指標として、絶対値による値(１乗)を用いたが差分の２乗等を用いてもよい。また、平均の代わりに、和等の値を用いても同等の効果を得ることができる。また、差分の最大値を検出して、その値を非対称指標として用いれば演算量が少なくて済む。
【００８３】
信頼性指標抽出部１２２では、顔画像に対して、特徴ベクトルｖと信頼性指標θ_contrastと信頼性指標θ_asymmetricを抽出し、顔メタデータとして出力する。前述したように、上記顔メタデータ生成手順をコンピュータプログラムによってコンピュータに実行させることもできる。
【００８４】
（２）顔類似度算出
次に、顔類似度算出部１４の動作について説明する。顔類似度算出部１４では、二つの顔メタデータの信頼性指標θ_contrast,1、θ_constrast,2とθ_{asymmetri c,1}、θ_asymmetric,2を用いて、分布推定部１４１が事後分布に関するパラメータ情報を推定し、二つの顔メタデータの二つの特徴ベクトルｖ₁、ｖ ₂と事後分布に関するパラメータ情報とを用いて距離算出部１４２が顔特徴間の類似度ｄを算出する。
【００８５】
ここでは、（数８）あるいは（数１８）によって顔の類似度を計算する場合について説明する。
【００８６】
（数８）あるいは（数１８）にて表記されている信頼性指標θ₁、θ₂は本実施形態ではベクトルであり、それぞれの要素はθ₁ = (θ_contrast,1, θ_asymmetric,1)^T、θ₂ = (θ_contrast,2, θ_asymmetric,2)^Tとなる。コントラスト指標、非対称性指標はそれぞれ4bitで表現されているので、θ_iのとり得る状態は256個の状態となる。与えられた信頼性指標θ_iによって、２５６個の状態の中の一つの状態が特定できる。
【００８７】
分布推定部１４１では、後述するように、予め求めておいた信頼性指標θ(256個の状態がある)に対する差ベクトルの分散値σε_{, k}(θ)や分散値σ_W,k(θ)、分散値σ_B,k (θ)をテーブルに記憶しておき、信頼性指標θ₁およびθ₂を用いて、それぞれの分散値テーブルの値を参照し、得られた分散値を事後分布情報として、距離算出部１４２に引き渡す。なお、顔同定の場合には、（数８）で必要な分散値σε_,k(θ₁)、分散値σε_,k (θ₂)の値を距離算出部１４２に出力すればよく、顔識別の場合には、（数１８）に必要な分散値σ_W,k(θ)、分散値σ_B,k(θ)を出力する。
【００８８】
距離算出部１４２では、（数８）あるいは（数１８）に従って適応的マハラノビス距離あるいは適応的判別距離を算出し、類似度ｄとして出力する。
【００８９】
前述した分散値テーブルの分散値は、予め用意した顔画像データサンプルを用いて計算しておく。顔画像サンプルにおける特徴ベクトル集合[ｖ_i]とその信頼性指標[θ_i]から、それぞれの分散値は次のように計算できる。
【００９０】
【数２３】

【００９１】
【数２４】

ここで、"(i, j)がクラスＷに属する"とは、データiとデータjが同一クラス(同一人物)から得られたデータである(クラス内である)ことを意味し、"(i, j)がクラスＢに属する"とは、データiとデータjが異なるクラス(異なる人物)から得られたデータである(クラス間である)ことを意味する。また、Ｎε(θ)、Ｎ_W(θ)、Ｎ_B(θ)はそれぞれのクラスに属するデータの組み合わせの数である。このようにθのビン毎に分散値を計算する際に、そのデータ数が少な過ぎる場合には、近傍のビンのデータを併合するようにして、サンプル数を確保するようにする(これは、分布推定におけるk近傍法(前述Bishop の文献,pp.53)によってbinを併合するのと同様な手法である)。
【００９２】
ここで、（数１８）の場合、（数４）と同様に、σ_W,k([θ_i])² = σ_W,k(θ₁)² + σ_W,k(θ₂)²、および、σ_B,k([θ_i])² = σ_B,k(θ₁)² +σ_B,k(θ₂)²となることに注意されたい。
【００９３】
なお、誤差分散σε_,k(θ)²とクラス内分散σ_W,k(θ)²が同じになるので、顔画像マッチングシステムで（数８）と（数１８）の両方の距離を計算する場合には、これらの分散値テーブルを共有して構わない。
【００９４】
また、誤差分布とクラス間分布は強い相関がある場合が多いので、誤差分 散σε_,k(θ)²の代わりにクラス間分散σ_B,k(θ)²を用いても、信頼性指標を用いない場合よりは精度が向上する(しかし、誤差分散を用いた方が精度が良い)。
【００９５】
このように顔メタデータ間の類似度を信頼性指標θ_contrastや信頼性指標θ_asymmetricを介した事後分布情報を用いて算出することで、精度のよい顔認識を行うことが可能である。前述したように、上記顔類似度算出手順をコンピュータプログラムによってコンピュータに実行させることもできる。
【００９６】
なお、ここでは（数８）と（数１８）を用いて、類似度を計算しているが、次のような様々な計算方法によって近似的に計算し、高速化等を図ることもできる。
【００９７】
【数２５】

【００９８】
【数２６】

上記各式の右辺第二項(lnの部分)を計算しないことで、さらなる高速演算を図ることができる。
【００９９】
また、（数６）や（数１６）によって類似度を計算する場合も、基本的には同様に予め用意した顔画像データサンプルから、それぞれの計算に必要な差ベクトルの誤差の共分散行列Σε(θ)や差ベクトルのクラス内の共分散行列Σ_W(θ)、クラス間の共分散行列Σ_B(θ)を算出して、共分散テーブルとして用意しておき、類似度計算の際にその共分散テーブルを参照するようにすればよい。この方法は共分散行列を用いて距離を計算するために演算量が増加するが、十分な学習サンプルがある場合には類似度計算の精度を向上させることができる。
【０１００】
（数３）の事後分布や（数１４）のクラス間分布、クラス間分布に対して混合正規分布を仮定して、分布関数を推定することで、それぞれ（数１０）や（数２０）の適応的混合マハラノビス距離や適応的混合判別距離を計算してもよい。この場合も、分散や共分散行列を用いて、事後分布情報を計算するのと同様に顔画像データサンプルから、混合正規分布を表す共分散行列Σ_s(θ₁, j)、P(j)等の混合分布を特定するパラメータを求めておき、テーブルとして記憶しておけばよい。なお、この推定については、一般的な推定方法である最尤推定法やＥＭアルゴリズムを用いて推定すればよい。
【０１０１】
ここまでの説明では、一枚の顔画像が登録され、一枚の顔画像を用いて検索する場合について説明したが、一人の顔に対して複数の画像が登録され、一枚の顔画像を用いて検索する場合には、例えば、次のようにすればよい。
【０１０２】
検索側の特徴ベクトルをｖ_queとし、登録側の特徴ベクトルをｖ_reg,kとし、複数画像登録の場合の類似度d_multi(ｖ_que, [ｖ_reg,1, ｖ_reg,2, ... ,ｖ_reg,n])として、次式（数２７、数２８）に示す計算式に基づき類似度を計算すればよい。
【０１０３】
【数２７】

あるいは、
【０１０４】
【数２８】

同様に１つの顔当たりの複数枚の画像登録と複数画像による検索の場合も、各組み合わせの類似度の平均や最小値を求め類似度を算出することで、一つの顔データに対する類似度を算出することができる。これは、動画像を複数画像と見倣すことで、本発明のマッチングシステムを動画像における顔認識に対しても適用できることを意味する。
【０１０５】
また、上記説明では、同一人物の顔の同定や顔の識別を中心に説明したが、例えば、男性の顔を一つのカテゴリー、女性の顔を一つのカテゴリーと考えて、それぞれの分布に関する情報を求め、男女の顔を識別する性別識別を行ったり、笑った顔、怒った顔、悲しい顔などの表情のカテゴリーを考えて、顔の表情を認識する表情認識に適用することも可能である。１０代、２０代、３０代、４０代というようにカテゴリーを設定し、そのカテゴリー毎に誤差分布、クラス内分布やクラス間分布を求めることで、年齢判別を行うことも可能であり、様々な顔認識に対して本発明を適用することが可能である。
【０１０６】
【発明の効果】
以上詳細に説明したように、本発明によれば、顔特徴量に対してその信頼性を表現するための信頼性指標を加えて顔メタデータとして生成することで、信頼性指標を通じた適応的な距離を導くことができ、その距離を用いて精度の高い顔認識を行うことが可能となる。
【図面の簡単な説明】
【図１】 本発明の一実施形態による顔画像マッチングシステムの構成を示すブロック図である。
【符号の説明】
11: 顔画像入力部
12: 顔メタデータ生成部
13: 顔メタデータ蓄積部
14: 顔類似度算出部
15: 顔画像データベース
16: 制御部
17: 表示部
121: 顔特徴抽出部
122: 信頼性指標抽出部
141: 分布推定部
142: 距離算出部[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a technique that can be used for face identification, face identification, facial expression recognition, gender identification by face, age discrimination by face, and the like, and in particular metadata related to face information displayed in still images and moving images. The present invention relates to metadata generation, face similarity calculation, and metadata matching technology.
[0002]
[Prior art]
Metadata generally refers to data that describes or represents the meaning of data. In the case of face recognition, it mainly refers to data related to face data such as still face images and moving images.
[0003]
  As a standardization activity of metadata for multimedia contents such as video, images and audio, MPEG-7 (MPEG (Moving Pictures Experts Group) International standard for multimedia content description interface standardized by ISO / IEC 15938) Is widely known. Among them, a face recognition descriptor has been proposed as a metadata descriptor for face recognition (A. Yamada et al., "MPEG-7 Visual part of eXperimental Model Version 9.0," ISO / IEC JTC1 / SC29 / WG11). N3914, 2001).
[0004]
In this face recognition descriptor, a base matrix for extracting feature values of a face image is obtained from a cut and normalized face image using a kind of subspace method generally called an eigenface. A face feature amount is extracted from the image using a base matrix, and this is used as metadata. It has also been proposed to use a weighted absolute value distance as the similarity to the face feature amount.
[0005]
In addition, it is known that there are various methods for face recognition technology.
Principal component analysis (Moghaddam et al., "Probalilistic Visual Learning for Object Detection", IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 17, No. 7, pp. 696-710, 1997), or
Discriminant analysis (W. Zhao et al., "Discriminant Analysis of Principal Components for Face Recognition," Proceedings of the IEEE Third International Conference on Automatic Face and Gesture Recognition, pp. 336-341, 1998)
There are known methods based on eigenfaces based on.
[0006]
In addition, when applying the subspace method to features obtained from fingerprint images, there is a method that introduces a quality index and adaptively measures the distance between patterns (T. Kamei and M. Mizoguchi, “Fingerprint Preselection Using Eigenfeatures, ”Proceedings of the 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp.918-923, 1998, JP-A-10-177650).
[0007]
[Problems to be solved by the invention]
However, the conventional technology described above cannot obtain sufficient face recognition accuracy. Accordingly, an object of the present invention is to provide a face metadata generation technique and a face similarity calculation technique capable of improving the accuracy of face recognition, and further provide a technique for constructing a practical face matching system. There is to do.
[0008]
[Means for Solving the Problems]
According to the present invention, the accuracy of face recognition can be improved by extracting the reliability from the face image and adaptively calculating the similarity between the patterns according to the reliability.
[0009]
  According to the present invention, face feature extraction means for extracting a face feature amount from a face image, and reliability index extraction means for extracting a reliability index representing the reliability of the face recognition result using the face feature amount, The face feature amount and the reliability index are output as face metadata, and the face feature extraction means is specified by a partial basis vector of the feature vector Λ when the feature vector obtained from the image is Λ. Feature vector v = U using the basis matrix U and the mean vector Ψ of feature vectors Λ^T(Λ−Ψ) is calculated and the feature vector v is extracted as a facial feature quantityThe reliability index extraction unit extracts a statistic obtained by sampling the pixel value in the face image as a reliability index representing the reliability of the face recognition accuracy of the face feature amount.It is characterized by that.
[0010]
The reliability index extraction unit can extract a contrast index of the image as the reliability index. As a contrast index of an image, a difference between a maximum value and a minimum value of pixel values in the image is extracted. Alternatively, a variance or standard deviation of pixel values in the image is extracted as a contrast index of the image.
[0011]
Furthermore, the asymmetry index of the image can be extracted as a reliability index. As an image asymmetry index, a maximum value of a difference between the image and an image obtained by horizontally inverting the image (hereinafter referred to as an inverted image) is extracted. Alternatively, as an asymmetry index of the image, either the sum or average of the powers of the pixel values of the difference image between the image and the inverted image is extracted.
[0012]
The reliability index extraction unit includes a first reliability index extraction unit that extracts a contrast index of the image and a second reliability index extraction unit that extracts an asymmetry index of the image. The first reliability index extracting unit extracts a difference between the maximum value and the minimum value of pixel values in the image as a contrast index of the image, and the second reliability index extracting unit is used as an asymmetry index of the image. The maximum value of the difference between the image and the reverse image is extracted. Alternatively, the first reliability index extracting unit extracts one of a variance and a standard deviation of pixel values in the image as a contrast index of the image, and the second reliability index extracting unit is used as an asymmetry index of the image. One of the sum and average of the powers of the pixel values of the difference image between the image and the reverse image may be extracted.
[0013]
SaidThe basis matrix U is characterized by being specified by a partial basis vector selected from basis vectors obtained by principal component analysis of the face image sample set [Λ]. The basis matrix U is a basis matrix specified by a partial basis vector selected from basis vectors obtained by principal component analysis of a feature vector set [Λ] in a face image sample, and conversion of pixel values for horizontally flipping the face image. Can be obtained by a linear sum of a basis matrix specified by an inverted partial basis vector in which elements of the basis vector are replaced so as to correspond to The feature vector Λ may be a vector having each pixel value in the image as an element, or may be a vector having the magnitude of each component of a complex Fourier component obtained by performing Fourier transform on the image as an element. .
[0014]
imageFor calculating similarity between face metadata of face informationIsThe similarity is calculated based on a facial feature amount extracted from the image and a reliability index representing reliability with respect to the facial feature amount.To do.
[0015]
The face similarity calculation device is derived from a distribution estimation means for estimating parameter information relating to a posterior distribution of a difference vector between face feature amounts using a reliability index for the face feature amount to be compared, and a log likelihood of the posterior distribution. Distance calculating means for calculating the distance between the facial features to be calculated as a similarity.
[0016]
The distance calculation means calculates an adaptive Mahalanobis distance as a similarity degree derived from a logarithmic likelihood of a normal distribution in the reliability index, assuming that a posterior distribution of a difference vector between facial feature quantities is a normal distribution. And
[0017]
The distribution estimation means uses a reliability index [θ_iThe variance σ of each element k of the difference vector s_{s, k}([θ_i])²And the distance calculation means calculates the variance σ of each element k_{s, k}([θ_i])²And calculating the adaptive Mahalanobis distance using.
[0018]
The distribution estimation means uses a reliability index [θ_iThe variance σ of each element k of the difference vector s_{s, k}([θ_i])²Is stored in advance, and a reliability index [θ_i] Is used to output a variance value necessary for calculating the adaptive Mahalanobis distance by referring to the variance value table.
[0019]
The distance calculation means calculates the adaptive mixed Mahalanobis distance derived from the logarithmic likelihood of the mixed distribution in the reliability index as the similarity, assuming that the posterior distribution of the difference vector between the facial feature quantities is a mixed distribution. It is characterized by.
[0020]
A posterior distribution (intraclass distribution) of difference vectors between face features in the class to be matched and imitated using the reliability index for the face feature to be compared, and face features between the classes to be imitated and inconsistent Estimate parameter information related to the posterior distribution (interclass distribution) of the difference vector between quantities, and calculate the distance between facial feature quantities derived from the log likelihood of the ratio between the intraclass distribution and the interclass distribution as the similarity It is characterized by.
[0021]
Assuming that the intra-class distribution and the inter-class distribution are normal distributions, an adaptive discriminant distance derived from the log likelihood of the ratio of the respective distributions in the reliability index is calculated as the similarity.
[0022]
Reliability index [θ_i] Intraclass variance σ of each element k of difference vector s_{W, k}([θ_i])²And interclass variance σ_{B, k}([θ_i])² The adaptive discriminant distance is calculated as the similarity degree by estimating.
[0023]
Reliability index [θ_i] Intraclass variance σ of each element k of difference vector s_{W, k}([θ_i])²And interclass variance σ_{B, k}([θ_i])²For the estimation of the intraclass variance σ_{W, k}([θ_i])²And a first variance value table for storing in advance, and an interclass variance σ_{B, k}([θ_i])²And a second variance value table that stores in advance a reliability index [θ_i], By referring to the first and second variance value tables, respectively, the variance value required in the adaptive discrimination distance is estimated.
[0024]
Assuming that the intraclass distribution and the interclass distribution are each a mixture distribution, an adaptive mixture discrimination distance derived from a logarithmic likelihood of a ratio of each mixture distribution in the reliability index is calculated as a similarity. To do.
[0025]
imageRecognition system having an image input unit for inputting a face and a face image database unit for storing face imagesIsA face metadata generation unit that generates a face feature amount extracted from the image and a reliability index face that represents reliability of the face feature amount as face metadata; and a face generated by the face metadata generation unit A face metadata accumulation unit that accumulates metadata, a face similarity calculation unit that calculates a face similarity based on a face feature amount and a reliability index that represents the reliability of the face feature amount, and an input command And a control unit that controls the face metadata generation unit, the face metadata storage unit, and the face similarity calculation unit, and executes matching of face images.Prepare.
[0026]
By using the face metadata generation unit and the face similarity calculation unit described above, highly accurate face image matching can be achieved.
DETAILED DESCRIPTION OF THE INVENTION
(Principle of the invention)
First, the principle of the present invention will be described. In general, when pattern recognition is performed, if it is possible to prepare a large amount of learning data for a class to be recognized, a pattern distribution function is estimated based on statistical analysis of the learning data. Can be built. However, in the face recognition application, only one registered image can be obtained for each individual, and only a few registered images are often allowed.
[0027]
Even in such a case, an index called a reliability index is given to the facial feature vector, and by considering a class based on that index, statistical analysis of the class for that reliability index is performed, and the pattern distribution function is calculated. By estimating, it is possible to construct a pattern recognition mechanism based on a distribution function through a reliability index even for face recognition applications in which only one registered image can be obtained.
[0028]
Hereinafter, the principle of face recognition regarding the Mahalanobis distance based on the error distribution and the discrimination distance based on the intraclass distribution and the interclass distribution will be described.
[0029]
Consider the case of observing one person's face. The feature vector obtained from the face image when there is no error is v₀It is assumed that the error vector ε is superimposed on the actually observed vector v (Equation 1).
[0030]
[Expression 1]

Where the feature vector v₀If two observations are taken, two observation vectors v₁And v₂Is obtained (Equation 2).
[0031]
[Expression 2]

Now, the error vector ε₁, Error vector ε₂Reliability index θ correlated with₁, Reliability index θ₂Are obtained, and the posterior distribution is p (ε | θ₁), P (ε | θ₂).
[0032]
V when such a distribution is obtained₁And v₂The posterior distribution of the difference vector s is p (s | θ₁, θ₂), The similarity d (v between patterns)₁, v₂) Can be the following log likelihood.
[0033]
[Equation 3]

Posterior distribution p (ε | θ₁), P (ε | θ₂) Are normal distributions, the distribution p (s | θ of the difference vector s₁, θ₂) Also has a normal distribution. Here, the posterior distribution p (ε | θ of the error vector_i) Covariance matrix Σε (θ with mean 0 as (i = 1,2)_i), The distribution of the difference vector s is 0 on average and the covariance matrix Σ_s(θ₁, θ₂) Is given by the following equation (Equation 4).
[0034]
[Expression 4]

That is, the posterior distribution p (s | θ₁, θ₂) Is expressed by the following equation (Equation 5).
[0035]
[Equation 5]

Therefore, (Equation 3) is expressed by the covariance matrix Σε (θ₁) Or Σε (θ₂), The reliability index θ₁, And θ₂Can be expressed as an adaptive Mahalanobis distance.
[0036]
[Formula 6]

Assuming the independence among the elements of the error vector, (Equation 5) becomes
[0037]
[Expression 7]

Where σ_{s, k}(θ₁, θ₂)²Is the covariance matrix Σ_s(θ₁, θ₂) K-th diagonal element, that is, the variance of the observation error. Also, σε_{, k}(θ₁)², Σε_{, k}(θ₂)²Is the covariance matrix Σε (θ₁), Σε (θ₂) K-th diagonal element. s_kIs the kth element of the difference vector s.
[0038]
Assuming a normal distribution in this way, (Equation 3) can be expressed by the variance σε for each element of each feature vector as shown in the following equation (Equation 8)._{, k}(θ₁), Σε_{, k}(θ₂), The reliability index θ₁And θ₂The similarity can be defined by an adaptive Mahalanobis distance.
[0039]
[Equation 8]

Where v_{1, k}, V_{2, k}Is the feature vector v₁, V₂K-th element.
[0040]
In the above description, the posterior distribution p (s | θ₁, θ₂), A normal distribution is assumed. Hereinafter, a mixed normal distribution is assumed. Now, the posterior distribution p (s | θ₁, θ₂), As shown by the following equation (Equation 9), the normal distribution p (s | θ₁, θ₂, j) The posterior distribution p (s | θ by the sum of (j = 1,2, ..., M)₁, θ₂) Can be expressed.
[0041]
[Equation 9]

Therefore, the adaptive mixed Mahalanobis distance can be defined by the following equation (Equation 10).
[0042]
[Expression 10]

Posterior distribution p (s | θ₁, θ₂, j) covariance matrix Σ_s(θ₁, θ₂, j) and P (j) can be estimated using general estimation methods such as maximum likelihood estimation and EM algorithm (CM Bishop, "Neural Networks for Pattern Recognition", Oxford University Express , 1995).
[0043]
By assuming a mixed normal distribution, the distribution can be approximated more accurately and the matching performance is improved. However, a large amount of learning data is required and the amount of calculation is also greatly increased.
[0044]
The Mahalanobis distance based on the above error distribution is a distance that is excellent for problems such as face identification to find out which face to search is closest to among other registered face data. is there.
[0045]
On the other hand, in the face verification problem, whether to accept or reject is important in determining the identity of the input face with the registered image. The distance named “discrimination distance” described below is a better similarity measure than the above-described Mahalanobis distance for the face identification problem.
[0046]
Now, it should be determined that the two feature vectors v of the face match, that is, if the two feature vectors belong to the same class (for example, the two feature vectors are the face data of the same person), Assume that the combination of these feature vectors belongs to class W. Also, if the two feature vectors v should be determined to be inconsistent, that is, feature vectors between classes (for example, the two feature vectors are different face data of a person), the combination thereof is class B. Suppose it belongs.
[0047]
Two feature vectors v₁, V₂Reliability index θ₁, Θ₂Is obtained. Difference vector s and two reliability indices θ₁, Θ₂(Hereafter, a set of two reliability indices is represented by [θ_i]. ) Is observed, the discriminant rule between the class W that is considered to be coincident with the class B that is to be imitated and the class B that is to be imitated is obtained as follows: .
[0048]
## EQU11 ##

The left side of the above (Equation 11) can be rewritten as the following equation (Equation 12) by the Bayes Theorem.
[0049]
[Expression 12]

Where W, B and [θ_i] Is assumed to be independent and P (W, [θ_i]) = P (W) P ([θ_i]), P (B, [θi]) = P (B) P ([θ_i]).
[0050]
Distance d (v between patterns₁, v₂) To calculate the log likelihood of (Equation 12), the similarity suitable for the face identification problem can be obtained as in the following equation (Equation 13).
[0051]
[Formula 13]

If the prior probabilities P (W) and P (B) are different for each individual matching and can be known, it is desirable to calculate the second term of (Equation 13). However, in many cases, since the prior probability cannot be known for each individual matching, assuming that the prior probability is constant, the second term is assumed to be constant and excluded from the calculation of the similarity.
[0052]
Posterior probability P (s | W, [θ_i]), P (s | B, [θ_i]) In each class distribution p_W(s | [θ_i]), Interclass distribution p_B(s | [θ_i]), The following expression (Expression 14) is obtained.
[0053]
[Expression 14]

Next, intraclass distribution p_W(s | [θ_i]), Interclass distribution p_B(s | [θ_i]) Are normally distributed, the average is 0, and the covariance matrix is Σ_W([θ_i]), Σ_B([θ_i]), The posterior distribution can be expressed by the following equation (Equation 15).
[0054]
[Expression 15]

Substituting the above expression into (Expression 14) (however, the second term of (Expression 14) is omitted), the distance expressed by the following expression (Expression 16) can be obtained. This is called “adaptive discrimination distance”.
[0055]
[Expression 16]

Assuming the independence between the elements of the difference vector s, (Equation 15) becomes the following equation.
[0056]
[Expression 17]

Where σ_{W, k}(θ_i)², Σ_{B, k}(θ_i)²Are the covariance matrices Σ_W(θ_i), Σ_B(θ_i) K-th diagonal element, that is, corresponding to intra-class variance and inter-class variance. s_kIs the kth element of the difference vector s.
[0057]
Assuming a normal distribution in this way, (Equation 16) can be obtained by calculating the intra-class variance σ for each element of each feature vector as shown in the following equation (Equation 18)._{W, k}(θ_i)², Interclass variance σ_{B, k}(θ_i)²Using the reliability index [θ_i], It is possible to define a similarity based on an adaptive discrimination distance.
[0058]
[Expression 18]

In the above explanation, intra-class variance σ_{W, k}(θ_i)², Interclass variance σ_{B, k}(θ_i)²A normal distribution is assumed as, but a mixed distribution is assumed below.
[0059]
Intraclass distribution p as in the following equation (Equation 19)_W(s | [θ_i]), Interclass distribution p_B(s | [θ_i]) As normal distribution p_W(s | [θ_i], j_W) (j_W= 1,2, ..., M_W), P_B(s | [θ_i], j_B) (j_B= 1,2, ..., M_B) Is assumed to be able to express the posterior distribution.
[0060]
[Equation 19]

Therefore, the adaptive mixed Mahalanobis distance of the following equation (Equation 20) can be derived using this log likelihood.
[0061]
[Expression 20]

Intraclass distribution p_W(s | [θ_i], j_W), Interclass distribution p_B(s | [θ_i], j_B) Covariance matrix Σ_W(s | [θ_i], j_W), Σ_B(s | [θ_i], j_B) And P (j_W), P (j_B) Can be estimated using a maximum likelihood estimation method or an EM algorithm.
[0062]
By assuming a mixed distribution, the distribution can be approximated more accurately and the matching performance can be improved, but at the same time, a large amount of learning data is required and the amount of computation is greatly increased.
[0063]
Thus, by extracting a reliability index further from the face feature amount, an adaptive distance criterion can be derived for the reliability index, and a highly accurate face recognition mechanism can be constructed. In the above, the reliability index for the feature vector is not specified as a scalar quantity (only one component) or a vector quantity (having a plurality of components), but it is an argument that holds in either case. Use of elements can be expected to improve performance.
[0064]
As for the specific reliability index, it is necessary to find an experimentally effective reliability index. In the case of face recognition, the contrast index indicating the contrast of the image, and the front face recognition, Using an asymmetric index that represents the amount of distortion in the face image caused by posture changes from left and right symmetry distortion is highly effective, and combining these reliability indices into a vector quantity is expected to improve accuracy. it can.
[0065]
(Embodiment)
FIG. 1 is a block diagram illustrating a face image matching system according to an embodiment of the present invention. Hereinafter, the face image matching system will be described in detail.
[0066]
As shown in FIG. 1, in the face image matching system according to the present invention, a face image input unit 11 for inputting a face image, and generating face metadata of the face feature amount and reliability index from the input face image. A face metadata generation unit 12, a face metadata storage unit 13 that stores the extracted face metadata, a face similarity calculation unit 14 that calculates a face similarity from the face metadata, and a face image. A face image database 15, a control unit 16 that controls input of images, generation of metadata, accumulation of metadata, and calculation of face similarity in response to an image registration request / search request; And a display unit 17 for displaying information.
[0067]
The face metadata generation unit 12 includes a face feature extraction unit 121 that extracts a face feature from an input face image and a reliability index extraction unit 122 that extracts a reliability index, and includes a face similarity calculation unit. 14 includes a distribution estimation unit 141 that estimates parameter information related to the posterior distribution from the reliability index, and a distance calculation unit 142 that calculates the distance between the facial feature amount and the posterior distribution information from the distribution estimation unit 141. The
[0068]
At the time of registration, the image input unit 11 inputs a face photograph or the like after matching the size and position of the face with a scanner or a video camera. Alternatively, a person's face may be input directly from a video camera or the like. In this case, it is better to detect the face position of the input image and automatically normalize the size of the face image using the face detection technique as shown in the above-mentioned Moghaddam document. Would be good.
[0069]
Further, the input face image is registered in the face image database 15 as necessary. Simultaneously with the registration of the face image, face metadata is generated by the face metadata generation unit 12 and stored in the face metadata storage unit 13.
[0070]
Similarly, at the time of retrieval, a face image is input by the face image input unit 11, and face metadata is generated by the face metadata generation unit 12. The generated face metadata is once registered in the face metadata accumulation unit 13 or directly sent to the face similarity calculation unit 14. In the search, when it is confirmed whether or not a face image inputted in advance is in the database (face identification), the similarity with each piece of data registered in the face metadata storage unit 13 is calculated. Based on the result with the highest similarity (small distance value), the control unit 16 selects a face image from the face image database 15, displays the face image on the display unit 17 and the like, and displays the face in the search image and the registered image. The operator confirms the identity of.
[0071]
On the other hand, when confirming whether the face image specified in advance by the ID number or the like matches the searched face image (face identification), it is determined whether or not the face image with the specified ID number matches. When the similarity is lower than the predetermined similarity (the distance value is large), it is determined not to match, and when the similarity is high, it is determined to match, and the result is calculated. It is displayed on the display unit 17. If this system is used for entry management, entry control can be performed by controlling the automatic door by sending an opening / closing control signal from the control unit 16 to the automatic door instead of displaying.
[0072]
As described above, the face image matching system operates, but such an operation can also be realized on a computer system. For example, a metadata generation program that executes metadata generation and a similarity calculation program that executes similarity calculation, which will be described in detail below, are stored in memory, respectively, and are executed by a program control processor. Face image matching can be realized.
[0073]
Next, the operation of the face image matching system, particularly the face metadata generation unit 12 and the face similarity calculation unit 14 will be described in detail.
[0074]
(1) Face metadata generation
The face metadata generation unit 12 extracts a face feature amount using an image I (x, y) whose position and size are normalized. For normalization of the position and size, for example, the image may be normalized so that the eye position is (16, 24), (31, 24), and the size is 46 × 56 pixels. Hereinafter, a case where an image is normalized to this size will be described.
[0075]
As a face feature amount, feature extraction is performed using a so-called eigenface technique (Moghaddam's paper described above). That is, the basis matrix U specified by the partial basis vector selected from the basis vectors obtained by the principal component analysis of the face image sample set [Λ] with respect to the feature vector Λ having the pixel values in the image as elements. And the feature vector v = U using the mean face Ψ which is the mean vector in the face image sample set [Λ].^T(Λ−Ψ) is calculated as a feature vector. In this way, the data amount of the input image can be reduced by using the partial basis vector instead of the input image. Reducing the amount of data in this way is an important factor for realizing high-speed matching as well as reducing the amount of storage in the meta database. As the number of dimensions of the feature vector, for example, a 48-dimensional feature amount may be used.
[0076]
In addition to the commonly used eigenface technique, partial basis vectors may be defined using a technique that combines discriminant analysis with principal component analysis (the above-mentioned W. Zhao paper). Absent.
[0077]
In addition, the base matrix Un specified by the partial base vector selected from the base vectors obtained by the principal component analysis or discriminant analysis of the feature vector set [Λ] in the face image sample as described above, and the face image are horizontally reversed. The matrix U (= aUn + bUm) obtained by linear summation of the basis matrix Um specified by the inverted partial basis vector with the basis vector elements replaced so as to correspond to the conversion of the pixel value to be A face feature vector may be extracted. For example, when a = b = 1, the obtained facial feature vector can extract only components that are symmetric with respect to the left and right transformations in the input image space. Since the face is essentially symmetric, the image component that has become asymmetric due to the effect of lighting and the asymmetric component that occurs because the orientation of the face is not directed to the front is essentially equivalent to noise. By extracting only symmetric components, it is possible to extract a facial feature quantity that is stable against changes in illumination and posture.
[0078]
Also, the image is subjected to Fourier transform, and a vector whose element is the size of each component of the obtained complex Fourier component is calculated as a feature vector Λ. The amount may be extracted. In this way, by performing Fourier transform on the image, it is possible to extract a facial feature amount that is strong against displacement. In this way, the facial feature extraction unit 121 extracts the facial feature quantity v.
[0079]
In the reliability index extraction unit 122, a contrast index θ that is effective as a reliability index of the face feature v_contrastAnd the asymmetry index θ_asymmetricTo extract. Contrast index θ_contrastThe standard deviation of the pixel value of the face image I (x, y) is calculated by the following equation (Equation 21).
[0080]
[Expression 21]

Here, round () means rounding of numerical values. Reliability index θ extracted in this way_contrastIf the value exceeds the range, the value is limited so that the value falls within the 4-bit range of [0,1,2, ..., 15]. In the above description, the standard deviation of the image is calculated as the contrast index. However, the variance or the difference between the maximum value and the minimum value of the pixel values in the image may be extracted. A contrast index based on the maximum and minimum pixel values in an image has a smaller amount of computation than the standard deviation and variance, but the effect is relatively small.
[0081]
Asymmetry index θ_asymmetricAs an average, the average of the absolute value (first power) of the difference between the face image I (x, y) and its inverted image is extracted by the following equation (Equation 22).
[0082]
[Expression 22]

Reliability index θ extracted in this way_asymmetricIf the value exceeds the range to fit within the 4-bit range of [0,1,2, ..., 15], the value is limited. In the above description, the absolute value (first power) is used as the asymmetric index, but the square of the difference or the like may be used. Further, the same effect can be obtained by using a value such as a sum instead of the average. Further, if the maximum value of the difference is detected and used as an asymmetric index, the amount of calculation can be reduced.
[0083]
In the reliability index extraction unit 122, the feature vector v and the reliability index θ are applied to the face image._contrastAnd reliability index θ_asymmetricIs extracted and output as face metadata. As described above, the face metadata generation procedure can be executed by a computer using a computer program.
[0084]
(2) Face similarity calculation
Next, the operation of the face similarity calculation unit 14 will be described. In the face similarity calculation unit 14, the reliability index θ of the two face metadata_{contrast, 1}, Θ_{constrast, 2}And θ_{asymmetri c, 1}, Θ_{asymmetric, 2}, The distribution estimation unit 141 estimates parameter information related to the posterior distribution, and the two feature vectors v of the two face metadata.₁, V₂And the parameter information regarding the posterior distribution, the distance calculation unit 142 calculates the similarity d between the facial features.
[0085]
Here, a case will be described in which the face similarity is calculated by (Equation 8) or (Equation 18).
[0086]
Reliability index θ expressed by (Equation 8) or (Equation 18)₁, Θ₂Is a vector in this embodiment, and each element is θ₁= (θ_{contrast, 1}, θ_{asymmetric, 1})^T, Θ₂ = (θ_{contrast, 2}, θ_{asymmetric, 2})^TIt becomes. Since the contrast index and the asymmetry index are each expressed in 4 bits, θ_iThere are 256 possible states. Given reliability index θ_iThus, one of the 256 states can be specified.
[0087]
As will be described later, the distribution estimation unit 141 distributes a difference vector σε with respect to a reliability index θ (having 256 states) obtained in advance._{, k}(θ) and variance σ_{W, k}(θ), variance σ_{B, k} (θ) is stored in a table, and the reliability index θ₁And θ₂Are used to refer to the values of the respective variance value tables, and the obtained variance values are delivered to the distance calculation unit 142 as posterior distribution information. In the case of face identification, the variance value σε required in (Equation 8)_{, k}(θ₁), Variance σε_{, k} (θ₂) Value may be output to the distance calculation unit 142. In the case of face identification, the variance value σ required for (Equation 18)_{W, k}(θ), variance σ_{B, k}Output (θ).
[0088]
The distance calculation unit 142 calculates an adaptive Mahalanobis distance or an adaptive discrimination distance according to (Equation 8) or (Equation 18), and outputs it as the similarity d.
[0089]
The variance values in the above-described variance value table are calculated using face image data samples prepared in advance. Feature vector set in face image sample [v_i] And its reliability index [θ_i], Each variance value can be calculated as follows:
[0090]
[Expression 23]

[0091]
[Expression 24]

Here, “(i, j) belongs to class W” means that data i and data j are data obtained from the same class (same person) (within the class), and “( “i, j) belongs to class B” means that data i and data j are data obtained from different classes (different persons) (between classes). Nε (θ), N_W(θ), N_B(θ) is the number of combinations of data belonging to each class. In this way, when calculating the variance value for each bin of θ, if the number of data is too small, the data of neighboring bins are merged to ensure the number of samples (this is K-nearest neighbor method in distribution estimation (similar to merging bin by Bishop literature, pp. 53).
[0092]
Here, in the case of (Equation 18), as in (Equation 4), σ_{W, k}([θ_i])² = σ_{W, k}(θ₁)² + σ_{W, k}(θ₂)²And σ_{B, k}([θ_i])² = σ_{B, k}(θ₁)² + σ_{B, k}(θ₂)²Note that
[0093]
The error variance σε_{, k}(θ)²And intra-class variance σ_{W, k}(θ)²Therefore, when the distances of both (Equation 8) and (Equation 18) are calculated by the face image matching system, these variance value tables may be shared.
[0094]
In addition, the error distribution and interclass distribution often have a strong correlation, so the error variance σε_{, k}(θ)²Instead of class variance σ_{B, k}(θ)²Even if is used, the accuracy is improved as compared with the case where the reliability index is not used (but the accuracy is better if error variance is used).
[0095]
In this way, the similarity between face metadata is represented by the reliability index θ_contrastAnd reliability index θ_asymmetricBy calculating using the posterior distribution information via, it is possible to perform face recognition with high accuracy. As described above, the computer can execute the face similarity calculation procedure using a computer program.
[0096]
Here, the degree of similarity is calculated using (Equation 8) and (Equation 18), but it is also possible to approximate the calculation by various calculation methods such as the following to increase the speed.
[0097]
[Expression 25]

[0098]
[Equation 26]

By not calculating the second term on the right side of each of the above formulas (the part of ln), it is possible to achieve further high-speed computation.
[0099]
Also, when calculating the similarity according to (Equation 6) or (Equation 16), basically, from the face image data sample prepared in advance, the covariance matrix Σε of the difference vector error necessary for each calculation is used. (θ) or covariance matrix Σ within the class of difference vectors_W(θ), covariance matrix Σ between classes_B(θ) may be calculated and prepared as a covariance table, and the covariance table may be referred to when calculating the similarity. In this method, the amount of calculation increases because the distance is calculated using the covariance matrix, but the accuracy of similarity calculation can be improved when there are sufficient learning samples.
[0100]
By assuming a mixed normal distribution for the posterior distribution of (Equation 3), the interclass distribution of (Equation 14), and the interclass distribution, and estimating the distribution function, respectively, (Equation 10) and (Equation 20) An adaptive mixture Mahalanobis distance or an adaptive mixture discrimination distance may be calculated. In this case, the covariance matrix Σ representing the mixed normal distribution is obtained from the face image data sample in the same manner as calculating the posterior distribution information using the variance and the covariance matrix._s(θ₁, j), P (j), and the like parameters for specifying the mixture distribution may be obtained and stored as a table. This estimation may be performed using a general likelihood estimation method such as a maximum likelihood estimation method or an EM algorithm.
[0101]
In the above description, a case where a single face image is registered and retrieval is performed using a single face image has been described. However, a plurality of images are registered for one face, and a single face image is registered. When using the search, for example, the following may be performed.
[0102]
The feature vector on the search side is v_queAnd the registration-side feature vector is v_{reg, k}And similarity d when registering multiple images_multi(v_que, [v_{reg, 1}, v_{reg, 2}, ..., v_{reg, n}]), The similarity may be calculated based on the calculation formulas shown in the following equations (Equations 27 and 28).
[0103]
[Expression 27]

Or
[0104]
[Expression 28]

Similarly, in the case of registration of a plurality of images per face and search by a plurality of images, the similarity for one face data is calculated by calculating the similarity by calculating the average or the minimum value of the similarity of each combination. can do. This means that the matching system of the present invention can be applied to face recognition in a moving image by imitating the moving image as a plurality of images.
[0105]
Also, in the above explanation, the explanation was centered on the identification of the same person's face and the identification of the face, but for example, considering the male face as one category and the female face as one category, information on each distribution is given. It can also be applied to facial expression recognition for recognizing facial expressions in consideration of gender identification that identifies male and female faces, or considering facial expression categories such as laughed faces, angry faces, and sad faces. Age can be determined by setting categories such as teens, 20s, 30s and 40s, and obtaining error distribution, intraclass distribution and interclass distribution for each category. The present invention can be applied to face recognition.
[0106]
【The invention's effect】
As described above in detail, according to the present invention, by adding a reliability index for expressing the reliability of a face feature amount as face metadata, adaptive generation through the reliability index is achieved. It is possible to guide a highly accurate face recognition using the distance.
[Brief description of the drawings]
FIG. 1 is a block diagram showing a configuration of a face image matching system according to an embodiment of the present invention.
[Explanation of symbols]
11: Face image input part
12: Face metadata generator
13: Face metadata storage
14: Face similarity calculator
15: Face image database
16: Control unit
17: Display
121: Facial feature extraction unit
122: Reliability index extraction unit
141: Distribution estimation part
142: Distance calculator

Claims (10)

Face feature extraction means for extracting a face feature value from a face image; and reliability index extraction means for extracting a reliability index representing the reliability of a face recognition result using the face feature value, the face feature value and In the face metadata generation device that outputs the reliability index as face metadata,
When the feature vector obtained from the face image is Λ, the face feature extracting means uses the basis matrix U specified by the partial basis vector of the feature vector Λ and the average vector Ψ of the feature vector Λ. A vector v = U ^T (Λ−Ψ) is calculated, and the feature vector v is extracted as a facial feature quantity ;
The reliability index extracting unit extracts a statistical quantity using pixel values in the face image as a sample as a reliability index representing reliability of face recognition accuracy of the face feature quantity. apparatus.   2. The face metadata generation according to claim 1, wherein the basis matrix U is specified by a partial basis vector selected from basis vectors obtained by principal component analysis of a face image sample set [Λ]. apparatus.   The basis matrix U includes a basis matrix specified by a partial basis vector selected from basis vectors obtained by principal component analysis of a feature vector set [Λ] in a face image sample, and pixel values for horizontally flipping a face image. 2. The face metadata generation apparatus according to claim 1, wherein the face metadata generation apparatus is obtained by a linear sum of a base matrix specified by an inverted partial base vector obtained by replacing elements of the base vector so as to correspond to a transformation.   4. The face metadata generation apparatus according to claim 1, wherein the feature vector Λ is a vector having each pixel value in the image as an element.   4. The face metadata generation according to claim 1, wherein the feature vector Λ is a vector whose element is the magnitude of each component of a complex Fourier component obtained by subjecting an image to Fourier transform. 5. apparatus.   In a face metadata generation method for outputting a face feature amount extracted from a face image and a reliability index representing the reliability of a face recognition result using the face feature amount as face metadata,
  When a feature vector obtained from the face image is Λ, a feature vector v = U using a basis matrix U specified by a partial basis vector of the feature vector Λ and an average vector Ψ of the feature vector Λ. ^T ( Λ-Ψ ) And extracting the feature vector v as the face feature amount,
  Extracting a statistic obtained by sampling pixel values in the face image as a reliability index representing the reliability of face recognition accuracy of the face feature amount;
  A method for generating face metadata.   The basis matrix U is a face image sample set. [ Λ ] The face metadata generation method according to claim 6, wherein the face metadata generation method is specified by a partial basis vector selected from basis vectors obtained by principal component analysis.   The basis matrix U is a feature vector set in a face image sample. [ Λ ] An inverted part in which elements of the basis vector are interchanged so as to correspond to a basis matrix specified by a partial basis vector selected from basis vectors obtained by principal component analysis of and a pixel value conversion that horizontally inverts a face image The face metadata generation method according to claim 6, wherein the face metadata generation method is obtained by a linear sum of a basis matrix specified by a basis vector.   The feature vector Λ is a vector having each pixel value in the image as an element. The face metadata generation method according to claim 6.   9. The face metadata generation according to claim 6, wherein the feature vector Λ is a vector whose element is the magnitude of each component of a complex Fourier component obtained by Fourier transforming an image. Method.

Images