JP4273748B2 - Image processing apparatus and method - Google Patents

Description

【００８８】
すなわち、εフィルタ５３は、フィルタ処理の中心画素ｐn の画素値Ｘn と画素ｐn-k の画素値Ｘn-k との差の絶対値｜Ｘn −Ｘn-k ｜を所定の閾値εと比較する。その結果、εフィルタ５３Ａは、絶対値｜Ｘn −Ｘn-k ｜が所定の閾値εよりも小さいと判断した場合には、ｗn-k に画素値Ｘn-k を代入し、ａk を各タップ係数とした通常の線形ローパスフィルタと同様の処理を実行することにより、中心画素ｐn を中心として画像を一様に平滑化する。
【００８９】
これに対して、εフィルタ５３は、絶対値｜Ｘn −Ｘn-k ｜が所定の閾値εよりも大きいと判断した場合には、ｗn-k に画素値Ｘn を代入し、画素ｐn-k の画素値Ｘn-k を中心画素ｐn の画素値Ｘn に置き換えた後に当該中心画素ｐn を中心としてローパスフィルタ処理を行うことにより、画素値Ｘn-k を無視して画素値Ｘn 近傍の画素値のみで平滑化を行う。
【００９０】
図１１は、εフィルタ５３Ａ（１７タップのεフィルタ）のレジスタ１０１乃至１１６に保持される画素値を説明する図である。
【００９１】
図１１において、フィルタ中心画素の画素値はＣとされ、中心画素より左側の画素値がＸ１，Ｘ２，Ｘ３，Ｌ１，Ｘ４，Ｘ５，Ｘ６，およびＬ２とされている。また、中心画素より右側の画素値がＹ１，Ｙ２，Ｙ３，Ｒ１，Ｙ４，Ｙ５，Ｙ６，およびＲ２とされている。
【００９２】
通常、ラインの走査は、左上から右下方向に行なわれるので、図中、最も左側の画素値Ｌ２が最も先に走査され、最も右側の画素値Ｒ２が最後に走査されるので、画素値Ｌ２がレジスタ１１６に保持され、画素値Ｘ６がレジスタ１１５に保持され、画素値Ｘ５がレジスタ１１４に保持される。以下、順次、図１１において左側の画素値から順番にレジスタ１１３，１１２，１１１，・・・に保持され、画素値Ｙ６がレジスタ１０１に保持され、画素値Ｒ２がセレクタ１４４に直接供給される（画素値Ｒ２をラッチするレジスタを設けてもよい）。
【００９３】
上述したようにεフィルタ５３Ａのタップ係数は｛１，０，０，０，２，０，０，０，２，０，０，０，２，０，０，０，１｝であるので、Ｌ２，Ｒ２の係数は１とされ、Ｌ１，Ｃ１，Ｒ１の係数は２とされ、Ｘ１乃至Ｘ６（それぞれ、レジスタ１１５，１１４，１１３，１１１，１１０，１０９に保持される）、並びに、Ｙ１乃至Ｙ６（それぞれ、レジスタ１０７，１０６，１０５，１０３，１０２，１０１に保持される）の係数は０とされる（各レジスタの出力が加算器１８１に供給されていない）。
【００９４】
上述した式（１）を適用すると、例えば、画素値Ｃと画素値Ｌ１との差の絶対値｜Ｃ−Ｌ１｜が、所定の閾値ε₁と比較され、所定の閾値ε₁よりも大きいとされた場合、画素値Ｌ１が、画素値Ｃに置き換えられる。式（１）によりＲ２，Ｙ６，Ｙ５，Ｙ４，Ｒ１，Ｙ３，Ｙ２，Ｙ１，Ｘ１，Ｘ２，Ｘ３，Ｌ１，Ｘ４，Ｘ５，Ｘ６，およびＬ２のそれぞれに対して演算された結果の画素値をＲ２’，Ｙ６’，Ｙ５’，Ｙ４’，Ｒ１’，Ｙ３’，Ｙ２’，Ｙ１’，Ｘ１’，Ｘ２’，Ｘ３’，Ｌ１’，Ｘ４’，Ｘ５’，Ｘ６’，およびＬ２’とすると、εフィルタ５３Ａから出力される画素値Ｃ’は、式（２）が演算されることにより行なわれる（式（２）は、図１０のステップＳ１０６乃至ステップＳ１０８の処理をまとめた式とされる）。
【数２】

【００９５】
図１２は、εフィルタ５３Ａ乃至εフィルタ５３Ｃにおける閾値εの関係を説明する図である。
【００９６】
εフィルタ５３Ａ乃至εフィルタ５３Ｃにおける閾値εは、後段のεフィルタ５３になるほど、大きくなる。すなわち、この場合、閾値ε₁＜ε₂＜ε₃とされる。
【００９７】
閾値εは、大きい値であるほど、ナマリが生じやすく（すなわち、ボケが生じるため、ノイズが取れる）、小さい値であるほど、ナマリが生じにくい（すなわち、ボケが生じにくく、ノイズが残る）。
【００９８】
例えば、タップ数が３とされ、タップ係数が｛１，２，１｝とされ、Ｌ１，Ｃ，Ｒ１が、それぞれＬ１＝３０，Ｃ＝１００およびＲ１＝８０とされ、閾値εが５０とされる場合、式（１）により、Ｌ１’＝１００とされる（Ｒ１’はそのままの値（Ｒ１）とされる）。また、式（２）により、Ｃ’＝｛１×１００＋２×１００＋１×８０｝／４が演算され、出力画素Ｃ’＝９５とされる。一方、例えば、タップ数が３とされ、タップ係数が｛１，２，１｝とされ、Ｌ１＝３０，Ｃ＝１００，Ｒ１＝８０とされ、閾値εが８０とされる場合（すなわち、上述した例と閾値εの値のみが異なる場合）、式（１）から、Ｌ１’とＲ１’はそのままの値（Ｌ１，Ｒ１）とされる。また、式（２）により、Ｃ’＝｛１×３０＋２×１００＋１×８０｝／４が演算され、出力画素Ｃ’＝７７．５とされる。このように、閾値εが大きいほど（いまの例の場合、閾値εが５０より８０であるほど）、出力画像にナマリが生じる（ボケが生じる）こととなる。
【００９９】
図１３は、εフィルタ５３Ｂ（９タップのεフィルタ）のレジスタ３０１乃至３０８に保持される画素値を説明する図であり、図１４は、εフィルタ５３Ｃのレジスタ３０１乃至３０８に保持されている画素値を説明する図である。
【０１００】
図１３において、フィルタ中心画素の画素値はＣとされ、中心画素より左側の画素値がＸ１，Ｌ１，Ｘ２，およびＬ２とされている。また、中心画素より右側の画素値がＹ１，Ｒ１，Ｙ２，およびＲ２とされている。上述したようにεフィルタ５３Ｂのタップ係数は｛１，０，２，０，２，０，２，０，１｝であるので、Ｌ２，Ｒ２の係数は１とされ、Ｌ１，Ｃ１，Ｒ１の係数は２とされ、Ｘ１，Ｘ２，Ｙ１，およびＹ２（それぞれ、レジスタ３０７，３０５，３０３，３０１に保持される）の係数は０とされる。
【０１０１】
また、図１４のεフィルタ５３Ｃにおいて、フィルタ中心画素の画素値はＣとされ、中心画素より左側の画素値がＬ１，およびＬ２とされている。また、中心画素より右側の画素値がＲ１，およびＲ２とされている。上述したようにεフィルタ５３Ｃのタップ係数は｛１，２，２，２，１｝であるので、Ｌ２，Ｒ２の係数は１とされ、Ｌ１，Ｃ１，Ｒ１の係数は２とされる。
【０１０２】
このように、εフィルタ５３Ａのインターバルは、４（＝２²）（インターバル設定のためのレジスタの数は３個）、εフィルタ５３Ｂのインターバルは、２（＝２¹）（インターバル設定の為のレジスタの数は１個）、εフィルタ５３Ｃのインターバルは、１（＝２⁰）（インターバル設定のためのレジスタの数は０個）とされる。
【０１０３】
ここで、それぞれの画素値に対して上述した式（１）と式（２）が適用されることにより、出力画素値Ｃ’が得られる。このとき、タップ数が大きいεフィルタ５３Ａに比較して、εフィルタ５３Ｂにより演算される出力画素値Ｃ’は、ノイズが多くなる。また、タップ数が大きいεフィルタ５３Ｂに比較して、εフィルタ５３Ｃにより演算される出力画素値Ｃ’は、ノイズが多くなる。すなわち、ノイズの大きさは、タップ数の多いεフィルタ５３Ａにおいてはノイズが少なく、後段のεフィルタ５３Ｃ（タップ数が少ない）になるに連れて、ノイズが多くなる。
【０１０４】
そこで、図１２に示されるように、εフィルタ５３の閾値εを、εフィルタのタップ数により変化させる（タップ数の多いεフィルタ５３Ａにおいては、閾値ε₁を小さくさせ、タップ数の少ないεフィルタ５３Ｃにおいては、閾値ε₃を大きくさせる）ことにより、εフィルタ５３Ａ乃至εフィルタ５３Ｃの全ての出力画素値Ｃ’のノイズを抑えることができる。
【０１０５】
図１５は、平滑化処理される画像データを説明する図である。
【０１０６】
図中左側の入力された画像データは、εフィルタ５３により平滑化処理されることによって、図中右側に示されるような画像データにされる。
【０１０７】
すなわち、急峻なエッジの前後で画素値の差の絶対値が所定の閾値εを超える場合、中心画素ｐn を中心とし、ローパスフィルタ処理を行う際には画素ｐm の画素値Ｘm を中心画素ｐn の画素値Ｘn に置き換えてローパスフィルタ処理を施すことにより、画素値Ｘn の近傍で平滑化するのに対して、画素ｐm を中心としてローパスフィルタ処理を行う際には画素値Ｘm 近傍で平滑化する。
【０１０８】
このとき、εフィルタ５３Ａは、フィルタ処理を行う範囲内に画素値Ｘの近い画素ｐが存在しないエッジ部分の画素ｐでは、当該エッジ部分の画素ｐの画素値Ｘをほぼそのまま出力することにより、エッジの急峻な変化をそのまま保存する（式（１））。因みに、εフィルタ５３Ａは、線形ローパスフィルタ５１と同様に、１次元のεフィルタを画像の水平方向および垂直方向にそれぞれ適用することによって構成される場合と、２次元のεフィルタによって構成される場合とがある。
【０１０９】
図１６は、εフィルタ５３Ａ乃至εフィルタ５３Ｃにおける周波数応答を説明する図である。
【０１１０】
図１６において、横軸はナイキスト周波数を０．５として正規化した周波数であり、縦軸はフィルタ周波数応答を表わしている。５３Ａ，５３Ｂ，５３Ｃは、それぞれ、εフィルタ５３Ａ、εフィルタ５３Ｂ、およびεフィルタ５３Ｃに対応する周波数応答を示す。
【０１１１】
εフィルタ５３（例えば、εフィルタ５３Ａ）のタップ数のインターバル（間引き）を２のべき乗とすることにより、図１６に示されるように、εフィルタ５３Ａ（例えば、インターバル＝４）の周波数応答の波形におけるサイドローブのピークを、εフィルタ５３Ｂ（例えば、インターバル＝２）の周波数応答により打ち消すことができる（εフィルタ５３Ａの周波数応答の波形におけるサイドローブのピーク点と、εフィルタ５３Ｂの周波数応答の波形における０点が同じであるため、打ち消している）。すなわち、前のεフィルタ５３のサイドローブを、次のεフィルタ５３により抑制することができ、もって、平滑化効果とサイドローブの抑制を両立することができる。
【０１１２】
εフィルタ５３により平滑化された平滑化画像データは、上述したように、演算部２３により演算され、画像処理回路１２の出力として、カメラ信号処理回路１３に供給される。このとき、画像処理回路１２から出力される出力画像データの各画素値ｙ（ｉ，ｊ）は、式（３）により示される。
【数３】

【０１１３】
式（３）において、ゲイン係数ｇ（ｉ，ｊ）を全画面に一様なゲイン係数Ｇに置き換えると共に、平滑化画像データの各画素値ｓ（ｉ，ｊ）を入力画像データのダイナミックレンジの中央値又は全画素値の平均値Ｃに置き換えると、この式（３）は、従来のトーンカーブ調整によるコントラスト強調方法を示す式（４）のように表わされる。
【数４】

【０１１４】
また、式（３）において、ゲイン係数ｇ（ｉ，ｊ）を全画面に一様なゲイン係数Ｇに置き換えると共に、平滑化画像データの各画素値ｓ（ｉ，ｊ）を入力画像データの各画素値ｘ（ｉ，ｊ）に線形ローパスフィルタ処理を施した画像データの各画素値ｆ（ｉ，ｊ）に置き換えると、この式（３）は、従来のアンシャープマスクによる高域成分強調方法を示す式（５）のように表わされる。
【数５】

【０１１５】
このように、従来のコントラスト強調方法は、画像の各画素毎に独立して画像強調処理を行う方法であると共に、従来の高域成分強調方法は、中心画素の周辺に存在する周辺画素との相対的なレベル差を基に画像強調処理を行う方法であるのに対して、本実施の形態による方法は、非線形平滑化フィルタを用いて従来のコントラスト強調方法及び高域成分強調方法を統合し、一段と高品位な画像強調処理を可能にしている。
【０１１６】
また、従来の非線形平滑化フィルタを用いた方法（入力画像データのうち画素値の変化が急峻なエッジを保存したまま当該エッジ以外の部分を増幅することにより、エッジ以外の部分を強調して表示する方法）に対して、本実施の形態による方法は、εフィルタ５３のタップ数のインターバルを２のべき乗とし、εフィルタ５３の閾値εをフィルタの帯域にあわせて変更することにより、画素値の変化が急峻なエッジを保存したまま当該エッジ以外の部分を平滑化する場合においても、計算量を抑えることができる。
【０１１７】
以上のように、帯域に応じた閾値εを設定し（狭帯域フィルタ（タップ数の多いεフィルタ）の場合は、閾値εを小さくし、広帯域フィルタ（タップ数の少ないεフィルタ）の場合は、閾値εを大きくする）、狭帯域フィルタから広帯域フィルタの順番にεフィルタをカスケード接続し、さらに、εフィルタのインターバル（実質的に処理する画素の抽出間隔）を２のべき乗とすることにより、画素値の変化が急峻なエッジを保存したまま当該エッジ以外の部分を平滑化する場合においても、計算量を抑えることができる。
【０１１８】
また、ノイズが生じにくく、かつ、計算量を抑えた出力画像を得ることができる。
【０１１９】
また、サイドローブを抑えた画像を出力させることができる。
【０１２０】
さらに上述の実施の形態においては、本発明をビデオカメラ１に適用した場合について述べたが、本発明はこれに限らず、例えば電子スチルカメラ、プリンタ、ディスプレイ、コンピュータのような他の種々の画像処理装置に本発明を広く適用し得る。この場合、コンピュータでは、画像コントラストを修正する際、ダイナミックレンジを維持しながら高品位なコントラスト修正画像を得ることができ、また異なる照明条件下で得られた画像同士を合成する際、それぞれのコントラスト成分の違いのみを補正でき、自然な合成画像を生成することができる。
【０１２１】
なお、以上の例においては、非線形平滑化器２２が備えるεフィルタの数を３個（εフィルタ５３Ａ，εフィルタ５３Ｂ、およびεフィルタ５３Ｃ）としたが、任意のＮ個のεフィルタ５３を備えるようにしてもよい。勿論、その場合、Ｎ番目のεフィルタをεフィルタ５３Ｎとすると、εフィルタ５３Ａの閾値ε₁は最小とされ、後段に接続されるεフィルタ５３Ｎの閾値ε_Nは最大とされる（ε₁＜ε_N）。
【０１２２】
また、以上の例においては、εフィルタを用いるようにしたが、εフィルタだけでなく、例えば、メディアンフィルタ等を用いるようにしても良い。
【０１２３】
なお、本明細書において、各フローチャートを記述するステップは、記載された順序に従って時系列的に行われる処理はもちろん、必ずしも時系列的に処理されなくとも、並列的あるいは個別に実行される処理をも含むものである。
【０１２４】
【発明の効果】
以上の如く、本願発明によれば、画素値の変化が急峻なエッジを保存したまま当該エッジ以外の部分を平滑化する場合においても、計算量を抑えることができる。特に、この発明によれば、ノイズが生じにくく、かつ、サイドローブを抑えた画像を出力させることができる。
【図面の簡単な説明】
【図１】本発明を適用したビデオカメラの機能的構成を示すブロック図である。
【図２】図１の画像処理回路における画像処理を説明するフローチャートである。
【図３】図１の非線形平滑化器の機能的構成を示すブロック図である。
【図４】図２のεフィルタの構成例を示すブロック図である。
【図５】図２の他のεフィルタの構成例を示すブロック図である。
【図６】図２のさらに他のεフィルタの構成例を示すブロック図である。
【図７】図４のセレクタの構成例を示すブロック図である。
【図８】図２の非線形平滑化器における非線形平滑化処理を説明するフローチャートである。
【図９】図８のステップＳ３４の処理を説明するフローチャートである。
【図１０】図９のステップＳ７１乃至ステップＳ７３の処理を説明するフローチャートである。
【図１１】図４のレジスタに保持される画素値を説明する図である。
【図１２】図３の各εフィルタにおける閾値の関係を説明する図である。
【図１３】図５のレジスタに保持される画素値を説明する図である。
【図１４】図６のレジスタに保持される画素値を説明する図である。
【図１５】平滑化処理される画像データを説明する図である。
【図１６】タップ数と周波数応答の関係を示す図である。
【符号の説明】
１ ビデオカメラ， １１ 撮像素子， １２ 画像処理回路， ２１ 遅延回路， ２２ 非線形平滑化器， ２３ 演算部， ３１ 減算器， ３２ 乗算器， ３３ 加算器， ５１ 線形ローパスフィルタ， ５２ ルックアップテーブル， ５３ εフィルタ， ５４ ルックアップテーブル， ５５ 線形ローパスフィルタ， １０１乃至１１６ レジスタ， １４１乃至１４４ セレクタ， １６１乃至１６５ 増幅器， １８１ 加算器， １８２ 割算器， １４０ 閾値設定部， ２００ 減算器， ２０１ セレクタ， ２０２ 絶対値変換器， ２０３ 大小比較器[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an image processing apparatus and method, and in particular, image processing capable of suppressing the amount of calculation even when smoothing a portion other than an edge while preserving an edge with a sharp change in pixel value. The present invention relates to an apparatus and a method.
[0002]
[Prior art]
Conventional video cameras improve the contrast (brightness and darkness) and sharpness (brightness of boundaries) of images captured by image sensors such as CCD (Charge Coupled Device) and CMOS (Complementary Metal-Oxide Semiconductor). As a method, a contrast enhancement method by gradation conversion and a high frequency component enhancement method for enhancing the contrast of a high frequency component in an image are considered.
[0003]
As a contrast enhancement method, tone curve adjustment for converting each pixel level of an image with a function having a predetermined input / output relationship (hereinafter referred to as a level conversion function), or frequency distribution of pixel levels A method called histogram equalization has been proposed in which the level conversion function is adaptively changed according to the frequency.In addition, as a high-frequency component enhancement method, so-called contour enhancement that extracts edges from an image and emphasizes the extracted edges is proposed. A method called an unsharp mask has been proposed.
[0004]
However, in contrast enhancement methods, in addition to the problem that the contrast can be improved only in a part of the luminance range in the entire dynamic range (difference between the maximum level and the minimum level) of the image, in addition, in the case of tone curve adjustment However, there is a problem that the contrast is lowered in the brightest and darkest parts of the image, and in the case of histogram equalization, in the vicinity of the luminance region where the frequency distribution is small. Further, in the high frequency component enhancement method, only the contrast of the high frequency component of the image is enhanced, and thus, there is a problem that it is inevitable that the vicinity of the edge of the image is unnaturally enhanced and the image quality is deteriorated.
[0005]
Therefore, the present applicant has first proposed that the portion other than the edge is emphasized and displayed by amplifying the portion other than the edge while preserving the edge where the pixel value changes sharply in the input image data. (For example, Patent Document 1).
[0006]
[Patent Document 1]
JP 2001-298621 A
[0007]
[Problems to be solved by the invention]
However, in the method described in Patent Document 1, when smoothing a portion other than the edge while preserving an edge with a sharp change in pixel value, in order to obtain a smoothing effect sufficiently, It is necessary to provide a non-linear filter having a large number of taps, which increases the amount of calculation.
[0008]
The present invention has been made in view of such a situation, and the amount of calculation can be reduced even when a portion other than the edge is smoothed while an edge with a sharp change in pixel value is preserved. It is what you want to do.
[0009]
[Means for Solving the Problems]
An image processing apparatus according to the present invention includes a smoothing unit that smoothes input image data and generates smoothed image data. series Connected to Multiple Nonlinear filter A plurality of nonlinear filters are filters having different smoothing thresholds and different tap numbers by setting different intervals for the taps. It is characterized by that.
[0010]
The smoothing means is a difference value between a central pixel and its neighboring pixels. And the threshold value On the basis of the, Above It is possible to adaptively change the degree of smoothing.
[0011]
Of the nonlinear filter The threshold The latter stage can be made larger than the previous stage.
[0012]
Of the nonlinear filter Above The interval value can be made smaller in the latter stage than in the former stage.
[0013]
Of the nonlinear filter Above The interval can be a power of two.
[0014]
The nonlinear filter is an ε filter Is Can be.
[0015]
The image processing method of the present invention includes a smoothing step of smoothing input image data and generating smoothed image data. series Connected to the Number of taps due to different smoothing thresholds and different intervals inserted Is performed by a plurality of different nonlinear filters.
[0016]
In the present invention, the input image data is smoothed. Also, smoothing of input image data is series Connected to the Number of taps due to different smoothing thresholds and different intervals set for taps Are performed by a plurality of nonlinear filters.
[0017]
DETAILED DESCRIPTION OF THE INVENTION
FIG. 1 shows a configuration example of a video camera 1 to which the present invention is applied.
[0018]
An image sensor 11 made of an image sensor such as a CCD or CMOS acquires a light image from a subject as input image data, and inputs the input image data to the delay circuit 21 and the nonlinear smoother 22 of the image processing circuit 12. The input image data is a two-dimensional digital image, and the pixel value corresponding to the position (i, j) on the image is represented by x (i, j).
[0019]
The non-linear smoother 22 smoothes the input image data and generates smoothed data.
[0020]
The delay circuit 21 delays the input image data picked up by the image pickup device 11 by a predetermined amount, and sends this to the subtracter 31 of the calculation unit 23 as delayed image data. The computing unit 23 constituting the image enhancement means is composed of a multiplier 32 and an adder 33 in addition to the subtractor 31, and generates output image data. The subtractor 31 subtracts each pixel value s (i, j) of the smoothed image data from each pixel value x (i, j) of the delayed image data and supplies the difference image data obtained as a result to the multiplier 32. Output.
[0021]
The multiplier 32 multiplies each pixel value (x (i, j) -s (i, j)) of the difference image data by a gain coefficient g (i, j), and amplifies the result. The difference image data is output to the adder 33. As the gain coefficient g (i, j), a uniform value for the entire screen or a value set for each pixel (i, j) is used.
[0022]
The adder 33 is a pixel value of the difference image data supplied from the multiplier 32.
g (i, j) × (x (i, j) −s (i, j))
Is added to each pixel value s (i, j) of the smoothed image data, which is the offset amount subtracted by the subtractor 31, and the output image data obtained as a result is output to the camera signal processing circuit 13. .
[0023]
The camera signal processing circuit 13 performs predetermined data processing on the output image data, and sends the output image data obtained as a result to a VTR (Video Tape Recorder) 14 for recording.
[0024]
Next, image processing in the image processing circuit 12 of FIG. 1 will be described with reference to the flowchart of FIG. This process is started when input image data is supplied from the image sensor 11 in accordance with a user instruction.
[0025]
In step S <b> 1, the delay circuit 21 acquires input image data supplied from the image sensor 11.
[0026]
In step S <b> 2, the nonlinear smoother 22 acquires the input image data supplied from the image sensor 11.
[0027]
In step S3, the nonlinear smoother 22 extracts an edge component having a sharp change in pixel value from the input image data acquired by the processing in step S2, and the edge component is directly smoothed without being smoothed. In contrast to output, small amplitude components other than edge components are smoothed to smooth the input image data while preserving the edge components of the input image data to generate smoothed image data. To do. Details of the processing will be described later with reference to FIGS. The non-linear smoother 22 supplies the generated smoothed image data to the subtracter 31 and the adder 33.
[0028]
In step S4, the delay circuit 21 delays the input image data acquired by the process of step S1 by a predetermined amount, and sends this to the subtractor 31 as delayed image data.
[0029]
In step S5, the subtractor 31 is supplied from the non-linear smoother 22 (process of step S3) from each pixel value x (i, j) of the delay image data (process of step S4) supplied from the delay circuit 21. ) Each pixel value s (i, j) of the smoothed image data is subtracted to generate difference image data. The subtractor 31 supplies the generated difference image data to the multiplier 32.
[0030]
In step S6, the multiplier 32 applies the gain coefficient g to each pixel value (x (i, j) −s (i, j)) of the difference image data supplied from the subtractor 31 (processing in step S5). (I, j) is multiplied and amplified, and the difference image data obtained as a result is sent to the adder 33. The gain coefficient g (i, j) is a uniform value for the entire screen or a value set for each pixel (i, j).
[0031]
In step S7, the adder 33 sets each pixel value of the difference image data supplied from the multiplier 32 by the process of step S6.
g (i, j) × (x (i, j) −s (i, j))
Is added to each pixel value s (i, j) of the smoothed image data (smoothed image data supplied from the nonlinear smoother 22 by the processing of step S3), and the resulting output image data Is sent to the camera signal processing circuit 13.
[0032]
With the processing in FIG. 2, the image processing circuit 12 can smooth the input image data and obtain output image data subjected to image enhancement processing.
[0033]
FIG. 3 is a block diagram illustrating a configuration example of the nonlinear smoother 22.
[0034]
The linear low-pass filter 51 attenuates the signal level of an extremely high frequency component in the input image data, and sends the image data obtained as a result to the lookup table 52. The lookup table 52 performs gradation conversion such as logarithmic conversion on the image data supplied from the linear low-pass filter 51 (non-linear conversion is performed on input image data to generate non-linear image data). The image data obtained as a result is sent to the ε filter 53.
[0035]
The ε filter 53 includes n (three in the example of FIG. 3) ε filters 53A to 53C connected in cascade (cascade connection). The ε filter 53A is a non-linear smoothing filter, that is, a digital filter effective in smoothing the pixel value without impairing a sharp change in the pixel value, and the edge of the image data supplied from the lookup table 52 The image data is smoothed with the image stored, and the smoothed image data obtained as a result is sent to the ε filter 53B.
[0036]
The ε filter 53B connected to the subsequent stage of the ε filter 53A and the ε filter 53C connected to the subsequent stage have the same configuration as the ε filter 53A, but are different in the threshold ε and the band (number of taps). Details of the ε filter 53A to the ε filter 53C will be described later with reference to FIGS.
[0037]
The ε filter 53B smoothes the image data, and sends the smoothed image data obtained as a result to the ε filter 53C. The ε filter 53C further smoothes the smoothed image data supplied from the ε filter 53B, and supplies the smoothed image data to the lookup table 54.
[0038]
The look-up table 54 performs gradation conversion (inverse conversion of logarithmic conversion performed by the look-up table 52) on the smoothed image data supplied from the ε filter 53C, and smoothed image data obtained as a result thereof. Is sent to the linear low-pass filter 55.
[0039]
The linear low-pass filter 55 generates smoothed image data by slightly dulling the edges of the smoothed image data that has been smoothed while preserving the edges, and sends the smoothed image data to the subtractor 31 and the adder 33. Thereby, the smoothness of the image in the vicinity of the edge can be maintained in the subsequent image enhancement processing (processing executed by the calculation unit 23).
[0040]
FIG. 4 is a block diagram showing a detailed configuration example of the ε filter 53A.
[0041]
The ε filter 53A has a one-dimensional filter processing pixel of 17 taps, and the threshold set by the threshold setting unit 140 is ε ₁ It is said. The ε filter 53A includes registers 101 to 116, a threshold setting unit 140, selectors 141 to 144, amplifiers 161 to 165, an adder 181, and a divider 182. Each of the registers 101 to 116 is a register that holds a pixel value for one pixel.
[0042]
The selector 141 includes the value Xm supplied from the register 116, the pixel value Xn of the central pixel pn supplied from the register 108, and the threshold value ε supplied from the threshold setting unit 140. ₁ Based on the above, the calculation is performed, and the result of the calculation is supplied to the amplifier 161. The selector 142 receives the value Xm supplied from the register 112, the pixel value Xn of the central pixel pn supplied from the register 108, and the threshold value ε supplied from the threshold setting unit 140. ₁ The calculation is performed based on the above and the calculation result is supplied to the amplifier 161.
[0043]
The selector 143 includes the value Xm supplied from the register 104, the pixel value Xn of the central pixel pn supplied from the register 108, and the threshold value ε supplied from the threshold setting unit 140. ₁ Based on the above, the calculation is performed, and the result of the calculation is supplied to the amplifier 164. The selector 144 receives the input value Xm, the pixel value Xn of the central pixel pn supplied from the register 108, and the threshold value ε supplied from the threshold setting unit 140. ₁ The operation is performed based on the above, and the operation result is supplied to the amplifier 165.
[0044]
A register 191 composed of registers 101 to 103, a register 192 composed of registers 105 to 107, a register 193 composed of registers 109 to 111, and a register 194 composed of registers 113 to 115 each set an interval. It is provided for this purpose.
[0045]
The amplifier 163 is supplied with the pixel value Xn of the center pixel pn from the register 108.
[0046]
Each of the amplifiers 161 to 165 multiplies the input pixel value X by a tap coefficient ak and sends the calculation result to the adder 181. In this case, the tap coefficients of the amplifiers 161 and 165 are set to 1, and the tap coefficients of the amplifiers 162 to 164 are set to 2. The adder 181 adds the supplied values (values supplied from the amplifiers 161 to 165) and supplies the result to the division unit 182. The division unit 182 divides the supplied value (in this example, divides by 8), and sends this to the ε filter 53B. The divider 182 may be omitted, the tap coefficients of the amplifiers 161 and 165 may be 1/8, and the tap coefficients of the amplifiers 162 to 164 may be 2/8.
[0047]
FIG. 5 is a block diagram illustrating a detailed configuration example of the ε filter 53B, and FIG. 6 is a block diagram illustrating a detailed configuration example of the ε filter 53C.
[0048]
As is clear by comparing FIG. 4 with FIGS. 5 and 6, the ε filters 53B and 53C in FIGS. 5 and 6 have basically the same configuration as the ε filter 53A in FIG.
[0049]
That is, the ε filter 53B of FIG. 5 corresponds to the registers 101 to 116, the threshold setting unit 140, the selectors 141 to 144, the amplifiers 161 to 165, the adder 181, and the divider 182 of the ε filter 53A of FIG. , Registers 301 to 308, threshold setting unit 340, selectors 341 to 344, amplifiers 361 to 365, adder 381, and divider 382. Corresponding names have corresponding functions.
[0050]
However, the ε filter 53B is set by the threshold setting unit 340 and is supplied to the selectors 341 to 344. ₂ 4 and the number of registers (in the case of the ε filter 53A in FIG. 6, the number of registers is 16 and in the case of the ε filter 53B in FIG. 8, the number of registers is 8). Different from the ε filter 53A. In the ε filter 53B, the registers 391 to 394 for setting the interval are configured by one register 301, 303, 305, or 307, respectively.
[0051]
Further, the ε filter 53C of FIG. 6 corresponds to the registers 101 to 116, the threshold setting unit 140, the selectors 141 to 144, the amplifiers 161 to 165, the adder 181, and the divider 182 of the ε filter 53A of FIG. , Registers 401 to 404, threshold value setting unit 440, selectors 441 to 444, amplifiers 461 to 465, adder 481, and divider 482. Corresponding names have corresponding functions.
[0052]
However, as in the above-described FIG. 5 (ε filter 53B), the ε filter 53C is set by the threshold setting unit 440 and is supplied to the selectors 401 to 404 as the threshold ε. _Three And the number of registers (the number of registers of the ε filter 53A is 16 and the number of registers of the ε filter 53C is 4) are different from the ε filter 53A of FIG. In the ε filter 53C, a register for setting an interval is not provided.
[0053]
FIG. 7 is a block diagram illustrating a configuration example of the selector 141 in FIG.
[0054]
The pixel value Xm of the pixel pm (in this example, the pixel value supplied from the register 116) and the pixel value Xn of the central pixel pn supplied from the register 108 are input to the subtracter 200 and the selector 201. The subtractor 200 takes the difference between the pixel value Xn and the pixel value Xm and outputs the result to the absolute value converter 202. The absolute value converter 202 obtains the absolute value | Xn−Xm | of the difference between the pixel value Xn and the pixel value Xm, and outputs this to the magnitude comparator 203.
[0055]
The magnitude comparator 203 calculates the absolute value | Xn−Xm | and a predetermined threshold ε (threshold ε in this example). ₁ ) And the comparison result is sent to the selector 201. When the absolute value | Xn−Xm | is smaller than the predetermined threshold value ε, the selector 201 selects the pixel value Xm and sends it to the subsequent amplifier 161, and the absolute value | Xn−Xm | ε ₁ If it is greater than the threshold value, the pixel value Xn is selected and sent to the subsequent amplifier 161.
[0056]
The selectors 142 to 144 are configured in the same manner as the selector 141, and send the pixel value X of the selected pixel p to the corresponding subsequent amplifiers 162, 164, and 165, respectively.
[0057]
The configurations of the selectors 341 to 344 in FIG. 5 and the selectors 441 to 444 in FIG. 6 are also the same as those of the selector 141, and thus description thereof is omitted.
[0058]
Next, the nonlinear smoothing process in the nonlinear smoother 22 of FIG. 3 will be described with reference to the flowchart of FIG. This flowchart explains in detail the processing of step S2 and step S3 of FIG. This process is started when input image data is supplied from the image sensor 11 to the linear low-pass filter 51 of the nonlinear smoother 22 in accordance with a user instruction.
[0059]
In step S <b> 31, the linear low-pass filter 51 acquires input image data supplied from the image sensor 11.
[0060]
In step S <b> 32, the linear low-pass filter 51 attenuates the signal level of an extremely high frequency component in the input image data, and transmits the image data obtained as a result to the lookup table 52.
[0061]
As described above, the linear low-pass filter 51 attenuates the signal level of an extremely high frequency component in the input image data, thereby performing the smoothing process in the subsequent ε filter 53 (ε filters 53A to 53C). Large high-frequency components are prevented from remaining on the image as dot noise without being sufficiently smoothed. The linear low-pass filter 51 is configured by applying a one-dimensional linear low-pass filter in the horizontal direction and the vertical direction of the image, or is configured by a two-dimensional linear low-pass filter.
[0062]
In step S33, the lookup table 52 acquires the image data supplied from the linear low-pass filter 51 by the process of step S32, and performs gradation conversion such as logarithmic conversion on the acquired image data, The image data obtained as a result is sent to the ε filter 53 (ε filter 53A).
[0063]
By the way, the downstream ε filters 53A to 53C have a threshold ε. ₁ To ε _Three However, the amplitude of the image data is generally the intensity of the illumination light that illuminates the object to be imaged. It increases in proportion to the height. Therefore, the look-up table 52 performs logarithmic transformation on the image data in advance, so that the same smoothing effect can be obtained regardless of the illumination conditions when performing the smoothing process in the subsequent ε filters 53A to 53C. To do. At the same time, the look-up table 52 controls the smoothing effect according to the pixel value, for example, by increasing the smoothing effect in a dark region or a bright region of the image, thereby increasing the degree of enhancement by the subsequent image enhancement processing. enable.
[0064]
In step S34, the ε filter 53 (ε filters 53A to 53C) executes a smoothing process. Details of the smoothing process will be described later with reference to FIG.
[0065]
In step S35, the look-up table 54 performs gradation conversion on the smoothed image data supplied from the ε filter 53 (ε filter 53C) (the inverse of the logarithmic conversion performed by the look-up table 52 by the processing in step S33). And the smoothed image data obtained as a result is sent to the linear low-pass filter 55.
[0066]
In step S36, the linear low-pass filter 55 slightly dulls the edges of the smoothed image data (smoothed image data supplied from the look-up table 54 by the processing in step S35) that has been smoothed while preserving the edges. To generate smoothed image data, which is sent to a subtractor 31 and an adder 33 (FIG. 1). Thereby, the smoothness of the image in the vicinity of the edge can be maintained in the image enhancement processing by the calculation unit 23 in the subsequent stage.
[0067]
The smoothed data subjected to the nonlinear smoothing process by the process of FIG. 8 is further subjected to the processes in steps S4 to S7 of FIG. 2 as described above.
[0068]
FIG. 9 is a flowchart for explaining the smoothing process in step S34 of FIG. This process is started when image data is supplied to the first-stage ε filter of the ε filter 53 (that is, in this case, the ε filter 53A) (after the process of step S33 in FIG. 8). The
[0069]
In step S71, the ε filter 53A determines that the first threshold value ε ₁ The first-stage smoothing process is executed according to the first interval. Details of this processing will be described later with reference to FIG. The ε filter 53A supplies the smoothed data subjected to the first smoothing process to the ε filter 53B at the subsequent stage.
[0070]
In step S72, the ε filter 53B determines that the second threshold value ε ₂ The second-stage smoothing process is executed by the second interval. The ε filter 53B supplies the smoothed data subjected to the second smoothing process to the subsequent ε filter 53C.
[0071]
In step S73, the ε filter 53C determines that the third threshold value ε ₂ The third-stage smoothing process is executed by the third interval. The ε filter 53C supplies the smoothed data subjected to the third smoothing process to the look-up table 54 as the output of the ε filter 53. As described above, the look-up table 54 performs gradation conversion on the smoothed data (processing in step S35 in FIG. 8).
[0072]
With the processing of FIG. 9, smoothing processing is performed by the ε filters 53A to 53C included in the ε filter 53, and smoothed data is generated.
[0073]
Specifically, the {1, 0, 0, 0, 2, 0, 0, 0, 2, 0, 0, 0, 2, 0, 0, The tap coefficient of 0,1} is calculated, the tap coefficient of {1,0,2,0,2,0,2,0,1} is calculated by the ε filter 53B, and the {1, 0,2} is calculated by the ε filter 53C. Calculation of tap coefficients of 2, 2, 2, 1} is performed. As a result, the input image data is substantially added to {1, 2, 3, 4, 6, 8, 10, 12, 13, 14, 15, 16, 16, 16, 16, 16, 15, 14 , 13, 12, 10, 8, 6, 4, 3, 2, 1} is equivalent to the calculation of the tap coefficients.
[0074]
That is, by cascading three ε filters (17-tap, 9-tap, and 5-tap ε filters), it is possible to obtain almost the same effect as the result of calculation using a 27-tap nonlinear filter. At this time, the comparison operation can be reduced from 26 times to 12 times, and the multiplication operation can be reduced from 27 times to 15 times, compared to the case where the operation is performed using a 27-tap non-linear filter.
[0075]
In this smoothing process (the process of step S34 in FIG. 8), it is desirable to generate an image with as little change as possible in the portion other than the edge. For this purpose, a very large filter is used in the ε filter 53A. There is a need. However, when the filter to be used is increased, if all the pixels are multiplied by the tap coefficient ak, the number of selectors 141 to 144 and amplifiers 161 to 165 in the subsequent stage must be increased, which increases the circuit scale.
[0076]
Therefore, in this embodiment, each of the register trains (registers 101 to 104, registers 105 to 108, registers 109 to 112, and registers 113 to 116) is composed of registers for four pixels, and the subsequent arithmetic processing is performed. On the other hand, by applying to only the first pixel (register 104, register 108, register 112, and register 116) of each register row, spatially without increasing the circuit scale of the subsequent stage. An ε filter covering a wide range is realized. However, a filter having a significant coefficient only every few pixels can be expected to have a large smoothing effect, but generally has a tendency to increase side lobes, and unnecessary high-frequency components remain in the smoothed image data. It becomes easy to do.
[0077]
Therefore, by connecting the ε filter 53B and the ε filter 53C in cascade (cascade), both the smoothing effect and the sidelobe suppression are achieved.
[0078]
FIG. 10 is a flowchart for explaining the processing of steps S71 to S73 in FIG. 9, that is, the smoothing processing (smoothing processing in an arbitrary ε filter) of the n-th stage (n = 1, 2, or 3). It is. This process is started when input image data is supplied to an arbitrary ε filter 53 that executes the process.
[0079]
In step S101, each register holds data. For example, when this process is executed in the ε filter 53A shown in FIG. 4, the registers 101 to 116 each hold the input data. At this time, the held data is supplied to the selectors 141 to 144 as the pixel value Xm of the pixel pm. In other words, the pixel values Xm held in the registers 104, 108, 112, and 116 at that time are supplied to the selectors 141 to 144, respectively. Further, the pixel value Xn of the central pixel pn supplied from the register 108 is input to the selectors 141 to 144 and the multiplier 163.
[0080]
In step S102, the subtractor 200 of the selectors 141 to 144 calculates a difference (Xn−Xm) between the pixel value Xm of the pixel pm and the pixel value Xn of the center pixel pn.
[0081]
In step S103, the absolute value converter 202 of the selectors 141 to 144 obtains an absolute value | Xn−Xm | of the difference (Xn−Xm) between the difference pixel value Xn and the pixel value Xm calculated by the processing in step S102. Is sent to the large / small comparator 203.
[0082]
In step S104, the magnitude comparator 203 of the selectors 141 to 144 determines the absolute value | Xn−Xm | and a predetermined threshold value ε (threshold value ε in this example). ₁ ) And the comparison result is sent to the selector 201.
[0083]
In step S105, when the absolute value | Xn−Xm | is smaller than the predetermined threshold ε, the selector 201 of the selectors 141 to 144 selects the pixel value Xm and sends it to the amplifier 161 at the subsequent stage. Xn−Xm | is a predetermined threshold value ε ₁ If it is greater than the value, the pixel value Xn is selected and sent to the amplifier 161. At this time, the pixel value supplied to the amplifier 161 is set to the pixel value X.
[0084]
In step S106, the amplifiers 161 to 165 multiply the input pixel value X by the tap coefficient ak and send the calculation result to the adder 181. That is, the amplifiers 161 to 165 weight the pixel value X. In the present example, since the tap coefficients of the amplifiers 161 and 165 are 1 and the tap coefficients of the amplifiers 162 to 164 are 2, the pixel value X is multiplied by 1 or 2.
[0085]
In step S <b> 107, the adder 181 adds the values supplied from the amplifiers 161 to 165, and supplies this to the division unit 182.
[0086]
In step S108, the division unit 182 divides the value supplied from the adder 181 (divides by 8 in the present example), and sends this to the ε filter 53B. Then, the processing returns to FIG. 9, and the processing after that (in this case, step S72) is performed.
[0087]
10, the ε filter 53 (ε filters 53A to 53C), which is a non-linear smoothing filter, smoothes the pixel value without impairing a sharp change in the pixel value, and the image supplied from the lookup table 52 The image data can be smoothed while the edges of the data (image data supplied from the lookup table 52 by the process of step S32 in FIG. 8) are preserved. Summarizing the smoothing process (process of FIG. 10) in the ε filter 53, when the filter processing pixel is one-dimensional and 2N + 1 taps, it is expressed by Expression (1).
[Expression 1]

[0088]
That is, the ε filter 53 compares the absolute value | Xn−Xn−k | of the difference between the pixel value Xn of the center pixel pn and the pixel value Xn−k of the pixel pn−k with a predetermined threshold ε. As a result, when the ε filter 53A determines that the absolute value | Xn−Xn−k | is smaller than the predetermined threshold value ε, the pixel value Xn−k is substituted for wn−k, and ak is used as each tap coefficient. By executing the same processing as that of the normal linear low-pass filter, the image is uniformly smoothed around the center pixel pn.
[0089]
On the other hand, if the ε filter 53 determines that the absolute value | Xn−Xn−k | is larger than the predetermined threshold value ε, the ε filter 53 substitutes the pixel value Xn for wn−k, and the pixel pn−k By replacing the pixel value Xn-k with the pixel value Xn of the center pixel pn and performing low-pass filter processing around the center pixel pn, the pixel value Xn-k is ignored and only the pixel value near the pixel value Xn is used. Perform smoothing.
[0090]
FIG. 11 is a diagram illustrating pixel values held in the registers 101 to 116 of the ε filter 53A (17-tap ε filter).
[0091]
In FIG. 11, the pixel value of the filter center pixel is C, and the pixel values on the left side of the center pixel are X1, X2, X3, L1, X4, X5, X6, and L2. The pixel values on the right side of the center pixel are Y1, Y2, Y3, R1, Y4, Y5, Y6, and R2.
[0092]
Usually, since the line is scanned from the upper left to the lower right, the leftmost pixel value L2 in the figure is scanned first and the rightmost pixel value R2 is scanned last, so that the pixel value L2 Is held in the register 116, the pixel value X6 is held in the register 115, and the pixel value X5 is held in the register 114. Hereinafter, sequentially from the pixel values on the left side in FIG. 11 are held in the registers 113, 112, 111,..., The pixel value Y6 is held in the register 101, and the pixel value R2 is directly supplied to the selector 144 ( A register for latching the pixel value R2 may be provided).
[0093]
As described above, the tap coefficient of the ε filter 53A is {1, 0, 0, 0, 2, 0, 0, 0, 2, 0, 0, 0, 2, 0, 0, 0, 1}. The coefficients of L2 and R2 are 1, the coefficients of L1, C1, and R1 are 2, X1 to X6 (retained in registers 115, 114, 113, 111, 110, and 109, respectively), and Y1 to The coefficient of Y6 (retained in the registers 107, 106, 105, 103, 102, and 101, respectively) is set to 0 (the output of each register is not supplied to the adder 181).
[0094]
When the above-described equation (1) is applied, for example, the absolute value | C−L1 | of the difference between the pixel value C and the pixel value L1 is a predetermined threshold value ε. ₁ And the predetermined threshold ε ₁ Is greater than the pixel value L1, the pixel value L1 is replaced with the pixel value C. The pixel values obtained as a result of calculation for each of R2, Y6, Y5, Y4, R1, Y3, Y2, Y1, X1, X2, X3, L1, X4, X5, X6, and L2 according to Expression (1) R2 ′, Y6 ′, Y5 ′, Y4 ′, R1 ′, Y3 ′, Y2 ′, Y1 ′, X1 ′, X2 ′, X3 ′, L1 ′, X4 ′, X5 ′, X6 ′, and L2 ′ The pixel value C ′ output from the ε filter 53A is obtained by calculating Expression (2) (Expression (2) is an expression that summarizes the processing of Step S106 to Step S108 of FIG. 10). ).
[Expression 2]

[0095]
FIG. 12 is a diagram for explaining the relationship of the threshold value ε in the ε filter 53A to the ε filter 53C.
[0096]
The threshold ε in the ε filter 53A to the ε filter 53C increases as the ε filter 53 in the subsequent stage increases. That is, in this case, the threshold ε ₁ <Ε ₂ <Ε _Three It is said.
[0097]
The larger the threshold value ε, the easier it is to create a nuisance (that is, noise can be taken because blur occurs), and the smaller the value ε, the less likely the image is blurred (that is, the blur hardly occurs and noise remains).
[0098]
For example, the number of taps is 3, the tap coefficients are {1, 2, 1}, L1, C, and R1 are L1 = 30, C = 100, and R1 = 80, respectively, and the threshold ε is 50. In this case, L1 ′ = 100 according to the expression (1) (R1 ′ is set to the value (R1) as it is). Further, C ′ = {1 × 100 + 2 × 100 + 1 × 80} / 4 is calculated from the expression (2), and the output pixel C ′ = 95. On the other hand, for example, when the number of taps is 3, the tap coefficients are {1, 2, 1}, L1 = 30, C = 100, R1 = 80, and the threshold ε is 80 (that is, the above-described case). In the case where only the value of the threshold value ε is different from the above example), L1 ′ and R1 ′ are set to the values (L1, R1) as they are from the equation (1). Further, C ′ = {1 × 30 + 2 × 100 + 1 × 80} / 4 is calculated by the expression (2), and the output pixel C ′ = 77.5 is obtained. Thus, the larger the threshold value ε (in the present example, the more the threshold value ε is 50 than 50), the more the output image is blurred (the image is blurred).
[0099]
13 is a diagram for explaining pixel values held in the registers 301 to 308 of the ε filter 53B (9-tap ε filter), and FIG. 14 illustrates pixels held in the registers 301 to 308 of the ε filter 53C. It is a figure explaining a value.
[0100]
In FIG. 13, the pixel value of the filter center pixel is C, and the pixel values on the left side of the center pixel are X1, L1, X2, and L2. The pixel values on the right side of the center pixel are Y1, R1, Y2, and R2. As described above, since the tap coefficient of the ε filter 53B is {1, 0, 2, 0, 2, 0, 2, 0, 1}, the coefficients of L2 and R2 are set to 1, and L1, C1, and R1 The coefficient is 2, and the coefficients of X1, X2, Y1, and Y2 (retained in registers 307, 305, 303, and 301, respectively) are 0.
[0101]
Further, in the ε filter 53C of FIG. 14, the pixel value of the filter center pixel is C, and the pixel values on the left side of the center pixel are L1 and L2. Further, the pixel values on the right side of the center pixel are R1, and R2. As described above, since the tap coefficient of the ε filter 53C is {1, 2, 2, 2, 1}, the coefficients of L2 and R2 are set to 1, and the coefficients of L1, C1, and R1 are set to 2.
[0102]
Thus, the interval of the ε filter 53A is 4 (= 2 ² ) (The number of registers for setting the interval is three), and the interval of the ε filter 53B is 2 (= 2 ¹ ) (The number of registers for setting the interval is one), and the interval of the ε filter 53C is 1 (= 2) ⁰ (The number of registers for setting the interval is 0).
[0103]
Here, the above-described expressions (1) and (2) are applied to the respective pixel values to obtain the output pixel value C ′. At this time, the output pixel value C ′ calculated by the ε filter 53B has more noise than the ε filter 53A having a large number of taps. Further, the output pixel value C ′ calculated by the ε filter 53C is more noisy than the ε filter 53B having a large number of taps. That is, the magnitude of noise is small in the ε filter 53A having a large number of taps, and the noise increases as the ε filter 53C in the subsequent stage (the number of taps is small).
[0104]
Therefore, as shown in FIG. 12, the threshold value ε of the ε filter 53 is changed depending on the number of taps of the ε filter (in the ε filter 53A having a large number of taps, the threshold value ε ₁ In the ε filter 53C having a small number of taps, the threshold ε _Three ), The noise of all the output pixel values C ′ of the ε filter 53A to the ε filter 53C can be suppressed.
[0105]
FIG. 15 is a diagram for explaining image data to be smoothed.
[0106]
The input image data on the left side in the figure is smoothed by the ε filter 53 to be converted into image data as shown on the right side in the figure.
[0107]
That is, when the absolute value of the difference between the pixel values before and after the steep edge exceeds the predetermined threshold value ε, the center pixel pn is the center, and the pixel value Xm of the pixel pm is set to the center pixel pn when performing low-pass filter processing. By performing low-pass filter processing instead of the pixel value Xn, smoothing is performed in the vicinity of the pixel value Xn. On the other hand, when performing low-pass filter processing around the pixel pm, smoothing is performed in the vicinity of the pixel value Xm.
[0108]
At this time, the ε filter 53A outputs the pixel value X of the pixel p of the edge portion almost as it is for the pixel p of the edge portion where the pixel p close to the pixel value X does not exist within the filter processing range. The sharp change of the edge is preserved as it is (formula (1)). Incidentally, as in the case of the linear low-pass filter 51, the ε filter 53A is configured by applying a one-dimensional ε filter in the horizontal direction and the vertical direction of an image, and by a two-dimensional ε filter. There is.
[0109]
FIG. 16 is a diagram illustrating the frequency response in the ε filter 53A to the ε filter 53C.
[0110]
In FIG. 16, the horizontal axis represents the frequency normalized with the Nyquist frequency being 0.5, and the vertical axis represents the filter frequency response. 53A, 53B, and 53C indicate frequency responses corresponding to the ε filter 53A, the ε filter 53B, and the ε filter 53C, respectively.
[0111]
By setting the interval (decimation) of the tap number of the ε filter 53 (for example, ε filter 53A) to a power of 2, as shown in FIG. 16, the waveform of the frequency response of the ε filter 53A (for example, interval = 4) Can be canceled by the frequency response of the ε filter 53B (for example, interval = 2) (the peak point of the side lobe in the frequency response waveform of the ε filter 53A and the frequency response waveform of the ε filter 53B). Since 0 points in the same are the same, they have been canceled). In other words, the side lobe of the previous ε filter 53 can be suppressed by the next ε filter 53, so that both the smoothing effect and the suppression of the side lobe can be achieved.
[0112]
As described above, the smoothed image data smoothed by the ε filter 53 is calculated by the calculation unit 23 and supplied to the camera signal processing circuit 13 as an output of the image processing circuit 12. At this time, each pixel value y (i, j) of the output image data output from the image processing circuit 12 is expressed by Expression (3).
[Equation 3]

[0113]
In Expression (3), the gain coefficient g (i, j) is replaced with a uniform gain coefficient G over the entire screen, and each pixel value s (i, j) of the smoothed image data is replaced with the dynamic range of the input image data. When replaced with the median value or the average value C of all pixel values, the equation (3) is expressed as an equation (4) indicating a contrast enhancement method by conventional tone curve adjustment.
[Expression 4]

[0114]
Further, in equation (3), the gain coefficient g (i, j) is replaced with a uniform gain coefficient G over the entire screen, and each pixel value s (i, j) of the smoothed image data is replaced with each input image data. When the pixel value x (i, j) is replaced with each pixel value f (i, j) of the image data obtained by performing the linear low-pass filter processing, this equation (3) is a high-frequency component enhancement method using a conventional unsharp mask. It is expressed as shown in equation (5).
[Equation 5]

[0115]
As described above, the conventional contrast enhancement method is a method in which image enhancement processing is performed independently for each pixel of the image, and the conventional high-frequency component enhancement method is performed with the surrounding pixels existing around the center pixel. In contrast to the method of performing image enhancement processing based on the relative level difference, the method according to the present embodiment integrates the conventional contrast enhancement method and high-frequency component enhancement method using a nonlinear smoothing filter. This enables higher-quality image enhancement processing.
[0116]
In addition, a method using a conventional non-linear smoothing filter (amplifies the part other than the edge by amplifying the part other than the edge while preserving the edge with a sharp change in pixel value in the input image data, and displays the highlighted part. On the other hand, in the method according to the present embodiment, the interval of the number of taps of the ε filter 53 is set to a power of 2, and the threshold value ε of the ε filter 53 is changed in accordance with the band of the filter. Even in the case where a portion other than the edge is smoothed while the sharply changing edge is preserved, the amount of calculation can be suppressed.
[0117]
As described above, the threshold value ε is set according to the band (in the case of a narrowband filter (ε filter with a large number of taps), the threshold value ε is reduced, and in the case of a wideband filter (an ε filter with a small number of taps), By increasing the threshold ε), cascading the ε filters in the order from the narrowband filter to the wideband filter, and further setting the interval of the ε filter (substantially the pixel extraction interval to be processed) to a power of two, Even in the case where the portion other than the edge is smoothed while the sharp edge of the value is preserved, the calculation amount can be suppressed.
[0118]
Further, it is possible to obtain an output image in which noise is hardly generated and the calculation amount is suppressed.
[0119]
In addition, an image with suppressed side lobes can be output.
[0120]
Further, in the above-described embodiment, the case where the present invention is applied to the video camera 1 has been described. However, the present invention is not limited to this, and various other images such as an electronic still camera, a printer, a display, and a computer are used. The present invention can be widely applied to processing apparatuses. In this case, when correcting the image contrast, the computer can obtain a high-quality contrast-corrected image while maintaining the dynamic range, and when synthesizing the images obtained under different illumination conditions, Only the difference in the components can be corrected, and a natural composite image can be generated.
[0121]
In the above example, the number of ε filters provided in the nonlinear smoother 22 is three (ε filter 53A, ε filter 53B, and ε filter 53C), but any N ε filters 53 are provided. You may do it. Of course, in this case, if the Nth ε filter is the ε filter 53N, the threshold ε of the ε filter 53A ₁ Is the minimum, and the threshold ε of the ε filter 53N connected to the subsequent stage _N Is the maximum (ε ₁ <Ε _N ).
[0122]
In the above example, the ε filter is used. However, not only the ε filter but also a median filter, for example, may be used.
[0123]
In the present specification, the steps describing each flowchart include not only processes performed in time series according to the described order, but also processes executed in parallel or individually even if not necessarily performed in time series. Is also included.
[0124]
【The invention's effect】
As described above, according to the present invention, it is possible to reduce the amount of calculation even when smoothing a portion other than the edge while preserving an edge having a sharp change in pixel value. In particular, according to the present invention, it is possible to output an image in which noise is hardly generated and side lobes are suppressed.
[Brief description of the drawings]
FIG. 1 is a block diagram showing a functional configuration of a video camera to which the present invention is applied.
FIG. 2 is a flowchart illustrating image processing in the image processing circuit of FIG.
FIG. 3 is a block diagram showing a functional configuration of the nonlinear smoother of FIG. 1;
4 is a block diagram illustrating a configuration example of an ε filter in FIG. 2. FIG.
5 is a block diagram showing a configuration example of another ε filter in FIG. 2;
6 is a block diagram showing a configuration example of still another ε filter of FIG. 2. FIG.
7 is a block diagram illustrating a configuration example of a selector in FIG. 4;
8 is a flowchart for explaining nonlinear smoothing processing in the nonlinear smoother of FIG. 2;
FIG. 9 is a flowchart illustrating the process of step S34 of FIG.
FIG. 10 is a flowchart illustrating processing in steps S71 to S73 in FIG.
11 is a diagram for explaining pixel values held in the register of FIG. 4;
12 is a diagram for explaining a relationship of threshold values in each ε filter of FIG. 3; FIG.
13 is a diagram illustrating pixel values held in the register of FIG.
14 is a diagram illustrating pixel values held in the register of FIG. 6. FIG.
FIG. 15 is a diagram illustrating image data to be smoothed.
FIG. 16 is a diagram illustrating a relationship between the number of taps and a frequency response.
[Explanation of symbols]
DESCRIPTION OF SYMBOLS 1 Video camera, 11 Image pick-up element, 12 Image processing circuit, 21 Delay circuit, 22 Nonlinear smoother, 23 Operation part, 31 Subtractor, 32 Multiplier, 33 Adder, 51 Linear low pass filter, 52 Look-up table, 53 ε filter, 54 look-up table, 55 linear low-pass filter, 101 to 116 registers, 141 to 144 selector, 161 to 165 amplifier, 181 adder, 182 divider, 140 threshold setting unit, 200 subtractor, 201 selector, 202 Absolute value converter, 203 Large / small comparator

Claims (7)

In an image processing apparatus that processes input image data,
Smoothing means for smoothing the input image data and generating smoothed image data;
The smoothing means includes a plurality of nonlinear filters connected in series ,
The image processing apparatus according to claim 1 , wherein each of the plurality of nonlinear filters is a filter having a different threshold value for smoothing and a different number of taps by setting different intervals for the taps . The image processing apparatus according to claim 1, wherein the smoothing unit adaptively changes the degree of smoothing based on a difference value between a central pixel and its neighboring pixels and the threshold value. The image processing apparatus according to claim 1, wherein the threshold value of the non-linear filter is larger in a rear stage than in a front stage. The image processing apparatus according to claim 1, wherein the value of the interval of the nonlinear filter is smaller in the latter stage than in the former stage. The image processing apparatus according to claim 4, wherein the interval of the nonlinear filter is a power of two. The nonlinear filter, the image processing apparatus according to claim 1, characterized in that the ε filter. In an image processing method of an image processing apparatus that processes input image data,
A smoothing step of smoothing the input image data and generating smoothed image data;
The processing of the smoothing step is performed by a plurality of nonlinear filters connected in series with different smoothing thresholds and different tap numbers by setting different intervals for the taps. Processing method.

Images