High dimensional model representation constructed by support vector regression. I. Independent variables with known probability distributions

High dimensional model representation (HDMR) is a general set of quantitative model assessment and analysis tools for capturing high dimensional input-output system behavior. In practice, the HDMR component functions are each approximated by an appropriate basis function expansion. This procedure often requires many input-output samples which can restrict the treatment of high dimensional systems. In order to address this problem we introduce svr-based HDMR to efficiently and effectively construct the HDMR expansion by support vector regression (SVR) for a function \(f(\mathbf{x})\). In this paper the results for independent variables sampled over known probability distributions are reported. The theoretical foundation of the new approach relies on the kernel used in SVR itself being an HDMR expansion (referred to as the HDMR kernel ), i.e., an ANOVA kernel whose component kernels are mutually orthogonal and all non-constant component kernels have zero expectation. Several HDMR kernels are constructed as illustrations. While preserving the characteristic properties of HDMR, the svr-based HDMR method enables efficient construction of high dimensional models with satisfactory prediction accuracy from a modest number of samples, which also permits accurate computation of the sensitivity indices. A genetic algorithm is employed to optimally determine all the parameters of the component HDMR kernels and in SVR. The svr-based HDMR introduces a new route to advance HDMR algorithms. Two examples are used to illustrate the capability of the method.

1. When the paper was published the authors changed to \(n=4\) with \(\mathbf{a} = (0, 1, 4.5, 99)\).

Support of this work for G. Li is provided by the NSF with account number CHE-1464569 and H. Rabitz was supported by the Templeton Foundation with Account No. 52265.

Authors and Affiliations

1. Department of Chemistry, Princeton University, Princeton, NJ, 08544, USA

   Genyuan Li, Xi Xing & Herschel Rabitz

2. Department of Pharmacology, Robert Wood Johnson Medical School and Division of Cheminformatics, Biomedical Informatics Shared Resource, Cancer Institute of New Jersey at Rutgers University, Piscataway, NJ, 08854, USA

   William Welsh

Corresponding author

Correspondence to Herschel Rabitz.

Appendices

The material below will summarize the features of the HDMR kernels employed in this paper.

Appendix 1: \(K_0(x_{i_s},x_i)\) is a kernel

The commonly used kernels \(K(x_{i_s},x_i)\) in SVR, like Gaussian rbf, polynomial kernels, etc. do not satisfy the zero expectation condition with respect to the measure \(\mu \) required for defining HDMR component functions:

or using \({\mathrm d}\mu (s)=p_i(s){\mathrm d}s\) and \({\mathrm d}\mu (t)=p_i(t){\mathrm d}t\)

where \(p_i(x_i)\) is the marginal pdf of \(x_i\). To solve this problem, define

For a uniform distribution of \(x_i\), Eq. 61 reduces to

From Eq. 61, it is easy to prove that \(K_0(x_{i_s},x_i)\) satisfies Eq. 60 by direct integration.

If \(K(x_i,x'_i)\) is a kernel, it can be proved that \(K_0(x_i,x'_i)\) is also a kernel satisfying Mercer’s condition [17]

The proof is the following:

Here, the inner product \(\langle *,* \rangle \) is defined with respect to the kernel \(K(x_i,x'_i)\)

and the symmetric property

along with the Cauchy-Schwarz inequality

were used.

Appendix 2: Formulas of some commonly used HDMR kernels

Using Eq. 61 some commonly used HDMR kernels are given below. To save the space, the detailed derivations of these formulas are omitted.

1. 1a.

   Gaussian rbf HDMR kernel with a uniform distribution in [−1,1]

   $$\begin{aligned} K_0(x_i,x'_i)= & {} \exp {\left[ \frac{(x_i-x'_i)^2}{2l_i^2}\right] } \nonumber \\&\quad -\frac{\left[ \mathrm{erf}\left( \frac{1+x_i}{\sqrt{2}l_i}\right) +\mathrm{erf}\left( \frac{1-x_i}{\sqrt{2}l_i}\right) \right] \left[ \mathrm{erf}\left( \frac{1+x'_i}{\sqrt{2}l_i}\right) +\mathrm{erf}\left( \frac{1-x'_i}{\sqrt{2}l_i}\right) \right] }{\frac{4}{\sqrt{\pi }}\frac{\sqrt{2}}{l_i}\, \mathrm{erf}\left( \frac{\sqrt{2}}{l_i}\right) +\frac{4}{\pi }\left( e^{-\frac{2}{l_i^2}}-1\right) }.\nonumber \\ \end{aligned}$$

   (69)
2. 1b.

   Gaussian rbf HDMR kernel with a standard normal distribution in \([-\infty ,\infty ]\)

   $$\begin{aligned} K_0(x_i,x'_i) =\exp {\left[ -\frac{(x_i-x'_i)^2}{2l_i^2}\right] } -\frac{l_i\sqrt{l_i^2+2}}{l_i^2+1} \exp {\left[ -\frac{x_i^2+(x'_i)^2}{2(l_i^2+1)}\right] }. \end{aligned}$$

   (70)
3. 2a.

   Exponential HDMR kernel with a uniform distribution in [−1,1]

   $$\begin{aligned} K_0(x_i,x'_i)= & {} \exp {\left[ -\frac{|x_i-x'_i|}{2l_i^2}\right] } \nonumber \\&\quad -\frac{\left( 2-\exp {\left[ -\frac{(1+x_i)(1-x_i)}{4l_i^4}\right] }\right) \left( 2-\exp {\left[ -\frac{(1+x'_i)(1-x'_i)}{4l_i^4}\right] }\right) }{2\left[ \frac{1}{l_i^2}-\left( 1-e^{-\frac{1}{l_i^2}}\right) \right] }.\nonumber \\ \end{aligned}$$

   (71)
4. 2b.

   Exponential HDMR kernel with a standard normal distribution in \([-\infty ,\infty ]\)

   $$\begin{aligned}&K_0(x_i,x'_i)=\exp {\left[ -\frac{|x_i-x'_i|}{2l_i^2}\right] } -\frac{1}{4\,\mathrm{erfc}(\frac{1}{2l_i^2})} \nonumber \\&\quad \times \left[ \exp \left( -\frac{x_i}{2l_i^2}\right) \mathrm{erfc} \left( \frac{1}{2\sqrt{2}l_i^2}-\frac{x_i}{\sqrt{2}}\right) +\exp \left( \frac{x_i}{2l_i^2}\right) \mathrm{erfc} \left( \frac{1}{2\sqrt{2}l_i^2}+\frac{x_i}{\sqrt{2}}\right) \right] \nonumber \\&\quad \times \left[ \exp \left( -\frac{x'_i}{2l_i^2}\right) \mathrm{erfc} \left( \frac{1}{2\sqrt{2}l_i^2}-\frac{x'_i}{\sqrt{2}}\right) +\exp \left( \frac{x'_i}{2l_i^2}\right) \mathrm{erfc} \left( \frac{1}{2\sqrt{2}l_i^2}+\frac{x'_i}{\sqrt{2}}\right) \right] . \nonumber \\ \end{aligned}$$

   (72)
5. 3a.

   Polynomial HDMR kernel with a uniform distribution in [0,1]

   $$\begin{aligned} K_0(x_i,x'_i) = (x_ix'_i+1)^k - \frac{\left[ \displaystyle {\sum _{j=1}^{k+1}} C_{k+1}^j x_i^{j-1}\right] \left[ \displaystyle {\sum _{j=1}^{k+1}}C_{k+1}^j (x_i')^{j-1}\right] }{(k+1)\displaystyle {\sum _{j=1}^{k+1}}\frac{2^j-1}{j}}, \end{aligned}$$

   (73)

   where

   $$\begin{aligned} C_{k+1}^j=\frac{(k+1)!}{j!(k+1-j)!} \end{aligned}$$

   (74)

   denotes the number of all combinations of \(k+1\) things taken j at a time.

6. 3b.

   Polynomial HDMR kernel with a uniform distribution in [−1,1]

   $$\begin{aligned} K_0(x_i,x'_i)= & {} \,(x_ix'_i+1)^k - \frac{1}{(k+1)\displaystyle {\sum _{j=1}^{\mathrm{Int}((k+1)/2)}} C_{k+1}^{2j-1}/(2j-1)} \times \nonumber \\&\quad \times \left[ \sum _{j=1}^{\mathrm{Int}((k+1)/2)} C_{k+1}^{2j-1} x_i^{2(j-1)} \right] \cdot \left[ \sum _{j=1}^{\mathrm{Int}((k+1)/2)} C_{k+1}^{2j-1}(x'_i)^{2(j-1)} \right] .\nonumber \\ \end{aligned}$$

   (75)
7. 4.

   Fourier HDMR kernel with a uniform distribution in \([-\pi ,\pi ]\)

   $$\begin{aligned} K_0(x_i,x'_i)= & {} \frac{1-q^2}{2(1-2q\cos (x_i-x'_i)+q^2)} - \frac{1}{2}\nonumber \\= & {} \frac{q\cos (x_i-x'_i)-q^2}{1-2q\cos (x_i-x'_i)+q^2}. \end{aligned}$$

   (76)

Complex and hybrid HDMR kernels

If \(K_{01}(x_i,x'_i)\) and \(K_{02}(x_i,x'_i)\) are two HDMR kernels, then

is another HDMR kernel. Using Eq. 31, we know that \(K_{03}(x_i,x'_i)\) is a kernel. Moreover, it can be also proved that \(K_{03}(x_i,x'_i)\) has zero expectation

Therefore \(K_{03}(x_i,x'_i)\) is an HDMR kernel. Using Eq. 77, various complex and hybrid HDMR kernels can be constructed.

Reprints and permissions