What means Gaussian Kernel?

Question

[JohnScott1]

Hello,

I am quite new in this field and I have a very beginner question. I always read about Gaussian Kernels in the context of Metropolis-Hasting Algorithms. It seems to be an Update Step within this algorithm.

But I do not understand what is meant by an Gaussian Kernel. I only know the term Kernel as an Mapping from x to y. But this makes no sense to me.

Sorry for this trivial question but I really have no clue about.

John

[h.mon]

I think the only thing trivial about your question is it is not directly related to bioinformatics, rather it is a statistics question.

Answer 1

[eric.kern13]

I am answering because I believe none of the other posts address the question completely and with the audience in mind.

In statistics and computer science, "kernel" indeed has many different meanings. In your case, you can replace "every step is given by a Gaussian kernel" with "every step is given by a random draw from a Normal/Gaussian distribution." (There should probably be an accept/reject step next, which you did not mention.) Metropolis-Hastings is used for search or integration, and either way it must explore the space of possible parameter values. For the math to work, they are usually randomized, and the Normal/Gaussian family is useful for that because it is simple and well-studied.

Some of the other meanings of "kernel":

Kernels appear in the context of kernel density estimation, which is a smoother alternative to histogramming. This is similar to the meaning in Ethan's answer--it's a probability distribution.

They also appear, as Matted mentioned, in machine learning, where the kernel function generalizes "parallel-ness" for two vectors in an abstract space. By "generalizes", I mean it adds options while preserving essential properties. Some algorithms can be painlessly adapted by replacing regular parallel-ness, measured by vector dot products, with kernel evaluations.

Matted also mentions differential equations and random diffusion processes. For phenomena that evolve over time, these "kernels" give a complete mathematical account of the inner workings. This is similar to what happens in metropolis-hastings, which is like an artificially constructed diffusion process.

In mathematics they can refer to a "dead zone": the kernel of a function (or matrix) includes everything that gets sent straight to zero by that function (or, when you multiply by the matrix).

In computer science, I don't know what a kernel is, but there is (at least) another meaning, as in the IPython Notebook's warning "Your Python kernel has died."

Kernels also refer to the seeds of corn. :)

Answer 2

[ethan.kaufman]

Gaussian is just another name for the familiar Normal probability distribution. Any probability distribution can be described by a density function (aka PDF), which maps events to probabilities. In this context, the kernel refers to the part(s) of the PDF that is dependent on the variables in the domain (i.e. the events/data), omitting the normalization constant that is required for the PDF to integrate to 1. If you've ever seen the normal density function, the kernel is just the exponential portion. PDFs are a foundational concept in statistics, but often only the kernel is required for computation (for example, building and maximizing the likelihood function).

To put this answer in context with the others: kernels/PDFs are naturally required for any kind of statistical modelling, two examples of which are mentioned in the answers by John and matted. Those answers, however, don't describe what a kernel is or where it comes from.

[JohnScott1]

Thank you for all the quick answers. In deed my question is most related to ethan's post. I have to implement an metroplis-hastings algorithm. Here the move of the parameter in every step is given by an 'Gaussian kernel'. More precisely I have to add an small value to the parameter p: p^new=p^old+epsilon. And everything I know (or not) is that this is based on this mysterious 'Gaussian kernel'. Originally I am from the field of image processing and therefore I have no clue and I am not able to find the right answer because it seems like 'kernel' is a word for everything and nothing.

I am REALLY confused.

[ethan.kaufman]

As h.mon pointed out, this is the wrong forum to post this question. Try stats.exchange instead, or (assuming this is for a course assignment) ask your professor or others in your class.

Answer 3

[matted]

It depends on the context, but in machine learning it's also known as the radial basis function kernel that can be used in methods that apply the kernel trick. They are also common in differential equations and stochastic processes, for instance various models of diffusion (https://en.wikipedia.org/wiki/Gaussian_kernel).