Guanyu’s Homepage

Optimization Part 1 Basics and Least Squares

2026-03-31T00:00:00+00:00

In common engineering problems, the optimization problem is to minimize the residual of a parametric model with respect to some observed data points.

Preliminaries

Given $N$ measurements $\left\{\left(\mathbf{x}_i, \mathbf{y}_i\right)\right\}_{i=1}^N$ where $\mathbf{x}_i\in\mathbb{R}^p$ and $\mathbf{y}_i\in\mathbb{R}^{m_i}$ . We define a prediction model parametrized by $\boldsymbol{\theta}\in\mathbb{R}^n$ as $\mathbf{h}\left(\cdot;\boldsymbol{\theta}\right):\mathbb{R}^p\to\mathbb{R}^{m_i}$ and the residual of a single datapoint as $\mathbf{r}_i:\mathbb{R}^n\to\mathbb{R}^{m_i}$ , where $\mathbf{r}_i\left(\boldsymbol{\theta}\right)\coloneqq \mathbf{h}\left(\mathbf{x}_i;\boldsymbol{\theta}\right)-\mathbf{y}_i$ . The full residual $\mathbf{r}:\mathbb{R}^{n}\to\mathbb{R}^m$ is simply a stack of every single residuals, i.e.,

\begin{align*} \mathbf{r}\left(\boldsymbol{\theta}\right)= \begin{bmatrix} \mathbf{r}_1\left(\boldsymbol{\theta}\right)\\ \mathbf{r}_2\left(\boldsymbol{\theta}\right)\\ \vdots \\ \mathbf{r}_N\left(\boldsymbol{\theta}\right)\\ \end{bmatrix} \in\mathbb{R}^m, && m=\sum_{i=1}^{N}m_i \end{align*}

Note that we require 𝑚≥𝑛 because otherwise the prediction model cannot be fully defined, i.e., there are infinitely many 𝜃 that perfectly fit the data.

The goal of an optimization problem is to find a model that best fits the data. To formalize this, we need to define some function $f:\mathbb{R}^n\to\mathbb{R}_{\geq 0}$ that properly maps the residual vector to a minimizable scalar, which is known as the objective function. In general, the function $f$ is assumed to be continuously differentiable. Some of the common objective functions are

Least Squares

f(\boldsymbol{\theta}) = \frac{1}{2}\|\mathbf{r}(\boldsymbol{\theta})\|^2 = \frac{1}{2}\sum_{i=1}^{N}\|\mathbf{r}_i(\boldsymbol{\theta})\|^2

Least Squares with $\ell_1$ regularization

f(\boldsymbol{\theta}) = \frac{1}{2}\|\mathbf{r}(\boldsymbol{\theta})\|^2 + \lambda\|\boldsymbol{\theta}\|_1 = \frac{1}{2}\sum_{i=1}^{N}\|\mathbf{r}_i(\boldsymbol{\theta})\|^2 + \lambda\sum_{j=1}^{n}|\theta_j|

Least Squares with $\ell_2$ regularization

f(\boldsymbol{\theta}) = \frac{1}{2}\|\mathbf{r}(\boldsymbol{\theta})\|^2 + \frac{\lambda}{2}\|\boldsymbol{\theta}\|^2= \frac{1}{2}\sum_{i=1}^{N}\|\mathbf{r}_i(\boldsymbol{\theta})\|^2 + \frac{\lambda}{2}\sum_{j=1}^{n}\theta_j^2

Weighted Least Squares

f(\boldsymbol{\theta}) = \frac{1}{2}\mathbf{r}(\boldsymbol{\theta})^\top \mathbf{W}\mathbf{r}(\boldsymbol{\theta}) = \frac{1}{2}\sum_{i=1}^{N} \mathbf{r}_i(\boldsymbol{\theta})^\top w_i\mathbf{r}_i(\boldsymbol{\theta})

Cauchy Loss

\begin{align*} f(\boldsymbol{\theta}) = \sum_{i=1}^{N} \rho\!\left(\|\mathbf{r}_i (\boldsymbol{\theta})\|\right), && \rho(r) = \frac{\alpha^2}{2}\log\!\left(1 + \frac{r^2}{\alpha^2}\right) \end{align*}

The following is a list of some of the theorems related to optimality conditions. Suppose $f$ is differentiable, and define the gradient (first-order derivative) and Hessian (second-order derivative) as

\begin{align*} \nabla f(\boldsymbol{\theta}) \coloneqq \begin{bmatrix} \dfrac{\partial f}{\partial \theta_1} \\[10pt] \vdots \\[6pt] \dfrac{\partial f}{\partial \theta_n} \end{bmatrix}\in\mathbb{R}^n &&\text{and}&& \mathbf{H}(\boldsymbol{\theta}) \coloneqq \begin{bmatrix} \dfrac{\partial^2 f}{\partial \theta_1^2} & \dfrac{\partial^2 f}{\partial \theta_1 \partial \theta_2} & \cdots & \dfrac{\partial^2 f}{\partial \theta_1 \partial \theta_n} \\[12pt] \dfrac{\partial^2 f}{\partial \theta_2 \partial \theta_1} & \dfrac{\partial^2 f}{\partial \theta_2^2} & \cdots & \dfrac{\partial^2 f}{\partial \theta_2 \partial \theta_n} \\[12pt] \vdots & \vdots & \ddots & \vdots \\[6pt] \dfrac{\partial^2 f}{\partial \theta_n \partial \theta_1} & \dfrac{\partial^2 f}{\partial \theta_n \partial \theta_2} & \cdots & \dfrac{\partial^2 f}{\partial \theta_n^2} \end{bmatrix}\in\mathbb{S}^n \end{align*}

Descent direction: If there is a vector $\mathbf{d}\in\mathbb{R}^n$ such that $\nabla f(\boldsymbol{\theta}_0)^\top\mathbf{d}<0$ , then for all sufficiently small $\lambda>0$ , $f(\boldsymbol{\theta}_0+\lambda\mathbf{d})$ . Here, $\mathbf{d}$ is a descent direction of $f$ at $\boldsymbol{\theta}_0$ .
Intuitively, the dot product less than 0 means that vector $\mathbf{d}$ is in the opposite direction as the gradient.

First-order necessary condition: If $\boldsymbol{\theta}^*$ is a local minimum, then $\nabla f(\boldsymbol{\theta}^*)=0$ .

Second-order necessary condition: If $\boldsymbol{\theta}^*$ is a local minimum, then $\nabla f(\boldsymbol{\theta}^*)=0$ and $\mathbf{H}(\boldsymbol{\theta}^*)\succeq 0$ (positive semi-definite).

Second-order sufficient condition: If $\nabla f(\boldsymbol{\theta}^*)=0$ and $\mathbf{H}(\boldsymbol{\theta}^*)\succ 0$ (positive definite), then $\boldsymbol{\theta}^*$ is a local minimum.

Convex function: $f$ is convex if and only if $\mathbf{H}(\boldsymbol{\theta})\succeq 0$ for all $\boldsymbol{\theta}$ .

Global minimum: Suppose $f$ is convex, then $\boldsymbol{\theta}^\star$ is a global minimum if an only if $\nabla f(\boldsymbol{\theta}^\star)=0$ .

Linear Least Squares

Let $\mathbf{A}\in\mathbb{R}^{m\times n}$ be a matrix and $\mathbf{b}\in\mathbb{R}^m$ be a vector, where $m>n$ and $\mathrm{rank}(\mathbf{A})=n$ . The Linear Least Squares (LLS) optimization problem is to find the optimal solution $\mathbf{x}\in\mathbb{R}^n$ to the linear system of equations $\mathbf{A}\mathbf{x}=\mathbf{b}$ . Define the objective function as

f(\mathbf{x})=\frac{1}{2}\left\|\mathbf{Ax-b}\right\|^2

and the goal is to find $\mathbf{x}^{\star}=\argmin_{\mathbf{x}\in\mathbb{R}^n}f(\mathbf{x})$ .

First, we can write out the gradient and Hessian matrix of $f$ , which are

\begin{align*} \nabla f(\mathbf{x})=\mathbf{A^\top A}\mathbf{x}-\mathbf{A^\top b} && \mathbf{H}(\mathbf{x})=\mathbf{A^\top A} \end{align*}

Derivation

Occupancy Grid Mapping

2026-03-22T00:00:00+00:00

Besides state estimation or localization, which provide a robot with knowledge of where it is, it’s equally important for a mobile robot to perceive its surrounding and form a knowledge of where is occupied, that’s where mapping comes in. This note covers a fundamental mapping algorithm, occupancy grid mapping.

Problem Statement

In occupancy grid mapping, a map $\mathbf{m}=\left\{m_i\right\}_{i=1}^N$ is discretized into $N$ different cells, where each cell $m_i$ is a binary random variable that models the occupancy, and $p(m_i)$ is the probability of a cell being occupied. The map is assumed to be static. The goal is to find the belief $\mathrm{bel}_t(\mathbf{m})=p(\mathbf{m}\mid \mathbf{z}_{1:t}, \mathbf{x}_{1:t})$ , where $\mathbf{z}$ is the observation and $\mathbf{x}$ is the state of the robot.

Two assumptions are made for tractability

The robot state $\mathbf{x}_t$ are known.

The occupancy of individual cells $m_i$ are independent, and thus

\begin{equation} \mathrm{bel}_t(\mathbf{m})=p(\mathbf{m}\mid \mathbf{z}_{1:t}, \mathbf{x}_{1:t})=\prod_{i=1}^{N}p(m_i\mid \mathbf{z}_{1:t}, \mathbf{x}_{1:t}) \end{equation}

At each timestep $t$ , the sensor performs some measurement and provide the information of whether a cell $m_i$ is believed to be occupied or not, in terms of $p(m_i\mid\mathbf{z}_t, \mathbf{x}_t)$ , which is known as the inverse sensor model.

Inverse sensor model of a laser beam range sensor, image taken from [1]

One simple inverse sensor model is shown in the figure above. Consider a laser beam range sensor that provides the reading $\mathbf{z}=(z_1^r,\dots, z_n^r,z_{1}^{\phi},\dots,z_{n}^{\phi})$ , where $z_k^r$ and $z_k^\phi$ are the range and bearing of the $k$ -th beam. For grid $i$ , let $r_i$ and $\phi_i$ be the range and bearing of the grid as observed from the current robot pose $\mathbf{x}$ , and $k=\argmin_j\left|\phi_i-z_j^{\phi}\right|$ be the beam going through that gird. Based on the relation of a grid’s position relative to the robot, we have three different cases:

If $r_i>\min(z_{\max},z_k^r+w_{\mathrm{grid}})$ or $\left|\phi_i-z_{k}^{\phi}\right|>w_{\mathrm{beam}} / 2$ , then return $p_{\mathrm{previous}}$ , the sensor has no information about this grid.

Else if $z_k^r < z_{\max}$ and $\left|r_i-z_k^r\right|$ , then return $p_{\mathrm{occupied}}$ , the sensor think the grid is occupied.

Else if $z_k^r$ and $r_i < z_k^r$ , then return $p_{\mathrm{free}}$ , the sensor think the grid is free.

where $z_{\max}$ is a sensor-specific parameter representing the maximum detectable range, $w_{\mathrm{grid}}$ is the grid size and $w_{\mathrm{beam}}$ is the width of each beam.

Mapping Algorithm

Following the Bayes filtering framework, a general algorithm can be derived for the occupancy grid mapping [2]. Starting from equation 1,

\begin{align*} p(m_i\mid \mathbf{z}_{1:t}, \mathbf{x}_{1:t})&=\frac{p(\mathbf{z}_t\mid m_i, \mathbf{z}_{1:t-1},\mathbf{x}_{1:t})p(m_i\mid\mathbf{z}_{1:t-1}, x_{1:t})}{p(\mathbf{z}_t\mid\mathbf{z}_{1:t-1},\mathbf{x}_{1:t})} && \text{(Bayes rule)}\\ &=\frac{p(\mathbf{z}_t\mid m_i, \mathbf{x}_t)p(m_i\mid\mathbf{z}_{1:t-1}, \mathbf{x}_{1:t-1})}{p(\mathbf{z}_t\mid\mathbf{z}_{1:t-1},\mathbf{x}_{1:t})} && \text{(Markov assumption)}\\ &=\frac{p(m_i \mid \mathbf{z}_t, \mathbf{x}_t) \, p(\mathbf{z}_t \mid \mathbf{x}_t) \, p(m_i \mid \mathbf{z}_{1:t-1}, \mathbf{x}_{1:t-1})}{p(m_i \mid \mathbf{x}_t) \, p(\mathbf{z}_t \mid \mathbf{z}_{1:t-1}, \mathbf{x}_{1:t})} && \text{(Bayes rule)} \end{align*}

Further notice that the map $\mathbf{m}$ is independent of the robot state $\mathbf{x}_t$ , we obtain

\begin{equation} p(m_i\mid \mathbf{z}_{1:t}, \mathbf{x}_{1:t})=\frac{p(m_i \mid \mathbf{z}_t, \mathbf{x}_t) \, p(\mathbf{z}_t \mid \mathbf{x}_t) \, p(m_i \mid \mathbf{z}_{1:t-1}, \mathbf{x}_{1:t-1})}{p(m_i) \, p(\mathbf{z}_t \mid \mathbf{z}_{1:t-1}, \mathbf{x}_{1:t})} \end{equation}

Similarly, for the negation

\begin{equation} p(\lnot m_i\mid \mathbf{z}_{1:t}, \mathbf{x}_{1:t})= \frac{p(\lnot m_i \mid \mathbf{z}_t, \mathbf{x}_t) \, p(\mathbf{z}_t \mid \mathbf{x}_t) \, p(\lnot m_i \mid \mathbf{z}_{1:t-1}, \mathbf{x}_{1:t-1})}{p(\lnot m_i) \, p(\mathbf{z}_t \mid \mathbf{z}_{1:t-1}, \mathbf{x}_{1:t})} \end{equation}

Compute the ratio of equation (2) and (3), we get

\begin{align*} \frac{p(m_i \mid \mathbf{z}_{1:t}, \mathbf{x}_{1:t})}{1 - p(m_i \mid \mathbf{z}_{1:t}, \mathbf{x}_{1:t})}&= \frac{p(m_i \mid \mathbf{z}_t, \mathbf{x}_t) \, p(m_i \mid \mathbf{z}_{1:t-1}, \mathbf{x}_{1:t-1}) \, p(\neg m_i)}{p(\neg m_i \mid \mathbf{z}_t, \mathbf{x}_t) \, p(\neg m_i \mid \mathbf{z}_{1:t-1}, \mathbf{x}_{1:t-1}) \, p(m_i)}\\ &= \underbrace{\frac{p(m_i \mid \mathbf{z}_t, \mathbf{x}_t)}{1 - p(m_i \mid \mathbf{z}_t, \mathbf{x}_t)}}_{\text{inverse sensor model}} \underbrace{\frac{p(m_i \mid \mathbf{z}_{1:t-1}, \mathbf{x}_{1:t-1})}{1 - p(m_i \mid \mathbf{z}_{1:t-1}, \mathbf{x}_{1:t-1})}}_{\text{recursive term}} \underbrace{\frac{1 - p(m_i)}{p(m_i)}}_{\text{prior}} \end{align*}

Then, change to log odds form by defining

l(x)\triangleq\log\frac{p(x)}{1-p(x)}

and the product chain turns into a sum

l(m_i\mid\mathbf{z}_{1:t}, \mathbf{x}_{1:t})=l(m_i\mid \mathbf{z}_{t},\mathbf{x}_t)+l(m_i\mid\mathbf{z}_{1:t-1}, \mathbf{x}_{1:t-1})-l(m_i)

Bayes Inference using Conjugate Prior

The previous section derived an equation for updating the belief of the map under the Bayes framework, however, it lacks the notion of uncertainty (i.e., how confident we are about the map). With the notion of conjugate prior, a proper selection of the underlying distribution function for $p(m_i)$ can both simplifies the updating rule and provide a closed form formula for uncertainty.

Recall that from Bayes rule

\begin{equation*} \underbrace{p(m_i\mid \mathbf{z}_{1:t}, \mathbf{x}_{1:t})}_{\text{posterior}}\propto\underbrace{p(\mathbf{z}_t\mid m_i, \mathbf{x}_t)}_{\text{likelihood}}\cdot\underbrace{p(m_i\mid\mathbf{z}_{1:t-1}, \mathbf{x}_{1:t-1})}_{\text{prior}} \end{equation*}

Bernoulli Likelihood and Beta Prior

First consider the case where the occupancy of a grid is binary, i.e., $m_i\in\{0, 1\}$ . Let $\theta_i\triangleq p(m_i=1)$ be the unknown occupancy probability. From the inverse sensor model described above, we know that each sensor measurement provide a binary result, denoted as $y_t$ , where

\begin{equation*} y_t=\left\{\begin{matrix} 1, & \text{if the inverse sensor model return }p_{\text{occupied}}\\ 0, & \text{if the inverse sensor model return }p_{\text{free}} \end{matrix}\right. \end{equation*}

Then, the likelihood is modeled as a Bernoulli with parameter $\theta_i$ , i.e.,

p(\mathbf{z}_t\mid m_i, \mathbf{x}_t)\propto p(y_t\mid \theta_i)=\theta_i^{y_t}\left(1-\theta_i\right)^{1-y_t}

Furthermore, if we model the prior as a Beta distribution, i.e.,

p(\theta_i\mid \mathbf{z}_{1:t-1},\mathbf{x}_{1:t-1})=\mathrm{Beta}(\theta_i;\alpha_{i,t-1},\beta_{i,t-1})=\frac{1}{B(\alpha_{i,t-1}, \beta_{i,t-1})}\theta_i^{\alpha_{i,t-1} -1}\left(1-\theta_i\right)^{\beta_{i,t-1} - 1}

where $B(\cdot, \cdot)$ is the beta function.

Then, it follows from the Bayes rule that the posterior is

\begin{align*} p(\theta_i\mid \mathbf{z}_{1:t}, \mathbf{x}_{1:t})&\propto p(y_t\mid \theta_i)\cdot p(\theta_i\mid\mathbf{z}_{1:t-1}, \mathbf{x}_{1:t-1})\\ &\propto\theta_i^{y_t}\left(1-\theta_i\right)^{1-y_t}\cdot \theta_i^{\alpha_{i,t-1} -1}\left(1-\theta_i\right)^{\beta_{i,t-1} - 1}\\ &=\theta_i^{(\alpha_{i,t-1}+y_t)-1}\left(1-\theta_i\right)^{(\beta_{i,t-1}+1-y_t)-1} \end{align*}

which still follows a Beta distribution, $p(\theta_i\mid \mathbf{z}_{1:t},\mathbf{x}_{1:t})=\mathrm{Beta}(\theta_i;\alpha_t, \beta_t)$ , where

\begin{align} \alpha_{i,t}=\alpha_{i,t-1}+y_t &&\text{and}&& \beta_{i,t} = \beta_{i,t-1}+(1-y_t) \end{align}

In this case, we call beta distribution the conjugate prior for the Bernoulli likelihood.

Conjugate Distribution and Conjugate Prior [2]

Lie Theory in Robot Motion

2026-03-10T00:00:00+00:00

This note covers some of the fundamental concepts in Lie group and Lie algebra, and their applications to representing rigid body motion in robotics.

Matrix Lie Groups

A group is a set $G$ equipped with a binary map $\cdot:G\times G\to G$ satisfying the following properties:

Closure: $\forall a,b\in G, a\cdot b\in G$ .

Associativity: $\forall a, b, c\in G,\, (a\cdot b)\cdot c = a\cdot(b\cdot c)$ .

Identity: $\exist e\in G,\,\mathrm{s.t.}\,\forall a\in G, \, a\cdot e=e\cdot a=a$ .

Inverse: $\forall a\in G,\, \exist a^{-1}\in G, \, \mathrm{s.t.}\, a\cdot a^{-1}=a^{-1}\cdot a = e$ .

Here, it’s better to understand the element of a group as a kind of operation or transformation (e.g., a rotation or a rigid body transformation) rather than a number or a vector. These groups turn out to be also differentiable manifolds, which makes them Lie groups.

Special Orthogonal Group, $\mathrm{SO}(n)$

To represent a rotation formally, we need to consider the properties that define a rotation. In general, a rotation should be

Linear, so it can be represented as a matrix $\mathbf{R}$ .

Preserves length and angles, so for any vector $\mathbf{v}$ and $\mathbf{w}$ , $\mathbf{v}\cdot \mathbf{w}=(\mathbf{R}\mathbf{v})\cdot (\mathbf{R}\mathbf{w})$ .
Note, from this property,
$\mathbf{v}^{\top}\mathbf{w}=(\mathbf{R}\mathbf{v})^{\top}(\mathbf{R}\mathbf{w})=\mathbf{v}\mathbf{R}^{\top}\mathbf{R}\mathbf{w}$
which means the matrix $\mathbf{R}$ has to satisfy the following
$\begin{equation} \mathbf{R}^{\top}\mathbf{R}=\mathbf{I} \end{equation}$

Preserves orientation (no mirroring), so $\det(\mathbf{R})>0$ .
Given the previous property
$\det(\mathbf{R}^{\top}\mathbf{R})=\det(\mathbf{R}^{\top})\cdot\det(\mathbf{R})=(\det(\mathbf{R}))^2=1$
we have
$\det(\mathbf{R})=1$

In fact, these properties give rise to a group called the special orthogonal group,

\mathrm{SO}(n)=\{\mathbf{R}\in\mathbb{R}^{n\times n}\mid\mathbf{R}^{\top}\mathbf{R}=\mathbf{I},\det(\mathbf{R})=1\}

which is a group of all rotations about the origin in an $n$ dimensional Euclidean space.

Special Euclidean Group, $\mathrm{SE}(n)$

The general way to represent the motion of a rigid body is a transformation that

Preserves the Euclidean distance, $\left\|f(\mathbf{x})-f(\mathbf{y})\right\|=\left\|\mathbf{x}-\mathbf{y}\right\|$ .

Preserves the orientation.

All such transformation forms a group called the special Euclidean group, $\mathrm{SE}(n)$ .

Intuitively, such transformation can be represented by a function $f:\mathbb{R}^n\to\mathbb{R}^n$ , where

\begin{align*} f(\mathbf{x})=\mathbf{R}\mathbf{x}+\mathbf{p}, && \mathbf{R}\in\mathrm{SO}(n)\,\,\text{and}\,\,\mathbf{p}\in\mathbb{R}^n \end{align*}

However, the function $f$ is not linear and thus cannot be represented by a matrix, which causes inconvenience. The general trick to solve this problem is to augment the dimension by 1. For $\mathbf{x}\in\mathbb{R}^n$ , define $\overline{\mathbf{x}}\in\mathbb{R}^{n+1}$ and $\mathbf{T}\in\mathbb{R}^{(n+1)\times(n+1)}$ such that

\begin{align*} \overline{\mathbf{x}}=\left[\begin{matrix}\mathbf{x}\\1\end{matrix}\right], && \mathbf{T}=\left[ \begin{matrix} \mathbf{R} & \mathbf{p}\\ \mathbf{0}_{1\times n} & 1 \end{matrix} \right], && \mathbf{T}\overline{\mathbf{x}}=\overline{f(\mathbf{x})} \end{align*}

so the transformation becomes matrix vector multiplication.

The special Euclidean group now can be written as

\mathrm{SE}(n)=\left\{\mathbf{T}\in\mathbb{R}^{(n+1)\times(n+1)}\mid\mathbf{T}=\left[ \begin{matrix} \mathbf{R} & \mathbf{p}\\ \mathbf{0}_{1\times n} & 1 \end{matrix} \right], \mathbf{R}\in\mathrm{SO}(n), \,\mathbf{p}\in\mathbb{R}^n \right\}

which represents all poses an object can take in an $n$ dimensional space.

Lie Algebra

A Lie algebra is a vector space $\mathfrak{g}$ together with a binary operation called the Lie bracket $[\cdot,\cdot]:\mathfrak{g}\times\mathfrak{g}\to\mathfrak{g}$ satisfying the following properties

Bilinearity: $[a\mathbf{x}+b\mathbf{y},\mathbf{z}]=a[\mathbf{x},\mathbf{z}]+b[\mathbf{y},z]$ , $[\mathbf{z},a\mathbf{x}+b\mathbf{y}]=a[\mathbf{z},\mathbf{x}]+b[\mathbf{z},\mathbf{y}]$ .

Alternating: $[\mathbf{x},\mathbf{x}]=\mathbf{0}$ .

Jacobi identity: $[\mathbf{x},[\mathbf{y},\mathbf{z}]]+[\mathbf{y},[\mathbf{z},\mathbf{x}]]+[\mathbf{z},[\mathbf{x},\mathbf{y}]]=\mathbf{0}$ .

Importantly, every Lie algebra $\mathfrak{g}$ is associated to a Lie group $G$ , which is the tangent space of the identity of the Lie group, denoted as $\mathfrak{g} = T_{e}G$ , and it completely captures the local structure of the group. In the case of robot motion, we can think of a trajectory that lives in the Lie group while the velocity lives in the Lie algebra.

Exponential Map

Let $G$ be a Lie group and $\mathfrak{g}$ be its Lie algebra, first we need to define a few notations:

A curve in $G$ is defined as a function $\gamma:\mathbb{R}\to G$ that maps each real number $t\in\mathbb{R}$ to an element of the group $G$ .
The derivative of $\gamma$ , denoted as $\dot\gamma$ , evaluated at a specific time $t$ , $\dot\gamma(t)$ , is a vector in the tangent space. Suppose $\gamma (t)=g$ , then $\dot\gamma(t)\in T_gG$ .

Left translation: For a specific element $g\in G$ , define the map $L_g:G\to G, x\mapsto gx$ as the left translation. Example: if $G=\mathrm{SO}(3)$ , and $g=\mathbf{R}$ , then $L_{\mathbf{R}}(\mathbf{Q})=\mathbf{R}\mathbf{Q}$ .

Left-invariant vector field: For a vector $\xi\in\mathfrak{g}$ , the left-invariant vector field $X_{\xi}$ is defined at any point $g\in G$ by $X_{\xi}(g)=(\mathrm{d}L_g)_e(\xi)$ .
Basically, start from a velocity vector $\xi\in\mathfrak{g}$ , which has to stars at the origin $e$ (identity). Then, you map this velocity vector to another vector in the tangent space to a group element $g$ , i.e., $T_g G$ .

Now, suppose you have a curve $\gamma_\xi$ that starts at the identity with velocity $\xi$ , then, for any $t\in\mathbb{R}$ , we have

\frac{\mathrm{d}}{\mathrm{d}t}\gamma_\xi(t)=\dot\gamma_\xi(t)=X_{\xi}(\gamma_\xi(t))

The above equation can be thought of as a “differential equation” whose solution is

\gamma_\xi(t)=\exp(t\xi)

Let $t=1$ , the exponential map can be defined as

\begin{align} \exp:\mathfrak{g}\to G,&& \exp(\xi)=\gamma_\xi(1) \end{align}

Takeaway: The exponential and logarithmic maps bridge the Lie group and its associated Lie algebra. Since Lie group is a “curved manifold” and some operations (e.g., optimizations) do not apply to it, and the Lie algebra is a flat vector space and most operations are well defined, a common paradigm is to lift group elements to Lie algebra using log map, solve the problem in Lie algebra, and bring it back to Lie group using exponential map.

Conjugation and Adjoint Map

Let $G$ be a matrix Lie group with Lie algebra $\mathfrak{g}$ . For a specific element $g\in G$ , the conjugation map is defined as $C_g:G\to G, x\mapsto gxg^{-1}$ .

The derivative of the conjugation map evaluated at the identity, $(\mathrm{d}C_g)_e:\mathfrak{g}\to\mathfrak{g}$ is defined as the adjoint map, denoted as $\mathrm{Ad}_g$ . For any $\xi\in\mathfrak{g}$ , there exist a curve $\gamma_{\xi}(t)\in G$ such that $\xi=\dot\gamma_{\xi}(0)$ and $\gamma_\xi(0)=e$ , then

\mathrm{Ad}_g(\xi)=(\mathrm{d}C_g)_e(\xi)=\left.\frac{\mathrm{d}}{\mathrm{d}t}g\gamma_\xi(t) g^{-1}\right|_{t=0}=g\xi g^{-1}

A conjugation map is a change of basis transformation for an element in the Lie group.

An adjoint map is a change of basis transformation for an element in the Lie algebra.

Angular Velocity and Twist, $\mathfrak{so}(3), \mathfrak{se}(3)$

Let $\mathbf{R}(t)\in\mathrm{SO}(3)$ be a smooth trajectory. By differentiating the orthogonality constraint, i.e., equation (1),

\frac{\mathrm{d}}{\mathrm{d}t}\left(\mathbf{R}^\top\mathbf{R}\right)=\dot{\mathbf{R}}^\top\mathbf{R}+\mathbf{R}^\top\dot{\mathbf{R}}=0

which means,

\begin{equation} \dot{\mathbf{R}}^\top\mathbf{R}=-\mathbf{R}^\top\dot{\mathbf{R}}=-\left(\dot{\mathbf{R}}^\top\mathbf{R}\right)^{\top} \end{equation}

Suppose $\mathbf{R}(0)=I$ , then $\dot{\mathbf{R}}(0)$ would be an element of the Lie algebra by definition, and we have $\dot{\mathbf{R}}(0)^{\top}=-\dot{\mathbf{R}}(0)$ . Therefore, the Lie algebra associated to the 3-dimensional rotation group is

\mathfrak{so}(3)=\left\{\mathbf{\Omega}\in\mathbb{R}^{3\times 3}\mid \mathbf{\Omega}^{\top}=-\mathbf{\Omega}\right\}

which consists of all skew-symmetric matrices. Obviously, a 3 by 3 skew-symmetric matrix should look like

\begin{align*} \mathbf{\Omega}=\left[\begin{matrix} 0 & -\omega_z & \omega_y\\ \omega_z & 0 & -\omega_x\\ -\omega_y& \omega_x & 0 \end{matrix}\right], && \omega_x, \omega_y,\omega_z \in\mathbb{R} \end{align*}

This means we can define a vector $\boldsymbol{\omega}=\left[\omega_x, \omega_y,\omega_z\right]^\top\in\mathbb{R}^3$ to represent the matrix $\mathbf{\Omega}$ , often denoted as $\boldsymbol{\omega}^{\wedge}=\mathbf{\Omega}$ or $\lfloor\boldsymbol{\omega}\rfloor_{\times}=\boldsymbol{\Omega}$

Notice that from equation (3), $\dot{\mathbf{R}}^\top\mathbf{R}$ and $\mathbf{R}^\top\dot{\mathbf{R}}$ are skew-symmetric, which makes them element of $\mathfrak{so}(3)$ . Thus, we can define two vectors $\boldsymbol{\omega}_b, \boldsymbol{\omega}_s\in\mathbb{R}^3$ such that

\begin{align} \boldsymbol{\omega}_b^{\wedge}=\mathbf{R}^{\top}\dot{\mathbf{R}}, && \boldsymbol{\omega}_s^{\wedge}=\dot{\mathbf{R}}\mathbf{R}^{\top} \end{align}

where $\boldsymbol{\omega}_s$ is the spatial angular velocity and $\boldsymbol{\omega}_b$ is the body angular velocity. From equation (4), naturally

\begin{align} \boxed{\dot{\mathbf{R}}=\boldsymbol{R}\boldsymbol{\omega}_b^\wedge} &&\boxed{\dot{\mathbf{R}}=\boldsymbol{\omega}_s^\wedge\mathbf{R}} \end{align}

These two equations are known as the reconstruction equation or the equation of motion. It’s better to interpret the subscript $b$ and $s$ as

If the rotation matrix is multiplied on the left, then the velocity is in the body frame.

If the rotation matrix is multiplied on the right, then the velocity is in the spatial frame.

Similarly, we can derive the Lie algebra of $\mathrm{SE}(3)$ as

\mathfrak{se}(3)=\left\{\mathbf{\Xi}\in\mathbb{R}^{4\times 4}\mid \boldsymbol{\Xi}=\left[\begin{matrix} \boldsymbol{\omega}^\wedge & \boldsymbol{v}\\ \boldsymbol{0} & 0 \end{matrix}\right], \boldsymbol{\omega}^\wedge\in\mathfrak{so}(3), \boldsymbol{v}\in\mathbb{R}^3 \right\}

which is a 6-dimensional vector space, each element of which can be identified with a vector known as the twist, $\boldsymbol{\xi}=[\boldsymbol{\omega},\boldsymbol{v}]\in\mathbb{R}^6$ , often denoted as $\boldsymbol{\xi}^\wedge=\mathbf{\Xi}$ . The reconstruction equation or equation of motion can be written as

\begin{align} \boxed{\dot{\mathbf{ T}}=\mathbf{T}\boldsymbol{\xi}_b^\wedge} && \boxed{\dot{\mathbf{ T}}=\boldsymbol{\xi}_s^\wedge \mathbf{T}} \end{align}

Reference

[1] T. D. Barfoot, State Estimation for Robotics. Cambridge University Press, 2024.
[2] M. Ghaffari, ROB530 Lecture Slides — Matrix Lie Groups, Robot Motion. University of Michigan, 2026.

Bayes Filtering and State Estimation

2026-03-04T00:00:00+00:00

In robot state estimation, the Bayes filter is a probabilistic approach that estimates the state from a sequence of controls and measurements by recursively performing prediction with a motion model and correction with a measurement model. In practice, some common instantiations of the Bayes framework are the Kalman filter, information filter, and particle filter.

Mathematical Formulation of Bayes Filter

Given a sequence of measurements $\mathrm{z}_{1:t}=\mathbf{z}_1,\dots,\mathbf{z}_t$ and controls $\mathbf{u}_{1:t}=\mathbf{u}_1,\dots,\mathbf{u}_t$ , and a measurement model $p(\mathbf{z}_t\mid\mathbf{x}_t)$ and a motion model $p(\mathbf{x}_t\mid\mathbf{x}_{t-1},\mathbf{u}_t)$ . The Bayes filter provides a recursive framework for estimating the state $\mathbf{x}_t$ in terms of a probability distribution function, known as the belief $\mathrm{bel}(\mathbf{x}_t)\triangleq p(\mathbf{x}_t\mid\mathbf{z}_{1:t},\mathbf{u}_{1,t})$ , or posterior.

The dynamic Bayes network that characterizes the evolution of controls, states, and measurements. [1]

The filtering algorithm is built upon two assumptions of conditional independence:

Motion model is (1st-order) Markov:

p(\mathbf{x}_t\mid\mathbf{x}_{0:t-1},\mathbf{u}_{1:t})=p(\mathbf{x}_t\mid\mathbf{x}_{t-1},\mathbf{u}_t)

The measurement model is memoryless:

p(\mathbf{z}_t\mid\mathbf{x}_{0:t},\mathbf{z}_{1:t-1},\mathbf{u}_{1:t})=p(\mathbf{z}_{t}\mid\mathbf{x}_t)

Given these two assumptions, we can propagate the belief over time:

\begin{align*} \mathrm{bel}(\mathbf{x}_t) &=p(\mathbf{x}_t\mid\mathbf{z}_{1:t},\mathbf{u}_{1:t})=p(\mathbf{x}_t\mid\mathbf{z}_t,\mathbf{z}_{1:t-1},\mathbf{u}_{1:t}) && \to\text{definition} \\ &=\eta p(\mathbf{z}_t\mid\mathbf{x}_t,\mathbf{z}_{1:t-1}, \mathbf{u}_{1:t})p(\mathbf{x}_t\mid\mathbf{z}_{1:t-1},\mathbf{u}_{1:t}) && \to\text{Bayes rule}\\ &=\eta p(\mathbf{z}_t\mid\mathbf{x}_t)p(\mathbf{x}_t\mid\mathbf{z}_{1:t-1},\mathbf{u}_{1:t}) && \to \text{memoryless} \end{align*}

We can define the predicted belief of the state at time $t$ to be the probability distribution before knowing the current measurement, i.e., $\overline{\mathrm{bel}}(\mathbf{x}_t)\triangleq p(\mathbf{x}_t\mid\mathbf{z}_{1:t-1},\mathbf{u}_{1:t})$ . Then, the above equation becomes

\begin{equation} \mathrm{bel}(\mathrm{x}_t)=\eta p(\mathbf{z}_t\mid\mathbf{x}_t)\overline{\mathrm{bel}}(\mathbf{x}_t) \end{equation}

where $\eta$ is the normalization coefficient, and

\frac{1}{\eta}=\int p(\mathbf{z}_t\mid\mathbf{x}_t)\overline{\mathrm{bel}}(\mathbf{x}_t)\mathrm{d}\mathbf{x}_t

which is essentially a correction applied to the belief after observing the current measurement.

What remains to be done, obviously, is to predict the belief $\overline{\mathrm{bel}}(\mathbf{x}_t)$ from the previous state:

\begin{align*} \overline{\mathrm{bel}}(\mathbf{x}_t) &=p(\mathbf{x}_t\mid\mathbf{z}_{1:t-1},\mathbf{u}_{1:t}) \\ &=\int p(\mathbf{x}_t\mid\mathbf{x}_{t-1},\mathbf{z}_{1:t-1},\mathbf{u}_{1:t})p(\mathbf{x}_{t-1}\mid\mathbf{z}_{1:t-1},\mathbf{u}_{1:t})\mathrm{d}\mathbf{x}_{t-1} && \to \text{marginalize}\,\mathbf{x}_{t-1}\\ &=\int p(\mathbf{x}_t\mid\mathbf{x}_{t-1}, \mathbf{u}_t)p(\mathbf{x}_{t-1}\mid\mathbf{z}_{1:t-1},\mathbf{u}_{1:t-1})\mathrm{d}\mathbf{x}_{t-1} &&\to\text{Markov assumption} \end{align*}

Then, recognizing $p(\mathbf{x}_{t-1}\mid\mathbf{z}_{1:t-1},\mathbf{u}_{1:t-1})=\mathrm{bel}(\mathbf{x}_{t-1})$ , we have

\begin{equation} \overline{\mathrm{bel}}(\mathbf{x}_t)=\int p(\mathbf{x}_t\mid\mathbf{u}_t,\mathbf{x}_{t-1})\mathrm{bel}(\mathbf{x}_{t-1})\mathrm{d}\mathbf{x}_{t-1} \end{equation}

In summary, the Bayes filter provides a two-step framework for updating the belief of the current state $\mathbf{x}_t$ from the belief of the previous state $\mathbf{x}_{t-1}$ :

Prediction using the motion model (Equation 1);

Correction using the measurement model (Equation 2).

These two steps can be performed recursively to estimate the state over time.

Kalman Filter

The Kalman filter is an instantiation of the Bayes filter with the assumption of linear Gaussian models. This means the motion model and the measurement model should be

linear with respect to their arguments:
Specifically, suppose $\mathbf{x}\in\mathbb{R}^m$ , $\mathbf{u}\in\mathbb{R}^n$ , and $\mathbf{z}\in\mathbb{R}^p$ . Let $\mathbf{F}\in \mathbb{R}^{m\times m}$ , $\mathbf{G}\in\mathbb{R}^{m\times n}$ , and $\mathbf{H}\in\mathbb{R}^{p\times m}$ be matrices (linear maps).
- Motion model:
$\begin{align} \mathbf{x}_t=\mathbf{F}_t\mathbf{x}_{t-1}+\mathbf{G}_t\mathbf{u}_t+\mathbf{w}_t, &&\mathbf{w}_t\sim\mathcal{N}(\mathbf{0},\mathbf{Q}_t) \end{align}$
- Measurement model:
$\begin{align} \mathbf{z}_t=\mathbf{H}_t\mathbf{x}_t+\mathbf{v}_t, &&\mathbf{v}_t \sim\mathcal{N}(\mathbf{0}, \mathbf{R}_t) \end{align}$
where $\mathbf{w}\in\mathbb{R}^m$ and $\mathbf{v}_t\in\mathbb{R}^p$ are white noise with variance $\mathbf{Q}\in\mathbb{R}^{m\times m}$ and $\mathbf{R}\in\mathbb{R}^{p\times p}$ , respectively.
- White noise

Acoustic Damping with a Helmholtz Resonator

2024-08-06T00:00:00+00:00

Wave Theory of Sound

Two fundamental equations in acoustics are

Continuity Equation

The continuity equation states that the rate at which mass enters a system is equal to the rate at which mass leaves the system plus the accumulation of mass within the system.

\begin{equation} \frac{\partial\rho}{\partial t}+\nabla\cdot \left(\rho \boldsymbol{v}\right)=0 \end{equation}

$\rho$ is fluid density

$\boldsymbol{v}$ is the velocity vector of the fluid

This equation is essentially $\sum m_{in} - \sum m_{out} = \Delta m_{system}$ .

Momentum Equation

The momentum equation states that the rate of change of momentum of a fluid element is equal to the sum of the pressure gradient force, convective acceleration, and body forces.

\begin{equation} \frac{\partial\boldsymbol{v}}{\partial t}+\boldsymbol{v}\cdot \nabla\boldsymbol{v}+\frac{\nabla p}{\rho}=\boldsymbol{g} \end{equation}

$p$ is the pressure

$\boldsymbol{g}$ is the body forces per unit mass.

This equation is essentially $F=ma$ .

Wave Equation

The ambient state is characterized by its pressure $p_0$ , density $\rho_0$ , and fluid velocity $\boldsymbol{u}_0$ . Acoustic disturbances are regarded as perturbations to the ambient state.

It can be derived from the previous two equations that the acoustic pressure $p$ , which is the difference between the pressure at $(x,t)$ and the ambient pressure, satisfies the wave equation

\frac{\partial ^2p}{\partial x^2} - \frac{1}{c^2}\frac{\partial^2p}{\partial t^2}=0

where $c$ is the speed of propagation. For simplicity, we only consider the one-dimensional case.

Sinusoidal Plane-wave

Suppose the sound we use in this experiment is a sinusoidal wave with a constant angular frequency $\omega$ . By introducing phasors, then by separation of variable, the solution is

p\left(x,t\right) = e^{i\omega t}f\left(x\right)

Plug back into the wave equation to get

\frac{\partial^2}{\partial t^2}\left(e^{i\omega t}f\left(x\right)\right)=c^2\frac{\partial^2}{\partial x^2}\left(e^{i\omega t}f\left(x\right)\right)

or simply

\frac{\mathrm{d}^2}{\mathrm{d}x^2}f\left(x\right) = -\left(\frac{\omega}{c}\right)^2f\left(x\right)

which is a very familiar second-order linear ODE. The general solution is

f\left(x\right)=Ae^{-ikx}+Be^{ikx}

where $k=\omega/c$ is called the wave number. Finally, the total solution is

p\left(x,t\right) = e^{i\omega t}\left(Ae^{-ikx}+Be^{ikx}\right)=Ae^{i\left(\omega t-kx\right)}+Be^{i\left(\omega t + kx\right)}

where $A,B\in\mathbb{C}$ are determined by initial and boundary conditions.

Acoustical Two-ports

Consider a model in the figure below where an acoustic element is located between two straight ducts.

This two-port system is fully characterized by an acoustic transfer matrix, which can be written as a relation between the pressure and velocities on each side of the two-port system.

We need to consider two variables , the acoustic pressure $p$ and the volume velocity $u$ , which are analogous to voltage $V$ and current $I$ in electric circuits respectively. Then, similar to Kirchhoff’s voltage and current law, we have the continuity of pressure law and continuity of volume law.

Continuity of Pressure

This law states that acoustic pressure does not vary appreciably over distances much less than a wavelength. Consider a path from $x_1$ to $x_2$ ,

Starting from the momentum equation (Equ. 2), neglecting nonlinear term $\left(\boldsymbol{v}\cdot \nabla\right)\boldsymbol{v}$ , and integrate along the path, we get,

\rho\frac{\partial}{\partial t}\int_{x_1}^{x_2}\boldsymbol{v}\cdot\mathrm{d}\ell = -\int_{x_1}^{x_2}\nabla p\cdot\mathrm{d}\ell = -\left(p_2-p_1\right)

where $\rho$ is the density of the medium, and $\boldsymbol{v}$ is the particle velocity vector.

Continuity of Volume Velocity

This law states that the net volume velocity flowing out of a volume is zero.

Starting from the continuity equation (Equ. 2), with an application of Gauss’s theorem, we obtain

\sum U=-\frac{\partial}{\partial t}\int\frac{p}{\rho c^2}\mathrm{d}V

Acoustic Impedance

We can define the acoustic impedance as the ratio of acoustic pressure and volume velocity,

Z = \frac{p}{u}

Notice that a proper definition can be found here.

For a Helmholtz Resonator, geometry shown below,

we have,

Z_{\mathrm{HR}}=Z_{\mathrm{vol}} + Z_{\mathrm{op}}

\begin{align*} Z_{\mathrm{vol}} = -\frac{1}{i\omega C_A} && \text{and} && Z_{\mathrm{op}} = -i\omega M_A \end{align*}

where $C_A = V/\rho c^2$ is the acoustic compliance, and $M_A = \rho l'/A$ is the acoustic inertance, $l'$ is the “effective neck length”.

Reflection and Transmission

The reason why Helmholtz Resonator can reduce noise is that it can “reflect” incident acoustic wave almost completely when the frequency of the wave is at the resonance frequency.

We define the reflection coefficient as the ratio of the intensity of the transmitted wave ( $I_t$ ) to the incident wave ( $I_i$ ),

\mathcal{T}=\frac{I_t}{I_i}

It can be derived that (but I don’t know how), for Helmholtz Resonator,

\mathcal{T}=1+\frac{-\rho c/A_D}{2Z_{HR}+\rho c/A_D}

At the resonance frequency,

\omega = c\sqrt{\frac{A}{l'V}}

The acoustic impedance of HR is

Z_{HR}=\frac{\rho c^2}{-i\omega V}-i\omega\frac{\rho l'}{A}=0

Then,

\mathcal{T}=0

Thus the resonator has the potentially useful property of causing nearly total reflection of acoustic waves at frequencies near its resonance frequency. That’s why it is useful for acoustic damping.

The transmission loss is defined as

\mathrm{TL} = 10\log\left|\frac{1}{\mathcal{T}}\right|^2

By setting proper value for $A, l',\rho, c, A_D$ , we can plot the transmission loss vs. frequency

Experiment

In the experiment, we want to measure the sound transmission loss of a Helmholtz Resonator for different frequency. The setup is shown below

where test specimen is a HR connected as a side branch.

We use the loudspeaker to generate a sinusoidal wave at frequency $f$ , then on the left hand side (upstream),

p_{\mathrm{up}} = Ae^{i(\omega t-kx)}+Be^{i(\omega t + kx)}

and on the right hand side (downstream),

p_{\mathrm{down}} = Ce^{i(\omega t-kx)}+De^{i(\omega t+kx)}

Here $Ae^{i(kx+\omega t)}$ is the incident wave, $Be^{i(kx+\omega t)}$ is the reflected wave, and $Ce^{-i(kx+\omega t)}$ is the transmitted wave. By definition, the transmission loss is

\mathrm{TL} = 10\log\left|\frac{A}{C}\right|^2

To solve for $A$ and $C$ , we use four microphone to measure the acoustic pressure at four different places,

\begin{align*} p_1 &= A e^{-ikx_1} + B e^{ikx_1}, \\ p_2 &= A e^{-ikx_2} + B e^{ikx_2}, \\ p_3 &= C e^{-ikx_3} + D e^{ikx_3}, \\ p_4 &= C e^{-ikx_4} + D e^{ikx_4}, \end{align*}

Solving for $A,B,C,D$ ,

\begin{align*} A &= \frac{i(p_1 e^{ikx_2} - p_2 e^{ikx_1})}{2 \sin k(x_1 - x_2)}, \\ B &= \frac{i(p_2 e^{-ikx_1} - p_1 e^{-ikx_2})}{2 \sin k(x_1 - x_2)}, \\ C &= \frac{i(p_3 e^{ikx_4} - p_4 e^{ikx_3})}{2 \sin k(x_3 - x_4)}, \\ D &= \frac{i(p_4 e^{-ikx_3} - p_3 e^{-ikx_4})}{2 \sin k(x_3 - x_4)}. \end{align*}

For convenience, we can take $s = \left|x_1-x_2\right|=\left|x_3-x_4\right|$ , then the transmission loss is

\mathrm{TL}=10\log\left|\frac{e^{iks}-p_2/p_1}{e^{iks}-p_4/p_3}\right|^2

The microphone record voltage signal, which is proportional to the acoustic pressure $p = p(t)$ , apply Fourier transform, take $p_1(t)=Ae^{i(\omega t-kx_1)}+Be^{i(\omega t+kx_1)}$

\hat{p}\left(\omega\right)=\int_{\mathbb{R}}e^{-i\omega t}p_1\left(t\right)\mathrm{d}t = A e^{-ikx_1} + B e^{ikx_1}

Thus, we can see that $p_1,\dots, p_4$ are just the Fourier coefficient at the given frequency. Since we only need to ratio, by applying a Fast Fourier Transform directly to the voltage signal, and take their ratio will give the same result.

Solving Basic PDEs

2023-12-07T00:00:00+00:00

A partial differential equation (PDE) is a relation involving one or more functions of several variables, and their partial derivatives. The order of a partial differential equation is the order of the highest partial derivative that appears in the equation.

There are three classical partial differential equations of order two which are very common in physical applications. They are known as the heat equation (1), the wave equation (2), and the Laplace equation (3).

\begin{align} &\frac{\partial u}{\partial t} = \alpha^2\frac{\partial^2u}{\partial x^2}\\ &\frac{\partial^2 u}{\partial t^2}=c^2\frac{\partial u^2}{\partial x^2}\\ &\frac{\partial ^2u}{\partial x^2}+\frac{\partial ^2u}{\partial y^2}=0 \end{align}

Heat Equation

Consider the boundary-value problem

\begin{align*} \frac{\partial u}{\partial t} = \alpha^2\frac{\partial^2u}{\partial x^2}; &&u\left(x,0\right)=f\left(x\right), 0

First, it’s easy to see that the equation is linear, which means any linear combination of solutions $u_1\left(x,t\right),\dots,u_n\left(x,t\right)$ is again a solution. This suggests the following “plan” for solving this boundary-value problem.

Find as many solutions $u_i\left(x,t\right)$ as we can that satisfies the boundary condition $u\left(0,t\right)=u\left(l,t\right)=0$ .

Find a proper linear combination of $u_i$ to satisfies the initial condition $u\left(x,0\right)=f\left(x\right)$ .

An important method to solve partial differential equations is separation of variables, which means we first assume $u\left(x,t\right)=X\left(x\right)T\left(t\right)$ . Then

\begin{align*} \frac{\partial u}{\partial t}=XT', && \frac{\partial^2u}{\partial x^2}=X''T \end{align*}

Plug this in the original equation and get

XT'=\alpha^2X''T

divide both side by $\alpha^2XT$ gives

\frac{X''}{X}=\frac{T'}{\alpha^2T}

Notice that the left-hand side only depend on $x$ , while the right-hand side only depend on $t$ . The only way that they can equal for any $x$ and $t$ is that both of them are constant. Thus,

\begin{align*} \frac{X''}{X}=-\lambda, && \frac{T'}{\alpha^2T}=-\lambda \end{align*}

By separation of variable, we have transformed a PDE into two solvable ODEs,

\begin{align*} X''+\lambda X=0, && T'+\alpha^2\lambda T=0 \end{align*}

Now, plug in the boundary condition.

\begin{align*} 0=u\left(0,t\right)=X\left(0\right)T\left(t\right), && 0=u\left(l,t\right)=X\left(l\right)T\left(t\right) \end{align*}

Since $T\left(t\right)\ne 0$ (not identically zero, otherwise $u$ would be identically zero, and we have a trivial solution), we have $X\left(0\right)=X\left(l\right)=0$ .

Now, we only need to solve

\begin{align*} X''+\lambda X = 0; && X\left(0\right)=0, X\left(l\right)=0 \end{align*}

and

T'+\lambda\alpha^2T=0

The first one is a second-order, linear, constant coefficient ODE, which we can directly write out the solution. There are three different cases:

$\lambda = 0$ : The general solution is $X\left(x\right)=c_1x+c_2$ for some constant $c_1, c_2$ . Plug in the boundary condition, we have $c_1=c_2=0$ . Thus, there are no non-trivial solution for $\lambda = 0$ .

$\lambda<0$ : The general solution is $X\left(x\right)=c_1e^{\sqrt{-\lambda}x}+c_1e^{\sqrt{-\lambda}x}$ for some constant $c_1,c_2$ . The boundary condition implies

\begin{align*} c_1+c_2=0, && c_1e^{\sqrt{-\lambda}l}+c_2e^{-\sqrt{-\lambda}l}=0 \end{align*}

This linear system of equation has non-trivial solution $c_1, c_2$ if and only if

\det\left(\begin{matrix}1 &1\\ e^{\sqrt{-\lambda}l}&e^{-\sqrt{-\lambda}l}\end{matrix}\right)=e^{-\sqrt{-\lambda}l}-e^{\sqrt{-\lambda}l}=0

This implies $e^{2\sqrt{-\lambda}l}=1$ , which is impossible since $2\sqrt{-\lambda}l>0$ . Therefore, there are also no non-trivial solution for $\lambda < 0$ .

$\lambda > 0$ : The general solution is $X\left(x\right)=c_1\cos\sqrt{\lambda}x+c_2\sin\sqrt{\lambda}x$ for some constant $c_1,c_2$ The boundary condition implies

\begin{align*} c_1=0, && c_2\sin\sqrt{\lambda}l=0 \end{align*}

For $c_2\ne0$ , we need to have $\sqrt{\lambda}l=n\pi$ for positive integer $n$ , or $\displaystyle \lambda_n=\frac{n^2\pi^2}{l^2}$ . Then the equation has non-trivial solutions

X\left(x\right)=X_n\left(x\right)=\sin\frac{n\pi x}{l}

Then, we have the second equation,

T'+\frac{n^2\pi^2\alpha^2}{l^2}T=0

which has solution

T\left(t\right)=T_n\left(t\right)=e^{-\alpha^2n^2\pi^2t/l^2}

Hence, we have

u_n\left(x,t\right)=X_n\left(x\right)T_n\left(t\right)=\sin\frac{n\pi x}{l}e^{-\alpha^2n^2\pi^2t/l^2}

Finally, we need to use the initial condition to find a proper linear combination of $u_n$ ’s, i.e., find coefficients $c_n$ ’s such that

u\left(x,t\right)=\sum_{n=0}^{\infty}c_n\sin\frac{n\pi x}{l}e^{-\alpha^2n^2\pi^2t/l^2}

satisfies $u\left(x,0\right)=f\left(x\right)$ . This means that

f\left(x\right)=\sum_{n=0}^{\infty}c_n\sin\frac{n\pi x}{l}

It’s easily seen that $c_n$ ’s should be the Fourier coefficient of the pure sine expansion, which is

c_n=\frac{2}{l}\int_{0}^{l}f\left(x\right)\sin\frac{n\pi x}{l}\mathrm{d}x

Finally,

u\left(x,t\right)=\frac{2}{l}\sum_{n=0}^{\infty}\left[\int_{0}^{l}f\left(x\right)\sin\frac{n\pi x}{l}\mathrm{d}x\right]\sin\frac{n\pi x}{l}e^{-\alpha^2n^2\pi^2t/l^2}

is the desired solution.

Wave Equation

Consider the boundary-value problem

\begin{align*} \frac{\partial^2u}{\partial t^2}=c^2\frac{\partial^2 u}{\partial x^2}; && u\left(x,0\right)=f\left(x\right), u_t\left(x,0\right)=g\left(x\right); &&u\left(0,t\right)=u\left(l,t\right)=0 \end{align*}

This problem can also be solved by separation of variable. Let $u\left(x,t\right)=X\left(x\right)T\left(t\right)$ , then

\begin{align*} \frac{\partial^2u}{\partial t^2}=XT'', && \frac{\partial^2u}{\partial x^2}=X''T \end{align*}

Plug into the original equation, we have

\frac{T''}{c^T}=\frac{X''}{X}=-\lambda

or two ODEs,

\begin{align*} X''+\lambda X=0; && X\left(0\right)=X\left(l\right)=0 \end{align*}

and

T''+\lambda c^2T=0

We’ve seen in the previous problem that the first equation has solutions

X\left(x\right)=X_n\left(x\right)=\sin\frac{n\pi x}{l}

For the second equation, it becomes

T''+\frac{n^2\pi^2c^2}{l^2}T=0

The solutions are

T\left(t\right)=T_n\left(t\right)=a_n\cos\frac{n\pi ct}{l}+b_n\sin\frac{n\pi ct}{l}

Thus,

u_n\left(x,t\right)=\sin\frac{n\pi x}{l}\left[a_n\cos\frac{n\pi ct}{l}+b_n\sin\frac{n\pi ct}{l}\right]

is a non-trivial solution of the BVP for every positive integer $n$ , and every suitable constants $a_n, b_n$ . The linear combination

u\left(x,t\right)=\sum_{n=1}^{\infty}\sin\frac{n\pi x}{l}\left[a_n\cos\frac{n\pi ct}{l}+b_n\sin\frac{n\pi ct}{l}\right]

satisfies

\begin{align*} u\left(x,0\right)=\sum_{n=1}^{\infty}a_n\sin\frac{n\pi x}{l}, && u_t\left(x,0\right)=\sum_{n=1}^{\infty}\frac{n\pi c}{l}b_n\sin\frac{n\pi x}{l} \end{align*}

To satisfy the initial condition, we only need to have

\begin{align*} f\left(x\right)=\sum_{n=1}^{\infty}a_n\sin\frac{n\pi x}{l}, && g\left(x\right)=\sum_{n=1}^{\infty}\frac{n\pi c}{l}b_n\sin\frac{n\pi x}{l} \end{align*}

By Fourier sine expansion, we have

\begin{align*} a_n=\frac{2}{l}\int_{0}^{l}f\left(x\right)\sin\frac{n\pi x}{l}\mathrm{d}x, && b_n=\frac{2}{n\pi c}\int_0^{l}g\left(x\right)\sin\frac{n\pi x}{l}\mathrm{d}x \end{align*}

Solving ODEs by Series

2023-12-03T00:00:00+00:00

Now, we consider a somehow more general case, where we want to solve a linear second-order homogeneous equation with coefficients that are polynomials. Consider the following equation

p\left( t \right) y''+q\left( t \right) y'+r\left( t \right) y=0

where $p\left(t\right)\ne 0$ in some interval and $p\left(t\right),q\left(t\right),r\left(t\right)$ are polynomials of $t$ .

Properties of Power Series

An infinite series

y\left( t \right) =\sum_{n=0}^{\infty}{a_n\left( t-t_0 \right) ^n}=a_0+a_1x+a_2x^2+\cdots

is called a power series about $t=t_0$ . The following are some of its properties.

All power series have an interval of convergence. This means that there exists a nonnegative number $\rho$ such that the infinite series converges for $\left|t-t_0\right|<\rho$ , and diverges for $\left|t-t_0\right|>\rho$ . The number $\rho$ is called the radius of convergence of the power series.

The power series can be differentiated and integrated term by term, and the resultant series have the same radius of convergence.

One simple method to determine the radius of convergence of a power series is the Cauchy ratio test. Suppose

\underset{n\rightarrow \infty}{\lim}\left| \frac{a_{n+1}}{a_n} \right|=\lambda

Then the radius of convergence is $\displaystyle \rho=\frac{1}{\lambda}$ .

Many functions $f\left(t\right)$ that arise in applications can be expanded in power series, i.e., there exists a series of coefficients $a_0,a_1,a_2,\dots$ so that

f\left(t\right)=\sum_{n=0}^{\infty}a_n\left(t-t_0\right)^n

Such functions are said to be analytic at $t=t_0$ , the series expansion is called the Taylor series of $f$ about $t=t_0$ , and the coefficients can be found by

a_n=\frac{f^{\left(n\right)}\left(t_0\right)}{n!}

The radius of convergence can be determined if we treat the function $f$ in the complex plane. Let $z_0\in\mathbb{C}$ be the point that is closest to $t_0$ that $f^{(n)}(z_0)$ fails to exist, then the radius of convergence is $\rho=\left\|z_0-t_0\right\|$ .

The product of two power series is still a power series, this product is called the Cauchy product

\sum_{n=0}^{\infty}{a_n\left( t-t_0 \right) ^n}\sum_{n=0}^{\infty}{b_n\left( t-t_0 \right) ^n}=\sum_{n=0}^{\infty}{c_n\left( t-t_0 \right) ^n}

where the coefficient is

c_n=\sum_{j+k=n}a_jb_k=a_0b_n+a_1b_{n-1}+\cdots+a_nb_0

which is called the convolution of $a_n$ and $b_n$ .

Important series expansions (Remember them!)

\begin{align*} \frac{1}{1-x}=1+x+x^2+\cdots=\sum_{n=0}^{\infty}x^n, && \rho=1 \end{align*}

\begin{align*} e^x=1+x+\frac{x^2}{2}+\frac{x^3}{6}+\cdots=\sum_{n=0}^{\infty}\frac{x^n}{n!}, && \rho=\infty \end{align*}

\begin{align*} \sin x=x-\frac{x^3}{3!}+\frac{x^5}{5!}-\cdots=\sum_{n=0}^{\infty}\frac{\left(-1\right)^nx^{2n+1}}{\left(2n+1\right)!}, && \rho=\infty \end{align*}

\begin{align*} \cos x=1-\frac{x^2}{2!}+\frac{x^4}{4!}-\cdots=\sum_{n=0}^{\infty}\frac{\left(-1\right)^nx^{2n}}{\left(2n\right)!}, && \rho=\infty \end{align*}

\begin{align*} \ln\left(1-x\right)=-x-\frac{x^2}{2}-\frac{x^3}{3}-\cdots=-\sum_{n=1}^{\infty}\frac{x^n}{n}, && \rho = 1 \end{align*}

\begin{align*} \ln\left(1+x\right)=x-\frac{x^2}{2}+\frac{x^3}{3}-\cdots=\sum_{n=1}^{\infty}\left(-1\right)^{n+1}\frac{x^n}{n}, && \rho = 1 \end{align*}

Series Solution

For a second-order equation with variable coefficients (but are analytic functions, at least), although it’s not very rigorous, we can first expect that the solution $y$ is also a power series. Then, by doing series expansion and setting the polynomial coefficients properly, we can expect the equation to hold. A formal statement of this is as follows.

Let the functions $q\left(t\right)/p\left(t\right)$ and $r\left(t\right)/p\left(t\right)$ have convergent Taylor series expansions about $t=t_0$ , for $\left|t-t_0\right|<\rho$ . Then, every solution $y\left(t\right)$ of the differential equation

p\left(t\right)y''+q\left(t\right)y'+r\left(t\right)y=0

is analytic at $t=t_0$ , and the radius of convergence of its Taylor series expansion about $t=t_0$ is at least $\rho$ . Set

y\left(t\right)=a_0+a_1\left(t-t_0\right)+a_2\left(t-t_0\right)^2+\cdots

the coefficients are determined by plugging the expression into the differential equation and setting the sum of the coefficients of like powers of $t$ in this expression equal to zero.

Example (1)

However, it is possible that sometime $p\left(t_0\right)=0$ , and the above method of series expansion will fail to work, but we still want to find a solution around the point $t=t_0$ . One possible case is Euler’s equation.

Euler’s Equation

The differential equation

t^2y''+\alpha ty'+\beta y=0

where $\alpha,\beta\in\mathbb{R}$ are constant is known as Euler’s equation.

This equation has a singular point at $t=0$ and the series solution method above cannot work. However, we can treat $t>0$ and $t<0$ separately.

First, assume that $t>0$ .

For $t<0$ , let $t=-x$ ( $x>0$ ), and let $y=u\left(x\right)$ . Then the equation is transformed into

x^2u''+\alpha x u'+\beta u = 0

which should have the same solution as the case when $t>0$ . Thus, for the final solution, we only need to put a modulus on $t$ .

Observe that $t^2y''$ , $ty'$ , and $y$ will have the same order if $y=t^r$ . Our goal is to find two independent solutions. Therefore, we plug in $y=t^r$ into the equation.

\begin{align*} \frac{\mathrm{d}}{\mathrm{d}t}t^r=rt^{r-1}, && \frac{\mathrm{d}^2}{\mathrm{d}t^2}t^r=r\left(r-1\right)t^{r-2} \end{align*}

The original equation then becomes,

\left[r\left(r-1\right)+\alpha r+\beta\right]t^r=0

Hence, $y=t^r$ is a solution if and only if $r$ is a solution of the quadratic equation

r^2+\left(\alpha-1\right)r+\beta=0

Two Different Real Roots, $\left(\alpha-1\right)^2-4\beta >0$

The general solution is given by

y\left(t\right)=c_1t^{r_1}+c_2t^{r_t}

where $\displaystyle r_{1,2}=-\frac{1}{2}\left[\left(\alpha-1\right)\pm\sqrt{\left(\alpha - 1\right)^2-4\beta}\right]$ .

Two Equal Roots, $\left(\alpha-1\right)^2-4\beta =0$

One solution that is obvious is

y_1\left(t\right)=t^{r_1}

where $\displaystyle r=\frac{1-\alpha}{2}$ .

To find the second independent solution, applying the differential operator

\displaystyle L=t^2\frac{\mathrm{d}^2}{\mathrm{d}t^2}+\alpha t\frac{\mathrm{d}}{\mathrm{d}t}+\beta

and rewrite the equation as $Ly=0$ . Plug in $y=t^r$ , we have

L\left[t^{r}\right]=\left(r-r_1\right)^2t^{r}

Taking partial derivatives of both sides with respect to $r$ gives

\frac{\partial}{\partial r}L\left[t^r\right]=L\left[\frac{\partial}{\partial r}t^{r}\right]=\frac{\partial}{\partial r}\left[\left(r-r_1\right)^2t^r\right]

That is

L\left[t^r\ln t\right]=\left(r-r_1\right)^2t^r\ln t+2\left(r-r_1\right)t^r

We can see that if $r=r_1$ , then the right-hand side will vanish. Therefore, the second solution is

y_2\left(t\right)=t^{r_1}\ln t

The general solution is

y=\left(c_1+c_2\ln t\right)t^{r_1}

Two Complex Solutions, $\left(\alpha-1\right)^2-4\beta <0$

Let

\begin{align*} r_1=\lambda+i\mu, && r_2=\lambda-i\mu \end{align*}

and

\phi\left(t\right)=t^{\lambda+i\mu}=t^{\lambda}\left[\cos\left(\mu \ln t\right)+i\sin\left(\mu \ln t\right)\right]

It can be shown that the real and imaginary parts are two independent solutions. Therefore,

\begin{align*} y_1\left(t\right)=\mathrm{Re}\left(\phi\left(t\right)\right)=t^{\lambda}\cos\left(\mu\ln t\right)\\ y_2\left(t\right)=\mathrm{Im}\left(\phi\left(t\right)\right)=t^{\lambda}\sin\left(\mu\ln t\right)\\ \end{align*}

and the general solution is

y\left(t\right)=t^{\lambda}\left[c_1\cos\left(\mu\ln t\right)+c_2\sin\left(\mu\ln t\right)\right]

where $\displaystyle \lambda=\frac{1-\alpha}{2}$ , $\displaystyle \mu = \frac{\sqrt{4\beta-\left(\alpha-1\right)^2}}{2}$ .

Method of Frobenius

Here, we want to find a class of singular differential equations that is more general than Euler’s equation. A very natural generalization is the following

L(y)=y''+p\left(t\right)y'+q\left(t\right)y=0

where $p\left(t\right)$ and $q\left(t\right)$ can be expanded in series of the form

\begin{align*} p\left(t\right)&=\frac{p_{0}}{t}+p_1+p_2t+p_3t^2+\cdots\\ q\left(t\right)&=\frac{q_{0}}{t^2}+\frac{q_{1}}{t}+q_2+q_3t+q_4t^2+\cdots \end{align*}

In this case, the equation is said to have a regular singular point at $t=0$ . Equivalently, an equation has a regular singular point at $t=0$ if functions $tp\left(t\right)$ and $t^2q\left(t\right)$ are analytic at $t=0$ . It has a regular singular point at $t=t_0$ if functions $\left(t-t_0\right)p\left(t\right)$ and $\left(t-t_0\right)^2q\left(t\right)$ are analytic at $t=t_0$ .

We focus on the interval $t>0$ and multiply through the equation by $t^2$ and get

t^2y''+t\left(tp\left(t\right)\right)y'+t^2q\left(t\right)=0

Compare the above equation with Euler’s equation, and find that it seems like adding higher order terms, i.e., by changing $\alpha$ to $\displaystyle tp\left(t\right)=\sum_{n=0}^{\infty}p_nt^n$ , and changing $\beta$ to $\displaystyle t^2q\left(t\right)=\sum_{n=0}^{\infty}q_nt^n$ . Therefore, we may be able to obtain a solution by plugging in higher order term to the original $t^r$ solution. Specifically, let

y\left(t\right)=\sum_{n=0}^{\infty}a_nt^{n+r}=t^{r}\sum_{n=0}^{\infty}a_nt^n

Then,

y'\left(t\right)=\sum_{n=0}^{\infty}\left(n+r\right)a_nt^{n+r-1}

and

y''\left(t\right)=\sum_{n=0}^{\infty}\left(n+r\right)\left(n+r-1\right)a_nt^{n+r-2}

Plug these in the original equation at get

\begin{align*} L(y)&=t^r\left\{\sum_{t=0}^{\infty}\left(n+r\right)\left(n+r-1\right)a_nt^n+\left(\sum_{m=0}^{\infty}p_mt^m\right)\left[\sum_{n=0}^{\infty}\left(n+r\right)a_nt^n\right]\right\}\\ &+t^r\left(\sum_{m=0}^{\infty}q_mt^m\right)\left(\sum_{n=0}^{\infty}a_nt^{n}\right)\\ &=\left[r\left(r-1\right)+0_0r+q_0\right]a_0t^r\\ &+\left\{\left[\left(1+r\right)r+p_0\left(1+r\right)+q_0\right]a_1+\left(rp_1+q_1\right)a_0\right\}t^\\ &+\dots\\ &+\left(\left[\left(n+1\right)\left(n+r-1\right)+p_0\left(n+r\right)+q_0\right]a_n+\sum_{n=0}^{\infty}\left[(k+r)p_{n-k}+q_{n-k}\right]a_k\right)t^{n+r} \end{align*}

If we set

F\left(r\right)=r\left(r-1\right)+p_0r+q_0

and set the coefficient of each like power equal to zero, we have an indicial equation

F\left(r\right)=r\left(r-1\right)+p_0r+q_0=0

and a recurrence relation

F\left(n+r\right)a_n=-\sum_{k=0}^{n-1}\left[\left(k+r\right)p_{n-k}+q_{m-k}\right]a_k

From these two relations, we can (hopefully) obtain two sets of coefficients $a_n(r_1)$ and $a_n(r_2)$ and the two independent solutions are

\begin{align*} y_1\left(t\right)=t^{r_1}\sum_{n=0}^{\infty}a_n\left(r_1\right)t^{n}, && y_2\left(t\right)=t^{r_2}\sum_{n=0}^{\infty}a_n\left(r_2\right)t^{n} \end{align*}

if we can obtain two different real solutions $r_1$ and $r_2$ , with $\left(r_1-r_2\right)\notin\mathbb{Z}$ .

Next, we discuss the possible situations that the above conditions are not satisfied.

Two Complex Roots

If $r$ is a complex root of the indicial equation, then

y\left(t\right)=t^r\sum_{n=0}^{\infty}a_nt^{n}

is a complex-valued solution, whose real and imaginary part are both real-valued solutions.

Equal Roots

If $r$ is the only root of the indicial equation, then we have only one independent solution.

y_1\left(t\right)=t^r\sum_{n=0}^{\infty}a_nt^{n}

To find the other solution, use the similar method of dealing with Euler’s equation. The second solution is

y_2\left(t\right)=y_1\left(t\right)\ln t+\sum_{n=0}^{\infty}a_n'\left(r\right)t^{n+r}

Roots Differing by a Positive Integer

Suppose that $r_2$ and $r_1=r_2+N$ , where $N\in\mathbb{Z}^{+}$ , are two roots of the indicial equation. Then, we can certainly find one solution

y_1\left(t\right)=t^r\sum_{n=0}^{\infty}a_n\left(r_1\right)t^{n}

It may be impossible to find another solution with coefficients $a_n\left(r_2\right)$ .

In this case, the equation will have a second solution of the form

y_2\left(t\right)=ay_1\left(t\right)\ln t+\sum_{n=0}^{\infty}a_n'\left(r_2\right)t^{n+r_2}

Elementary Complex Analysis

2023-11-16T00:00:00+00:00

Holomorphic Functions

A function $f:\mathbb{C}\to \mathbb{C}$ is complex differentiable, or holomorphic, at $z\in\mathbb{C}$ if

f'\left(z\right):=\lim_{h\to0,h\in\mathbb{C}}\frac{f\left(z+h\right)-f\left(z\right)}{h}

exists. The following are some of the properties of holomorphic functions.

Cauchy-Riemann Equations

Suppose that $f$ is holomorphic, and

f\left(x+iy\right)=u\left(x,y\right)+iv\left(x,y\right)

then, the functions $u,v:\mathbb{R}^2\to \mathbb{R}$ must satisfies the Cauchy-Riemann differential equations,

\begin{align*} \frac{\partial u}{\partial x}=\frac{\partial v}{\partial y}, && \frac{\partial u}{\partial y}=-\frac{\partial v}{\partial x} \end{align*}

In addition, we define

\begin{align*} \frac{\partial}{\partial z}:=\frac{1}{2}\left(\frac{\partial}{\partial x}+\frac{1}{i}\frac{\partial}{\partial y}\right), && \frac{\partial }{\partial \bar{z}}:=\frac{1}{2}\left(\frac{\partial}{\partial x}-\frac{1}{i}\frac{\partial}{\partial y}\right) \end{align*}

then

\begin{align*} f'\left(z\right)=\frac{\partial f}{\partial z}=2\frac{\partial u}{\partial z}, && \frac{\partial f}{\partial \bar{z}}=0 \end{align*}

Cauchy’s Theorem

Let $\Omega\sub\mathbb{C}$ be an open set, $f$ holomorphic on $\Omega$ and $\mathcal{C}\sub \Omega$ and oriented smooth curve. We define the integral of $f$ along $\mathcal{C}$ by

\int_{\mathcal{C}}f\left(z\right)\mathrm{d}z:=\int_{I}f\left(\gamma\left(t\right)\right)\cdot\gamma'\left(t\right)\mathrm{d}t

where $\gamma:I\to \mathcal{C}$ is a parametrization.

One of the most important properties of holomorphic function is known as Cauchy’s theorem.

If $f$ is holomorphic in a disc, then,

\oint_{\mathcal{C}}f\left(z\right)\mathrm{d}z=0

for any closed curve $\mathcal{C}$ in that disc.

With this properties, we can actually calculate many integrals along the real axis by extending it into an integration along a contour in the complex plane. The situation that we integrate over a semi-circle of radius $R$ and want that part to vanish is quite common, and Jordan’s lemma present a useful general result.

Assume that for some $R_0>0$ the function $g:\mathbb{C}\backslash\overline{B_{R_0}(0)}\to \mathbb{C}$ is holomorphic. Let

f\left(z\right)=e^{iaz}g\left(z\right)

for some $a>0$ . Let

\mathcal{C}_{R}=\left\{z\in\mathbb{C}:z=Re^{i\theta},0\le\theta\le\pi\right\}

be a semi-circle segment in the upper half-plane and assume that

\max_{0\le\theta\le\pi}\left|g\left(Re^{i\theta}\right)\right|\to0

as $R\to \infty$ . Then

\lim_{R\to\infty}\int_{\mathcal{C}_R}f\left(z\right)\mathrm{d}z=0

Cauchy’s Integral Formula

Suppose $f$ is a holomorphic function in an open set $\Omega\sub\mathbb{C}$ . If a closed disc $D$ defined as

D\:=\left\{z:\left|z-z_0\right|\le r\right\}

is completely contained in $\Omega$ . Let $C$ be the positively oriented circle forming the boundary of $D$ , i.e., $C=\partial D$ . Then, for all $a\in D$

\oint_{C}\frac{f\left(z\right)}{z-a}\mathrm{d}z=2\pi if\left(a\right)

This formula is known as the Cauchy’s integral formula. Furthermore, $f$ has infinitely many complex derivatives in $\Omega$ , and for all $z\in D$ ,

f^{\left(n\right)}\left(z\right)=\frac{n!}{2\pi i}\oint_{C}\frac{f\left(\zeta\right)}{\left(\zeta-z\right)^{n+1}}\mathrm{d}\zeta

It then follows that holomorphic functions are analytic, specifically, for all $z\in D$

f\left(z\right)=\sum_{n=0}^{\infty}a_n\left(z-z_0\right)^n

where

a_n=\frac{f^{\left(n\right)}\left(z_0\right)}{n!}

Residue Calculus

If $f:\Omega\to\mathbb{C}$ has a pole of order $n$ at $z_0\in \Omega$ , then there exists a neighborhood $U\sub\Omega$ of $z_0$ , numbers $a_{-n},\dots, a_{-1}\in\mathbb{C}$ and a holomorphic function $G: U\to \mathbb{C}$ such that

f\left(z\right)=\frac{a_{-n}}{\left(z-z_0\right)^n}+\frac{a_{-n+1}}{\left(z-z_0\right)^{n-1}}+\cdots+\frac{a_{-1}}{z-z_0}+G\left(z\right)

for all $z\in U$ . The term

P\left(z\right):=\frac{a_{-n}}{\left(z-z_0\right)^n}+\frac{a_{-n+1}}{\left(z-z_0\right)^{n-1}}+\cdots+\frac{a_{-1}}{z-z_0}

is called the principal part of $f$ at the pole $z_0$ . To calculate the coefficients of the principal part, we have

a_{-n}=\lim_{z\to z_0}\left(z-z_0\right)^nf\left(z\right)

The coefficient $a_{-1}$ of $\left(z-z_0\right)^{-1}$ is called the residue of $f$ at the pole $z_0$ , denoted as

\mathrm{res}_{z_0}f=a_{-1}

In fact, we can calculate the residue without knowing the series expansion. In the case where $f$ has a simple pole at $z_0$ , it is clear that

\mathrm{res}_{z_0}f=\lim_{z\to z_0}\left(z-z_0\right)f\left(z\right)

If $f$ has a pole of order $n$ at $z_0$ , then

\mathrm{res}_{z_0}f=\frac{1}{\left(n-1\right)!}\lim_{z\to z_0}\frac{\mathrm{d}^{n-1}}{\mathrm{d}z^{n-1}}\left(\left(z-z_0\right)^nf\left(z\right)\right)

Residues are very useful at calculating contour integration in the complex plane.

Suppose that $f$ is holomorphic in an open set containing a (positively oriented) toy contour $\mathcal{C}$ and its interior, except for poles at points $z_1,\dots, z_n$ inside $\mathcal{C}$ . Then,

\oint_{\mathcal{C}}f\left(z\right)\mathrm{d}z=2\pi i\sum_{k=1}^{n}\mathrm{res}_{z_k}f

In addition, let $p$ and $q$ be polynomials of degree $m$ and $n$ respectively, where $n\ge m+2$ . If $q\left(x\right)\ne 0$ for $x>0$ , if $q$ has a zero of order at most 1 at the orgin, and if

f\left(z\right)=\frac{z^{\alpha}p\left(z\right)}{q\left(z\right)}

for $0<\alpha<1$ , then

\int_{0}^{\infty}\frac{z^{\alpha}p\left(x\right)}{q\left(x\right)}\mathrm{d}x=\frac{2\pi i}{1-e^{2\pi \alpha i}}\sum_{k=1}^{n}\mathrm{res}_{z_k}f

where $z_1,\dots, z_n$ are the non zero poles of $p/q$ .

Guanyu’s Homepage

Optimization Part 1 Basics and Least Squares

Preliminaries

Linear Least Squares

Occupancy Grid Mapping

Problem Statement

Mapping Algorithm

Bayes Inference using Conjugate Prior

Bernoulli Likelihood and Beta Prior

Lie Theory in Robot Motion

Matrix Lie Groups

Special Orthogonal Group, SO(n)\mathrm{SO}(n)SO(n)﻿

Special Euclidean Group, SE(n)\mathrm{SE}(n)SE(n)﻿

Lie Algebra

Exponential Map

Conjugation and Adjoint Map

Angular Velocity and Twist, so(3),se(3)\mathfrak{so}(3), \mathfrak{se}(3)so(3),se(3)﻿

Reference

Bayes Filtering and State Estimation

Mathematical Formulation of Bayes Filter

Kalman Filter

Acoustic Damping with a Helmholtz Resonator

Wave Theory of Sound

Wave Equation

Sinusoidal Plane-wave

Acoustical Two-ports

Continuity of Pressure

Continuity of Volume Velocity

Acoustic Impedance

Reflection and Transmission

Experiment

Solving Basic PDEs

Heat Equation

Wave Equation

Solving ODEs by Series

Properties of Power Series

Series Solution

Euler’s Equation

Two Different Real Roots, (α−1)2−4β>0\left(\alpha-1\right)^2-4\beta >0(α−1)2−4β>0﻿

Two Equal Roots, (α−1)2−4β=0\left(\alpha-1\right)^2-4\beta =0(α−1)2−4β=0﻿

Two Complex Solutions, (α−1)2−4β<0\left(\alpha-1\right)^2-4\beta <0(α−1)2−4β<0﻿

Method of Frobenius

Two Complex Roots

Equal Roots

Roots Differing by a Positive Integer

Elementary Complex Analysis

Holomorphic Functions

Cauchy-Riemann Equations

Cauchy’s Theorem

Cauchy’s Integral Formula

Residue Calculus

Special Orthogonal Group, $\mathrm{SO}(n)$

Special Euclidean Group, $\mathrm{SE}(n)$

Angular Velocity and Twist, $\mathfrak{so}(3), \mathfrak{se}(3)$

Two Different Real Roots, $\left(\alpha-1\right)^2-4\beta >0$

Two Equal Roots, $\left(\alpha-1\right)^2-4\beta =0$

Two Complex Solutions, $\left(\alpha-1\right)^2-4\beta <0$