重庆理工大学理学院，重庆

收稿日期：2023年4月28日；录用日期：2023年5月21日；发布日期：2023年5月31日

摘要

研究了离散时间时变Markov跳变系统在有限时域上的最优控制问题。证明了任何离散时间时变Markov跳变系统的最优控制问题都可以通过广义代数Riccati方程的解来确定；也证明了最优控制问题的可达性是广义代数Riccati方程解的存在性的充分条件；最后通过数值举例来验证结果的正确性。

关键词

线性二次最优控制，离散时间时变系统，广义代数Riccati方程，Markov跳变系统

Linear Quadratic Optimal Control for Discrete Time Time-Varying Markov Jump Systems

Hongxia Zhao^*, Xin He, Yaqi Jia, Chunmei Zhang

College of Science, Chongqing University of Technology, Chongqing

Received: Apr. 28^th, 2023; accepted: May 21^st, 2023; published: May 31^st, 2023

ABSTRACT

The optimal control of discrete-time time-varying Markov jump systems in infinite horizon is studied. It is proved that the optimal control problem of any discrete time time-varying Markov jump system can be determined by the solution of generalized algebraic Riccati equation. It is also proved that the attainablility of the optimal control problem is a sufficient condition for the existence of solutions of generalized algebraic Riccati equations. Finally, numerical examples are given to verify the correctness of the results.

Keywords:Linear Quadratic Optimal Control, Discrete Time-Varying Systems, Generalized Algebraic Riccati Equation, Markov Jump System

Copyright © 2023 by author(s) and Hans Publishers Inc.

This work is licensed under the Creative Commons Attribution International License (CC BY 4.0).

http://creativecommons.org/licenses/by/4.0/

1. 引言

最优性是我们生活生产中所追求的目标之一。研究控制系统时，若系统是线性的，二次指标表示成状态变量 $x\left(t\right)$ 和输入控制变量 $u\left(t\right)$ 的二次函数，则把求得系统所对应的二次指标达到最优的控制问题，称为最优控制问题 [1] 。线性二次最优控制问题最早实在1958年由Bellman，Glicksberg，Gross等人研究的 [2] 。在此基础之上，R.E. Kalman针对二次最优控制建立了状态反馈 [3] ，并且将Riccati方程引入控制理论中，建立了最优线性反馈调节器 [4] [5] [6] [7] 。参考文献 [8] - [15] 中针对线性系统的系数问题作出假设，并且研究了任何的线性系统的稳定解可以用广义Riccati方程的可解性来表示 [16] [17] [18] 。在文献 [19] 中，为了保证二次线性控制问题是适定的和反馈稳定控制的存在性，要定义一些稳定性概念。

本文与参考文献 [20] 不同之处在于，本文考虑了具体非齐次马尔可夫切换的无限时域离散时间时变Markov跳变系统的线性二次最优控制，介绍了与KKT定理相关的最优线性反馈的存在性，且证明了广义Riccati方程的可解性与二次最优问题的适定性与可达性是等价的。要保证线性二次最优问题是适定的，则对应的广义Riccati方程是可解的，并且存在最优控制。在文章的第二节介绍了一些相关的定义和后续文章中所需要的一些引理；第三节对所研究的模型进行详细的描述，对模型中所出现的符号进行说明。

2. 预备知识

为了能够方便在后续文章中对一些相关性定理引理的应用，先对文章中将要用到的定理引理进行简单的介绍：

数学规划：

$\begin{array}{l}\min f\left(x\right)\\ \text{s}\text{.t}\text{.}\{\begin{array}{c}g\left(x\right)\le 0\\ h\left(x\right)=0\end{array}\end{array}$

其中：

$\begin{array}{l}g\left(x\right)=\left(g_1\left(x\right),g_2\left(x\right),\cdots ,g_m\left(x\right)\right)\\ h\left(x\right)=(h_1\left(x\right),h_2\left(x\right),\cdots ,h_m(\; x\; ))\end{array}$

定义1 (正则点)：对于数学规划中的可行点 $x^{\ast }$ ，令 $I^{\ast }=\left\{i|g_1\left(x^{\ast }\right)=0\right\}$ ，如果梯度向量 $\nabla g_i\left(x^{\ast }\right),i\in I^{\ast }$ 和 $\nabla h_j\left(x^{\ast }\right),j=1,2,\cdots ,n$ 都是线性独立的，则 $x^{\ast }$ 成为约束正则点。

定义2 (正则性条件)：令 $I^{\ast }=\left\{i|g_1\left(x^{\ast }\right)=0\right\}$ ，如果梯度向量 $\nabla g_i\left(x^{\ast }\right),i\in I^{\ast }$ 和 $\nabla h_j\left(x^{\ast }\right),j=1,2,\cdots ,n$ 都是线性独立的，则称之为正则性条件。

定义3 (KKT定理)：设正则点 $x^{\ast }$ 是数学规划的局部最优解，且目标函数 $f\left(x\right)$ 以及约束函数 $g\left(x\right)=\left(g_1\left(x\right),g_2\left(x\right),\cdots ,g_m\left(x\right)\right)$ ， $h\left(x\right)=\left(h_1\left(x\right),h_2\left(x\right),\cdots ,h_m\left(x\right)\right)$ ，在点 $x^{\ast }$ 处是连续可微的，则存在 $\lambda \in {ℝ}^m,\lambda \ge 0$ 及 $\mu \in {ℝ}^n$ 叫做KKT乘数，使得下列等式成立：

$\{\begin{array}{l}{\nabla }_x\mathcal{L}\left(x^{\ast },{\lambda }^{\ast },{\mu }^{\ast }\right)=0\\ {\lambda }^{\text{T}}g\left(x^{\ast }\right)=0\end{array}$

其中拉格朗日函数为： $\mathcal{L}\left(x,\lambda ,\mu \right)=f\left(x\right)+{\lambda }^{\text{T}}g\left(x\right)+{\mu }^{\text{T}}h(\; x\; )$

引理1 [12] ：给定矩阵 $Q\in {ℝ}^{m\times n}$ ，矩阵 $Q^{†}\in {ℝ}^{n\times m}$ 叫做Q的广义逆矩阵，则：

$\{\begin{array}{l}Q{Q}^{†}Q=Q,Q^{†}Q{Q}^{†}=Q^{†}\\ {\left(Q{Q}^{†}\right)}^{\text{T}}=Q{Q}^{†},{\left(Q^{†}Q\right)}^{\text{T}}=Q^{†}Q\end{array}$

引理2 [12] ：给定对称矩阵Q，则：

$\{\begin{array}{l}{\left(Q^{†}\right)}^{\text{T}}={\left(Q^{\text{T}}\right)}^{†},Q^{†}Q=Q{Q}^{†}\\ Q\ge 0,当且仅当Q^{†}\ge 0\end{array}$

引理3 (舒尔补引理) [13] ：设具有适当维数的矩阵 $Q^{\text{T}}=Q,N=N^{\text{T}},R=R^{\text{T}}$ ，则下面的条件是等价的：

(I) $Q-N{R}^{†}{N}^{\text{T}}\ge 0,R\ge 0,N\left(I-R{R}^{†}\right)=0$

(II) $\left[\begin{array}{cc}Q& N\\ N^{\text{T}}& R\end{array}\right]\ge 0$

(III) $\left[\begin{array}{cc}R& N^{\text{T}}\\ N& Q\end{array}\right]\ge 0$

引理4 [14] ：给定矩阵 $Q,K,L$ ，当且仅当 $Q,K,L$ ，当且仅当 $Q{Q}^{†}LK{K}^{†}=L$ 时，矩阵方程 $QXK=L$ 有一个解X，另外，X满足 $X=Q^{†}L{K}^{†}+Y-Q^{†}QK{K}^{†}$ ，其中Y是一个适当维数的矩阵。

3. 系统描述

本节主要考虑以下随机离散时间系统：

$\begin{array}{c}x\left(t+1\right)=A_0\left(t,{\theta }_t\right)x\left(t\right)+{\displaystyle \underset{k=1}{\overset{r}{\sum }}{A}_k\left(t,{\theta }_t\right)x\left(t\right)\omega \left(t\right)}+B_0\left(t,{\theta }_t\right)u\left(t\right)\\ +{\displaystyle \underset{k=1}{\overset{r}{\sum }}{B}_k\left(t,{\theta }_t\right)u\left(t\right)\omega \left(t\right)}+\omega \left(t\right)\end{array}$ (2-1)

$a_{i1}{x}_1\left(T\right)+a_{i2}{x}_2\left(T\right)+\cdots +a_{i1}{x}_n\left(T\right)\le {\xi }_i$

其中 $A_0\left(t,{\theta }_t\right)\in {ℝ}^n,A_k\left(t,{\theta }_t\right)\in {ℝ}^n$ ， $B_0\left(t,{\theta }_t\right)$ 和 $B_K\left(t,{\theta }_t\right)$ 是具有适当维数的向量， $x\in R^n$ 是系统的输出状态， $u\in R^m$ 是系统的输入控制， $x_0\in R^n$ 是给定的系统的初始状态， ${\left\{\omega \left(t\right)\right\}}_{t\ge 0}$ 是定义在完备概率空间 $\left(\Omega ,\mathcal{F},\mathcal{P}\right)$ 上的一维独立随机变量。 $\omega \left(t\right)$ 是噪声， $t\in \left\{t_0,t_0+1,\cdots ,T-1\right\},N_T\in \left\{0,1,\cdots ,T\right\}$ 。令 $N_{r\times n}={\left(a_{ij}\right)}_{r\times n},\xi ={\left({\xi }_1,{\xi }_2,\cdots ,{\xi }_r\right)}^{\text{T}}$ ，其中 $a_{ij}$ 是常数。

系统(2-1)所对应的二次指标为：

$\begin{array}{l}\mathcal{J}\left(t_0,x_0;u\left(t_0\right),u\left(t_0+1\right),\cdots ,u\left(T-1\right)\right)\\ =E\left\{{{\displaystyle \underset{t=t_0}{\overset{T-1}{\sum }}\left(\begin{array}{c}x\left(t\right)\\ u\left(t\right)\end{array}\right)}}^{\text{T}}\left(\begin{array}{cc}Q\left(t,{\theta }_t\right)& L\left(t,{\theta }_t\right)\\ {\left(L\left(t,{\theta }_t\right)\right)}^{\text{T}}& R\left(t,{\theta }_t\right)\end{array}\right)\left(\begin{array}{c}x\left(t\right)\\ u\left(t\right)\end{array}\right)+x^{\text{T}}\left(T\right)Mx\left(T\right)\right\}\end{array}$ (2-2)

其中， $\left(\begin{array}{cc}Q\left(t,{\theta }_t\right)& L\left(t,{\theta }_t\right)\\ {\left(L\left(t,{\theta }_t\right)\right)}^{\text{T}}& R\left(t,{\theta }_t\right)\end{array}\right)$ 与M均为对称矩阵。值函数定义为：

$V\left(t_0,x_0\right)=\underset{u\left(t_0\right),u(t_0+1),\cdots ,u\left(T-1\right)}{\inf }\mathcal{J}\left(t_0,x_0;u(\cdot )\right)$ (2-3)

假设1：为方便后续研究，现作出以下假设：

1) 上述矩阵都是有界的矩阵值序列。

2) 对于任意的 $t\ge 0$ ， $P_t={\left(p_t\left(i,j\right)\right)}_{i,j\in \Im \times \Im }$ 是一个非退化的随机矩阵，对于任意的 $\forall i,j\in \Im \times \Im$ 满足：

$\{\begin{array}{c}0\le p_t\left(i,j\right)\le 1\\ {\displaystyle \underset{k=1}{\overset{N}{\sum }}{p}_t\left(i,k\right)=1}\\ {\displaystyle \underset{k=1}{\overset{N}{\sum }}{p}_t\left(k,j\right)>0}\end{array}$ (2-4)

假设2： ${\left\{\omega \left(t\right)\right\}}_{t\ge 0}$ 是定义在完备概率空间 $\left(\Omega ,\mathcal{F},\mathcal{P}\right)$ 上的一维独立随机变量，与初值条件 $x\left(t_0\right)=x_0$ 是相互独立的，且具有以下性质：

$\begin{array}{l}E\left[\omega \left(t\right)\right]=0,\\ E\left[\omega \left(s\right){\left(\omega \left(t\right)\right)}^{\text{T}}\right]={\delta }_{st},\\ E\left[{\left(\omega \left(t\right)\right)}^2\right]=1\end{array}$ (2-5)

其中：

${\delta }_{st}=\{\begin{array}{c}{I}_r,s=t\\ 0,s\ne t\end{array}$ (2-6)

假设3：对于任意的 $t\ge 0$ ， $\sigma$ -代数 ${\mathcal{F}}_t$ 与 $\sigma$ -代数 ${\mathcal{G}}_t$ 是相互独立的，其中， ${\mathcal{F}}_t=\sigma \left\{\omega \left(s\right),0\le s\le t\right\}$ ， ${\mathcal{G}}_t=\sigma \left\{\vartheta \left(s\right),0\le s\le t\right\}$ 。

定义 4：无限随机线性二次问题(2-1)、(2-2)、(2-3)是适定的，若对于任意的 $\left(t_0,x_0\right)\in N_T\times R^n$ ， $V\left(t_0,x_0\right)>-\infty$ 。

定义 5：无限随机线性二次问题(2-1)、(2-2)、(2-3)是可达的，若对于任意的 $\left(t_0,x_0\right)\in N_T\times R^n$ ，存在一个序列 $\left\{u\left(t\right),t=t_0,t_0+1,\cdots ,T-1\right\}$ 为系统的最优控制，使得 $V\left(t_0,x_0\right)=\mathcal{J}\left(t_0,x_0;u\left(t_0+1\right),\cdots ,u\left(T-1\right)\right)$ 。

注记 1：若无限随机线性二次问题(2-1)、(2-2)、(2-3)的一个线性反馈控制是最优的，则必存在一个最优线性反馈控制具有以下形式：

$u\left(t\right)=K\left(t,{\theta }_t\right)x\left(t\right),t\in N_T$

其中， $K\left(t\right)$ 是一个矩阵值函数。

4. 主要成果

在研究中发现，系统(2-1)、(2-2)、(2-3)的最优控制问题可以通过广义Riccati差分方程的解来表示。

4.1. 随机线性系统与Riccati方程

定理1：如果系统(2-1)、(2-2)、(2-3)的最优控制问题是可达的， $u\left(t\right)=K\left(t,{\theta }_t\right)x\left(t\right),t\in N_T$ ，而且正则点 $\left(u^{\ast }\left(t\right),x^{\ast }\left(t\right)\right)$ 是系统(2-1)、(2-2)、(2-3)的局部最优解，则下列广义Riccati方程(GDRE)有解 $\left(P\left(k\right),\mu \right),0\le \mu \in R^1,t\in N_T$ ：

$\begin{array}{c}P\left(t\right)={\left(A_0\left(t,{\theta }_t\right)\right)}^{\text{T}}E\left(t\right)\left[P\left(t+1\right)\right]\left({\theta }_t\right)\times A_0\left(t,{\theta }_t\right)\\ +{\displaystyle \underset{k=1}{\overset{r}{\sum }}{\left(A_k\left(t,{\theta }_t\right)\right)}^{\text{T}}}E\left(t\right)\left[P\left(t+1\right)\right]\left({\theta }_t\right)A_k\left(t,{\theta }_t\right)+Q\left(t,{\theta }_t\right)\\ -{\left(T\left(t,{\theta }_t\right)\right)}^{\text{T}}{\left(W\left(t,{\theta }_t\right)\right)}^{+}T\left(t,{\theta }_t\right)\end{array}$ (2-7)

$\begin{array}{c}T\left(t,{\theta }_t\right)={\left(B_0\left(t,{\theta }_t\right)\right)}^{\text{T}}E\left(t\right)\left[P\left(t+1\right)\right]\left({\theta }_t\right)\times A_0\left(t,{\theta }_t\right)\\ +{\displaystyle \underset{k=1}{\overset{r}{\sum }}{\left(B_k\left(t,{\theta }_t\right)\right)}^{\text{T}}}E\left(t\right)\left[P\left(t+1\right)\right]\left({\theta }_t\right)A_k\left(t,{\theta }_t\right)\end{array}$ (2-8)

$\begin{array}{c}W\left(t,{\theta }_t\right)={\left(B_0\left(t,{\theta }_t\right)\right)}^{\text{T}}E\left(t\right)\left[P\left(t+1\right)\right]\left({\theta }_t\right)\times B_0\left(t,{\theta }_t\right)\\ +{\displaystyle \underset{k=1}{\overset{r}{\sum }}{\left(B_k\left(t,{\theta }_t\right)\right)}^{\text{T}}}E\left(t\right)\left[P\left(t+1\right)\right]\left({\theta }_t\right)B_k\left(t,{\theta }_t\right)+R\left(t,{\theta }_t\right)\end{array}$ (2-9)

$W\left(t,{\theta }_t\right)\ge 0$

$\mu =\frac{Tr\left\{P\left(T\right)-M\right\}}{Tr\left\{N^{T}N\right\}}$ (2-10)

此外，

$\begin{array}{l}{u}^{*}\left(t\right)=\left[-{\left(W\left(t,{\theta }_t\right)\right)}^{†}T\left(t,{\theta }_t\right)+Y\left(t,{\theta }_t\right)-{\left(W\left(t,{\theta }_t\right)\right)}^{†}W\left(t,{\theta }_t\right)Y\left(t,{\theta }_t\right)\right]x\left(t\right)\\ u^{*}\left(t\right)\in R^{m\times n},t\in N_T\end{array}$ (2-11)

$V\left(x_0,{\theta }_t\right)={\displaystyle \underset{t=1}{\overset{T-1}{\sum }}Tr\left\{V\left(t,{\theta }_t\right)E\left(t\right)\left[P\left(t+1\right)\right]\left({\theta }_t\right)\right\}}+x_0^{\text{T}}P\left(0,{\theta }_t\right)x_0-\mu Tr\left\{M\right\}$ (2-12)

证明：对于任意的 $t\in N_T$ ，令 $X\left(t,{\theta }_t\right)=E\left({\left[x\left(t\right)x{\left(t\right)}^{\text{T}}\right]}_{\left\{{\theta }_{t_0}=i\right\}}\right)$ ，其中 $u\left(t\right)=K\left(t,{\theta }_t\right)x\left(t\right),t\in N_T$ ，则线

性二次控制问题(2-1)、(2-2)、(2-3)可以转化成下面的最优化问题：

$\begin{array}{l}J\left(t_0,x_0;u\left(t_0\right),u\left(t_0+1\right),\cdots ,u\left(\infty \right)\right)\\ =E\left\{{\displaystyle \underset{t=t_0}{\overset{\infty }{\sum }}{\left(\begin{array}{c}x\left(t\right)\\ u\left(t\right)\end{array}\right)}^{\text{T}}\left(\begin{array}{cc}Q\left(t,{\theta }_t\right)& L\left(t,{\theta }_t\right)\\ {\left(L\left(t,{\theta }_t\right)\right)}^{\text{T}}& R\left(t,{\theta }_t\right)\end{array}\right)\left(\begin{array}{c}x\left(t\right)\\ u\left(t\right)\end{array}\right)+\underset{T\to \infty }{\lim }{x}^{\text{T}}\left(T\right)Mx\left(T\right)}\right\}\\ ={\displaystyle \underset{t=t_0}{\overset{\infty }{\sum }}E\{{\left(x\left(t\right)\right)}^{\text{T}}Q\left(t,{\theta }_t\right)x\left(t\right)+{\left(K\left(t,{\theta }_t\right)\right)}^{\text{T}}{\left(x\left(t\right)\right)}^{\text{T}}{\left(L\left(t,{\theta }_t\right)\right)}^{\text{T}}x\left(t\right)+{\left(x\left(t\right)\right)}^{\text{T}}L\left(t,{\theta }_t\right)K\left(t,{\theta }_t\right)x\left(t\right)}\\ +{\left(K\left(t,{\theta }_t\right)\right)}^{\text{T}}{\left(x\left(t\right)\right)}^{\text{T}}R\left(t,{\theta }_t\right)x\left(t\right)K\left(t,{\theta }_t\right)+\underset{T\to \infty }{\lim }{x}^{\text{T}}\left(T\right)Mx\left(T\right)\}\\ ={\displaystyle \underset{t=t_0}{\overset{\infty }{\sum }}Tr}\{Q\left(t,{\theta }_t\right)X\left(t\right)+{\left(K\left(t,{\theta }_t\right)\right)}^{\text{T}}{\left(L\left(t,{\theta }_t\right)\right)}^{\text{T}}K\left(t,{\theta }_t\right)X\left(t\right)+{\left(K\left(t,{\theta }_t\right)\right)}^{\text{T}}{\left(L\left(t,{\theta }_t\right)\right)}^{\text{T}}X\left(t\right)\\ +L\left(t,{\theta }_t\right)K\left(t,{\theta }_t\right)X\left(t\right)+\underset{T\to \infty }{\lim }MX\left(T\right)\}\end{array}$ (2-13)

s.t.下式均成立：

$\begin{array}{c}X\left(t+1\right)=A_0\left(t,{\theta }_t\right)X\left(t\right){\left(A_0\left(t,{\theta }_t\right)\right)}^{\text{T}}+A_0\left(t,{\theta }_t\right)X\left(t\right){\left(K\left(t,{\theta }_t\right)\right)}^{\text{T}}{\left(B_0\left(t,{\theta }_t\right)\right)}^{\text{T}}\\ +B_0\left(t,{\theta }_t\right)K\left(t,{\theta }_t\right)X\left(t\right){\left(A_0\left(t,{\theta }_t\right)\right)}^{\text{T}}\\ +B_0\left(t,{\theta }_t\right)K\left(t,{\theta }_t\right)X\left(t\right){\left(K\left(t,{\theta }_t\right)\right)}^{\text{T}}{\left(B_0\left(t,{\theta }_t\right)\right)}^{\text{T}}\\ +{\displaystyle \underset{k=1}{\overset{r}{\sum }}{\left(A_k\left(t,{\theta }_t\right)\right)}^{\text{T}}}X\left(t\right)A_k\left(t,{\theta }_t\right)\\ +{\displaystyle \underset{k=1}{\overset{r}{\sum }}{\left(A_k\left(t,{\theta }_t\right)\right)}^{\text{T}}}X\left(t\right){\left(K\left(t,{\theta }_t\right)\right)}^{\text{T}}{\left(B_k\left(t,{\theta }_t\right)\right)}^{\text{T}}\\ +{\displaystyle \underset{k=1}{\overset{r}{\sum }}{B}_k\left(t,{\theta }_t\right)K\left(t,{\theta }_t\right)}X\left(t\right){\left(K\left(t,{\theta }_t\right)\right)}^{\text{T}}{\left(B_k\left(t,{\theta }_t\right)\right)}^{\text{T}}+V\left(t\right)\end{array}$ (2-14)

$\begin{array}{l}X\left(t_0\right)=E\left[x_0{x}_0^{\text{T}}\right]=x_0{x}_0^{\text{T}}\\ Tr\left(X\left(T\right)N^{\text{T}}N\right)\le TrS,S=E\left[\xi {\xi }^{\text{T}}\right]\end{array}$

即：

$\underset{t\in N_{T-1}}{\min }f\left[X\left(t\right),K\left(t,{\theta }_t\right)\right]$

$\begin{array}{l}\text{s}\text{.t}.\mathcal{H}\left[X\left(t\right),K\left(t,{\theta }_t\right)\right]=0\\ \mathcal{G}\left(X\left(T\right)\right)\le 0\end{array}$

其中， $K\left(1,{\theta }_t\right),K\left(2,{\theta }_t\right),\cdots ,K\left(T-1,{\theta }_t\right)$ 为待定控制项。

$f\left[X\left(t\right),K\left(t,{\theta }_t\right)\right]=J\left(t_0,x_0;u\left(t_0\right),u\left(t_0+1\right),\cdots ,u\left(T-1\right)\right)$ (2-15)

$\begin{array}{l}\mathcal{H}\left[X\left(t\right),K\left(t,{\theta }_t\right)\right]-X\left(t+1\right)=0\\ \mathcal{G}\left(X\left(T\right)\right)=Tr\left(X\left(T\right)N^{\text{T}}N\right)-TrS\end{array}$

根据KKT定理，设拉格朗日函数为：

$\begin{array}{l}\mathcal{L}[X\left(t\right),K\left(t,{\theta }_t\right),P\left(t+1,{\theta }_t\right),\mu ]\\ =f\left[X\left(t\right),K\left(t,{\theta }_t\right)\right]+{\displaystyle \underset{t=t_0}{\overset{\infty }{\sum }}Tr\left\{P\left(t+1,{\theta }_t\right)H\left[X\left(t\right),K\left(t,{\theta }_t\right)\right]\right\}}+\mu g\left(X\left(T\right)\right)\end{array}$ (2-16)

对 $\mathcal{L}\left[X\left(t\right),K\left(t,{\theta }_t\right),P\left(t+1,{\theta }_t\right),\mu \right]$ 中的 $X\left(t\right)$ 进行求导，且根据KKT定理得：

$\frac{\partial \mathcal{L}\left[X\left(t\right),K\left(t,{\theta }_t\right),P\left(t+1,{\theta }_t\right),\mu \right]}{\partial X\left(t\right)}=0$

通过计算可得：若 $X\left(0\right)=0$ ，则有

$TrM\Delta X\left(T\right)-Tr\Delta X\left(T\right)P\left(T,{\theta }_t\right)+Tr\left(\Delta X\left(T\right)N^{\text{T}}N\right)=0$

即：

$P\left(T,{\theta }_t\right)=M+\mu N^{\text{T}}N$ (2-17)

其中， $T\to \infty$ 。

又由于 $\Delta X\left(T\right)$ 与 $\Delta X\left(t\right)$ 相互独立，则有：

$\begin{array}{l}P\left(t,{\theta }_t\right)=Q\left(t,{\theta }_t\right)+{\left(K\left(t,{\theta }_t\right)\right)}^{\text{T}}[R\left(t,{\theta }_t\right)+{\left(B_0\left(t,{\theta }_t\right)\right)}^{\text{T}}E\left(t\right)\left[P\left(t+1\right)\right]\left({\theta }_t\right)B_0\left(t,{\theta }_t\right)\\ +{\displaystyle \underset{k=1}{\overset{r}{\sum }}{\left(B_k\left(t,{\theta }_t\right)\right)}^{\text{T}}}K\left(t,{\theta }_t\right)E\left(t\right)\left[P\left(t+1\right)\right]\left({\theta }_t\right)B_k\left(t,{\theta }_t\right)]K\left(t,{\theta }_t\right)\\ +{[\left(L\left(t,{\theta }_t\right)\right)}^{\text{T}}+{\left(A_0\left(t,{\theta }_t\right)\right)}^{\text{T}}E\left(t\right)\left[P\left(t+1\right)\right]\left({\theta }_t\right)B_0\left(t,{\theta }_t\right)\\ +{\displaystyle \underset{k=1}{\overset{r}{\sum }}{\left(A_k\left(t,{\theta }_t\right)\right)}^{\text{T}}}E\left(t\right)\left[P\left(t+1\right)\right]\left({\theta }_t\right)B_k\left(t,{\theta }_t\right)]K\left(t,{\theta }_t\right)\end{array}$

$\begin{array}{l}+{\left(K\left(t,{\theta }_t\right)\right)}^{\text{T}}[L\left(t,{\theta }_t\right)+{\left(B_0\left(t,{\theta }_t\right)\right)}^{\text{T}}E\left(t\right)\left[P\left(t+1\right)\right]\left({\theta }_t\right)A_0\left(t,{\theta }_t\right)\\ +{\displaystyle \underset{k=1}{\overset{r}{\sum }}{\left(B_k\left(t,{\theta }_t\right)\right)}^{\text{T}}}E\left(t\right)\left[P\left(t+1\right)\right]\left({\theta }_t\right)A_k\left(t,{\theta }_t\right)]\\ +A_0\left(t,{\theta }_t\right)E\left(t\right)\left[P\left(t+1\right)\right]\left({\theta }_t\right){\left(A_0\left(t,{\theta }_t\right)\right)}^{\text{T}}\\ +{\displaystyle \underset{k=1}{\overset{r}{\sum }}{B}_k}\left(t,{\theta }_t\right)E\left(t\right)\left[P\left(t+1\right)\right]\left({\theta }_t\right){\left(B_k\left(t,{\theta }_t\right)\right)}^{\text{T}}\end{array}$ (2-18)

同理，对 $K\left(t,{\theta }_t\right)$ 求导可得：

$\begin{array}{l}[R\left(t,{\theta }_t\right)+{\left(B_0\left(t,{\theta }_t\right)\right)}^{\text{T}}E\left(t\right)\left[P\left(t+1\right)\right]\left({\theta }_t\right)B_0\left(t,{\theta }_t\right)\\ +{\displaystyle \underset{k=1}{\overset{r}{\sum }}{\left(B_k\left(t,{\theta }_t\right)\right)}^{\text{T}}K\left(t,{\theta }_t\right)E\left(t\right)\left[P\left(t+1\right)\right]\left({\theta }_t\right)B_k\left(t,{\theta }_t\right)}]K\left(t,{\theta }_t\right)\\ +{\left(L\left(t,{\theta }_t\right)\right)}^{\text{T}}+{\left(A_0\left(t,{\theta }_t\right)\right)}^{\text{T}}E\left(t\right)\left[P\left(t+1\right)\right]\left({\theta }_t\right)B_0(t,{\theta }_t)\\ +{\displaystyle \underset{k=1}{\overset{r}{\sum }}{\left(A_k\left(t,{\theta }_t\right)\right)}^{\text{T}}E\left(t\right)\left[P\left(t+1\right)\right]\left({\theta }_t\right)B_k\left(t,{\theta }_t\right)}=0\end{array}$ (2-19)

取：

$\begin{array}{c}W\left(t,{\theta }_t\right)=R\left(t,{\theta }_t\right)+{\left(B_0\left(t,{\theta }_t\right)\right)}^{\text{T}}E\left(t\right)\left[P\left(t+1\right)\right]\left({\theta }_t\right)B_0\left(t,{\theta }_t\right)\\ +{\displaystyle \underset{k=1}{\overset{r}{\sum }}{\left(B_k\left(t,{\theta }_t\right)\right)}^{\text{T}}}K\left(t,{\theta }_t\right)E\left(t\right)\left[P\left(t+1\right)\right]\left({\theta }_t\right)B_k\left(t,{\theta }_t\right)\end{array}$ (2-20)

$\begin{array}{c}T\left(t,{\theta }_t\right)={\left(L\left(t,{\theta }_t\right)\right)}^{\text{T}}+{\left(A_0\left(t,{\theta }_t\right)\right)}^{\text{T}}E\left(t\right)\left[P\left(t+1\right)\right]\left({\theta }_t\right)B_0\left(t,{\theta }_t\right)\\ +{\displaystyle \underset{k=1}{\overset{r}{\sum }}{\left(A_k\left(t,{\theta }_t\right)\right)}^{\text{T}}}E\left(t\right)\left[P\left(t+1\right)\right]\left({\theta }_t\right)B_k\left(t,{\theta }_t\right)\end{array}$ (2-21)

综合上述(2-20)、(2-21)则第(2-18)、(2-19)式可写做：

$\begin{array}{c}P\left(t\right)=Q\left(t,{\theta }_t\right)+{\left(K\left(t,{\theta }_t\right)\right)}^{\text{T}}W\left(t,{\theta }_t\right)K\left(t,{\theta }_t\right)+T\left(t,{\theta }_t\right)K\left(t,{\theta }_t\right)\\ +{\left(K\left(t,{\theta }_t\right)\right)}^{\text{T}}{\left(T\left(t,{\theta }_t\right)\right)}^{\text{T}}+A_0\left(t,{\theta }_t\right)E\left(t\right)\left[P\left(t+1\right)\right]\left({\theta }_t\right){\left(A_0\left(t,{\theta }_t\right)\right)}^{\text{T}}\\ +{\displaystyle \underset{k=1}{\overset{r}{\sum }}{B}_k\left(t,{\theta }_t\right)E\left(t\right)\left[P\left(t+1\right)\right]\left({\theta }_t\right){\left(B_k\left(t,{\theta }_t\right)\right)}^{\text{T}}}\end{array}$ (2-22)

$W\left(t,{\theta }_t\right)K\left(t,{\theta }_t\right)+T\left(t,{\theta }_t\right)=0$ (2-23)

根据引理4可得：若 $W\left(t,{\theta }_t\right)K\left(t,{\theta }_t\right)+T\left(t,{\theta }_t\right)=0$ 有解 $K\left(t,{\theta }_t\right)$ ，当且仅当 $W\left(t,{\theta }_t\right){\left(W\left(t,{\theta }_t\right)\right)}^{†}K\left(t,{\theta }_t\right)=-T\left(t,{\theta }_t\right)$ ，则此时方程的解 $K\left(t,{\theta }_t\right)$ 表示如下：

$K\left(t,{\theta }_t\right)=-{\left(W\left(t,{\theta }_t\right)\right)}^{+}T\left(t,{\theta }_t\right)+Y\left(t,{\theta }_t\right)-{\left(W\left(t,{\theta }_t\right)\right)}^{+}W\left(t,{\theta }_t\right)Y\left(t,{\theta }_t\right)$ (2-24)

将 $K\left(t,{\theta }_t\right)$ 带入(2-22)式得以下广义Riccati方程：

$\begin{array}{c}P\left(t\right)=Q\left(t,{\theta }_t\right)+A_0\left(t,{\theta }_t\right)E\left(t\right)\left[P\left(t+1\right)\right]\left({\theta }_t\right){\left(A_0\left(t,{\theta }_t\right)\right)}^{\text{T}}\\ +{\displaystyle \underset{k=1}{\overset{r}{\sum }}{B}_k\left(t,{\theta }_t\right)E\left(t\right)\left[P\left(t+1\right)\right]\left({\theta }_t\right){\left(B_k\left(t,{\theta }_t\right)\right)}^{\text{T}}}\\ +{\left(T\left(t,{\theta }_t\right)\right)}^{\text{T}}{\left(W\left(t,{\theta }_t\right)\right)}^{+}T\left(t,{\theta }_t\right)\end{array}$ (2-25)

即为(2-7)式。

$\begin{array}{c}W\left(t,{\theta }_t\right)=R\left(t,{\theta }_t\right)+{\left(B_0\left(t,{\theta }_t\right)\right)}^{\text{T}}E\left(t\right)\left[P\left(t+1\right)\right]\left({\theta }_t\right)B_0\left(t,{\theta }_t\right)\\ +{\displaystyle \underset{k=1}{\overset{r}{\sum }}{\left(B_k\left(t,{\theta }_t\right)\right)}^{\text{T}}}K\left(t,{\theta }_t\right)E\left(t\right)\left[P\left(t+1\right)\right]\left({\theta }_t\right)B_k\left(t,{\theta }_t\right)\end{array}$ (2-26)

即为(2-9)式。

$\begin{array}{c}T\left(t,{\theta }_t\right)={\left(L\left(t,{\theta }_t\right)\right)}^{\text{T}}+{\left(A_0\left(t,{\theta }_t\right)\right)}^{\text{T}}E\left(t\right)\left[P\left(t+1\right)\right]\left({\theta }_t\right)B_0\left(t,{\theta }_t\right)\\ +{\displaystyle \underset{k=1}{\overset{r}{\sum }}{\left(A_k\left(t,{\theta }_t\right)\right)}^{\text{T}}}E\left(t\right)\left[P\left(t+1\right)\right]\left({\theta }_t\right)B_k\left(t,{\theta }_t\right)\end{array}$ (2-27)

即为(2-10)式。

$\mu =\frac{Tr\left\{P\left(T,{\theta }_t\right)-M\right\}}{Tr\left\{N^{T}N\right\}}$

且得到：

$K\left(t,{\theta }_t\right)=-{\left(W\left(t,{\theta }_t\right)\right)}^{+}T\left(t,{\theta }_t\right)+Y\left(t,{\theta }_t\right)-{\left(W\left(t,{\theta }_t\right)\right)}^{+}W\left(t,{\theta }_t\right)Y\left(t,{\theta }_t\right)$ (2-28)

由于 $u\left(t\right)=K\left(t,{\theta }_t\right)x\left(t\right),t\in N_T$ ，故有：

$\begin{array}{l}{u}^{*}\left(t\right)=\left[-{\left(W\left(t,{\theta }_t\right)\right)}^{+}T\left(t,{\theta }_t\right)+Y\left(t,{\theta }_t\right)-{\left(W\left(t,{\theta }_t\right)\right)}^{+}W\left(t,{\theta }_t\right)Y\left(t,{\theta }_t\right)\right]x\left(t\right)\\ u^{*}\left(t\right)\in R^{m\times n},t\in N_T\end{array}$ (2-29)

即为(2-11)式。

假设 $P\left(t+1,{\theta }_t\right)$ 是一个对称矩阵，若 $P\left(t+1,{\theta }_t\right)$ 不是对称矩阵，则取：

$\tilde{P}\left(t+1,{\theta }_t\right)=\frac{P\left(t+1,{\theta }_t\right)+{\left(P\left(t+1,{\theta }_t\right)\right)}^{\text{T}}}{2}$

显然， $\tilde{P}\left(t+1,{\theta }_t\right)$ 也是一个对称矩阵。

$\begin{array}{l}{\displaystyle \underset{t=0}{\overset{T-1}{\sum }}E\left[{\left(x\left(t+1\right)\right)}^{\text{T}}E\left(t\right)\left[P\left(t+1\right)\right]\left({\theta }_t\right)x\left(t+1\right)-{\left(x\left(t\right)\right)}^{\text{T}}E\left(t\right)\left[P\left(t\right)\right]\left({\theta }_t\right)x\left(t\right)\right]}\\ =E\left[{\left(x\left(T\right)\right)}^{\text{T}}E\left(t\right)\left[P\left(T\right)\right]\left({\theta }_T\right)x\left(T\right)-{\left(x\left(0\right)\right)}^{\text{T}}E\left(t\right)\left[P\left(0\right)\right]\left({\theta }_0\right)x\left(0\right)\right]\end{array}$ (2-30)

将第(2-30)式变形得到：

$\begin{array}{l}{\displaystyle \underset{t=0}{\overset{T-1}{\sum }}E\left[{\left(x\left(t+1\right)\right)}^{\text{T}}E\left(t\right)\left[P\left(t+1\right)\right]\left({\theta }_t\right)x\left(t+1\right)-{\left(x\left(t\right)\right)}^{\text{T}}E\left(t\right)\left[P\left(t\right)\right]\left({\theta }_t\right)x\left(t\right)\right]}\\ -E\left[{\left(x\left(T\right)\right)}^{\text{T}}E\left(t\right)\left[P\left(T\right)\right]\left({\theta }_T\right)x\left(T\right)+{\left(x\left(0\right)\right)}^{\text{T}}E\left(t\right)\left[P\left(0\right)\right]\left({\theta }_0\right)x\left(0\right)\right]\\ =0\end{array}$ (2-31)

将第(2-31)式与第(2-2)式相加，得：

$\begin{array}{l}J\left(t_0,x_0;u\left(t_0\right),u\left(t_0+1\right),\cdots ,u\left(T-1\right)\right)\\ =E\left\{{\displaystyle \underset{t=t_0}{\overset{T-1}{\sum }}{\left(\begin{array}{c}x\left(t\right)\\ u\left(t\right)\end{array}\right)}^{\text{T}}\left(\begin{array}{cc}Q\left(t,{\theta }_t\right)& L\left(t,{\theta }_t\right)\\ {\left(L\left(t,{\theta }_t\right)\right)}^T& R\left(t,{\theta }_t\right)\end{array}\right)\left(\begin{array}{c}x\left(t\right)\\ u\left(t\right)\end{array}\right)+x^{\text{T}}\left(T\right)Mx\left(T\right)}\right\}\\ +{\displaystyle \underset{t=0}{\overset{T-1}{\sum }}E\left[{\left(x\left(t+1\right)\right)}^{\text{T}}E\left(t\right)\left[P\left(t+1\right)\right]\left({\theta }_t\right)x\left(t+1\right)-{\left(x\left(t\right)\right)}^{\text{T}}E\left(t\right)\left[P\left(t\right)\right]\left({\theta }_t\right)x\left(t\right)\right]}\\ -E\left[{\left(x\left(T\right)\right)}^{\text{T}}E\left(t\right)\left[P\left(T\right)\right]\left({\theta }_T\right)x\left(T\right)-{\left(x\left(0\right)\right)}^{\text{T}}P\left(0,{\theta }_0\right)x\left(0\right)\right]\end{array}$