设为首页
加入收藏
期刊导航
网站地图
首页
期刊
数学与物理
地球与环境
信息通讯
经济与管理
生命科学
工程技术
医药卫生
人文社科
化学与材料
会议
合作
新闻
我们
招聘
千人智库
我要投稿
办刊
期刊菜单
●领域
●编委
●投稿须知
●最新文章
●检索
●投稿
文章导航
●Abstract
●Full-Text PDF
●Full-Text HTML
●Full-Text ePUB
●Linked References
●How to Cite this Article
AdvancesinAppliedMathematics
A^
ê
Æ
?
Ð
,2023,12(1),183-202
PublishedOnlineJanuary2023inHans.https://www.hanspub.org/journal/aam
https://doi.org/10.12677/aam.2023.121022
Ä
u
F
Ý
6
Ä
Ú
BB
Ú
•
S
“
Â
K
Š
©
Û
hŽ
{
©©©
www
§§§
$$$
½½½
777
∗
B
²
Œ
Æ
§
ê
Æ
†
Ú
O
Æ
§
B
²
B
Â
v
F
Ï
µ
2022
c
12
28
F
¶
¹
^
F
Ï
µ
2023
c
1
23
F
¶
u
Ù
F
Ï
µ
2023
c
1
30
F
Á
‡
©
ï
Ä
Û
h
o
e
‘
k
š
à
K
²
º
x
4
z
¯
K
§
Ù
¥
›
”
¼
ê´
à
¼
ê
§
K
‘
•
MCP
¼
ê
"
J
Ñ
Ä
u
F
Ý
6
Ä
Ú
Barzilar-Borwein(BB)
Ú
•
S
“
Â
K
Š
©
Û
hŽ
{
(ISTDP)
"
Ä
k
§
Ä
u
Ž
{
z
g
S
“
þ
é
F
Ý
V
\
p
d
D
(
§
y
²
T
Ž
{
ä
k
©
Û
h
o
5
Ÿ
"
Ù
g
§
Ä
u
±
BB
Ú
•
‰
Á
&
Ú
?
1
‚
|¢
S
“
Â
K
Š
Ž
{
§
y
²
T
Ž
{
Œ
±
Â
ñ
u
?
¿
‰
½
°
Ý
"
Ï
d
§
ISTDP
Ž
{
´
˜
«
Œ
±
÷
v
Û
h
o
‡
¦
Å
ì
Æ
S
`
z
Ž
{
"
'
…
c
©
Û
h
o
§
F
Ý
6
Ä
§
Â
K
Š
Ž
{
§
MCP
K
IterativeShrinkThresholdDifferential
PrivacyAlgorithmBasedonGradient
PerturbationandBBStepSize
WenliYuan,DingtaoPeng
∗
∗
Ï
Õ
Š
ö
"
©
Ù
Ú
^
:
©
w
,
$
½
7
.
Ä
u
F
Ý
6
Ä
Ú
BB
Ú
•
S
“
Â
K
Š
©
Û
hŽ
{
[J].
A^
ê
Æ
?
Ð
,2023,12(1):
183-202.DOI:10.12677/aam.2023.121022
©
w
§
$
½
7
SchoolofMathematicsandStatistics,GuizhouUniversity,GuiyangGuizhou
Received:Dec.28
th
,2022;accepted:Jan.23
rd
,2023;published:Jan.30
th
,2023
Abstract
Inthispaper,westudytheproblemofempiricalriskminimizationwithnonconvex
regularizationunderprivacyprotection,wherethelossfunctionisaconvexfunction
and the regular term is theMCP function.An iterative shrinkage threshold difference
privacyalgorithm(ISTDP)basedongradientperturbationandBarzir-Borwein(BB)
stepsizeisproposed.First,ISTDPalgorithmisprovedtohavethepropertyof
differentialprivacyprotectionsinceGaussnoiseisaddedtothegradientforeach
iterationinthealgorithm.Secondly,itisprovedthattheISTDPalgorithmcan
convergeatanygivenaccuracyduetothefactthatISTDPalgorithmadoptsan
iterative shrinkage thresholdalgorithmtogether withalinesearch beginningwithBB
stepsize.Therefore,ISTDPalgorithmisakindofmachinelearningoptimization
algorithmwhichcansatisfytherequirementofprivacyprotection.
Keywords
DifferentialPrivacyProtection,GradientPerturbation,ShrinkageThreshold
Algorithm,MCPRegularization
Copyright
c
2023byauthor(s)andHansPublishersInc.
This work is licensed undertheCreative Commons Attribution InternationalLicense(CCBY4.0).
http://creativecommons.org/licenses/by/4.0/
1.
Ú
ó
3
Œ
ê
â
†
<
ó
œ
U
ž
“
,
|
„
Ú
Å
Â
8
ê
â
´
8
&
E
ž
“
'
…
]
.
ù
ê
â
8
DOI:10.12677/aam.2023.121022184
A^
ê
Æ
?
Ð
©
w
§
$
½
7
Œ
U
´
¯
•
¿
•
¹
¯
a
&
E
,
ù
Û
h
&
E
³
é
‡
<
Û
h
¤
î
-
%
,
l
¦
J
ø
˜
‡
ˆ
Û
h
o
Ó
ž
±
ê
â
Œ
^
5
Ž
{
¤
•
˜
‡
-
‡
¯
K
[1].2006
c
‡
^
ú
i
Dwork
J
Ñ
©
Û
h
(DifferentialPrivacy,
{
¡
DP)
V
g
[2–4],
Ò
´
•
Û
h
ê
â
©
Û
þ
½
›
Û
h
V
g
,
Ù
8
´
3
|
^
Û
h
ê
â
N
&
E
Ó
ž
o
z
‡‡
N
&
E
.
ä
N
5
`
,
Ò
´
3
˜
g
Ú
O
Î
ê
â
8
¥
O
\
½
~
˜
^
P
¹
,
Œ
¼
A
ƒ
Ó
Ñ
Ñ
[5].
•
Ò
´
`
?
Û
˜
^
P
¹
,
§
3
Ø
3
ê
â
8
¥
,
é
(
J
K
•
Œ
Ñ
Ø
O
,
l
Ã
{
l
(
J
¥
„
Ñ
?
Û
˜
^
©
P
¹
.
•
y
‡
<
Û
h
Ø
³
,
Ó
ž
,
q
U
¦
©
Û
hŽ
{
Ñ
Ñ
(
J
¦
þ
O
(
.
©
Û
h
.
ï
~
†
²
º
x
4
z
E
â
ƒ
(
Ü
,
˜
„
²
º
x
4
z
.
X
e
[6–8]:
min
θ
L
(
θ
;
D
) :=
1
n
n
X
i
=1
`
(
θ,z
i
)
Ù
¥
D
=
{
z
1
,
···
,z
n
}
•
•
¹
n
^
¯
a
ê
â
z
i
=
{
x
i
,y
i
}∈
R
d
+1
ê
â
8
.
`
(
θ,z
i
)
´
Š
^u
ê
â
z
i
þ
›
”
¼
ê
,
L
(
θ
;
D
):
R
d
→
R
L
«
Š
^u
‡
ê
â
8
D
þ
›
”
¼
ê
,
§
´ê
â
8
D
þ
¤
k
ê
â
z
i
ƒ
A
›
”
¼
ê
`
(
θ,z
i
)
²
þ
Š
,
Ø
”
˜
„
5
,
§
´
Y
²
k
.
.
•
;
•
þ
ã
.
L
[
Ü
,
˜
„
I
‡
\
\
K
‘
,
=
X
e
K
z
²
º
x
4
z
.
:
min
θ
F
(
θ
;
D
) :=
L
(
θ
;
D
)+
R
(
θ
)
.
(1)
Ù
¥
R
(
θ
)
´
K
¼
ê
,
~
R
(
θ
) =
λ
k
θ
k
1
½
ö
λ
k
θ
k
2
2
[9],
λ>
0
´
K
z
ë
ê
.
é
.
(1),
›
”
¼
ê
Ú
K
¼
ê
þ
•
à
¼
êž
,
÷
v
©
Û
h
o
Ž
{
8
c
~
^
k
n
«
µ
1
˜
«
´
Ä
u
Ñ
Ñ
6
Ä
Ž
{
[9,10],
Ì
‡
´
é
ê
â
u
Ù
¥
Î
(
JÚ
Å
ì
Æ
S
Ž
{
Ñ
Ñ
(
J
V
\
D
(
,
du
Ñ
Ñ
´
3
•
¹
k
Û
h
ê
â
þ
,
Ï
d
Ñ
Ñ
B
k
Œ
U
³
^
r
Û
h
[1];
1
«
´
Ä
u
8
I
6
Ä
Ž
{
[11],
Ì
‡
´
é
©
Û
h
Æ
S
µ
e
e
8
I
¼
ê
V
\
D
(
,
•
Œ
±
y
Û
h
5
,
‡
3
8
I
6
Ä
œ
¹
e
¼
•
`
)
´
5
Ã
;
1
n
«
´
Ä
u
F
Ý
6
Ä
Ž
{
[12–15],
´
•
3
Ž
{
S
“
L
§
¥
é
z
˜
S
“
Ú
¥
F
Ý
V
\
D
(
,
ï
Ä
L
²
Ð
Ž
{
Œ
±
Ž
Ñ
Û
h
³
Ú
O
Ž
¯
K
[16].
K
¼
ê
R
(
θ
)
´
•
“
Ž
L
[
Ü
Ú
\
,
R
(
θ
)=
k
θ
k
1
½
ö
k
θ
k
2
2
K
¯
K
)
%
´
k
,
=
O
þ
ê
Æ
Ï
"
Ø
u
O
ë
ê
ý
¢
Š
.
Ï
d
,
X
Û
À
J
K
¼
ê
,
¦
T
K
z
¯
K
Q
U
^u
“
Ž
L
[
Ü
q
U
¦
(
J
•
O
(
´
©
•
Ä
-
:
.Fan
Ú
Li[17]
•
Ñ
˜
‡
Ð
K
¼
ê
A
¦
)
O
þ
ä
k
e
ã
o
‡
5
Ÿ
:(1)
Ã
5
,(2)
D
Õ
5
,(3)
ë
Y5
,(4)
Oracle
5
Ÿ
.
ï
Ä
ö
®
²
y
²
µ
3
•
¦
›
”
œ
¹
e
,
ò
U
]
(
š
à
)
K
¼
ê
SCAD[17,18],
MCP[19]
Ú
Capped-
`
1
[20–25]
)
O
þ
ä
k
þ
ã
5
Ÿ
[26,27].
É
[17,19,20,24,25,27–30]
š
à
K
é
u
,
©
?
Ø
±
e
©
Û
h
o
e
‘
k
MCP
K
‘
DOI:10.12677/aam.2023.121022185
A^
ê
Æ
?
Ð
©
w
§
$
½
7
²
º
x
4
z
¯
K
:
min
θ
F
(
θ
;
D
) :=
L
(
θ
;
D
)+
R
(
θ
)
,
(2)
Ù
¥
D
=
{
z
1
,...,z
n
}∈
Z
n
L
«
‘
k
n
^
P
¹
ê
â
8
,
Ù
¥
Z
⊆
R
d
+1
¡
•
ê
â
Š
˜
m
,
z
i
=(
x
i
,y
i
)
∈
R
d
+1
,
i
∈
[
n
]=
{
1
,
2
,...,n
}
¡
•
ê
â
P
¹
,
¿
…
z
˜
^
P
¹
é
A
˜
‡
ê
â
z
ö
,
…
•
¹
T
^
r
¤
k
&
E
,
n
L
«ê
â
8
¥
•
¹
P
¹
^
ê
,
x
i
∈
R
d
¡
•
ê
â
z
ö
5
Ÿ
&
E
,
y
i
∈{
0
,
1
}⊂
R
¡
•
^
r
I
\
.
L
(
θ
;
D
)
´
1
w
à
¼
ê
…
ä
k
F
Ý
Lipschitz
ë
Y
,
=
•
3
˜
‡
~
ê
L>
0,
¦
k∇
L
(
w,D
)
−∇
L
(
u,D
)
k≤
L
k
w
−
u
k
,
∀
w,u
∈
R
d
.
R
(
θ
)
MCP(MinimaxConcavePenalty)
K
¼
ê
,
=
R
(
θ
) =
P
d
i
=1
φ
(
θ
i
),
Ù
¥
φ
(
t
) :=
λ
|
t
|−
t
2
/
(2
α
)
,
|
t
|≤
αλ,
αλ
2
/
2
,
|
t
|
>αλ,
λ>
0
,α>
1
.
Å
ì
Æ
S
¥
N
õ
¯
K
Ñ
·
^u
±
þ
.
,
Ï
L
À
Ø
Ó
›
”
¼
ê
Œ
±
¢
y
Ø
Ó
Æ
S
?
Ö
,
'
X
~
„
Ü
6
£
8
[11]
!
Ø
•{
[31]
.
©
Ì
‡
(
X
e
µ
1
Ü
©
¥
,
©
J
Ñ
Ä
u
F
Ý
6
Ä
Ú
Barzilai-Borwein(BB)
Ú
•
S
“
Â
K
Š
©
Û
h
(ISTDP)
Ž
{
.
1
n
Ü
©
é
¤
J
Ñ
ISTDP
Ž
{
?
1
n
Ø
©
Û
,
y
²
ISTDP
Ž
{
Ø
=
÷
v
©
Û
h
o
‡
¦
,
„
Œ
±
Â
ñ
?
¿
‰
½
°
Ý
.
1
o
Ü
©
?
1
{
ü
o
(
.
Î
Ò
`
²
:
±
e
Î
Ò0
B
©
,
∀
t
∈
R
,
|
t
|
L
«
t
ý
é
Š
.
k
x
k
1
L
«
•
þ
x
`
1
‰
ê
.
k
x
k
L
«
•
þ
x
`
2
‰
ê
.
g
k
=
∇
L
θ
k
;
D
L
«
›
”
¼
ê
3
θ
k
?
F
Ý
.
θ
∗
L
«
8
I
¼
ê
•
`
)
.Pr
L
«
V
Ç
.sign(
t
)
•
Î
Ò¼
ê
sign(
t
) :=
1
,t>
0
,
0
,t
= 0
,
−
1
,t<
0
.
2.
Ž
{
µ
e
!
J
Ñ
˜
«
Ê
·
Û
h
o
Ž
{
))
Ä
u
F
Ý
6
Ä
Ú
BB
Ú
•
S
“
Â
K
Š
©
Û
hŽ
DOI:10.12677/aam.2023.121022186
A^
ê
Æ
?
Ð
©
w
§
$
½
7
{
(IterativeShrinkageThresholdDifferentialPrivacyAlgorithmbasedonGradientPerturbation
andBarzilar-BorweinStepSize,
{
¡
ISTDP).
3
ISTDP
Ž
{
z
˜
S
“
Ú
,
é
F
Ý
g
k
V
\
Ñ
l
p
d
©
Ù
N
(0
,σ
2
I
)
D
(
,
1
k
Ú
S
“
Û
h
ý
Ž
•
ε
k
,
Û
h
ë
ê
•
δ
k
ž
,
D
(
F
Ý
ξ
k
=
g
k
+
b
k
,
b
k
∼
N
0
,σ
2
I
.
d
ž
Œ
y
D
(
F
Ý
ξ
k
(
½
L
§
÷
v
(
ε
k
,δ
k
)
−
©
Û
h
(
½
Â
3.3).
Ð
Ú
•
•
#
ü
Ñ
Œ
±
ŒŒ
~
‚
|¢
s
¤
ž
m
,
é
Ž
{
¯
„
Â
ñ
–
'
-
‡
.Barzilai-
Borwein(BB)
Ú
•
5
K
[32]
®
²
y
²
´
š
~
k
Ú
•
ü
Ñ
.
Ï
d
,ISTDP
Ž
{
æ
^
BB
Ú
•
Š
•
z
g
S
“
Á
&
Ú
,
=
α
BB
k
=
d
k
θ
T
d
k
θ
d
k
θ
T
d
k
g
Ù
¥
d
k
θ
=
θ
k
−
θ
k
−
1
,
d
k
g
=
g
k
−
g
k
−
1
.
é
.
(2),ISTDP
Ž
{
Ï
L
¦
)
e
ã
f
¯
K
5
•
#
θ
:
θ
k
+1
= argmin
θ
Q
(
α
BB
k
,θ
k
,θ
) :=
L
θ
k
;
D
+
h
ξ
k
,θ
−
θ
k
i
+
1
2
α
BB
k
θ
−
θ
k
2
+
R
(
θ
)
.
(3)
¯
K
(3)
d
u
±
e
C
:
¯
K
µ
θ
k
+1
= argmin
θ
1
2
θ
−
u
k
2
+
α
BB
k
R
(
θ
)
,
(4)
Ù
¥
u
k
=
θ
k
−
α
BB
k
ξ
k
.
5
¿
k·k
2
Ú
R
(
θ
)
Œ
©
5
,
¯
K
(4)
Œ
±
d
/
©
)
•
d
‡
Õ
á
ü
C
þ
`
z
¯
K
µ
θ
k
+1
i
= argmin
θ
i
{
h
(
θ
i
,u
k
i
) :=
1
2
(
θ
i
−
u
k
i
)
2
+
α
BB
k
φ
i
(
θ
i
)
}
,i
= 1
,
2
,
···
,d.
DOI:10.12677/aam.2023.121022187
A^
ê
Æ
?
Ð
©
w
§
$
½
7
Ù
¥
u
k
i
=
θ
k
i
−
α
BB
k
ξ
k
i
.
ù
d
‡
ü
C
þ
`
z
¯
K
ä
k
ƒ
Ó
(
,
•
{
z
Î
Ò
,
·
‚
Ï
L
í
Ø
þ
e
I
5
n
¤
X
e
Ú
˜
/
ª
µ
t
(
s
) = argmin
t
{
h
(
t,s
) :=
1
2
(
t
−
s
)
2
+
γφ
(
t
)
}
.
Š
â
[30],
þ
ã
¯
K
ä
k
X
e
4
ª
)
µ
t
(
s
) =
t
1
,h
1
(
t
1
,s
)
≤
h
2
(
t
2
,s
)
,
t
2
,h
1
(
t
1
,s
)
>h
2
(
t
2
,s
)
.
Ù
¥
t
1
= argmin
t
h
1
(
t,s
) :=
1
2
(
t
−
s
)
2
+
λγ
|
t
|−
t
2
2
α
s.t.
|
t
|≤
αλ
,
t
2
= argmin
t
h
2
(
t,s
) :=
1
2
(
t
−
s
)
2
+
α
(
λγ
)
2
2
s.t.
|
t
|≥
αλ
.
•
ä
N
,
Ï
L
O
Ž
Œ
±
t
1
= sign(
s
)
z,
t
2
= sign(
s
)max(
αλ,
|
s
|
)
,
ù
p
z
= argmin
t
∈C
1
2
(
t
−|
s
|
)
2
+
λγt
−
t
2
2
α
,
Ù
¥
C
=
0
,αλ,
min
αλ,
max
0
,
α
(
|
s
|−
γλ
)
α
−
1
,
α
6
= 1
ž
{
0
,αλ
}
,
Ä
K
.
S
.
þ
•
¡
t
(
s
)
•
Â
K
Š
¼
ê
.
3
ISTDP
Ž
{
¥
,
Ø
^
BB
Ú
•
‰
Á
&
Ú
,
ï
Æ
É
Ú
•
I
÷
v
±
ee
ü
Y
²
[30]
µ
E
[
F
(
θ
k
+1
;
D
)]
≤
E
[
F
(
θ
k
;
D
)]
−
βα
BB
k
2
E
[
θ
k
+1
−
θ
k
2
]
,β
∈
(0
,
1)
,
(5)
´
„
,
3
d
Ú
•
O
K
ƒ
e
,
8
I
¼
ê
Š
Ï
"
´
ü
N4
~
.
¦
)
¯
K
(2)
ISTDP
Ž
{
µ
e
X
e
µ
DOI:10.12677/aam.2023.121022188
A^
ê
Æ
?
Ð
©
w
§
$
½
7
Ž
{
1
(ISTDP
Ž
{
)
•
Ñ
\
:
ê
â
8
D
=
{
z
1
,z
2
,...,z
n
}
.
•
À
J
ë
ê
:
T>
0,
η
∈
(0
,
1),
β
∈
(0
,
1)
Ú
÷
v
0
<α
min
<α
max
α
min
,
α
max
.
•
Ð
©
z
:
À
θ
0
,θ
1
∈
R
d
,
θ
0
6
=
θ
1
,
O
Ž
g
0
=
∇
θ
L
(
θ
0
;
D
),
k
= 1;
•
Ì
‚
:
while
k<T
,do
1.1.
O
Ž
F
Ý
:
g
k
=
∇
θ
L
θ
k
;
D
,
‰
F
Ý
V
\
D
(
:
ξ
k
=
g
k
+
b
k
,b
k
∼
N
(0
,σ
2
I
);
1.2.
O
Ž
BB
Ú
•
:
d
k
θ
=
θ
k
−
θ
k
−
1
,d
k
g
=
g
k
−
g
k
−
1
,
α
BB
k
=
d
k
θ
T
d
k
θ
d
k
θ
T
d
k
g
,α
BB
k
:= min
{
max
{
α
BB
k
,α
min
}
,α
max
}
;
1.3.
S
Ì
‚
:
•
#
θ
k
+1
1.3.1.
O
Ž
:
ˆ
θ
k
+1
= argmin
θ
{
Q
(
α
BB
k
,θ
k
,θ
) =
L
(
θ
k
;
D
)+
h
ξ
k
,θ
−
θ
k
i
+
1
2
α
BB
k
k
θ
−
θ
k
k
2
+
R
(
θ
)
}
;
1.3.2.
X
J
ˆ
θ
k
+1
÷
v
:
E
(
F
(
θ
k
;
D
)
−
F
(
ˆ
θ
k
+1
;
D
))
≥
βα
BB
k
2
E
k
ˆ
θ
k
+1
−
θ
k
k
2
,
K
θ
k
+1
=
ˆ
θ
k
+1
,
Ê
Ž
S
Ì
‚
,
=
Ú
1.4;
Ä
K
,
α
BB
k
=
ηα
BB
k
,
=
Ú
1.3.1;
1.4.
•
#
S
“
Ú
:
k
:=
k
+1,
=
Ú
1.1;
•
end
while;
•
Ñ
Ñ
:
θ
priv
=
θ
T
.
3.
n
Ø
©
Û
!
©
Û
ISTDP
Ž
{
n
Ø
5
Ÿ
,
ò
y
²
§
Ø
=
´
˜
‡
÷
v
©
Û
h
‡
¦
Ž
{
,
Ó
ž
•
´Â
ñ
,
§
Ñ
Ñ
(
J
Œ
±
÷
v
?
Û
‰
½
°
Ý
‡
¦
.
3.1.
Ž
{
1
Û
h
o
5
e
¡
k
0
©
Û
h
o
Ä
V
g
Ú
(
Ø
,
2
ï
Ä
Ž
{
1
©
Û
h
o
5
Ÿ
.
½
Â
3.1
[6,33]
‰
½
ê
â
8
D,D
0
∈
Z
n
,
e
|
(
D
\
D
0
)
∪
(
D
0
\
D
)
|
= 1
,
K
¡
D,D
0
∈
Z
n
•
ƒ
ê
â
8
,
P
•
D
∼
D
0
.
5
:
e
ê
â
8
D,D
0
∈
Z
n
´
ƒ
,
K
ê
â
8
D
0
Œ
±
Ï
L
3
ê
â
8
D
þ
V
\
½
ö
í
Ø
˜
^
P
¹
.
½
Â
3.2
[5,33]
‰
½
?
¿
ü
‡
ƒ
ê
â
8
D,D
0
∈
Z
n
,
M
•
3
ê
â
8
D,D
0
∈
Z
n
þ
$
1
˜
‡
‘
ÅÅ
›
,
P
M
•
M
¤
k
Œ
U
Ñ
Ñ
(
J
¤
8
Ü
.
e
é
?
¿
f
8
S
⊆
P
M
÷
v
Pr[
M
(
D
)
∈
S
]
≤
e
ε
Pr
h
M
D
0
∈
S
i
,
DOI:10.12677/aam.2023.121022189
A^
ê
Æ
?
Ð
©
w
§
$
½
7
Ù
¥
ε
¡
•
Û
h
ý
Ž
,
K
¡
Å
›
M
÷
v
X
©
Û
h
,
P
•
ε
-DifferentialPrivacy,
{
¡
ε
−
DP
.
©
Û
h
Œ
±
y
ê
â
Û
h
ô
Â
ö
Ã
{
«
©
ƒ
ê
â
8
,
?
¢
y
3
ê
â
?
n
L
§
¥
Ø
³
?
Û
k
'
ê
â
8
¥
?
¿˜
‡
^
r
&
E
8
.
½
Â
3.3
[8]
‰
½
?
¿
ü
‡
ƒ
ê
â
8
D,D
0
∈
Z
n
,
M
•
3
ê
â
8
D,D
0
∈
Z
n
þ
$
1
˜
‡
‘
Å
Å
›
,
P
M
•
M
¤
k
Œ
U
Ñ
Ñ
(
J
¤
8
Ü
.
e
é
?
¿
f
8
S
⊆
P
n
÷
v
Pr[
M
(
D
)
∈
S
]
≤
e
ε
Pr[
M
(
D
0
)
∈
S
]+
δ,
Ù
¥
ε
¡
•
Û
h
ý
Ž
,
δ>
0
¡
•
X
©
Û
h
Š
‡
§
Ý
(
˜
„
C
u
0
Š
),
K
¡
Å
›
M
÷
v
C
q
©
Û
h
,
P
•
(
ε,δ
)
-DifferentialPrivacy,
{
¡
(
ε,δ
)
−
DP
.
C
q
©
Û
h
V
g
´
é
X
©
Û
h
í
2
,
…
ä
k
•
2
•
A^
.
½
Â
3.4
[1]
‰
½
¼
ê
f
:
Z
n
→
R
d
,
f
L
i
¯
a
Ý
∆
i
(
f
)(
i
= 1
,
2)
½
Â
X
e
∆
i
(
f
) =max
D
∼
D
0
k
f
(
D
)
−
f
(
D
0
)
k
i
,
Ù
¥
D,D
0
∈
Z
n
•
?
¿
ƒ
ê
â
8
.
¯¢
þ
,
¯
a
5
•
x
¼
ê
f
Š
^
3
?
Û
ƒ
ê
â
8
e
Ñ
Ñ
Šƒ
m
•
Œ
C
z
.
3
Ø
—
Ú
å
·
Ï
œ
¹
e
,
ò
¯
a
5
¤
∆(
f
).
ï
Ä
ö
~~
Ä
u
¼
ê
¯
a
Ý
∆(
f
)
Ú
Û
h
ý
Ž
ε
)
¤
D
(
,
?
ï
á
ƒ
A
©
Û
h
oÅ
›
.
½
n
3.1
‰
½
Š
^u
ê
â
8
D
⊂
Z
n
¼
ê
f
:
D
→
R
d
,
∀
ε
∈
(0
,
1)
,
e
‘
ÅÅ
›
M
÷
v
M
(
D
) =
f
(
D
)+
N
0
,σ
2
I
,
Ù
¥
σ
÷
v
σ
≥
∆
2
(
f
)
ε
s
2log
1
.
25
δ
∆
2
(
f
)
•
¼
ê
f
L
2
¯
a
Ý
,
N
(0
,σ
2
I
)
L
«
d
‘
p
d
©
Ù
,
K
Å
›
M
÷
v
(
ε,δ
)
−
DP
.
y
²
.
k
?
Ø
d
=1
œ
/
.
Š
â
.
(2)
Œ
•
,
d
ž
é
¢
Š
¼
ê
f
:
Z
n
→
R
V
\
Î
Ü
I
O
©
Ù
D
(
.
Š
â
¯
a
ݽ
Â
3.4,∆(
f
)=∆
1
(
f
)=∆
2
(
f
)=max
D
∼
D
0
|
f
(
D
)
−
f
(
D
0
)
|
.
D
(
x
∼
N
(0
,σ
2
),
Š
â
©
z
[33],
d
ž
Û
h›
”
•
DOI:10.12677/aam.2023.121022190
A^
ê
Æ
?
Ð
©
w
§
$
½
7
log
e
(
−
1
/
2
σ
2
)
x
2
e
(
−
1
/
2
σ
2
)(
x
+∆
f
)
2
=
log
e
(
−
1
/
2
σ
2
)[
x
2
−
(
x
+∆
f
)
2
]
=
−
1
2
σ
2
x
2
−
x
2
+2
x
∆
f
+∆
f
2
=
−
1
2
σ
2
2
x
∆
f
+(∆
f
)
2
.
|
x
|
<
σ
2
ε
∆
f
−
∆
f
2
ž
,
log
e
(
−
1
/
2
σ
2
)
x
2
e
(
−
1
/
2
σ
2
)
(
x
+∆
f
)
2
≤
ε
.
•
(
Pr
log
e
(
−
1
/
2
σ
2
)
x
2
e
(
−
1
/
2
σ
2
)
(
x
+∆
f
)
2
≤
ε
≥
1
−
δ
,
I
‡
Pr
|
x
|≥
σ
2
ε
∆
f
−
∆
f
2
<δ,
(6)
l
I
‡
Pr
x
≥
σ
2
ε
∆
f
−
∆
f
2
<
δ
2
.
0
<ε<
1,
t
=
σ
2
ε
∆
f
−
∆
f
2
,
K
Pr[
x
≥
t
] =
Z
∞
t
1
√
2
πσ
e
−
x
2
2
σ
2
dx
≤
σ
√
2
πt
e
−
t
2
2
σ
2
.
Ï
d
,
•
I
σ
√
2
π
1
t
e
−
t
2
2
σ
2
<
δ
2
,
ù
d
u
I
‡
σ
t
e
−
t
2
2
σ
2
<
√
2
πδ
2
⇔
t
σ
e
t
2
2
σ
2
>
2
√
2
πδ
⇔
log(
t
σ
)+
t
2
2
σ
2
>
log(
2
√
2
πδ
)
.
du
t
=
σ
2
ε
∆
f
−
∆
f
2
,
þ
ª
q
d
u
I
‡
log
σ
2
ε
∆
f
−
∆
f
2
1
σ
+
"
σ
2
ε
∆
f
−
∆
f
2
2
1
2
σ
2
#
>
log
2
√
2
πδ
= log
r
2
π
1
δ
!
.
•
d
•
I
‡
e
ã
ü
ª
Ó
ž
¤
á
=
Œ
log
σ
2
ε
∆
f
−
∆
f
2
1
σ
>
0;
"
σ
2
ε
∆
f
−
∆
f
2
2
1
2
σ
2
#
≥
log
r
2
π
1
δ
!
.
(7)
DOI:10.12677/aam.2023.121022191
A^
ê
Æ
?
Ð
©
w
§
$
½
7
P
c
=
εσ
∆
f
,
K
σ
=
c
∆
f
ε
.
du
1
σ
σ
2
ε
∆
f
−
∆
f
2
=
1
σ
(
c
∆
f
)
2
ε
2
ε
∆
f
−
∆
f
2
=
1
σ
c
2
∆
f
ε
−
∆
f
2
=
ε
c
∆
f
c
2
∆
f
ε
−
∆
f
2
=
c
−
ε
2
c
,
5
¿
0
<ε
≤
1,
Ï
d
,
c
≥
3
2
ž
,
c
−
ε
2
c
≥
7
6
,
d
ž
,log
σ
2
ε
∆
f
−
∆
f
2
1
σ
>
0
÷
v
.
,
˜
•
¡
,
1
2
σ
2
σ
2
ε
∆
f
−
∆
f
2
2
=
1
2
c
−
ε
2
c
2
=
1
2
c
2
−
ε
+
ε
2
4
c
2
.
c
≥
3
2
…
0
<ε
≤
1
ž
,
du
c
2
−
ε
+
ε
2
4
c
2
0
c
>
0,
¤
±
c
2
−
ε
+
ε
2
4
c
2
≥
c
2
−
8
9
.
Ï
d
,
‡
¦
"
σ
2
ε
∆
f
−
∆
f
2
2
1
2
σ
2
#
≥
log
q
2
π
1
δ
,
•
I
c
2
−
8
9
>
2log
r
2
π
1
δ
!
,
=
•
I
c
2
>
2log
r
2
π
!
+2log
1
δ
+log
e
8
9
= log
2
π
+log
e
8
9
+2log
1
δ
= log
2
π
e
8
9
+2log
1
δ
,
5
¿
r
2
π
e
8
9
<
1
.
25,
l
•
I
c
2
>
2log
1
.
25
δ
,
=
•
I
σ
≥
∆
2
(
f
)
ε
s
2log
1
.
25
δ
.
-
R
1
=
{
x
∈
R
:
|
x
|≤
c
2
∆
f
ε
−
∆
f
2
}
,
R
2
=
{
x
∈
R
:
|
x
|
>
c
2
∆
f
ε
−
∆
f
2
}
,
K
R
=
R
1
∪
R
2
.
∀
S
⊆
R
,
½
Â
S
1
=
{
f
(
D
)+
x
|
x
∈
R
1
}
,S
2
=
{
f
(
D
)+
x
|
x
∈
R
2
}
,
DOI:10.12677/aam.2023.121022192
A^
ê
Æ
?
Ð
©
w
§
$
½
7
K
Pr
x
∼N
(0
,σ
2
)
[
f
(
D
)+
x
∈
S
] =Pr
x
∼N
(0
,σ
2
)
[
f
(
D
)+
x
∈
S
1
]+Pr
x
∼N
(0
,σ
2
)
[
f
(
D
)+
x
∈
S
2
]
≤
Pr
x
∼N
(0
,σ
2
)
[
f
(
D
)+
x
∈
S
1
]+
δ
≤
e
ε
Pr
x
∼N
(0
,σ
2
)
[
f
(
D
0
)+
x
∈
S
1
]
+
δ,
l
y
é
˜
‘
œ
¹
e
p
d
Å
›
÷
v
(
ε,δ
)-DP.
2
?
Ø
d>
1
œ
/
.
é
u
f
:
D
→
R
d
Ú
?
¿
ƒ
ê
â
8
D,D
0
,
P
∆
f
=∆
2
(
f
)
Ú
v
=
f
(
D
)
−
f
(
D
0
)(
§
“
L
V
\
F
Ý
6
Ä
¤
ù
X
&
E
),
K
•
þ
v
÷
v
k
v
k≤
∆
f
.
D
(
•
þ
x
∼N
(0
,σ
2
I
),
K
Š
â
©
z
[33],
Û
h›
”
•
log
e
(
−
1
2
σ
2
)
k
x
−
µ
k
2
e
(
−
1
2
σ
2
)
k
x
+
v
−
µ
k
2
=
log
e
(
−
1
2
σ
2
)[
k
x
−
µ
k
2
−k
x
+
v
−
µ
k
2
]
=
1
2
σ
2
(
k
x
k
2
−k
x
+
v
k
2
)
,
Ù
¥
µ
=(0
,...,
0)
T
∈
R
d
.
-
a
1
=
v
k
v
k
,
¿
ò
a
1
*
¿¤
R
d
¥
˜
|
I
O
Ä
a
1
,...,a
d
,
K
•
3
β
1
,
···
,β
d
∈
R
…
β
i
∼N
(0
,σ
2
)(
i
= 1
,
···
,d
),
¦
x
=
P
m
i
=1
β
i
a
i
.
P
x
[
i
]
=
β
i
a
i
.
5
¿
,
d
Ä
5
Œ
h
x
[
i
]
,v
i
= 0,
i
= 2
,...,d
,
?
h
m
P
i
=2
x
[
i
]
,v
i
= 0.
Ï
d
,
k
x
k
2
=
m
X
i
=1
x
[
i
]
2
=
x
[1]
2
+
m
X
i
=2
x
[
i
]
2
,
k
x
+
v
k
2
=
x
[1]
+
m
X
i
=2
x
[
i
]
+
v
2
=
v
+
x
[1]
2
+
m
X
i
=2
x
[
i
]
2
.
d
x
[1]
=
β
1
b
1
=
β
1
v
k
v
k
,
v
+
x
[1]
2
=
k
v
k
(1+
β
1
k
v
k
)
2
= (
k
v
k
+
β
1
)
2
.
Ï
d
,
k
x
+
v
k
2
−k
x
k
2
=
k
v
+
x
[1]
k
2
−k
x
[1]
k
2
= (
k
v
k
+
β
1
)
2
−
β
2
1
=
k
v
k
2
+2
β
1
k
v
k
.
Ï
k
v
k≤
∆
f
,
1
2
σ
2
(
k
x
k
2
−k
x
+
v
k
2
)
=
1
2
σ
2
(
k
v
k
2
+2
β
1
k
v
k
2
)
≤
1
2
σ
2
[2
β
1
∆
f
+(∆
f
)
2
]
.
du
β
1
∼N
(0
,σ
2
),
þ
ª
L
²
Û
h›
”
•
6
u
˜
‘
‘
Å
C
þ
β
1
,
=
q
£
˜
‘
œ
/
.
d
c
ã
˜
‘
œ
/
(
Ø
Œ
,
‘
ê
d>
1
œ
¹
e
p
d
Å
›
•
,
÷
v
(
ε,δ
)-DP.
½
n
3.2
Å
›
M
1
(
·
)
÷
v
(
,δ
)
−
DP
,
M
2
´
?
¿
Ø
ž
Ñ
Û
h
ý
Ž
Ž
{
,
K
M
1
Ú
M
2
E
Ü
M
2
(
M
1
(
·
))
÷
v
(
,δ
)
−
DP
.
DOI:10.12677/aam.2023.121022193
A^
ê
Æ
?
Ð
©
w
§
$
½
7
y
²
.
D
Ú
D
0
´
?
¿
ü
‡
ƒ
ê
â
8
.
-
Range(
M
1
) =
S
L
«
M
1
Š
•
8
Ü
.
(i)
e
S
´
l
Ñ
8
Ü
,
∀
t
∈
Range(
M
2
),
k
Pr[
M
2
(
M
1
(
D
)) =
t
)] =
X
s
∈S
Pr[
M
1
(
D
) =
s
]Pr[
M
2
(
s
) =
t
]
≤
X
s
∈S
(
e
ε
Pr[
M
1
(
D
0
) =
s
]+
δ
)Pr[
M
2
(
s
) =
t
]
=
e
ε
Pr[
M
2
(
M
1
(
D
0
)) =
t
]+
δ
(ii)
e
S
´
ë
Y
8
Ü
,
∀
t
∈
Range(
M
2
),
Pr[
M
2
(
M
1
(
D
)) =
t
]=
R
s
∈S
Pr[
M
1
(
D
) =
s
]Pr[
M
2
(
s
) =
t
]d
s
,
æ
^†
þ
ã
y
²
a
q
Ü
6
,
Œ
y
²
(
Ø
¤
á
.
©
Û
h
˜
‡
-
‡
A
´
§
Û
h
o
5
3
õ
-
E
Ü
e
Œ
±
\
È
.(
ε,δ
)-DP
\
È
½
n
X
e
.
Ú
n
3.1
[5]
∀
ε>
0
,δ>
0
,δ
0
>
0
,
K
(
ε,δ
)
−
DP
Å
›
3
k
-
g
·
A
E
Ü
(
=
Ž
{
z
g
‰
1
(
J
Ñ
´
d
ƒ
c
(
J
g
·
A
¤
)
e
÷
v
(
p
2
k
log(1
/δ
0
)
ε
+
kε
(
e
ε
−
1)
,kδ
+
δ
0
)
−
DP
.
ù
˜
5
Ÿ
¦
©
Û
h
·
^u
¬
z
Ž
{
O
Ú
©
Û
:
˜
‡E
,
Ž
{
•
¹
˜
X
÷
v
©
Û
h
Ú
½
ž
,
Œ
±
(
½
T
Ž
{
N
E
÷
v
©
Û
h
5
.
½
n
3.3
∇
θ
L
(
θ
;
D
)
3
Z
n
þ
˜
—
k
þ
.
U
,
‰
½
Û
h
ý
Ž
ε
9
ë
ê
δ
,
e
1
k
g
S
“
Û
h
ý
Ž
ε
k
=
min
{
ε
2
√
2
T
log(2
/δ
)
,
1
2
p
ε
T
}
,
ë
ê
δ
k
=
δ
2
T
,
D
(
Y
²
σ
=
2
U
ε
k
r
2log
1
.
25
δ
k
,
K
Ž
{
1
÷
v
(
ε,δ
)
−
DP
.
y
²
.
3
Ž
{
1
1
k
Ú
S
“
¥
,
=
k
D
(
F
Ý
ξ
k
(
½
I
‡
^
Û
h
ý
Ž
ε
k
9
ë
ê
δ
k
.
3D
(
Ñ
l
p
d
©
Ù
N
(0
,σ
2
I
)
¥
À
I
O
σ
÷
v
σ
=
2
U
ε
k
r
2log
1
.
25
δ
k
≥
∆
2
(
g
k
)
δ
k
r
2log
1
.
25
δ
k
,
K
d
½
n
3.1
Œ
•
,
D
(
F
Ý
)
÷
v
(
ε
k
,δ
k
)
−
DP.
1
k
Ú
S
“
´
D
(
F
Ý
)
¤
†
S
“
:
•
#
Ž
{
E
Ü
,
Š
â
½
n
3.2,
1
k
Ú
S
“
•
÷
v
(
ε
k
,δ
k
)
−
DP.
du
Ž
{
N
•
õS
“
T
Ú
,
¿
…
z
Ú
S
“
¥
D
(
F
ÝÑ
d
þ
˜
Ú
S
“
:
5
(
½
,
Ï
d
Ž
{
÷
v
T
-
g
·
A
E
Ü
.
Š
â
Ú
n
3.1,
δ
k
=
δ
2
T
,δ
0
=
δ
2
,ε
k
= min
{
ε
2
√
2
T
log(2
/δ
)
,
1
2
p
ε
T
}
ž
,
k
p
2
T
log(1
/δ
0
)
ε
k
+
Tε
k
(
e
ε
k
−
1)
≤
ε
2
+2
Tε
2
k
≤
ε
2
+
ε
2
=
ε,Tδ
k
+
δ
0
=
δ,
Ï
d
,
‡
Ž
{
S
“
L
§
÷
v
(
ε,δ
)
−
DP.
DOI:10.12677/aam.2023.121022194
A^
ê
Æ
?
Ð
©
w
§
$
½
7
3.2.
Ž
{
1
Â
ñ
5
!
©
Û
ISTDP
Ž
{
Â
ñ
5
,
ò
y
²
,
‘
X
S
“g
ê
O
\
§
Ñ
Ñ
(
J
Œ
±
÷
v
?
¿
‰
½
°
Ý
‡
¦
.
e
¡
Ä
k
y
²
é
z
‡
k
,
S
Ì
‚
1.3
k
•
Ú
=
Œ
÷
v
.
Ú
n
3.2
é
z
‡
k
≥
0
,
S
Ì
‚
Ê
Ž
O
K
(
Ú
1.3.2
Ø
ª
)
–
õ
l
log(
βα
2
max
+
α
max
L
)
−
log
η
+1
m
Ú
Ò
¬
÷
v
.
y
²
.
d
Ú
1.3.1
Œ
•
,
Q
(
α
BB
k
,
ˆ
θ
k
+1
,θ
k
)
≤
Q
(
α
BB
k
,θ
k
,θ
k
),
=
L
(
θ
k
;
D
)+
h
g
k
+
b
k
,
ˆ
θ
k
+1
−
θ
k
i
+
1
2
α
BB
k
k
ˆ
θ
k
+1
−
θ
k
k
2
+
R
(
ˆ
θ
k
+1
)
≤
L
(
θ
k
;
D
)+
R
(
θ
k
)
.
(8)
Ï
•
∇
L
(
θ
;
D
)is
L
-Lipschitz
ë
Y
,
L
(
ˆ
θ
k
+1
;
D
)
≤
L
(
θ
k
;
D
)+
h
g
k
,
ˆ
θ
k
+1
−
θ
k
i
+
L
2
k
ˆ
θ
k
+1
−
θ
k
k
2
.
(9)
d
(8)
Ú
(9),
L
(
ˆ
θ
k
+1
;
D
)+
R
(
ˆ
θ
k
+1
)+
h
b
k
,
ˆ
θ
k
+1
−
θ
k
i
+
1
2
(
1
α
BB
k
−
L
)
k
ˆ
θ
k
+1
−
θ
k
k
2
≤
L
(
θ
k
;
D
)+
R
(
θ
k
)
.
(10)
d
F
(
ˆ
θ
k
+1
;
D
) =
L
(
ˆ
θ
k
+1
;
D
)+
R
(
ˆ
θ
k
+1
)
,
F
(
θ
k
;
D
) =
L
(
θ
k
;
D
)+
R
(
θ
k
)
,
Ú
Ø
ª
(10),
E
(
F
(
θ
k
;
D
)
−
F
(
ˆ
θ
k
+1
;
D
))
≥
1
2
(
1
α
BB
k
−
L
)
E
k
ˆ
θ
k
+1
−
θ
k
k
2
.
(11)
Ï
d
,
1
α
BB
k
−
L
≥
βα
BB
k
(
=
1
α
BB
k
−
βα
BB
k
≥
L
)
ž
,
Ú
1.3.2
÷
v
.
du
1
α
−
αβ
'
u
α>
0
´
ü
N4
~
¼
ê
…
lim
α
→
0+
(
1
α
−
αβ
)=+
∞
,
Ú
1.3.2
Ø
÷
v
ž
,
α
BB
k
=
ηα
BB
k
´
U
'
~
η
(0
<η<
1)
Ø
α
BB
k
,
–
õ
k
•
Ú
ƒ
Ú
1.3.2
=
Œ
÷
v
.
-
¯
α
BB
k
L
«
1
k
Ú
S
“
α
BB
k
•
ªŠ
,
K
e
¡
ü
ª
Ó
ž
DOI:10.12677/aam.2023.121022195
A^
ê
Æ
?
Ð
©
w
§
$
½
7
¤
á
1
¯
α
BB
k
−
β
¯
α
BB
k
≥
L,
η
¯
α
BB
k
−
β
¯
α
BB
k
η
<L.
-
m
k
L
«
1
k
g
Ì
‚
ž
S
Ì
‚
(
Ú
1.3)
S
“g
ê
,
T
S
Ì
‚
Ð
©
Ú
•
•
α
BB
k
,
K
¯
α
BB
k
=
α
BB
k
η
m
k
,
…
1
α
BB
k
η
m
k
−
1
−
βα
BB
k
η
m
k
−
1
<L.
du
1
α
−
αβ
'
u
α>
0
´
ü
N4
~
¼
ê
…
α
min
≤
α
BB
k
≤
α
max
,
1
α
max
η
m
k
−
1
−
βα
max
η
m
k
−
1
<L.
d
0
<η<
1,
1
α
max
η
m
k
−
1
≤
βα
max
η
m
k
−
1
+
L
≤
βα
max
+
L.
Ï
d
,
m
k
≤
l
log(
βα
2
max
+
α
max
L
)
−
log
η
+1
m
,
y
.
.
5
1:
Š
â
Ú
n
3.2
Œ
•
,
3
‡
Ž
{
¥
Ú
•
{
α
BB
k
}
þ
k
e
.
Ú
þ
.
:
0
<α
min
:=
α
min
η
log(
βα
2
max
+
α
max
L
)
−
log
η
+1
≤
α
BB
k
≤
α
max
.
5
2:
Š
â
S
Ì
‚
1.3
Œ
•
,
1
k
d
Ì
‚
(
åž
,
e
ã
ü
ª
Ó
ž
¤
á
:
θ
k
+1
= argmin
θ
{
Q
(
α
BB
k
,θ
k
,θ
) =
L
(
θ
k
;
D
)+
h
ξ
k
,θ
−
θ
k
i
+
1
2
α
BB
k
k
θ
−
θ
k
k
2
+
R
(
θ
)
}
;
E
(
F
(
θ
k
;
D
)
−
F
(
θ
k
+1
;
D
))
≥
βα
BB
k
2
E
k
θ
k
+1
−
θ
k
k
2
.
1
ª
¿
›
X
{
E
(
F
(
θ
k
;
D
)
}
k
´
ü
N
~
.
Ú
n
3.3
[34]
Œ
‡
¼
ê
f
:
R
n
→
R
½
Â
•
domf
=
R
n
,
…
F
Ý
L
−
Lipschitz
ë
Y
,
K
e
ã
Ø
ª
¤
á
:
f
(
y
)
≤
f
(
x
)+
∇
f
(
x
)
T
(
y
−
x
)+
L
2
k
y
−
x
k
2
,
∀
x,y
∈
domf.
DOI:10.12677/aam.2023.121022196
A^
ê
Æ
?
Ð
©
w
§
$
½
7
½
n
3.4
é
.
(2),
Ž
{
1
÷
v
E
[
F
(
θ
priv
;
D
)
−
F
(
θ
∗
;
D
)]
≤
k
θ
1
−
θ
∗
k
2
2
Tα
min
,
Ù
¥
T
L
«
Ž
{
1
•
Œ
S
“
Š
,
α
min
L
«
α
BB
k
e
.
,
θ
∗
∈
R
d
´
{
θ
k
}
?
˜
à
:
.
y
²
.
Ï
{
E
F
θ
k
;
D
}
ü
N4
~
k
e
.
,
{
E
F
θ
k
;
D
}
7
Â
ñ
.
du
F
(
θ
;
D
)
´
Y
²
k
.
,
{
θ
k
}
7
k
à
:
,
θ
∗
´
{
θ
k
}
?
¿˜
‡
à
:
,
K
k
→∞
ž
,
E
F
(
θ
k
;
D
)
→
E
(
F
(
θ
∗
;
D
)).
Š
â
.
(2)
Œ
±
Ñ
E
F
(
θ
k
+1
;
D
)
−
F
(
θ
∗
;
D
)
=
E
L
(
θ
k
+1
;
D
)
−
L
(
θ
∗
;
D
)+
R
(
θ
k
+1
)
−
R
(
θ
∗
)
(10
.
1)
≤
E
L
(
θ
k
;
D
)+
g
k
,θ
k
+1
−
θ
k
+
L
2
θ
k
+1
−
θ
k
2
−
L
(
θ
∗
;
D
)+
R
(
θ
k
+1
)
−
R
(
θ
∗
)
(10
.
2)
≤
E
g
k
,θ
k
−
θ
∗
+
g
k
,θ
k
+1
−
θ
k
+
L
2
θ
k
+1
−
θ
k
2
+
R
(
θ
k
+1
)
−
R
(
θ
∗
)
(10
.
3)
=
E
ξ
k
,θ
k
−
θ
∗
+
g
k
,θ
k
+1
−
θ
k
+
L
2
θ
k
+1
−
θ
k
2
+
R
(
θ
k
+1
)
−
R
(
θ
∗
)
.
(12)
Ù
¥
Ø
ª
(10.1)
Ä
u
L
(
·
;
D
)
r
1
w
5
,
Ø
ª
(10.2)
Ä
u
L
(
·
;
D
)
à
5
,
ª
(10.3)
´
Ï
•
E
(
b
k
) = 0.
du
θ
k
+1
=argmin
θ
{
L
(
θ
k
;
D
)+
h
ξ
k
,θ
−
θ
k
i
+
1
2
α
BB
k
k
θ
−
θ
k
k
2
+
R
(
θ
)
}
=argmin
θ
∈
R
d
{
1
2
θ
−
θ
k
+
α
BB
k
ξ
k
2
+
α
BB
k
R
(
θ
)
}
,
•
3
g
F
Ý
ν
k
+1
∈
∂R
(
θ
k
+1
)[35],
¦
θ
k
+1
−
θ
k
+
α
BB
k
(
ξ
k
+
ν
k
+1
) = 0.
Š
â
g
F
Ý
5
Ÿ
,
k
R
(
θ
∗
)
−
R
(
θ
k
+1
)
≥
ν
k
+1
,θ
∗
−
θ
k
+1
.
l
,
R
(
θ
∗
)
−
R
(
θ
k
+1
)+
h
1
α
BB
k
(
θ
k
+1
−
θ
k
)+
ξ
k
,θ
∗
−
θ
k
+1
i
≥h
1
α
BB
k
(
θ
k
+1
−
θ
k
)+
ξ
k
+
ν
k
+1
,θ
∗
−
θ
k
+1
i
= 0
,
R
(
θ
∗
)
−
R
(
θ
k
+1
)+
h
ξ
k
,θ
∗
−
θ
k
+1
i≥
1
α
BB
k
h
θ
k
−
θ
k
+1
,θ
∗
−
θ
k
+1
i
.
DOI:10.12677/aam.2023.121022197
A^
ê
Æ
?
Ð
©
w
§
$
½
7
?
,
ξ
k
,θ
k
−
θ
∗
+
R
(
θ
k
+1
)
−
R
(
θ
∗
)
=
ξ
k
,θ
k
−
θ
k
+1
+
ξ
k
,θ
k
+1
−
θ
∗
+
R
(
θ
k
+1
)
−
R
(
θ
∗
)
≤
ξ
k
,θ
k
−
θ
k
+1
−
1
α
BB
k
θ
k
+1
−
θ
k
,θ
k
+1
−
θ
∗
=
ξ
k
,θ
k
−
θ
k
+1
+
θ
k
−
θ
k
+1
,θ
k
−
θ
∗
2
α
BB
k
+
θ
k
+1
−
θ
∗
,θ
k
−
θ
∗
2
α
BB
k
−
θ
k
+1
−
θ
k
,θ
k
+1
−
θ
∗
2
α
BB
k
−
θ
k
−
θ
∗
,θ
k
+1
−
θ
∗
2
α
BB
k
−
θ
k
+1
−
θ
k
,θ
k
+1
−
θ
∗
2
α
BB
k
−
θ
k
+1
−
θ
k
,θ
∗
−
θ
k
2
α
BB
k
=
ξ
k
,θ
k
−
θ
k
+1
+
θ
k
−
θ
∗
,θ
k
−
θ
∗
2
α
BB
k
−
θ
k
+1
−
θ
∗
,θ
k
+1
−
θ
∗
2
α
BB
k
−
θ
k
+1
−
θ
k
,θ
k
+1
−
θ
k
2
α
BB
k
=
ξ
k
,θ
k
−
θ
k
+1
+
θ
k
−
θ
∗
2
2
α
BB
k
−
θ
k
+1
−
θ
∗
2
2
α
BB
k
−
θ
k
+1
−
θ
k
2
2
α
BB
k
.
(13)
(
Ü
(12)
Ú
(13)
ü
‡
Ø
ª
,
k
E
F
θ
k
+1
;
D
−
F
(
θ
∗
;
D
)
≤
E
"
ξ
k
−
g
k
,θ
k
−
θ
k
+1
−
1
−
α
BB
k
L
2
α
BB
k
θ
k
+1
−
θ
k
2
+
θ
k
−
θ
∗
2
−
θ
k
+1
−
θ
∗
2
2
α
BB
k
#
(12
.
1)
≤
E
"
α
BB
k
2(1
−
α
BB
k
L
)
ξ
k
−
g
k
2
+
θ
k
−
θ
∗
2
−
θ
k
+1
−
θ
∗
2
2
α
BB
k
#
=
E
"
θ
k
−
θ
∗
2
−
θ
k
+1
−
θ
∗
2
2
α
BB
k
#
(12
.
2)
≤
E
"
θ
k
−
θ
∗
2
−
θ
k
+1
−
θ
∗
2
2
α
min
#
,
(14)
Ù
¥
Ø
ª
(12.1)
d
α
BB
k
1
−
α
BB
k
L
(
ξ
k
−
g
k
)
−
(
θ
k
−
θ
k
+1
)
2
≥
0
Ð
m
ª
n
¤
;
d
Ú
n
3.1
5
1
Œ
•
,
α
BB
k
k
e
.
α
min
>
0,
l
Ø
ª
(12.2)
¤
á
.
d
Ø
ª
(14)
(
Ü
Ú
n
3.1
5
2(
{
E
[
F
θ
k
+1
;
D
]
}
ü
N4
~
5
Ÿ
),
k
E
F
θ
T
;
D
−
F
(
θ
∗
;
D
)
≤
E
"
T
−
1
X
k
=0
F
θ
k
+1
;
D
T
−
F
(
θ
∗
;
D
)
#
=
E
"
1
T
T
−
1
X
k
=0
F
(
θ
k
+1
;
D
)
−
F
(
θ
∗
;
D
)
#
≤
E
"
k
θ
1
−
θ
∗
k
2
−
θ
k
+1
−
θ
∗
2
2
Tα
min
#
≤
E
"
k
θ
1
−
θ
∗
k
2
2
Tα
min
#
=
k
θ
1
−
θ
∗
k
2
2
Tα
min
,
=
E
F
θ
priv
;
D
−
F
(
θ
∗
;
D
)
=
E
F
θ
T
;
D
−
F
(
θ
∗
;
D
)
≤
k
θ
1
−
θ
∗
k
2
2
Tα
min
.
DOI:10.12677/aam.2023.121022198
A^
ê
Æ
?
Ð
©
w
§
$
½
7
5
:
Š
â
½
n
3.4,
‰
½
°
Ý
>
0,
‡
ˆ
T
°
Ý
Ž
{
S
“g
ê
T
A
÷
v
T >
l
m
2
(
θ
1
)
2
α
min
m
,
Ù
¥
m
(
θ
1
)
L
«
Y
²
8
L
(
θ
1
) :=
{
θ
∈
R
d
:
F
(
θ
;
D
)
≤
F
(
θ
1
;
D
)
}
†
»
.
4.
o
(
©
ï
Ä
Ä
u
F
Ý
6
Ä
©
Û
h
o
e
‘
k
š
à
K
²
º
x
4
z
¯
K
.
Ä
u
±
BB
Ú
•
•
Á
&
Ú
‚
|ƒ
Ú
S
“
Â
K
Š
Ž
{
J
Ñ
ISTDP
Ž
{
,
¿
y
²
ISTDP
Ž
{
Q
÷
v
©
Û
h
o
‡
¦
q
Œ
±
Â
ñ
?
¿
‰
½
°
Ý
,
´
˜
«
Œ
±
¢
y
Û
h
o
Å
ì
Æ
S
`
z
Ž
{
.
e
˜
Ú
ò
Ï
L
ê
Š
¢
Ú
Ž
~
?
˜
Ú
u
Ž
{
¢
S
J
.
Ä
7
‘
8
I
[
g
,
‰
Æ
Ä
7
‘
8
(11861020,12261020)
!
B
²
Ž
p
g
3
Æ
<
â
M
#
M
’
J
`
]
Ï-
:
‘
8
([2018]03)
!
B
²
Ž
‰
EO
y
‘
8
(ZK[2021]009,[2018]5781)
!
B
²
Ž
“
c
‰
E
<
â
¤•
‘
8
([2018]121).
ë
•
©
z
[1]Li,N.,Lyu,M.,Su,D.andYang,W.(2016)DifferentialPrivacy:FromTheorytoPractice.
In:
SynthesisLecturesonInformationSecurityPrivacyandTrust
,Springer,Cham,1-138.
https://doi.org/10.1007/978-3-031-02350-7
[2]Dwork,C.,McSherry,F.,Nissim,K.andSmith,A(2016)CalibratingNoisetoSensitivity
inPrivateDataAnalysis.
ProceedingsoftheThirdConferenceonTheoryofCryptography
,
7
,
17-51.https://doi.org/10.29012/jpc.v7i3.405
[3]Jiang,B.,Li,J.andYue,G.(2021)DifferentialPrivacyforIndustrialInternetofThings:
Opportunities,Applications,andChallenges.
IEEEInternetofThingsJournal
,
8
,10430-
10451.https://doi.org/10.1109/JIOT.2021.3057419
[4]El Ouadrhiri,A.andAbdelhadi,A.(2022)Differential Privacy forDeep andFederated Learn-
ing:A Survey.
IEEE Access
,
10
, 22359-22380.https://doi.org/10.1109/ACCESS.2022.3151670
[5]Dwork, C., Rothblum, G. and Vadhan, S. (2010) Boosting and Differential Privacy.
2010IEEE
51stAnnualSymposiumonFoundationsofComputerScience
,LasVegas,NV,23-26October
2010,23-26.https://doi.org/10.1109/FOCS.2010.12
DOI:10.12677/aam.2023.121022199
A^
ê
Æ
?
Ð
©
w
§
$
½
7
[6]Girgis,A.,Data,D.and Diggavi, S. (2021)Shuffled Model of Differential Privacy in Federated
Learning.
InternationalConferenceonArtificialIntelligenceand Statistics,PMLR
,
130
, 2521-
2529.
[7]Adnan,M., Kalra,S. andCresswell,J. (2022)Federated Learningand Differential Privacyfor
MedicalImageAnalysis.
ScientificReports
,
12
,ArticleNo.1953.
https://doi.org/10.1038/s41598-022-05539-7
[8]Abadi,M.,Andy,C.and Goodfellow,L.(2016)Deep LearningwithDifferentialPrivacy.
Pro-
ceedingsofthe2016ACMSIGSACConferenceonComputerandCommunicationsSecurity
,
Vienna,24-28October2016,308-318.https://doi.org/10.1145/2976749.2978318
[9]Chaudhuri,K.andMonteleoni,C.(2011)DifferentiallyPrivateEmpiricalRiskMinimization.
JournalofMachineLearningResearch
,
12
,1069-1109.
[10]Zhang, J.,Zheng,K., Mou, W. andWang,L.(2017) Efficient Private ERM for SmoothObjec-
tives.
Proceedingsofthe Twenty-SixthInternationalJoint ConferenceonArtificial Intelligence
,
Melbourne,19-25August2017,3922-3928.https://doi.org/10.24963/ijcai.2017/548
[11]Chaudhuri, K. and Monteleoni, C.(2009) Privacy-Preserving Logistic Regression.
Advancesin
NeuralInformationProcessingSystems
,
21
,289-296.
[12]Wang, Y., Wang, Y. and Singh,A. (2015)DifferentiallyPrivate SubspaceClustering.
Advances
inNeuralInformationProcessingSystems28:AnnualConferenceonNeuralInformationPro-
cessingSystems
,
28
,1000-1008.
[13]Bassily,R., Smith,A. and Thakurta,A. (2014) Private Empirical Risk Minimization:Efficient
AlgorithmsandTightErrorBounds.
2014IEEE55thAnnualSymposiumonFoundationsof
ComputerScience
,Philadelphia,PA,18-21October2014,464-473.
https://doi.org/10.1109/FOCS.2014.56
[14]Beimel,A., Kasiviswanathan, S.and Nissim,K.(2010)Bounds onthe SampleComplexityfor
PrivateLearningandPrivateDataRelease.InMicciancio,D.,Ed.,
TheoryofCryptography.
TCC2010.LectureNotesinComputerScience
,Vol.5978,Springer,Berlin,Heidelberg,437-
454.https://doi.org/10.1007/978-3-642-11799-226
[15]Williams, O.andMcsherry, F.(2010)ProbabilisticInferenceandDifferentialPrivacy.
Advances
inNeuralInformationProcessingSystems
,
23
,2451-2459.
[16]Wang,D.,Ye, M.and Xu,J.(2017)Differentially PrivateEmpiricalRiskMinimizationRevis-
ited:FasterandMoreGeneral.
31stAdvancesinNeuralInformationProcessingSystems
,
30
,
2722-2731.
DOI:10.12677/aam.2023.121022200
A^
ê
Æ
?
Ð
©
w
§
$
½
7
[17]Fan,J.andLi,R.(2001)VariableSelectionviaNonconcavePenalizedLikelihoodandIts
OracleProperties.
JournaloftheAmericanStatisticalAssociation
,
96
,1348-1360.
https://doi.org/10.1198/016214501753382273
[18]
Û
¯
,
$
½
7
,
ü
‡
ý
é
Š
`
z
¯
K
)
d
5
[J].
-
Ÿ
ó
û
Œ
ÆÆ
:
g
,
‰
Æ
‡
,2020,
37(5):37-42.
[19]Zhang, C.(2010) Nearly Unbiased Variable Selection under MinimaxConcave Penalty.
Annals
ofStatistics
,
38
,894-942.https://doi.org/10.1214/09-AOS729
[20]Ong,C.andAn,L.(2013)LearningSparseClassifierswithDifferenceofConvexFunctions
Algorithms.
OptimizationMethodsandSoftware
,
28
,830-854.
https://doi.org/10.1080/10556788.2011.652630
[21]Peleg, D. and Meir, R. (2008)A Bilinear Formulation for Vector Sparsity Optimization.
Signal
Processing
,
88
,375-389.https://doi.org/10.1016/j.sigpro.2007.08.015
[22]Thi,H.,Dinh,T.,Le,H.andVo,X.(2015)DCApproximationApproachesforSparseOpti-
mization.
EuropeanJournalofOperationalResearch
,
244
,26-46.
https://doi.org/10.1016/j.ejor.2014.11.031
[23]Zhang,T.(2013)Multi-StageConvexRelaxationforFeatureSelection.
Bernoulli
,
19
,2277-
2293.https://doi.org/10.3150/12-BEJ452
[24]
$
½
7
,
/
l
,
Ü
u
,
|
D
Õ
`
z
¯
K
°
(
ë
Y
Capped-L1
t
µ
[J].
ê
ÆÆ
,2022,65(2):
243-262.
[25]Zhang,X.andPeng,D.(2022)SolvingConstrainedNonsmoothGroupSparseOptimization
viaGroupCapped-L1RelaxationandGroup SmoothingProximalGradientAlgorithm.
Com-
putationalOptimizationandApplications
,
83
,801-844.
https://doi.org/10.1007/s10589-022-00419-2
[26]Bian,W.andChen,X.(2017)OptimalityandComplexityforConstrainedOptimization
Problemswith NonconvexRegularization.
Mathematicsof OperationsResearch
,
42
,1063-1084.
https://doi.org/10.1287/moor.2016.0837
[27]Chartrand,R.andStaneva,V.(2008)RestrictedIsometryPropertiesandNonconvexCom-
pressiveSensing.
InverseProblems
,
24
,Article035020.
https://doi.org/10.1088/0266-5611/24/3/035020
[28]Peng, D. andChen,X. (2020) ComputationofSecond-OrderDirectionalStationaryPointsfor
GroupSparseOptimization.
OptimizationMethodsandSoftware
,
35
,348-376.
https://doi.org/10.1080/10556788.2019.1684492
DOI:10.12677/aam.2023.121022201
A^
ê
Æ
?
Ð
©
w
§
$
½
7
[29]
Û
¯
,
$
½
7
,
Ü
u
,
Ä
u
MCP
K
•
˜
¦
£
8
¯
K
ï
Ä
[J].
X
Ú
‰
Æ
†
ê
Æ
,2021,
41(8):2327-2337.
[30]Gong,P.andZhang,C.(2013)AGeneralIterativeShrinkageandThresholdingAlgorithm
forNon-ConvexRegularizedOptimizationProblems.
Proceedingsofthe30thInternational
ConferenceonInternationalConferenceonMachineLearning
,
28
,37-45.
[31]Jain,P.andThakurta,A.(2013)DifferentiallyPrivateLearningwithKernels.
Proceedingsof
the30thInternationalConferenceonMachineLearning
,
28
,118-126.
[32]Barzilai,J.andBorwein,J.M.(1988)Two-PointStepSizeGradientMethods.
IMAJournal
ofNumericalAnalysis
,
8
,141-148.https://doi.org/10.1093/imanum/8.1.141
[33]Dwork, C.andRoth, A.(2014)The AlgorithmicFoundationsofDifferentialPrivacy.
Founda-
tionsandTrendsinTheoreticalComputerScience
,
9
,211-407.
https://doi.org/10.1561/0400000042
[34]
4
Ó
,
r
ò
,
o
]
¹
,
©
2
©
.
•
`
z
:
ï
!
Ž
{
†
n
Ø
[M].
®
:
p
Ñ
‡
,2021.
[35]
p
ñ
.
š
1
w
`
z
[M].
®
:
‰
Æ
Ñ
‡
,2008.
DOI:10.12677/aam.2023.121022202
A^
ê
Æ
?
Ð