﻿ 基于遗传模拟退火算法的Hadoop系统性能配置优化 Performance Configuration Optimization of Hadoop System Based on Genetic Simulated Annealing Algorithm

Artificial Intelligence and Robotics Research
Vol. 09  No. 02 ( 2020 ), Article ID: 35537 , 9 pages
10.12677/AIRR.2020.92015

Performance Configuration Optimization of Hadoop System Based on Genetic Simulated Annealing Algorithm

Wanying Bao, Xiaoling Luo*

College of Computer and Information Engineering, Inner Mongolia Agricultural University, Hohhot Inner Mongolia

Received: Apr. 21st, 2020; accepted: May 6th, 2020; published: May 13th, 2020

ABSTRACT

In order to improve the performance of Hadoop on open-source cloud computing platform, a performance configuration optimization method based on genetic simulated annealing algorithm is proposed. Based on genetic algorithm, the configuration scheme is selected, crossed and mutated as chromosome. Combined with the principle of simulated annealing, the survival rate of new chromosome and the number of iterations of the whole algorithm are controlled to find the optimal scheme of system configuration. According to the genetic simulated annealing algorithm, the overall performance is better, and the optimization speed is faster in the long-term optimization. It can be used to solve the problem of finding the approximate optimal allocation scheme of the system through random search in the global space. Experimental results show that this method can effectively improve the efficiency of finding the optimal configuration. The proposed configuration method improves the operation speed, makes full use of resources and increases the throughput of the system.

Keywords:Genetic Simulated Annealing Algorithm, Hadoop, System Performance, Configuration Optimization

1. 前言

2. 概述

2.1. 云计算技术

3. 架构与方法

Table 1. Hadoop basic configuration properties

Table 2. Main configuration attributes affecting Hadoop performance

Table 3. Some properties and default values that affect Hadoop performance

Table 4. Hadoop partial basic test program

Table 5. HiBench test case and its resource characteristics

3.3. 实验方法

$a\prime =\left(1-\alpha \right)a+\beta \ast b$ $b\prime =\left(1-\beta \right)b+\alpha \ast a$ (1)

4. 实验和结果分析

Reduce类以map的输出作为输入，因此Reduce的输入类型是 ，它的输出类型是 。Reduce类也要实现reduce方法，在本实验中，reduce函数将输入的key值作为输出的key值，然后将获得多个value值加起来，作为输出的值。

GeneName，GeneRange和GeneType三个文件都是在进行配置属性值的的规范化和染色体基因值的反规范化时使用。分别用来保存配置属性，属性值范围和属性值类型，通过这三个文件，任何类型的属性值都规范化为浮点类型。算法运行时，种群信息记录在日志文件中。

Figure 2. Algorithm optimization when the tasksize is 1.2 g

Table 6. Compare the best configuration and default configuration under different working conditions

5. 讨论

5.1. 配置优化方法的比较

Table 7. Comparison between genetic simulated annealing algorithm and chaos particle swarm optimization algorithm

5.2. 自适应配置

6. 结论

Performance Configuration Optimization of Hadoop System Based on Genetic Simulated Annealing Algorithm[J]. 人工智能与机器人研究, 2020, 09(02): 131-139. https://doi.org/10.12677/AIRR.2020.92015

1. 1. 吴浩宇. 基于Hadoop的同源性搜索GO功能注释平台的研究[D]: [硕士学位论文]. 南京: 南京农业大学, 2013.

2. 2. 柳香. 云计算环境下的计算模型性能优化研究[D]: [硕士学位论文]. 石家庄: 河北师范大学, 2012.

3. 3. 杨润芝, 肖卫青, 胡开喜, 等. 云计算平台上实现30年气候资料整编的方法[J]. 计算机技术与自动化, 2013, 32(3): 137-140.

4. 4. 童颖. 基于机器学习的hadoop参数调优方法[D]: [硕士学位论文]. 武汉: 华中科技大学, 2016.

5. 5. KEl-Shorbagy, M.A., Ayoub, A.Y., El-Desoky, I.M. and Mousa, A.A. (2018) Anovel Genetic Algorithm Based K-Means Algorithm for Cluster Analysis. In: Hassanien, A., Tolba, M., Elhoseny, M. and Mostafa, M., Eds., The International Conference on Advanced Machine Learning Technologies and Applications (AMLTA2018). Advances in Intelligent Systems and Computing, Springer, Cham, 92-101. https://doi.org/10.1007/978-3-319-74690-6_10

6. 6. Gao, C., Gao, Y. and Lv, S. (2016) Improved Simulated An-nealing Algorithm for Flexible Job Shop Scheduling Problems. 2016 Control and Decision Conference, 2191-2196.

7. 7. Kang, Z. and Qu, Z. (2017) Application of BP Neural Network Optimized by Genetic Simulated An-nealing Algorithm to Prediction of Air Quality Index in Lanzhou. 2017 2nd IEEE International Conference on Com-putational Intelligence and Applications, Beijing, 8-11 September 2017, 155-160. https://doi.org/10.1109/CIAPP.2017.8167199

8. 8. 郭彩杏, 郭晓金, 柏林江. 改进遗传模拟退火算法优化BP算法研究[J]. 小型微型计算机系统, 2019, 10(10): 2063-2067.

9. 9. 杨从锐, 钱谦, 王锋, 等. 改进的自适应遗传算法在函数优化中的应用[J]. 计算机应用研究, 2018, 35(4): 1042-1045.

10. 10. 陈闯, Ryad Chellali, 邢尹. 改进遗传算法优化BP神经网络的语音情感识别[J]. 计算机应用研究, 2019, 36(2): 344-346+361.

11. NOTES

*通讯作者。