yum -y install openssl-devel zlib-devel bzip2-devel sqlite-devel readline-devel libffi-devel systemtap-sdt-devel
3、下载安装包

tar zvxf Python-3.7.0.tgz
cd Python-3.7.0
./configure --prefix=/usr/local/python3.7 --enable-optimizations
make && make install

# 编译完成后，创建软链接文件到执行文件路径：
ln -s /usr/local/python3/bin/python3 /usr/bin/python3
ln -s /usr/local/python3/bin/pip3 /usr/bin/pip3
# 我们可以清除之前编译的可执行文件及配置文件 && 清除所有生成的文件：
make clean && make distclean

if [ -z ${PYTHON37_HOME} ]; then
export PYTHON37_HOME=/usr/local/python3.7
export PATH=${PYTHON37_HOME}/bin:${PATH}
fi
6、加载环境变量

2，git2 安装

centos7系统默认的git安装版本是1.8，但是在项目构建中发现git版本过低，于是用源码编译的方式进行升级.

作者：一介布衣q
链接：https://www.imooc.com/article/275738
来源：慕课网
本文原创发布于慕课网，转载请注明出处，谢谢合作

——————————————————————————————————————————

4，使用ESRally压测ElasticSearch性能

------------------------------------------------------
    _______             __   _____
   / ____(_)___  ____ _/ /  / ___/_________  ________
  / /_  / / __ / __ `/ /   \__ / ___/ __ / ___/ _ 
 / __/ / / / / / /_/ / /   ___/ / /__/ /_/ / /  /  __/
/_/   /_/_/ /_/\__,_/_/   /____/\___/\____/_/   \___/
------------------------------------------------------

|                         Metric |                 Task |     Value |   Unit |
|-------------------------------:|---------------------:|----------:|-------:|
|            Total indexing time |                      |   28.0997 |    min |
|               Total merge time |                      |   6.84378 |    min |
|             Total refresh time |                      |   3.06045 |    min |
|               Total flush time |                      |  0.106517 |    min |
|      Total merge throttle time |                      |   1.28193 |    min |
|               Median CPU usage |                      |     471.6 |      % |
|             Total Young Gen GC |                      |    16.237 |      s |
|               Total Old Gen GC |                      |     1.796 |      s |
|                     Index size |                      |   2.60124 |     GB |
|                  Total written |                      |   11.8144 |     GB |
|         Heap used for segments |                      |   14.7326 |     MB |
|       Heap used for doc values |                      |  0.115917 |     MB |
|            Heap used for terms |                      |   13.3203 |     MB |
|            Heap used for norms |                      | 0.0734253 |     MB |
|           Heap used for points |                      |    0.5793 |     MB |
|    Heap used for stored fields |                      |  0.643608 |     MB |
|                  Segment count |                      |        97 |        |
|                 Min Throughput |         index-append |   31925.2 | docs/s |
|              Median Throughput |         index-append |   39137.5 | docs/s |
|                 Max Throughput |         index-append |   39633.6 | docs/s |
|      50.0th percentile latency |         index-append |   872.513 |     ms |
|      90.0th percentile latency |         index-append |   1457.13 |     ms |
|      99.0th percentile latency |         index-append |   1874.89 |     ms |
|       100th percentile latency |         index-append |   2711.71 |     ms |
| 50.0th percentile service time |         index-append |   872.513 |     ms |
| 90.0th percentile service time |         index-append |   1457.13 |     ms |
| 99.0th percentile service time |         index-append |   1874.89 |     ms |
|  100th percentile service time |         index-append |   2711.71 |     ms |
|                           ...  |                  ... |       ... |    ... |
|                           ...  |                  ... |       ... |    ... |
|                 Min Throughput |     painless_dynamic |   2.53292 |  ops/s |
|              Median Throughput |     painless_dynamic |   2.53813 |  ops/s |
|                 Max Throughput |     painless_dynamic |   2.54401 |  ops/s |
|      50.0th percentile latency |     painless_dynamic |    172208 |     ms |
|      90.0th percentile latency |     painless_dynamic |    310401 |     ms |
|      99.0th percentile latency |     painless_dynamic |    341341 |     ms |
|      99.9th percentile latency |     painless_dynamic |    344404 |     ms |
|       100th percentile latency |     painless_dynamic |    344754 |     ms |
| 50.0th percentile service time |     painless_dynamic |    393.02 |     ms |
| 90.0th percentile service time |     painless_dynamic |   407.579 |     ms |
| 99.0th percentile service time |     painless_dynamic |   430.806 |     ms |
| 99.9th percentile service time |     painless_dynamic |   457.352 |     ms |
|  100th percentile service time |     painless_dynamic |   459.474 |     ms |

----------------------------------
[INFO] SUCCESS (took 2634 seconds)
----------------------------------

在部署完一套ES集群之后，我们肯定想知道这套集群性能如何？是否可以支撑未来业务发展？存不存在性能上的瓶颈？要想有依据的回答这些问题，我们需要通过压力测试结果中找答案。

介绍

Rally是Elasticsearch的基准测试框架，由官方提供维护。

安装

安装Python3.5及以上版本，系统默认可能是2.x版本，如果需要升级请参考《在CentOS7上安装Python3》。
安装git1.9及以上版本
安装esrally pip3 install esrally
配置esrally esrally configure，执行此命令后会在当前用户根目录下生成 .rally 目录，可以 ll ~/.rally这样来确认。

使用

快速开始

如果想测试当前机器上某个版本单点ES性能，可以像下面这样：

esrally --distribution-version=6.5.3

# 同样的如果你想测试其它版本
esrally --distribution-version=6.8.1   --car="4gheap"

当执行上面的命令之后会自动下载对应es版本软件，并在本地启动，接着执行测试。这个过程在rally中被称为比赛，而赛道是用默认的，即geonames。

测试远程集群

上面的示例不能测试存在的es集群，下面介绍使用方法：

指定跑道和ES集群地址后就可以执行测试。

esrally --pipeline=benchmark-only 
 --track=http_logs 
 --target-hosts=192.168.1.100:9200,192.168.1.101:9200,192.168.1.102:9200 
 --report-file=/tmp/report_http_logs.md
--track-params="bulk_indexing_clients:96"
 --include-tasks="index-append"
--challenge=append-fast-with-no-conflicts  (只测试写）

 --report-format=csv



备注： esrally list tracks 命令可以查看可用跑道（–track）

可以使用 --report-file=/path/to/your/report.md 将此报告也保存到文件中，并使用 --report-format=csv将其另存为CSV。

修改默认跑道参数

如果直接在默认跑道上修改，会被还原，所以只能通过增加跑道的方式。

在 .rally/benchmarks/tracks 下面创建新的赛道，比如 custom。
在 custom/http_logs/challenges/default.json 文件中调整赛道的配置并保存，例如下面修改default对应的操作，由10个客户端发起，每个用户端发出100次操作。
1
2
3
4
5
6
7

{
"operation": "default",
"clients": 10,
"warmup-iterations": 500,
"iterations": 100,
"target-throughput": 100
}

在启动时指定跑道：esrally --track=custom ....，例如：

esrally --pipeline=benchmark-only 
 --track=custom 
 --track=http_logs 
 --target-hosts=192.168.1.100:9200,192.168.1.101:9200,192.168.1.102:9200 
 --report-file=/tmp/report_http_logs.md

参考文献：

https://esrally.readthedocs.io/en/stable/index.html
https://github.com/elastic/rally
比较全的压测介绍：https://www.jianshu.com/p/c89975b50447
使用docker 运行 esrally（包含离线数据集）：https://www.jianshu.com/p/3a019c135e2a
测试参数解释：https://www.jianshu.com/p/979f548c233e
讨论为什么elasticsearch没有被压满：https://discuss.elastic.co/t/es-benchmark-using-rally-to-stress-a-2-node-setup/150020/6
有压测结果对比：https://www.jianshu.com/p/e7de3b24f505

数据集：
国内下载慢，可以先执行一遍

esrally --distribution-version=6.5.3 --track=geonames
这样即使下载测试数据失败，但是目录结构都生成好了。可以自行下载bz文件，在

http://benchmarks.elasticsearch.org.s3.amazonaws.com/corpora/geonames/documents.json.bz2
存成默认定义的：
documents-2.json.bz2
documents-2.json.offset
然后直接拷贝过去即可：~/.rally/benchmarks/data/geonames


esrally list tracks

This will show the following list:

Name        Description                                          Documents  Compressed Size    Uncompressed Size    Default Challenge        All Challenges
----------  -------------------------------------------------  -----------  -----------------  -------------------  -----------------------  ---------------------------
geonames    POIs from Geonames                                    11396505  252.4 MB           3.3 GB               append-no-conflicts      append-no-conflicts,appe...
geopoint    Point coordinates from PlanetOSM                      60844404  481.9 MB           2.3 GB               append-no-conflicts      append-no-conflicts,appe...
http_logs   HTTP server log data                                 247249096  1.2 GB             31.1 GB              append-no-conflicts      append-no-conflicts,appe...
nested      StackOverflow Q&A stored as nested docs               11203029  663.1 MB           3.4 GB               nested-search-challenge  nested-search-challenge,...
noaa        Global daily weather measurements from NOAA           33659481  947.3 MB           9.0 GB               append-no-conflicts      append-no-conflicts,appe...
nyc_taxis   Taxi rides in New York in 2015                       165346692  4.5 GB             74.3 GB              append-no-conflicts      append-no-conflicts,appe...
percolator  Percolator benchmark based on AOL queries              2000000  102.7 kB           104.9 MB             append-no-conflicts      append-no-conflicts,appe...
pmc         Full text benchmark with academic papers from PMC       574199  5.5 GB             21.7 GB      

这个地址里面 https://github.com/elastic/rally-tracks/tree/master/，进入子目录有各个数据集可配置的参数：

如http_logs：

Parameters （--track-params="bulk_indexing_clients:96"）

This track allows to overwrite the following parameters with Rally 0.8.0+ using --track-params:

bulk_size (default: 5000)
bulk_indexing_clients (default: 8): Number of clients that issue bulk indexing requests.

5,自己定义压测：

参考：https://esrally.readthedocs.io/en/latest/adding_tracks.html

主要步骤：

1，创建目录：~/rally-tracks/tutorial

2，手动下载 geonames 压测数据：　http://download.geonames.org/export/dump/allCountries.zip　。它里面是用ｔａｂ分开的文本文件。需要转成ｊｓｏｎ格式。

用这段ｐｙｔｈｏｎ代码转：

import json

cols = (("geonameid", "int", True),
        ("name", "string", True),
        ("asciiname", "string", False),
        ("alternatenames", "string", False),
        ("latitude", "double", True),
        ("longitude", "double", True),
        ("feature_class", "string", False),
        ("feature_code", "string", False),
        ("country_code", "string", True),
        ("cc2", "string", False),
        ("admin1_code", "string", False),
        ("admin2_code", "string", False),
        ("admin3_code", "string", False),
        ("admin4_code", "string", False),
        ("population", "long", True),
        ("elevation", "int", False),
        ("dem", "string", False),
        ("timezone", "string", False))


def main():
    with open("allCountries.txt", "rt", encoding="UTF-8") as f:
        for line in f:
            tup = line.strip().split("	")
            record = {}
            for i in range(len(cols)):
                name, type, include = cols[i]
                if tup[i] != "" and include:
                    if type in ("int", "long"):
                        record[name] = int(tup[i])
                    elif type == "double":
                        record[name] = float(tup[i])
                    elif type == "string":
                        record[name] = tup[i]
            print(json.dumps(record, ensure_ascii=False))


if __name__ == "__main__":
    main()

存到刚才~/rally-tracks/tutorial目录下，python3 toJSON.py > documents.json

３，对于７.０以下的ｅｓ，保持这个成ｉｎｄｅｘ．ｊｓｏｎ

{
  "settings": {
    "index.number_of_replicas": 0
  },
  "mappings": {
    "docs": {
      "dynamic": "strict",
      "properties": {
        "geonameid": {
          "type": "long"
        },
        "name": {
          "type": "text"
        },
        "latitude": {
          "type": "double"
        },
        "longitude": {
          "type": "double"
        },
        "country_code": {
          "type": "text"
        },
        "population": {
          "type": "long"
        }
      }
    }
  }
}

５，再保存一个ｔｒａｃｋ．ｊｓｏｎ

{
  "version": 2,
  "description": "Tutorial benchmark for Rally",
  "indices": [
    {
      "name": "geonames",
      "body": "index.json",
      "types": [ "docs" ]
    }
  ],
  "corpora": [
    {
      "name": "rally-tutorial",
      "documents": [
        {
          "source-file": "documents.json",
          "document-count": 11658903,
          "uncompressed-bytes": 1544799789
        }
      ]
    }
  ],
  "schedule": [
    {
      "operation": {
        "operation-type": "delete-index"
      }
    },
    {
      "operation": {
        "operation-type": "create-index"
      }
    },
    {
      "operation": {
        "operation-type": "cluster-health",
        "request-params": {
          "wait_for_status": "green"
        }
      }
    },
    {
      "operation": {
        "operation-type": "bulk",
        "bulk-size": 5000
      },
      "warmup-time-period": 120,
      "clients": 8
    },
    {
      "operation": {
        "operation-type": "force-merge"
      }
    },
    {
      "operation": {
        "name": "query-match-all",
        "operation-type": "search",
        "body": {
          "query": {
            "match_all": {}
          }
        }
      },
      "clients": 8,
      "warmup-iterations": 1000,
      "iterations": 1000,
      "target-throughput": 100
    }
  ]
}

其中documents 属性里面的字段值是这么来的：

wc -l documents.json　ｊｓｏｎ个数

stat -f "%z" documents.json　文件大小

７.０以后版本要去掉types 。

６，检查建立成功没有：esrally list tracks --track-path=~/rally-tracks/tutorial

７，执行自己的ｔｒａｃｋ：esrally --distribution-version=6.0.0 --track-path=~/rally-tracks/tutorial

--test-mode　来检测配置文件对否。

这个来生成１０００条数据：head -n 1000 documents.json > documents-1k.json

ESRally压测ElasticSearch性能 CentOS 7.5 安装 Python3.7

1，CentOS 7.5 安装 Python3.7