• presto-admin 安装及管理presto集群


    rpm:https://repo1.maven.org/maven2/com/facebook/presto/presto-server-rpm/
    https://prestodb.io/presto-admin/docs/current/presto-admin-commands.html#configuration-deploy

    presto-admin:
    https://docs.starburstdata.com/302-e/presto-admin/presto-admin-commands.html
    https://repo1.maven.org/maven2/com/facebook/presto/presto-server-rpm/
    https://prestodb.io/presto-admin/docs/current/presto-admin-commands.html#configuration-deploy
    https://prestodb.io/presto-admin/docs/current/presto-admin-commands.html#server-install
    安装presto-admin依赖的库文件
    yum install postgresql-devel*
    pip3 install psycopg2
    pip3 install PyHive
    pip3 install sqlalchemy
    pip3 install pandas
    pip3 install requests
    pip3 install sasl
    pip3 install thrift
    pip3 install thrift-sasl


    presto-admin 安装
    python 2.6 2.7(需要安装pip)

    curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py

    ll get-pip.py
    -rw-r--r--. 1 root root 1885433 Sep 25 15:38 get-pip.py

    python get-pip.py

    Collecting pip
    Downloading pip-20.2.3-py2.py3-none-any.whl (1.5 MB)
    |████████████████████████████████| 1.5 MB 126 kB/s
    Collecting setuptools
    Downloading setuptools-44.1.1-py2.py3-none-any.whl (583 kB)
    |████████████████████████████████| 583 kB 67 kB/s
    Collecting wheel
    Downloading wheel-0.35.1-py2.py3-none-any.whl (33 kB)
    Installing collected packages: pip, setuptools, wheel
    Successfully installed pip-20.2.3 setuptools-44.1.1 wheel-0.35.1

    clone presto-admin
    git clone https://github.com/prestodb/presto-admin.git
    presto-admin
    cd presto-admin
    pip install -r requirements.txt
    python setup.py develop

    Using /usr/lib/python2.7/site-packages
    Searching for virtualenv==12.0.7
    Best match: virtualenv 12.0.7
    Adding virtualenv 12.0.7 to easy-install.pth file
    Installing virtualenv script to /usr/bin
    Installing virtualenv-3.4 script to /usr/bin

    Using /usr/lib/python2.7/site-packages
    Searching for pyflakes==1.0.0
    Best match: pyflakes 1.0.0
    Processing pyflakes-1.0.0-py2.7.egg
    pyflakes 1.0.0 is already the active version in easy-install.pth
    Installing pyflakes script to /usr/bin

    Using /usr/lib/python2.7/site-packages/pyflakes-1.0.0-py2.7.egg
    Searching for pep8==1.7.1
    Best match: pep8 1.7.1
    Processing pep8-1.7.1-py2.7.egg
    pep8 1.7.1 is already the active version in easy-install.pth
    Installing pep8 script to /usr/bin

    Using /usr/lib/python2.7/site-packages/pep8-1.7.1-py2.7.egg
    Searching for mccabe==0.4.0
    Best match: mccabe 0.4.0
    Processing mccabe-0.4.0-py2.7.egg
    mccabe 0.4.0 is already the active version in easy-install.pth

    Using /usr/lib/python2.7/site-packages/mccabe-0.4.0-py2.7.egg
    Searching for ecdsa==0.16.0
    Best match: ecdsa 0.16.0
    Processing ecdsa-0.16.0-py2.7.egg
    ecdsa 0.16.0 is already the active version in easy-install.pth

    Using /usr/lib/python2.7/site-packages/ecdsa-0.16.0-py2.7.egg
    Searching for typing==3.7.4.3
    Best match: typing 3.7.4.3
    Adding typing 3.7.4.3 to easy-install.pth file

    Using /usr/lib/python2.7/site-packages
    Searching for enum34==1.1.10
    Best match: enum34 1.1.10
    Adding enum34 1.1.10 to easy-install.pth file

    Using /usr/lib/python2.7/site-packages
    Finished processing dependencies for prestoadmin===2.6-SNAPSHOT

    [root@DM02 presto-admin]# whereis presto-admin
    presto-admin: /usr/bin/presto-admin

    安装完之后会自动在/root/目录下产生一个目录/root/.prestoadmin:
    [root@DM02 ~]# cd /root/.prestoadmin/
    [root@DM02 .prestoadmin]# ls
    catalog config.json coordinator log workers
    catalog有用添加catalog,通过prestoadmin添加的catalog都会先写到该 目录下,
    然后分发到集群中的其他节点。
    修改config.json 指定coordinator,workers 相关信息
    [root@DM02 .prestoadmin]# cat config.json
    {
    "username":"root",
    "coordinator":"172.25.5.38",
    "workers":[
    "172.25.5.38","172.25.5.39","172.25.5.40"
    ],
    "port":22
    }
    修改config.properties 指定discovery.uri=http://172.25.5.38:8080 节点
    [root@DM02 .prestoadmin]# cd coordinator/
    [root@DM02 coordinator]# ls
    config.properties jvm.config node.properties
    [root@DM02 coordinator]# cat config.properties
    query.max-memory=50GB
    node-scheduler.include-coordinator=true
    discovery.uri=http://localhost:8080
    discovery-server.enabled=true
    http-server.http.port=8080
    coordinator=true
    query.max-memory-per-node=8GB
    [root@DM02 coordinator]# cat jvm.config
    -server
    -Xmx16G
    -XX:-UseBiasedLocking
    -XX:+UseG1GC
    -XX:G1HeapRegionSize=32M
    -XX:+ExplicitGCInvokesConcurrent
    -XX:+HeapDumpOnOutOfMemoryError
    -XX:+UseGCOverheadLimit
    -XX:+ExitOnOutOfMemoryError
    -XX:ReservedCodeCacheSize=512M
    -DHADOOP_USER_NAME=hive
    [root@DM02 coordinator]# cat node.properties
    node.launcher-log-file=/var/log/presto/launcher.log
    node.server-log-file=/var/log/presto/server.log
    catalog.config-dir=/etc/presto/catalog
    node.data-dir=/var/lib/presto/data
    node.environment=presto
    plugin.dir=/usr/lib/presto/lib/plugin

    准备好rpm安装包
    /opt/presto-server-rpm-0.224.rpm
    [root@DM02 ~]# presto-admin server install /opt/presto-server-rpm-0.224.rpm
    Using rpm_specifier as a local path
    Fetching local presto rpm at path: /opt/presto-server-rpm-0.224.rpm
    Found existing rpm at: /opt/presto-server-rpm-0.224.rpm
    Deploying rpm on 172.25.5.40...
    Deploying rpm on 172.25.5.39...
    Deploying rpm on 172.25.5.38...
    Package deployed successfully on: 172.25.5.40
    Package deployed successfully on: 172.25.5.38
    Package deployed successfully on: 172.25.5.39
    Package installed successfully on: 172.25.5.40
    Deploying configuration on: 172.25.5.40
    Package installed successfully on: 172.25.5.39
    Deploying configuration on: 172.25.5.39
    Deploying tpch.properties catalog configurations on: 172.25.5.40
    Deploying tpch.properties catalog configurations on: 172.25.5.39
    Package installed successfully on: 172.25.5.38
    Deploying configuration on: 172.25.5.38
    Deploying tpch.properties catalog configurations on: 172.25.5.38

    [root@DM02 presto]# presto-admin server start
    [172.25.5.40] out: Starting presto
    [172.25.5.40] out: Started as 1702
    [172.25.5.40] out:
    Waiting to make sure we can connect to the Presto server on 172.25.5.40, please wait. This check will time out after 2 minutes if the server does not respond.
    [172.25.5.38] out: Starting presto
    [172.25.5.39] out: Starting presto
    [172.25.5.39] out: Started as 27667
    [172.25.5.39] out:
    Waiting to make sure we can connect to the Presto server on 172.25.5.39, please wait. This check will time out after 2 minutes if the server does not respond.
    [172.25.5.38] out: Started as 29199
    [172.25.5.38] out:
    Waiting to make sure we can connect to the Presto server on 172.25.5.38, please wait. This check will time out after 2 minutes if the server does not respond.
    Server started successfully on: 172.25.5.38
    Server started successfully on: 172.25.5.40
    Server started successfully on: 172.25.5.39


    [root@DM02 presto]# presto-admin server status
    Server Status:
    172.25.5.38(IP: 172.25.5.38, Roles: coordinator, worker): Running
    Node URI(http): http://172.25.5.38:8080
    Presto Version: 0.224-b3a49f7
    Node status: active
    Catalogs: system, tpch
    Server Status:
    172.25.5.39(IP: 172.25.5.39, Roles: worker): Running
    Node URI(http): http://172.25.5.39:8080
    Presto Version: 0.224-b3a49f7
    Node status: active
    Catalogs: system, tpch
    Server Status:
    172.25.5.40(IP: 172.25.5.40, Roles: worker): Running
    Node URI(http): http://172.25.5.40:8080
    Presto Version: 0.224-b3a49f7
    Node status: active
    Catalogs: system, tpch

    有报错看日志:
    This could mean that the server failed to start or that there was no coordinator or worker up.
    Please check /var/log/presto/server.log and /var/log/presto/launcher.log

    http://172.25.5.38:8080/ui/

  • 相关阅读:
    BZOJ2241 [SDOI2011]打地鼠 【模拟】
    K-D tree入门
    BZOJ1924 [Sdoi2010]所驼门王的宝藏 【建图 + tarjan】
    BZOJ1925 [Sdoi2010]地精部落 【dp】
    BZOJ1926 [Sdoi2010]粟粟的书架 【主席树 + 二分 + 前缀和】
    BZOJ2457 [BeiJing2011]双端队列 【贪心】
    二进制集合枚举子集
    BZOJ1923 [Sdoi2010]外星千足虫 【高斯消元】
    BZOJ1922 [Sdoi2010]大陆争霸 【最短路】
    HDU3157:Crazy Circuits——题解
  • 原文地址:https://www.cnblogs.com/songyuejie/p/13710872.html
Copyright © 2020-2023  润新知