Azkaban.Sqoop

1-Azkaban介绍

 ============================================================================================================================================================

2-Azkaban编译和安装模式

========================================================================================================================================================== 

 3-Azkaban-solo模式-安装

cd /export/softwares

tar ‐zxvf azkaban‐solo‐server‐0.1.0‐SNAPSHOT.tar.gz ‐C ../servers/

 

 

 ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

cd /export/servers/azkaban‐solo‐server‐0.1.0‐SNAPSHOT/conf

vim azkaban.properties

 

```shell

```properties

default.timezone.id=Asia/Shanghai

 --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

memCheck.enabled=false 

 ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

cd /export/servers/azkaban‐solo‐server‐0.1.0‐SNAPSHOT

bin/start‐solo.sh

 ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

 ===========================================================================================================================================================

4-Azkaban-solo模式-使用

 =============================================================================================================================================================

5-Azkaban-two_server模式-安装1

需要登录mysql:   MYSQL的作用是用来存储azkaban中间产生的数据

CREATE DATABASE azkaban;

CREATE USER 'azkaban'@'%' IDENTIFIED BY 'azkaban';

GRANT all privileges ON azkaban.* to 'azkaban'@'%' identified by

'azkaban' WITH GRANT OPTION;

flush privileges;

use azkaban;

source /export/software/create‐all‐sql‐0.1.0‐SNAPSHOT.sql;

----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

第二步：centos6.9安装mysql

第一步：在线安装mysql相关的软件包

yum install mysql mysql-server mysql-devel

第二步：启动mysql的服务

/etc/init.d/mysqld start

第三步：通过mysql安装自带脚本进行设置

/usr/bin/mysql_secure_installation

第四步：进入mysql的客户端然后进行授权

grant all privileges on *.* to 'root'@'%' identified by '123456' with grant option;

flush privileges;

重新打开一个终端: 

 

 回到前一个终端:

 ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

 

cd /export/softwares

tar ‐zxvf azkaban‐web‐server‐0.1.0‐SNAPSHOT.tar.gz ‐C ../servers/

cd /export/servers

mv azkaban‐web‐server‐0.1.0‐SNAPSHOT/ azkaban‐web‐server‐3.51.0

* 解压azkaban‐exec‐server

```shell

cd /export/softwares

tar ‐zxvf azkaban‐exec‐server‐0.1.0‐SNAPSHOT.tar.gz ‐C ../servers/

cd /export/servers

mv azkaban‐exec‐server‐0.1.0‐SNAPSHOT/ azkaban‐exec‐server‐3.51.0

 

 

 

 ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

 

安装ssl安全认证，允许我们使用https的方式访问我们的azkaban的web服务(SSL为网络通信中的一种协议,保证网络通信的安全性;装了之后,以后访问azkaban的时候可以采用https这种方式访问)

密码一定要一个个的字母输入，或者粘贴也行

cd /export/servers/azkaban-web-server-3.51.0

keytool -keystore keystore -alias jetty -genkey -keyalg RSA

keytool为安全认证的工具

 

 ===============================================================================================================================================================

6-Azkaban-two_server模式-安装2

vim azkaban.properties

 

# Azkaban Personalization Settings

azkaban.name=Azkaban

azkaban.label=My Azkaban

azkaban.color=#FF3601

azkaban.default.servlet.path=/index

web.resource.dir=web/

default.timezone.id=Asia/Shanghai

# Azkaban UserManager class

user.manager.class=azkaban.user.XmlUserManager

user.manager.xml.file=conf/azkaban‐users.xml

# Loader for projects

executor.global.properties=conf/global.properties

azkaban.project.dir=projects

# Velocity dev mode

velocity.dev.mode=false

# Azkaban Jetty server properties.

jetty.use.ssl=true

jetty.maxThreads=25

jetty.port=8081

jetty.ssl.port=8443

jetty.keystore=/export/servers/azkaban‐web‐server‐3.51.0/keystore

jetty.password=azkaban

jetty.keypassword=azkaban

jetty.truststore=/export/servers/azkaban‐web‐server‐3.51.0/keystore

jetty.trustpassword=azkaban

# Azkaban Executor settings

# mail settings

mail.sender=

mail.host=

# User facing web server configurations used to construct the user facing

server URLs. They are useful when there is a reverse proxy between

Azkaban web servers and users.

# enduser ‐> myazkabanhost:443 ‐> proxy ‐> localhost:8081

# when this parameters set then these parameters are used to generate

email links.

# if these parameters are not set then jetty.hostname, and jetty.port(ifssl configured jetty.ssl.port) are used.

# azkaban.webserver.external_hostname=myazkabanhost.com

# azkaban.webserver.external_ssl_port=443

# azkaban.webserver.external_port=8081

job.failure.email=

job.success.email=

lockdown.create.projects=false

cache.directory=cache

# JMX stats

jetty.connector.stats=true

executor.connector.stats=true

# Azkaban mysql settings by default. Users should configure their own

username and password.

database.type=mysql

mysql.port=3306

mysql.host=node03

mysql.database=azkaban

mysql.user=azkaban

mysql.password=azkaban

mysql.numconnections=100

#Multiple Executor

azkaban.use.multiple.executors=true

#azkaban.executorselector.filters=StaticRemainingFlowSize,MinimumFreeMemo

ry,CpuStatus

azkaban.executorselector.comparator.NumberOfAssignedFlowComparator=1

azkaban.executorselector.comparator.Memory=1

azkaban.executorselector.comparator.LastDispatched=1

azkaban.executorselector.comparator.CpuUsage=1

azkaban.activeexecutor.refresh.milisecinterval=10000

azkaban.queueprocessing.enabled=true

azkaban.activeexecutor.refresh.flowinterval=10

azkaban.executorinfo.refresh.maxThreads=10 

 

 ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

vim azkaban.properties

 

# Azkaban Personalization Settings

azkaban.name=Azkaban

azkaban.label=My Azkaban

azkaban.color=#FF3601

azkaban.default.servlet.path=/index

web.resource.dir=web/

default.timezone.id=Asia/Shanghai

# Azkaban UserManager class

user.manager.class=azkaban.user.XmlUserManager

user.manager.xml.file=conf/azkaban‐users.xml

# Loader for projects

executor.global.properties=conf/global.properties

azkaban.project.dir=projects

# Velocity dev mode

velocity.dev.mode=false

# Azkaban Jetty server properties.

jetty.use.ssl=true

jetty.maxThreads=25

jetty.port=8081

jetty.keystore=/export/servers/azkaban‐web‐server‐3.51.0/keystore

jetty.password=azkaban

jetty.keypassword=azkaban

jetty.truststore=/export/servers/azkaban‐web‐server‐3.51.0/keystore

jetty.trustpassword=azkaban

# Where the Azkaban web server is located

azkaban.webserver.url=https://node03:8443

# mail settings

mail.sender=

mail.host=

# User facing web server configurations used to construct the user facing

server URLs. They are useful when there is a reverse proxy between

Azkaban web servers and users.

# enduser ‐> myazkabanhost:443 ‐> proxy ‐> localhost:8081

# when this parameters set then these parameters are used to generate

email links.

# if these parameters are not set then jetty.hostname, and jetty.port(ifStep 6: azkaban executor server 安装 第二步：添加插件

将我们编译后的C文件execute-as-user.c

上传到这个目录来/export/servers/azkaban-exec-server-3.51.0/plugins/jobtypes

或者直接将我们/export/softwares下面的文件拷贝过来也行

然后执行以下命令生成execute-as-user

ssl configured jetty.ssl.port) are used.

# azkaban.webserver.external_hostname=myazkabanhost.com

# azkaban.webserver.external_ssl_port=443

# azkaban.webserver.external_port=8081

job.failure.email=

job.success.email=

lockdown.create.projects=false

cache.directory=cache

# JMX stats

jetty.connector.stats=true

executor.connector.stats=true

# Azkaban plugin settings

azkaban.jobtype.plugin.dir=plugins/jobtypes

# Azkaban mysql settings by default. Users should configure their own

username and password.

database.type=mysql

mysql.port=3306

mysql.host=node03

mysql.database=azkaban

mysql.user=azkaban

mysql.password=azkaban

mysql.numconnections=100

# Azkaban Executor settings

executor.maxThreads=50

executor.flow.threads=30 

 

 

============================================================================================================================================================

7-Azkaban-two_server模式-安装3和启动运行

/export/servers/azkaban-exec-server-3.51.0/plugins/jobtypes

 

yum ‐y install gcc‐c++

cd /export/servers/azkaban‐exec‐server‐3.51.0/plugins/jobtypes

gcc execute‐as‐user.c ‐o execute‐as‐user

chown root execute‐as‐user

chmod 6050 execute‐as‐user

 ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

cd /export/servers/azkaban‐exec‐server‐3.47.0/plugins/jobtypes

 

vim commonprivate.properties

 

execute.as.user=false

memCheck.enabled=false

azkaban.native.lib=/export/servers/azkaban‐exec‐server‐3.51.0/plugins/jobtypes 

 

 ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

(1)第一步：启动azkaban exec server

cd /export/servers/azkaban‐exec‐server‐3.51.0

bin/start‐exec.sh 

(2)第二步：激活我们的exec-server

node03机器任意目录下执行以下命令 

curl ‐G "node03:$(<./executor.port)/executor?action=activate" && echo

 (3)第三步：启动azkaban-web-server

cd /export/servers/azkaban‐web‐server‐3.51.0/

bin/start‐web.sh 

 ============================================================================================================================================================