azkaban 编译部署

安装azkaban单机版和集群版

一、安装单机版

1、安装git

yum install git 

2、拉取azkaban源码

git clone https://github.com/azkaban/azkaban.git

　　

3、开始编译

cd azkaban; 
开始编译
./gradlew build installDist

　　

编译过程很漫长，一直在下载包，可能出现超时的情况，然后再继续编译就好

如果出现JCE报错，则需要手动下载相关的包

testV1_1 FAILED
    java.lang.RuntimeException: java.lang.RuntimeException: org.jasypt.exceptions.EncryptionOperationNotPossibleException: Encryption raised an exception. A possible cause is you are using strong encryption algorithms and you have not installed the Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files in this Java Virtual Machine
        at azkaban.crypto.Crypto.decrypt(Crypto.java:76)
        at azkaban.crypto.DecryptionTest.testV1_1(DecryptionTest.java:35)

        Caused by:
        java.lang.RuntimeException: org.jasypt.exceptions.EncryptionOperationNotPossibleException: Encryption raised an exception. A possible cause is you are using strong encryption algorithms and you have not installed the Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files in this Java Virtual Machine
            at azkaban.crypto.CryptoV1_1.decrypt(CryptoV1_1.java:57)
            at azkaban.crypto.Crypto.decrypt(Crypto.java:74)
            ... 1 more

            Caused by:
            org.jasypt.exceptions.EncryptionOperationNotPossibleException: Encryption raised an exception. A possible cause is you are using strong encryption algorithms and you have not installed the Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files in this Java Virtual Machine
                at org.jasypt.encryption.pbe.StandardPBEByteEncryptor.handleInvalidKeyException(StandardPBEByteEncryptor.java:1073)
                at org.jasypt.encryption.pbe.StandardPBEByteEncryptor.decrypt(StandardPBEByteEncryptor.java:1050)
                at org.jasypt.encryption.pbe.StandardPBEStringEncryptor.decrypt(StandardPBEStringEncryptor.java:725)
                at azkaban.crypto.CryptoV1_1.decrypt(CryptoV1_1.java:55)
                ... 2 more
azkaban.crypto.EncryptionTest > testEncryption FAILED
    org.jasypt.exceptions.EncryptionOperationNotPossibleException: Encryption raised an exception. A possible cause is you are using strong encryption algorithms and you have not installed the Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files in this Java Virtual Machine
        at org.jasypt.encryption.pbe.StandardPBEByteEncryptor.handleInvalidKeyException(StandardPBEByteEncryptor.java:1073)
        at org.jasypt.encryption.pbe.StandardPBEByteEncryptor.encrypt(StandardPBEByteEncryptor.java:924)
        at org.jasypt.encryption.pbe.StandardPBEStringEncryptor.encrypt(StandardPBEStringEncryptor.java:642)
        at azkaban.crypto.CryptoV1_1.encrypt(CryptoV1_1.java:42)
        at azkaban.crypto.Crypto.encrypt(Crypto.java:58)
        at azkaban.crypto.EncryptionTest.testEncryption(EncryptionTest.java:28)

　　解决方法：
下载JCE
https://www.oracle.com/technetwork/cn/java/javase/downloads/jce8-download-2133166-zhs.html
我这边使用的JDK8，包含了JCE所需要的jre8 ，解压出来 ，放到jdk安装目录的 jre/lib/security下面

参考 https://yq.aliyun.com/articles/648399

4、启动服务

cd azkaban-solo-server/build/install/azkaban-solo-server; 
bin/start-solo.sh

　　

注意，这里不能直接进入到bin目录启动，否则会报找不到database.properties的错误

java.io.IOException: Cannot find 'database.properties' file in /opt/local/azkaban-solo-server/build/install/azkaban-solo-server/bin/sql/database.properties
	at azkaban.database.AzkabanDatabaseSetup.loadDBProps(AzkabanDatabaseSetup.java:178)
	at azkaban.database.AzkabanDatabaseSetup.loadTableInfo(AzkabanDatabaseSetup.java:102)
	at azkaban.database.AzkabanDatabaseUpdater.runDatabaseUpdater(AzkabanDatabaseUpdater.java:82)
	at azkaban.soloserver.AzkabanSingleServer.start(AzkabanSingleServer.java:93)
	at azkaban.soloserver.AzkabanSingleServer.main(AzkabanSingleServer.java:58)　　

参考： https://blog.csdn.net/SUDDEV/article/details/101369524

5、查看服务

访问 http://ip:8081 

默认账号密码是 azkaban azkaban

6、停止服务 

bin/shutdown-solo.sh

　　

7、如果是离线安装，可以在可以上网的机器编译，然后将编译好后的

azkaban/azkaban-solo-server/build/install/azkaban-solo-server  复制到安装机器上，然后启动即可。

 二、安装集群版

1、在上面编译好的目录中，azkaban-solo-server是单机版服务，azkaban-exec-server是后台服务，azkaban-web-server 是网页服务
我们现在需要在上面编译好的文件中，将后台服务和网页服务复制到安装目录

cp -r azkaban/azkaban-exec-server/build/install/azkaban-exec-server /opt/local
cp -r azkaban/azkaban-web-server/build/install/azkaban-web-server /opt/local

　　

2、创建azkaban依赖的MySQL数据库

mysql> CREATE DATABASE azkaban;

　　

3、生成初始化MySQL数据库文件

cd azkaban/azkaban-db
../gradlew build installDist

　　

文件在azkaban/azkaban-db/build/sql，放到MySQL执行

4、配置和启动后台服务

cd /opt/local/azkaban-exec-server/conf
vim azkaban.properties，修改MySQL配置、时区

default.timezone.id=Asia/Shanghai
executor.port=12321
mysql.port=3306
mysql.host=数据库ip
mysql.database=azkaban
mysql.user=数据库账号
mysql.password=数据库密码

　

启动服务

bin/start-exec.sh

5、配置和启动网页服务

cd /opt/local/azkaban-web-server/conf

vim azkaban.properties

default.timezone.id=Asia/Shanghai

mysql.port=3306
mysql.host=数据库ip
mysql.database=azkaban
mysql.user=数据库账号
mysql.password=数据库密码

　　

启动服务 bin/start-web.sh 

然后发现报错了，发现没有可用的executor

2020/05/15 15:05:46.300 +0800  INFO [AzkabanWebServer] [main] [Azkaban] ************* loginLoggerObjName is null, m
ake sure there is a logger with name azkaban.webapp.servlet.LoginAbstractAzkabanServlet
2020/05/15 15:05:46.301 +0800  INFO [ExecutorManager] [main] [Azkaban] Initializing executors from database.
2020/05/15 15:05:46.302 +0800 ERROR [ExecutorManager] [main] [Azkaban] No active executors found
2020/05/15 15:05:46.303 +0800 ERROR [StdOutErrRedirect] [main] [Azkaban] Exception in thread "main" 
2020/05/15 15:05:46.303 +0800 ERROR [StdOutErrRedirect] [main] [Azkaban] azkaban.executor.ExecutorManagerException:
 No active executors found
2020/05/15 15:05:46.303 +0800 ERROR [StdOutErrRedirect] [main] [Azkaban] 	at azkaban.executor.ActiveExecutors
.setupExecutors(ActiveExecutors.java:52)
2020/05/15 15:05:46.303 +0800 ERROR [StdOutErrRedirect] [main] [Azkaban] 	at azkaban.executor.ExecutorManager
.setupExecutors(ExecutorManager.java:192)
2020/05/15 15:05:46.303 +0800 ERROR [StdOutErrRedirect] [main] [Azkaban] 	at azkaban.executor.ExecutorManager
.initialize(ExecutorManager.java:127)
2020/05/15 15:05:46.303 +0800 ERROR [StdOutErrRedirect] [main] [Azkaban] 	at azkaban.executor.ExecutorManager
.start(ExecutorManager.java:141)
2020/05/15 15:05:46.303 +0800 ERROR [StdOutErrRedirect] [main] [Azkaban] 	at azkaban.webapp.AzkabanWebServer.
launch(AzkabanWebServer.java:234)
2020/05/15 15:05:46.303 +0800 ERROR [StdOutErrRedirect] [main] [Azkaban] 	at azkaban.webapp.AzkabanWebServer.
main(AzkabanWebServer.java:227)

　　

我们改一下数据库，激活executor

use azkaban;
update azkaban.executors set active=1;

　　

再启动一次服务，即可

对于集群部署，将azkaban-web-server按照上面的步骤部署，然后执行 curl -G "http:ip:port/executor?action=activate" 激活节点，或者去MySQL改数据激活