极狐gitlab集成 sonarqube 7.6
场景：
- 极狐 gitlab 旗舰版已经具备扫描能力，但有时候希望集成第三方扫描工具作为补充。但是，单纯的通过 Runner 调用的方式，只是松散的集成，无法真正形成「深度集成」的体验。
- 若可以「在极狐 gitlab 的漏洞报告中展现第三方扫描工具的扫描结果」，那么，对于集成的体验则会提升很多。本文即针对此目的进行展开说明。
这里，我们引进非常流行的扫描工具 sonarqube 社区版为例加以说明。
环境介绍：
- centos 7.9
- 极狐gitlab v14.9.2 旗舰版
- 极狐gitlab-runner v14.9.1
- sonarqube 7.6-community
- sonar-scanner-cli-4.7.0.2747-linux
1. 部署 sonarqube ce
采用 docker 方式部署
1.1 安装 postgresql
创建目录
mkdir -p /home/sonar/postgres/postgresql
mkdir -p /home/sonar/postgres/data
创建网络
docker network create sonarqube-network
部署 pg
docker run --name postgres -d -p 5432:5432 --network sonarqube-network \
-v /home/sonar/postgres/postgresql:/var/lib/postgresql \
-v /home/sonar/postgres/data:/var/lib/postgresql/data \
-v /etc/localtime:/etc/localtime:ro \
-e POSTGRES_USER=sonar \
-e POSTGRES_PASSWORD=sonar \
-e POSTGRES_DB=sonar \
-e TZ=Asia/Shanghai \
--restart always \
--privileged=true \
--network-alias postgres \
postgres:14.2
1.2 安装 sonarqube
创建工作目录
mkdir -p /data/sonarqube_dir
修改系统参数
echo "vm.max_map_count=262144" >> /etc/sysctl.conf
sysctl -p
运行测试容器
docker run -d --name sonartest sonarqube:7.6-community
拷贝必须文件到本地，并修改权限为777
docker cp sonartest:/opt/sonarqube/conf /data/sonarqube_dir
docker cp sonartest:/opt/sonarqube/data /data/sonarqube_dir
docker cp sonartest:/opt/sonarqube/logs /data/sonarqube_dir
docker cp sonartest:/opt/sonarqube/extensions /data/sonarqube_dir
chmod -R 777 /data/sonarqube_dir/
删除容器
docker stop sonartest
docker rm sonartest
启动 SonarQube，其中SONARQUBE_JDBC_URL需要修改为实际 Postgresql 数据库的 IP
docker run -d --name sonar -p 9000:9000 \
-e ALLOW_EMPTY_PASSWORD=yes \
-e SONARQUBE_DATABASE_USER=sonar \
-e SONARQUBE_DATABASE_NAME=sonar \
-e SONARQUBE_DATABASE_PASSWORD=sonar \
-e SONARQUBE_JDBC_URL="jdbc:postgresql://<postgresql_server_ip>:5432/sonar" \
--privileged=true \
--network sonarqube-network \
--restart always \
-v /data/sonarqube_dir/logs:/opt/sonarqube/logs \
-v /data/sonarqube_dir/conf:/opt/sonarqube/conf \
-v /data/sonarqube_dir/data:/opt/sonarqube/data \
-v /data/sonarqube_dir/extensions:/opt/sonarqube/extensions \
sonarqube:7.6-community
- 必须是 7.6 版本，因为 7.7 版本后官方已经不支持 preview 模式，无法在本地生成测试数据 report.json：How to get sonar-report.json file to display sonar issues at gerrit level itself
新版本可以 docker-compose 一键部署，docker-compose.yml 参考
version: "3.0"
services:
postgresql:
image: postgres:14.2
restart: always
container_name: postgres
networks:
- sonarnet
environment:
POSTGRES_DB: sonar
POSTGRES_USER: sonar
POSTGRES_PASSWORD: sonar
TZ: Asia/Shanghai
volumes:
- /data/sonarqube/postgres/postgresql:/var/lib/postgresql
- /data/sonarqube/postgres/postgresql_data:/var/lib/postgresql/data
- /etc/localtime:/etc/localtime:ro
sonarqube:
image: sonarqube:9.4-community
container_name: sonar
depends_on:
- postgresql
ports:
- "9000:9000"
networks:
- sonarnet
environment:
TZ: Asia/Shanghai
SONARQUBE_JDBC_USERNAME: sonar
SONARQUBE_JDBC_PASSWORD: sonar
SONARQUBE_DATABASE_NAME: sonar
SONARQUBE_JDBC_URL: jdbc:postgresql://postgresql:5432/sonar
volumes:
- /data/sonarqube/conf:/opt/sonarqube/conf
- /data/sonarqube/data:/opt/sonarqube/data
- /data/sonarqube/log:/opt/sonarqube/log
- /data/sonarqube/extensions:/opt/sonarqube/extensions
networks:
sonarnet:
driver: bridge
1.3 生成 token
登陆地址：http://<sonarqube_server_ip>:9000
默认账号：admin/admin
设置必须登陆后才能查看信息：
登陆 -- Administration -- Security -- 开启 Force user authentication -- save
生成 token：
admin 登陆后点击右上角
My Account-Security中生成Token
2. 扫描器镜像制作
2.1 下载 sonar-scanner
参考官方：SonarScanner | SonarQube Docs
下载：
curl -O https://binaries.sonarsource.com/Distribution/sonar-scanner-cli/sonar-scanner-cli-4.7.0.2747-linux.zip
unzip sonar-scanner-cli-4.7.0.2747-linux.zip
简单使用方法：
./sonar-scanner \
-Dsonar.host.url=http://<sonarqube_server_ip>:9000 \
-Dsonar.login=333f3410ce3e575d559329e8f3d0a5d4ec8a499d \
-Dsonar.projectKey=my:test \
-Dsonar.sources=/path_to_codes
如果想要在本地生成json格式报告，则增加如下参数
-Dsonar.report.export.path=report.json \
-Dsonar.analysis.mode=preview
2.2 json 格式转换
sonar-scanner 默认生成 json 格式：
{
"version":"7.6.0.21501",
"issues":[
{
"key":"01803B75AC297CF54F",
"component":"my:test:main.py",
"line":94,
"startLine":94,
"startOffset":4,
"endLine":94,
"endOffset":52,
"message":"Remove this commented out code.",
"severity":"MAJOR",
"rule":"python:S125",
"status":"OPEN",
"isNew":true
},
{
"key":"01803B75AC297CF550",
"component":"my:test:main.py",
"line":28,
"startLine":28,
"startOffset":4,
"endLine":28,
"endOffset":13,
"message":"Rename this local variable \"startLine\" to match the regular expression ^[_a-z][a-z0-9_]*$.",
"severity":"MINOR",
"rule":"python:S117",
"status":"OPEN",
"isNew":true
},
{
"key":"01803B75AC297CF551",
"component":"my:test:main.py",
"line":29,
"startLine":29,
"startOffset":4,
"endLine":29,
"endOffset":11,
"message":"Rename this local variable \"endLine\" to match the regular expression ^[_a-z][a-z0-9_]*$.",
"severity":"MINOR",
"rule":"python:S117",
"status":"OPEN",
"isNew":true
}
],
"components":[
{
"key":"my:test"
},
{
"key":"my:test:main.py",
"path":"main.py",
"status":"ADDED"
}
],
"rules":[
{
"key":"python:S125",
"rule":"S125",
"repository":"python",
"name":"Sections of code should not be commented out"
},
{
"key":"python:S117",
"rule":"S117",
"repository":"python",
"name":"Local variable and function parameter names should comply with a naming convention"
}
],
"users":[
]
}
而 gitlab 报告 json 格式：
{
"category": "test",
"message": "这个问题不怎么严重",
"cve": "python-webhook/MicroService/Service.py:960662f9bd521d32692b07bd8d5b10538924c23c37cec891847f40e436c5c2f:B104",
"severity": "Medium",
"confidence": "Medium",
"scanner": {
"id": "test",
"name": "test"
},
"location": {
"file": "python-webhook/MicroService/Service.py",
"start_line": 26,
"end_line": 28
},
"identifiers": [
{
"type": "bandit_test_id",
"name": "Bandit Test ID B104",
"value": "B104",
"url": "https://bandit.readthedocs.io/en/latest/plugins/b104_hardcoded_bind_all_interfaces.htl"
}
]
}
2.2.1 转换器 converter.py
# coding=utf-8
from datetime import datetime
import json
import hashlib
# {u'INFO': 50, u'BLOCKER': 3, u'MAJOR': 5724, u'CRITICAL': 1089, u'MINOR': 1103}
severity_mapper = {
'INFO': 'Info',
'BLOCKER': 'Unknown',
'MAJOR': 'High',
'CRITICAL': 'Critical',
'MINOR': 'Low'
}
# gitlab values ["Ignore", "Unknown", "Experimental", "Low", "Medium", "High", "Confirmed"]
confidence_mapper = {
'INFO': 'Ignore',
'BLOCKER': 'Unknown',
'MAJOR': 'High',
'CRITICAL': 'Confirmed',
'MINOR': 'Low'
}
def conv(issue):
component = issue.get('component')
start_line = issue.get('startLine')
end_line = issue.get('endLine')
message = issue.get('message')
severity = issue.get('severity')
rule = issue.get('rule')
ret = {
'category': 'sast',
'message': message,
'cve': '',
'severity': severity_mapper.get(severity, 'Unknown'),
'confidence': confidence_mapper.get(severity, 'Unknown'),
'scanner': {
'id': 'sonarqube',
'name': 'sonarqube'
},
'location': {
'file': component.split(':')[-1],
'start_line': start_line,
'end_line': end_line
},
'identifiers': [
{
'type': rule,
'name': rule,
'value': rule,
# 'url': ''
}
]
}
hash_id = hashlib.sha256(json.dumps(ret, sort_keys=True).encode('utf-8')).hexdigest()
ret['id'] = hash_id
return ret
def sonarqube2gitlab(source_file, destination_file):
datetime_obj = datetime.now()
time_str = datetime_obj.strftime('%Y-%m-%dT%H:%M:%S')
gitlab_sast_report = {
'version': '3.0.0',
'vulnerabilities': [],
'remediations': [],
'scan': {
'scanner': {
'id': 'sonarqube',
'name': 'SonarQube',
'url': 'https://docs.sonarqube.org/',
'vendor': {
"name": 'GitLab'
},
'version': '1.7.0'
},
'type': 'sast',
'start_time': time_str,
'end_time': time_str,
'status': 'success'
}
}
with open(source_file, 'r') as f:
report = json.loads(f.read())
issues = report.get('issues', list())
for issue in issues:
issue_gitlab = conv(issue)
gitlab_sast_report['vulnerabilities'].append(issue_gitlab)
with open(destination_file, 'w') as gitlab_sast_report_file:
gitlab_sast_report_file.write(json.dumps(gitlab_sast_report, indent=4, sort_keys=True))
if __name__ == '__main__':
sonarqube2gitlab('.scannerwork/report.json', 'gl-sast-report.json')
2.3 打包镜像
新建 scan.sh
/sonar-scanner/bin/sonar-scanner -Dsonar.host.url=$sonar_host_url -Dsonar.login=$sonar_login -Dsonar.report.export.path=report.json -Dsonar.analysis.mode=preview
python /sonar-scanner/converter.py
- 使用 preview 模式
Dockerfile
from python:3.9.12-slim
workdir /sonar-scanner
copy sonar-scanner-4.7.0.2747-linux /sonar-scanner
add converter.py /sonar-scanner
add scan.sh /sonar-scanner
ENV LANG C.UTF-8
ENV TZ='Asia/Shanghai'
RUN echo 'Asia/Shanghai' > /etc/timezone
构建
docker build -t mysonarscanner:4.7 .
3. 扫描测试
3.1 创建 python 代码
创建测试项目，添加代码 main.py:
# coding=utf-8
# Copyright 2022 Xuefeng Yin, All Rights Reserved
from datetime import datetime
import json
import hashlib
f = open(".scannerwork/report.json", "r")
report = json.loads(f.read())
issues = report.get("issues")
# {u'INFO': 50, u'BLOCKER': 3, u'MAJOR': 5724, u'CRITICAL': 1089, u'MINOR': 1103}
severitys_mapper = {
"INFO": "info",
"BLOCKER":"Unknown",
"MAJOR":"High",
"CRITICAL":"Critical",
"MINOR":"Low",
}
def conv(issue):
component = issue.get("component")
startLine = issue.get("startLine")
endLine = issue.get("endLine")
message = issue.get("message")
severity = issue.get("severity")
rule = issue.get("rule")
ret = {
"category": "sast",
"message": message,
"cve": "",
"severity": severitys_mapper.get(severity, "Unknown"),
"confidence": severitys_mapper.get(severity, "Unknown"),
"scanner": {
"id": "sonarqube",
"name": "sonarqube"
},
"location": {
"file": component.split(":")[-1],
"start_line": startLine,
"end_line": endLine
},
"identifiers": [
{
"type": rule,
"name": rule,
"value": rule,
"url": ""
}
]
}
id = hashlib.sha256(json.dumps(ret, sort_keys=True)).hexdigest()
ret["id"] = id
return ret
dateTimeObj = datetime.now()
timeStr = dateTimeObj.strftime("%Y-%m-%dT%H:%M:%S")
gl_sast_report = {
"version": "3.0.0",
"vulnerabilities": [],
"remediations": [],
"scan": {
"scanner": {
"id": "sonarqube",
"name": "SonarQube",
"url": "https://docs.sonarqube.org/",
"vendor": {
"name": "GitLab"
},
"version": "1.7.0"
},
"type": "sast",
"start_time": timeStr,
"end_time": timeStr,
"status": "success"
}
}
for i, issue in enumerate(issues[:]):
#print("Issue No. %s ---------------------" % i)
#print("SonarQube: %s" % issue)
issue_gitlab = conv(issue)
#print("GitLab: %s" % issue_gitlab)
gl_sast_report["vulnerabilities"].append(issue_gitlab)
gl_sast_report_file = open("gl-sast-report.json", "w")
gl_sast_report_file.write(json.dumps(gl_sast_report, indent=4, sort_keys=True))
gl_sast_report_file.close()
3.2 添加 .gitlab-ci.yml
variables:
sonar_host_url: http://<sonarqube_server_ip>:9000
sonar_login: <sonarqube_server_token>
sonarqube:
image: mysonarscanner:4.7
script:
# 生成扫描参数
- echo -e "sonar.sourceEncoding=UTF-8\nsonar.projectKey=$CI_PROJECT_NAME\nsonar.sources=." >> sonar-project.properties
# 开始扫描并转换 json 格式
- sh /sonar-scanner/scan.sh
artifacts:
reports:
sast:
- gl-sast-report.json
- 注意替换 <sonarqube_server_ip> 与 <sonarqube_server_token> 为真实值
扫描结果：
如果将检查设置为合并时执行，还可以将结果显示到 GitLab 合并请求小部件中：
.gitlab-ci.yml
variables:
sonar_host_url: http://<sonarqube_server_ip>:9000
sonar_login: <sonarqube_server_token>
sonarqube:
image: mysonarscanner:4.7
script:
- echo "sonar.sourceEncoding=UTF-8\nsonar.projectKey=$CI_PROJECT_NAME\nsonar.sources=." >> sonar-project.properties
- sh /sonar-scanner/scan.sh
artifacts:
reports:
sast:
- gl-sast-report.json
only:
- merge_requests
- 注意替换 <sonarqube_server_ip> 与 <sonarqube_server_token> 为真实值
扫描结果：
4. Findbugs 支持
sonarqube 7.6 最高支持 4.0.0 版本：SpotBugs. 4.0.0, sb-contrib 7.4.7, and findsecbugs 1.10.1，2020.2.28 发布；
最新版本 4.1.4：SpotBugs. 4.6.0, sb-contrib 7.4.7, and findsecbugs 1.12.0，2022.4.28 发布；
4.1 安装插件
下载插件：
wget https://github.com/spotbugs/sonar-findbugs/releases/download/4.0.0/sonar-findbugs-plugin-4.0.0.jar
复制插件到 sonarqube server 插件目录
cp sonar-findbugs-plugin-4.0.0.jar /data/sonarqube_dir/extensions/plugins
chmod 777 /data/sonarqube_dir/extensions/plugins/sonar-findbugs-plugin-4.0.0.jar
重启 sonarqube server 生效
docker restart sonar
4.2 启用插件
查看插件
这里设置 FindBugs + FB-Contrib 为默认，也可根据需求选择其他配置
4.3 测试 maven 项目
下载测试项目 push 到 gitlab 示例：https://jihulab.com/ffli/simple-java-maven-app
添加 .gitlab-ci.yml
stages:
- compile
- scan
- package
- test
variables:
sonar_host_url: http://<sonar_server_ip>:9000
sonar_login: <sonar_server_token>
# java 项目需要编译后才能进行扫描
# 这里使用 maven 编译后，然后使用
# gitlab 的 artifacts 功能将结果
# 传递给下一个 sonarqube 任务扫描
compile:
image: maven:3.8.5-jdk-11
stage: compile
variables:
MAVEN_CLI_OPTS: "-s .m2/settings.xml --batch-mode"
MAVEN_OPTS: "-Dmaven.repo.local=.m2/repository"
# 开启 job cache 缓存，可以极大的提高第 2 次以后的编译速度
cache:
paths:
- .m2/repository/
- target/
key: $CI_PROJECT_NAME
script:
# - mvn package
- mvn compile
artifacts:
paths:
- target/
only:
- merge_requests
# sonarqube 扫描任务
sonarqube:
image: mysonarscanner:4.7
stage: scan
script:
# 生成扫描参数，这里的 sonar.sources 设置源码目录，可以设置多个，sonar.java.binaries 设置编译后目录
- echo "sonar.sourceEncoding=UTF-8" >> sonar-project.properties
- echo "sonar.projectKey=$CI_PROJECT_NAME" >> sonar-project.properties
- echo "sonar.sources=./src" >> sonar-project.properties
- echo "sonar.java.binaries=./target" >> sonar-project.properties
- echo "sonar.language=java" >> sonar-project.properties
- echo "sonar.exclusions=**/*.js" >> sonar-project.properties
# 开始扫描并转换 json 格式
- sh /sonar-scanner/scan.sh
artifacts:
reports:
sast:
- gl-sast-report.json
only:
- merge_requests
# 打包
package:
image: maven:3.8.5-jdk-11
stage: package
variables:
MAVEN_CLI_OPTS: "-s .m2/settings.xml --batch-mode"
MAVEN_OPTS: "-Dmaven.repo.local=.m2/repository"
cache:
paths:
- .m2/repository/
- target/
key: $CI_PROJECT_NAME
script:
- mvn -B -DskipTests clean package
artifacts:
paths:
- target/
only:
- merge_requests
# 测试，并使用 gitlab artifacts 功能将 junit 测试结果显示到 gitlab web 页面上
test:
image: maven:3.8.5-jdk-11
stage: test
variables:
MAVEN_CLI_OPTS: "-s .m2/settings.xml --batch-mode"
MAVEN_OPTS: "-Dmaven.repo.local=.m2/repository"
cache:
paths:
- .m2/repository/
- target/
key: $CI_PROJECT_NAME
script:
- mvn test
artifacts:
reports:
junit: target/surefire-reports/*.xml
only:
- merge_requests
- 注意替换 <sonarqube_server_ip> 与 <sonarqube_server_token> 为真实值
结果：
5. 总结
优点：
- 只需使用 sonar token，不需要多余配置，并且不会在 sonarqube server 中生成项目数据与结果数据
缺点：
- 只支持到 sonarqube 7.6，高于这个版本不支持在 scanner 端直接生成 json 报告