写在前面：kestrel当前发展很快，官方文档更新也不及时，比如这个：https://opencybersecurityalliance.org/posts/kestrel-2021-07-26/，巨坑！用最新版本的kestrel，里面的语法都出现解析错误，你说尴尬不。。。没办法，自己看源码搞吧，源码测试里仅仅有单元测试的，没有端到端的测试，只能去看底层源码揣摩使用方法。。。总之，我最终修改了elastic_ecs模块下stix_transmission/api_client.py的源码才搞定。
本文目标：kestrel对接elasticsearch，最终能够使用kestrel查询出ES的数据来。最终效果图：
kestrel查询脚本：
browsers = GET process FROM stixshifter://host110 WHERE [process:name IN ('firefox.exe', 'chrome.exe')] START t'2021-01-01T00:00:00Z' STOP t'2021-12-31T00:00:00Z'
DISP browsers ATTR name, pid
输出：
name pid chrome.exe 12132112121 firefox.exe 121321 firefox.exe 9121321
ES数据源，主要写入了几条：
curl -k -uelastic:changeme -H "Content-Type: application/json" -XPUT https://YOUR_IP:9200/host110/_doc/1 -d '{
"process": {
"name": "firefox.exe",
"content": "I hava a friend who loves smile, gymbo is his name",
"pid": "121321"},
"@timestamp": "2021-11-02T14:44:23.453+0000"
}'
curl -k -uelastic:changeme -H "Content-Type: application/json" -XPUT https://YOUR_IP:9200/host110/_doc/2 -d '{
"process": {
"name": "chrome.exe",
"content": "I hava a friend who loves smile, gymbo is his name",
"pid": "12132112121"},
"@timestamp": "2021-11-02T14:44:23.453+0000"
}'
curl -k -uelastic:changeme -H "Content-Type: application/json" -XPUT https://YOUR_IP:9200/host110/_doc/3 -d '{
"process": {
"name": "twitter.exe",
"content": "I hava a friend who loves smile, gymbo is his name",
"pid": "1213211242123"},
"@timestamp": "2021-11-02T14:44:23.453+0000"
}'
curl -k -uelastic:changeme -H "Content-Type: application/json" -XPUT https://YOUR_IP:9200/host110/_doc/4 -d '{
"process": {
"name": "firefox.exe",
"content": "I hava a friend who loves smile, gymbo is his name",
"pid": "9121321"},
"@timestamp": "2021-11-02T14:44:23.453+0000"
}'
在已经安装好了kestrel的前提下，操作如下：
第一步，设置ES的api_key，可以参考：https://blog.csdn.net/UbuntuTouch/article/details/107181440，我的ES配置：
node.name: node-1 network.host: 0.0.0.0 http.port: 9200 cluster.initial_master_nodes: ["node-1"] xpack.security.enabled: true xpack.security.authc.api_key.enabled: true xpack.security.http.ssl.enabled: true xpack.security.http.ssl.keystore.path: /home/es_user/elasticsearch-7.15.1/config/elastic-stack-ca.p12 xpack.security.http.ssl.truststore.path: /home/es_user/elasticsearch-7.15.1/config/elastic-stack-ca.p12 xpack.security.transport.ssl.enabled: true xpack.security.transport.ssl.verification_mode: certificate xpack.security.transport.ssl.keystore.path: /home/es_user/elasticsearch-7.15.1/config/elastic-stack-ca.p12 xpack.security.transport.ssl.truststore.path: /home/es_user/elasticsearch-7.15.1/config/elastic-stack-ca.p12 http.cors.enabled: true http.cors.allow-origin: "*"
然后就是生成api_key了。
第二步：安装stix_shifter好以后，设置connector环境变量。
pip install stix-shifter-modules-elastic_ecs
export STIXSHIFTER_HOST110_CONFIG='{"auth":{"id":"ZT3QznwBhSK3ri59dnDv", "api_key":"oY8lmKTpTOOXxNqcwJuiqA"}}'
export STIXSHIFTER_HOST110_CONNECTION='{"host":"localhost", "port":9200, "indices":"host110"}'
export STIXSHIFTER_HOST110_CONNECTOR=elastic_ecs
补充说下stix_shifter的用途，本质上是将kestrel lang对ES数据的查询语句转换为ES的查询语法。
例如，我的脚本中：
GET process FROM stixshifter://host110 WHERE [process:name IN ('firefox.exe', 'chrome.exe')] START t'2021-01-01T00:00:00Z' STOP t'2021-12-31T00:00:00Z'
这条语句会被stix_shifter转换为如下ES查询：
{'Content-Type': 'application/json'} search data==> {'_source': {'includes': ['@timestamp', 'source.*', 'destination.*', 'event.*', 'client.*', 'server.*', 'host.*', 'network.*', 'process.*', 'user.*', 'file.*', 'url.*', 'registry.*', 'dns.*']}, 'query': {'query_string': {'query': '(process.pid : ("12132112121" OR "9121321" OR "121321") OR process.ppid : ("12132112121" OR "9121321" OR "121321") OR process.parent.pid : ("12132112121" OR "9121321" OR "121321") OR process.parent.ppid : ("12132112121" OR "9121321" OR "121321")) AND (@timestamp:["2021-01-01T00:00:00.000Z" TO "2021-12-31T00:00:00.000Z"])'}}}
当然要通过追踪kestrel源码追踪分析才知道。
第三步：kestrel在对接ES自签名证书的时候，有bug会一致报这样错误，真是蛋疼啊，我是通过源码分析才发现的。
错误如下：
File "/root/bone/huntingspace/lib/python3.8/site-packages/kestrel_datasource_stixshifter/interface.py", line 151, in query
raise DataSourceError(
kestrel.exceptions.DataSourceError: [ERROR] DataSourceError: data source internal error: STIX-shifter transmission.results() failed. please test data source manually.
然后你深入源码分析才知道是这个错误：
"Wrong certificate: HTTPSConnectionPool(host='localhost', port=9200): Max retries exceeded with url: /_cluster/health?pretty (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate (_ssl.c:1131)')))", 'code': 'authentication_fail'}
就是因为自签名证书的问题，解决方式如下：
修改源码：site-packages/stix_shifter_modules/elastic_ecs/stix_transmission/api_client.py
42 self.client = RestApiClient(connection.get('host'),
43 connection.get('port'),
44 headers,
45 url_modifier_function=url_modifier_function,
46 cert_verify= connection.get('selfSignedCert', True),
47 sni=connection.get('sni', None)
48 )
将46行的True修改为False就好了！
第四步，ES写入mapping和数据，一个bash脚本：
curl -k -uelastic:changeme -H "Content-Type: application/json" -XPUT https://YOUR_IP:9200/host110?pretty
curl -k -uelastic:changeme -H "Content-Type: application/json" -XPUT https://YOUR_IP:9200/host110?pretty -d '{"mappings": {
"properties": {
"process": {
"type": "object",
"properties": {
"@timestamp": {"type": "date"},
"name": {"type": "text"},
"pid": {"type": "text"},
"content": {"type": "text"}
}
}}}}'
curl -k -uelastic:changeme -H "Content-Type: application/json" -XPUT https://YOUR_IP:9200/host110/_doc/1 -d '{
"process": {
"name": "firefox.exe",
"content": "I hava a friend who loves smile, gymbo is his name",
"pid": "121321"},
"@timestamp": "2021-11-02T14:44:23.453+0000"
}'
curl -k -uelastic:changeme -H "Content-Type: application/json" -XPUT https://YOUR_IP:9200/host110/_doc/2 -d '{
"process": {
"name": "chrome.exe",
"content": "I hava a friend who loves smile, gymbo is his name",
"pid": "12132112121"},
"@timestamp": "2021-11-02T14:44:23.453+0000"
}'
curl -k -uelastic:changeme -H "Content-Type: application/json" -XPUT https://YOUR_IP:9200/host110/_doc/3 -d '{
"process": {
"name": "twitter.exe",
"content": "I hava a friend who loves smile, gymbo is his name",
"pid": "1213211242123"},
"@timestamp": "2021-11-02T14:44:23.453+0000"
}'
curl -k -uelastic:changeme -H "Content-Type: application/json" -XPUT https://YOUR_IP:9200/host110/_doc/4 -d '{
"process": {
"name": "firefox.exe",
"content": "I hava a friend who loves smile, gymbo is his name",
"pid": "9121321"},
"@timestamp": "2021-11-02T14:44:23.453+0000"
}'
第5步： 使用最初的kestrel脚本运行即可出现目标结果了。