• Elasticsearch基本操作RestApi


    Elasticsearch基础Api:

    大家一定是使用过关系型数据库的,那么我们现在从关系型数据库的概念扩展讲解一下ES对应关系型数据库的内容

    对应关系表
      Elasticsearch                =>     MySQL

      索引库(indices)        =>   Database 数据库
      类型(type)                   =>  Table 数据表
      文档(Document)       =>   Row 行
      域字段(Field)            =>  Columns 列
      映射配置(mappings) =>  每个列的约束(类型、长度)

    知识点思维导图

    基础增删改查

    语法

    (1) 创建/全量替换

    记录不存在就是创建, 否则是全量替换.

    PUT /index/type/id  
    {
        "属性名" : "value"
        ...
    }

    (2) 创建/修改创建

    //如果不传id, 则系统自动生成一个UUID
      POST /index/type/
      {
          "属性名":修改值
      }
      或者
      POST /index/type/id
      {
          "属性名":修改值
      }

    修改

    //没有带上的属性会被清除
    POST /index/type/id
    {
        "属性名":修改值
    }

    (3) 查询

    GET /index/type/id

    (4) 删除

    只是逻辑删除, 将其标记为delete, 当数据越来越多时, ES会自动物理删除

    DELETE /index/type/id

    实例

    (1) 新增

    PUT /company/employee/1
    {
      "age":25,
      "salary":"20k",
      "skill":["java","mysql","es"]
    }

    或者

    POST /company/employee
    {
      "age":30,
      "salary":"30k",
      "skill":["python","redis","hadoop"]
    }

    POST /company/employee/2
    { "age":22, "salary":"10k", "skill":["python","java"] }

    POST 命令新增数据时, 如果不传id, 则系统自动生成一个UUID.

    (2) 查询

    GET /company/employee/1

    (3) 修改

    POST /company/employee/1
    {
      "age":40,
      "salary":"100k"
    }

    PUT /company/employee/1
    {
      "age":18,
      "salary":"50K",
      "skill":["c++"]
    }

    PUT 是全量替换, 且是幂等操作. 上面的POST修改, skill属性被清除了, 属性现在只有age和salary.

    幂等操作: GET, PUT, DELETE都是幂等操作, 执行多次与执行一次结果一样.

    (4) 删除

    DELETE /company/employee/1

    Partial Update

    部分更新.

    (1) 语法

    post /index/type/id/_update 
    {
       "doc": {
          "属性名" : "value"
          ...
       }
    }

    2) 与全量替换和普通修改的区别

    在应用程序中, 全量替换是先从ES中查询出记录, 然后在修改, 一个属性都不能少. 而部分更新就不用先从ES查询记录了, 可以直接修改, 且不用所有属性都列出.

    部分更新如果只包含部分属性, 那么其他没有被包含的属性仍然存在, 但普通修改其他没有被包含的属性就直接清除了.

    (3) 实例

    a. 先添加一条数据

    PUT /school/student/1
    {
      "name": "alice",
      "age": 17
    }

    b. 测试部分更新

    //部分更新
    POST /school/student/1/_update
    {
      "doc": {
        "name" : "Tom"
      }
    }
    
    //查询
    GET /school/student/1
    
    //结果
    {
      "_index": "school",
      "_type": "student",
      "_id": "1",
      "_version": 7,
      "found": true,
      "_source": {
        "name": "Tom",
        "age": 17
      }
    }

    c. 测试普通修改

    //普通修改
    POST /school/student/1
    {
      "name" : "Linda"
    }
    
    //查询
    GET school/student/1
    
    //结果
    {
      "_index": "school",
      "_type": "student",
      "_id": "1",
      "_version": 8,
      "found": true,
      "_source": {
        "name": "Linda"
      }
    }

    Bulk Operation

    批量查询和批量增删改的语法不同, 所以分开来介绍.

    批量查询

    GET /_mget
    {
      "docs" : [
        {
          "_index" : "company",
          "_type" : "employee",
          "_id" : 1
        },
         {
          "_index" : "company",
          "_type" : "employee",
          "_id" : 2
        }
      ]
    }
    
    GET /company/_mget
    {
      "docs" : [
        {
          "_type" : "employee",
          "_id" : 1
        },
         {
          "_type" : "employee",
          "_id" : 2
        }
      ]
    }
    
    GET /company/employee/_mget
    {
      "ids" : [1,2]
    }

    批量增删改

    (1) 语法

    每一个操作要两个json串,语法如下:

    POST /index/type/_bulk
    {"action": {"metadata"}}
    {"data"}

    ndex和type可以放入metadata中. 每个json串不能换行, 只能放一行. 同时一个json串和一个json串之间, 必须有一个换行.

    (2) action类型

      delete  删除.

      create  强制创建. PUT /index/type/id/_create

      index  创建或替换.

      update  属于部分更新.

    (3) 实例

    POST /_bulk
    {"delete" : {"_index":"company", "_type":"employee","_id":"1"}}
    {
    "create" :{"_index":"company","_type":"employee","_id":"2"}} {"name":"tyshawn", "age":18}
    {
    "index":{"_index":"company","_type":"employee","_id":"3"}} {"name":"lee", "age":24}
    {
    "update":{"_index":"company","_type":"employee","_id":"2"}} {"doc":{"age":30}}

    搜索

    添加搜索实例数据.

    PUT /website/article/1
    {
      "post_date": "2017-01-01",
      "title": "my first article",
      "content": "this is my first article in this website",
      "author_id": 11401,
      "tags": [
          "java",
          "c"
        ]
    }
    
    PUT /website/article/2
    {
      "post_date": "2017-01-02",
      "title": "my second article",
      "content": "this is my second article in this website",
      "author_id": 11402,
      "tags": [
          "redis",
          "linux"
        ]
    }
    
    PUT /website/article/3
    {
      "post_date": "2017-01-03",
      "title": "my third article",
      "content": "this is my third article in this website",
      "author_id": 11403,
      "tags": [
          "elaticsearch",
          "kafka"
        ]
    }

     mapping结构如下:

    GET /website/_mapping/article
    
    {
      "website": {
        "mappings": {
          "article": {
            "properties": {
              "author_id": {
                "type": "long"
              },
              "content": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "post_date": {
                "type": "date"
              },
              "tags": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "title": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              }
            }
          }
        }
      }
    }

    当type = text时, ES默认会设置两个field, 一个分词, 一个不分词. 如content会分词, 而content.keyword不分词, content.keyword最多保留256个字符.

    通用语法

    (1) 搜索所有index数据.

    GET /_search

    (2) 搜索指定index, type下的数据(index和type可以有多个)

    GET /index1/_search
    GET /index1,index2/_search
    GET /index1/type1/_search
    GET /index1/type1,type2/_search
    GET /index1,index2/type1,type2/_search

    (3) 搜索所有index下的指定type的数据.

    GET /_all/employee,product/_search

    下面再介绍Elasticsearch搜索的条件语法.

    query string search

    这个查询就类似于HTTP里的GET请求, 参数放在URL上.

    (1) 语法

    GET /index/type/_search?q=属性名:属性值
    GET /index/type/_search?q=+属性名:属性值
    GET /index/type/_search?q=-属性名:属性值
    也可以省略属性名, 直接q=属性值
    GET /index/type/_search?q=属性值

    (2) + 和 - 的区别

    默认是+, 指必须包含该字段, - 指不包含该字段.

    (3) 实例

    GET /website/article/_search?q=author_id:11403
    GET /website/article/_search?q=-author_id:11403
    GET /website/article/_search?q=11403

    (4) _all metadata的原理

    GET /index/type/_search?q=属性值.

    这个语句是从所有属性中搜索包含指定的关键字的数据. 那么ES在搜索时是遍历所有document中的每一个field吗? 不是的, 我们在插入一条document时, ES会自动将多个field的值, 全部用字符串的方式串联起来, 变成一个长的字符串(以空格作为分隔符),作为_all field的值,同时进行分词建立倒排索引. 如果在搜索时没有指定属性名, 就会默认搜索_all field. (生产环境不使用)

    query DSL

    DSL, Domain Specified Language,特定领域的语言. 这个查询就类似于HTTP里的POST请求, 参数放在body中.

    实例: 从website索引中查询所有文章

    GET /website/article/_search
    {
      "query": {
        "match_all": {}
      }
    }

    full-text search

    全文搜索.

    (1) 基础使用

    a. 搜索标题中包含first或second的文章

    GET /website/article/_search
    {
      "query": {
        "match": {
          "title": "first second"
        }
      }
    }
    
    或者
    
    GET /website/article/_search
    {
      "query": {
        "match": {
          "title": {
            "query": "first second"
            , "operator": "or"
          }
        }
      }
    }
    
    或者
    
    GET /website/article/_search
    {
      "query": {
        "bool": {
          "should": [
            {"match": {
              "title":"first"
            }},
            {"match": {
              "title":"second"
            }}
          ]
        }
      }
    }

    b. 搜索标题中包含first和second的文章

    GET /website/article/_search
    {
      "query": {
        "match": {
          "title": {
            "query": "first second"
            , "operator": "and"
          }
        }
      }
    }
    
    或者
    
    GET /website/article/_search
    {
      "query": {
        "bool": {
          "must": [
            {"match": {
              "title": "first"
            }},
            {"match": {
              "title": "second"
            }}
          ]
        }
      }
    }

    c. 搜索标题中至少包含first, second, third, fourth中三个单词的文章.

    GET /website/article/_search
    {
      "query": {
        "match": {
          "title": {
            "query": "first second third fourth",
            "minimum_should_match": "75%"
          }
        }
      }
    }
    
    或者
    
    GET /website/article/_search
    {
      "query": {
        "bool": {
          "should": [
            {"match": {
              "title": "first"
            }},
            {"match": {
              "title": "second"
            }},
            {"match": {
              "title": "third"
            }},
            {"match": {
              "title": "fourth"
            }}
          ],
          "minimum_number_should_match": 3
        }
      }
    }

    d. 从website索引中查询, 标题必须包含elasticsearch,内容可以包含elasticsearch也可以不包含,作者id必须不为111.

    GET /website/article/_search
    {
      "query": {
        "bool": {
          "must": [
            {
              "match": {
                "title":"elasticsearch"
              }
            }
          ],
          "should": [
            {
              "match": {
                "content":"elasticsearch"
              }
            }
          ], 
          "must_not": [
            {
              "match": {
                 "author_id":"111"
              }
            }
          ]
        }
      }
    }

    e. 从website索引中查询, 标题包含first, 同时按作者id降序排序

    GET /website/article/_search
    {
      "query": {
        "match": {
          "title": "first"
        }
      },
      "sort": [
        {
          "author_id": {
            "order": "desc"
          }
        }
      ]
    }

    f. 从website索引中分页查询,总共3篇文章,假设每页就显示1篇文章,现在显示第2页

    GET /website/article/_search
    {
      "query": {
        "match_all": {}
      },
      "from": 1,
      "size": 1
    }

    g. 从website索引中查询所有文章, 只显示post_date, title两个属性.

    GET /website/article/_search
    {
      "query": {
        "match_all": {}
      },
      "_source": ["post_date","title"]
    }

    h. 搜索标题中包含 article 的文章, 如果标题中包含first或second就优先搜索出来, 同时, 如果一个文章标题包含first article, 另一个文章标题包含second article, 包含first article的文章要优先搜索出来.

    //通过加权重来处理, 默认权重为1
    GET /website/article/_search
    {
      "query": {
        "bool": {
          "must": [
            {"match": {
              "title": "article"
            }}
          ],
          "should": [
            {"match": {
              "title": {
                "query": "first",
                "boost" : 3
              }
            }},
            {"match": {
              "title": {
                "query": "second",
                "boost": 2
              }
            }}
          ]
        }
      }
    }

    (2) multi_match

    multi_match 用于查询词匹配多个属性. 这里涉及到几种匹配策略:

      best-fields

        doc的某个属性匹配尽可能多的关键词, 那么这个doc会优先返回.
      most-fields
        某个关键词匹配doc尽可能多的属性, 那么这个doc会优先返回.
      cross_fields
        跨越多个field搜索一个关键词.

    best-fields和most-fields的区别:
    比如, doc1的field1匹配的三个关键词, doc2的field1, field2都匹配上了同一个关键词. 如果是best-fields策略, 则doc1的相关度分数要更高, 如果是most-fields策略, 则doc2的相关度分数要更高.

    实例:

    a. 使用best_fields策略, 从title和content中搜索"my third article".

    GET /website/article/_search
    {
      "query": {
        "multi_match": {
          "query": "my third article",
          "type": "best_fields", 
          "fields": ["title","content"]
        }
      }
    }

    b. 从title和content中搜索"my third article", 且这三个单词要连在一起.

    GET /website/article/_search
    {
      "query": {
        "multi_match": {
          "query": "my third article",
          "type": "cross_fields",
          "operator": "and",
          "fields": ["title","content"]
        }
      }
    }

    3) dis_max

    在我们进行多个条件的全文搜索时, 最后的计算出的相关度分数是根据多个条件的匹配分数综合而来的, 比如score = (score1 + score2) / 2, 如果我们想让最终的相关度分数等于多个条件匹配分数中的最大值, 即score = max(score1, score2), 则可以使用dis_max.

    实例: 搜索title或content中包含first或article的文章

    GET /website/article/_search
    {
      "query": {
        "bool": {
          "should": [
            {
              "match": {
                "title": "first article"
              }
            },
            {
              "match": {
                "content": "first article"
              }
            }
          ]
        }
      }
    }

    使用dis_max:

    GET /website/article/_search
    {
      "query": {
        "dis_max": {
          "queries": [
              {
              "match": {
                "title": "first article"
                  }
              },
              {
                "match": {
                  "content": "first article"
                }
              }
            ]
        }
      }
    }

    (4) tie_breaker

    tie_breaker是与dis_max配套使用的. dis_max只取分数最大的那个条件的分数, 完全不考虑其他条件的分数, 但如果在某些场景下也需要考虑其他条件的分数呢? 我们可以指定一个系数值tie_breaker, 将其他条件的分数乘以tie_breaker, 然后和最大分数综合起来计算最终得分.

    tie_breaker的取值为 0 ~ 1之间.

    a. 将上面的实例优化下:

    GET /website/article/_search
    {
      "query": {
        "dis_max": {
          "tie_breaker": 0.7, 
          "queries": [
              {
              "match": {
                "title": "first article"
                  }
              },
              {
                "match": {
                  "content": "first article"
                }
              }
            ]
        }
      }
    }

    b. 继续优化. 如果搜索词包含多个关键字, 我们要求至少匹配多个关键词, 且多个条件的权重不同.

    GET /website/article/_search
    {
      "query": {
        "dis_max": {
          "tie_breaker": 0.7, 
          "queries": [
              {
              "match": {
                "title": {
                  "query": "first article hello world",
                  "minimum_should_match": "50%",
                  "boost": 2
                  }
                }
              },
              {
                "match": {
                  "content": {
                    "query": "first article is my hero",
                    "minimum_should_match": "20%",
                    "boost": 1
                  }
                }
              }
            ]
        }
      }
    }

    phrase search

    (1) 短语搜索, 与全文搜索有什么区别呢?

    全文搜索会将"查询词"拆解开来, 去倒排索引中一一匹配, 只要能匹配上任意一个拆解后的关键词, 就可以作为结果返回. 而短语搜索在全文搜索的基础上, 要求关键词必须相邻. (注意短语搜索的"查询词"也是会被分词的)

    GET /website/article/_search
    {
      "query": {
        "match": {
          "title": "first article"
        }
      }
    }
    //三条记录都会搜索出来
    
    GET /website/article/_search
    {
      "query": {
        "match_phrase": {
          "title": "first article"
        }
      }
    }
    //只有一条记录

    (2) 原理:

    短语搜索的原理实际上是相邻匹配(proximity match). Lucene建立的倒排索引结构为: 关键词 -> 文档号, 在文档中的位置, 在文档出现的频率等, 当一个"查询词"包含多个关键词时, Lucene先通过关键词找到对应的文档号, 判断多个关键词所在的文档号是否相同, 然后再判断在文档中的位置是否相邻.

    (3) 实例

    短语搜索默认是搜索相邻的关键词, 但也可以搜索间隔几个位置的关键词. 间隔越小相关度分数越高.

    a. 从content中搜索"first website", first和website必须在同一个doc中, 且间隔不能超过10.

    ET /website/article/_search
    {
      "query": {
        "match_phrase": {
          "content": {
            "query": "first website",
            "slop": 10
          }
        }
      }
    }

    b. 全文搜索和短语搜索配合使用. 从content中搜索"first website", 在优先满足召回率的前提下, 尽可能提高精准度.

    召回率: 从n个doc中搜索, 有多少个doc返回.
    精准度: 让两个关键词间隔越小的doc相关度分数越高.
    GET /website/article/_search
    {
      "query": {
        "bool": {
          "must": [
            {
              "match": {
                "content": "first website"
              }
            }
          ],
          "should": [
            {
              "match_phrase": {
                "content": {
                  "query": "first website",
                  "slop": 10
                }
              }
            }
          ]
        }
      }
    }

    (4) 优化短语搜索: rescore

    短语搜索的性能要比全文搜索的性能低10倍以上, 所以一般我们要用短语搜索时都会配合全文搜索使用. 先通过全文搜索出匹配的doc, 然后对相关度分数最高的前n条doc进行rescore短语搜索. (这里只能用于分页搜索)

    GET /website/article/_search
    {
      "query": {
        "match": {
          "content": "first website"
        }
      },
      "rescore": {
        "window_size": 20,
        "query": {
          "rescore_query": {
            "match_phrase": {
              "content": {
                "query": "first website",
                "slop": 10
              }
            }
          }
        }
      }
    }

    (5) match_phrase_prefix

    匹配短语的前缀, 用于做搜索推荐. 比如我们在百度输入一个关键词, 立马就会推荐一系列查询词, 这个就是搜索推荐.

    这个功能不推荐使用, 因为性能太差, 我们一般通过ngram分词机制来实现搜索推荐.
    GET /website/article/_search
    {
      "query": {
        "match_phrase_prefix": {
          "title": "my sec"
        }
      }
    }

    fuzzy search

    模糊搜索, 自动将拼写错误的搜索文本进行纠正, 然后去匹配索引中的数据.

    (1) 语法一

    GET /website/article/_search
    {
      "query": {
        "fuzzy": {
          "title.keyword": {
            "value": "my wecond article",
            "fuzziness": 2
          }
        }
      }
    }

    fuzziness代表最多纠正多少个字母, 默认为2. 搜索文本不会被分词.

    (2) 语法二

    GET /website/article/_search
    {
      "query": {
        "match": {
          "title": {
            "query": "my wecond article",
            "fuzziness": "auto",
            "operator": "and"
          }
        }
      }
    }

    fuzziness可以给定个数, 也可以设置为auto.

    term search

    term查询, 是一种结构化查询, "查询词"不会被分词, 结果要么存在要么不存在, 不关心结果的score相关度. 如果查询text属性, 需要改为查询filed.keyword.

    (1) 实例

    和短语搜索对比一下可以更好的理解:

    1. phrase搜索
    GET /website/article/_search
    {
      "query": {
        "match_phrase": {
          "title": "my first article"
        }
      }
    }
    //存在一条结果
    
    2. term搜索text
    GET /website/article/_search
    {
      "query": {
        "term": {
          "title": {
            "value": "my first article"
          }
        }
      }
    }
    //不存在结果, 原因是词语查询的value值不会被分词, 也就是直接查询"my first article".
    
    3. term搜索keyword
    GET /website/article/_search
    {
      "query": {
        "term": {
          "title.keyword": {
            "value": "my first article"
          }
        }
      }
    }
    //存在一条结果, filed.keyword属性不分词.

    2) 常用语法

    为了提高效率, term搜索一般与filter和constant_score联用. constant_score 以固定的评分来执行查询(默认为1), 而filter不计算score相关度, 因此执行效率非常高.

    GET /website/article/_search
    {
      "query": {
        "constant_score": {
          "filter": {
            "term": {
              "post_date": "2017-01-03"
            }
          }
        }
      }
    }

    query filter

    (1) 语法

    query filter 用于过滤数据, 不参与score相关度计算, 效率很高. 适用于范围查询以及不计算相关度score的精确查询(filter + term)

    (2) 实例

    a. 从website索引中查询, 作者id必须大于等于11402,同时发表时间必须是2017-01-02.

    GET /website/article/_search
    {
      "query": {
        "constant_score": {
          "filter": {
            "bool": {
              "must": [
                  {
                    "term": {
                      "post_date": "2017-01-02"
                    }
                  },
                  {
                    "range": {
                      "author_id": {
                         "gte": 11402
                      }
                    }
                  }
                ]
            }
          }
        }
      }
    }

    b. 搜索发布日期为2017-01-01, 或者文章标题为"my first article"的帖子, 同时要求文章的发布日期绝对不为2017-01-02.

    GET /website/article/_search
    {
      "query": {
        "constant_score": {
          "filter": {
            "bool": {
              "should": [
                {
                  "term":{
                    "post_date": "2017-01-01"
                  }
                },
                {
                  "term":{
                    "title.keyword": "my first article"
                  }
                }
                ],
                "must_not": {
                  "term": {
                    "post_date": "2017-01-02"
                  }
                }
            }
          }
        }
      }
    }

    c. 搜索文章标题为"my first article", 或者是文章标题为"my second article", 而且发布日期为"2017-01-01"的文章.

    GET /website/article/_search
    {
      "query": {
        "constant_score": {
          "filter": {
            "bool": {
              "should": [
                {
                  "term":{
                    "title.keyword": "my first article"
                  }
                },
                {
                  "bool": {
                    "must": [
                        {
                           "term":{
                             "title.keyword": "my second article"
                           }
                        },
                        {
                          "term":{
                             "post_date": "2017-01-01"
                           }
                        }
                      ]
                  }
                }
                ]
            }
          }
        }
      }
    }

    d. 搜索文章标题为"my first article"或"my second article"的文章

    GET /website/article/_search
    {
      "query": {
        "constant_score": {
          "filter": {
            "terms": {
              "title.keyword": [
                "my first article",
                "my second article"
              ]
            }
          }
        }
      }
    }

    e. 搜索tags中包含java的帖子.

    GET /website/article/_search
    {
      "query": {
        "constant_score": {
          "filter": {
            "terms": {
              "tags.keyword": [
                "java"
              ]
            }
          }
        }
      }
    }

    注意, 这里必须用terms, 因为term不支持数组.

    d. 搜索tags中只包含java的帖子.

    如果想搜索tags中只包含java的帖子, 就需要新增一个字段tags_count, 表示tags中有几个tag, 否则就无法搜索.

    GET /website/article/_search
    {
      "query": {
        "constant_score": {
          "filter": {
            "bool": {
              "must": [
                  {
                    "terms": {
                      "tags.keyword": [
                          "java"
                        ]
                    }
                  },
                  {
                    "term": {
                      "tags_count": "1"
                    }
                  }
                ]
            }
          }
        }
      }
    }
    对上面这几实例做个总结, should表示或, must表示且.
    
    scroll search
    
    scroll滚动搜索,可以先搜索一批数据,然后下次再搜索一批数据,以此类推,直到搜索出全部的数据来. 它的原理是每次查询时都生成一个游标scroll_id, 后续的查询根据这个游标去获取数据, 直到返回的hits字段为空. scroll_id相当于建立了一个历史快照, 在此之后的写操作不会影响到这个快照的结果, 也就意味着其不能用于实时查询.
    滚动查询用来解决深度分页的问题, 就类似于sql语句: select * from comment where id > 1000 order by id asc limit 1000.

    (1) 语法

    查询时指定一个参数scroll, 代表scroll_id的有效期, 过期后scroll_id会被ES自动清除.

    如果不需要特定的排序, 按照文档创建时间排序更高效.

    scroll_id只能使用一次, 使用过后会被自动删除.

    最后一次查询, hits为空时也会返回一个scroll_id, 我们需要手动删除来释放资源.

    (2) 实例

    a. 首次查询

    GET /website/article/_search?scroll=1s
    {
      "query": {
        "match_all": {}
      },
      "sort": [
        {
          "_doc": {
            "order": "asc"
          }
        }
      ],
      "size": 1
    }

    b. 后续查询

    scroll_id只能使用一次.

    GET /_search/scroll?scroll=1s&scroll_id=DnF1ZXJ5VGhlbkZldGNoBQAAAAAAAATqFmRBQ3FwUFVrUUw2VHgyU2I5UWRMRlEAAAAAAAAE6xZkQUNxcFBVa1FMNlR4MlNiOVFkTEZRAAAAAAAABOwWZEFDcXBQVWtRTDZUeDJTYjlRZExGUQAAAAAAAATtFmRBQ3FwUFVrUUw2VHgyU2I5UWRMRlEAAAAAAAAE7hZkQUNxcFBVa1FMNlR4MlNiOVFkTEZR
    {
    }
    
    或者
    
    GET /_search/scroll
    {
      "scroll":"1s", "scroll_id":"DnF1ZXJ5VGhlbkZldGNoBQAAAAAAAATqFmRBQ3FwUFVrUUw2VHgyU2I5UWRMRlEAAAAAAAAE6xZkQUNxcFBVa1FMNlR4MlNiOVFkTEZRAAAAAAAABOwWZEFDcXBQVWtRTDZUeDJTYjlRZExGUQAAAAAAAATtFmRBQ3FwUFVrUUw2VHgyU2I5UWRMRlEAAAAAAAAE7hZkQUNxcFBVa1FMNlR4MlNiOVFkTEZR"
    }

    c. 删除指定scroll_id

    DELETE /_search/scroll
    {
      "scroll_id": "scroll_id=DnF1ZXJ5VGhlbkZldGNoBQAAAAAAAAiXFmRBQ3FwUFVrUUw2VHgyU2I5UWRMRlEAAAAAAAAImBZkQUNxcFBVa1FMNlR4MlNiOVFkTEZRAAAAAAAACJkWZEFDcXBQVWtRTDZUeDJTYjlRZExGUQAAAAAAAAiaFmRBQ3FwUFVrUUw2VHgyU2I5UWRMRlEAAAAAAAAImxZkQUNxcFBVa1FMNlR4MlNiOVFkTEZR=="
    }

    d. 删除所有scroll_id

    DELETE /_search/scroll/_all

    聚合

    聚合包括分组和统计, 其中分组操作包括term, histogram, date_histogram, filter. 统计操作包括count, avg, max, min, sum, cardinality, percentiles, percentile_ranks等.

    注意: 注意: 聚合的属性不能被分词.

    语法

    GET /index/type/_search
    {
      size: 0,
      "aggs": {
        "NAME": {
          "AGG_TYPE": {
             "field": "field_name"
           }
        }
      }
    }

    NAME为聚合操作的名称, 可以取一个有参考意义的名称. AGG_TYPE为分组或统计操作, 当进行分组操作时, 会自动生成一个doc_count, 统计了组内数据的数量. 默认按照doc_count降序排列.

    size=0的原因是不需要搜索结果, 如果需要搜索结果, 则去除size=0.

    实例

    (1) 实例数据

    新增电视机销售记录, 用于接下来的实例分析.

    POST /televisions/sales/_bulk
    { "index": {}}
    { "price" : 1000, "color" : "红色", "brand" : "长虹", "sold_date" : "2016-10-28" }
    { "index": {}}
    { "price" : 2000, "color" : "红色", "brand" : "长虹", "sold_date" : "2016-11-05" }
    { "index": {}}
    { "price" : 3000, "color" : "绿色", "brand" : "小米", "sold_date" : "2016-05-18" }
    { "index": {}}
    { "price" : 1500, "color" : "蓝色", "brand" : "TCL", "sold_date" : "2016-07-02" }
    { "index": {}}
    { "price" : 1200, "color" : "绿色", "brand" : "TCL", "sold_date" : "2016-08-19" }
    { "index": {}}
    { "price" : 2000, "color" : "红色", "brand" : "长虹", "sold_date" : "2016-11-05" }
    { "index": {}}
    { "price" : 8000, "color" : "红色", "brand" : "三星", "sold_date" : "2017-01-01" }
    { "index": {}}
    { "price" : 2500, "color" : "蓝色", "brand" : "小米", "sold_date" : "2017-02-12" }

    mapping结构如下:

    GET /televisions/_mapping/sales
    {
      "televisions": {
        "mappings": {
          "sales": {
            "properties": {
              "brand": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "color": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "price": {
                "type": "long"
              },
              "sold_date": {
                "type": "date"
              }
            }
          }
        }
      }
    }

    (2) 基础聚合实例

    a. 统计哪种颜色的电视销量最高

    GET /televisions/sales/_search
    {
      "size": 0,
      "aggs": {
        "max_sales_color": {
          "terms": {
            "field": "color.keyword"
          }
        }
      }
    }

    分组后会计算出每组数据个数(doc_count), 默认按照doc_count降序显示.

    b. 统计每种颜色电视的平均价格

    GET /televisions/sales/_search
    {
      "size": 0,
      "aggs": {
        "group_by_color": {
          "terms": {
            "field": "color.keyword"
          },
          "aggs": {
            "avg_price": {
              "avg": {
                "field": "price"
              }
            }
          }
        }
      }
    }

    c. 统计每种颜色电视的平均价格, 以及统计每种颜色下每个品牌的平均价格.

    这里就涉及到嵌套分组了, 也叫做多层下钻分析.

    GET /televisions/sales/_search
    {
      "size": 0,
      "aggs": {
        "group_by_color": {
          "terms": {
            "field": "color.keyword"
          },
          "aggs": {
            "color_avg_price": {
              "avg": {
                "field": "price"
              }
            },
            "group_by_brand": {
              "terms": {
                "field": "brand.keyword"
              },
              "aggs": {
                "brand_avg_price": {
                  "avg": {
                    "field": "price"
                  }
                }
              }
            }
          }
        }
      }
    }

    d. 统计每种颜色电视机的最大最小价格

    GET /televisions/sales/_search
    {
      "size": 0,
      "aggs": {
        "group_by_color": {
          "terms": {
            "field": "color.keyword"
          },
          "aggs": {
            "max_price": {
              "max": {
                "field": "price"
              }
            },
            "min_price": {
              "min": {
                "field": "price"
              }
            }
          }
        }
      }
    }

    e. 统计每种颜色电视机的总销售额

    GET /televisions/sales/_search
    {
      "size": 0,
      "aggs": {
        "group_by_color": {
          "terms": {
            "field": "color.keyword"
          },
          "aggs": {
            "sum_price": {
              "sum": {
                "field": "price"
              }
            }
          }
        }
      }
    }

    (2) 高级聚合实例

    a. 按价格区间统计电视销量和销售额

    GET /televisions/sales/_search
    {
      "size": 0,
      "aggs": {
        "group_by_price_range": {
          "histogram": {
            "field": "price",
            "interval": 2000
          },
          "aggs": {
            "sum_price": {
              "sum": {
                "field": "price"
              }
            }
          }
        }
      }
    }

    b. 统计 2016-01-01 ~ 2017-12-31 范围内每个月的电视机销量.

    GET /televisions/sales/_search
    {
      "size": 0,
      "aggs": {
        "group_by_sold_date": {
          "date_histogram": {
            "field": "sold_date",
            "interval": "month",
            "format": "yyyy-MM-dd",
            "min_doc_count": 0,
            "extended_bounds": {
              "min": "2016-01-01",
              "max": "2017-12-31"
            }
          }
        }
      }
    }

    c. 统计 2016-01-01 ~ 2017-12-31 范围内每个季度的销售额以及该季度下每个品牌的销售额

    GET /televisions/sales/_search
    {
      "size": 0,
      "aggs": {
        "group_by_sold_date": {
          "date_histogram": {
            "field": "sold_date",
            "interval": "quarter",
            "format": "yyyy-MM-dd",
            "min_doc_count": 0,
            "extended_bounds": {
              "min": "2016-01-01",
              "max": "2017-12-31"
            }
          },
          "aggs": {
            "sum_price": {
              "sum": {
                "field": "price"
              }
            },
            "group_by_brand": {
              "terms": {
                "field": "brand.keyword"
              },
              "aggs": {
                "brand_sum_price": {
                  "sum": {
                    "field": "price"
                  }
                }
              }
            }
          }
        }
      }
    }

    d. 统计每种颜色电视的销售额, 按照销售额升序排序

    GET /televisions/sales/_search
    {
      "size": 0, 
      "aggs": {
        "group_by_color": {
          "terms": {
            "field": "color.keyword",
            "order": {
              "sum_price": "asc"
            }
          },
          "aggs": {
            "sum_price": {
              "sum": {
                "field": "price"
              }
            }
          }
        }
      }
    }

    e. 统计每种颜色下的每个品牌电视机的总销售额, 并按这个销售额升序排序.

    GET /televisions/sales/_search
    {
      "size": 0, 
      "aggs": {
        "group_by_color": {
          "terms": {
            "field": "color.keyword"
          },
          "aggs": {
            "group_by_brand": {
              "terms": {
                "field": "brand.keyword",
                "order": {
                  "color_brand_sum_price": "asc"
                }
              },
              "aggs": {
                "color_brand_sum_price": {
                  "sum": {
                    "field": "price"
                  }
                }
              }
            }
          }
        }
      }
    }

    f. 统计每个月的电视销量, 并按品牌去重.

    GET /televisions/sales/_search
    {
      "size": 0, 
      "aggs": {
        "group_by_sold_date": {
          "date_histogram": {
            "field": "sold_date",
            "interval": "month",
            "format": "yyyy-MM-dd"
          },
          "aggs": {
            "distinct_brand": {
              "cardinality": {
                "field": "brand.keyword",
                "precision_threshold": 100 
              }
            }
          }
        }
      }
    }

    cardinality 去重采用的是近似估计的算法, 错误率在5%左右, 其中precision_threshold指定的值为100%准确去重的数量, 值设置的越大, 内存开销也就越大.

    g. 统计50%, 90% 和 99%的电视的最大价格(一般用于统计api请求的最长延迟时间)

    GET /televisions/sales/_search
    {
      "size": 0, 
      "aggs": {
        "price_percentiles": {
          "percentiles": {
            "field": "price",
            "percents": [
              50,
              90,
              99
            ]
          }
        }
      }
    }

    h. 统计每个品牌的电视机的价格, 在1000以内, 2000以内, 3000以内, 4000以内的所占比例.

    GET /televisions/sales/_search
    {
      "size": 0,
      "aggs": {
        "group_by_brand": {
          "terms": {
            "field": "brand.keyword"
          },
          "aggs": {
            "price_percentile_ranks": {
              "percentile_ranks": {
                "field": "price",
                "values": [
                  1000,
                  2000,
                  3000,
                  4000
                ]
              }
            }
          }
        }
      }
    }

    3) 搜索+聚合

    a. 统计指定品牌下(小米)每个颜色的销量

    GET /televisions/sales/_search
    {
      "size": 0, 
      "query": {
        "term": {
          "brand.keyword": {
            "value": "小米"
          }
        }
      }, 
      "aggs": {
        "group_by_color": {
          "terms": {
            "field": "color.keyword"
          }
        }
      }
    }

     或者

    GET /televisions/sales/_search
    {
      "size": 0, 
      "query": {
        "constant_score": {
          "filter": {
            "term": {
              "brand.keyword": "小米"
            }
          }
        }
      }, 
      "aggs": {
        "group_by_color": {
          "terms": {
            "field": "color.keyword"
          }
        }
      }
    }    

    其实对于聚合来说, 因为不需要搜索结果, 可以直接用filter, 效率更高.

    b. 统计单个品牌(长虹)与所有品牌销售额对比

    GET /televisions/sales/_search
    {
      "size": 0, 
      "query": {
        "constant_score": {
          "filter": {
            "term": {
              "brand.keyword": "长虹"
            }
          }
        }
      }, 
      "aggs": {
        "single_brand_sum_price": {
          "sum": {
            "field": "price"
          }
        },
        "all_brand": {
          "global": {},
          "aggs": {
            "all_brand_sum_price": {
              "sum": {
                "field": "price"
              }
            }
          }
        }
      }
    }

    global 表示将所有数据纳入聚合的scope,忽视前面的query过滤.

    c. 统计指定品牌(长虹)最近一个月和最近半年的平均价格

    GET /televisions/sales/_search
    {
      "size": 0, 
      "query": {
        "term": {
          "brand.keyword": {
            "value": "长虹"
          }
        }
      }, 
      "aggs": {
        "recent_one_month": {
          "filter": {
            "range": {
              "sold_date": {
                "gte": "now-1M"
              }
            }
          },
          "aggs": {
            "recent_one_month_avg_price": {
              "avg": {
                "field": "price"
              }
            }
          }
        },
        "recent_six_month": {
          "filter": {
            "range": {
              "sold_date": {
                "gte": "now-6M"
              }
            }
          },
          "aggs": {
            "recent_six_month_avg_price": {
              "avg": {
                "field": "price"
              }
            }
          }
        }
      }
    }

    ————————————————
    版权声明:本文为CSDN博主「椰子Tyshawn」的原创文章,遵循CC 4.0 BY-SA版权协议,转载请附上原文出处链接及本声明。
    原文链接:https://blog.csdn.net/litianxiang_kaola/article/details/102936324

  • 相关阅读:
    laravel本地化
    laravel加载javascript库
    laravel安装第三方包3rd generators
    openfire的smack和asmack
    openfire+spark+smack实现即时通讯
    openfire开发
    即时通信Spark安装和配置
    ubuntu eclipse CDT 问题
    网络编程TCP总结及实践-C语言
    [Openstack]使用devstack自己主动化安装
  • 原文地址:https://www.cnblogs.com/hanzeng1993/p/15075253.html
Copyright © 2020-2023  润新知