• GitLab CI/CD实践记录


    1. GitLab Runner

    参考:《Configuring GitLab Runners

    In GitLab CI/CD, Runners run the code defined in .gitlab-ci.yml. A GitLab Runner is a lightweight, highly-scalable agent that picks up a CI job through the coordinator API of GitLab CI/CD, runs the job, and sends the result back to the GitLab instance.

    GitLab Runner的有多种形式:Repositories、Binaries、Docker service。参考:《Create a group Runner

    1.1 安装Git Lab Runner

    1.1.1 Ubuntu中安装gitlab-runner

    参考:《Install GitLab Runner manually on GNU/Linux

    curl -L https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh | sudo bash sudo apt-get install gitlab-runner

    安装好Runner之后,还需要将Runner注册到Gitlab。

    Ubuntu下注册Runner为:GNU/Linux

    sudo gitlab-runner register

    一行命令注册:

    docker run --rm -v /srv/gitlab-runner/config:/etc/gitlab-runner gitlab/gitlab-runner register 
      --non-interactive 
      --executor "docker" 
      --docker-image alpine:latest 
      --url "https://gitlab.com/" 
      --registration-token "PROJECT_REGISTRATION_TOKEN" 
      --description "docker-runner" 
      --tag-list "docker,aws" 
      --run-untagged="true" 
      --locked="false" 
      --access-level="not_protected"

    1.1.3 docker中安装gitlab-runner

    参考:《Run GitLab Runner in a container

    1.2 Git Lab Runner使用

    docker中对Runner进行注册有两种方法:交互式注册一行命令注册

    sudo gitlab-runner register 
      --non-interactive 
      --url "https://gitlab.com/" 
      --registration-token "PROJECT_REGISTRATION_TOKEN" 
      --executor "docker" 
      --docker-image alpine:latest 
      --description "docker-runner" 
      --tag-list "docker,aws" 
      --run-untagged="true" 
      --locked="false" 
      --access-level="not_protected"

    这句话的意思是:注册一个executor为docker,docker image为alpine:latest的gitlab-runner,到https://gitlab.com/,其token为PROJECT_REGISTRATION_TOKEN。

    并且此gitlab-runner的tag为docker或aws,并设置interactive、locked、access等属性。

    上述参数url和registration-token从Settings->CI/CD->Runners中获取:

    另外--docker-image为指定的Runner的Docker image文件;--tag-list为Runner的tag列表。

    查看所有注册的GitLab Runner:

    sudo gitlab-runner list

    去注册选定的GitLab Runner:

    sudo gitlab-runner unregister --token g_L6R5T1RZDxbm1ttbx8 --url http://192.168.70.8/

     运行所有注册的Runner:

    sudo gitlab-runner run

    1.3 Git Lab Runner的config.toml配置

    config.toml默认位置在/etc/gitlab-runner/config.toml,详细配置参考《Advanced configuration》。

    Pipelines简要介绍

    Pipelines组成

    参考:《CI/CD pipelines

    Pipelines are the top-level component of continuous integration, delivery, and deployment.

    Pipelines comprise:

    Jobs, which define what to do. For example, jobs that compile or test code.

    Stages, which define when to run the jobs. For example, stages that run tests after stages that compile the code.

    Pipelines即流水线是持续集成、持续发布、持续部署的最顶层组件,由Stages和Jobs组成。

    Jobs是真正执行单元,定义了做什么。Stags是对Jobs串接,定义了什么时候执行Jobs。

    Pipelines调度

    参考:《Pipeline schedules

    Pipelines的调度按触发模式可以分为:自动触发和手动触发,自动触发又分为定时触发和Merge请求触发Pipelines for Merge Requests

    默认通过commit触发Pipeline,还可以被Merge Requests触发。

    比如下面build job 1可以被Merge Requests或者master上的分支触发。

    build job 1:
      stage: build
      tags:
      - mytag
      script:
      only:
        - merge_requests
        - master

    还可以通过except关键字,排除触发条件。

    另外还可以打开Settings->CI/CD->Pipeline triggers,在其他pipeline中触发。

    输入Decsription,然后Add trigger,即可增加Pipeline trigger。

    可以命令行触发,也可以在.gitlab-ci.yml中其他job中触发。

    trigger_build:
      stage: deploy
      script:
        - "curl -X POST -F token=<Pipeline trigger token> -F ref=<branch or tag> http://<gitlab server>/api/v4/projects/<project id>/trigger/pipeline"

    在Pipeline之间传递artifacts:

    build_submodule:
      image: debian
      stage: test
      script:
    - curl --location --output artifacts.zip "https://gitlab.example.com/api/v4/projects/1/jobs/artifacts/master/download?job=test&job_token=$CI_JOB_TOKEN"
        - unzip artifacts.zip
      only:
        - tags

    如果要在curl中传递参数可以通过:

    curl --request POST 
      --form token=TOKEN 
      --form ref=master 
      --form "variables[UPLOAD_TO_S3]=true" 
      https://gitlab.example.com/api/v4/projects/9/trigger/pipeline

    gitlab-ci.yml配置

    stages配置 stage配置 .pre和.post

    参考:《stages》《stage》《.pre and .post

    stages is used to define stages that contain jobs and is defined globally for the pipeline.

    Jobs of the same stage are run in parallel.Jobs of the next stage are run after the jobs from the previous stage complete successfully.

    stage is defined per-job and relies on stages which is defined globally. It allows to group jobs into different stages, and jobs of the same stage are executed in parallel (subject to certain conditions).

    The following stages are available to every pipeline:

    .pre, which is guaranteed to always be the first stage in a pipeline.

    .post, which is guaranteed to always be the last stage in a pipeline.

    User-defined stages are executed after .pre and before .post.

    The order of .pre and .post can’t be changed, even if defined out of order in .gitlab-ci.yml.

    .pre和.post在所有stage的最前和最后执行,而不受pipelines中定义的顺序。

    一个示例如下:

    stages:
      - build
      - test
      - deploy
    
    job 0:
      stage: .pre
      script: make something useful before build stage
    
    job 1:
      stage: build
      script: make build dependencies
    
    job 2:
      stage: build
      script: make build artifacts
    
    job 3:
      stage: test
      script: make test
    
    job 4:
      stage: deploy
      script: make deploy
    
    job 5:
      stage: .post
      script: make something useful at the end of pipeline

    这个Pilelines中定义了5个stages,分别是.pre、build、test、deploy、.post;定义了6个jobs:分别是job 0、job 1、job 2、job 3、job 4、job 5。

    所以此Pipellines的执行顺序是.pre-[job 0]->build-[job 1/job 2]->test-[job 3]->deploy-[job 4]->.post[job 5],其中job1和job2并行执行

    include配置

    参考:《include

    include requires the external YAML file to have the extensions .yml or .yaml, otherwise the external file won’t be included.

    include包含如下类型:

    MethodDescription
    local Include a file from the local project repository.
    file Include a file from a different project repository.
    remote Include a file from a remote URL. Must be publicly accessible.
    template Include templates which are provided by GitLab.

    local示例如下:

    include:
      - local: '/templates/.gitlab-ci-template.yml'
    
    include: '.gitlab-ci-production.yml'

    file示例如下:

    include:
      - project: 'my-group/my-project'
        ref: master
        file: '/templates/.gitlab-ci-template.yml'
    
      - project: 'my-group/my-project'
        ref: v1.0.0
        file: '/templates/.gitlab-ci-template.yml'
    
      - project: 'my-group/my-project'
        ref: 787123b47f14b552955ca2786bc9542ae66fee5b # Git SHA
        file: '/templates/.gitlab-ci-template.yml'

    remote示例如下:

    include:
      - remote: 'https://gitlab.com/awesome-project/raw/master/.gitlab-ci-template.yml'

    template示例如下:

    include:
      - template: Android-Fastlane.gitlab-ci.yml
      - template: Auto-DevOps.gitlab-ci.yml

    script before_script和after_script

    参考:《script》《before_scrip and after_script

    script is the only required keyword that a job needs. It’s a shell script which is executed by the Runner. 

    before_script is used to define a command that should be run before each job, including deploy jobs, but after the restoration of any artifacts. This must be an array.

    Scripts specified in before_script are concatenated with any scripts specified in the main script, and executed together in a single shell.

    after_script is used to define the command that will be run after each job, including failed ones. This must be an array.

    Scripts specified in after_script are executed in a new shell, separate from any before_script or script scripts.

    before_script是在script运行之前,恢复artifacts之后运行,但是和script在一个shell中运行。

    after_script单独在一个shell中运行,并且在scirpt之后。

    image services

    参考:《image》《service

    image用于指定Docker image,service用于指定连接到image指定Docker image的service Docker image。

    The image keyword is the name of the Docker image the Docker executor will run to perform the CI tasks.
    The services keyword defines just another Docker image that is run during your job and is linked to the Docker image that the image keyword defines. This allows you to access the service image during build time.

    image可配置选项:

    SettingRequiredGitLab versionDescription
    name yes, when used with any other option 9.4 Full name of the image that should be used. It should contain the Registry part if needed.
    entrypoint no 9.4 Command or script that should be executed as the container’s entrypoint. It will be translated to Docker’s --entrypoint option while creating the container. The syntax is similar to Dockerfile’s ENTRYPOINT directive, where each shell token is a separate string in the array.

    service可配置选项:

    SettingRequiredGitLab versionDescription
    name yes, when used with any other option 9.4 Full name of the image that should be used. It should contain the Registry part if needed.
    entrypoint no 9.4 Command or script that should be executed as the container’s entrypoint. It will be translated to Docker’s --entrypoint option while creating the container. The syntax is similar to Dockerfile’s ENTRYPOINT directive, where each shell token is a separate string in the array.
    command no 9.4 Command or script that should be used as the container’s command. It will be translated to arguments passed to Docker after the image’s name. The syntax is similar to Dockerfile’s CMD directive, where each shell token is a separate string in the array.
    alias no 9.4 Additional alias that can be used to access the service from the job’s container. Read Accessing the services for more information.

    示例:

    default:
      image:
        name: ruby:2.6
        entrypoint: ["/bin/bash"]
    
      services:
        - name: my-postgres:11.7
          alias: db-postgres
          entrypoint: ["/usr/local/bin/db-postgres"]
          command: ["start"]
    
      before_script:
        - bundle install
    
    test:
      script:
        - bundle exec rake spec

    only/except

    参考:《only/except (basic)

    only and except are two parameters that set a job policy to limit when jobs are created:

    1. only defines the names of branches and tags for which the job will run.
    2. except defines the names of branches and tags for which the job will not run.
    ValueDescription
    branches When the Git reference for a pipeline is a branch.
    tags When the Git reference for a pipeline is a tag.
    api For pipelines triggered by the pipelines API.
    external When using CI services other than GitLab.
    pipelines For multi-project pipelines created by using the API with CI_JOB_TOKEN.
    pushes For pipelines triggered by a git push event, including for branches and tags.
    schedules For scheduled pipelines.
    triggers For pipelines created by using a trigger token.
    web For pipelines created by using Run pipeline button in the GitLab UI, from the project’s CI/CD > Pipelines section.
    merge_requests For pipelines created when a merge request is created or updated. Enables merge request pipelinesmerged results pipelines, and merge trains.
    external_pull_requests When an external pull request on GitHub is created or updated (See Pipelines for external pull requests).
    chat For pipelines created by using a GitLab ChatOps command.

    示例1:

    job:
      # use regexp
      only:
        - /^issue-.*$/---仅执行issues-开头的分支、tags等。
      # use special keyword
      except:
        - branches----排除所有的branches

    示例2:

    job:
      only:
        - branches@gitlab-org/gitlab-----------在gitlab-org/gitlab的所有分支上运行。
      except:
        - master@gitlab-org/gitlab------------但是要排除master分支。
        - /^release/.*$/@gitlab-org/gitlab----以及release/xxx开头。

    参考:《only/except(advanced)

    If you use multiple keys under only or except, the keys will be evaluated as a single conjoined expression. That is:

    • only: means “include this job if all of the conditions match”.
    • except: means “exclude this job if any of the conditions match”.

    With only, individual keys are logically joined by an AND

    示例1:

    test:
      script: npm run test
      only:----下面refs、variables、kubernetes是与的关系,也即必须都满足才会执行job;refs内部是或的而关系。
        refs:
          - master
          - schedules
        variables:
          - $CI_COMMIT_MESSAGE =~ /run-end-to-end-tests/
        kubernetes: active

    示例2:

    test:
      script: npm run test
      except:--------下面refs、changes是或的关系,也即只要满足一个条件都不会执行job。
        refs:
          - master
        changes:
          - "README.md"

    rules

    参考:《rules

    The rules keyword can be used to include or exclude jobs in pipelines.

    Rules are evaluated in order until the first match.

    When matched, the job is either included or excluded from the pipeline, depending on the configuration.

    ClauseDescription
    if Add or exclude jobs from a pipeline by evaluating an if statement.
    changes Add or exclude jobs from a pipeline based on what files are changed.
    exists Add or exclude jobs from a pipeline based on the presence of specific files.

    Rules are evaluated in order until a match is found.

    If a match is found, the attributes are checked to see if the job should be added to the pipeline.

    If no attributes are defined, the defaults are:

    • when: on_success
    • allow_failure: false

    The job is added to the pipeline:

    • If a rule matches and has when: on_successwhen: delayed or when: always.
    • If no rules match, but the last clause is when: on_successwhen: delayed or when: always (with no rule).

    The job is not added to the pipeline:

    • If no rules match, and there is no standalone when: on_successwhen: delayed or when: always.
    • If a rule matches, and has when: never as the attribute.

    参考:《when

    when is used to implement jobs that are run in case of failure or despite the failure.

    when can be set to one of the following values:

    1. on_success - execute job only when all jobs from prior stages succeed (or are considered succeeding because they are marked allow_failure). This is the default.
    2. on_failure - execute job only when at least one job from prior stages fails.
    3. always - execute job regardless of the status of jobs from prior stages.
    4. manual - execute job manually (added in GitLab 8.10).
    5. delayed - execute job after a certain period (added in GitLab 11.14).

    参考:《allow_failure

    allow_failure allows a job to fail without impacting the rest of the CI suite.

    The default value is false, except for manual jobs using the when: manual syntax, unless using rules: syntax, where all jobs default to false, including when: manual jobs.

    When enabled and the job fails, the job will show an orange warning in the UI.

    However, the logical flow of the pipeline will consider the job a success/passed, and is not blocked.

    job:
      script: "echo Hello, Rules!"
      rules:
        - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
          when: never-----------所有Merge Request触发Pipeline,不运行此job。
        - if: '$CI_PIPELINE_SOURCE == "schedule"'
          when: never-----------所有Schedule触发Pipeline,不运行此job。
        - when: on_success------所有其他情况,在前面所有job都成功情况下,运行此job。

    参考:《rules:if

    $CI_PIPELINE_SOURCE的取值可能情况包括:

    ValueDescription
    push For pipelines triggered by a git push event, including for branches and tags.
    web For pipelines created by using Run pipeline button in the GitLab UI, from the project’s CI/CD > Pipelines section.
    trigger For pipelines created by using a trigger token.
    schedule For scheduled pipelines.
    api For pipelines triggered by the pipelines API.
    external When using CI services other than GitLab.
    pipeline For multi-project pipelines created by using the API with CI_JOB_TOKEN.
    chat For pipelines created by using a GitLab ChatOps command.
    webide For pipelines created by using the WebIDE.
    merge_request_event For pipelines created when a merge request is created or updated. Required to enable merge request pipelinesmerged results pipelines, and merge trains.
    external_pull_request_event When an external pull request on GitHub is created or updated. See Pipelines for external pull requests.
    parent_pipeline For pipelines triggered by parent/child pipeline with rules, use this in the child pipeline configuration so that it can be triggered by the parent pipeline.

    Other commonly used variables for if clauses:

    • if: $CI_COMMIT_TAG: If changes are pushed for a tag.
    • if: $CI_COMMIT_BRANCH: If changes are pushed to any branch.
    • if: '$CI_COMMIT_BRANCH == "master"': If changes are pushed to master.
    • if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH': If changes are pushed to the default branch (usually master). Useful if reusing the same configuration in multiple projects with potentially different default branches.
    • if: '$CI_COMMIT_BRANCH =~ /regex-expression/': If the commit branch matches a regular expression.
    • if: '$CUSTOM_VARIABLE !~ /regex-expression/': If the custom variable CUSTOM_VARIABLE does not match a regular expression.
    • if: '$CUSTOM_VARIABLE == "value1"': If the custom variable CUSTOM_VARIABLE is exactly value1.

    还有一些其他常用的变量:

    参考:《Predefined environment variables reference

    CI_MERGE_REQUEST_SOURCE_BRANCH_NAME: The URL of the source project of the merge request if the pipelines are for merge requests. Available only if only: [merge_requests] or rules syntax is used and the merge request is created.
    CI_MERGE_REQUEST_TARGET_BRANCH_NAME: The target branch name of the merge request if the pipelines are for merge requests. Available only if only: [merge_requests] or rules syntax is used and the merge request is created.

    variables

    参考:《variables

    GitLab CI/CD allows you to define variables inside .gitlab-ci.yml that are then passed in the job environment.

    They can be set globally and per-job.

    When the variables keyword is used on a job level, it will override the global YAML variables and predefined ones of the same name.

    build b1:
      stage: .post
      only:
        - merge_requests
      variables:
        TRIGGER_SOURCE: "repo-a1"
      trigger:
        project: DevOps-testfarm/repo-b1
        branch: master

    Git strategy

    参考:《Git strategy

    GIT_STRATEGY is used for getting recent application code, either globally or per-job in the variables section.

    There are three possible values: clone, fetch, and none.

    clone is the slowest option. It clones the repository from scratch for every job, ensuring that the local working copy is always pristine.
    fetch is faster as it re-uses the local working copy (falling back to clone if it does not exist).
    none also re-uses the local working copy, but skips all Git operations (including GitLab Runner’s pre-clone script, if present).

    variables:
      GIT_STRATEGY: fetch

    Git submodule strategy

     参考:《Git submodule strategy

    The GIT_SUBMODULE_STRATEGY variable is used to control if / how Git submodules are included when fetching the code before a build. You can set them globally or per-job in the variables section.

    There are three possible values: none, normal, and recursive:

    none means that submodules won’t be included when fetching the project code. This is the default, which matches the pre-v1.10 behavior.

    normal means that only the top-level submodules will be included.

    git submodule sync
    git submodule update --init

    recursive means that all submodules (including submodules of submodules) will be included.

    git submodule sync --recursive
    git submodule update --init --recursive

    Package Registry

    参考《GitLab Package Registry

    Conan

    apt-get install python3-pip
    pip install conan
  • 相关阅读:
    设置Linux环境的欢迎登陆信息
    负载均衡实现之随机数
    数据库连接池原理分析及模拟实现
    SpringBoot入门 (十四) Security安全控制
    SpringBoot入门 (十三) WebSocket使用
    SpringBoot入门 (十二) 定时任务
    SpringBoot入门 (十一) 数据校验
    SpringBoot入门 (十) 发送邮件
    SpringBoot入门 (九) MQ使用
    SpringBoot入门 (八) Cache使用
  • 原文地址:https://www.cnblogs.com/arnoldlu/p/13367340.html
Copyright © 2020-2023  润新知