• Supervisor安装与配置


    介绍

    总览

    Supervisor是一个客户端/服务器系统,允许其用户控制类似UNIX的操作系统上的许多进程。它受到以下方面的启发:

    方便

    需要为每个单个流程实例编写rc.d脚本通常很不方便。 rc.d脚本是进程初始化/自动启动/管理的一个很好的最低公分母形式,但是编写和维护它们可能很麻烦。此外,rc.d脚本无法自动重新启动崩溃的进程,并且许多程序在崩溃时无法正确地自行重启。Supervisord将进程作为其子进程启动,并且可以配置为在崩溃时自动重新启动它们。也可以将其自动配置为自行调用启动进程。

    准确性

    在UNIX上,通常很难获得准确的启动/关闭状态信息。Pidfile经常说谎。Supervisord将流程作为子流程启动,因此它始终了解其子代的真实上/下状态,并且可以方便地查询该数据。

    代表团

    需要控制流程状态的用户通常只需要这样做。他们不希望或不需要完全的Shell访问运行这些进程的计算机。侦听“低” TCP端口的进程通常需要以root用户身份启动和重新启动(UNIX功能不全)。通常情况下,允许“正常”人员停止或重新启动这样的过程是完全可以的,但是为他们提供shell访问通常是不切实际的,并且为他们提供root访问或sudo访问通常是不可能的。(正确)也很难向他们解释为什么存在此问题。如果超级用户以root用户身份启动,则可以允许“普通”用户控制此类过程,而无需向他们解释问题的复杂性。Supervisorctl允许以非常有限的形式访问计算机,

    工艺组

    流程通常需要成组地启动和停止,有时甚至需要按照“优先级顺序”进行。通常很难向人们解释如何执行此操作。Supervisor允许您为进程分配优先级,并允许用户通过supervisorctl客户端发出命令,如“全部启动”和“全部重新启动”,这将按照预先分配的优先级顺序启动它们。此外,可以将流程分为“流程组”,并且可以将一组逻辑相关的流程作为一个单元停止和启动。

    特征

    简单

    通过简单易懂的INI样式配置文件配置Supervisor。它提供了许多每个进程的选项,使您的工作变得更轻松,例如重新启动失败的进程和自动日志轮换。

    集中

    主管为您提供了一个开始,停止和监视过程的地方。可以单独或成组控制过程。您可以配置Supervisor以提供本地或远程命令行和Web界面。

    高效的

    主管通过fork / exec启动其子进程,并且子进程不守护。进程终止时,操作系统会立即向Supervisor发送信号,这与某些依赖麻烦的PID文件和定期轮询以重新启动失败的进程的解决方案不同。

    可扩展的

    Supervisor具有一个简单的事件通知协议,该协议可以使用任何语言编写的程序对其进行监视,并且具有用于控制的XML-RPC接口。它还使用扩展点构建,Python开发人员可以利用这些扩展点。

    兼容

    除Windows外,Supervisor几乎适用于所有其他方面。它已在Linux,Mac OS X,Solaris和FreeBSD上经过测试和支持。它完全用Python编写,因此安装不需要C编译器。

    久经考验

    尽管Supervisor如今非常活跃,但它不是新软件。Supervisor已经存在了很多年,并且已经在许多服务器上使用。

    主管组件

    监督的

    服务器主管的名字叫supervisor它负责自行调用启动子程序,响应来自客户端的命令,重新启动崩溃或退出的子进程,记录其子进程stdoutstderr 输出以及生成和处理与子进程生存期中的点相对应的“事件”。

    服务器进程使用配置文件。这通常位于/etc/supervisord.conf中此配置文件是“ Windows-INI”样式的配置文件。通过适当的文件系统权限来确保此文件的安全很重要,因为它可能包含未加密的用户名和密码。

    监督员

    主管的命令行客户端名为 supervisorctl它为supervisor提供的功能提供了类似shell的界面supervisorctl,用户可以连接到不同的 supervisord进程(一次一个),对,停止控制的子流程获得地位和启动的子进程,并获得运行的进程的列表supervisord

    命令行客户端通过UNIX域套接字或Internet(TCP)套接字与服务器对话。服务器可以断言客户端的用户应在允许客户端执行命令之前出示身份验证凭据。客户端进程通常使用与服务器相同的配置文件,但是其中包含[supervisorctl]节的任何配置文件都可以使用。

    网络服务器

    与功能媲美A(稀疏)的Web用户界面 supervisorctl可以通过浏览器,如果你开始访问supervisord对互联网插座。激活配置文件的[inet_http_server]部分后,请访问服务器URL(例如http:// localhost:9001 /)以通过Web界面查看和控制进程状态

     安装

    [root@topcheer ~]# sudo yum install python-setuptools-devel
    已加载插件:fastestmirror, langpacks
    Determining fastest mirrors
     * base: mirror.jdcloud.com
     * extras: mirror.jdcloud.com
     * updates: mirrors.163.com
    base                                                                                                                                | 3.6 kB  00:00:00
    extras                                                                                                                              | 2.9 kB  00:00:00
    updates                                                                                                                             | 2.9 kB  00:00:00
    (1/4): extras/7/x86_64/primary_db                                                                                                   | 153 kB  00:00:01
    (2/4): base/7/x86_64/group_gz                                                                                                       | 165 kB  00:00:01
    (3/4): base/7/x86_64/primary_db                                                                                                     | 6.0 MB  00:00:03
    (4/4): updates/7/x86_64/primary_db                                                                                                  | 5.1 MB  00:00:03
    软件包 python-setuptools-0.9.8-7.el7.noarch 已安装并且是最新版本
    无须任何处理
    [root@topcheer ~]# easy_install supervisor
    Searching for supervisor
    Reading https://pypi.python.org/simple/supervisor/
    Best match: supervisor 4.1.0
    Downloading https://files.pythonhosted.org/packages/de/87/ee1ad8fa533a4b5f2c7623f4a2b585d3c1947af7bed8e65bc7772274320e/supervisor-4.1.0.tar.gz#sha256=2dc86fe0476e945e61483d614ceb2cf4f93b95282eb243bdf792621994360383
    Processing supervisor-4.1.0.tar.gz
    Writing /tmp/easy_install-q0ws8c/supervisor-4.1.0/setup.cfg
    Running supervisor-4.1.0/setup.py -q bdist_egg --dist-dir /tmp/easy_install-q0ws8c/supervisor-4.1.0/egg-dist-tmp-YY6JL0
    warning: no previously-included files matching '*' found under directory 'docs/.build'
    Adding supervisor 4.1.0 to easy-install.pth file
    Installing echo_supervisord_conf script to /usr/bin
    Installing pidproxy script to /usr/bin
    Installing supervisorctl script to /usr/bin
    Installing supervisord script to /usr/bin
    
    Installed /usr/lib/python2.7/site-packages/supervisor-4.1.0-py2.7.egg
    Processing dependencies for supervisor
    Finished processing dependencies for supervisor
    [root@topcheer ~]# cat /etc/supervisord.conf
    cat: /etc/supervisord.conf: 没有那个文件或目录
    [root@topcheer ~]# echo_supervisord_conf
    ; Sample supervisor config file.
    ;
    ; For more information on the config file, please see:
    ; http://supervisord.org/configuration.html
    ;
    ; Notes:
    ;  - Shell expansion ("~" or "$HOME") is not supported.  Environment
    ;    variables can be expanded using this syntax: "%(ENV_HOME)s".
    ;  - Quotes around values are not supported, except in the case of
    ;    the environment= options as shown below.
    ;  - Comments must have a leading space: "a=b ;comment" not "a=b;comment".
    ;  - Command will be truncated if it looks like a config file comment, e.g.
    ;    "command=bash -c 'foo ; bar'" will truncate to "command=bash -c 'foo ".
    ;
    ; Warning:
    ;  Paths throughout this example file use /tmp because it is available on most
    ;  systems.  You will likely need to change these to locations more appropriate
    ;  for your system.  Some systems periodically delete older files in /tmp.
    ;  Notably, if the socket file defined in the [unix_http_server] section below
    ;  is deleted, supervisorctl will be unable to connect to supervisord.
    
    [unix_http_server]
    file=/tmp/supervisor.sock   ; the path to the socket file
    ;chmod=0700                 ; socket file mode (default 0700)
    ;chown=nobody:nogroup       ; socket file uid:gid owner
    ;username=user              ; default is no username (open server)
    ;password=123               ; default is no password (open server)
    
    ; Security Warning:
    ;  The inet HTTP server is not enabled by default.  The inet HTTP server is
    ;  enabled by uncommenting the [inet_http_server] section below.  The inet
    ;  HTTP server is intended for use within a trusted environment only.  It
    ;  should only be bound to localhost or only accessible from within an
    ;  isolated, trusted network.  The inet HTTP server does not support any
    。。。。。

    生成配置文件

    [root@topcheer ~]# mkdir /etc/supervisor
    [root@topcheer ~]# echo_supervisord_conf > /etc/supervisor/supervisord.conf
    [root@topcheer ~]# cat /etc/supervisor/supervisord.conf
    ; Sample supervisor config file.
    ;
    ; For more information on the config file, please see:
    ; http://supervisord.org/configuration.html
    ;
    ; Notes:
    ;  - Shell expansion ("~" or "$HOME") is not supported.  Environment
    ;    variables can be expanded using this syntax: "%(ENV_HOME)s".
    ;  - Quotes around values are not supported, except in the case of
    ;    the environment= options as shown below.
    ;  - Comments must have a leading space: "a=b ;comment" not "a=b;comment".
    ;  - Command will be truncated if it looks like a config file comment, e.g.
    ;    "command=bash -c 'foo ; bar'" will truncate to "command=bash -c 'foo ".
    ;
    ; Warning:
    ;  Paths throughout this example file use /tmp because it is available on most
    ;  systems.  You will likely need to change these to locations more appropriate
    ;  for your system.  Some systems periodically delete older files in /tmp.
    ;  Notably, if the socket file defined in the [unix_http_server] section below
    ;  is deleted, supervisorctl will be unable to connect to supervisord.
    
    [unix_http_server]
    file=/tmp/supervisor.sock   ; the path to the socket file
    ;chmod=0700                 ; socket file mode (default 0700)
    ;chown=nobody:nogroup       ; socket file uid:gid owner
    ;username=user              ; default is no username (open server)
    ;password=123               ; default is no password (open server)
    
    ; Security Warning:
    ;  The inet HTTP server is not enabled by default.  The inet HTTP server is
    ;  enabled by uncommenting the [inet_http_server] section below.  The inet
    ;  HTTP server is intended for use within a trusted environment only.  It
    ;  should only be bound to localhost or only accessible from within an
    ;  isolated, trusted network.  The inet HTTP server does not support any
    ;  form of encryption.  The inet HTTP server does not use authentication
    ;  by default (see the username= and password= options to add authentication).
    ;  Never expose the inet HTTP server to the public internet.
    
    ;[inet_http_server]         ; inet (TCP) server disabled by default
    ;port=127.0.0.1:9001        ; ip_address:port specifier, *:port for all iface
    ;username=user              ; default is no username (open server)
    ;password=123               ; default is no password (open server)

    加入Solr管理

    [root@topcheer ~]# cat /etc/supervisor/supervisord.conf
    ; Sample supervisor config file.
    ;
    ; For more information on the config file, please see:
    ; http://supervisord.org/configuration.html
    ;
    ; Notes:
    ;  - Shell expansion ("~" or "$HOME") is not supported.  Environment
    ;    variables can be expanded using this syntax: "%(ENV_HOME)s".
    ;  - Quotes around values are not supported, except in the case of
    ;    the environment= options as shown below.
    ;  - Comments must have a leading space: "a=b ;comment" not "a=b;comment".
    ;  - Command will be truncated if it looks like a config file comment, e.g.
    ;    "command=bash -c 'foo ; bar'" will truncate to "command=bash -c 'foo ".
    ;
    ; Warning:
    ;  Paths throughout this example file use /tmp because it is available on most
    ;  systems.  You will likely need to change these to locations more appropriate
    ;  for your system.  Some systems periodically delete older files in /tmp.
    ;  Notably, if the socket file defined in the [unix_http_server] section below
    ;  is deleted, supervisorctl will be unable to connect to supervisord.
    
    [unix_http_server]
    file=/tmp/supervisor.sock   ; the path to the socket file
    ;chmod=0700                 ; socket file mode (default 0700)
    ;chown=nobody:nogroup       ; socket file uid:gid owner
    ;username=user              ; default is no username (open server)
    ;password=123               ; default is no password (open server)
    
    ; Security Warning:
    ;  The inet HTTP server is not enabled by default.  The inet HTTP server is
    ;  enabled by uncommenting the [inet_http_server] section below.  The inet
    ;  HTTP server is intended for use within a trusted environment only.  It
    ;  should only be bound to localhost or only accessible from within an
    ;  isolated, trusted network.  The inet HTTP server does not support any
    ;  form of encryption.  The inet HTTP server does not use authentication
    ;  by default (see the username= and password= options to add authentication).
    ;  Never expose the inet HTTP server to the public internet.
    
    [inet_http_server]         ; inet (TCP) server disabled by default
    port=0.0.0.0:9001        ; ip_address:port specifier, *:port for all iface
    username=user              ; default is no username (open server)
    password=123               ; default is no password (open server)
    
    [supervisord]
    logfile=/tmp/supervisord.log ; main log file; default $CWD/supervisord.log
    logfile_maxbytes=50MB        ; max main logfile bytes b4 rotation; default 50MB
    logfile_backups=10           ; # of main logfile backups; 0 means none, default 10
    loglevel=info                ; log level; default info; others: debug,warn,trace
    pidfile=/tmp/supervisord.pid ; supervisord pidfile; default supervisord.pid
    nodaemon=false               ; start in foreground if true; default false
    minfds=1024                  ; min. avail startup file descriptors; default 1024
    minprocs=200                 ; min. avail process descriptors;default 200
    ;umask=022                   ; process file creation umask; default 022
    ;user=supervisord            ; setuid to this UNIX account at startup; recommended if root
    ;identifier=supervisor       ; supervisord identifier, default is 'supervisor'
    ;directory=/tmp              ; default is not to cd during start
    ;nocleanup=true              ; don't clean up tempfiles at start; default false
    ;childlogdir=/tmp            ; 'AUTO' child log dir, default $TEMP
    ;environment=KEY="value"     ; key value pairs to add to environment
    ;strip_ansi=false            ; strip ansi escape codes in logs; def. false
    
    ; The rpcinterface:supervisor section must remain in the config file for
    ; RPC (supervisorctl/web interface) to work.  Additional interfaces may be
    ; added by defining them in separate [rpcinterface:x] sections.
    
    [rpcinterface:supervisor]
    supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
    
    ; The supervisorctl section configures how supervisorctl will connect to
    ; supervisord.  configure it match the settings in either the unix_http_server
    ; or inet_http_server section.
    
    [supervisorctl]
    serverurl=unix:///tmp/supervisor.sock ; use a unix:// URL  for a unix socket
    ;serverurl=http://127.0.0.1:9001 ; use an http:// url to specify an inet socket
    ;username=chris              ; should be same as in [*_http_server] if set
    ;password=123                ; should be same as in [*_http_server] if set
    ;prompt=mysupervisor         ; cmd line prompt (default "supervisor")
    ;history_file=~/.sc_history  ; use readline history if available
    
    ; The sample program section below shows all possible program subsection values.
    ; Create one or more 'real' program: sections to be able to control them under
    ; supervisor.
    
    ;[program:theprogramname]
    ;command=/bin/cat              ; the program (relative uses PATH, can take args)
    ;process_name=%(program_name)s ; process_name expr (default %(program_name)s)
    ;numprocs=1                    ; number of processes copies to start (def 1)
    ;directory=/tmp                ; directory to cwd to before exec (def no cwd)
    ;umask=022                     ; umask for process (default None)
    ;priority=999                  ; the relative start priority (default 999)
    ;autostart=true                ; start at supervisord start (default: true)
    ;startsecs=1                   ; # of secs prog must stay up to be running (def. 1)
    ;startretries=3                ; max # of serial start failures when starting (default 3)
    ;autorestart=unexpected        ; when to restart if exited after running (def: unexpected)
    ;exitcodes=0                   ; 'expected' exit codes used with autorestart (default 0)
    ;stopsignal=QUIT               ; signal used to kill process (default TERM)
    ;stopwaitsecs=10               ; max num secs to wait b4 SIGKILL (default 10)
    ;stopasgroup=false             ; send stop signal to the UNIX process group (default false)
    ;killasgroup=false             ; SIGKILL the UNIX process group (def false)
    ;user=chrism                   ; setuid to this UNIX account to run the program
    ;redirect_stderr=true          ; redirect proc stderr to stdout (default false)
    ;stdout_logfile=/a/path        ; stdout log path, NONE for none; default AUTO
    ;stdout_logfile_maxbytes=1MB   ; max # logfile bytes b4 rotation (default 50MB)
    ;stdout_logfile_backups=10     ; # of stdout logfile backups (0 means none, default 10)
    ;stdout_capture_maxbytes=1MB   ; number of bytes in 'capturemode' (default 0)
    ;stdout_events_enabled=false   ; emit events on stdout writes (default false)
    ;stdout_syslog=false           ; send stdout to syslog with process name (default false)
    ;stderr_logfile=/a/path        ; stderr log path, NONE for none; default AUTO
    ;stderr_logfile_maxbytes=1MB   ; max # logfile bytes b4 rotation (default 50MB)
    ;stderr_logfile_backups=10     ; # of stderr logfile backups (0 means none, default 10)
    ;stderr_capture_maxbytes=1MB   ; number of bytes in 'capturemode' (default 0)
    ;stderr_events_enabled=false   ; emit events on stderr writes (default false)
    ;stderr_syslog=false           ; send stderr to syslog with process name (default false)
    ;environment=A="1",B="2"       ; process environment additions (def no adds)
    ;serverurl=AUTO                ; override serverurl computation (childutils)
    
    ; The sample eventlistener section below shows all possible eventlistener
    ; subsection values.  Create one or more 'real' eventlistener: sections to be
    ; able to handle event notifications sent by supervisord.
    
    ;[eventlistener:theeventlistenername]
    ;command=/bin/eventlistener    ; the program (relative uses PATH, can take args)
    ;process_name=%(program_name)s ; process_name expr (default %(program_name)s)
    ;numprocs=1                    ; number of processes copies to start (def 1)
    ;events=EVENT                  ; event notif. types to subscribe to (req'd)
    ;buffer_size=10                ; event buffer queue size (default 10)
    ;directory=/tmp                ; directory to cwd to before exec (def no cwd)
    ;umask=022                     ; umask for process (default None)
    ;priority=-1                   ; the relative start priority (default -1)
    ;autostart=true                ; start at supervisord start (default: true)
    ;startsecs=1                   ; # of secs prog must stay up to be running (def. 1)
    ;startretries=3                ; max # of serial start failures when starting (default 3)
    ;autorestart=unexpected        ; autorestart if exited after running (def: unexpected)
    ;exitcodes=0                   ; 'expected' exit codes used with autorestart (default 0)
    ;stopsignal=QUIT               ; signal used to kill process (default TERM)
    ;stopwaitsecs=10               ; max num secs to wait b4 SIGKILL (default 10)
    ;stopasgroup=false             ; send stop signal to the UNIX process group (default false)
    ;killasgroup=false             ; SIGKILL the UNIX process group (def false)
    ;user=chrism                   ; setuid to this UNIX account to run the program
    ;redirect_stderr=false         ; redirect_stderr=true is not allowed for eventlisteners
    ;stdout_logfile=/a/path        ; stdout log path, NONE for none; default AUTO
    ;stdout_logfile_maxbytes=1MB   ; max # logfile bytes b4 rotation (default 50MB)
    ;stdout_logfile_backups=10     ; # of stdout logfile backups (0 means none, default 10)
    ;stdout_events_enabled=false   ; emit events on stdout writes (default false)
    ;stdout_syslog=false           ; send stdout to syslog with process name (default false)
    ;stderr_logfile=/a/path        ; stderr log path, NONE for none; default AUTO
    ;stderr_logfile_maxbytes=1MB   ; max # logfile bytes b4 rotation (default 50MB)
    ;stderr_logfile_backups=10     ; # of stderr logfile backups (0 means none, default 10)
    ;stderr_events_enabled=false   ; emit events on stderr writes (default false)
    ;stderr_syslog=false           ; send stderr to syslog with process name (default false)
    ;environment=A="1",B="2"       ; process environment additions
    ;serverurl=AUTO                ; override serverurl computation (childutils)
    
    ; The sample group section below shows all possible group values.  Create one
    ; or more 'real' group: sections to create "heterogeneous" process groups.
    
    ;[group:thegroupname]
    ;programs=progname1,progname2  ; each refers to 'x' in [program:x] definitions
    ;priority=999                  ; the relative start priority (default 999)
    
    ; The [include] section can just contain the "files" setting.  This
    ; setting can list multiple files (separated by whitespace or
    ; newlines).  It can also contain wildcards.  The filenames are
    ; interpreted as relative to this file.  Included files *cannot*
    ; include files themselves.
    
    ;[include]
    ;files = relative/directory/*.ini
    
    [program:solr]
    command= /mnt/solr-7.6.0/bin/solr start -force -p 8983
    stdout_logfile          = /mnt/solr.log
    [root@topcheer ~]#

    以上配置文件用到几个部分:

    • [unix_http_server]:这部分设置HTTP服务器监听的UNIX domain socket
      • file: 指向UNIX domain socket,即file=/var/run/supervisor.sock
      • chmod:启动时改变supervisor.sock的权限
    • [supervisord]:与supervisord有关的全局配置需要在这部分设置
      • logfile: 指向记录supervisord进程的log文件
      • pidfile:pidfile保存子进程的路径
      • childlogdir:子进程log目录设为AUTO的log目录
    • [supervisorctl]:
      • serverurl:进入supervisord的URL, 对于UNIX domain sockets, 应设为 unix:///absolute/path/to/file.sock
    • [include]:如果配置文件包含该部分,则该部分必须包含一个files键:
      • files:包含一个或多个文件,这里包含了/etc/supervisor/conf.d/目录下所有的.conf文件,可以在该目录下增加我们自己的配置文件,在该配置文件中增加[program:x]部分,用来运行我们自己的程序,如下:
    • [program:x]:配置文件必须包括至少一个program,x是program名称,必须写上,不能为空
      • command:包含一个命令,当这个program启动时执行
      • directory:执行子进程时supervisord暂时切换到该目录
      • user:账户名
      • startsecs:进程从STARING状态转换到RUNNING状态program所需要保持运行的时间(单位:秒)
      • redirect_stderr:如果是true,则进程的stderr输出被发送回其stdout文件描述符上的supervisord
      • stdout_logfile:将进程stdout输出到指定文件
      • stdout_logfile_maxbytes:stdout_logfile指定日志文件最大字节数,默认为50MB,可以加KB、MB或GB等单位
      • stdout_logfile_backups:要保存的stdout_logfile备份的数量

     常用命令

    supervisorctl status
    supervisorctl stop solr
    supervisorctl start solr
    supervisorctl restart solr
    supervisorctl reread
    supervisorctl update

    启动

    supervisord -c /etc/supervisor/supervisord.conf

    修改配置重启生效

    [root@topcheer supervisor]# vim supervisord.conf
    [root@topcheer supervisor]# supervisorctl reread
    solr: changed
    [root@topcheer supervisor]# supervisorctl update
    solr: stopped
    solr: updated process group
    [root@topcheer supervisor]# cd /mnt

     发现Solr起来了,但是状态显示Exit,比较奇怪。

     

    Web管理界面

    于安全考虑,默认配置是没有开启web管理界面,需要修改supervisord.conf配置文件打开http访权限,将下面的配置:

    ;[inet_http_server] ; inet (TCP) server disabled by default
    ;port=127.0.0.1:9001 ; (ip_address:port specifier, *:port for all iface)
    ;username=user ; (default is no username (open server))
    ;password=123 ; (default is no password (open server))
    修改成:
    
    [inet_http_server] ; inet (TCP) server disabled by default
    port=0.0.0.0:9001 ; (ip_address:port specifier, *:port for all iface)
    username=user ; (default is no username (open server))
    password=123 ; (default is no password (open server))

    port:绑定访问IP和端口,这里是绑定的是本地IP和9001端口
    username:登录管理后台的用户名
    password:登录管理后台的密码

     具体的用法和配置可以参考:官网

  • 相关阅读:
    kafka consumer防止数据丢失(转)
    Cglib源码分析 invoke和invokeSuper的差别(转)
    如何查看k8s存在etcd中的数据(转)
    k8s Nodeport方式下service访问,iptables处理逻辑(转)
    项目演化系列--验证体系
    项目演化系列--路由解析
    项目演化系列--开篇
    基于SWFUpload的angular上传组件
    Query Object--查询对象模式(下)
    Query Object--查询对象模式(上)
  • 原文地址:https://www.cnblogs.com/dalianpai/p/11994815.html
Copyright © 2020-2023  润新知