kubernets controller 和 CRD的扩展

Kubernetes 状态管理与扩展 

Kubernetes中文指南/云原生应用架构实践手册(201910) 

如何从零开始编写一个 Kubernetes CRD。 

sample git repo

各个组件开发指导

operator 介绍   （官网 operator Tutorial）

 此图来自谷歌员工的实践介绍

client-go的使用和源码分析 

(dlv) p pods
*k8s.io/api/core/v1.PodList {
        TypeMeta: k8s.io/apimachinery/pkg/apis/meta/v1.TypeMeta {Kind: "", APIVersion: ""},
        ListMeta: k8s.io/apimachinery/pkg/apis/meta/v1.ListMeta {
                SelfLink: "/api/v1/pods",
                ResourceVersion: "145208",
                Continue: "",},
        Items: []k8s.io/api/core/v1.Pod len: 8, cap: 8, [
                (*k8s.io/api/core/v1.Pod)(0xc00033a000),
                (*k8s.io/api/core/v1.Pod)(0xc00033a360),
                (*k8s.io/api/core/v1.Pod)(0xc00033a6c0),
                (*k8s.io/api/core/v1.Pod)(0xc00033aa20),
                (*k8s.io/api/core/v1.Pod)(0xc00033ad80),
                (*k8s.io/api/core/v1.Pod)(0xc00033b0e0),
                (*k8s.io/api/core/v1.Pod)(0xc00033b440),
                (*k8s.io/api/core/v1.Pod)(0xc00033b7a0),
        ],}

　　

从运行流程和list-watch看kubernetes系统的设计理念

 开发operator扩展kubernetes 调研整理

如何基于Kubernetes开发自定义的Controller

Kubernetes CRD (CustomResourceDefinition) 自定义资源类型

kubernetes系列之十四：Kubernetes CRD(CustomResourceDefinition)概览

kubernetes系列之十六：Kubernetes CRD sample-controller的编译和测试

This Picture from this blog and official doc

 Analyzing value of Operator Framework for Kubernetes community

Install GO:

wget -O- https://raw.githubusercontent.com/udhos/update-golang/master/update-golang.sh  | sudo bash

export PATH=$PATH:/usr/local/go/bin

go env |grep GOPATH >> ~/.profile
go env |grep GOROOT >> ~/.profile
echo "export GOPATH" >> ~/.profile
echo "export GOROOT " >> ~/.profile
echo "export PATH=$PATH:/usr/local/go/bin:$GOPATH/bin:$GOROOT/bin" >> ~/.profile

source ~/.profile

install sample: 

go get github.com/tools/godep
cd $GOPATH/src
git clone https://github.com/kubernetes/sample-controller.git
cd sample-controller
godep restore
cd $GOPATH/src/k8s.io/sample-controller
go build -o sample-controller .

 用户关心的实现， sample： type Controller struct

debug:

go get github.com/derekparker/delve/cmd/dlv
dlv exec ./sample-controller -- -kubeconfig=$HOME/.kube/config

分析:

main.go

1.  获取配置信息（构建client使用）

 clientcmd.BuildConfigFromFlags(masterURL, kubeconfig)

 从"/home/ubuntu/.kube/config" 获取登陆的证书, key和CA

2. 生成一个client，来登陆api server

// "k8s.io/client-go/kubernetes"

kubeClient, err := kubernetes.NewForConfig(cfg)

3. 生成一个sample的client，用户来登陆controller

// clientset "k8s.io/sample-controller/pkg/client/clientset/versioned"

clientset.NewForConfig(cfg)

4.  分别生成 kubeInformerFactory 和 exampleInformerFactory

// kubeinformers "k8s.io/client-go/informers"

// informers "k8s.io/sample-controller/pkg/client/informers/externalversions"

// SharedInformerFactory a small interface to allow for adding an informer without an import cycle
type SharedInformerFactory interface {
        Start(stopCh <-chan struct{})
        InformerFor(obj runtime.Object, newFunc NewInformerFunc) cache.SharedIndexInformer
}

// SharedInformerOption defines the functional option type for SharedInformerFactory.
type SharedInformerOption func(*sharedInformerFactory) *sharedInformerFactory

type sharedInformerFactory struct {
        client           versioned.Interface
        namespace        string
        tweakListOptions internalinterfaces.TweakListOptionsFunc
        lock             sync.Mutex
        defaultResync    time.Duration
        customResync     map[reflect.Type]time.Duration

        informers map[reflect.Type]cache.SharedIndexInformer
        // startedInformers is used for tracking which informers have been started.
        // This allows Start() to be called multiple times safely.
        startedInformers map[reflect.Type]bool
}

　reflect.Type  Go Reflect 

SharedInformer具有共享数据缓存，并且能够将对缓存更改的通知分发给通过AddEventHandler注册的多个listeners。 如果使用此方法，则与标准Informer相比，有一个行为不同。 当您收到通知时，缓存将至少与通知一样的fresh，甚至更fresh。 您不应该依赖缓存的内容与处理函数中收到的通知完全匹配。 如果创建后紧接着是删除，则缓存可能没有这个条目了。 这比广播者有优势，因为它允许我们在多个控制器之间共享公共缓存。 扩展广播者需要我们为每个wach保留重复的缓存。

// SharedInformer has a shared data cache and is capable of distributing notifications for changes
// to the cache to multiple listeners who registered via AddEventHandler. If you use this, there is
// one behavior change compared to a standard Informer.  When you receive a notification, the cache
// will be AT LEAST as fresh as the notification, but it MAY be more fresh.  You should NOT depend
// on the contents of the cache exactly matching the notification you've received in handler
// functions.  If there was a create, followed by a delete, the cache may NOT have your item.  This
// has advantages over the broadcaster since it allows us to share a common cache across many
// controllers. Extending the broadcaster would have required us keep duplicate caches for each
// watch.
type SharedInformer interface {
        // AddEventHandler adds an event handler to the shared informer using the shared informer's resync
        // period.  Events to a single handler are delivered sequentially, but there is no coordination
        // between different handlers.
        AddEventHandler(handler ResourceEventHandler)
        // AddEventHandlerWithResyncPeriod adds an event handler to the shared informer using the
        // specified resync period.  Events to a single handler are delivered sequentially, but there is
        // no coordination between different handlers.
        AddEventHandlerWithResyncPeriod(handler ResourceEventHandler, resyncPeriod time.Duration)
        // GetStore returns the Store.
        GetStore() Store
        // GetController gives back a synthetic interface that "votes" to start the informer
        GetController() Controller
        // Run starts the shared informer, which will be stopped when stopCh is closed.
        Run(stopCh <-chan struct{})
        // HasSynced returns true if the shared informer's store has synced.
        HasSynced() bool
        // LastSyncResourceVersion is the resource version observed when last synced with the underlying
        // store. The value returned is not synchronized with access to the underlying store and is not
        // thread-safe.
        LastSyncResourceVersion() string
}

type SharedIndexInformer interface {
        SharedInformer
        // AddIndexers add indexers to the informer before it starts.
        AddIndexers(indexers Indexers) error
        GetIndexer() Indexer
}

　　

怀疑是代码自动生成的。

git grep "func NewSharedInformerFactory"
pkg/client/informers/externalversions/factory.go:func NewSharedInformerFactory(client versioned.Interface, defaultResync time.Duration) SharedInformerFactory {
pkg/client/informers/externalversions/factory.go:func NewSharedInformerFactoryWithOptions(client versioned.Interface, defaultResync time.Duration, options ...SharedInformerOption) SharedInformerFactory {
vendor/k8s.io/client-go/informers/factory.go:func NewSharedInformerFactory(client kubernetes.Interface, defaultResync time.Duration) SharedInformerFactory {
vendor/k8s.io/client-go/informers/factory.go:func NewSharedInformerFactoryWithOptions(client kubernetes.Interface, defaultResync time.Duration, options ...SharedInformerOption) SharedInformerFactory {
vendor/k8s.io/code-generator/_examples/MixedCase/informers/externalversions/factory.go:func NewSharedInformerFactory(client versioned.Interface, defaultResync time.Duration) SharedInformerFactory {
vendor/k8s.io/code-generator/_examples/MixedCase/informers/externalversions/factory.go:func NewSharedInformerFactoryWithOptions(client versioned.Interface, defaultResync time.Duration, options ...SharedInformerOption) SharedInformerFactory {
vendor/k8s.io/code-generator/_examples/apiserver/informers/externalversions/factory.go:func NewSharedInformerFactory(client versioned.Interface, defaultResync time.Duration) SharedInformerFactory {
vendor/k8s.io/code-generator/_examples/apiserver/informers/externalversions/factory.go:func NewSharedInformerFactoryWithOptions(client versioned.Interface, defaultResync time.Duration, options ...SharedInformerOption) SharedInformerFactory {
vendor/k8s.io/code-generator/_examples/apiserver/informers/internalversion/factory.go:func NewSharedInformerFactory(client internalversion.Interface, defaultResync time.Duration) SharedInformerFactory {
vendor/k8s.io/code-generator/_examples/apiserver/informers/internalversion/factory.go:func NewSharedInformerFactoryWithOptions(client internalversion.Interface, defaultResync time.Duration, options ...SharedInformerOption) SharedInformerFactory {
vendor/k8s.io/code-generator/_examples/crd/informers/externalversions/factory.go:func NewSharedInformerFactory(client versioned.Interface, defaultResync time.Duration) SharedInformerFactory {
vendor/k8s.io/code-generator/_examples/crd/informers/externalversions/factory.go:func NewSharedInformerFactoryWithOptions(client versioned.Interface, defaultResync time.Duration, options ...SharedInformerOption) SharedInformerFactory {
vendor/k8s.io/code-generator/cmd/informer-gen/generators/factory.go:func NewSharedInformerFactory(client {{.clientSetInterface|raw}}, defaultResync {{.timeDuration|raw}}) SharedInformerFactory {
vendor/k8s.io/code-generator/cmd/informer-gen/generators/factory.go:func NewSharedInformerFactoryWithOptions(client {{.clientSetInterface|raw}}, defaultResync {{.timeDuration|raw}}, options ...SharedInformerOption) SharedInformerFactory {

　　

5.  生成 controller 

"controller.go" 定义了Controller

低层定义见 pkg/client/informers/externalversions/samplecontroller/v1alpha1/foo.go

6.  运行 kubeInformerFactory， exampleInformerFactory 和 controller

(dlv) b k8s.io/sample-controller/pkg/client/informers/externalversions.(*sharedInformerFactory).Start
Breakpoint 5 set at 0xf42f08 for k8s.io/sample-controller/pkg/client/informers/externalversions.(*sharedInformerFactory).Start() ./pkg/client/informers/externalversions/factory.go:111
(dlv) b k8s.io/sample-controller/vendor/k8s.io/client-go/informers.(*sharedInformerFactory).Start
Breakpoint 6 set at 0xefb888 for k8s.io/sample-controller/vendor/k8s.io/client-go/informers.(*sharedInformerFactory).Start() ./vendor/k8s.io/client-go/informers/factory.go:126

// https://github.com/kubernetes/sample-controller/blob/master/pkg/client/informers/externalversions/factory.go

// Start initializes all requested informers.
func (f *sharedInformerFactory) Start(stopCh <-chan struct{}) {
	f.lock.Lock()
	defer f.lock.Unlock()

	for informerType, informer := range f.informers {
		if !f.startedInformers[informerType] {
			go informer.Run(stopCh)
			f.startedInformers[informerType] = true
		}
	}
}

　　

 0  0x0000000000f46733 in main.(*Controller).enqueueFoo
    at ./controller.go:338
 1  0x0000000000f4785e in main.NewController.func1
    at ./controller.go:120
 2  0x0000000000e6f96d in k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnUpdate
    at ./vendor/k8s.io/client-go/tools/cache/controller.go:202
 3  0x0000000000e81066 in k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.(*ResourceEventHandlerFuncs).OnUpdate
    at <autogenerated>:1
 4  0x0000000000e7ee3b in k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.(*processorListener).run.func1.1
    at ./vendor/k8s.io/client-go/tools/cache/shared_informer.go:552
 5  0x00000000009b478c in k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/util/wait.ExponentialBackoff
    at ./vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:203
 6  0x0000000000e7efa9 in k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.(*processorListener).run.func1
    at ./vendor/k8s.io/client-go/tools/cache/shared_informer.go:548
 7  0x00000000009b4c14 in k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1
    at ./vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
 8  0x00000000009b451e in k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil
    at ./vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134
 9  0x00000000009b444d in k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/util/wait.Until
    at ./vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88
10  0x0000000000e7a58d in k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.(*processorListener).run
    at ./vendor/k8s.io/client-go/tools/cache/shared_informer.go:546
11  0x0000000000e7f50a in k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.(*processorListener).run-fm
    at ./vendor/k8s.io/client-go/tools/cache/shared_informer.go:390
12  0x00000000009b4b8f in k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1
    at ./vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71
13  0x00000000004594e1 in runtime.goexit
    at /usr/local/go/src/runtime/asm_amd64.s:1333

　　

0  0x0000000000f469f0 in main.(*Controller).handleObject
    at ./controller.go:378
 1  0x0000000000f47f5e in main.(*Controller).handleObject-fm
    at ./controller.go:130
 2  0x0000000000e6f8f9 in k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnAdd
    at ./vendor/k8s.io/client-go/tools/cache/controller.go:195
 3  0x0000000000e80f32 in k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.(*ResourceEventHandlerFuncs).OnAdd
    at <autogenerated>:1
 4  0x0000000000e7eecd in k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.(*processorListener).run.func1.1
    at ./vendor/k8s.io/client-go/tools/cache/shared_informer.go:554
 5  0x00000000009b478c in k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/util/wait.ExponentialBackoff
    at ./vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:203
 6  0x0000000000e7efa9 in k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.(*processorListener).run.func1
    at ./vendor/k8s.io/client-go/tools/cache/shared_informer.go:548
 7  0x00000000009b4c14 in k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1
    at ./vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
 8  0x00000000009b451e in k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil
    at ./vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134
 9  0x00000000009b444d in k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/util/wait.Until
    at ./vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88
10  0x0000000000e7a58d in k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.(*processorListener).run
    at ./vendor/k8s.io/client-go/tools/cache/shared_informer.go:546
11  0x0000000000e7f50a in k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.(*processorListener).run-fm
    at ./vendor/k8s.io/client-go/tools/cache/shared_informer.go:390
12  0x00000000009b4b8f in k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1
    at ./vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71
13  0x00000000004594e1 in runtime.goexit
    at /usr/local/go/src/runtime/asm_amd64.s:1333

　　

(dlv) print ownerRef
*k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/apis/meta/v1.OwnerReference {
        APIVersion: "samplecontroller.k8s.io/v1alpha1",
        Kind: "Foo",
        Name: "example-foo",
        UID: "118da85e-00fa-11e9-96e2-fa163e199d30",
        Controller: *true,
        BlockOwnerDeletion: *true,}

　　

goroutines                                                                                                                                  [41/326]
  Goroutine 1 - User: ./controller.go:171 main.(*Controller).Run (0xf45a11)
  Goroutine 2 - User: /usr/local/go/src/runtime/asm_amd64.s:311 runtime.systemstack_switch (0x457400)
  Goroutine 3 - User: /usr/local/go/src/runtime/proc.go:303 runtime.gopark (0x42e3db)
  Goroutine 4 - User: /usr/local/go/src/runtime/proc.go:303 runtime.gopark (0x42e3db)
  Goroutine 5 - User: ./vendor/k8s.io/klog/klog.go:941 k8s.io/sample-controller/vendor/k8s.io/klog.(*loggingT).flushDaemon (0x5f185b)
  Goroutine 6 - User: /usr/local/go/src/runtime/sigqueue.go:139 os/signal.signal_recv (0x44243c)
  Goroutine 7 - User: /usr/local/go/src/runtime/proc.go:303 runtime.gopark (0x42e3db)
  Goroutine 8 - User: ./pkg/signals/signal.go:36 k8s.io/sample-controller/pkg/signals.SetupSignalHandler.func1 (0xf449f4)
  Goroutine 9 - User: ./vendor/k8s.io/apimachinery/pkg/watch/mux.go:207 k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/watch.(*Broadcaster).loop
(0x772ab6)
  Goroutine 10 - User: ./vendor/k8s.io/client-go/tools/record/event.go:231 k8s.io/sample-controller/vendor/k8s.io/client-go/tools/record.(*eventBroadcast$
rImpl).StartEventWatcher.func1 (0xe9d054)
  Goroutine 11 - User: ./vendor/k8s.io/client-go/tools/record/event.go:231 k8s.io/sample-controller/vendor/k8s.io/client-go/tools/record.(*eventBroadcast$
rImpl).StartEventWatcher.func1 (0xe9d054)
  Goroutine 12 - User: ./vendor/k8s.io/client-go/util/workqueue/queue.go:198 k8s.io/sample-controller/vendor/k8s.io/client-go/util/workqueue.(*Type).upda$
eUnfinishedWorkLoop (0xea1212)
  Goroutine 13 - User: /usr/local/go/src/runtime/lock_futex.go:228 runtime.notetsleepg (0x40c4f7)
  Goroutine 14 - User: ./vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:206 k8s.io/sample-controller/vendor/k8s.io/client-go/util/workqueue.(*d$
layingType).waitingLoop (0xe9f2cf)
  Goroutine 15 - User: /usr/local/go/src/runtime/sema.go:510 sync.runtime_notifyListWait (0x43f4fb)
  Goroutine 16 - User: /usr/local/go/src/runtime/sema.go:510 sync.runtime_notifyListWait (0x43f4fb)
  Goroutine 18 - User: /usr/local/go/src/runtime/proc.go:3063 runtime.exitsyscall (0x4353a9)
  Goroutine 20 - User: /usr/local/go/src/runtime/proc.go:303 runtime.gopark (0x42e3db)
  Goroutine 21 - User: /usr/local/go/src/runtime/proc.go:303 runtime.gopark (0x42e3db)
  Goroutine 22 - User: /usr/local/go/src/runtime/proc.go:303 runtime.gopark (0x42e3db)
  Goroutine 34 - User: /usr/local/go/src/runtime/proc.go:3063 runtime.exitsyscall (0x4353a9)
  Goroutine 36 - User: ./vendor/k8s.io/client-go/tools/cache/shared_informer.go:425 k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.(*sharedP
rocessor).run (0xe79b46)
  Goroutine 37 - User: ./vendor/k8s.io/client-go/tools/cache/controller.go:103 k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.(*controller).
Run.func1 (0xe7e0d4)
  Goroutine 38 - User: ./vendor/k8s.io/client-go/tools/cache/reflector.go:302 k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.(*Reflector).w$
tchHandler (0xe764e5)
  Goroutine 41 - User: /usr/local/go/src/runtime/proc.go:303 runtime.gopark (0x42e3db)
  Goroutine 42 - User: /usr/local/go/src/runtime/proc.go:303 runtime.gopark (0x42e3db)
  Goroutine 43 - User: /usr/local/go/src/runtime/proc.go:303 runtime.gopark (0x42e3db)
  Goroutine 44 - User: /usr/local/go/src/runtime/proc.go:303 runtime.gopark (0x42e3db)
  Goroutine 45 - User: /usr/local/go/src/runtime/proc.go:303 runtime.gopark (0x42e3db)
  Goroutine 50 - User: ./vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:374 k8s.io/sample-controller/vendor/k8s.io/apimachinery/pkg/util/wait.poller.fu$
c1.1 (0x9b4d93)
  Goroutine 52 - User: ./vendor/k8s.io/client-go/tools/cache/shared_informer.go:425 k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.(*shared$
rocessor).run (0xe79b46)
  Goroutine 53 - User: ./vendor/k8s.io/client-go/tools/cache/controller.go:103 k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.(*controller)$
Run.func1 (0xe7e0d4)
  Goroutine 54 - User: ./vendor/k8s.io/client-go/tools/cache/reflector.go:302 k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.(*Reflector).w$
tchHandler (0xe764e5)
  Goroutine 58 - User: /usr/local/go/src/runtime/proc.go:303 runtime.gopark (0x42e3db)
  Goroutine 59 - User: /usr/local/go/src/runtime/proc.go:303 runtime.gopark (0x42e3db)
  Goroutine 60 - User: /usr/local/go/src/runtime/proc.go:303 runtime.gopark (0x42e3db)
  Goroutine 61 - User: /usr/local/go/src/runtime/proc.go:303 runtime.gopark (0x42e3db)
  Goroutine 62 - User: /usr/local/go/src/runtime/proc.go:303 runtime.gopark (0x42e3db)
  Goroutine 63 - User: /usr/local/go/src/runtime/proc.go:303 runtime.gopark (0x42e3db)
  Goroutine 65 - User: /usr/local/go/src/runtime/netpoll.go:173 internal/poll.runtime_pollWait (0x428dc6)
  Goroutine 66 - User: ./vendor/k8s.io/client-go/tools/cache/shared_informer.go:549 k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.(*proces$
orListener).run.func1.1 (0xe7ed04)
  Goroutine 67 - User: ./vendor/k8s.io/client-go/tools/cache/shared_informer.go:517 k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.(*proces$
orListener).pop (0xe7a356)
  Goroutine 68 - User: /usr/local/go/src/runtime/proc.go:303 runtime.gopark (0x42e3db)
  Goroutine 69 - User: /usr/local/go/src/runtime/proc.go:303 runtime.gopark (0x42e3db)
  Goroutine 71 - User: ./vendor/k8s.io/client-go/tools/cache/reflector.go:207 k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.(*Reflector).L$
stAndWatch.func1 (0xe7e56d)
  Goroutine 72 - User: /usr/local/go/src/runtime/lock_futex.go:228 runtime.notetsleepg (0x40c4f7)
  Goroutine 74 - User: /usr/local/go/src/runtime/sema.go:510 sync.runtime_notifyListWait (0x43f4fb)
* Goroutine 82 - User: ./controller.go:341 main.(*Controller).enqueueFoo (0xf46741) (thread 12081)
  Goroutine 83 - User: ./vendor/k8s.io/client-go/tools/cache/shared_informer.go:517 k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.(*process
orListener).pop (0xe7a356)
  Goroutine 99 - User: ./vendor/k8s.io/client-go/tools/cache/reflector.go:207 k8s.io/sample-controller/vendor/k8s.io/client-go/tools/cache.(*Reflector).Li
stAndWatch.func1 (0xe7e56d)
  Goroutine 102 - User: /usr/local/go/src/runtime/sema.go:510 sync.runtime_notifyListWait (0x43f4fb)
  Goroutine 103 - User: /usr/local/go/src/runtime/sema.go:510 sync.runtime_notifyListWait (0x43f4fb)
  Goroutine 114 - User: /usr/local/go/src/runtime/proc.go:303 runtime.gopark (0x42e3db)
  Goroutine 146 - User: /usr/local/go/src/runtime/proc.go:303 runtime.gopark (0x42e3db)
  Goroutine 179 - User: /usr/local/go/src/runtime/sema.go:510 sync.runtime_notifyListWait (0x43f4fb)
  Goroutine 187 - User: /usr/local/go/src/runtime/proc.go:303 runtime.gopark (0x42e3db)

　　

Other:

Understanding kubernetes networking: ingress

Load Balance East-West Traffic in Kubernetes Environment Using NetScaler CPX

Kubernetes网络分析之Flannel

How we run Kubernetes in Kubernetes aka Kubeception

Istio Handbook——Istio中文指南/服务网格实践手册

Istio docs   

Kubernetes与云原生应用概览 

Controllers (offical)  

Kubernetes中文文档 v1.14  

kubernetes 客户端KubeClient使用及常用api  (c#)

比较典型CRD的例子是：

基于Helm和Operator的K8S应用管理

带你入门operator-framework

CoreOS有状态应用管理的框架--Operator详解

如何在GO语言中使用Kubernetes API？

视频： 沃趣：袁琳峰-使用 Operator 来扩展 Kubernetes

 kubebuilder [feature] exponential backoff 指数回退 (github code), k8s RequeueAfter 重新进队列机制 

Kubernetes Operators Best Practices   

Kubernetes 实战-Operator Finalizers 实现  

How we built a controller using KubeBuilder with Test Driven development, Part 1  

kubebuilder-starwars-exampl  

kubebuilder 构建与相关代码分析

kubebuilder 中文文档由云原生社区主导翻译

Kubebuilder 中文文档 （翻译一半）

kubebuilder-book-v1.0   kubebuilder-v2.0   book-v1.book.kubebuilder.io

The Cluster API Book （详细的开发文档）

Error Back-off with Controller Runtime   

Extending the Kubernetes API   

Kubernetes之controller-runtime事件再处理  （详细代码分析）

K8S中编写自己的CRD及Controller简明指南  

Timer based reconciliation (SyncPeriod *time.Duration)

深入解析 Kubebuilder：让编写 CRD 变得更简单  (分析的很到位）

一般来说 Kinds 和 Resources 是 1:1 的，比如 pods Resource 对应 Pod Kind，但是有时候相同的 Kind 可能对应多个 Resources，比如 Scale Kind 可能对应很多 Resources：deployments/scale，replicasets/scale，对于 CRD 来说，只会是 1:1 的关系。

使用 kubebuilder 创建 operator 示例  

Groups、Versions 和 Kinds 之间的关系   

Kubernetes operator 模式开发实践   

kubebuilder 设计哲学   (kubebuilder 详细设计)

Fire `reconcile()` on a timer?