{"id":4243,"date":"2023-05-30T16:54:05","date_gmt":"2023-05-30T14:54:05","guid":{"rendered":"https:\/\/blog.outscale.com\/developing-a-cluster-api-provider-at-outscale\/"},"modified":"2023-07-07T13:36:50","modified_gmt":"2023-07-07T11:36:50","slug":"developing-a-cluster-api-provider-at-outscale","status":"publish","type":"post","link":"https:\/\/blog.outscale.com\/en\/developing-a-cluster-api-provider-at-outscale\/","title":{"rendered":"Developing a Cluster API provider at OUTSCALE"},"content":{"rendered":"<p><strong>In this post, we will share the journey of developing a Cluster API provider. Cluster API is a Kubernetes operator based on the Kubebuilder<sup>[1]<\/sup> framework.<\/strong><\/p>\n<p><span style=\"font-weight: 400;\">The objective of Cluster API is to reduce the complexity of managing and maintaining a cluster, to ease the provisioning of its infrastructure on cloud and on-premise with a Kubernetes-style API. It automates the cluster lifecycle of Kubernetes clusters (create, upgrade, delete) and manages a massive fleet of Kubernetes clusters. Users can also choose the bootstrapper provider like kubeadm or microk8s and infrastructure providers.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">In this article, you will learn how to build your own Cluster API provider:<\/span><\/p>\n<ul>\n<li><span style=\"font-weight: 400;\">You will see the architecture of a Cluster API.<\/span><\/li>\n<li><span style=\"font-weight: 400;\">The main components you must develop.<\/span><\/li>\n<li><span style=\"font-weight: 400;\">Some focal points you must be careful of.<\/span><\/li>\n<li><span style=\"font-weight: 400;\">You will learn about the unit, functional and E2E tests for Cluster API.<\/span><\/li>\n<\/ul>\n<h1><\/h1>\n<p>&nbsp;<\/p>\n<h1><span style=\"font-weight: 400;\">Architecture of Cluster API infrastructure provider<\/span><\/h1>\n<h2><\/h2>\n<h2><span style=\"font-weight: 400;\">a.<\/span><span style=\"font-weight: 400;\"> \u00a0 <\/span><span style=\"font-weight: 400;\">Choice of Architecture<\/span><\/h2>\n<p><span style=\"font-weight: 400;\">It is important to define the default target cluster architecture based on services like machines, networks, etc. <\/span><span style=\"font-weight: 400;\"><sup>[1]<\/sup> <sup>[2]<\/sup> <sup>[3]<\/sup> <sup>[4]<\/sup>\u00a0<\/span><\/p>\n<p><span style=\"font-weight: 400;\">For example, we have chosen to adopt a single AZ (availability zone) architecture with separate master, worker, and public subnets.<\/span><\/p>\n<p><span style=\"font-weight: 400;\"><img fetchpriority=\"high\" decoding=\"async\" class=\" wp-image-3931 aligncenter\" src=\"https:\/\/blog.outscale.com\/wp-content\/uploads\/2023\/05\/operator_OUTSCALE-1-300x212.png\" alt=\"Architecture of Cluster API infrastructure provider: a single AZ (availability zone) architecture with separate master, worker, and public subnets\" width=\"627\" height=\"443\" srcset=\"https:\/\/blog.outscale.com\/wp-content\/uploads\/2023\/05\/operator_OUTSCALE-1-300x212.png 300w, https:\/\/blog.outscale.com\/wp-content\/uploads\/2023\/05\/operator_OUTSCALE-1-1024x724.png 1024w, https:\/\/blog.outscale.com\/wp-content\/uploads\/2023\/05\/operator_OUTSCALE-1-768x543.png 768w, https:\/\/blog.outscale.com\/wp-content\/uploads\/2023\/05\/operator_OUTSCALE-1-1170x827.png 1170w, https:\/\/blog.outscale.com\/wp-content\/uploads\/2023\/05\/operator_OUTSCALE-1-585x414.png 585w, https:\/\/blog.outscale.com\/wp-content\/uploads\/2023\/05\/operator_OUTSCALE-1.png 1236w\" sizes=\"(max-width: 627px) 100vw, 627px\" \/><\/span><\/p>\n<h2><\/h2>\n<p>&nbsp;<\/p>\n<h2><span style=\"font-weight: 400;\">b.<\/span><span style=\"font-weight: 400;\"> \u00a0 <\/span><span style=\"font-weight: 400;\">Build your application<\/span><\/h2>\n<p>&nbsp;<\/p>\n<p><img decoding=\"async\" class=\" wp-image-3927 aligncenter\" src=\"https:\/\/blog.outscale.com\/wp-content\/uploads\/2023\/05\/architecture_OUTSCALE-300x118.png\" alt=\"Creating a provider for Cluster API\" width=\"419\" height=\"165\" srcset=\"https:\/\/blog.outscale.com\/wp-content\/uploads\/2023\/05\/architecture_OUTSCALE-300x118.png 300w, https:\/\/blog.outscale.com\/wp-content\/uploads\/2023\/05\/architecture_OUTSCALE-1024x402.png 1024w, https:\/\/blog.outscale.com\/wp-content\/uploads\/2023\/05\/architecture_OUTSCALE-768x301.png 768w, https:\/\/blog.outscale.com\/wp-content\/uploads\/2023\/05\/architecture_OUTSCALE-1170x459.png 1170w, https:\/\/blog.outscale.com\/wp-content\/uploads\/2023\/05\/architecture_OUTSCALE-585x229.png 585w, https:\/\/blog.outscale.com\/wp-content\/uploads\/2023\/05\/architecture_OUTSCALE.png 1280w\" sizes=\"(max-width: 419px) 100vw, 419px\" \/><\/p>\n<p>&nbsp;<\/p>\n<p><span style=\"font-weight: 400;\">When creating a provider for Cluster API, it is important to follow these rules. <\/span><span style=\"font-weight: 400;\">The first step is to create an operator using Kubebuilder to create the infrastructure controller that will be used to manage your cluster provider object and machine provider object.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Kubebuilder is a framework to quickly build and publish Kubernetes API and controllers using CRDs (Custom Resource Definitions).<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Futhermore, Kubebuilder will generate your CRD based on your API definition, the Kustomize<\/span><span style=\"font-weight: 400;\"><sup>[2]<\/sup><\/span><span style=\"font-weight: 400;\"> YAML deployment of your controller and create the skeleton for the reconcile function.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Each time you change the API definition, you will use Kubebuilder to regenerate the CRD and the Kustomize YAML deployment. For more information, please refer to the Cluster API Developer Guide<\/span><span style=\"font-weight: 400;\"><sup>[3]<\/sup><\/span><span style=\"font-weight: 400;\">.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">\u00a0<\/span><\/p>\n<h2><span style=\"font-weight: 400;\">c.<\/span><span style=\"font-weight: 400;\"> \u00a0 <\/span><span style=\"font-weight: 400;\">Create Provider contract<\/span><\/h2>\n<p><span style=\"font-weight: 400;\">In order to publish your Cluster API provider, you will need to create a provider contract with:<\/span><\/p>\n<ul>\n<li><span style=\"font-weight: 400;\">metadata.yaml: a map file between provider release and Cluster API release<\/span><\/li>\n<li><span style=\"font-weight: 400;\">components.yaml: a YAML file that contains all the components for your provider installation<\/span><\/li>\n<li><span style=\"font-weight: 400;\">cluster-template<\/span><span style=\"font-weight: 400;\"><sup>[4]<\/sup><\/span><span style=\"font-weight: 400;\">: a YAML file with all the objects needed in order to create your cluster through environment variables<\/span><\/li>\n<\/ul>\n<p><span style=\"font-weight: 400;\">For example, you can find a release<\/span><span style=\"font-weight: 400;\"><sup>[5]<\/sup><\/span><span style=\"font-weight: 400;\"> of Cluster API at <\/span><span style=\"font-weight: 400;\">OUTSCALE<\/span><span style=\"font-weight: 400;\">.\u00a0<\/span><\/p>\n<p><span style=\"font-weight: 400;\">\u00a0<\/span><\/p>\n<h1><span style=\"font-weight: 400;\">Kubernetes API style with Custom Resource Definitions, controller and cloud scope<\/span><\/h1>\n<p><span style=\"font-weight: 400;\">Your Cluster API provider will be based on the Kubebuilder framework utilizing Kubernetes API with Custom Resource Definitions (CRDs), controllers and cloud scope.<\/span><span style=\"font-weight: 400;\"><br \/>\n<\/span><span style=\"font-weight: 400;\">Furthermore, the cloud scope is the method getter and setter for each cluster and each machine object.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">\u00a0<\/span><\/p>\n<h2><span style=\"font-weight: 400;\">a.<\/span><span style=\"font-weight: 400;\"> \u00a0 <\/span><span style=\"font-weight: 400;\">Kubernetes-style API with Custom Resource Definition<\/span><\/h2>\n<p><span style=\"font-weight: 400;\">The next step is to define your Kubernetes-style API for each controller (machine and cluster) based on services like VM, network, etc. which are controlled through your provider\u2019s resource API in order to create a cluster with machines.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">The idea then is to build into the cluster controller the API definition based on the parameters of each component that is part of the cluster, such as network, loadbalancer, public IPs, etc.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">In our provider, we\u2019ve chosen to define <\/span><i><span style=\"font-weight: 400;\">OscNetwork<\/span><\/i><span style=\"font-weight: 400;\"> in order to handle each cluster component and its parameters.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">It is composed of LoadBalancer, Net, Subnets, InternetService, NatService, RoutesTable, SecurityGroups and PublicIps.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">For a machine controller, the best is to build into it the API definition of each component that will consist of a machine such as a Virtual Machine, disks, SSH Keys, etc.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">We have decided to define OscNode to handle each machine component\u2019s parameters composed of VM, Image, Volumes, KeyPair.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">There are some mandatory status fields for the cluster controller:<\/span><\/p>\n<ul>\n<li><span style=\"font-weight: 400;\">\u201cReady\u201d which is a boolean field that is \u201ctrue\u201d when the infrastructure is ready to be used.<\/span><\/li>\n<li><span style=\"font-weight: 400;\">\u201cControlPlaneEndpoint\u201d which is the endpoint used to expose the targeting cluster API server.<\/span><\/li>\n<\/ul>\n<p><span style=\"font-weight: 400;\">There are also some mandatory fields for a machine controller:<\/span><\/p>\n<ul>\n<li><span style=\"font-weight: 400;\">\u201cReady\u201d which is a boolean field that is \u201ctrue\u201d when the bootstrap config data is ready and ready to be used.<\/span><\/li>\n<li><span style=\"font-weight: 400;\"> \u201cDataSecretName\u201d is a string field which is a reference to the secret name that is stored generated bootstrap data.<\/span><\/li>\n<\/ul>\n<p><span style=\"font-weight: 400;\">\u00a0<\/span><span style=\"font-weight: 400;\">\u00a0<\/span><\/p>\n<h2><span style=\"font-weight: 400;\">b.<\/span><span style=\"font-weight: 400;\"> \u00a0 <\/span><span style=\"font-weight: 400;\">Infrastructure Controllers<\/span><\/h2>\n<p><span style=\"font-weight: 400;\">You need to implement the cluster infrastructure controller and the machine infrastructure controller within the infrastructure controller<\/span><span style=\"font-weight: 400;\"><sup>[5]<\/sup> <\/span><span style=\"font-weight: 400;\">.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">The cluster infrastructure controller manages the K8s cluster lifecycle, and the machine infrastructure controller manages the K8s machine lifecycle.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">At OUTSCALE, the cluster infrastructure will create a cluster with load balancer, net, subnet and route<\/span> <span style=\"font-weight: 400;\">table and it will expose the cluster endpoint.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">After the infrastructure is provisioned, the machine infrastructure will create a virtual machine with a bootstrap mechanism.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">It is possible to create a cluster reconciliation function (\u201creconcileDelete<\/span><span style=\"font-weight: 400;\">\u201d<\/span><span style=\"font-weight: 400;\"> and \u201creconcile\u201d) for each component in the cluster infrastructure controller and the machine infrastructure controller.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">As a controller, you must implement a control loop that observes the desired state of the object, checks the difference between the current state and the desired state, brings the current state in line with the desired state and it repeats continuously.<\/span><\/p>\n<h2><\/h2>\n<p>&nbsp;<\/p>\n<h2><span style=\"font-weight: 400;\">c.<\/span><span style=\"font-weight: 400;\"> \u00a0 <\/span><span style=\"font-weight: 400;\">Webhooks<\/span><\/h2>\n<p><span style=\"font-weight: 400;\">Admission webhooks are controlled by the admission controller.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Admission webhooks are HTTP callbacks which get admission requests, process them and send admission responses. They arbitrate the API request route path between Kubernetes components.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">When you change an object with an operation like creation, deletion, etc. the mutation webhook will modify the object and validation determines which custom policies are accepted or rejected.\u00a0<\/span><\/p>\n<p><span style=\"font-weight: 400;\">There are three webhooks that can be carefully implemented in your Cluster API provider to avoid impacting the system:<\/span><\/p>\n<ul>\n<li><span style=\"font-weight: 400;\">Validation webhook which is used to validate user webhook and which is an implementation of Kubernetes webhook.<\/span><\/li>\n<\/ul>\n<p><span style=\"font-weight: 400;\">It will get an admission request which has an object and accepts or denies the request<\/span> <span style=\"font-weight: 400;\">with a reason message.<\/span><\/p>\n<ul>\n<li><span style=\"font-weight: 400;\">Default webhook which is used to set default values, and which is an implementation of Kubernetes mutation webhook.<\/span><\/li>\n<li><span style=\"font-weight: 400;\">Conversion webhook to have Cluster API to manage multiple versions of the API.<\/span><\/li>\n<\/ul>\n<p><span style=\"font-weight: 400;\">\u00a0<\/span><\/p>\n<h1><span style=\"font-weight: 400;\">Unexpected Behavior<\/span><\/h1>\n<h2><\/h2>\n<h2><span style=\"font-weight: 400;\">a.<\/span><span style=\"font-weight: 400;\"> \u00a0 <\/span><span style=\"font-weight: 400;\">Check API parameters<\/span><\/h2>\n<p><span style=\"font-weight: 400;\">It is necessary to check that the provider&#8217;s resources APIs are never called with bad parameters (resources such as VM, network, load<\/span> <span style=\"font-weight: 400;\">balancer, etc.) If you send bad parameters to the controller, it should be stopped before launching the provider resource API call.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">It avoids having the controller fail over and over on the control loop with bad API call parameters that will keep creating resources from the previous successful API call for each loop.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">\u00a0<\/span><\/p>\n<h2><span style=\"font-weight: 400;\">b.<\/span><span style=\"font-weight: 400;\"> \u00a0 <\/span><span style=\"font-weight: 400;\">Two clusters instead of one<\/span><\/h2>\n<p><span style=\"font-weight: 400;\">Depending on your provider resource API, you should store the ID of your provider resource (e.g.,<\/span> <span style=\"font-weight: 400;\">VM, network, load balancer, etc.) of each cluster, each machine<\/span><span style=\"font-weight: 400;\"><sup>[6]<\/sup> <\/span><span style=\"font-weight: 400;\">\u00a0so that you can use this API to create provider resources on the first reconciliation loop, validate your object still exists in both Kubernetes and your provider on the second reconciliation loop and thus avoid recreating this provider resource.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">It should be stored in your CRD with a status parameter which will contain a map with resourceName and resourceId.<\/span><\/p>\n<h2><\/h2>\n<p>&nbsp;<\/p>\n<h2><span style=\"font-weight: 400;\">c.<\/span><span style=\"font-weight: 400;\"> \u00a0 <\/span><span style=\"font-weight: 400;\">Webhook behavior<\/span><\/h2>\n<p><span style=\"font-weight: 400;\">Your cluster template must be defined with the minimum of parameters as it should use default parameters. Each component parameter can be overridden.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">When you set your webhook, you should only validate the parameters which are in the cluster-template.<\/span><\/p>\n<h1><\/h1>\n<p>&nbsp;<\/p>\n<h1><span style=\"font-weight: 400;\">Cluster API Unit and Functional test<\/span><\/h1>\n<p><span style=\"font-weight: 400;\">It is time then to create a test pyramid with unit-tests and functional tests. Cluster API advises you how to create unit and functional tests which are <\/span><span style=\"font-weight: 400;\">strongly suggested<\/span><span style=\"font-weight: 400;\"> for Cluster API providers.<\/span><\/p>\n<p>&nbsp;<\/p>\n<h2><span style=\"font-weight: 400;\">a.<\/span><span style=\"font-weight: 400;\"> \u00a0 <\/span><span style=\"font-weight: 400;\">Unit Tests<\/span><\/h2>\n<p><span style=\"font-weight: 400;\">With unit tests, you can mock only the cloud scope instead of mocking your API, so you mock functions which call the API instead of mocking the API.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">In fact, there is no need in mocking the API but only mock some functions which call the API.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">With unit tests, we use \u201cgo testing package\u201d<\/span><span style=\"font-weight: 400;\"><sup>[6]<\/sup><\/span><span style=\"font-weight: 400;\"> and \u201cmock\u201d to validate the expected behavior of each function individually.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">\u00a0<\/span><\/p>\n<h2><span style=\"font-weight: 400;\">b.<\/span><span style=\"font-weight: 400;\"> \u00a0 <\/span><span style=\"font-weight: 400;\">Functional tests<\/span><\/h2>\n<p><span style=\"font-weight: 400;\">It is possible to create functional tests for cluster controllers to verify the creation and the deletion of a cluster.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">It is also possible to create a functional test for a machine controller with a cluster controller to verify the creation and deletion of individual machines.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">In those tests, you don\u2019t have \u201ckubeadm\u201d bootstrap configuration<\/span><span style=\"font-weight: 400;\"><sup>[7]<\/sup><\/span><span style=\"font-weight: 400;\"> so it will only test one machine and cluster controller creation and destruction.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">In Cluster API, it is necessary to define the kubeadm bootstrap configuration template which will be used by the bootstrap controller<\/span><span style=\"font-weight: 400;\"><sup>[8]<\/sup><\/span><span style=\"font-weight: 400;\"> to generate a cloud-init<\/span><span style=\"font-weight: 400;\"><sup>[9]<\/sup><\/span><span style=\"font-weight: 400;\"> configuration. In order to validate the cloud init configuration which will be generated by the bootstrap controller, you can add a test which creates a machine with the same cloud initialization configuration (or bootstrap data) as the one defined in cluster templates.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Furthermore, you will test that your machine has the same behavior as a master node, but it is not a master node<\/span><span style=\"font-weight: 400;\"><sup>[10]<\/sup><\/span><span style=\"font-weight: 400;\"> because you don\u2019t use a kubeadm bootstrap configuration.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Each Cluster API bootstrapper like kubeadm, rke, and microk8s will create and update nodes using cloud-init. In fact, the configuration of the node will be in cloud-init format generated by the bootstrap controller using the bootstrap K8s config object.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">\u00a0<\/span><\/p>\n<h1><span style=\"font-weight: 400;\">5.<\/span> <span style=\"font-weight: 400;\">Cluster Api E2E Tests<\/span><\/h1>\n<h2><\/h2>\n<h2><span style=\"font-weight: 400;\">a.<\/span><span style=\"font-weight: 400;\"> \u00a0 <\/span><span style=\"font-weight: 400;\">Validate real use case<\/span><\/h2>\n<p><span style=\"font-weight: 400;\">Creating E2E tests based on the Cluster API framework allows us to check the good behavior on Cluster API in an environment similar to a real production environment. Cluster API advises how to create E2E tests which are <\/span><span style=\"font-weight: 400;\">strongly suggested<\/span><span style=\"font-weight: 400;\"> for Cluster API providers.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">E2E tests help validate that your Cluster API provider controller has the expected behavior with simple use-cases such as upgrade, remedication, high availability control plane.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">E2E test will use all the controllers<\/span><span style=\"font-weight: 400;\"><sup>[11]<\/sup><\/span><span style=\"font-weight: 400;\"> (cluster, machine, bootstrap, control-plane) of cluster API<\/span><span style=\"font-weight: 400;\"> to test the behavior of the previously mentioned use-cases<\/span><span style=\"font-weight: 400;\">.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">For example, we have used a fixed private IP as a virtual machine input parameter. However, if a machine is rolling update it is not possible to have two virtual machines with the same private IP.<\/span><span style=\"font-weight: 400;\">\u00a0<\/span><\/p>\n<h2><\/h2>\n<p>&nbsp;<\/p>\n<h2><span style=\"font-weight: 400;\">b.<\/span><span style=\"font-weight: 400;\"> \u00a0 <\/span><span style=\"font-weight: 400;\">Missing CCM<\/span><\/h2>\n<p><span style=\"font-weight: 400;\">After we launched our first E2E test (ApplyClusterAndWaitTemplate) with both machine and cluster infrastructure controller, we found that this test had never checked the status of pod or node.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">As we discovered afterwards, the cloud controller manager is mandatory for a Cluster API to untaint <\/span><b><i>node.cloudprovider.kubernetes.io\/uninitialized<\/i><\/b><span style=\"font-weight: 400;\"> from nodes.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">To have a cluster and nodes ready, you need to set cni (calico, kube-router, etc.) and remove your node taint <\/span><b><i>node.cloudprovider.kubernetes.io\/uninitialized<\/i><\/b><span style=\"font-weight: 400;\"> from nodes as Cluster API uses a cluster-provider-id to turn your machine into a running node with ccm (cloud provider, metallb, etc.) or untaint nodes one by one.<\/span><\/p>\n<p>&nbsp;<\/p>\n<h2><span style=\"font-weight: 400;\">c.<\/span><span style=\"font-weight: 400;\"> \u00a0 <\/span><span style=\"font-weight: 400;\">Bootstrap Cluster<\/span><\/h2>\n<p><span style=\"font-weight: 400;\">The E2E<\/span> <span style=\"font-weight: 400;\">test framework from Cluster API can be used to have a list of E2E tests. Although, there are several functions that you can call in the framework (ClusterUpgradeConformanceSpec, MachineRemediationSpec, etc.) that need to have a management cluster to access some object statuses (the status of machineList).<\/span><\/p>\n<p><span style=\"font-weight: 400;\">It is possible to launch your E2E tests with the creation of K8s cluster with kind<\/span><span style=\"font-weight: 400;\"><sup>[12]<\/sup><\/span><span style=\"font-weight: 400;\">.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">\u00a0<\/span><\/p>\n<h2><span style=\"font-weight: 400;\">d.<\/span><span style=\"font-weight: 400;\"> \u00a0 <\/span><span style=\"font-weight: 400;\">Conformance test<\/span><\/h2>\n<p><span style=\"font-weight: 400;\">Adding a conformance test that uses sonobuoy<\/span><span style=\"font-weight: 400;\"><sup>[13]<\/sup><\/span><span style=\"font-weight: 400;\"> to validate your cluster allows you to understand what is missing in your configuration or architecture.<\/span><span style=\"font-weight: 400;\"><br \/>\n<\/span><span style=\"font-weight: 400;\">This, for example, helped us understand that we needed to open a bgp port (179) for calico<\/span><span style=\"font-weight: 400;\"><sup>[14]<\/sup><\/span><span style=\"font-weight: 400;\">.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">\u00a0<\/span><\/p>\n<h1><span style=\"font-weight: 400;\">Conclusion<\/span><\/h1>\n<p><span style=\"font-weight: 400;\">Creating your own Cluster API provider is a long journey that should be overseen by a Cluster API project and supported by <\/span><span style=\"font-weight: 400;\">comprehensive documentation<\/span><span style=\"font-weight: 400;\">.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">In this post, we saw that the steps included choosing architecture, setting your API definition, creating the skeleton with Kubebuilder, then implementing your controller to manage your provider resources and create unit, functional and E2E tests.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">OUTSCALE Cluster API provider<\/span><span style=\"font-weight: 400;\"><sup>[15]<\/sup><\/span><span style=\"font-weight: 400;\"> is available to deploy a fleet of clusters at OUTSCALE with kubeadm<\/span><span style=\"font-weight: 400;\"><sup>[16]<\/sup><\/span><span style=\"font-weight: 400;\">.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Currently, our Cluster API provider<\/span><span style=\"font-weight: 400;\"><sup>[17]<\/sup><\/span><span style=\"font-weight: 400;\"> is only available in a mono availability zone. One of the next major features will consist of having our cluster available on two availability zones and having integration with new bootstrappers like microk8s<\/span><span style=\"font-weight: 400;\"><sup>[18]<\/sup><\/span><span style=\"font-weight: 400;\"> and rke<\/span><span style=\"font-weight: 400;\"><sup>[19]<\/sup><\/span><span style=\"font-weight: 400;\">.<\/span><\/p>\n<p>&nbsp;<\/p>\n<p><span style=\"font-weight: 400;\"><sup>[1]<\/sup><\/span><span style=\"font-weight: 400;\"> https:\/\/github.com\/kubernetes-sigs\/kubebuilder<\/span><\/p>\n<p><span style=\"font-weight: 400;\"><sup>[2]<\/sup><\/span><span style=\"font-weight: 400;\"> https:\/\/kustomize.io\/<\/span><\/p>\n<p><span style=\"font-weight: 400;\"><sup>[3]<\/sup><\/span> <span style=\"font-weight: 400;\">https:\/\/cluster-api.sigs.k8s.io\/developer\/guide.html<\/span><span style=\"font-weight: 400;\">\u00a0<\/span><\/p>\n<p><span style=\"font-weight: 400;\"><sup>[4]<\/sup><\/span><span style=\"font-weight: 400;\"> https:\/\/cluster-api.sigs.k8s.io\/developer\/providers\/contracts.html<\/span><\/p>\n<p><span style=\"font-weight: 400;\"><sup>[5]<\/sup><\/span><span style=\"font-weight: 400;\"> https:\/\/github.com\/outscale-dev\/cluster-api-provider-outscale\/releases\/tag\/v0.1.4<\/span><\/p>\n<p><span style=\"font-weight: 400;\"><sup>[6]<\/sup><\/span><span style=\"font-weight: 400;\"> https:\/\/pkg.go.dev\/testing<\/span><\/p>\n<p><span style=\"font-weight: 400;\"><sup>[7]<\/sup><\/span><span style=\"font-weight: 400;\"> https:\/\/cluster-api.sigs.k8s.io\/tasks\/bootstrap\/kubeadm-bootstrap.html<\/span><\/p>\n<p><span style=\"font-weight: 400;\"><sup>[8]<\/sup><\/span><span style=\"font-weight: 400;\"> https:\/\/cluster-api.sigs.k8s.io\/developer\/architecture\/controllers\/bootstrap.html<\/span><\/p>\n<p><span style=\"font-weight: 400;\"><sup>[9]<\/sup><\/span><span style=\"font-weight: 400;\"> https:\/\/cloud-init.io\/<\/span><\/p>\n<p><span style=\"font-weight: 400;\"><sup>[10]<\/sup><\/span><span style=\"font-weight: 400;\"> https:\/\/cluster-api.sigs.k8s.io\/developer\/architecture\/controllers\/machine.html<\/span><\/p>\n<p><span style=\"font-weight: 400;\"><sup>[11]<\/sup><\/span><span style=\"font-weight: 400;\"> https:\/\/cluster-api.sigs.k8s.io\/user\/concepts.html<\/span><\/p>\n<p><span style=\"font-weight: 400;\"><sup>[12]<\/sup><\/span><span style=\"font-weight: 400;\"> https:\/\/kind.sigs.k8s.io\/docs\/user\/quick-start\/<\/span><\/p>\n<p><span style=\"font-weight: 400;\"><sup>[13]<\/sup><\/span><span style=\"font-weight: 400;\"> https:\/\/github.com\/vmware-tanzu\/sonobuoy<\/span><\/p>\n<p><span style=\"font-weight: 400;\"><sup>[14]<\/sup><\/span><span style=\"font-weight: 400;\"> https:\/\/projectcalico.docs.tigera.io\/about\/about-calico<\/span><span style=\"font-weight: 400;\">\u00a0<\/span><\/p>\n<p><span style=\"font-weight: 400;\"><sup>[15]<\/sup><\/span><span style=\"font-weight: 400;\">https:\/\/cluster-api-outscale.oos-website.eu-west-2.outscale.com\/topics\/get-started-with-clusterctl.html<\/span><\/p>\n<p><span style=\"font-weight: 400;\"><sup>[16]<\/sup><\/span><span style=\"font-weight: 400;\"> https:\/\/github.com\/kubernetes-sigs\/cluster-api<\/span><\/p>\n<p><span style=\"font-weight: 400;\"><sup>[17]<\/sup><\/span><span style=\"font-weight: 400;\"> https:\/\/github.com\/outscale\/cluster-api-provider-outscale<\/span><\/p>\n<p><span style=\"font-weight: 400;\"><sup>[18]<\/sup><\/span><span style=\"font-weight: 400;\"> https:\/\/cluster-api.sigs.k8s.io\/tasks\/control-plane\/microk8s-control-plane.html<\/span><\/p>\n<p><span style=\"font-weight: 400;\"><sup>[19]<\/sup><\/span><span style=\"font-weight: 400;\"> https:\/\/github.com\/rancher-sandbox\/cluster-api-provider-rke2<\/span><\/p>\n<p>&nbsp;<\/p>\n<h1><span style=\"font-weight: 400;\">Glossary<\/span><\/h1>\n<p><b><i>outscale-bsu-csi-driver<\/i><\/b><span style=\"font-weight: 400;\">: Expose block storage (BSU) to container workload for container orchestration.<\/span><\/p>\n<p><b><i>BSU<\/i><\/b><span style=\"font-weight: 400;\">:\u00a0 Block storage unit to have the storage capacity (in GiB) you provisioned a standard\/gp2\/io1 volume.<\/span><\/p>\n<p><b><i>CCM<\/i><\/b><span style=\"font-weight: 400;\">: Cloud provider manager to have load balancer service which is associated with LBU.<\/span><\/p>\n<p><b><i>LBU<\/i><\/b><span style=\"font-weight: 400;\">: Load balancer unit to have a running load balancer in public cloud or in a Virtual Private Cloud (Net)<\/span><\/p>\n<p><b><i>OUTSCALE<\/i><\/b><span style=\"font-weight: 400;\">: <\/span><span style=\"font-weight: 400;\">OUTSCALE is a French cloud provider that offers Cloud services on robust and secure IaaS infrastructures, available on demand.<\/span><\/p>\n<p><b><i>OscMachine<\/i><\/b><span style=\"font-weight: 400;\">: The OUTSCALE machine controller.<\/span><\/p>\n<p><b><i>OscMachineTemplate<\/i><\/b><span style=\"font-weight: 400;\">: The OUTSCALE machine template.<\/span><\/p>\n<p><b><i>OscCluster<\/i><\/b><span style=\"font-weight: 400;\">: The OUTSCALE cluster controller.<\/span><\/p>\n<p><b><i>OscClusterTemplate<\/i><\/b><span style=\"font-weight: 400;\">: The OUTSCALE cluster template.<\/span><\/p>\n<p><b>CRD<\/b><span style=\"font-weight: 400;\">: (Custom Resource Definition) Add custom object to Kubernetes resource which included an open-api schema.<\/span><\/p>\n<p><b>Kustomize<\/b><span style=\"font-weight: 400;\">: Templating to customize applications deployed on Kubernetes.<\/span><\/p>\n<p><b><i>Kubernetes operator<\/i><\/b><span style=\"font-weight: 400;\">: Application specific controller to extend functionality of Kubernetes api to create, manage and delete complex applications.<\/span><\/p>\n<p><b><i>Microservices<\/i><\/b><span style=\"font-weight: 400;\">: Microservice are an architectural pattern for applications to be independently deployed, loosely coupled, highly available and handle simple business capabilities, owned by a small team.<\/span><\/p>\n<p><b><i>Bootstrap Provider: <\/i><\/b><span style=\"font-weight: 400;\">It creates a bootstrap data used to bootstrapping a Kubernetes node.<\/span><\/p>\n<p><b><i>CNI<\/i><\/b><span style=\"font-weight: 400;\">: (container network interface) Plugin which configures Kubernetes network.<\/span><\/p>\n<p><b><i>E2E test<\/i><\/b><span style=\"font-weight: 400;\">: (End to end test) Test the entire application to have the expected behavior of the application.<\/span><\/p>\n<p><b><i>ProviderID<\/i><\/b><span style=\"font-weight: 400;\">: A cloud provider ID to identify the machine.<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"<p>In this post, we will share the journey of developing a Cluster API provider. Cluster API&hellip;<\/p>\n","protected":false},"author":33,"featured_media":3937,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"om_disable_all_campaigns":false,"_lmt_disableupdate":"","_lmt_disable":"","footnotes":""},"categories":[356],"tags":[],"class_list":["post-4243","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-tech"],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/blog.outscale.com\/en\/wp-json\/wp\/v2\/posts\/4243","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.outscale.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.outscale.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.outscale.com\/en\/wp-json\/wp\/v2\/users\/33"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.outscale.com\/en\/wp-json\/wp\/v2\/comments?post=4243"}],"version-history":[{"count":1,"href":"https:\/\/blog.outscale.com\/en\/wp-json\/wp\/v2\/posts\/4243\/revisions"}],"predecessor-version":[{"id":4244,"href":"https:\/\/blog.outscale.com\/en\/wp-json\/wp\/v2\/posts\/4243\/revisions\/4244"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blog.outscale.com\/en\/wp-json\/wp\/v2\/media\/3937"}],"wp:attachment":[{"href":"https:\/\/blog.outscale.com\/en\/wp-json\/wp\/v2\/media?parent=4243"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.outscale.com\/en\/wp-json\/wp\/v2\/categories?post=4243"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.outscale.com\/en\/wp-json\/wp\/v2\/tags?post=4243"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}