Setting up Multicluster Service Mesh with Rafay CLI
This is the second of a two-part series. Read Part 1.
Over the past several months, our team has been working on scaling Rafay’s SaaS controller. As a crucial part of this, we embarked on setting up multicluster Istio environments. During this process, we encountered and successfully tackled the challenges previously mentioned. These challenges encompassed managing the complexity of the configuration, ensuring consistent settings across clusters, establishing secure network connectivity and handling service discovery, monitoring and troubleshooting complexities.
To overcome these challenges, we adopted Infrastructure as Code (IaC) approaches for configuration management and developed a command line interface (CLI) automation tool to ensure consistent and streamlined multicluster Istio deployments. The CLI follows the “multi-primary on different networks” model described in the Istio documentation. The topology we use in our multicluster Istio deployments looks like the image below.
The CLI uses a straightforward configuration. Below is an example of the configuration format:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 |
$ cat examples/mesh.yaml apiVersion: ristioctl.k8smgmt.io/v3 kind: Certificate metadata: name: ristioctl-certs spec: validityHours: 2190 password: false sanSuffix: istio.io # Subject Alternative Name Suffix meshID: uswestmesh --- apiVersion: ristioctl.k8smgmt.io/v3 kind: Cluster metadata: name: cluster1 spec: kubeconfigFile: "kubeconfig-istio-demo.yaml" context: cluster1 meshID: uswestmesh version: "1.18.0" installHelloWorld: true #deploy sample HelloWorld application --- apiVersion: ristioctl.k8smgmt.io/v3 kind: Cluster metadata: name: cluster2 spec: kubeconfigFile: "kubeconfig-istio-demo.yaml" context: cluster2 meshID: uswestmesh version: "1.18.0" installHelloWorld: true #deploy sample HelloWorld application |
(Note: The example above is a generic representation.)
In this configuration, the CLI is set up to work with two Kubernetes clusters: cluster1 and cluster2. Each cluster is defined with its respective details, including the Kubernetes kubeconfig file, the context and the version of Istio to be installed. The CLI uses this configuration to establish connectivity between services across the clusters and create the multicluster service mesh.
Explanation of the configuration:
Certificate: The CLI establishes trust between all clusters in the mesh using this configuration. It will generate and deploy distinct certificates for each cluster. All cluster certificates are issued by the same root certificate authority (CA). Internally, the CLI uses the step-ca tool.
Explanation:
- apiVersion: The version of the API being used, in this case, it’s ristioctl.k8smgmt.io/v3.
- kind: The type of resource, which is Certificate in this case.
- metadata: Metadata associated with the resource, such as the resource name.
- spec: This section contains the specifications or settings for the resource.
- validityHours: Specifies the validity period of the certificate in hours.
- password: Indicates whether a password is required.
- sanSuffix: Subject Alternative Name (SAN) Suffix for the certificate.
- meshID: Identifier for the multicluster service mesh.
Cluster: These are cluster resources used to define individual Kubernetes clusters that will be part of the multicluster service mesh. Each cluster resource represents a different Kubernetes cluster.
Explanation:
- kubeconfigFile: Specifies the path to the kubeconfig file for the respective cluster, which contains authentication details and cluster information.
- context: The Kubernetes context associated with the cluster, which defines a named set of access parameters.
- meshID: Identifies the multicluster service mesh that these clusters will be connected to.
- version: Specifies the version of Istio to be deployed in the clusters.
- installHelloWorld: Indicates whether to deploy a sample HelloWorld application in each cluster.
Overall, this configuration describes the necessary settings to set up a multicluster service mesh using the ristioctl CLI tool. It includes the specification for a certificate and Kubernetes clusters that will be part of the service mesh. The ristioctl CLI tool will use this configuration to deploy Istio and other required configurations to create a unified and scalable mesh over these clusters. The steps below outline the tasks the CLI tool handles internally to set up a multicluster service mesh. Let’s further explain each step:
- Configure trust across all clusters in the mesh: The CLI tool establishes trust between the Kubernetes clusters participating in the multicluster service mesh. This trust allows secure communication and authentication between services in different clusters. This involves generating and distributing certificates and keys for mutual TLS (Transport Layer Security) authentication.
- Deploy Istio into the clusters: The CLI deploys Istio into each Kubernetes cluster within the mesh.
- Deploy east-west gateway into the clusters: The east-west gateway is an Istio component responsible for handling traffic within the service mesh, specifically the traffic flowing between services in different clusters (east-west traffic). The CLI deploys the east-west gateway into each cluster to enable cross-cluster communication.
- Expose services in the clusters: The CLI ensures that services run within each cluster are appropriately exposed and accessible to the other clusters in the multicluster service mesh.
- Provision cross-cluster service discovery using Rafay ZTKA-based secure channel: Rafay ZTKA (Zero Trust Kubectl Access) is a secure channel technology that enables cross-cluster Kube API server communication.
By automating these steps, the CLI simplifies setting up a multicluster service mesh, reducing the operational complexity for users and ensuring a unified and scalable mesh over clusters in different environments. This approach enhances connectivity, security and observability, allowing organizations to adopt a multicloud or hybrid cloud strategy easily.
To use it:
ristioctl apply -f examples/mesh.yaml
The CLI is open source. You can find more details at https://github.com/RafaySystems/rafay-istio-multicluster/blob/main/README.md.
We use Rafay Zero Trust Kubectl Access (ZTKA) to prevent exposing the Kubernetes Cluster Kube API server to a different network for improved security. To implement this, you need to incorporate Rafay’s ZTKA kubeconfig in the configuration. The resulting topology will resemble the following:
Conclusion
Multicluster service connectivity is crucial for various organizational needs. While Istio provides multicluster connectivity, configuring it can be complex and cumbersome. Therefore, we have developed a tool to simplify the configuration process. Ensuring secure network connectivity between clusters is paramount to safeguarding data in the multicluster environment. With our tool, organizations can streamline the setup of multicluster service mesh and establish a secure and scalable infrastructure to support their distributed applications effectively.
