Multi-Node Kubernetes Setup

This guide provides instructions for setting up a multi-node Kubernetes cluster using k3s. The guide assumes a minimum three-node cluster configuration for high availability.

Prerequisites

Hardware Requirements

Per node:

• 4 Cores

• 16 GB RAM

• 50GB disk space

• Static private IP

Network Requirements

• TCP/UDP communication between nodes enabled

• All nodes can reach each other

• Firewall/Security Group rules configured for inter-node communication

Example Setup

Assume three nodes:

• Node1: Primary control plane

• Node2: Secondary control plane

• Node3: Secondary control plane

Installation

1. Set Up First Server Node (Node1)

# Install k3s server
curl -sfL https://get.k3s.io | sh -s - server \
  --cluster-init \
  --token=YOUR_TOKEN

2. Join Additional Server Nodes

# On Node2 and Node3
curl -sfL https://get.k3s.io | sh -s - server \
  --server https://NODE1_IP:6443 \
  --token=YOUR_TOKEN

3. Verify Cluster

# Check cluster status
kubectl get nodes -o wide

# Expected output showing all nodes with control-plane roles
NAME    STATUS   ROLES                       AGE   VERSION
node1   Ready    control-plane,etcd,master   10m   v1.28.5+k3s1
node2   Ready    control-plane,etcd,master   8m    v1.28.5+k3s1
node3   Ready    control-plane,etcd,master   8m    v1.28.5+k3s1

Deploy Observo Site

1. Configure Access (On Node1)

sudo cat /etc/rancher/k3s/k3s.yaml > ~/.kube/config

2. Deploy Site

# Apply cert-manager CRDs
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.12.0/cert-manager.crds.yaml

# Deploy Observo Site
helm upgrade --install -n observo-client observo-site \
    oci://public.ecr.aws/e4z0a1h1/observo-site \
    --create-namespace \
    --values=/path/to/downloaded_config.yaml

Configure Access

1. Configure nginx on Each Node

# Install nginx
sudo apt install nginx-full nginx-extras

# Get service IP
export SERVICE_IP=$(kubectl get svc -n observo-client data-plane-gateway-service -o jsonpath='{.spec.clusterIP}')

# Configure nginx stream
sudo tee /etc/nginx/conf.d/stream/tcp-proxy.conf << EOF
stream {
    map \$server_port \$destination_port {
        default 10001; 
        ~^(\d+)$ \$1; 
    }

    server {
        listen 10001-10050;
        proxy_pass ${SERVICE_IP}:\$destination_port;
        proxy_connect_timeout 1s;
    }
} 
EOF

# Update nginx config
sudo sed -i '/^http {/i include /etc/nginx/conf.d/stream/*.conf;' /etc/nginx/nginx.conf

# Restart nginx
sudo systemctl restart nginx

Verify Installation

1. Check Node Status

kubectl get nodes
kubectl get pods -n observo-client -o wide

2. Test Connectivity

# Test from each node
curl -v http://localhost:10001/health

High Availability Testing

1. Test Node Failover

# Stop k3s on one node
sudo systemctl stop k3s

# Check cluster status
kubectl get nodes

2. Check Pod Distribution

kubectl get pods -n observo-client -o wide --sort-by='.spec.nodeName'

Troubleshooting

Network Connectivity

# Test inter-node communication
ping <node-ip>

# Test k8s API server
curl -k https://NODE1_IP:6443

Cluster Status

# Check etcd status
kubectl get events -A | grep etcd

Node Issues

# Check node conditions
kubectl describe node <node-name>

# Check system logs
journalctl -u k3s

Maintenance

Backup Etcd (k3s)

k3s etcd-snapshot save

Update Nodes

curl -sfL https://get.k3s.io | sh -

For additional assistance, refer to:

• Single-Node Guide

• Cleanup Guide