Task:
Install HAProxy for use as Layer 4 load balancer - Raw TCP / SSL Passthrough mode.
Notes:
- HAProxy, the DNS, and DHCP are outside the cluster typically. LBs and the API are set-up before the cluster is initialized.
- API: Must be stateless, with no session persistence
- /readyz: Configure load balancer to take a maximum of 30 seconds from the time the API server turns off the /readyz endpoint to the removal of the API server instance from the pool. Within the time frame after /readyz returns an error or becomes healthy, the endpoint must have been removed or added. Probing every 5 or 10 seconds, with two successful requests to become healthy and three to become unhealthy, are well-tested values.
API Load Balancer Ports:
- 6443: K8s API server
- - Internal and External
- - Bootstrap until after cluster initialized, and afterwards, the Control Planes.
- - Must have /readyz endpoint for API server health check probe configured.
- 22623: Machine config server
- - Internal only
- - Bootstrap until after cluster initialized, and afterwards, the Control Planes.
Applications Ingress Load Balancer Ports:
- 443: HTTPS apps traffic
- - Internal and External
- - Host pool members that run the Ingress Controller, used by pods, compute/worker nodes (or control nodes if 0 workers)
- 80: HTTP apps traffic
- - Internal and External
- - Host pool members that run the Ingress Controller, used by pods, compute/worker nodes (or control nodes if 0 workers)
Steps:
1. Install HAProxy VMs.
- Include a VIP with a primary (master) and a secondary where the VIP fails over to the secondary until the primary comes back up.
- On both VMs, set to handle SELinux.
$ sudo setsebool -P haproxy_connect_any=1
<confirm success>
2. Configure HAProxy:
- Set-up the API to not use session persistence
- Set-up the API /readyz endpoint
- Set-up the App Ingress with session persistence if the container workloads (apps) require it
- Use the below HAProxy OKD documentation sample configuration file as a guide
- Remove the bootstrap entries after cluster initialization
- If running 0 compute/worker nodes, change the sample configuration of compute0 and compute1 to the 3 control nodes
3. Validate HAProxy:
a. Verify the listening ports (6443, 22623, 443, and 80) on the HAProxy VMs/nodes:
$ netstat -nltupe
<confirm ports listening>
b. Shutdown the primary HAProxy VM and confirm the secondary uses the VIP to process requests.
OKD HAProxy Documentation Sample:
global
log 127.0.0.1 local2
pidfile /var/run/haproxy.pid
maxconn 4000
daemon
defaults
mode http
log global
option dontlognull
option http-server-close
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout check 10s
maxconn 3000
listen api-server-6443
bind *:6443
mode tcp
option httpchk GET /readyz HTTP/1.0
option log-health-checks
balance roundrobin
server bootstrap bootstrap.ocp4.example.com:6443 verify none check check-ssl inter 10s fall 2 rise 3 backup
server master0 master0.ocp4.example.com:6443 weight 1 verify none check check-ssl inter 10s fall 2 rise 3
server master1 master1.ocp4.example.com:6443 weight 1 verify none check check-ssl inter 10s fall 2 rise 3
server master2 master2.ocp4.example.com:6443 weight 1 verify none check check-ssl inter 10s fall 2 rise 3
listen machine-config-server-22623
bind *:22623
mode tcp
server bootstrap bootstrap.ocp4.example.com:22623 check inter 1s backup
server master0 master0.ocp4.example.com:22623 check inter 1s
server master1 master1.ocp4.example.com:22623 check inter 1s
server master2 master2.ocp4.example.com:22623 check inter 1s
listen ingress-router-443
bind *:443
mode tcp
balance source
server compute0 compute0.ocp4.example.com:443 check inter 1s
server compute1 compute1.ocp4.example.com:443 check inter 1s
listen ingress-router-80
bind *:80
mode tcp
balance source
server compute0 compute0.ocp4.example.com:80 check inter 1s
server compute1 compute1.ocp4.example.com:80 check inter 1s
OKD HAProxy OCP4-HelperNode Sample:
#---------------------------------------------------------------------
# Example configuration for a possible web application. See the
# full configuration options online.
#
# http://haproxy.1wt.eu/download/1.4/doc/configuration.txt
#
#---------------------------------------------------------------------
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
# to have these messages end up in /var/log/haproxy.log you will
# need to:
#
# 1) configure syslog to accept network log events. This is done
# by adding the '-r' option to the SYSLOGD_OPTIONS in
# /etc/sysconfig/syslog
#
# 2) configure local2 events to go to the /var/log/haproxy.log
# file. A line like the following can be added to
# /etc/sysconfig/syslog
#
# local2.* /var/log/haproxy.log
#
log 127.0.0.1 local2
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
daemon
# turn on stats unix socket
stats socket /var/lib/haproxy/stats
#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
mode tcp
log global
option httplog
option dontlognull
option http-server-close
option forwardfor except 127.0.0.0/8
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 4h
timeout server 4h
timeout http-keep-alive 10s
timeout check 10s
maxconn 3000
#---------------------------------------------------------------------
listen stats
bind :9000
mode http
stats enable
stats uri /
stats refresh 15s
monitor-uri /healthz
frontend openshift-api-server
bind *:6443
{% if ipi %}
acl sni-api req_ssl_sni -i api.{{ dns.clusterid }}.{{ dns.domain | lower }}
use_backend openshift-api-server if sni-api
{% else %}
default_backend openshift-api-server
option tcplog
{% endif %}
backend openshift-api-server
{% if haproxy_apiserver_healthcheck %}
mode tcp
option log-health-checks
option httpchk GET /readyz HTTP/1.0
http-check expect status 200
default-server check check-ssl inter 3s fall 2 rise 3 verify none
{% endif %}
{% if ipi %}
balance roundrobin
server backend-api {{ helper.api_ipaddr }}:6443 check
{% else %}
balance source
{% if bootstrap is defined %}
server {{ bootstrap.name | lower }} {{ bootstrap.ipaddr }}:6443 check
{% endif %}
{% for m in masters %}
server {{ m.name | lower }} {{ m.ipaddr }}:6443 check
{% endfor %}
{% endif %}
{% if not ipi %}
frontend machine-config-server
bind *:22623
default_backend machine-config-server
option tcplog
backend machine-config-server
balance source
{% if bootstrap is defined %}
server {{ bootstrap.name | lower }} {{ bootstrap.ipaddr }}:22623 check
{% endif %}
{% for m in masters %}
server {{ m.name | lower }} {{ m.ipaddr }}:22623 check
{% endfor %}
{% endif %}
frontend ingress-http
bind *:80
{% if ipi %}
acl sni-ingress req_ssl_sni -i apps.{{ dns.clusterid }}.{{ dns.domain | lower }}
use_backend ingress-http if sni-ingress
{% else %}
default_backend ingress-http
option tcplog
{% endif %}
backend ingress-http
{% if ipi %}
balance roundrobin
server backend-ingress {{ helper.ingress_ipaddr }}:80 check
{% else %}
balance source
{% if workers is defined %}
{% for w in workers %}
server {{ w.name | lower }}-http-router{{ loop.index0 }} {{ w.ipaddr }}:80 check
{% endfor %}
{% else %}
{% for m in masters %}
server {{ m.name | lower }}-http-router{{ loop.index0 }} {{ m.ipaddr }}:80 check
{% endfor %}
{% endif %}
{% endif %}
frontend ingress-https
bind *:443
{% if ipi %}
acl sni-https req_ssl_sni -i apps.{{ dns.clusterid }}.{{ dns.domain | lower }}
use_backend ingress-https if sni-https
{% else %}
default_backend ingress-https
option tcplog
{% endif %}
backend ingress-https
{% if ipi %}
balance roundrobin
server backend-https {{ helper.ingress_ipaddr }}:443 check
{% else %}
balance source
{% if workers is defined %}
{% for w in workers %}
server {{ w.name | lower }}-https-router{{ loop.index0 }} {{ w.ipaddr }}:443 check
{% endfor %}
{% else %}
{% for m in masters %}
server {{ m.name | lower }}-https-router{{ loop.index0 }} {{ m.ipaddr }}:443 check
{% endfor %}
{% endif %}
{% endif %}
#---------------------------------------------------------------------
previous page
|