前情提要
基于以下需求,博主在Kubernetes集群上部署并配置了MetalLB和Traefik。
局域网内特定IP(VIP)访问整个Kubernetes集群的所有服务。
通过独立IP实现故障转移。
控制特定IP允许访问Kubernetes集群内的部分服务,即部分服务的IP白名单模式。
实现Ingress Gateway,即使用域名暴露服务。
技术选型
本章节将根据上文介绍的需求,并经过技术调研,最终博主选择了Traefik和MetalLB技术栈。
选型考量了以下维度:
需求1和需求2可以通过MetalLB实现,因为MetalLB可以具有Layer2模式,通过OSI模型的第二层实现局域网内VIP发现。
需求3和需求4可以通过Traefik实现,因为Traefik可以针对IngressRoute配置Middleware实现IP白名单功能。其实需求3和需求4在理论上使用Nginx Ingress Controller也可以实现,但是博主根据官方文档在Ingress中配置了Annotation后仍然是所有IP均可访问对应Ingress的服务。通过进入Controller DaemonSet各个Pod的Controller容器查看配置文件,发现配置文件中并没有allow和deny指令,故此方案被博主放弃。若有了解原因的大佬,可以在评论区指导一下博主。
实现原理简介
本章节将简单介绍和剖析此方案的实现原理。
MetalLB
MetalLB具有以下两种工作模式:
BGP:通过BGP路由协议的在局域网内进行IP的自动发现,此模式需要内网的路由器支持BGP路由协议。
Layer2:通过在局域网内发送ARP广播包的方式进行IP的自动发现。
需要注意的是,上述两种模式都不仅可以实现VIP的发现,而是可以给Kubernetes集群的每个LoadBalancer模式的Service都分配一个内网IP,实现通过该内网IP直接访问集群内部服务。
Traefik
Traefik是一个具有更多高级特性的网关,可以替代Nginx Ingress Controller实现7层转发(Traefik也可以实现4层转发),博主会根据浅薄的认知,在下文简单介绍一下Traefik的一些特性。
IngressRoute:可以认为是Kubernetes原生的Ingress增强版,可以实现添加Middleware实现更多的功能,也可以更精细的控制(例如基于Header和Query的控制)路由走向和流量转发路径,甚至可以实现TCP/UDP转发和跨Namespace的资源引用。并且对于多命名空间而言,TLS证书只需要配置在Traefik所在的命名空间,Traefik会自动匹配部署时指定的默认证书实现HTTPS访问,而不需要手动指定TLS证书。这个特点可以简化TLS证书的管理和IngressRoute资源清单的编写。
Middleware:这个特性可以理解为在进入IngressRoute前对请求的处理,根据在资源清单里面的列表顺序不同,有处理顺序的差异。
Dashboard:Traefik可以部署Dashboard,这个Dashboard非常直观的展示了Router和对应的Service的状态,也可以查看各个Router的后端服务的对应关系。
Traefik的特性和功能远不止于此,有需要的读者可以自行在Traefik官网和Traefik官方文档查阅相关信息。
部署介绍
本文使用的MetalLB和Traefik均使用Helm部署,博主默认读者会使用Helm的命令且能阅读YAML文件。
MetalLB
本章节将介绍MetalLB的部署方式。
Chart仓库
部署的第一步是添加Chart仓库,仓库地址:https://metallb.github.io/metallb
修改values文件
以下是博主使用的values文件,读者可参考配置。
controller:
affinity: {}
enabled: true
extraContainers: []
image:
pullPolicy: null
repository: quay.io/metallb/controller
tag: null
labels: {}
livenessProbe:
enabled: true
failureThreshold: 20
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
logLevel: info
nodeSelector: {}
podAnnotations: {}
priorityClassName: ''
readinessProbe:
enabled: true
failureThreshold: 10
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
resources: {}
runtimeClassName: ''
securityContext:
fsGroup: 65534
runAsNonRoot: true
runAsUser: 65534
serviceAccount:
annotations: {}
create: true
name: ''
strategy:
type: RollingUpdate
tlsCipherSuites: ''
tlsMinVersion: VersionTLS12
tolerations: []
crds:
enabled: true
validationFailurePolicy: Fail
frrk8s:
enabled: false
external: false
namespace: ''
fullnameOverride: ''
imagePullSecrets: []
loadBalancerClass: ''
nameOverride: ''
prometheus:
controllerMetricsTLSSecret: ''
metricsPort: 7472
namespace: ''
podMonitor:
additionalLabels: {}
annotations: {}
enabled: false
interval: null
jobLabel: app.kubernetes.io/name
metricRelabelings: []
relabelings: []
prometheusRule:
additionalLabels: {}
addressPoolExhausted:
enabled: true
excludePools: ''
labels:
severity: critical
addressPoolUsage:
enabled: true
excludePools: ''
thresholds:
- labels:
severity: warning
percent: 75
- labels:
severity: warning
percent: 85
- labels:
severity: critical
percent: 95
annotations: {}
bgpSessionDown:
enabled: true
labels:
severity: critical
configNotLoaded:
enabled: true
labels:
severity: warning
enabled: false
extraAlerts: []
staleConfig:
enabled: true
labels:
severity: warning
rbacPrometheus: true
rbacProxy:
pullPolicy: null
repository: gcr.io/kubebuilder/kube-rbac-proxy
tag: v0.12.0
scrapeAnnotations: false
serviceAccount: ''
serviceMonitor:
controller:
additionalLabels: {}
annotations: {}
tlsConfig:
insecureSkipVerify: true
enabled: false
interval: null
jobLabel: app.kubernetes.io/name
metricRelabelings: []
relabelings: []
speaker:
additionalLabels: {}
annotations: {}
tlsConfig:
insecureSkipVerify: true
speakerMetricsTLSSecret: ''
rbac:
create: true
speaker:
affinity: {}
enabled: true
excludeInterfaces:
enabled: true
extraContainers: []
frr:
enabled: true
image:
pullPolicy: null
repository: quay.io/frrouting/frr
tag: 9.1.0
metricsPort: 7473
resources: {}
frrMetrics:
resources: {}
ignoreExcludeLB: false
image:
pullPolicy: null
repository: quay.io/metallb/speaker
tag: null
labels: {}
livenessProbe:
enabled: true
failureThreshold: 10
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
logLevel: info
memberlist:
enabled: true
mlBindAddrOverride: ''
mlBindPort: 7946
mlSecretKeyPath: /etc/ml_secret_key
nodeSelector: {}
podAnnotations: {}
priorityClassName: ''
readinessProbe:
enabled: true
failureThreshold: 10
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
reloader:
resources: {}
resources: {}
runtimeClassName: ''
securityContext: {}
serviceAccount:
annotations: {}
create: true
name: ''
startupProbe:
enabled: true
failureThreshold: 30
periodSeconds: 5
tolerateMaster: true
tolerations: []
updateStrategy:
type: RollingUpdate
global:
cattle:
systemProjectId: p-mz6m6
该values文件基本没有需要修改的,但请读者务必确认以下几个字段保持一致,并且自行调整controller.livenessProbe.failureThreshold
和controller.readinessProbe.failureThreshold
:
controller.enable: true
frr.enable: true
crds.enable: true
rbac.create: true
部署
通过Helm命令行工具或者其他工具使用上文的values部署MetalLB。
检查
检查MetalLB部署的命名空间的所有Pod是否正常启动,确保连续15分钟各个容器不重启。若发生容器重启,可以查看容器日志,排除问题。博主在部署后发现frr容器经常重启,发现是探针过于敏感导致的,所以博主修改了values文件里探针的阈值。
配置
MetalLB部署完成后需要进行以下配置,以实现layer2模式的IP发现。
# IPAddressPool
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
name: ingress
namespace: metallb
spec:
addresses:
- 172.16.0.240/32
---
# L2Advertisement
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
name: ingress
namespace: metallb
spec:
ipAddressPools:
- ingress
其中,IPAddressPool的spec.addresses需要以CIDR列表的方式配置需要分配的内网IP。由于博主只需要暴露Traefik的Service,所以此处只分配了一个IP,读者可以根据需要修改。
需要注意的是L2Advertisement
中的spec.ipAddressPools
需要与IPAddressPool
中的metadata.name
一致。
Traefik
Chart仓库
部署的第一步是添加Chart仓库,仓库地址:https://traefik.github.io/charts
修改values文件
以下是博主使用的values文件,读者可参考配置。
additionalArguments:
- '--api.insecure=true'
- '--entryPoints.web.forwardedHeaders.trustedIPs=10.0.0.0/8'
- '--entryPoints.websecure.forwardedHeaders.trustedIPs=10.0.0.0/8'
additionalVolumeMounts:
- mountPath: /plugins-storage
name: plugins
affinity: {}
autoscaling:
enabled: false
certificatesResolvers: {}
commonLabels: {}
core:
defaultRuleSyntax: ''
deployment:
additionalContainers: []
additionalVolumes:
- name: plugins
persistentVolumeClaim:
claimName: plugins-pvc
annotations: {}
dnsConfig: {}
dnsPolicy: ''
enabled: true
healthchecksHost: ''
healthchecksPort: null
healthchecksScheme: null
hostAliases: []
imagePullSecrets: []
initContainers: []
kind: DaemonSet
labels: {}
lifecycle: {}
livenessPath: ''
minReadySeconds: 0
podAnnotations: {}
podLabels: {}
readinessPath: ''
replicas: 1
revisionHistoryLimit: null
runtimeClassName: ''
shareProcessNamespace: false
terminationGracePeriodSeconds: 60
env: null
envFrom: []
experimental:
abortOnPluginFailure: false
fastProxy:
debug: false
enabled: false
kubernetesGateway:
enabled: false
plugins: {}
extraObjects:
- apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: plugins-pvc
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 1Gi
- apiVersion: v1
kind: Service
metadata:
name: traefik-api
spec:
ports:
- name: traefik
port: 8080
protocol: TCP
targetPort: 8080
selector:
app.kubernetes.io/instance: traefik-traefik
app.kubernetes.io/name: traefik
type: ClusterIP
- apiVersion: v1
kind: Secret
metadata:
name: traefik-dashboard-auth-secret
stringData:
password: pass
username: user
type: kubernetes.io/basic-auth
- apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
name: traefik-dashboard-auth
spec:
basicAuth:
secret: traefik-dashboard-auth-secret
- apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
name: traefik-dashboard
namespace: traefik
spec:
entryPoints:
- websecure
routes:
- kind: Rule
match: Host(`traefik.example.com`) && PathPrefix(`/`)
services:
- kind: TraefikService
name: api@internal
gateway:
annotations: {}
enabled: true
infrastructure: {}
listeners:
web:
hostname: ''
namespacePolicy: null
port: 8000
protocol: HTTP
name: ''
namespace: ''
gatewayClass:
enabled: true
labels: {}
name: ''
global:
azure:
enabled: false
images:
hub:
image: traefik-hub
registry: ghcr.io/traefik
tag: latest
proxy:
image: traefik
registry: docker.io/library
tag: latest
cattle:
systemProjectId: p-mz6m6
globalArguments:
- '--global.checknewversion'
- '--global.sendanonymoususage'
hostNetwork: false
hub:
apimanagement:
admission:
customWebhookCertificate: {}
listenAddr: ''
restartOnCertificateChange: true
secretName: hub-agent-cert
enabled: false
openApi:
validateRequestMethodAndPath: false
experimental:
aigateway: false
namespaces: []
providers:
consulCatalogEnterprise:
cache: false
connectAware: false
connectByDefault: false
constraints: ''
defaultRule: Host(`{{ normalize .Name }}`)
enabled: false
endpoint:
address: ''
datacenter: ''
endpointWaitTime: 0
httpauth:
password: ''
username: ''
scheme: ''
tls:
ca: ''
cert: ''
insecureSkipVerify: false
key: ''
token: ''
exposedByDefault: true
namespaces: ''
partition: ''
prefix: traefik
refreshInterval: 15
requireConsistent: false
serviceName: traefik
stale: false
strictChecks: passing, warning
watch: false
microcks:
auth:
clientId: ''
clientSecret: ''
endpoint: ''
token: ''
enabled: false
endpoint: ''
pollInterval: 30
pollTimeout: 5
tls:
ca: ''
cert: ''
insecureSkipVerify: false
key: ''
redis:
cluster: null
database: null
endpoints: ''
password: ''
sentinel:
masterset: ''
password: ''
username: ''
timeout: ''
tls:
ca: ''
cert: ''
insecureSkipVerify: false
key: ''
username: ''
sendlogs: null
token: ''
tracing:
additionalTraceHeaders:
enabled: false
traceContext:
parentId: ''
traceId: ''
traceParent: ''
traceState: ''
image:
pullPolicy: IfNotPresent
registry: docker.io
repository: traefik
tag: null
ingressClass:
enabled: false
isDefaultClass: true
name: traefik
ingressRoute:
dashboard:
annotations: {}
enabled: false
entryPoints:
- traefik
labels: {}
matchRule: PathPrefix(`/dashboard`) || PathPrefix(`/api`)
middlewares: []
services:
- kind: TraefikService
name: api@internal
tls: {}
healthcheck:
annotations: {}
enabled: false
entryPoints:
- traefik
labels: {}
matchRule: PathPrefix(`/ping`)
middlewares: []
services:
- kind: TraefikService
name: ping@internal
tls: {}
instanceLabelOverride: ''
livenessProbe:
failureThreshold: 3
initialDelaySeconds: 2
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 2
logs:
access:
addInternals: false
bufferingSize: null
enabled: true
fields:
general:
defaultmode: keep
names: {}
headers:
defaultmode: drop
names: {}
filters:
minduration: ''
retryattempts: false
statuscodes: ''
format: null
general:
filePath: ''
format: null
level: INFO
noColor: false
metrics:
addInternals: false
otlp:
addEntryPointsLabels: null
addRoutersLabels: null
addServicesLabels: null
enabled: false
explicitBoundaries: []
grpc:
enabled: false
endpoint: ''
insecure: false
tls:
ca: ''
cert: ''
insecureSkipVerify: false
key: ''
http:
enabled: false
endpoint: ''
headers: {}
tls:
ca: ''
cert: ''
insecureSkipVerify: null
key: ''
pushInterval: ''
serviceName: null
prometheus:
addEntryPointsLabels: null
addRoutersLabels: null
addServicesLabels: null
buckets: ''
disableAPICheck: null
entryPoint: metrics
headerLabels: {}
manualRouting: false
prometheusRule:
additionalLabels: {}
enabled: false
namespace: ''
service:
annotations: {}
enabled: false
labels: {}
serviceMonitor:
additionalLabels: {}
enableHttp2: false
enabled: false
followRedirects: false
honorLabels: false
honorTimestamps: false
interval: ''
jobLabel: ''
metricRelabelings: []
namespace: ''
namespaceSelector: {}
relabelings: []
scrapeTimeout: ''
namespaceOverride: ''
nodeSelector: {}
oci_meta:
enabled: false
images:
hub:
image: traefik-hub
tag: latest
proxy:
image: traefik
tag: latest
repo: traefik
persistence:
accessMode: ReadWriteMany
annotations: {}
enabled: false
existingClaim: ''
name: data
path: /data
size: 1Gi
storageClass: longhorn
subPath: ''
volumeName: ''
podDisruptionBudget:
enabled: false
maxUnavailable: null
minAvailable: null
podSecurityContext:
runAsGroup: 65532
runAsNonRoot: true
runAsUser: 65532
podSecurityPolicy:
enabled: false
ports:
metrics:
expose:
default: false
exposedPort: 9100
port: 9100
protocol: TCP
traefik:
expose:
default: false
exposedPort: 8080
hostIP: null
hostPort: null
port: 8080
protocol: TCP
web:
expose:
default: true
exposedPort: 80
forwardedHeaders:
insecure: false
trustedIPs: []
nodePort: null
port: 8000
protocol: TCP
proxyProtocol:
insecure: false
trustedIPs: []
redirections:
entryPoint: {}
targetPort: null
transport:
keepAliveMaxRequests: null
keepAliveMaxTime: null
lifeCycle:
graceTimeOut: null
requestAcceptGraceTimeout: null
respondingTimeouts:
idleTimeout: null
readTimeout: null
writeTimeout: null
websecure:
allowACMEByPass: false
appProtocol: null
containerPort: null
expose:
default: true
exposedPort: 443
forwardedHeaders:
insecure: false
trustedIPs: []
hostPort: null
http3:
advertisedPort: null
enabled: false
middlewares: []
nodePort: null
port: 8443
protocol: TCP
proxyProtocol:
insecure: false
trustedIPs: []
targetPort: null
tls:
certResolver: ''
domains: []
enabled: true
options: ''
transport:
keepAliveMaxRequests: null
keepAliveMaxTime: null
lifeCycle:
graceTimeOut: null
requestAcceptGraceTimeout: null
respondingTimeouts:
idleTimeout: null
readTimeout: null
writeTimeout: null
priorityClassName: ''
providers:
file:
content: ''
enabled: false
watch: true
kubernetesCRD:
allowCrossNamespace: true
allowEmptyServices: true
allowExternalNameServices: false
enabled: true
ingressClass: ''
namespaces: []
nativeLBByDefault: false
kubernetesGateway:
enabled: false
experimentalChannel: false
labelselector: ''
namespaces: []
nativeLBByDefault: false
statusAddress:
hostname: ''
ip: ''
service:
enabled: true
name: ''
namespace: ''
kubernetesIngress:
allowEmptyServices: true
allowExternalNameServices: false
enabled: true
ingressClass: ''
namespaces: []
nativeLBByDefault: false
publishedService:
enabled: true
pathOverride: ''
rbac:
aggregateTo: []
enabled: true
namespaced: false
secretResourceNames: []
readinessProbe:
failureThreshold: 1
initialDelaySeconds: 2
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 2
resources: {}
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
readOnlyRootFilesystem: true
service:
additionalServices: {}
annotations: {}
annotationsTCP: {}
annotationsUDP: {}
enabled: true
externalIPs: []
labels: {}
loadBalancerSourceRanges: []
single: true
spec:
externalTrafficPolicy: Local
type: LoadBalancer
serviceAccount:
name: ''
serviceAccountAnnotations: {}
startupProbe: {}
tlsOptions: {}
tlsStore:
default:
defaultCertificate:
secretName: secret
tolerations: []
topologySpreadConstraints: []
tracing:
addInternals: false
capturedRequestHeaders: []
capturedResponseHeaders: []
otlp:
enabled: false
grpc:
enabled: false
endpoint: ''
insecure: false
tls:
ca: ''
cert: ''
insecureSkipVerify: false
key: ''
http:
enabled: false
endpoint: ''
headers: {}
tls:
ca: ''
cert: ''
insecureSkipVerify: false
key: ''
resourceAttributes: {}
safeQueryParams: []
sampleRate: null
serviceName: null
updateStrategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
type: RollingUpdate
versionOverride: ''
volumes: []
default:
defaultCertificate:
secretName: secret
此values文件有以下字段需要修改
additionalArguments
:添加以下列表,其中10.0.0.0/8
需要改成集群CNI的CIDR。deployment.kind
:必须将此字段设置为DaemonSet
,否则后续正确转发流量,导致访问异常。extraObjects
:使用以下YAML分别配置dashboard使用的PVC、SVC、Secret、Middleware和IngressRoute。- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: plugins-pvc spec: accessModes: - ReadWriteMany resources: requests: storage: 1Gi - apiVersion: v1 kind: Service metadata: name: traefik-api spec: ports: - name: traefik port: 8080 protocol: TCP targetPort: 8080 selector: app.kubernetes.io/instance: traefik-traefik app.kubernetes.io/name: traefik type: ClusterIP - apiVersion: v1 kind: Secret metadata: name: traefik-dashboard-auth-secret stringData: password: pass username: user type: kubernetes.io/basic-auth - apiVersion: traefik.io/v1alpha1 kind: Middleware metadata: name: traefik-dashboard-auth spec: basicAuth: secret: traefik-dashboard-auth-secret - apiVersion: traefik.io/v1alpha1 kind: IngressRoute metadata: name: traefik-dashboard namespace: traefik spec: entryPoints: - websecure routes: - kind: Rule match: Host(`traefik.example.com`) && PathPrefix(`/`) services: - kind: TraefikService name: api@internal
ingressClass.enable
:建议将此字段设置为false
,否则在配置IngressRoute时需要通过Annotation显式指定IngressClass。persistence.enabled
:必须将此字段设置为false
,否则DeamonSet将无法部署。default.defaultCertificate.secretName
:填写HTTPS默认使用的证书TLS,以Secret的形式提供。service.spec.externalTrafficPolicy
:必须将此字段配置为Local
,否则无法正确透传客户端 IP,导致IP白名单无法正常工作。service.type
:必须将此字段配置为LoadBalancer
,否则无法正确向MetalLB申请VIP。tlsStore.default.defaultCertificate.secretName
:填写HTTPS默认使用的证书TLS,以Secret的形式提供。ports.websecure.tls.enable
:必须将此字段设置为true,否则无法正常使用HTTPS协议。
部署
通过Helm命令行工具或者其他工具使用上文的values部署Traefik。
检查
检查Traefik部署的命名空间的所有Pod是否正常启动,确保连续15分钟各个容器不重启。若发生容器重启,可以查看容器日志,排除问题。
配置
为实现Gateway和Whitelist,需要进行如下配置
Middleware
需要配置Middleware实现IP白名单
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
name: whitelist
namespace: default
spec:
ipWhiteList:
sourceRange:
- 172.16.0.0/16
其中spec.ipWhiteList.sourceRange
以CIDR列表方式填入,不分顺序。
IngressRoute
需要配置IngressRoute暴露服务
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
name: nginx
namespace: default
spec:
entryPoints:
- websecure
routes:
- kind: Rule
match: Host(`nginx.example.com`)
middlewares:
- name: whiitelist
services:
- name: nginx
port: 80
scheme: http
其中需要注意的有:
spec.entryPoints
:配置为websecure
以启用HTTPS。spec.routes.match
:内部需要填写主机名。spec.routes.middlewares
:以列表方式填入需要使用的Middleware,与上文定义的middleware保持同名,此处是whitelist
。若有多个Middleware,请注意列表有序,即按顺序经过Middleware处理,最后到达服务。spec.routes.services.name
:需要访问的服务的后端服务名。spec.routes.services.port
:需要访问的服务的后端端口。spec.routes.services.scheme
:需要访问的服务的后端协议。
验证
至此处,所有的部署和配置工作已完成。以博主提供的YAML文件为例,可以在浏览器访问https://nginx.example.com
和https://traefik.example.com
,以验证部署及配置是否成功。