k8s部署gitlab

一、准备工作

1.1 规划

想要使用 SSH 协议，正好使用 metallb 负载均衡器，分配一个虚拟 ip，正好和宿主机上的 22 端口不冲突。需要修改 traefik 的端口。

需要提前调整 worker 节点的内容和 CPU 4C8G.

1.2 暴露22端口

1. 让 Traefik 的 LoadBalancer Service 多暴露一个 22 端口

vim traefik-values-with-ssh.yaml

# traefik-values-with-ssh.yaml
ingressClass:
  enabled: true
  isDefaultClass: true       # 设为默认 ingressClass，Ingress 不写 class 也走 Traefik

api:
  dashboard: true            # 打开 dashboard（下一步用我们自己的 IngressRoute 暴露它）

ingressRoute:
  dashboard:
    enabled: false           # 关掉自带的 dashboard 路由，避免和我们带 TLS 的那个冲突

# service 默认就是 LoadBalancer 类型，会自动向 MetalLB 要一个 IP
ports:                       # 新增的顶层块，顶格写
  ssh:
    port: 2222               # Traefik 容器内监听端口
    exposedPort: 22          # 对外(MetalLB VIP)端口
    protocol: TCP
    expose:
      default: true          # 新版 chart 写法

2. 更新 helm 的版本

helm upgrade traefik traefik/traefik -n traefik -f traefik-values-with-ssh.yaml

3. 验证

kubectl get svc -n traefik traefik

二、创建资源清单文件

1. 命名空间

vim namespace.yaml

# namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: gitlab

2. PVC卷，数据卷、配置卷

vim pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: gitlab-data-pvc
  namespace: gitlab
spec:
  storageClassName: nfs-client
  accessModes: ["ReadWriteOnce"]
  resources: { requests: { storage: 50Gi } }
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: gitlab-config-pvc
  namespace: gitlab
spec:
  storageClassName: nfs-client
  accessModes: ["ReadWriteOnce"]
  resources: { requests: { storage: 5Gi } }

3. Deployment

vim deployment.yaml

# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: gitlab
  namespace: gitlab
spec:
  selector:
    matchLabels: { app: gitlab }
  replicas: 1
  template:
    metadata:
      labels: { app: gitlab }
    spec:
      containers:
        - name: gitlab
          image: gitlab/gitlab-ce:18.4.3-ce.0
          env:
            - name: GITLAB_SKIP_UNMIGRATED_DATA_CHECK
              value: "true"
            - name: GITLAB_OMNIBUS_CONFIG
              value: |
                external_url 'https://gitlab.qx.lab'
                nginx['listen_port'] = 80
                nginx['listen_https'] = false
                gitlab_rails['gitlab_shell_ssh_port'] = 22
                prometheus['enable'] = false
                alertmanager['enable'] = false
                gitlab_rails['time_zone'] = 'Asia/Shanghai'
                gitlab_rails['gitlab_email_enabled'] = false
                gitlab_rails['smtp_enable'] = false
                nginx['worker_processes'] = 2
                postgresql['max_connections'] = 100
                postgresql['shared_buffers'] = "128MB"
          ports:
            - { containerPort: 80, name: http }
            - { containerPort: 22, name: ssh }
          readinessProbe:
            exec: { command: ["sh","-c","curl -s http://127.0.0.1/-/health"] }
          livenessProbe:
            exec: { command: ["sh","-c","curl -s http://127.0.0.1/-/health"] }
            timeoutSeconds: 5
            failureThreshold: 3
            periodSeconds: 60
          startupProbe:
            exec: { command: ["sh","-c","curl -s http://127.0.0.1/-/health"] }
            failureThreshold: 20
            periodSeconds: 120
          resources:
            requests: { memory: "4Gi", cpu: "2" }
            limits:   { memory: "8Gi", cpu: "4" }
          volumeMounts:
            - { name: data,   mountPath: /var/opt/gitlab }
            - { name: config, mountPath: /etc/gitlab }
            - { name: log,    mountPath: /var/log/gitlab }
            - { name: cache-volume, mountPath: /dev/shm }
      volumes:
        - { name: data,   persistentVolumeClaim: { claimName: gitlab-data-pvc } }
        - { name: config, persistentVolumeClaim: { claimName: gitlab-config-pvc } }
        - { name: log,    emptyDir: {} }
        - name: cache-volume
          emptyDir: { medium: Memory, sizeLimit: 256Mi }

4. Service

vim service.yaml

# service.yaml
apiVersion: v1
kind: Service
metadata:
  name: gitlab-svc
  namespace: gitlab
spec:
  selector: { app: gitlab }
  ports:
    - { port: 80, targetPort: 80, name: http }
    - { port: 22, targetPort: 22, name: ssh }

5. Ingress

vim ingress.yaml

# ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: gitlab
  namespace: gitlab
  annotations:
    cert-manager.io/cluster-issuer: qx-ca-issuer   # ★ 自动签
spec:
  ingressClassName: traefik
  tls:
    - hosts: [gitlab.qx.lab]
      secretName: gitlab-tls                        # ★ 自动创建
  rules:
    - host: gitlab.qx.lab
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: gitlab-svc
                port: { number: 80 }

6. IngressRouteTCP 透传

vim ingressroutetcp.yaml

# ingressroutetcp.yaml
apiVersion: traefik.io/v1alpha1
kind: IngressRouteTCP
metadata:
  name: gitlab-ssh
  namespace: gitlab
spec:
  entryPoints: [ssh]
  routes:
    - match: HostSNI(`*`)
      services:
        - name: gitlab-svc
          port: 22

三、创建资源

1. 批量创建资源

kubectl apply -f .

四、验证

https://gitlab.qx.lab

1. 获取初始化密码

kubectl exec -it deploy/gitlab -n gitlab -- cat /etc/gitlab/initial_root_password

root
wBY6ATYeQNdUbK6leKUJAtkfH8LAas9Lta1bsmE5sbo=