[2.23] Update k8s and etcd hashes + default to latest patch version (#10797)

* k8s: add hashes for 1.25.16, 1.26.12, 1.27.9

Make 1.27.9 default

* [etcd] add 3.5.10 hashes (#10566)

* Update etcd version for 1.26 and 1.27

---------

Co-authored-by: Mohamed Omar Zaian <mohamedzaian@gmail.com>

.ansible-lint

@@ -36,4 +36,3 @@ exclude_paths:
   # Generated files
   - tests/files/custom_cni/cilium.yaml
   - venv
-  - .github

.ansible-lint-ignore

@@ -5,4 +5,4 @@ roles/kubernetes/control-plane/defaults/main/main.yml jinja[spacing]
 roles/kubernetes/kubeadm/defaults/main.yml jinja[spacing]
 roles/kubernetes/node/defaults/main.yml jinja[spacing]
 roles/kubernetes/preinstall/defaults/main.yml jinja[spacing]
-roles/kubespray-defaults/defaults/main/main.yml jinja[spacing]
+roles/kubespray-defaults/defaults/main.yaml jinja[spacing]

.gitattributes

@@ -1 +0,0 @@
-docs/_sidebar.md linguist-generated=true

.github/ISSUE_TEMPLATE/bug-report.md

@@ -0,0 +1,44 @@
+---
+name: Bug Report
+about: Report a bug encountered while operating Kubernetes
+labels: kind/bug
+
+---
+<!--
+Please, be ready for followup questions, and please respond in a timely
+manner.  If we can't reproduce a bug or think a feature already exists, we
+might close your issue.  If we're wrong, PLEASE feel free to reopen it and
+explain why.
+-->
+
+**Environment**:
+- **Cloud provider or hardware configuration:**
+
+- **OS (`printf "$(uname -srm)\n$(cat /etc/os-release)\n"`):**
+
+- **Version of Ansible** (`ansible --version`):
+
+- **Version of Python** (`python --version`):
+
+
+**Kubespray version (commit) (`git rev-parse --short HEAD`):**
+
+
+**Network plugin used**:
+
+
+**Full inventory with variables (`ansible -i inventory/sample/inventory.ini all -m debug -a "var=hostvars[inventory_hostname]"`):**
+<!-- We recommend using snippets services like https://gist.github.com/ etc. -->
+
+**Command used to invoke ansible**:
+
+
+**Output of ansible run**:
+<!-- We recommend using snippets services like https://gist.github.com/ etc. -->
+
+**Anything else do we need to know**:
+<!-- By running scripts/collect-info.yaml you can get a lot of useful informations.
+Script can be started by:
+ansible-playbook -i <inventory_file_path> -u <ssh_user> -e ansible_ssh_user=<ssh_user> -b --become-user=root -e dir=`pwd` scripts/collect-info.yaml
+(If you using CoreOS remember to add '-e ansible_python_interpreter=/opt/bin/python').
+After running this command you can find logs in `pwd`/logs.tar.gz. You can even upload somewhere entire file and paste link here.-->

.github/ISSUE_TEMPLATE/bug-report.yaml

@@ -1,124 +0,0 @@
----
-name: Bug Report
-description: Report a bug encountered while using Kubespray
-labels: kind/bug
-body:
-  - type: markdown
-    attributes:
-      value: |
-        Please, be ready for followup questions, and please respond in a timely
-        manner.  If we can't reproduce a bug or think a feature already exists, we
-        might close your issue.  If we're wrong, PLEASE feel free to reopen it and
-        explain why.
-  - type: textarea
-    id: problem
-    attributes:
-      label: What happened?
-      description: |
-        Please provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner.
-    validations:
-      required: true
-  - type: textarea
-    id: expected
-    attributes:
-      label: What did you expect to happen?
-    validations:
-      required: true
-
-  - type: textarea
-    id: repro
-    attributes:
-      label: How can we reproduce it (as minimally and precisely as possible)?
-    validations:
-      required: true
-
-  - type: markdown
-    attributes:
-      value: '### Environment'
-
-  - type: textarea
-    id: os
-    attributes:
-      label: OS
-      placeholder: 'printf "$(uname -srm)\n$(cat /etc/os-release)\n"'
-    validations:
-      required: true
-
-  - type: textarea
-    id: ansible_version
-    attributes:
-      label: Version of Ansible
-      placeholder: 'ansible --version'
-    validations:
-      required: true
-
-  - type: input
-    id: python_version
-    attributes:
-      label: Version of Python
-      placeholder: 'python --version'
-    validations:
-      required: true
-
-  - type: input
-    id: kubespray_version
-    attributes:
-      label: Version of Kubespray (commit)
-      placeholder: 'git rev-parse --short HEAD'
-    validations:
-      required: true
-
-  - type: dropdown
-    id: network_plugin
-    attributes:
-      label: Network plugin used
-      options:
-        - calico
-        - cilium
-        - cni
-        - custom_cni
-        - flannel
-        - kube-ovn
-        - kube-router
-        - macvlan
-        - meta
-        - multus
-        - ovn4nfv
-        - weave
-    validations:
-      required: true
-
-  - type: textarea
-    id: inventory
-    attributes:
-      label: Full inventory with variables
-      placeholder: 'ansible -i inventory/sample/inventory.ini all -m debug -a "var=hostvars[inventory_hostname]"'
-      description: We recommend using snippets services like https://gist.github.com/ etc.
-    validations:
-      required: true
-
-  - type: input
-    id: ansible_command
-    attributes:
-      label: Command used to invoke ansible
-    validations:
-      required: true
-
-  - type: textarea
-    id: ansible_output
-    attributes:
-      label: Output of ansible run
-      description: We recommend using snippets services like https://gist.github.com/ etc.
-    validations:
-      required: true
-
-  - type: textarea
-    id: anything_else
-    attributes:
-      label: Anything else we need to know
-      description: |
-        By running scripts/collect-info.yaml you can get a lot of useful informations.
-        Script can be started by:
-        ansible-playbook -i <inventory_file_path> -u <ssh_user> -e ansible_ssh_user=<ssh_user> -b --become-user=root -e dir=`pwd` scripts/collect-info.yaml
-        (If you using CoreOS remember to add '-e ansible_python_interpreter=/opt/bin/python').
-        After running this command you can find logs in `pwd`/logs.tar.gz. You can even upload somewhere entire file and paste link here

.github/ISSUE_TEMPLATE/config.yml

@@ -1,5 +0,0 @@
----
-contact_links:
-  - name: Support Request
-    url: https://kubernetes.slack.com/channels/kubespray
-    about: Support request or question relating to Kubernetes

.github/ISSUE_TEMPLATE/enhancement.md

@@ -0,0 +1,11 @@
+---
+name: Enhancement Request
+about: Suggest an enhancement to the Kubespray project
+labels: kind/feature
+
+---
+<!-- Please only use this template for submitting enhancement requests -->
+
+**What would you like to be added**:
+
+**Why is this needed**:

.github/ISSUE_TEMPLATE/enhancement.yaml

@@ -1,20 +0,0 @@
----
-name: Enhancement Request
-description: Suggest an enhancement to the Kubespray project
-labels: kind/feature
-body:
-  - type: markdown
-    attributes:
-      value: Please only use this template for submitting enhancement requests
-  - type: textarea
-    id: what
-    attributes:
-      label: What would you like to be added
-    validations:
-      required: true
-  - type: textarea
-    id: why
-    attributes:
-      label: Why is this needed
-    validations:
-      required: true

.github/ISSUE_TEMPLATE/failing-test.md

@@ -0,0 +1,20 @@
+---
+name: Failing Test
+about: Report test failures in Kubespray CI jobs
+labels: kind/failing-test
+
+---
+
+<!-- Please only use this template for submitting reports about failing tests in Kubespray CI jobs -->
+
+**Which jobs are failing**:
+
+**Which test(s) are failing**:
+
+**Since when has it been failing**:
+
+**Testgrid link**:
+
+**Reason for failure**:
+
+**Anything else we need to know**:

.github/ISSUE_TEMPLATE/failing-test.yaml

@@ -1,41 +0,0 @@
----
-name: Failing Test
-description: Report test failures in Kubespray CI jobs
-labels: kind/failing-test
-body:
-  - type: markdown
-    attributes:
-      value: Please only use this template for submitting reports about failing tests in Kubespray CI jobs
-  - type: textarea
-    id: failing_jobs
-    attributes:
-      label: Which jobs are failing ?
-    validations:
-      required: true
-
-  - type: textarea
-    id: failing_tests
-    attributes:
-      label: Which tests are failing ?
-    validations:
-      required: true
-
-  - type: input
-    id: since_when
-    attributes:
-      label: Since when has it been failing ?
-    validations:
-      required: true
-
-  - type: textarea
-    id: failure_reason
-    attributes:
-      label: Reason for failure
-      description: If you don't know and have no guess, just put "Unknown"
-    validations:
-      required: true
-
-  - type: textarea
-    id: anything_else
-    attributes:
-      label: Anything else we need to know

.github/ISSUE_TEMPLATE/support.md

@@ -0,0 +1,18 @@
+---
+name: Support Request
+about: Support request or question relating to Kubespray
+labels: kind/support
+
+---
+
+<!--
+STOP -- PLEASE READ!
+
+GitHub is not the right place for support requests.
+
+If you're looking for help, check [Stack Overflow](https://stackoverflow.com/questions/tagged/kubespray) and the [troubleshooting guide](https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/).
+
+You can also post your question on the [Kubernetes Slack](http://slack.k8s.io/) or the [Discuss Kubernetes](https://discuss.kubernetes.io/) forum.
+
+If the matter is security related, please disclose it privately via https://kubernetes.io/security/.
+-->

.github/dependabot.yml

@@ -1,7 +0,0 @@
-version: 2
-updates:
-  - package-ecosystem: "pip"
-    directory: "/"
-    schedule:
-      interval: "weekly"
-    labels: [ "dependencies" ]

.gitignore

@@ -3,8 +3,6 @@
 **/vagrant_ansible_inventory
 *.iml
 temp
-contrib/offline/container-images
-contrib/offline/container-images.tar.gz
 contrib/offline/offline-files
 contrib/offline/offline-files.tar.gz
 .idea

.gitlab-ci.yml

@@ -9,7 +9,7 @@ stages:
   - deploy-special
 
 variables:
-  KUBESPRAY_VERSION: v2.25.0
+  KUBESPRAY_VERSION: v2.22.1
   FAILFASTCI_NAMESPACE: 'kargo-ci'
   GITLAB_REPOSITORY: 'kargo-ci/kubernetes-sigs-kubespray'
   ANSIBLE_FORCE_COLOR: "true"

.gitlab-ci/lint.yml

@@ -27,14 +27,6 @@ ansible-lint:
     - ansible-lint -v
   except: ['triggers', 'master']
 
-jinja-syntax-check:
-  extends: .job
-  stage: unit-tests
-  tags: [light]
-  script:
-    - "find -name '*.j2' -exec tests/scripts/check-templates.py {} +"
-  except: ['triggers', 'master']
-
 syntax-check:
   extends: .job
   stage: unit-tests
@@ -89,14 +81,6 @@ markdownlint:
   script:
     - markdownlint $(find . -name '*.md' | grep -vF './.git') --ignore docs/_sidebar.md --ignore contrib/dind/README.md
 
-generate-sidebar:
-  extends: .job
-  stage: unit-tests
-  tags: [light]
-  script:
-    - scripts/gen_docs_sidebar.sh
-    - git diff --exit-code
-
 check-readme-versions:
   stage: unit-tests
   tags: [light]

.gitlab-ci/molecule.yml

@@ -61,23 +61,23 @@ molecule_cri-o:
 molecule_kata:
   extends: .molecule
   stage: deploy-part3
+  allow_failure: true
   script:
     - ./tests/scripts/molecule_run.sh -i container-engine/kata-containers
-  when: manual
+  when: on_success
-# FIXME: this test is broken (perma-failing)
 
 molecule_gvisor:
   extends: .molecule
   stage: deploy-part3
+  allow_failure: true
   script:
     - ./tests/scripts/molecule_run.sh -i container-engine/gvisor
-  when: manual
+  when: on_success
-# FIXME: this test is broken (perma-failing)
 
 molecule_youki:
   extends: .molecule
   stage: deploy-part3
+  allow_failure: true
   script:
     - ./tests/scripts/molecule_run.sh -i container-engine/youki
-  when: manual
+  when: on_success
-# FIXME: this test is broken (perma-failing)

.gitlab-ci/packet.yml

@@ -61,17 +61,7 @@ packet_ubuntu22-calico-all-in-one:
   extends: .packet_pr
   when: on_success
 
-packet_ubuntu24-all-in-one-docker:
+packet_ubuntu22-calico-etcd-datastore:
-  stage: deploy-part2
-  extends: .packet_pr
-  when: on_success
-
-packet_ubuntu24-calico-all-in-one:
-  stage: deploy-part2
-  extends: .packet_pr
-  when: on_success
-
-packet_ubuntu24-calico-etcd-datastore:
   stage: deploy-part2
   extends: .packet_pr
   when: on_success
@@ -184,11 +174,6 @@ packet_almalinux8-docker:
   extends: .packet_pr
   when: on_success
 
-packet_amazon-linux-2-all-in-one:
-  stage: deploy-part2
-  extends: .packet_pr
-  when: on_success
-
 packet_fedora38-docker-weave:
   stage: deploy-part2
   extends: .packet_pr
@@ -198,7 +183,7 @@ packet_fedora38-docker-weave:
 packet_opensuse-docker-cilium:
   stage: deploy-part2
   extends: .packet_pr
-  when: on_success
+  when: manual
 
 # ### MANUAL JOBS
 
@@ -255,6 +240,11 @@ packet_fedora37-calico-swap-selinux:
   extends: .packet_pr
   when: manual
 
+packet_amazon-linux-2-all-in-one:
+  stage: deploy-part2
+  extends: .packet_pr
+  when: manual
+
 packet_almalinux8-calico-nodelocaldns-secondary:
   stage: deploy-part2
   extends: .packet_pr
@@ -275,11 +265,6 @@ packet_debian11-kubelet-csr-approver:
   extends: .packet_pr
   when: manual
 
-packet_debian12-custom-cni-helm:
-  stage: deploy-part2
-  extends: .packet_pr
-  when: manual
-
 # ### PR JOBS PART3
 # Long jobs (45min+)
 

.gitlab-ci/vagrant.yml

@@ -18,12 +18,12 @@
     - ./tests/scripts/testcases_run.sh
   after_script:
     - chronic ./tests/scripts/testcases_cleanup.sh
+  allow_failure: true
 
 vagrant_ubuntu20-calico-dual-stack:
   stage: deploy-part2
   extends: .vagrant
-  when: manual
+  when: on_success
-# FIXME: this test if broken (perma-failing)
 
 vagrant_ubuntu20-weave-medium:
   stage: deploy-part2
@@ -55,8 +55,7 @@ vagrant_ubuntu20-kube-router-svc-proxy:
 vagrant_fedora37-kube-router:
   stage: deploy-part2
   extends: .vagrant
-  when: manual
+  when: on_success
-# FIXME: this test if broken (perma-failing)
 
 vagrant_centos7-kube-router:
   stage: deploy-part2

.pre-commit-config.yaml

@@ -1,5 +1,6 @@
 ---
 repos:
+
   - repo: https://github.com/pre-commit/pre-commit-hooks
     rev: v3.4.0
     hooks:
@@ -24,14 +25,14 @@ repos:
     rev: v0.11.0
     hooks:
       - id: markdownlint
-        args: [-r, "~MD013,~MD029"]
+        args: [ -r, "~MD013,~MD029" ]
         exclude: "^.git"
 
   - repo: https://github.com/jumanjihouse/pre-commit-hooks
     rev: 3.0.0
     hooks:
       - id: shellcheck
-        args: [--severity, "error"]
+        args: [ --severity, "error" ]
         exclude: "^.git"
         files: "\\.sh$"
 
@@ -63,23 +64,8 @@ repos:
         language: script
         pass_filenames: false
 
-      - id: generate-docs-sidebar
-        name: generate-docs-sidebar
-        entry: scripts/gen_docs_sidebar.sh
-        language: script
-        pass_filenames: false
-
       - id: ci-matrix
         name: ci-matrix
         entry: tests/scripts/md-table/test.sh
         language: script
         pass_filenames: false
-
-      - id: jinja-syntax-check
-        name: jinja-syntax-check
-        entry: tests/scripts/check-templates.py
-        language: python
-        types:
-          - jinja
-        additional_dependencies:
-          - Jinja2

.yamllint

@@ -3,7 +3,6 @@ extends: default
 
 ignore: |
   .git/
-  .github/
   # Generated file
   tests/files/custom_cni/cilium.yaml
 

Dockerfile

@@ -1,8 +1,5 @@
-# syntax=docker/dockerfile:1
-
 # Use imutable image tags rather than mutable tags (like ubuntu:22.04)
-FROM ubuntu:22.04@sha256:149d67e29f765f4db62aa52161009e99e389544e25a8f43c8c89d4a445a7ca37
+FROM ubuntu:jammy-20230308
-
 # Some tools like yamllint need this
 # Pip needs this as well at the moment to install ansible
 # (and potentially other packages)
@@ -10,37 +7,7 @@ FROM ubuntu:22.04@sha256:149d67e29f765f4db62aa52161009e99e389544e25a8f43c8c89d4a
 ENV LANG=C.UTF-8 \
     DEBIAN_FRONTEND=noninteractive \
     PYTHONDONTWRITEBYTECODE=1
-
 WORKDIR /kubespray
-
-# hadolint ignore=DL3008
-RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
-    apt-get update -q \
-    && apt-get install -yq --no-install-recommends \
-    curl \
-    python3 \
-    python3-pip \
-    sshpass \
-    vim \
-    rsync \
-    openssh-client \
-    && apt-get clean \
-    && rm -rf /var/lib/apt/lists/* /var/log/*
-
-RUN --mount=type=bind,source=requirements.txt,target=requirements.txt \
-    --mount=type=cache,sharing=locked,id=pipcache,mode=0777,target=/root/.cache/pip \
-    pip install --no-compile --no-cache-dir -r requirements.txt \
-    && find /usr -type d -name '*__pycache__' -prune -exec rm -rf {} \;
-
-SHELL ["/bin/bash", "-o", "pipefail", "-c"]
-
-RUN --mount=type=bind,source=roles/kubespray-defaults/defaults/main/main.yml,target=roles/kubespray-defaults/defaults/main/main.yml \
-    KUBE_VERSION=$(sed -n 's/^kube_version: //p' roles/kubespray-defaults/defaults/main/main.yml) \
-    OS_ARCHITECTURE=$(dpkg --print-architecture) \
-    && curl -L "https://dl.k8s.io/release/${KUBE_VERSION}/bin/linux/${OS_ARCHITECTURE}/kubectl" -o /usr/local/bin/kubectl \
-    && echo "$(curl -L "https://dl.k8s.io/release/${KUBE_VERSION}/bin/linux/${OS_ARCHITECTURE}/kubectl.sha256")" /usr/local/bin/kubectl | sha256sum --check \
-    && chmod a+x /usr/local/bin/kubectl
-
 COPY *.yml ./
 COPY *.cfg ./
 COPY roles ./roles
@@ -50,3 +17,29 @@ COPY library ./library
 COPY extra_playbooks ./extra_playbooks
 COPY playbooks ./playbooks
 COPY plugins ./plugins
+
+RUN apt update -q \
+    && apt install -yq --no-install-recommends \
+       curl \
+       python3 \
+       python3-pip \
+       sshpass \
+       vim \
+       rsync \
+       openssh-client \
+    && pip install --no-compile --no-cache-dir \
+       ansible==7.6.0 \
+       ansible-core==2.14.6 \
+       cryptography==41.0.1 \
+       jinja2==3.1.2 \
+       netaddr==0.8.0 \
+       jmespath==1.0.1 \
+       MarkupSafe==2.1.3 \
+       ruamel.yaml==0.17.21 \
+       passlib==1.7.4 \
+    && KUBE_VERSION=$(sed -n 's/^kube_version: //p' roles/kubespray-defaults/defaults/main.yaml) \
+    && curl -L https://dl.k8s.io/release/$KUBE_VERSION/bin/linux/$(dpkg --print-architecture)/kubectl -o /usr/local/bin/kubectl \
+    && echo $(curl -L https://dl.k8s.io/release/$KUBE_VERSION/bin/linux/$(dpkg --print-architecture)/kubectl.sha256) /usr/local/bin/kubectl | sha256sum --check \
+    && chmod a+x /usr/local/bin/kubectl \
+    && rm -rf /var/lib/apt/lists/* /var/log/* \
+    && find /usr -type d -name '*__pycache__' -prune -exec rm -rf {} \;

OWNERS_ALIASES

@@ -1,24 +1,31 @@
 aliases:
   kubespray-approvers:
-    - cristicalin
-    - floryut
-    - liupeng0518
-    - mzaian
-    - oomichi
-    - yankay
-  kubespray-reviewers:
-    - cyclinder
-    - erikjiang
-    - mrfreezeex
-    - mzaian
-    - vannten
-    - yankay
-  kubespray-emeritus_approvers:
-    - ant31
-    - atoms
-    - chadswen
-    - luckysb
     - mattymo
+    - chadswen
+    - mirwan
     - miouge1
+    - luckysb
+    - floryut
+    - oomichi
+    - cristicalin
+    - liupeng0518
+    - yankay
+    - mzaian
+  kubespray-reviewers:
+    - holmsten
+    - bozzo
+    - eppo
+    - oomichi
+    - jayonlau
+    - cristicalin
+    - liupeng0518
+    - yankay
+    - cyclinder
+    - mzaian
+    - mrfreezeex
+    - erikjiang
+  kubespray-emeritus_approvers:
     - riverzhang
+    - atoms
+    - ant31
     - woopstar

README.md

@@ -1,5 +1,3 @@
-TEST CI
-
 # Deploy a Production Ready Kubernetes Cluster
 
 ![Kubernetes Logo](https://raw.githubusercontent.com/kubernetes-sigs/kubespray/master/docs/img/kubernetes-logo.png)
@@ -7,7 +5,7 @@ TEST CI
 If you have questions, check the documentation at [kubespray.io](https://kubespray.io) and join us on the [kubernetes slack](https://kubernetes.slack.com), channel **\#kubespray**.
 You can get your invite [here](http://slack.k8s.io/)
 
-- Can be deployed on **[AWS](docs/cloud_providers/aws.md), GCE, [Azure](docs/cloud_providers/azure.md), [OpenStack](docs/cloud_providers/openstack.md), [vSphere](docs/cloud_providers/vsphere.md), [Equinix Metal](docs/cloud_providers/equinix-metal.md) (bare metal), Oracle Cloud Infrastructure (Experimental), or Baremetal**
+- Can be deployed on **[AWS](docs/aws.md), GCE, [Azure](docs/azure.md), [OpenStack](docs/openstack.md), [vSphere](docs/vsphere.md), [Equinix Metal](docs/equinix-metal.md) (bare metal), Oracle Cloud Infrastructure (Experimental), or Baremetal**
 - **Highly available** cluster
 - **Composable** (Choice of the network plugin for instance)
 - Supports most popular **Linux distributions**
@@ -21,7 +19,7 @@ Below are several ways to use Kubespray to deploy a Kubernetes cluster.
 
 #### Usage
 
-Install Ansible according to [Ansible installation guide](/docs/ansible/ansible.md#installing-ansible)
+Install Ansible according to [Ansible installation guide](/docs/ansible.md#installing-ansible)
 then run the following steps:
 
 ```ShellSession
@@ -77,18 +75,18 @@ You will then need to use [bind mounts](https://docs.docker.com/storage/bind-mou
 to access the inventory and SSH key in the container, like this:
 
 ```ShellSession
-git checkout v2.25.0
+git checkout v2.22.1
-docker pull quay.io/kubespray/kubespray:v2.25.0
+docker pull quay.io/kubespray/kubespray:v2.22.1
 docker run --rm -it --mount type=bind,source="$(pwd)"/inventory/sample,dst=/inventory \
   --mount type=bind,source="${HOME}"/.ssh/id_rsa,dst=/root/.ssh/id_rsa \
-  quay.io/kubespray/kubespray:v2.25.0 bash
+  quay.io/kubespray/kubespray:v2.22.1 bash
 # Inside the container you may now run the kubespray playbooks:
 ansible-playbook -i /inventory/inventory.ini --private-key /root/.ssh/id_rsa cluster.yml
 ```
 
 #### Collection
 
-See [here](docs/ansible/ansible_collection.md) if you wish to use this repository as an Ansible collection
+See [here](docs/ansible_collection.md) if you wish to use this repository as an Ansible collection
 
 ### Vagrant
 
@@ -101,7 +99,7 @@ python -V && pip -V
 
 If this returns the version of the software, you're good to go. If not, download and install Python from here <https://www.python.org/downloads/source/>
 
-Install Ansible according to [Ansible installation guide](/docs/ansible/ansible.md#installing-ansible)
+Install Ansible according to [Ansible installation guide](/docs/ansible.md#installing-ansible)
 then run the following step:
 
 ```ShellSession
@@ -111,79 +109,80 @@ vagrant up
 ## Documents
 
 - [Requirements](#requirements)
-- [Kubespray vs ...](docs/getting_started/comparisons.md)
+- [Kubespray vs ...](docs/comparisons.md)
-- [Getting started](docs/getting_started/getting-started.md)
+- [Getting started](docs/getting-started.md)
-- [Setting up your first cluster](docs/getting_started/setting-up-your-first-cluster.md)
+- [Setting up your first cluster](docs/setting-up-your-first-cluster.md)
-- [Ansible inventory and tags](docs/ansible/ansible.md)
+- [Ansible inventory and tags](docs/ansible.md)
-- [Integration with existing ansible repo](docs/operations/integration.md)
+- [Integration with existing ansible repo](docs/integration.md)
-- [Deployment data variables](docs/ansible/vars.md)
+- [Deployment data variables](docs/vars.md)
-- [DNS stack](docs/advanced/dns-stack.md)
+- [DNS stack](docs/dns-stack.md)
-- [HA mode](docs/operations/ha-mode.md)
+- [HA mode](docs/ha-mode.md)
 - [Network plugins](#network-plugins)
-- [Vagrant install](docs/developers/vagrant.md)
+- [Vagrant install](docs/vagrant.md)
-- [Flatcar Container Linux bootstrap](docs/operating_systems/flatcar.md)
+- [Flatcar Container Linux bootstrap](docs/flatcar.md)
-- [Fedora CoreOS bootstrap](docs/operating_systems/fcos.md)
+- [Fedora CoreOS bootstrap](docs/fcos.md)
-- [openSUSE setup](docs/operating_systems/opensuse.md)
+- [Debian Jessie setup](docs/debian.md)
-- [Downloaded artifacts](docs/advanced/downloads.md)
+- [openSUSE setup](docs/opensuse.md)
-- [Cloud providers](docs/cloud_providers/cloud.md)
+- [Downloaded artifacts](docs/downloads.md)
-- [OpenStack](docs/cloud_providers/openstack.md)
+- [Cloud providers](docs/cloud.md)
-- [AWS](docs/cloud_providers/aws.md)
+- [OpenStack](docs/openstack.md)
-- [Azure](docs/cloud_providers/azure.md)
+- [AWS](docs/aws.md)
-- [vSphere](docs/cloud_providers/vsphere.md)
+- [Azure](docs/azure.md)
-- [Equinix Metal](docs/cloud_providers/equinix-metal.md)
+- [vSphere](docs/vsphere.md)
-- [Large deployments](docs/operations/large-deployments.md)
+- [Equinix Metal](docs/equinix-metal.md)
-- [Adding/replacing a node](docs/operations/nodes.md)
+- [Large deployments](docs/large-deployments.md)
-- [Upgrades basics](docs/operations/upgrades.md)
+- [Adding/replacing a node](docs/nodes.md)
-- [Air-Gap installation](docs/operations/offline-environment.md)
+- [Upgrades basics](docs/upgrades.md)
-- [NTP](docs/advanced/ntp.md)
+- [Air-Gap installation](docs/offline-environment.md)
-- [Hardening](docs/operations/hardening.md)
+- [NTP](docs/ntp.md)
-- [Mirror](docs/operations/mirror.md)
+- [Hardening](docs/hardening.md)
-- [Roadmap](docs/roadmap/roadmap.md)
+- [Mirror](docs/mirror.md)
+- [Roadmap](docs/roadmap.md)
 
 ## Supported Linux Distributions
 
 - **Flatcar Container Linux by Kinvolk**
 - **Debian** Bookworm, Bullseye, Buster
 - **Ubuntu** 20.04, 22.04
-- **CentOS/RHEL** 7, [8, 9](docs/operating_systems/centos.md#centos-8)
+- **CentOS/RHEL** 7, [8, 9](docs/centos.md#centos-8)
 - **Fedora** 37, 38
-- **Fedora CoreOS** (see [fcos Note](docs/operating_systems/fcos.md))
+- **Fedora CoreOS** (see [fcos Note](docs/fcos.md))
 - **openSUSE** Leap 15.x/Tumbleweed
-- **Oracle Linux** 7, [8, 9](docs/operating_systems/centos.md#centos-8)
+- **Oracle Linux** 7, [8, 9](docs/centos.md#centos-8)
-- **Alma Linux** [8, 9](docs/operating_systems/centos.md#centos-8)
+- **Alma Linux** [8, 9](docs/centos.md#centos-8)
-- **Rocky Linux** [8, 9](docs/operating_systems/centos.md#centos-8)
+- **Rocky Linux** [8, 9](docs/centos.md#centos-8)
-- **Kylin Linux Advanced Server V10** (experimental: see [kylin linux notes](docs/operating_systems/kylinlinux.md))
+- **Kylin Linux Advanced Server V10** (experimental: see [kylin linux notes](docs/kylinlinux.md))
-- **Amazon Linux 2** (experimental: see [amazon linux notes](docs/operating_systems/amazonlinux.md))
+- **Amazon Linux 2** (experimental: see [amazon linux notes](docs/amazonlinux.md))
-- **UOS Linux** (experimental: see [uos linux notes](docs/operating_systems/uoslinux.md))
+- **UOS Linux** (experimental: see [uos linux notes](docs/uoslinux.md))
-- **openEuler** (experimental: see [openEuler notes](docs/operating_systems/openeuler.md))
+- **openEuler** (experimental: see [openEuler notes](docs/openeuler.md))
 
 Note: Upstart/SysV init based OS types are not supported.
 
 ## Supported Components
 
 - Core
-  - [kubernetes](https://github.com/kubernetes/kubernetes) v1.29.5
+  - [kubernetes](https://github.com/kubernetes/kubernetes) v1.27.9
-  - [etcd](https://github.com/etcd-io/etcd) v3.5.12
+  - [etcd](https://github.com/etcd-io/etcd) v3.5.10
-  - [docker](https://www.docker.com/) v24.0 (see [Note](#container-runtime-notes))
+  - [docker](https://www.docker.com/) v20.10 (see note)
-  - [containerd](https://containerd.io/) v1.7.16
+  - [containerd](https://containerd.io/) v1.7.5
-  - [cri-o](http://cri-o.io/) v1.29.1 (experimental: see [CRI-O Note](docs/CRI/cri-o.md). Only on fedora, ubuntu and centos based OS)
+  - [cri-o](http://cri-o.io/) v1.27 (experimental: see [CRI-O Note](docs/cri-o.md). Only on fedora, ubuntu and centos based OS)
 - Network Plugin
   - [cni-plugins](https://github.com/containernetworking/plugins) v1.2.0
-  - [calico](https://github.com/projectcalico/calico) v3.27.3
+  - [calico](https://github.com/projectcalico/calico) v3.25.2
-  - [cilium](https://github.com/cilium/cilium) v1.15.4
+  - [cilium](https://github.com/cilium/cilium) v1.13.4
   - [flannel](https://github.com/flannel-io/flannel) v0.22.0
   - [kube-ovn](https://github.com/alauda/kube-ovn) v1.11.5
-  - [kube-router](https://github.com/cloudnativelabs/kube-router) v2.0.0
+  - [kube-router](https://github.com/cloudnativelabs/kube-router) v1.5.1
   - [multus](https://github.com/k8snetworkplumbingwg/multus-cni) v3.8
   - [weave](https://github.com/weaveworks/weave) v2.8.1
-  - [kube-vip](https://github.com/kube-vip/kube-vip) v0.8.0
+  - [kube-vip](https://github.com/kube-vip/kube-vip) v0.5.12
 - Application
-  - [cert-manager](https://github.com/jetstack/cert-manager) v1.13.2
+  - [cert-manager](https://github.com/jetstack/cert-manager) v1.11.1
-  - [coredns](https://github.com/coredns/coredns) v1.11.1
+  - [coredns](https://github.com/coredns/coredns) v1.10.1
-  - [ingress-nginx](https://github.com/kubernetes/ingress-nginx) v1.10.1
+  - [ingress-nginx](https://github.com/kubernetes/ingress-nginx) v1.8.1
   - [krew](https://github.com/kubernetes-sigs/krew) v0.4.4
-  - [argocd](https://argoproj.github.io/) v2.11.0
+  - [argocd](https://argoproj.github.io/) v2.8.0
-  - [helm](https://helm.sh/) v3.14.2
+  - [helm](https://helm.sh/) v3.12.3
   - [metallb](https://metallb.universe.tf/)  v0.13.9
   - [registry](https://github.com/distribution/distribution) v2.8.1
 - Storage Plugin
@@ -191,22 +190,21 @@ Note: Upstart/SysV init based OS types are not supported.
   - [rbd-provisioner](https://github.com/kubernetes-incubator/external-storage) v2.1.1-k8s1.11
   - [aws-ebs-csi-plugin](https://github.com/kubernetes-sigs/aws-ebs-csi-driver) v0.5.0
   - [azure-csi-plugin](https://github.com/kubernetes-sigs/azuredisk-csi-driver) v1.10.0
-  - [cinder-csi-plugin](https://github.com/kubernetes/cloud-provider-openstack/blob/master/docs/cinder-csi-plugin/using-cinder-csi-plugin.md) v1.29.0
+  - [cinder-csi-plugin](https://github.com/kubernetes/cloud-provider-openstack/blob/master/docs/cinder-csi-plugin/using-cinder-csi-plugin.md) v1.22.0
   - [gcp-pd-csi-plugin](https://github.com/kubernetes-sigs/gcp-compute-persistent-disk-csi-driver) v1.9.2
   - [local-path-provisioner](https://github.com/rancher/local-path-provisioner) v0.0.24
   - [local-volume-provisioner](https://github.com/kubernetes-sigs/sig-storage-local-static-provisioner) v2.5.0
-  - [node-feature-discovery](https://github.com/kubernetes-sigs/node-feature-discovery) v0.14.2
 
 ## Container Runtime Notes
 
-- Supported Docker versions are 18.09, 19.03, 20.10, 23.0 and 24.0. The *recommended* Docker version is 24.0. `Kubelet` might break on docker's non-standard version numbering (it no longer uses semantic versioning). To ensure auto-updates don't break your cluster look into e.g. the YUM  ``versionlock`` plugin or ``apt pin``).
+- Supported Docker versions are 18.09, 19.03, 20.10, 23.0 and 24.0. The *recommended* Docker version is 20.10 (except on Debian bookworm which without supporting for 20.10 and below any more). `Kubelet` might break on docker's non-standard version numbering (it no longer uses semantic versioning). To ensure auto-updates don't break your cluster look into e.g. the YUM  ``versionlock`` plugin or ``apt pin``).
 - The cri-o version should be aligned with the respective kubernetes version (i.e. kube_version=1.20.x, crio_version=1.20)
 
 ## Requirements
 
-- **Minimum required version of Kubernetes is v1.28**
+- **Minimum required version of Kubernetes is v1.25**
 - **Ansible v2.14+, Jinja 2.11+ and python-netaddr is installed on the machine that will run Ansible commands**
-- The target servers must have **access to the Internet** in order to pull docker images. Otherwise, additional configuration is required (See [Offline Environment](docs/operations/offline-environment.md))
+- The target servers must have **access to the Internet** in order to pull docker images. Otherwise, additional configuration is required (See [Offline Environment](docs/offline-environment.md))
 - The target servers are configured to allow **IPv4 forwarding**.
 - If using IPv6 for pods and services, the target servers are configured to allow **IPv6 forwarding**.
 - The **firewalls are not managed**, you'll need to implement your own rules the way you used to.
@@ -227,7 +225,7 @@ These limits are safeguarded by Kubespray. Actual requirements for your workload
 
 You can choose among ten network plugins. (default: `calico`, except Vagrant uses `flannel`)
 
-- [flannel](docs/CNI/flannel.md): gre/vxlan (layer 2) networking.
+- [flannel](docs/flannel.md): gre/vxlan (layer 2) networking.
 
 - [Calico](https://docs.tigera.io/calico/latest/about/) is a networking and network policy provider. Calico supports a flexible set of networking options
     designed to give you the most efficient networking across a range of situations, including non-overlay
@@ -236,32 +234,32 @@ You can choose among ten network plugins. (default: `calico`, except Vagrant use
 
 - [cilium](http://docs.cilium.io/en/latest/): layer 3/4 networking (as well as layer 7 to protect and secure application protocols), supports dynamic insertion of BPF bytecode into the Linux kernel to implement security services, networking and visibility logic.
 
-- [weave](docs/CNI/weave.md): Weave is a lightweight container overlay network that doesn't require an external K/V database cluster.
+- [weave](docs/weave.md): Weave is a lightweight container overlay network that doesn't require an external K/V database cluster.
     (Please refer to `weave` [troubleshooting documentation](https://www.weave.works/docs/net/latest/troubleshooting/)).
 
-- [kube-ovn](docs/CNI/kube-ovn.md): Kube-OVN integrates the OVN-based Network Virtualization with Kubernetes. It offers an advanced Container Network Fabric for Enterprises.
+- [kube-ovn](docs/kube-ovn.md): Kube-OVN integrates the OVN-based Network Virtualization with Kubernetes. It offers an advanced Container Network Fabric for Enterprises.
 
-- [kube-router](docs/CNI/kube-router.md): Kube-router is a L3 CNI for Kubernetes networking aiming to provide operational
+- [kube-router](docs/kube-router.md): Kube-router is a L3 CNI for Kubernetes networking aiming to provide operational
     simplicity and high performance: it uses IPVS to provide Kube Services Proxy (if setup to replace kube-proxy),
     iptables for network policies, and BGP for ods L3 networking (with optionally BGP peering with out-of-cluster BGP peers).
     It can also optionally advertise routes to Kubernetes cluster Pods CIDRs, ClusterIPs, ExternalIPs and LoadBalancerIPs.
 
-- [macvlan](docs/CNI/macvlan.md): Macvlan is a Linux network driver. Pods have their own unique Mac and Ip address, connected directly the physical (layer 2) network.
+- [macvlan](docs/macvlan.md): Macvlan is a Linux network driver. Pods have their own unique Mac and Ip address, connected directly the physical (layer 2) network.
 
-- [multus](docs/CNI/multus.md): Multus is a meta CNI plugin that provides multiple network interface support to pods. For each interface Multus delegates CNI calls to secondary CNI plugins such as Calico, macvlan, etc.
+- [multus](docs/multus.md): Multus is a meta CNI plugin that provides multiple network interface support to pods. For each interface Multus delegates CNI calls to secondary CNI plugins such as Calico, macvlan, etc.
 
 - [custom_cni](roles/network-plugin/custom_cni/) : You can specify some manifests that will be applied to the clusters to bring you own CNI and use non-supported ones by Kubespray.
   See `tests/files/custom_cni/README.md` and `tests/files/custom_cni/values.yaml`for an example with a CNI provided by a Helm Chart.
 
 The network plugin to use is defined by the variable `kube_network_plugin`. There is also an
 option to leverage built-in cloud provider networking instead.
-See also [Network checker](docs/advanced/netcheck.md).
+See also [Network checker](docs/netcheck.md).
 
 ## Ingress Plugins
 
 - [nginx](https://kubernetes.github.io/ingress-nginx): the NGINX Ingress Controller.
 
-- [metallb](docs/ingress/metallb.md): the MetalLB bare-metal service LoadBalancer provider.
+- [metallb](docs/metallb.md): the MetalLB bare-metal service LoadBalancer provider.
 
 ## Community docs and resources
 
@@ -282,4 +280,4 @@ See also [Network checker](docs/advanced/netcheck.md).
 
 CI/end-to-end tests sponsored by: [CNCF](https://cncf.io), [Equinix Metal](https://metal.equinix.com/), [OVHcloud](https://www.ovhcloud.com/), [ELASTX](https://elastx.se/).
 
-See the [test matrix](docs/developers/test_cases.md) for details.
+See the [test matrix](docs/test_cases.md) for details.

RELEASE.md

@@ -3,19 +3,17 @@
 The Kubespray Project is released on an as-needed basis. The process is as follows:
 
 1. An issue is proposing a new release with a changelog since the last release. Please see [a good sample issue](https://github.com/kubernetes-sigs/kubespray/issues/8325)
-1. At least one of the [approvers](OWNERS_ALIASES) must approve this release
+2. At least one of the [approvers](OWNERS_ALIASES) must approve this release
-1. (Only for major releases) The `kube_version_min_required` variable is set to `n-1`
+3. The `kube_version_min_required` variable is set to `n-1`
-1. (Only for major releases) Remove hashes for [EOL versions](https://github.com/kubernetes/website/blob/main/content/en/releases/patch-releases.md) of kubernetes from `*_checksums` variables.
+4. Remove hashes for [EOL versions](https://github.com/kubernetes/website/blob/main/content/en/releases/patch-releases.md) of kubernetes from `*_checksums` variables.
-1. Create the release note with [Kubernetes Release Notes Generator](https://github.com/kubernetes/release/blob/master/cmd/release-notes/README.md). See the following `Release note creation` section for the details.
+5. Create the release note with [Kubernetes Release Notes Generator](https://github.com/kubernetes/release/blob/master/cmd/release-notes/README.md). See the following `Release note creation` section for the details.
-1. An approver creates [new release in GitHub](https://github.com/kubernetes-sigs/kubespray/releases/new) using a version and tag name like `vX.Y.Z` and attaching the release notes
+6. An approver creates [new release in GitHub](https://github.com/kubernetes-sigs/kubespray/releases/new) using a version and tag name like `vX.Y.Z` and attaching the release notes
-1. (Only for major releases) An approver creates a release branch in the form `release-X.Y`
+7. An approver creates a release branch in the form `release-X.Y`
-1. (For major releases) On the `master` branch: bump the version in `galaxy.yml` to the next expected major release (X.y.0 with y = Y + 1), make a Pull Request.
+8. The corresponding version of [quay.io/kubespray/kubespray:vX.Y.Z](https://quay.io/repository/kubespray/kubespray) and [quay.io/kubespray/vagrant:vX.Y.Z](https://quay.io/repository/kubespray/vagrant) container images are built and tagged. See the following `Container image creation` section for the details.
-1. (For minor releases) On the `release-X.Y` branch: bump the version in `galaxy.yml` to the next expected minor release (X.Y.z with z = Z + 1), make a Pull Request.
+9. The `KUBESPRAY_VERSION` variable is updated in `.gitlab-ci.yml`
-1. The corresponding version of [quay.io/kubespray/kubespray:vX.Y.Z](https://quay.io/repository/kubespray/kubespray) and [quay.io/kubespray/vagrant:vX.Y.Z](https://quay.io/repository/kubespray/vagrant) container images are built and tagged. See the following `Container image creation` section for the details.
+10. The release issue is closed
-1. (Only for major releases) The `KUBESPRAY_VERSION` in `.gitlab-ci.yml` is upgraded to the version we just released # TODO clarify this, this variable is for testing upgrades.
+11. An announcement email is sent to `dev@kubernetes.io` with the subject `[ANNOUNCE] Kubespray $VERSION is released`
-1. The release issue is closed
+12. The topic of the #kubespray channel is updated with `vX.Y.Z is released! | ...`
-1. An announcement email is sent to `dev@kubernetes.io` with the subject `[ANNOUNCE] Kubespray $VERSION is released`
-1. The topic of the #kubespray channel is updated with `vX.Y.Z is released! | ...`
 
 ## Major/minor releases and milestones
 

Vagrantfile

@@ -21,15 +21,13 @@ SUPPORTED_OS = {
   "flatcar-edge"        => {box: "flatcar-edge",               user: "core", box_url: FLATCAR_URL_TEMPLATE % ["edge"]},
   "ubuntu2004"          => {box: "generic/ubuntu2004",         user: "vagrant"},
   "ubuntu2204"          => {box: "generic/ubuntu2204",         user: "vagrant"},
-  "ubuntu2404"          => {box: "bento/ubuntu-24.04",         user: "vagrant"},
   "centos"              => {box: "centos/7",                   user: "vagrant"},
   "centos-bento"        => {box: "bento/centos-7.6",           user: "vagrant"},
   "centos8"             => {box: "centos/8",                   user: "vagrant"},
   "centos8-bento"       => {box: "bento/centos-8",             user: "vagrant"},
   "almalinux8"          => {box: "almalinux/8",                user: "vagrant"},
   "almalinux8-bento"    => {box: "bento/almalinux-8",          user: "vagrant"},
-  "rockylinux8"         => {box: "rockylinux/8",               user: "vagrant"},
+  "rockylinux8"         => {box: "generic/rocky8",             user: "vagrant"},
-  "rockylinux9"         => {box: "rockylinux/9",               user: "vagrant"},
   "fedora37"            => {box: "fedora/37-cloud-base",       user: "vagrant"},
   "fedora38"            => {box: "fedora/38-cloud-base",       user: "vagrant"},
   "opensuse"            => {box: "opensuse/Leap-15.4.x86_64",  user: "vagrant"},
@@ -38,8 +36,6 @@ SUPPORTED_OS = {
   "oraclelinux8"        => {box: "generic/oracle8",            user: "vagrant"},
   "rhel7"               => {box: "generic/rhel7",              user: "vagrant"},
   "rhel8"               => {box: "generic/rhel8",              user: "vagrant"},
-  "debian11"            => {box: "debian/bullseye64",          user: "vagrant"},
-  "debian12"            => {box: "debian/bookworm64",          user: "vagrant"},
 }
 
 if File.exist?(CONFIG)
@@ -81,10 +77,7 @@ $libvirt_nested ||= false
 $ansible_verbosity ||= false
 $ansible_tags ||= ENV['VAGRANT_ANSIBLE_TAGS'] || ""
 
-$vagrant_dir ||= File.join(File.dirname(__FILE__), ".vagrant")
-
 $playbook ||= "cluster.yml"
-$extra_vars ||= {}
 
 host_vars = {}
 
@@ -103,7 +96,7 @@ $inventory = File.absolute_path($inventory, File.dirname(__FILE__))
 # if $inventory has a hosts.ini file use it, otherwise copy over
 # vars etc to where vagrant expects dynamic inventory to be
 if ! File.exist?(File.join(File.dirname($inventory), "hosts.ini"))
-  $vagrant_ansible = File.join(File.absolute_path($vagrant_dir), "provisioners", "ansible")
+  $vagrant_ansible = File.join(File.dirname(__FILE__), ".vagrant", "provisioners", "ansible")
   FileUtils.mkdir_p($vagrant_ansible) if ! File.exist?($vagrant_ansible)
   $vagrant_inventory = File.join($vagrant_ansible,"inventory")
   FileUtils.rm_f($vagrant_inventory)
@@ -189,14 +182,6 @@ Vagrant.configure("2") do |config|
             lv.storage :file, :device => "hd#{driverletters[d]}", :path => "disk-#{i}-#{d}-#{DISK_UUID}.disk", :size => $kube_node_instances_with_disks_size, :bus => "scsi"
           end
         end
-        node.vm.provider :virtualbox do |vb|
-          # always make /dev/sd{a/b/c} so that CI can ensure that
-          # virtualbox and libvirt will have the same devices to use for OSDs
-          (1..$kube_node_instances_with_disks_number).each do |d|
-            vb.customize ['createhd', '--filename', "disk-#{i}-#{driverletters[d]}-#{DISK_UUID}.disk", '--size', $kube_node_instances_with_disks_size] # 10GB disk
-            vb.customize ['storageattach', :id, '--storagectl', 'SATA Controller', '--port', d, '--device', 0, '--type', 'hdd', '--medium', "disk-#{i}-#{driverletters[d]}-#{DISK_UUID}.disk", '--nonrotational', 'on', '--mtype', 'normal']
-          end
-        end
       end
 
       if $expose_docker_tcp
@@ -247,13 +232,6 @@ Vagrant.configure("2") do |config|
         SHELL
       end
 
-      # Rockylinux boxes needs UEFI
-      if ["rockylinux8", "rockylinux9"].include? $os
-        config.vm.provider "libvirt" do |domain|
-          domain.loader = "/usr/share/OVMF/x64/OVMF_CODE.fd"
-        end
-      end
-
       # Disable firewalld on oraclelinux/redhat vms
       if ["oraclelinux","oraclelinux8","rhel7","rhel8","rockylinux8"].include? $os
         node.vm.provision "shell", inline: "systemctl stop firewalld; systemctl disable firewalld"
@@ -277,8 +255,7 @@ Vagrant.configure("2") do |config|
         "kubectl_localhost": "True",
         "local_path_provisioner_enabled": "#{$local_path_provisioner_enabled}",
         "local_path_provisioner_claim_root": "#{$local_path_provisioner_claim_root}",
-        "ansible_ssh_user": SUPPORTED_OS[$os][:user],
+        "ansible_ssh_user": SUPPORTED_OS[$os][:user]
-        "unsafe_show_logs": "True"
       }
 
       # Only execute the Ansible provisioner once, when all the machines are up and ready.
@@ -297,7 +274,6 @@ Vagrant.configure("2") do |config|
           ansible.host_key_checking = false
           ansible.raw_arguments = ["--forks=#{$num_instances}", "--flush-cache", "-e ansible_become_pass=vagrant"]
           ansible.host_vars = host_vars
-          ansible.extra_vars = $extra_vars
           if $ansible_tags != ""
             ansible.tags = [$ansible_tags]
           end

ansible.cfg

@@ -1,6 +1,6 @@
 [ssh_connection]
 pipelining=True
-ssh_args = -o ControlMaster=auto -o ControlPersist=30m -o ConnectionAttempts=100 -o UserKnownHostsFile=/dev/null
+ansible_ssh_args = -o ControlMaster=auto -o ControlPersist=30m -o ConnectionAttempts=100 -o UserKnownHostsFile=/dev/null
 #control_path = ~/.ssh/ansible-%%r@%%h:%%p
 [defaults]
 # https://github.com/ansible/ansible/issues/56930 (to ignore group names with - and .)

contrib/azurerm/README.md

@@ -49,7 +49,7 @@ If you need to delete all resources from a resource group, simply call:
 
 ## Installing Ansible and the dependencies
 
-Install Ansible according to [Ansible installation guide](/docs/ansible/ansible.md#installing-ansible)
+Install Ansible according to [Ansible installation guide](/docs/ansible.md#installing-ansible)
 
 ## Generating an inventory for kubespray
 

contrib/offline/README.md

@@ -5,17 +5,13 @@
 Container image collecting script for offline deployment
 
 This script has two features:
-(1) Get container images from an environment which is deployed online, or set IMAGES_FROM_FILE
+(1) Get container images from an environment which is deployed online.
-    environment variable to get images from a file (e.g. temp/images.list after running the
-    ./generate_list.sh script).
 (2) Deploy local container registry and register the container images to the registry.
 
 Step(1) should be done online site as a preparation, then we bring the gotten images
 to the target offline environment. if images are from a private registry,
 you need to set `PRIVATE_REGISTRY` environment variable.
-Then we will run step(2) for registering the images to local registry, or to an existing
+Then we will run step(2) for registering the images to local registry.
-registry set by the `DESTINATION_REGISTRY` environment variable. By default, the local registry
-will run on port 5000. This can be changed with the `REGISTRY_PORT` environment variable
 
 Step(1) can be operated with:
 
@@ -31,7 +27,7 @@ manage-offline-container-images.sh   register
 
 ## generate_list.sh
 
-This script generates the list of downloaded files and the list of container images by `roles/kubespray-defaults/defaults/main/download.yml` file.
+This script generates the list of downloaded files and the list of container images by `roles/download/defaults/main/main.yml` file.
 
 Run this script will execute `generate_list.yml` playbook in kubespray root directory and generate four files,
 all downloaded files url in files.list, all container images in images.list, jinja2 templates in *.template.

contrib/offline/generate_list.sh

@@ -5,7 +5,7 @@ CURRENT_DIR=$(cd $(dirname $0); pwd)
 TEMP_DIR="${CURRENT_DIR}/temp"
 REPO_ROOT_DIR="${CURRENT_DIR%/contrib/offline}"
 
-: ${DOWNLOAD_YML:="roles/kubespray-defaults/defaults/main/download.yml"}
+: ${DOWNLOAD_YML:="roles/download/defaults/main/main.yml"}
 
 mkdir -p ${TEMP_DIR}
 
@@ -19,7 +19,7 @@ sed -n '/^downloads:/,/download_defaults:/p' ${REPO_ROOT_DIR}/${DOWNLOAD_YML} \
     | sed 'N;s#\n# #g' | tr ' ' ':' | sed 's/\"//g' > ${TEMP_DIR}/images.list.template
 
 # add kube-* images to images list template
-# Those container images are downloaded by kubeadm, then roles/kubespray-defaults/defaults/main/download.yml
+# Those container images are downloaded by kubeadm, then roles/download/defaults/main/main.yml
 # doesn't contain those images. That is reason why here needs to put those images into the
 # list separately.
 KUBE_IMAGES="kube-apiserver kube-controller-manager kube-scheduler kube-proxy"

contrib/offline/manage-offline-container-images.sh

@@ -12,40 +12,27 @@ RETRY_COUNT=5
 function create_container_image_tar() {
 	set -e
 
-	if [ -z "${IMAGES_FROM_FILE}" ]; then
+	IMAGES=$(kubectl describe pods --all-namespaces | grep " Image:" | awk '{print $2}' | sort | uniq)
-		echo "Getting images from current \"$(kubectl config current-context)\""
+	# NOTE: etcd and pause cannot be seen as pods.
-
+	# The pause image is used for --pod-infra-container-image option of kubelet.
-		IMAGES=$(mktemp --suffix=-images)
+	EXT_IMAGES=$(kubectl cluster-info dump | egrep "quay.io/coreos/etcd:|registry.k8s.io/pause:" | sed s@\"@@g)
-		trap 'rm -f "${IMAGES}"' EXIT
+	IMAGES="${IMAGES} ${EXT_IMAGES}"
-
-		kubectl describe cronjobs,jobs,pods --all-namespaces | grep " Image:" | awk '{print $2}' | sort | uniq > "${IMAGES}"
-		# NOTE: etcd and pause cannot be seen as pods.
-		# The pause image is used for --pod-infra-container-image option of kubelet.
-		kubectl cluster-info dump | grep -E "quay.io/coreos/etcd:|registry.k8s.io/pause:" | sed s@\"@@g >> "${IMAGES}"
-	else
-		echo "Getting images from file \"${IMAGES_FROM_FILE}\""
-		if [ ! -f "${IMAGES_FROM_FILE}" ]; then
-			echo "${IMAGES_FROM_FILE} is not a file"
-			exit 1
-		fi
-		IMAGES=$(realpath $IMAGES_FROM_FILE)
-	fi
 
 	rm -f  ${IMAGE_TAR_FILE}
 	rm -rf ${IMAGE_DIR}
 	mkdir  ${IMAGE_DIR}
 	cd     ${IMAGE_DIR}
 
-	sudo ${runtime} pull registry:latest
+	sudo docker pull registry:latest
-	sudo ${runtime} save -o registry-latest.tar registry:latest
+	sudo docker save -o registry-latest.tar registry:latest
 
-	while read -r image
+	for image in ${IMAGES}
 	do
-		FILE_NAME="$(echo ${image} | sed s@"/"@"-"@g | sed s/":"/"-"/g | sed -E 's/\@.*//g')".tar
+		FILE_NAME="$(echo ${image} | sed s@"/"@"-"@g | sed s/":"/"-"/g)".tar
 		set +e
 		for step in $(seq 1 ${RETRY_COUNT})
 		do
-			sudo ${runtime} pull ${image}
+			sudo docker pull ${image}
 			if [ $? -eq 0 ]; then
 				break
 			fi
@@ -55,26 +42,24 @@ function create_container_image_tar() {
 			fi
 		done
 		set -e
-		sudo ${runtime} save -o ${FILE_NAME}  ${image}
+		sudo docker save -o ${FILE_NAME}  ${image}
 
 		# NOTE: Here removes the following repo parts from each image
 		# so that these parts will be replaced with Kubespray.
 		# - kube_image_repo: "registry.k8s.io"
 		# - gcr_image_repo: "gcr.io"
-		# - ghcr_image_repo: "ghcr.io"
 		# - docker_image_repo: "docker.io"
 		# - quay_image_repo: "quay.io"
 		FIRST_PART=$(echo ${image} | awk -F"/" '{print $1}')
 		if [ "${FIRST_PART}" = "registry.k8s.io" ] ||
 		   [ "${FIRST_PART}" = "gcr.io" ] ||
-		   [ "${FIRST_PART}" = "ghcr.io" ] ||
 		   [ "${FIRST_PART}" = "docker.io" ] ||
 		   [ "${FIRST_PART}" = "quay.io" ] ||
 		   [ "${FIRST_PART}" = "${PRIVATE_REGISTRY}" ]; then
-			image=$(echo ${image} | sed s@"${FIRST_PART}/"@@ | sed -E 's/\@.*/\n/g')
+			image=$(echo ${image} | sed s@"${FIRST_PART}/"@@)
 		fi
 		echo "${FILE_NAME}  ${image}" >> ${IMAGE_LIST}
-	done < "${IMAGES}"
+	done
 
 	cd ..
 	sudo chown ${USER} ${IMAGE_DIR}/*
@@ -87,16 +72,6 @@ function create_container_image_tar() {
 }
 
 function register_container_images() {
-	create_registry=false
-	REGISTRY_PORT=${REGISTRY_PORT:-"5000"}
-
-	if [ -z "${DESTINATION_REGISTRY}" ]; then
-		echo "DESTINATION_REGISTRY not set, will create local registry"
-		create_registry=true
-		DESTINATION_REGISTRY="$(hostname):${REGISTRY_PORT}"
-	fi
-	echo "Images will be pushed to ${DESTINATION_REGISTRY}"
-
 	if [ ! -f ${IMAGE_TAR_FILE} ]; then
 		echo "${IMAGE_TAR_FILE} should exist."
 		exit 1
@@ -106,47 +81,39 @@ function register_container_images() {
 	fi
 
 	# To avoid "http: server gave http response to https client" error.
+	LOCALHOST_NAME=$(hostname)
 	if [ -d /etc/docker/ ]; then
 		set -e
 		# Ubuntu18.04, RHEL7/CentOS7
 		cp ${CURRENT_DIR}/docker-daemon.json      ${TEMP_DIR}/docker-daemon.json
-		sed -i s@"HOSTNAME"@"$(hostname)"@  ${TEMP_DIR}/docker-daemon.json
+		sed -i s@"HOSTNAME"@"${LOCALHOST_NAME}"@  ${TEMP_DIR}/docker-daemon.json
 		sudo cp ${TEMP_DIR}/docker-daemon.json           /etc/docker/daemon.json
 	elif [ -d /etc/containers/ ]; then
 		set -e
 		# RHEL8/CentOS8
 		cp ${CURRENT_DIR}/registries.conf         ${TEMP_DIR}/registries.conf
-		sed -i s@"HOSTNAME"@"$(hostname)"@  ${TEMP_DIR}/registries.conf
+		sed -i s@"HOSTNAME"@"${LOCALHOST_NAME}"@  ${TEMP_DIR}/registries.conf
 		sudo cp ${TEMP_DIR}/registries.conf   /etc/containers/registries.conf
 	else
-		echo "runtime package(docker-ce, podman, nerctl, etc.) should be installed"
+		echo "docker package(docker-ce, etc.) should be installed"
 		exit 1
 	fi
 
 	tar -zxvf ${IMAGE_TAR_FILE}
-
+	sudo docker load -i ${IMAGE_DIR}/registry-latest.tar
-	if [ "${create_registry}" ]; then
+	set +e
-		sudo ${runtime} load -i ${IMAGE_DIR}/registry-latest.tar
+	sudo docker container inspect registry >/dev/null 2>&1
-		set +e
+	if [ $? -ne 0 ]; then
-
+		sudo docker run --restart=always -d -p 5000:5000 --name registry registry:latest
-		sudo ${runtime} container inspect registry >/dev/null 2>&1
-		if [ $? -ne 0 ]; then
-			sudo ${runtime} run --restart=always -d -p "${REGISTRY_PORT}":"${REGISTRY_PORT}" --name registry registry:latest
-		fi
-		set -e
 	fi
+	set -e
 
 	while read -r line; do
 		file_name=$(echo ${line} | awk '{print $1}')
 		raw_image=$(echo ${line} | awk '{print $2}')
-		new_image="${DESTINATION_REGISTRY}/${raw_image}"
+		new_image="${LOCALHOST_NAME}:5000/${raw_image}"
-		load_image=$(sudo ${runtime} load -i ${IMAGE_DIR}/${file_name} | head -n1)
+		org_image=$(sudo docker load -i ${IMAGE_DIR}/${file_name} | head -n1 | awk '{print $3}')
-		org_image=$(echo "${load_image}"  | awk '{print $3}')
+		image_id=$(sudo docker image inspect ${org_image} | grep "\"Id\":" | awk -F: '{print $3}'| sed s/'\",'//)
-		# special case for tags containing the digest when using docker or podman as the container runtime
-		if [ "${org_image}" == "ID:" ]; then
-		  org_image=$(echo "${load_image}"  | awk '{print $4}')
-		fi
-		image_id=$(sudo ${runtime} image inspect ${org_image} | grep "\"Id\":" | awk -F: '{print $3}'| sed s/'\",'//)
 		if [ -z "${file_name}" ]; then
 			echo "Failed to get file_name for line ${line}"
 			exit 1
@@ -163,48 +130,32 @@ function register_container_images() {
 			echo "Failed to get image_id for file ${file_name}"
 			exit 1
 		fi
-		sudo ${runtime} load -i ${IMAGE_DIR}/${file_name}
+		sudo docker load -i ${IMAGE_DIR}/${file_name}
-		sudo ${runtime} tag  ${image_id} ${new_image}
+		sudo docker tag  ${image_id} ${new_image}
-		sudo ${runtime} push ${new_image}
+		sudo docker push ${new_image}
 	done <<< "$(cat ${IMAGE_LIST})"
 
 	echo "Succeeded to register container images to local registry."
-	echo "Please specify \"${DESTINATION_REGISTRY}\" for the following options in your inventry:"
+	echo "Please specify ${LOCALHOST_NAME}:5000 for the following options in your inventry:"
 	echo "- kube_image_repo"
 	echo "- gcr_image_repo"
 	echo "- docker_image_repo"
 	echo "- quay_image_repo"
 }
 
-# get runtime command
-if command -v nerdctl 1>/dev/null 2>&1; then
-    runtime="nerdctl"
-elif command -v podman 1>/dev/null 2>&1; then
-    runtime="podman"
-elif command -v docker 1>/dev/null 2>&1; then
-    runtime="docker"
-else
-    echo "No supported container runtime found"
-    exit 1
-fi
-
 if [ "${OPTION}" == "create" ]; then
 	create_container_image_tar
 elif [ "${OPTION}" == "register" ]; then
 	register_container_images
 else
 	echo "This script has two features:"
-	echo "(1) Get container images from an environment which is deployed online, or set IMAGES_FROM_FILE"
+	echo "(1) Get container images from an environment which is deployed online."
-	echo "    environment variable to get images from a file (e.g. temp/images.list after running the"
-	echo "    ./generate_list.sh script)."
 	echo "(2) Deploy local container registry and register the container images to the registry."
 	echo ""
 	echo "Step(1) should be done online site as a preparation, then we bring"
 	echo "the gotten images to the target offline environment. if images are from"
 	echo "a private registry, you need to set PRIVATE_REGISTRY environment variable."
-	echo "Then we will run step(2) for registering the images to local registry, or to an existing"
+	echo "Then we will run step(2) for registering the images to local registry."
-	echo "registry set by the DESTINATION_REGISTRY environment variable. By default, the local registry"
-	echo "will run on port 5000. This can be changed with the REGISTRY_PORT environment variable"
 	echo ""
 	echo "${IMAGE_TAR_FILE} is created to contain your container images."
 	echo "Please keep this file and bring it to your offline environment."

contrib/offline/manage-offline-files.sh

@@ -17,12 +17,7 @@ rm -rf "${OFFLINE_FILES_DIR}"
 rm "${OFFLINE_FILES_ARCHIVE}"
 mkdir  "${OFFLINE_FILES_DIR}"
 
-while read -r url; do
+wget -x -P "${OFFLINE_FILES_DIR}" -i "${FILES_LIST}"
-  if ! wget -x -P "${OFFLINE_FILES_DIR}" "${url}"; then
-    exit 1
-  fi
-done < "${FILES_LIST}"
-
 tar -czvf "${OFFLINE_FILES_ARCHIVE}"  "${OFFLINE_FILES_DIR_NAME}"
 
 [ -n "$NO_HTTP_SERVER" ] && echo "skip to run nginx" && exit 0
@@ -43,7 +38,7 @@ sudo "${runtime}" container inspect nginx >/dev/null 2>&1
 if [ $? -ne 0 ]; then
     sudo "${runtime}" run \
         --restart=always -d -p ${NGINX_PORT}:80 \
-        --volume "${OFFLINE_FILES_DIR}":/usr/share/nginx/html/download \
+        --volume "${OFFLINE_FILES_DIR}:/usr/share/nginx/html/download" \
         --volume "${CURRENT_DIR}"/nginx.conf:/etc/nginx/nginx.conf \
         --name nginx nginx:alpine
 fi

contrib/terraform/OWNERS

@@ -1,3 +1,5 @@
 # See the OWNERS docs at https://go.k8s.io/owners
+
 approvers:
+  - holmsten
   - miouge1

contrib/terraform/aws/README.md

@@ -50,32 +50,70 @@ Example (this one assumes you are using Ubuntu)
 ansible-playbook -i ./inventory/hosts ./cluster.yml -e ansible_user=ubuntu -b --become-user=root --flush-cache
 ```
 
-## Using other distrib than Ubuntu***
+***Using other distrib than Ubuntu***
+If you want to use another distribution than Ubuntu 18.04 (Bionic) LTS, you can modify the search filters of the 'data "aws_ami" "distro"' in variables.tf.
 
-To leverage a Linux distribution other than Ubuntu 18.04 (Bionic) LTS for your Terraform configurations, you can adjust the AMI search filters within the 'data "aws_ami" "distro"' block by utilizing variables in your `terraform.tfvars` file. This approach ensures a flexible configuration that adapts to various Linux distributions without directly modifying the core Terraform files.
+For example, to use:
 
-### Example Usages
+- Debian Jessie, replace 'data "aws_ami" "distro"' in variables.tf with
 
-- **Debian Jessie**: To configure the usage of Debian Jessie, insert the subsequent lines into your `terraform.tfvars`:
+```ini
+data "aws_ami" "distro" {
+  most_recent = true
 
-  ```hcl
+  filter {
-  ami_name_pattern        = "debian-jessie-amd64-hvm-*"
+    name   = "name"
-  ami_owners              = ["379101102735"]
+    values = ["debian-jessie-amd64-hvm-*"]
-  ```
+  }
 
-- **Ubuntu 16.04**: To utilize Ubuntu 16.04 instead, apply the following configuration in your `terraform.tfvars`:
+  filter {
+    name   = "virtualization-type"
+    values = ["hvm"]
+  }
 
-  ```hcl
+  owners = ["379101102735"]
-  ami_name_pattern        = "ubuntu/images/hvm-ssd/ubuntu-xenial-16.04-amd64-*"
+}
-  ami_owners              = ["099720109477"]
+```
-  ```
 
-- **Centos 7**: For employing Centos 7, incorporate these lines into your `terraform.tfvars`:
+- Ubuntu 16.04, replace 'data "aws_ami" "distro"' in variables.tf with
 
-  ```hcl
+```ini
-  ami_name_pattern        = "dcos-centos7-*"
+data "aws_ami" "distro" {
-  ami_owners              = ["688023202711"]
+  most_recent = true
-  ```
+
+  filter {
+    name   = "name"
+    values = ["ubuntu/images/hvm-ssd/ubuntu-xenial-16.04-amd64-*"]
+  }
+
+  filter {
+    name   = "virtualization-type"
+    values = ["hvm"]
+  }
+
+  owners = ["099720109477"]
+}
+```
+
+- Centos 7, replace 'data "aws_ami" "distro"' in variables.tf with
+
+```ini
+data "aws_ami" "distro" {
+  most_recent = true
+
+  filter {
+    name   = "name"
+    values = ["dcos-centos7-*"]
+  }
+
+  filter {
+    name   = "virtualization-type"
+    values = ["hvm"]
+  }
+
+  owners = ["688023202711"]
+}
+```
 
 ## Connecting to Kubernetes
 

contrib/terraform/aws/variables.tf

@@ -20,38 +20,20 @@ variable "aws_cluster_name" {
   description = "Name of AWS Cluster"
 }
 
-variable "ami_name_pattern" {
-  description = "The name pattern to use for AMI lookup"
-  type        = string
-  default     = "debian-10-amd64-*"
-}
-
-variable "ami_virtualization_type" {
-  description = "The virtualization type to use for AMI lookup"
-  type        = string
-  default     = "hvm"
-}
-
-variable "ami_owners" {
-  description = "The owners to use for AMI lookup"
-  type        = list(string)
-  default     = ["136693071363"]
-}
-
 data "aws_ami" "distro" {
   most_recent = true
 
   filter {
     name   = "name"
-    values = [var.ami_name_pattern]
+    values = ["debian-10-amd64-*"]
   }
 
   filter {
     name   = "virtualization-type"
-    values = [var.ami_virtualization_type]
+    values = ["hvm"]
   }
 
-  owners = var.ami_owners
+  owners = ["136693071363"] # Debian-10
 }
 
 //AWS VPC Variables

contrib/terraform/equinix/README.md

@@ -35,7 +35,7 @@ now six total etcd replicas.
 ## Requirements
 
 - [Install Terraform](https://www.terraform.io/intro/getting-started/install.html)
-- [Install Ansible dependencies](/docs/ansible/ansible.md#installing-ansible)
+- [Install Ansible dependencies](/docs/ansible.md#installing-ansible)
 - Account with Equinix Metal
 - An SSH key pair
 

contrib/terraform/equinix/provider.tf

@@ -7,7 +7,7 @@ terraform {
   required_providers {
     equinix = {
       source  = "equinix/equinix"
-      version = "1.24.0"
+      version = "~> 1.14"
     }
   }
 }

contrib/terraform/openstack/.gitignore

@@ -1,5 +1,5 @@
 .terraform
 *.tfvars
-!sample-inventory/cluster.tfvars
+!sample-inventory\/cluster.tfvars
 *.tfstate
 *.tfstate.backup

contrib/terraform/openstack/README.md

@@ -24,7 +24,6 @@ most modern installs of OpenStack that support the basic services.
 - [Ultimum](https://ultimum.io/)
 - [VexxHost](https://vexxhost.com/)
 - [Zetta](https://www.zetta.io/)
-- [Cloudify](https://www.cloudify.ro/en)
 
 ## Approach
 
@@ -98,10 +97,9 @@ binaries available on hyperkube v1.4.3_coreos.0 or higher.
 
 ## Module Architecture
 
-The configuration is divided into four modules:
+The configuration is divided into three modules:
 
 - Network
-- Loadbalancer
 - IPs
 - Compute
 
@@ -271,18 +269,11 @@ For your cluster, edit `inventory/$CLUSTER/cluster.tfvars`.
 |`supplementary_master_groups` | To add ansible groups to the masters, such as `kube_node` for tainting them as nodes, empty by default. |
 |`supplementary_node_groups` | To add ansible groups to the nodes, such as `kube_ingress` for running ingress controller pods, empty by default. |
 |`bastion_allowed_remote_ips` | List of CIDR allowed to initiate a SSH connection, `["0.0.0.0/0"]` by default |
-|`bastion_allowed_remote_ipv6_ips` | List of IPv6 CIDR allowed to initiate a SSH connection, `["::/0"]` by default |
 |`master_allowed_remote_ips` | List of CIDR blocks allowed to initiate an API connection, `["0.0.0.0/0"]` by default |
-|`master_allowed_remote_ipv6_ips` | List of IPv6 CIDR blocks allowed to initiate an API connection, `["::/0"]` by default |
 |`bastion_allowed_ports` | List of ports to open on bastion node, `[]` by default |
-|`bastion_allowed_ports_ipv6` | List of ports to open on bastion node for IPv6 CIDR blocks, `[]` by default |
 |`k8s_allowed_remote_ips` | List of CIDR allowed to initiate a SSH connection, empty by default |
-|`k8s_allowed_remote_ips_ipv6` | List of IPv6 CIDR allowed to initiate a SSH connection, empty by default |
-|`k8s_allowed_egress_ipv6_ips` | List of IPv6 CIDRs allowed for egress traffic, `["::/0"]` by default |
 |`worker_allowed_ports` | List of ports to open on worker nodes, `[{ "protocol" = "tcp", "port_range_min" = 30000, "port_range_max" = 32767, "remote_ip_prefix" = "0.0.0.0/0"}]` by default |
-|`worker_allowed_ports_ipv6` | List of ports to open on worker nodes for IPv6 CIDR blocks, `[{ "protocol" = "tcp", "port_range_min" = 30000, "port_range_max" = 32767, "remote_ip_prefix" = "::/0"}]` by default |
 |`master_allowed_ports` | List of ports to open on master nodes, expected format is `[{ "protocol" = "tcp", "port_range_min" = 443, "port_range_max" = 443, "remote_ip_prefix" = "0.0.0.0/0"}]`, empty by default |
-|`master_allowed_ports_ipv6` | List of ports to open on master nodes for IPv6 CIDR blocks, expected format is `[{ "protocol" = "tcp", "port_range_min" = 443, "port_range_max" = 443, "remote_ip_prefix" = "::/0"}]`, empty by default |
 |`node_root_volume_size_in_gb` | Size of the root volume for nodes, 0 to use ephemeral storage |
 |`master_root_volume_size_in_gb` | Size of the root volume for masters, 0 to use ephemeral storage |
 |`master_volume_type` | Volume type of the root volume for control_plane, 'Default' by default |
@@ -299,10 +290,6 @@ For your cluster, edit `inventory/$CLUSTER/cluster.tfvars`.
 |`force_null_port_security` | Set `null` instead of `true` or `false` for `port_security`. `false` by default |
 |`k8s_nodes` | Map containing worker node definition, see explanation below |
 |`k8s_masters` | Map containing master node definition, see explanation for k8s_nodes and `sample-inventory/cluster.tfvars` |
-| `k8s_master_loadbalancer_enabled`| Enable and use an Octavia load balancer for the K8s master nodes |
-| `k8s_master_loadbalancer_listener_port` | Define via which port the K8s Api should be exposed. `6443` by default  |
-| `k8s_master_loadbalancer_server_port` | Define via which port the K8S api is available on the mas. `6443` by default |
-| `k8s_master_loadbalancer_public_ip` | Specify if an existing floating IP should be used for the load balancer. A new floating IP is assigned by default  |
 
 ##### k8s_nodes
 
@@ -331,7 +318,6 @@ k8s_nodes:
         mount_path: string # Path to where the partition should be mounted
         partition_start: string # Where the partition should start (e.g. 10GB ). Note, if you set the partition_start to 0 there will be no space left for the root partition
         partition_end: string # Where the partition should end (e.g. 10GB or -1 for end of volume)
-      netplan_critical_dhcp_interface: string # Name of interface to set the dhcp flag critical = true, to circumvent [this issue](https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1776013).
 ```
 
 For example:
@@ -619,7 +605,7 @@ Edit `inventory/$CLUSTER/group_vars/k8s_cluster/k8s_cluster.yml`:
 
 - Set variable **kube_network_plugin** to your desired networking plugin.
   - **flannel** works out-of-the-box
-  - **calico** requires [configuring OpenStack Neutron ports](/docs/cloud_providers/openstack.md) to allow service and pod subnets
+  - **calico** requires [configuring OpenStack Neutron ports](/docs/openstack.md) to allow service and pod subnets
 
 ```yml
 # Choose network plugin (calico, weave or flannel)

contrib/terraform/openstack/kubespray.tf

@@ -77,21 +77,14 @@ module "compute" {
   k8s_nodes_fips                               = module.ips.k8s_nodes_fips
   bastion_fips                                 = module.ips.bastion_fips
   bastion_allowed_remote_ips                   = var.bastion_allowed_remote_ips
-  bastion_allowed_remote_ipv6_ips              = var.bastion_allowed_remote_ipv6_ips
   master_allowed_remote_ips                    = var.master_allowed_remote_ips
-  master_allowed_remote_ipv6_ips               = var.master_allowed_remote_ipv6_ips
   k8s_allowed_remote_ips                       = var.k8s_allowed_remote_ips
-  k8s_allowed_remote_ips_ipv6                  = var.k8s_allowed_remote_ips_ipv6
   k8s_allowed_egress_ips                       = var.k8s_allowed_egress_ips
-  k8s_allowed_egress_ipv6_ips                  = var.k8s_allowed_egress_ipv6_ips
   supplementary_master_groups                  = var.supplementary_master_groups
   supplementary_node_groups                    = var.supplementary_node_groups
   master_allowed_ports                         = var.master_allowed_ports
-  master_allowed_ports_ipv6                    = var.master_allowed_ports_ipv6
   worker_allowed_ports                         = var.worker_allowed_ports
-  worker_allowed_ports_ipv6                    = var.worker_allowed_ports_ipv6
   bastion_allowed_ports                        = var.bastion_allowed_ports
-  bastion_allowed_ports_ipv6                   = var.bastion_allowed_ports_ipv6
   use_access_ip                                = var.use_access_ip
   master_server_group_policy                   = var.master_server_group_policy
   node_server_group_policy                     = var.node_server_group_policy
@@ -112,24 +105,6 @@ module "compute" {
   ]
 }
 
-module "loadbalancer" {
-  source = "./modules/loadbalancer"
-
-  cluster_name                          = var.cluster_name
-  subnet_id                             = module.network.subnet_id
-  floatingip_pool                       = var.floatingip_pool
-  k8s_master_ips                        = module.compute.k8s_master_ips
-  k8s_master_loadbalancer_enabled       = var.k8s_master_loadbalancer_enabled
-  k8s_master_loadbalancer_listener_port = var.k8s_master_loadbalancer_listener_port
-  k8s_master_loadbalancer_server_port   = var.k8s_master_loadbalancer_server_port
-  k8s_master_loadbalancer_public_ip     = var.k8s_master_loadbalancer_public_ip
-
-  depends_on = [
-    module.compute.k8s_master
-  ]
-}
-
-
 output "private_subnet_id" {
   value = module.network.subnet_id
 }

contrib/terraform/openstack/modules/compute/main.tf

@@ -19,8 +19,8 @@ data "cloudinit_config" "cloudinit" {
   part {
     content_type =  "text/cloud-config"
     content = templatefile("${path.module}/templates/cloudinit.yaml.tmpl", {
-      extra_partitions = [],
+      # template_file doesn't support lists
-      netplan_critical_dhcp_interface = ""
+      extra_partitions = ""
     })
   }
 }
@@ -70,36 +70,6 @@ resource "openstack_networking_secgroup_rule_v2" "k8s_master_ports" {
   security_group_id = openstack_networking_secgroup_v2.k8s_master.id
 }
 
-resource "openstack_networking_secgroup_rule_v2" "k8s_master_ipv6_ingress" {
-  count             = length(var.master_allowed_remote_ipv6_ips)
-  direction         = "ingress"
-  ethertype         = "IPv6"
-  protocol          = "tcp"
-  port_range_min    = "6443"
-  port_range_max    = "6443"
-  remote_ip_prefix  = var.master_allowed_remote_ipv6_ips[count.index]
-  security_group_id = openstack_networking_secgroup_v2.k8s_master.id
-}
-
-resource "openstack_networking_secgroup_rule_v2" "k8s_master_ports_ipv6_ingress" {
-  count             = length(var.master_allowed_ports_ipv6)
-  direction         = "ingress"
-  ethertype         = "IPv6"
-  protocol          = lookup(var.master_allowed_ports_ipv6[count.index], "protocol", "tcp")
-  port_range_min    = lookup(var.master_allowed_ports_ipv6[count.index], "port_range_min")
-  port_range_max    = lookup(var.master_allowed_ports_ipv6[count.index], "port_range_max")
-  remote_ip_prefix  = lookup(var.master_allowed_ports_ipv6[count.index], "remote_ip_prefix", "::/0")
-  security_group_id = openstack_networking_secgroup_v2.k8s_master.id
-}
-
-resource "openstack_networking_secgroup_rule_v2" "master_egress_ipv6" {
-  count             = length(var.k8s_allowed_egress_ipv6_ips)
-  direction         = "egress"
-  ethertype         = "IPv6"
-  remote_ip_prefix  = var.k8s_allowed_egress_ipv6_ips[count.index]
-  security_group_id = openstack_networking_secgroup_v2.k8s_master.id
-}
-
 resource "openstack_networking_secgroup_v2" "bastion" {
   name                 = "${var.cluster_name}-bastion"
   count                = var.number_of_bastions != "" ? 1 : 0
@@ -129,28 +99,6 @@ resource "openstack_networking_secgroup_rule_v2" "k8s_bastion_ports" {
   security_group_id = openstack_networking_secgroup_v2.bastion[0].id
 }
 
-resource "openstack_networking_secgroup_rule_v2" "bastion_ipv6_ingress" {
-  count             = var.number_of_bastions != "" ? length(var.bastion_allowed_remote_ipv6_ips) : 0
-  direction         = "ingress"
-  ethertype         = "IPv6"
-  protocol          = "tcp"
-  port_range_min    = "22"
-  port_range_max    = "22"
-  remote_ip_prefix  = var.bastion_allowed_remote_ipv6_ips[count.index]
-  security_group_id = openstack_networking_secgroup_v2.bastion[0].id
-}
-
-resource "openstack_networking_secgroup_rule_v2" "k8s_bastion_ports_ipv6_ingress" {
-  count             = length(var.bastion_allowed_ports_ipv6)
-  direction         = "ingress"
-  ethertype         = "IPv6"
-  protocol          = lookup(var.bastion_allowed_ports_ipv6[count.index], "protocol", "tcp")
-  port_range_min    = lookup(var.bastion_allowed_ports_ipv6[count.index], "port_range_min")
-  port_range_max    = lookup(var.bastion_allowed_ports_ipv6[count.index], "port_range_max")
-  remote_ip_prefix  = lookup(var.bastion_allowed_ports_ipv6[count.index], "remote_ip_prefix", "::/0")
-  security_group_id = openstack_networking_secgroup_v2.bastion[0].id
-}
-
 resource "openstack_networking_secgroup_v2" "k8s" {
   name                 = "${var.cluster_name}-k8s"
   description          = "${var.cluster_name} - Kubernetes"
@@ -164,13 +112,6 @@ resource "openstack_networking_secgroup_rule_v2" "k8s" {
   security_group_id = openstack_networking_secgroup_v2.k8s.id
 }
 
-resource "openstack_networking_secgroup_rule_v2" "k8s_ipv6" {
-  direction         = "ingress"
-  ethertype         = "IPv6"
-  remote_group_id   = openstack_networking_secgroup_v2.k8s.id
-  security_group_id = openstack_networking_secgroup_v2.k8s.id
-}
-
 resource "openstack_networking_secgroup_rule_v2" "k8s_allowed_remote_ips" {
   count             = length(var.k8s_allowed_remote_ips)
   direction         = "ingress"
@@ -182,17 +123,6 @@ resource "openstack_networking_secgroup_rule_v2" "k8s_allowed_remote_ips" {
   security_group_id = openstack_networking_secgroup_v2.k8s.id
 }
 
-resource "openstack_networking_secgroup_rule_v2" "k8s_allowed_remote_ips_ipv6" {
-  count             = length(var.k8s_allowed_remote_ips_ipv6)
-  direction         = "ingress"
-  ethertype         = "IPv6"
-  protocol          = "tcp"
-  port_range_min    = "22"
-  port_range_max    = "22"
-  remote_ip_prefix  = var.k8s_allowed_remote_ips_ipv6[count.index]
-  security_group_id = openstack_networking_secgroup_v2.k8s.id
-}
-
 resource "openstack_networking_secgroup_rule_v2" "egress" {
   count             = length(var.k8s_allowed_egress_ips)
   direction         = "egress"
@@ -201,14 +131,6 @@ resource "openstack_networking_secgroup_rule_v2" "egress" {
   security_group_id = openstack_networking_secgroup_v2.k8s.id
 }
 
-resource "openstack_networking_secgroup_rule_v2" "egress_ipv6" {
-  count             = length(var.k8s_allowed_egress_ipv6_ips)
-  direction         = "egress"
-  ethertype         = "IPv6"
-  remote_ip_prefix  = var.k8s_allowed_egress_ipv6_ips[count.index]
-  security_group_id = openstack_networking_secgroup_v2.k8s.id
-}
-
 resource "openstack_networking_secgroup_v2" "worker" {
   name                 = "${var.cluster_name}-k8s-worker"
   description          = "${var.cluster_name} - Kubernetes worker nodes"
@@ -233,17 +155,6 @@ resource "openstack_networking_secgroup_rule_v2" "worker" {
   security_group_id = openstack_networking_secgroup_v2.worker.id
 }
 
-resource "openstack_networking_secgroup_rule_v2" "worker_ipv6_ingress" {
-  count             = length(var.worker_allowed_ports_ipv6)
-  direction         = "ingress"
-  ethertype         = "IPv6"
-  protocol          = lookup(var.worker_allowed_ports_ipv6[count.index], "protocol", "tcp")
-  port_range_min    = lookup(var.worker_allowed_ports_ipv6[count.index], "port_range_min")
-  port_range_max    = lookup(var.worker_allowed_ports_ipv6[count.index], "port_range_max")
-  remote_ip_prefix  = lookup(var.worker_allowed_ports_ipv6[count.index], "remote_ip_prefix", "::/0")
-  security_group_id = openstack_networking_secgroup_v2.worker.id
-}
-
 resource "openstack_compute_servergroup_v2" "k8s_master" {
   count    = var.master_server_group_policy != "" ? 1 : 0
   name     = "k8s-master-srvgrp"
@@ -393,10 +304,6 @@ resource "openstack_networking_port_v2" "k8s_master_port" {
     }
   }
 
-  lifecycle {
-    ignore_changes = [ allowed_address_pairs ]
-  }
-
   depends_on = [
     var.network_router_id
   ]
@@ -463,10 +370,6 @@ resource "openstack_networking_port_v2" "k8s_masters_port" {
     }
   }
 
-  lifecycle {
-    ignore_changes = [ allowed_address_pairs ]
-  }
-
   depends_on = [
     var.network_router_id
   ]
@@ -531,10 +434,6 @@ resource "openstack_networking_port_v2" "k8s_master_no_etcd_port" {
     }
   }
 
-  lifecycle {
-    ignore_changes = [ allowed_address_pairs ]
-  }
-
   depends_on = [
     var.network_router_id
   ]
@@ -661,10 +560,6 @@ resource "openstack_networking_port_v2" "k8s_master_no_floating_ip_port" {
     }
   }
 
-  lifecycle {
-    ignore_changes = [ allowed_address_pairs ]
-  }
-
   depends_on = [
     var.network_router_id
   ]
@@ -725,10 +620,6 @@ resource "openstack_networking_port_v2" "k8s_master_no_floating_ip_no_etcd_port"
     }
   }
 
-  lifecycle {
-    ignore_changes = [ allowed_address_pairs ]
-  }
-
   depends_on = [
     var.network_router_id
   ]
@@ -790,10 +681,6 @@ resource "openstack_networking_port_v2" "k8s_node_port" {
     }
   }
 
-  lifecycle {
-    ignore_changes = [ allowed_address_pairs ]
-  }
-
   depends_on = [
     var.network_router_id
   ]
@@ -860,10 +747,6 @@ resource "openstack_networking_port_v2" "k8s_node_no_floating_ip_port" {
     }
   }
 
-  lifecycle {
-    ignore_changes = [ allowed_address_pairs ]
-  }
-
   depends_on = [
     var.network_router_id
   ]
@@ -925,10 +808,6 @@ resource "openstack_networking_port_v2" "k8s_nodes_port" {
     }
   }
 
-  lifecycle {
-    ignore_changes = [ allowed_address_pairs ]
-  }
-
   depends_on = [
     var.network_router_id
   ]
@@ -942,8 +821,7 @@ resource "openstack_compute_instance_v2" "k8s_nodes" {
   flavor_id         = each.value.flavor
   key_pair          = openstack_compute_keypair_v2.k8s.name
   user_data         = each.value.cloudinit != null ? templatefile("${path.module}/templates/cloudinit.yaml.tmpl", {
-    extra_partitions = each.value.cloudinit.extra_partitions,
+    extra_partitions = each.value.cloudinit.extra_partitions
-    netplan_critical_dhcp_interface = each.value.cloudinit.netplan_critical_dhcp_interface,
   }) : data.cloudinit_config.cloudinit.rendered
 
   dynamic "block_device" {
@@ -972,7 +850,7 @@ resource "openstack_compute_instance_v2" "k8s_nodes" {
 
   metadata = {
     ssh_user         = var.ssh_user
-    kubespray_groups = "kube_node,k8s_cluster,%{if !each.value.floating_ip}no_floating,%{endif}${var.supplementary_node_groups}${each.value.extra_groups != null ? ",${each.value.extra_groups}" : ""}"
+    kubespray_groups = "kube_node,k8s_cluster,%{if each.value.floating_ip == false}no_floating,%{endif}${var.supplementary_node_groups}${each.value.extra_groups != null ? ",${each.value.extra_groups}" : ""}"
     depends_on       = var.network_router_id
     use_access_ip    = var.use_access_ip
   }

contrib/terraform/openstack/modules/compute/outputs.tf

@@ -1,3 +0,0 @@
-output "k8s_master_ips" {
-  value = concat(openstack_compute_instance_v2.k8s_master_no_floating_ip.*, openstack_compute_instance_v2.k8s_master_no_floating_ip_no_etcd.*)
-}

contrib/terraform/openstack/modules/compute/templates/cloudinit.yaml.tmpl

@@ -1,4 +1,4 @@
-%{~ if length(extra_partitions) > 0 || netplan_critical_dhcp_interface != "" }
+%{~ if length(extra_partitions) > 0 }
 #cloud-config
 bootcmd:
 %{~ for idx, partition in extra_partitions }
@@ -8,26 +8,11 @@ bootcmd:
 %{~ endfor }
 
 runcmd:
-%{~ if netplan_critical_dhcp_interface != "" }
-  - netplan apply
-%{~ endif }
 %{~ for idx, partition in extra_partitions }
   - mkdir -p ${partition.mount_path}
   - chown nobody:nogroup ${partition.mount_path}
   - mount ${partition.partition_path} ${partition.mount_path}
-%{~ endfor ~}
+%{~ endfor }
-
-%{~ if netplan_critical_dhcp_interface != "" }
-write_files:
-  - path: /etc/netplan/90-critical-dhcp.yaml
-    content: |
-      network:
-        version: 2
-        ethernets:
-          ${ netplan_critical_dhcp_interface }:
-            dhcp4: true
-            critical: true
-%{~ endif }
 
 mounts:
 %{~ for idx, partition in extra_partitions }

contrib/terraform/openstack/modules/compute/variables.tf

@@ -104,34 +104,18 @@ variable "bastion_allowed_remote_ips" {
   type = list
 }
 
-variable "bastion_allowed_remote_ipv6_ips" {
-  type = list
-}
-
 variable "master_allowed_remote_ips" {
   type = list
 }
 
-variable "master_allowed_remote_ipv6_ips" {
-  type = list
-}
-
 variable "k8s_allowed_remote_ips" {
   type = list
 }
 
-variable "k8s_allowed_remote_ips_ipv6" {
-  type = list
-}
-
 variable "k8s_allowed_egress_ips" {
   type = list
 }
 
-variable "k8s_allowed_egress_ipv6_ips" {
-  type = list
-}
-
 variable "k8s_masters" {
   type = map(object({
     az                     = string
@@ -158,14 +142,13 @@ variable "k8s_nodes" {
     additional_server_groups = optional(list(string))
     server_group           = optional(string)
     cloudinit              = optional(object({
-      extra_partitions = optional(list(object({
+      extra_partitions = list(object({
         volume_path     = string
         partition_path  = string
         partition_start = string
         partition_end   = string
         mount_path      = string
-      })), [])
+      }))
-      netplan_critical_dhcp_interface = optional(string, "")
     }))
   }))
 }
@@ -188,26 +171,14 @@ variable "master_allowed_ports" {
   type = list
 }
 
-variable "master_allowed_ports_ipv6" {
-  type = list
-}
-
 variable "worker_allowed_ports" {
   type = list
 }
 
-variable "worker_allowed_ports_ipv6" {
-  type = list
-}
-
 variable "bastion_allowed_ports" {
   type = list
 }
 
-variable "bastion_allowed_ports_ipv6" {
-  type = list
-}
-
 variable "use_access_ip" {}
 
 variable "master_server_group_policy" {

contrib/terraform/openstack/modules/loadbalancer/main.tf

@@ -1,54 +0,0 @@
-resource "openstack_lb_loadbalancer_v2" "k8s_lb" {
-  count             = var.k8s_master_loadbalancer_enabled ? 1 : 0
-  name              = "${var.cluster_name}-api-loadbalancer"
-  vip_subnet_id     = var.subnet_id
-}
-
-resource "openstack_lb_listener_v2" "api_listener"{
-  count             = var.k8s_master_loadbalancer_enabled ? 1 : 0
-  name              = "api-listener"
-  protocol          = "TCP"
-  protocol_port     = var.k8s_master_loadbalancer_listener_port
-  loadbalancer_id   = openstack_lb_loadbalancer_v2.k8s_lb[0].id
-  depends_on        = [ openstack_lb_loadbalancer_v2.k8s_lb ]
-}
-
-resource "openstack_lb_pool_v2" "api_pool" {
-  count             = var.k8s_master_loadbalancer_enabled ? 1 : 0
-  name              = "api-pool"
-  protocol          = "TCP"
-  lb_method         = "ROUND_ROBIN"
-  listener_id       = openstack_lb_listener_v2.api_listener[0].id
-  depends_on        = [ openstack_lb_listener_v2.api_listener ]
-}
-
-resource "openstack_lb_member_v2" "lb_member" {
-  count             = var.k8s_master_loadbalancer_enabled ? length(var.k8s_master_ips) : 0
-  name              = var.k8s_master_ips[count.index].name
-  pool_id           = openstack_lb_pool_v2.api_pool[0].id
-  address           = var.k8s_master_ips[count.index].access_ip_v4
-  protocol_port     = var.k8s_master_loadbalancer_server_port
-  depends_on        = [ openstack_lb_pool_v2.api_pool ]
-}
-
-resource "openstack_lb_monitor_v2" "monitor" {
-  count       = var.k8s_master_loadbalancer_enabled ? 1 : 0
-  name        = "Api Monitor"
-  pool_id     = openstack_lb_pool_v2.api_pool[0].id
-  type        = "TCP"
-  delay       = 10
-  timeout     = 5
-  max_retries = 5
-}
-
-resource "openstack_networking_floatingip_v2" "floatip_1" {
-  count = var.k8s_master_loadbalancer_enabled && var.k8s_master_loadbalancer_public_ip == "" ? 1 : 0
-  pool = var.floatingip_pool
-}
-
-resource "openstack_networking_floatingip_associate_v2" "public_ip" {
-  count             = var.k8s_master_loadbalancer_enabled ? 1 : 0
-  floating_ip       = var.k8s_master_loadbalancer_public_ip != "" ? var.k8s_master_loadbalancer_public_ip : openstack_networking_floatingip_v2.floatip_1[0].address
-  port_id           = openstack_lb_loadbalancer_v2.k8s_lb[0].vip_port_id
-  depends_on        = [ openstack_lb_loadbalancer_v2.k8s_lb ]
-}

contrib/terraform/openstack/modules/loadbalancer/variables.tf

@@ -1,15 +0,0 @@
-variable "cluster_name" {}
-
-variable "subnet_id" {}
-
-variable "floatingip_pool" {}
-
-variable "k8s_master_ips" {}
-
-variable "k8s_master_loadbalancer_enabled" {}
-
-variable "k8s_master_loadbalancer_listener_port" {}
-
-variable "k8s_master_loadbalancer_server_port" {}
-
-variable "k8s_master_loadbalancer_public_ip" {}

contrib/terraform/openstack/modules/loadbalancer/versions.tf

@@ -1,8 +0,0 @@
-terraform {
-  required_providers {
-    openstack = {
-      source = "terraform-provider-openstack/openstack"
-    }
-  }
-  required_version = ">= 0.12.26"
-}

contrib/terraform/openstack/variables.tf

@@ -220,60 +220,30 @@ variable "bastion_allowed_remote_ips" {
   default     = ["0.0.0.0/0"]
 }
 
-variable "bastion_allowed_remote_ipv6_ips" {
-  description = "An array of IPv6 CIDRs allowed to SSH to hosts"
-  type        = list(string)
-  default     = ["::/0"]
-}
-
 variable "master_allowed_remote_ips" {
   description = "An array of CIDRs allowed to access API of masters"
   type        = list(string)
   default     = ["0.0.0.0/0"]
 }
 
-variable "master_allowed_remote_ipv6_ips" {
-  description = "An array of IPv6 CIDRs allowed to access API of masters"
-  type        = list(string)
-  default     = ["::/0"]
-}
-
 variable "k8s_allowed_remote_ips" {
   description = "An array of CIDRs allowed to SSH to hosts"
   type        = list(string)
   default     = []
 }
 
-variable "k8s_allowed_remote_ips_ipv6" {
-  description = "An array of IPv6 CIDRs allowed to SSH to hosts"
-  type        = list(string)
-  default     = []
-}
-
 variable "k8s_allowed_egress_ips" {
   description = "An array of CIDRs allowed for egress traffic"
   type        = list(string)
   default     = ["0.0.0.0/0"]
 }
 
-variable "k8s_allowed_egress_ipv6_ips" {
-  description = "An array of CIDRs allowed for egress IPv6 traffic"
-  type        = list(string)
-  default     = ["::/0"]
-}
-
 variable "master_allowed_ports" {
   type = list(any)
 
   default = []
 }
 
-variable "master_allowed_ports_ipv6" {
-  type = list(any)
-
-  default = []
-}
-
 variable "worker_allowed_ports" {
   type = list(any)
 
@@ -287,31 +257,12 @@ variable "worker_allowed_ports" {
   ]
 }
 
-variable "worker_allowed_ports_ipv6" {
-  type = list(any)
-
-  default = [
-    {
-      "protocol"         = "tcp"
-      "port_range_min"   = 30000
-      "port_range_max"   = 32767
-      "remote_ip_prefix" = "::/0"
-    },
-  ]
-}
-
 variable "bastion_allowed_ports" {
   type = list(any)
 
   default = []
 }
 
-variable "bastion_allowed_ports_ipv6" {
-  type = list(any)
-
-  default = []
-}
-
 variable "use_access_ip" {
   default = 1
 }
@@ -389,23 +340,3 @@ variable "group_vars_path" {
   type        = string
   default     = "./group_vars"
 }
-
-variable "k8s_master_loadbalancer_enabled" {
-  type    = bool
-  default = "false"
-}
-
-variable "k8s_master_loadbalancer_listener_port" {
-  type    = string
-  default = "6443"
-}
-
-variable "k8s_master_loadbalancer_server_port" {
-  type    = string
-  default = 6443
-}
-
-variable "k8s_master_loadbalancer_public_ip" {
-  type    = string
-  default = ""
-}

contrib/terraform/upcloud/README.md

@@ -140,4 +140,4 @@ terraform destroy --var-file cluster-settings.tfvars \
   * `backend_servers`: List of servers that traffic to the port should be forwarded to.
 * `server_groups`: Group servers together
   * `servers`: The servers that should be included in the group.
-  * `anti_affinity_policy`: Defines if a server group is an anti-affinity group. Setting this to "strict" or yes" will result in all servers in the group being placed on separate compute hosts. The value can be "strict", "yes" or "no". "strict" refers to strict policy doesn't allow servers in the same server group to be on the same host. "yes" refers to best-effort policy and tries to put servers on different hosts, but this is not guaranteed.
+  * `anti_affinity`: If anti-affinity should be enabled, try to spread the VMs out on separate nodes.

contrib/terraform/upcloud/cluster-settings.tfvars

@@ -18,7 +18,7 @@ ssh_public_keys = [
 
 # check list of available plan https://developers.upcloud.com/1.3/7-plans/
 machines = {
-  "control-plane-0" : {
+  "master-0" : {
     "node_type" : "master",
     # plan to use instead of custom cpu/mem
     "plan" : null,
@@ -116,9 +116,8 @@ k8s_allowed_remote_ips = [
 master_allowed_ports = []
 worker_allowed_ports = []
 
-loadbalancer_enabled        = false
+loadbalancer_enabled = false
-loadbalancer_plan           = "development"
+loadbalancer_plan    = "development"
-loadbalancer_proxy_protocol = false
 loadbalancers = {
   # "http" : {
   #   "port" : 80,
@@ -134,9 +133,9 @@ loadbalancers = {
 server_groups = {
   # "control-plane" = {
   #   servers = [
-  #     "control-plane-0"
+  #     "master-0"
   #   ]
-  #   anti_affinity_policy = "strict"
+  #   anti_affinity = true
   # },
   # "workers" = {
   #   servers = [
@@ -144,6 +143,6 @@ server_groups = {
   #     "worker-1",
   #     "worker-2"
   #   ]
-  #   anti_affinity_policy = "yes"
+  #   anti_affinity = true
   # }
-}
+}

contrib/terraform/upcloud/main.tf

@@ -31,10 +31,9 @@ module "kubernetes" {
   master_allowed_ports      = var.master_allowed_ports
   worker_allowed_ports      = var.worker_allowed_ports
 
-  loadbalancer_enabled                 = var.loadbalancer_enabled
+  loadbalancer_enabled = var.loadbalancer_enabled
-  loadbalancer_plan                    = var.loadbalancer_plan
+  loadbalancer_plan    = var.loadbalancer_plan
-  loadbalancer_outbound_proxy_protocol = var.loadbalancer_proxy_protocol ? "v2" : ""
+  loadbalancers        = var.loadbalancers
-  loadbalancers                        = var.loadbalancers
 
   server_groups = var.server_groups
 }

contrib/terraform/upcloud/modules/kubernetes-cluster/main.tf

@@ -3,7 +3,7 @@ locals {
   disks = flatten([
     for node_name, machine in var.machines : [
       for disk_name, disk in machine.additional_disks : {
-        disk      = disk
+        disk = disk
         disk_name = disk_name
         node_name = node_name
       }
@@ -13,8 +13,8 @@ locals {
   lb_backend_servers = flatten([
     for lb_name, loadbalancer in var.loadbalancers : [
       for backend_server in loadbalancer.backend_servers : {
-        port        = loadbalancer.target_port
+        port = loadbalancer.target_port
-        lb_name     = lb_name
+        lb_name = lb_name
         server_name = backend_server
       }
     ]
@@ -22,7 +22,7 @@ locals {
 
   # If prefix is set, all resources will be prefixed with "${var.prefix}-"
   # Else don't prefix with anything
-  resource-prefix = "%{if var.prefix != ""}${var.prefix}-%{endif}"
+  resource-prefix = "%{ if var.prefix != ""}${var.prefix}-%{ endif }"
 }
 
 resource "upcloud_network" "private" {
@@ -38,7 +38,7 @@ resource "upcloud_network" "private" {
 
 resource "upcloud_storage" "additional_disks" {
   for_each = {
-    for disk in local.disks : "${disk.node_name}_${disk.disk_name}" => disk.disk
+    for disk in local.disks: "${disk.node_name}_${disk.disk_name}" => disk.disk
   }
 
   size  = each.value.size
@@ -61,8 +61,8 @@ resource "upcloud_server" "master" {
   zone     = var.zone
 
   template {
-    storage = var.template_name
+  storage = var.template_name
-    size    = each.value.disk_size
+  size    = each.value.disk_size
   }
 
   # Public network interface
@@ -81,14 +81,14 @@ resource "upcloud_server" "master" {
     ignore_changes = [storage_devices]
   }
 
-  firewall = var.firewall_enabled
+  firewall  = var.firewall_enabled
 
   dynamic "storage_devices" {
     for_each = {
       for disk_key_name, disk in upcloud_storage.additional_disks :
-      disk_key_name => disk
+        disk_key_name => disk
-      # Only add the disk if it matches the node name in the start of its name
+        # Only add the disk if it matches the node name in the start of its name
-      if length(regexall("^${each.key}_.+", disk_key_name)) > 0
+        if length(regexall("^${each.key}_.+", disk_key_name)) > 0
     }
 
     content {
@@ -138,14 +138,14 @@ resource "upcloud_server" "worker" {
     ignore_changes = [storage_devices]
   }
 
-  firewall = var.firewall_enabled
+  firewall  = var.firewall_enabled
 
   dynamic "storage_devices" {
     for_each = {
       for disk_key_name, disk in upcloud_storage.additional_disks :
-      disk_key_name => disk
+        disk_key_name => disk
-      # Only add the disk if it matches the node name in the start of its name
+        # Only add the disk if it matches the node name in the start of its name
-      if length(regexall("^${each.key}_.+", disk_key_name)) > 0
+        if length(regexall("^${each.key}_.+", disk_key_name)) > 0
     }
 
     content {
@@ -162,10 +162,10 @@ resource "upcloud_server" "worker" {
 }
 
 resource "upcloud_firewall_rules" "master" {
-  for_each  = upcloud_server.master
+  for_each = upcloud_server.master
   server_id = each.value.id
 
-  dynamic "firewall_rule" {
+  dynamic firewall_rule {
     for_each = var.master_allowed_remote_ips
 
     content {
@@ -181,7 +181,7 @@ resource "upcloud_firewall_rules" "master" {
     }
   }
 
-  dynamic "firewall_rule" {
+  dynamic firewall_rule {
     for_each = length(var.master_allowed_remote_ips) > 0 ? [1] : []
 
     content {
@@ -197,7 +197,7 @@ resource "upcloud_firewall_rules" "master" {
     }
   }
 
-  dynamic "firewall_rule" {
+  dynamic firewall_rule {
     for_each = var.k8s_allowed_remote_ips
 
     content {
@@ -213,7 +213,7 @@ resource "upcloud_firewall_rules" "master" {
     }
   }
 
-  dynamic "firewall_rule" {
+  dynamic firewall_rule {
     for_each = length(var.k8s_allowed_remote_ips) > 0 ? [1] : []
 
     content {
@@ -229,7 +229,7 @@ resource "upcloud_firewall_rules" "master" {
     }
   }
 
-  dynamic "firewall_rule" {
+  dynamic firewall_rule {
     for_each = var.master_allowed_ports
 
     content {
@@ -245,97 +245,97 @@ resource "upcloud_firewall_rules" "master" {
     }
   }
 
-  dynamic "firewall_rule" {
+  dynamic firewall_rule {
     for_each = var.firewall_default_deny_in ? ["tcp", "udp"] : []
 
     content {
-      action               = "accept"
+      action                 = "accept"
-      comment              = "UpCloud DNS"
+      comment                = "UpCloud DNS"
-      source_port_end      = "53"
+      source_port_end        = "53"
-      source_port_start    = "53"
+      source_port_start      = "53"
-      direction            = "in"
+      direction              = "in"
-      family               = "IPv4"
+      family                 = "IPv4"
-      protocol             = firewall_rule.value
+      protocol               = firewall_rule.value
-      source_address_end   = "94.237.40.9"
+      source_address_end     = "94.237.40.9"
-      source_address_start = "94.237.40.9"
+      source_address_start   = "94.237.40.9"
     }
   }
 
-  dynamic "firewall_rule" {
+  dynamic firewall_rule {
     for_each = var.firewall_default_deny_in ? ["tcp", "udp"] : []
 
     content {
-      action               = "accept"
+      action                 = "accept"
-      comment              = "UpCloud DNS"
+      comment                = "UpCloud DNS"
-      source_port_end      = "53"
+      source_port_end        = "53"
-      source_port_start    = "53"
+      source_port_start      = "53"
-      direction            = "in"
+      direction              = "in"
-      family               = "IPv4"
+      family                 = "IPv4"
-      protocol             = firewall_rule.value
+      protocol               = firewall_rule.value
-      source_address_end   = "94.237.127.9"
+      source_address_end     = "94.237.127.9"
-      source_address_start = "94.237.127.9"
+      source_address_start   = "94.237.127.9"
     }
   }
 
-  dynamic "firewall_rule" {
+  dynamic firewall_rule {
     for_each = var.firewall_default_deny_in ? ["tcp", "udp"] : []
 
     content {
-      action               = "accept"
+      action                 = "accept"
-      comment              = "UpCloud DNS"
+      comment                = "UpCloud DNS"
-      source_port_end      = "53"
+      source_port_end        = "53"
-      source_port_start    = "53"
+      source_port_start      = "53"
-      direction            = "in"
+      direction              = "in"
-      family               = "IPv6"
+      family                 = "IPv6"
-      protocol             = firewall_rule.value
+      protocol               = firewall_rule.value
-      source_address_end   = "2a04:3540:53::1"
+      source_address_end     = "2a04:3540:53::1"
-      source_address_start = "2a04:3540:53::1"
+      source_address_start   = "2a04:3540:53::1"
     }
   }
 
-  dynamic "firewall_rule" {
+  dynamic firewall_rule {
     for_each = var.firewall_default_deny_in ? ["tcp", "udp"] : []
 
     content {
-      action               = "accept"
+      action                 = "accept"
-      comment              = "UpCloud DNS"
+      comment                = "UpCloud DNS"
-      source_port_end      = "53"
+      source_port_end        = "53"
-      source_port_start    = "53"
+      source_port_start      = "53"
-      direction            = "in"
+      direction              = "in"
-      family               = "IPv6"
+      family                 = "IPv6"
-      protocol             = firewall_rule.value
+      protocol               = firewall_rule.value
-      source_address_end   = "2a04:3544:53::1"
+      source_address_end     = "2a04:3544:53::1"
-      source_address_start = "2a04:3544:53::1"
+      source_address_start   = "2a04:3544:53::1"
     }
   }
 
-  dynamic "firewall_rule" {
+  dynamic firewall_rule {
     for_each = var.firewall_default_deny_in ? ["udp"] : []
 
     content {
-      action               = "accept"
+      action                 = "accept"
-      comment              = "NTP Port"
+      comment                = "NTP Port"
-      source_port_end      = "123"
+      source_port_end        = "123"
-      source_port_start    = "123"
+      source_port_start      = "123"
-      direction            = "in"
+      direction              = "in"
-      family               = "IPv4"
+      family                 = "IPv4"
-      protocol             = firewall_rule.value
+      protocol               = firewall_rule.value
-      source_address_end   = "255.255.255.255"
+      source_address_end     = "255.255.255.255"
-      source_address_start = "0.0.0.0"
+      source_address_start   = "0.0.0.0"
     }
   }
 
-  dynamic "firewall_rule" {
+  dynamic firewall_rule {
     for_each = var.firewall_default_deny_in ? ["udp"] : []
 
     content {
-      action            = "accept"
+      action                 = "accept"
-      comment           = "NTP Port"
+      comment                = "NTP Port"
-      source_port_end   = "123"
+      source_port_end        = "123"
-      source_port_start = "123"
+      source_port_start      = "123"
-      direction         = "in"
+      direction              = "in"
-      family            = "IPv6"
+      family                 = "IPv6"
-      protocol          = firewall_rule.value
+      protocol               = firewall_rule.value
     }
   }
 
@@ -351,10 +351,10 @@ resource "upcloud_firewall_rules" "master" {
 }
 
 resource "upcloud_firewall_rules" "k8s" {
-  for_each  = upcloud_server.worker
+  for_each = upcloud_server.worker
   server_id = each.value.id
 
-  dynamic "firewall_rule" {
+  dynamic firewall_rule {
     for_each = var.k8s_allowed_remote_ips
 
     content {
@@ -370,7 +370,7 @@ resource "upcloud_firewall_rules" "k8s" {
     }
   }
 
-  dynamic "firewall_rule" {
+  dynamic firewall_rule {
     for_each = length(var.k8s_allowed_remote_ips) > 0 ? [1] : []
 
     content {
@@ -386,7 +386,7 @@ resource "upcloud_firewall_rules" "k8s" {
     }
   }
 
-  dynamic "firewall_rule" {
+  dynamic firewall_rule {
     for_each = var.worker_allowed_ports
 
     content {
@@ -402,97 +402,97 @@ resource "upcloud_firewall_rules" "k8s" {
     }
   }
 
-  dynamic "firewall_rule" {
+  dynamic firewall_rule {
     for_each = var.firewall_default_deny_in ? ["tcp", "udp"] : []
 
     content {
-      action               = "accept"
+      action                 = "accept"
-      comment              = "UpCloud DNS"
+      comment                = "UpCloud DNS"
-      source_port_end      = "53"
+      source_port_end        = "53"
-      source_port_start    = "53"
+      source_port_start      = "53"
-      direction            = "in"
+      direction              = "in"
-      family               = "IPv4"
+      family                 = "IPv4"
-      protocol             = firewall_rule.value
+      protocol               = firewall_rule.value
-      source_address_end   = "94.237.40.9"
+      source_address_end     = "94.237.40.9"
-      source_address_start = "94.237.40.9"
+      source_address_start   = "94.237.40.9"
     }
   }
 
-  dynamic "firewall_rule" {
+  dynamic firewall_rule {
     for_each = var.firewall_default_deny_in ? ["tcp", "udp"] : []
 
     content {
-      action               = "accept"
+      action                 = "accept"
-      comment              = "UpCloud DNS"
+      comment                = "UpCloud DNS"
-      source_port_end      = "53"
+      source_port_end        = "53"
-      source_port_start    = "53"
+      source_port_start      = "53"
-      direction            = "in"
+      direction              = "in"
-      family               = "IPv4"
+      family                 = "IPv4"
-      protocol             = firewall_rule.value
+      protocol               = firewall_rule.value
-      source_address_end   = "94.237.127.9"
+      source_address_end     = "94.237.127.9"
-      source_address_start = "94.237.127.9"
+      source_address_start   = "94.237.127.9"
     }
   }
 
-  dynamic "firewall_rule" {
+  dynamic firewall_rule {
     for_each = var.firewall_default_deny_in ? ["tcp", "udp"] : []
 
     content {
-      action               = "accept"
+      action                 = "accept"
-      comment              = "UpCloud DNS"
+      comment                = "UpCloud DNS"
-      source_port_end      = "53"
+      source_port_end        = "53"
-      source_port_start    = "53"
+      source_port_start      = "53"
-      direction            = "in"
+      direction              = "in"
-      family               = "IPv6"
+      family                 = "IPv6"
-      protocol             = firewall_rule.value
+      protocol               = firewall_rule.value
-      source_address_end   = "2a04:3540:53::1"
+      source_address_end     = "2a04:3540:53::1"
-      source_address_start = "2a04:3540:53::1"
+      source_address_start   = "2a04:3540:53::1"
     }
   }
 
-  dynamic "firewall_rule" {
+  dynamic firewall_rule {
     for_each = var.firewall_default_deny_in ? ["tcp", "udp"] : []
 
     content {
-      action               = "accept"
+      action                 = "accept"
-      comment              = "UpCloud DNS"
+      comment                = "UpCloud DNS"
-      source_port_end      = "53"
+      source_port_end        = "53"
-      source_port_start    = "53"
+      source_port_start      = "53"
-      direction            = "in"
+      direction              = "in"
-      family               = "IPv6"
+      family                 = "IPv6"
-      protocol             = firewall_rule.value
+      protocol               = firewall_rule.value
-      source_address_end   = "2a04:3544:53::1"
+      source_address_end     = "2a04:3544:53::1"
-      source_address_start = "2a04:3544:53::1"
+      source_address_start   = "2a04:3544:53::1"
     }
   }
 
-  dynamic "firewall_rule" {
+  dynamic firewall_rule {
     for_each = var.firewall_default_deny_in ? ["udp"] : []
 
     content {
-      action               = "accept"
+      action                 = "accept"
-      comment              = "NTP Port"
+      comment                = "NTP Port"
-      source_port_end      = "123"
+      source_port_end        = "123"
-      source_port_start    = "123"
+      source_port_start      = "123"
-      direction            = "in"
+      direction              = "in"
-      family               = "IPv4"
+      family                 = "IPv4"
-      protocol             = firewall_rule.value
+      protocol               = firewall_rule.value
-      source_address_end   = "255.255.255.255"
+      source_address_end     = "255.255.255.255"
-      source_address_start = "0.0.0.0"
+      source_address_start   = "0.0.0.0"
     }
   }
 
-  dynamic "firewall_rule" {
+  dynamic firewall_rule {
     for_each = var.firewall_default_deny_in ? ["udp"] : []
 
     content {
-      action            = "accept"
+      action                 = "accept"
-      comment           = "NTP Port"
+      comment                = "NTP Port"
-      source_port_end   = "123"
+      source_port_end        = "123"
-      source_port_start = "123"
+      source_port_start      = "123"
-      direction         = "in"
+      direction              = "in"
-      family            = "IPv6"
+      family                 = "IPv6"
-      protocol          = firewall_rule.value
+      protocol               = firewall_rule.value
     }
   }
 
@@ -521,9 +521,6 @@ resource "upcloud_loadbalancer_backend" "lb_backend" {
 
   loadbalancer = upcloud_loadbalancer.lb[0].id
   name         = "lb-backend-${each.key}"
-  properties {
-    outbound_proxy_protocol = var.loadbalancer_outbound_proxy_protocol
-  }
 }
 
 resource "upcloud_loadbalancer_frontend" "lb_frontend" {
@@ -538,9 +535,9 @@ resource "upcloud_loadbalancer_frontend" "lb_frontend" {
 
 resource "upcloud_loadbalancer_static_backend_member" "lb_backend_member" {
   for_each = {
-    for be_server in local.lb_backend_servers :
+    for be_server in local.lb_backend_servers:
-    "${be_server.server_name}-lb-backend-${be_server.lb_name}" => be_server
+      "${be_server.server_name}-lb-backend-${be_server.lb_name}" => be_server
-    if var.loadbalancer_enabled
+      if var.loadbalancer_enabled
   }
 
   backend      = upcloud_loadbalancer_backend.lb_backend[each.value.lb_name].id
@@ -553,9 +550,9 @@ resource "upcloud_loadbalancer_static_backend_member" "lb_backend_member" {
 }
 
 resource "upcloud_server_group" "server_groups" {
-  for_each             = var.server_groups
+  for_each      = var.server_groups
-  title                = each.key
+  title         = each.key
-  anti_affinity_policy = each.value.anti_affinity_policy
+  anti_affinity = each.value.anti_affinity
-  labels               = {}
+  labels        = {}
-  members              = [for server in each.value.servers : merge(upcloud_server.master, upcloud_server.worker)[server].id]
+  members       = [for server in each.value.servers : merge(upcloud_server.master, upcloud_server.worker)[server].id]
-}
+}

contrib/terraform/upcloud/modules/kubernetes-cluster/output.tf

@@ -3,8 +3,8 @@ output "master_ip" {
   value = {
     for instance in upcloud_server.master :
     instance.hostname => {
-      "public_ip" : instance.network_interface[0].ip_address
+      "public_ip": instance.network_interface[0].ip_address
-      "private_ip" : instance.network_interface[1].ip_address
+      "private_ip": instance.network_interface[1].ip_address
     }
   }
 }
@@ -13,8 +13,8 @@ output "worker_ip" {
   value = {
     for instance in upcloud_server.worker :
     instance.hostname => {
-      "public_ip" : instance.network_interface[0].ip_address
+      "public_ip": instance.network_interface[0].ip_address
-      "private_ip" : instance.network_interface[1].ip_address
+      "private_ip": instance.network_interface[1].ip_address
     }
   }
 }

contrib/terraform/upcloud/modules/kubernetes-cluster/variables.tf

@@ -15,11 +15,11 @@ variable "private_network_cidr" {}
 variable "machines" {
   description = "Cluster machines"
   type = map(object({
-    node_type = string
+    node_type       = string
-    plan      = string
+    plan            = string
-    cpu       = string
+    cpu             = string
-    mem       = string
+    mem             = string
-    disk_size = number
+    disk_size       =  number
     additional_disks = map(object({
       size = number
       tier = string
@@ -85,10 +85,6 @@ variable "loadbalancer_plan" {
   type = string
 }
 
-variable "loadbalancer_outbound_proxy_protocol" {
-  type = string
-}
-
 variable "loadbalancers" {
   description = "Load balancers"
 
@@ -103,7 +99,7 @@ variable "server_groups" {
   description = "Server groups"
 
   type = map(object({
-    anti_affinity_policy = string
+    anti_affinity   = bool
-    servers              = list(string)
+    servers = list(string)
   }))
-}
+}

contrib/terraform/upcloud/modules/kubernetes-cluster/versions.tf

@@ -2,8 +2,8 @@
 terraform {
   required_providers {
     upcloud = {
-      source  = "UpCloudLtd/upcloud"
+      source = "UpCloudLtd/upcloud"
-      version = "~>2.12.0"
+      version = "~>2.7.1"
     }
   }
   required_version = ">= 0.13"

contrib/terraform/upcloud/sample-inventory/cluster.tfvars

@@ -18,7 +18,7 @@ ssh_public_keys = [
 
 # check list of available plan https://developers.upcloud.com/1.3/7-plans/
 machines = {
-  "control-plane-0" : {
+  "master-0" : {
     "node_type" : "master",
     # plan to use instead of custom cpu/mem
     "plan" : null,
@@ -28,7 +28,7 @@ machines = {
     "mem" : "4096"
     # The size of the storage in GB
     "disk_size" : 250
-    "additional_disks" : {}
+    "additional_disks": {}
   },
   "worker-0" : {
     "node_type" : "worker",
@@ -40,7 +40,7 @@ machines = {
     "mem" : "4096"
     # The size of the storage in GB
     "disk_size" : 250
-    "additional_disks" : {
+    "additional_disks": {
       # "some-disk-name-1": {
       #   "size": 100,
       #   "tier": "maxiops",
@@ -61,7 +61,7 @@ machines = {
     "mem" : "4096"
     # The size of the storage in GB
     "disk_size" : 250
-    "additional_disks" : {
+    "additional_disks": {
       # "some-disk-name-1": {
       #   "size": 100,
       #   "tier": "maxiops",
@@ -82,7 +82,7 @@ machines = {
     "mem" : "4096"
     # The size of the storage in GB
     "disk_size" : 250
-    "additional_disks" : {
+    "additional_disks": {
       # "some-disk-name-1": {
       #   "size": 100,
       #   "tier": "maxiops",
@@ -118,7 +118,7 @@ master_allowed_ports = []
 worker_allowed_ports = []
 
 loadbalancer_enabled = false
-loadbalancer_plan    = "development"
+loadbalancer_plan = "development"
 loadbalancers = {
   # "http" : {
   #   "port" : 80,
@@ -134,9 +134,9 @@ loadbalancers = {
 server_groups = {
   # "control-plane" = {
   #   servers = [
-  #     "control-plane-0"
+  #     "master-0"
   #   ]
-  #   anti_affinity_policy = "strict"
+  #   anti_affinity = true
   # },
   # "workers" = {
   #   servers = [
@@ -144,6 +144,6 @@ server_groups = {
   #     "worker-1",
   #     "worker-2"
   #   ]
-  #   anti_affinity_policy = "yes"
+  #   anti_affinity = true
   # }
-}
+}

contrib/terraform/upcloud/variables.tf

@@ -121,11 +121,6 @@ variable "loadbalancer_plan" {
   default     = "development"
 }
 
-variable "loadbalancer_proxy_protocol" {
-  type    = bool
-  default = false
-}
-
 variable "loadbalancers" {
   description = "Load balancers"
 
@@ -141,8 +136,8 @@ variable "server_groups" {
   description = "Server groups"
 
   type = map(object({
-    anti_affinity_policy = string
+    anti_affinity = bool
-    servers              = list(string)
+    servers       = list(string)
   }))
 
   default = {}

contrib/terraform/upcloud/versions.tf

@@ -3,7 +3,7 @@ terraform {
   required_providers {
     upcloud = {
       source  = "UpCloudLtd/upcloud"
-      version = "~>2.12.0"
+      version = "~>2.7.1"
     }
   }
   required_version = ">= 0.13"

docs/_sidebar.md

@@ -1,94 +1,66 @@
 * [Readme](/)
-* Advanced
+* [Comparisons](/docs/comparisons.md)
-  * [Arch](/docs/advanced/arch.md)
+* [Getting started](/docs/getting-started.md)
-  * [Cert Manager](/docs/advanced/cert_manager.md)
+* [Ansible](docs/ansible.md)
-  * [Dns-stack](/docs/advanced/dns-stack.md)
+* [Variables](/docs/vars.md)
-  * [Downloads](/docs/advanced/downloads.md)
-  * [Gcp-lb](/docs/advanced/gcp-lb.md)
-  * [Kubernetes-reliability](/docs/advanced/kubernetes-reliability.md)
-  * [Mitogen](/docs/advanced/mitogen.md)
-  * [Netcheck](/docs/advanced/netcheck.md)
-  * [Ntp](/docs/advanced/ntp.md)
-  * [Proxy](/docs/advanced/proxy.md)
-  * [Registry](/docs/advanced/registry.md)
-* Ansible
-  * [Ansible](/docs/ansible/ansible.md)
-  * [Ansible Collection](/docs/ansible/ansible_collection.md)
-  * [Vars](/docs/ansible/vars.md)
-* Cloud Providers
-  * [Aws](/docs/cloud_providers/aws.md)
-  * [Azure](/docs/cloud_providers/azure.md)
-  * [Cloud](/docs/cloud_providers/cloud.md)
-  * [Equinix-metal](/docs/cloud_providers/equinix-metal.md)
-  * [Openstack](/docs/cloud_providers/openstack.md)
-  * [Vsphere](/docs/cloud_providers/vsphere.md)
-* CNI
-  * [Calico](/docs/CNI/calico.md)
-  * [Cilium](/docs/CNI/cilium.md)
-  * [Cni](/docs/CNI/cni.md)
-  * [Flannel](/docs/CNI/flannel.md)
-  * [Kube-ovn](/docs/CNI/kube-ovn.md)
-  * [Kube-router](/docs/CNI/kube-router.md)
-  * [Macvlan](/docs/CNI/macvlan.md)
-  * [Multus](/docs/CNI/multus.md)
-  * [Weave](/docs/CNI/weave.md)
-* CRI
-  * [Containerd](/docs/CRI/containerd.md)
-  * [Cri-o](/docs/CRI/cri-o.md)
-  * [Docker](/docs/CRI/docker.md)
-  * [Gvisor](/docs/CRI/gvisor.md)
-  * [Kata-containers](/docs/CRI/kata-containers.md)
-* CSI
-  * [Aws-ebs-csi](/docs/CSI/aws-ebs-csi.md)
-  * [Azure-csi](/docs/CSI/azure-csi.md)
-  * [Cinder-csi](/docs/CSI/cinder-csi.md)
-  * [Gcp-pd-csi](/docs/CSI/gcp-pd-csi.md)
-  * [Vsphere-csi](/docs/CSI/vsphere-csi.md)
-* Developers
-  * [Ci-setup](/docs/developers/ci-setup.md)
-  * [Ci](/docs/developers/ci.md)
-  * [Test Cases](/docs/developers/test_cases.md)
-  * [Vagrant](/docs/developers/vagrant.md)
-* External Storage Provisioners
-  * [Cephfs Provisioner](/docs/external_storage_provisioners/cephfs_provisioner.md)
-  * [Local Volume Provisioner](/docs/external_storage_provisioners/local_volume_provisioner.md)
-  * [Rbd Provisioner](/docs/external_storage_provisioners/rbd_provisioner.md)
-  * [Scheduler Plugins](/docs/external_storage_provisioners/scheduler_plugins.md)
-* Getting Started
-  * [Comparisons](/docs/getting_started/comparisons.md)
-  * [Getting-started](/docs/getting_started/getting-started.md)
-  * [Setting-up-your-first-cluster](/docs/getting_started/setting-up-your-first-cluster.md)
-* Ingress
-  * [Alb Ingress Controller](/docs/ingress/alb_ingress_controller.md)
-  * [Ingress Nginx](/docs/ingress/ingress_nginx.md)
-  * [Kube-vip](/docs/ingress/kube-vip.md)
-  * [Metallb](/docs/ingress/metallb.md)
-* Operating Systems
-  * [Amazonlinux](/docs/operating_systems/amazonlinux.md)
-  * [Bootstrap-os](/docs/operating_systems/bootstrap-os.md)
-  * [Centos](/docs/operating_systems/centos.md)
-  * [Fcos](/docs/operating_systems/fcos.md)
-  * [Flatcar](/docs/operating_systems/flatcar.md)
-  * [Kylinlinux](/docs/operating_systems/kylinlinux.md)
-  * [Openeuler](/docs/operating_systems/openeuler.md)
-  * [Opensuse](/docs/operating_systems/opensuse.md)
-  * [Rhel](/docs/operating_systems/rhel.md)
-  * [Uoslinux](/docs/operating_systems/uoslinux.md)
 * Operations
-  * [Cgroups](/docs/operations/cgroups.md)
+  * [Integration](docs/integration.md)
-  * [Encrypting-secret-data-at-rest](/docs/operations/encrypting-secret-data-at-rest.md)
+  * [Upgrades](/docs/upgrades.md)
-  * [Etcd](/docs/operations/etcd.md)
+  * [HA Mode](docs/ha-mode.md)
-  * [Ha-mode](/docs/operations/ha-mode.md)
+  * [Adding/replacing a node](docs/nodes.md)
-  * [Hardening](/docs/operations/hardening.md)
+  * [Large deployments](docs/large-deployments.md)
-  * [Integration](/docs/operations/integration.md)
+  * [Air-Gap Installation](docs/offline-environment.md)
-  * [Large-deployments](/docs/operations/large-deployments.md)
+* CNI
-  * [Mirror](/docs/operations/mirror.md)
+  * [Calico](docs/calico.md)
-  * [Nodes](/docs/operations/nodes.md)
+  * [Flannel](docs/flannel.md)
-  * [Offline-environment](/docs/operations/offline-environment.md)
+  * [Kube Router](docs/kube-router.md)
-  * [Port-requirements](/docs/operations/port-requirements.md)
+  * [Kube OVN](docs/kube-ovn.md)
-  * [Recover-control-plane](/docs/operations/recover-control-plane.md)
+  * [Weave](docs/weave.md)
-  * [Upgrades](/docs/operations/upgrades.md)
+  * [Multus](docs/multus.md)
-* Roadmap
+* Ingress
-  * [Roadmap](/docs/roadmap/roadmap.md)
+  * [kube-vip](docs/kube-vip.md)
-* Upgrades
+  * [ALB Ingress](docs/ingress_controller/alb_ingress_controller.md)
-  * [Migrate Docker2containerd](/docs/upgrades/migrate_docker2containerd.md)
+  * [MetalLB](docs/metallb.md)
+  * [Nginx Ingress](docs/ingress_controller/ingress_nginx.md)
+* [Cloud providers](docs/cloud.md)
+  * [AWS](docs/aws.md)
+  * [Azure](docs/azure.md)
+  * [OpenStack](/docs/openstack.md)
+  * [Equinix Metal](/docs/equinix-metal.md)
+  * [vSphere](/docs/vsphere.md)
+* [Operating Systems](docs/bootstrap-os.md)
+  * [Debian](docs/debian.md)
+  * [Flatcar Container Linux](docs/flatcar.md)
+  * [Fedora CoreOS](docs/fcos.md)
+  * [OpenSUSE](docs/opensuse.md)
+  * [RedHat Enterprise Linux](docs/rhel.md)
+  * [CentOS/OracleLinux/AlmaLinux/Rocky Linux](docs/centos.md)
+  * [Kylin Linux Advanced Server V10](docs/kylinlinux.md)
+  * [Amazon Linux 2](docs/amazonlinux.md)
+  * [UOS Linux](docs/uoslinux.md)
+  * [openEuler notes](docs/openeuler.md)
+* CRI
+  * [Containerd](docs/containerd.md)
+  * [Docker](docs/docker.md)
+  * [CRI-O](docs/cri-o.md)
+  * [Kata Containers](docs/kata-containers.md)
+  * [gVisor](docs/gvisor.md)
+* Advanced
+  * [Proxy](/docs/proxy.md)
+  * [Downloads](docs/downloads.md)
+  * [Netcheck](docs/netcheck.md)
+  * [Cert Manager](docs/cert_manager.md)
+  * [DNS Stack](docs/dns-stack.md)
+  * [Kubernetes reliability](docs/kubernetes-reliability.md)
+  * [Local Registry](docs/kubernetes-apps/registry.md)
+  * [NTP](docs/ntp.md)
+* External Storage Provisioners
+  * [RBD Provisioner](docs/kubernetes-apps/rbd_provisioner.md)
+  * [CEPHFS Provisioner](docs/kubernetes-apps/cephfs_provisioner.md)
+  * [Local Volume Provisioner](docs/kubernetes-apps/local_volume_provisioner.md)
+* Developers
+  * [Test cases](docs/test_cases.md)
+  * [Vagrant](docs/vagrant.md)
+  * [CI Matrix](docs/ci.md)
+  * [CI Setup](docs/ci-setup.md)
+* [Roadmap](docs/roadmap.md)

docs/ansible.md

@@ -32,7 +32,7 @@ Based on the table below and the available python version for your ansible host
 
 | Ansible Version | Python Version |
 |-----------------|----------------|
-| >= 2.16.4       | 3.10-3.12      |
+| 2.14            | 3.9-3.11       |
 
 ## Inventory
 
@@ -59,7 +59,7 @@ not _kube_node_.
 
 There are also two special groups:
 
-* **calico_rr** : explained for [advanced Calico networking cases](/docs/CNI/calico.md)
+* **calico_rr** : explained for [advanced Calico networking cases](/docs/calico.md)
 * **bastion** : configure a bastion host if your nodes are not directly reachable
 
 Below is a complete inventory example:
@@ -285,7 +285,7 @@ For more information about Ansible and bastion hosts, read
 
 ## Mitogen
 
-Mitogen support is deprecated, please see [mitogen related docs](/docs/advanced/mitogen.md) for usage and reasons for deprecation.
+Mitogen support is deprecated, please see [mitogen related docs](/docs/mitogen.md) for usage and reasons for deprecation.
 
 ## Beyond ansible 2.9
 

docs/calico.md

@@ -222,14 +222,6 @@ calico_node_livenessprobe_timeout: 10
 calico_node_readinessprobe_timeout: 10
 ```
 
-### Optional :  Enable NAT with IPv6
-
-To allow outgoing IPv6 traffic going from pods to the Internet, enable the following:
-
-```yml
-nat_outgoing_ipv6: true  # NAT outgoing ipv6 (default value: false).
-```
-
 ## Config encapsulation for cross server traffic
 
 Calico supports two types of encapsulation: [VXLAN and IP in IP](https://docs.projectcalico.org/v3.11/networking/vxlan-ipip). VXLAN is the more mature implementation and enabled by default, please check your environment if you need *IP in IP* encapsulation.
@@ -243,7 +235,7 @@ If you are running your cluster with the default calico settings and are upgradi
 * perform a manual migration to vxlan before upgrading kubespray (see migrating from IP in IP to VXLAN below)
 * pin the pre-2.19 settings in your ansible inventory (see IP in IP mode settings below)
 
-**Note:**: Vxlan in ipv6 only supported when kernel >= 3.12. So if your kernel version < 3.12, Please don't set `calico_vxlan_mode_ipv6: Always`. More details see [#Issue 6877](https://github.com/projectcalico/calico/issues/6877).
+**Note:**: Vxlan in ipv6 only supported when kernel >= 3.12. So if your kernel version < 3.12, Please don't set `calico_vxlan_mode_ipv6: vxlanAlways`. More details see [#Issue 6877](https://github.com/projectcalico/calico/issues/6877).
 
 ### IP in IP mode
 
@@ -382,7 +374,7 @@ To clean up any ipvs leftovers:
 
 Calico node, typha and kube-controllers need to be able to talk to the kubernetes API. Please reference the [Enabling eBPF Calico Docs](https://docs.projectcalico.org/maintenance/ebpf/enabling-bpf) for guidelines on how to do this.
 
-Kubespray sets up the `kubernetes-services-endpoint` configmap based on the contents of the `loadbalancer_apiserver` inventory variable documented in [HA Mode](/docs/operations/ha-mode.md).
+Kubespray sets up the `kubernetes-services-endpoint` configmap based on the contents of the `loadbalancer_apiserver` inventory variable documented in [HA Mode](/docs/ha-mode.md).
 
 If no external loadbalancer is used, Calico eBPF can also use the localhost loadbalancer option. We are able to do so only if you use the same port for the localhost apiserver loadbalancer and the kube-apiserver. In this case Calico Automatic Host Endpoints need to be enabled to allow services like `coredns` and `metrics-server` to communicate with the kubernetes host endpoint. See [this blog post](https://www.projectcalico.org/securing-kubernetes-nodes-with-calico-automatic-host-endpoints/) on enabling automatic host endpoints.
 

docs/centos.md

@@ -7,6 +7,10 @@ Kubespray supports multiple ansible versions but only the default (5.x) gets wid
 
 ## CentOS 8
 
+CentOS 8 / Oracle Linux 8,9 / AlmaLinux 8,9 / Rocky Linux 8,9 ship only with iptables-nft (ie without iptables-legacy similar to RHEL8)
+The only tested configuration for now is using Calico CNI
+You need to add `calico_iptables_backend: "NFT"` to your configuration.
+
 If you have containers that are using iptables in the host network namespace (`hostNetwork=true`),
 you need to ensure they are using iptables-nft.
 An example how k8s do the autodetection can be found [in this PR](https://github.com/kubernetes/kubernetes/pull/82966)

docs/ci.md

@@ -11,7 +11,7 @@ amazon |  :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
 centos7 |  :white_check_mark: | :x: | :x: | :white_check_mark: | :x: | :white_check_mark: | :x: | :white_check_mark: |
 debian10 |  :white_check_mark: | :white_check_mark: | :x: | :x: | :x: | :x: | :white_check_mark: | :x: |
 debian11 |  :white_check_mark: | :x: | :white_check_mark: | :x: | :x: | :x: | :x: | :x: |
-debian12 |  :white_check_mark: | :white_check_mark: | :white_check_mark: | :x: | :x: | :x: | :x: | :x: |
+debian12 |  :white_check_mark: | :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: |
 fedora37 |  :white_check_mark: | :x: | :x: | :x: | :x: | :white_check_mark: | :x: | :x: |
 fedora38 |  :x: | :x: | :x: | :x: | :white_check_mark: | :x: | :x: | :x: |
 opensuse |  :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
@@ -19,7 +19,6 @@ rockylinux8 |  :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
 rockylinux9 |  :white_check_mark: | :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: |
 ubuntu20 |  :white_check_mark: | :white_check_mark: | :x: | :white_check_mark: | :x: | :white_check_mark: | :x: | :white_check_mark: |
 ubuntu22 |  :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
-ubuntu24 |  :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
 
 ## crio
 
@@ -38,7 +37,6 @@ rockylinux8 |  :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
 rockylinux9 |  :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
 ubuntu20 |  :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
 ubuntu22 |  :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
-ubuntu24 |  :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
 
 ## docker
 
@@ -57,4 +55,3 @@ rockylinux8 |  :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
 rockylinux9 |  :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
 ubuntu20 |  :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: | :white_check_mark: |
 ubuntu22 |  :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
-ubuntu24 |  :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |

docs/cilium.md

@@ -141,7 +141,7 @@ cilium_encryption_enabled: true
 cilium_encryption_type: "ipsec"
 ```
 
-The third variable is `cilium_ipsec_key`. You need to create a secret key string for this variable.
+The third variable is `cilium_ipsec_key.` You need to create a secret key string for this variable.
 Kubespray does not automate this process.
 Cilium documentation currently recommends creating a key using the following command:
 
@@ -149,11 +149,7 @@ Cilium documentation currently recommends creating a key using the following com
 echo "3 rfc4106(gcm(aes)) $(echo $(dd if=/dev/urandom count=20 bs=1 2> /dev/null | xxd -p -c 64)) 128"
 ```
 
-Note that Kubespray handles secret creation. So you only need to pass the key as the `cilium_ipsec_key` variable, base64 encoded:
+Note that Kubespray handles secret creation. So you only need to pass the key as the `cilium_ipsec_key` variable.
-
-```shell
-echo "cilium_ipsec_key: "$(echo -n "3 rfc4106(gcm(aes)) $(echo $(dd if=/dev/urandom count=20 bs=1 2> /dev/null | xxd -p -c 64)) 128" | base64 -w0)
-```
 
 ### Wireguard Encryption
 

docs/containerd.md

@@ -35,24 +35,15 @@ containerd_registries_mirrors:
         skip_verify: false
 ```
 
-containerd falls back to `https://{{ prefix }}` when none of the mirrors have the image.
+`containerd_registries_mirrors` is ignored for pulling images when `image_command_tool=nerdctl`
-This can be changed with the [`server` field](https://github.com/containerd/containerd/blob/main/docs/hosts.md#server-field):
+(the default for `container_manager=containerd`). Use `crictl` instead, it supports
+`containerd_registries_mirrors` but lacks proper multi-arch support (see
+[#8375](https://github.com/kubernetes-sigs/kubespray/issues/8375)):
 
 ```yaml
-containerd_registries_mirrors:
+image_command_tool: crictl
-  - prefix: docker.io
-    mirrors:
-      - host: https://mirror.gcr.io
-        capabilities: ["pull", "resolve"]
-        skip_verify: false
-      - host: https://registry-1.docker.io
-        capabilities: ["pull", "resolve"]
-        skip_verify: false
-    server: https://mirror.example.org
 ```
 
-The `containerd_registries` and `containerd_insecure_registries` configs are deprecated.
-
 ### Containerd Runtimes
 
 Containerd supports multiple runtime configurations that can be used with

docs/cri-o.md

@@ -42,22 +42,6 @@ crio_registries:
 
 [CRI-O]: https://cri-o.io/
 
-The following is a method to enable insecure registries.
-
-```yaml
-crio_insecure_registries:
-  - 10.0.0.2:5000
-```
-
-And you can config authentication for these registries after `crio_insecure_registries`.
-
-```yaml
-crio_registry_auth:
-  - registry: 10.0.0.2:5000
-    username: user
-    password: pass
-```
-
 ## Note about user namespaces
 
 CRI-O has support for user namespaces. This feature is optional and can be enabled by setting the following two variables.

docs/debian.md

@@ -0,0 +1,41 @@
+# Debian Jessie
+
+Debian Jessie installation Notes:
+
+- Add
+
+  ```ini
+  GRUB_CMDLINE_LINUX="cgroup_enable=memory swapaccount=1"
+  ```
+
+  to `/etc/default/grub`. Then update with
+
+  ```ShellSession
+  sudo update-grub
+  sudo update-grub2
+  sudo reboot
+  ```
+
+- Add the [backports](https://backports.debian.org/Instructions/) which contain Systemd 2.30 and update Systemd.
+
+  ```ShellSession
+  apt-get -t jessie-backports install systemd
+  ```
+
+  (Necessary because the default Systemd version (2.15) does not support the "Delegate" directive in service files)
+
+- Add the Ansible repository and install Ansible to get a proper version
+
+  ```ShellSession
+  sudo add-apt-repository ppa:ansible/ansible
+  sudo apt-get update
+  sudo apt-get install ansible
+  ```
+
+- Install Jinja2 and Python-Netaddr
+
+  ```ShellSession
+  sudo apt-get install python-jinja2=2.8-1~bpo8+1 python-netaddr
+  ```
+
+Now you can continue with [Preparing your deployment](getting-started.md#starting-custom-deployment)

docs/dns-stack.md

@@ -143,22 +143,6 @@ coredns_default_zone_cache_block: |
   }
 ```
 
-### Handle old/extra dns_domains
-
-If you need to change the dns_domain of your cluster for whatever reason (switching to or from `cluster.local` for example),
-and you have workloads that embed it in their configuration you can use the variable `old_dns_domains`.
-This will add some configuration to coredns and nodelocaldns to ensure the DNS requests using the old domain are handled correctly.
-Example:
-
-```yaml
-old_dns_domains:
-- example1.com
-- example2.com
-dns_domain: cluster.local
-```
-
-will make `my-svc.my-ns.svc.example1.com`, `my-svc.my-ns.svc.example2.com` and `my-svc.my-ns.svc.cluster.local` have the same DNS answer.
-
 ### systemd_resolved_disable_stub_listener
 
 Whether or not to set `DNSStubListener=no` when using systemd-resolved. Defaults to `true` on Flatcar.

docs/docker.md

@@ -97,9 +97,3 @@ Adding extra options to pass to the docker daemon:
 ## This string should be exactly as you wish it to appear.
 docker_options: ""
 ```
-
-For Debian based distributions, set the path to store the GPG key to avoid using the default one used in `apt_key` module (e.g. /etc/apt/trusted.gpg)
-
-```yaml
-docker_repo_key_keyring: /etc/apt/trusted.gpg.d/docker.gpg
-```

docs/equinix-metal.md

@@ -54,7 +54,7 @@ cd kubespray
 
 ## Install Ansible
 
-Install Ansible according to [Ansible installation guide](/docs/ansible/ansible.md#installing-ansible)
+Install Ansible according to [Ansible installation guide](/docs/ansible.md#installing-ansible)
 
 ## Cluster Definition
 

docs/external_storage_provisioners/scheduler_plugins.md

@@ -1,49 +0,0 @@
-# Scheduler plugins for Kubernetes
-
-[scheduler-plugins](https://github.com/kubernetes-sigs/scheduler-plugins) is out-of-tree scheduler plugins based on the [scheduler framework](https://kubernetes.io/docs/concepts/scheduling-eviction/scheduling-framework/).
-
-The kube-scheduler binary includes a list of plugins:
-
-- [CapacityScheduling](https://github.com/kubernetes-sigs/scheduler-plugins/tree/master/pkg/capacityscheduling) [Beta]
-- [CoScheduling](https://github.com/kubernetes-sigs/scheduler-plugins/tree/master/pkg/coscheduling) [Beta]
-- [NodeResources](https://github.com/kubernetes-sigs/scheduler-plugins/tree/master/pkg/noderesources) [Beta]
-- [NodeResouceTopology](https://github.com/kubernetes-sigs/scheduler-plugins/blob/master/pkg/noderesourcetopology/README.md) [Beta]
-- [PreemptionToleration](https://github.com/kubernetes-sigs/scheduler-plugins/blob/master/pkg/preemptiontoleration/README.md) [Alpha]
-- [Trimaran](https://github.com/kubernetes-sigs/scheduler-plugins/blob/master/pkg/trimaran/README.md) [Alpha]
-- [NetworkAware](https://github.com/kubernetes-sigs/scheduler-plugins/blob/master/pkg/networkaware/README.md) [Sample]
-- [CrossNodePreemption](https://github.com/kubernetes-sigs/scheduler-plugins/blob/master/pkg/crossnodepreemption/README.md) [Sample]
-- [PodState](https://github.com/kubernetes-sigs/scheduler-plugins/blob/master/pkg/podstate/README.md) [Sample]
-- [QualityOfService](https://github.com/kubernetes-sigs/scheduler-plugins/blob/master/pkg/qos/README.md) [Sample]
-
-Currently, we use [helm chart](https://github.com/kubernetes-sigs/scheduler-plugins/blob/master/manifests/install/charts/as-a-second-scheduler/README.md#installing-the-chart) to install the scheduler plugins, so that a second scheduler would be created and running. **Note that running multi-scheduler will inevitably encounter resource conflicts when the cluster is short of resources**.
-
-## Compatibility Matrix
-
-There are requirements for the version of Kubernetes, please see [Compatibility Matrix
-](https://github.com/kubernetes-sigs/scheduler-plugins/tree/master?tab=readme-ov-file#compatibility-matrix). It deserves our attention.
-
-| Scheduler Plugins | Compiled With K8s Version |
-| ----------------- | ------------------------- |
-| v0.28.9           | v1.28.9                   |
-| v0.27.8           | v1.27.8                   |
-
-## Turning it on
-
-  The `scheduler_plugins_enabled` option is used to enable the installation of scheduler plugins.
-
-  You can enable or disable some plugins by setting the `scheduler_plugins_enabled_plugins` or `scheduler_plugins_disabled_plugins` option. They must be in the list we mentioned above.
-
-  In addition, to use custom plugin configuration, set a value for `scheduler_plugins_plugin_config` option.
-
-  For example, for Coscheduling plugin, you want to customize the permit waiting timeout to 10 seconds:
-
-  ```yaml
-  scheduler_plugins_plugin_config:
-    - name: Coscheduling
-      args:
-        permitWaitingTimeSeconds: 10 # default is 60
-  ```
-
-## Leverage plugin
-
-  Once the plugin is installed, we can apply CRs into cluster. For example, if using `CoScheduling`, we can apply the CR and test the deployment in the [example](https://github.com/kubernetes-sigs/scheduler-plugins/blob/master/doc/install.md#test-coscheduling).

docs/getting-started.md

@@ -36,7 +36,7 @@ ansible-playbook -i inventory/mycluster/hosts.yml cluster.yml -b -v \
   --private-key=~/.ssh/private_key
 ```
 
-See more details in the [ansible guide](/docs/ansible/ansible.md).
+See more details in the [ansible guide](/docs/ansible.md).
 
 ### Adding nodes
 
@@ -81,7 +81,7 @@ kube-apiserver via port 8080. A kubeconfig file is not necessary in this case,
 because kubectl will use <http://localhost:8080> to connect. The kubeconfig files
 generated will point to localhost (on kube_control_planes) and kube_node hosts will
 connect either to a localhost nginx proxy or to a loadbalancer if configured.
-More details on this process are in the [HA guide](/docs/operations/ha-mode.md).
+More details on this process are in the [HA guide](/docs/ha-mode.md).
 
 Kubespray permits connecting to the cluster remotely on any IP of any
 kube_control_plane host on port 6443 by default. However, this requires
@@ -140,5 +140,5 @@ If desired, copy admin.conf to ~/.kube/config.
 
 ## Setting up your first cluster
 
-[Setting up your first cluster](/docs/getting_started/setting-up-your-first-cluster.md) is an
+[Setting up your first cluster](/docs/setting-up-your-first-cluster.md) is an
  applied step-by-step guide for setting up your first cluster with Kubespray.

docs/hardening.md

@@ -54,11 +54,6 @@ kube_apiserver_enable_admission_plugins:
   - PodNodeSelector
   - PodSecurity
 kube_apiserver_admission_control_config_file: true
-# Creates config file for PodNodeSelector
-# kube_apiserver_admission_plugins_needs_configuration: [PodNodeSelector]
-# Define the default node selector, by default all the workloads will be scheduled on nodes
-# with label network=srv1
-# kube_apiserver_admission_plugins_podnodeselector_default_node_selector: "network=srv1"
 # EventRateLimit plugin configuration
 kube_apiserver_admission_event_rate_limits:
   limit_1:
@@ -71,8 +66,6 @@ kube_apiserver_admission_event_rate_limits:
     qps: 50
     burst: 100
 kube_profiling: false
-# Remove anonymous access to cluster
-remove_anonymous_access: true
 
 ## kube-controller-manager
 kube_controller_manager_bind_address: 127.0.0.1
@@ -99,7 +92,7 @@ kubelet_event_record_qps: 1
 kubelet_rotate_certificates: true
 kubelet_streaming_connection_idle_timeout: "5m"
 kubelet_make_iptables_util_chains: true
-kubelet_feature_gates: ["RotateKubeletServerCertificate=true"]
+kubelet_feature_gates: ["RotateKubeletServerCertificate=true", "SeccompDefault=true"]
 kubelet_seccomp_default: true
 kubelet_systemd_hardening: true
 # In case you have multiple interfaces in your
@@ -107,7 +100,7 @@ kubelet_systemd_hardening: true
 # IP addresses, kubelet_secure_addresses allows you
 # to specify the IP from which the kubelet
 # will receive the packets.
-kubelet_secure_addresses: "localhost link-local {{ kube_pods_subnet }} 192.168.10.110 192.168.10.111 192.168.10.112"
+kubelet_secure_addresses: "192.168.10.110 192.168.10.111 192.168.10.112"
 
 # additional configurations
 kube_owner: root
@@ -122,7 +115,7 @@ kube_pod_security_default_enforce: restricted
 Let's take a deep look to the resultant **kubernetes** configuration:
 
 * The `anonymous-auth` (on `kube-apiserver`) is set to `true` by default. This is fine, because it is considered safe if you enable `RBAC` for the `authorization-mode`.
-* The `enable-admission-plugins` includes `PodSecurity` (for more details, please take a look here: <https://kubernetes.io/docs/concepts/security/pod-security-admission/>). Then, we set the `EventRateLimit` plugin, providing additional configuration files (that are automatically created under the hood and mounted inside the `kube-apiserver` container) to make it work.
+* The `enable-admission-plugins` has not the `PodSecurityPolicy` admission plugin. This because it is going to be definitely removed from **kubernetes** `v1.25`. For this reason we decided to set the newest `PodSecurity` (for more details, please take a look here: <https://kubernetes.io/docs/concepts/security/pod-security-admission/>). Then, we set the `EventRateLimit` plugin, providing additional configuration files (that are automatically created under the hood and mounted inside the `kube-apiserver` container) to make it work.
 * The `encryption-provider-config` provide encryption at rest. This means that the `kube-apiserver` encrypt data that is going to be stored before they reach `etcd`. So the data is completely unreadable from `etcd` (in case an attacker is able to exploit this).
 * The `rotateCertificates` in `KubeletConfiguration` is set to `true` along with `serverTLSBootstrap`. This could be used in alternative to `tlsCertFile` and `tlsPrivateKeyFile` parameters. Additionally it automatically generates certificates by itself. By default the CSRs are approved automatically via [kubelet-csr-approver](https://github.com/postfinance/kubelet-csr-approver). You can customize approval configuration by modifying Helm values via `kubelet_csr_approver_values`.
   See <https://kubernetes.io/docs/reference/access-authn-authz/kubelet-tls-bootstrapping/> for more information on the subject.

docs/ingress_controller/alb_ingress_controller.md

@@ -1,6 +1,6 @@
 # AWS ALB Ingress Controller
 
-**NOTE:** The current image version is `v1.1.9`. Please file any issues you find and note the version used.
+**NOTE:** The current image version is `v1.1.6`. Please file any issues you find and note the version used.
 
 The AWS ALB Ingress Controller satisfies Kubernetes [ingress resources](https://kubernetes.io/docs/concepts/services-networking/ingress/) by provisioning [Application Load Balancers](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/introduction.html).
 

docs/ingress_controller/ingress_nginx.md

@@ -35,7 +35,7 @@ kubectl create clusterrolebinding cluster-admin-binding \
 The following **Mandatory Command** is required for all deployments except for AWS. See below for the AWS version.
 
 ```console
-kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.10.1/deploy/static/provider/cloud/deploy.yaml
+kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.40.2/deploy/static/provider/cloud/deploy.yaml
 ```
 
 ### Provider Specific Steps