CI: reduce VM resources requests to improve scheduling

Add 'system-packages' tag to control installing packages from OS repositories (#10872 )
Merge pull request #11239 from VannTen/cleanup/collection-build-test
2024-05-31 11:36:15 +02:00 · 2024-05-30 04:25:21 -07:00 · 2024-05-30 04:10:29 -07:00 · 2024-05-30 03:01:38 -07:00 · 2024-05-30 02:32:28 -07:00 · 2024-05-29 19:37:56 -07:00
23 changed files with 35 additions and 61 deletions
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@ -43,6 +43,7 @@ repos:
          - jsonschema==4.22.0
          - jmespath==1.0.1
          - netaddr==1.2.1
+          - distlib

  - repo: https://github.com/VannTen/misspell
    # Waiting on https://github.com/golangci/misspell/pull/19 to get merged
@ -80,6 +81,7 @@ repos:
        language: python
        additional_dependencies:
          - ansible-core>=2.16.4
+          - distlib
        entry: tests/scripts/collection-build-install.sh
        pass_filenames: false

--- a/2
+++ b/2
@ -6,6 +6,7 @@ aliases:
    - mzaian
    - oomichi
    - yankay
+    - ant31
  kubespray-reviewers:
    - cyclinder
    - erikjiang
@ -14,7 +15,6 @@ aliases:
    - vannten
    - yankay
  kubespray-emeritus_approvers:
-    - ant31
    - atoms
    - chadswen
    - luckysb
--- a/docs/ansible/ansible.md
+++ b/docs/ansible/ansible.md
@ -231,6 +231,7 @@ The following tags are defined in playbooks:
 | services                       | Remove services (etcd, kubelet etc...) when resetting |
 | snapshot                       | Enabling csi snapshot                                 |
 | snapshot-controller            | Configuring csi snapshot controller                   |
+| system-packages                | Install packages using OS package manager             |
 | upgrade                        | Upgrading, f.e. container images/binaries             |
 | upload                         | Distributing images/binaries across hosts             |
 | vsphere-csi-driver             | Configuring csi driver: vsphere                       |
--- a/docs/operations/offline-environment.md
+++ b/docs/operations/offline-environment.md
@ -103,7 +103,9 @@ If you use the settings like the one above, you'll need to define in your invent
  can store them anywhere as long as it's accessible by kubespray. It's recommended to use `*_version` in the path so
  that you don't need to modify this setting everytime kubespray upgrades one of these components.
 * `yum_repo`/`debian_repo`/`ubuntu_repo`: OS package repository depending on your OS, should point to your internal
-  repository. Adjust the path accordingly.
+  repository. Adjust the path accordingly. Used only for Docker/Containerd packages (if needed); other packages might
+  be installed from other repositories. You might disable installing packages from other repositories by skipping
+  the `system-packages` tag

 ## Install Kubespray Python Packages

--- a/galaxy.yml
+++ b/galaxy.yml
@ -9,42 +9,12 @@ authors:
 tags:
  - infrastructure
 repository: https://github.com/kubernetes-sigs/kubespray
+issues: https://github.com/kubernetes-sigs/kubespray/issues
+documentation: https://kubespray.io
 license_file: LICENSE
 dependencies:
  ansible.utils: '>=2.5.0'
  community.general: '>=3.0.0'
-build_ignore:
-  - .github
-  - '*.tar.gz'
-  - extra_playbooks
-  - inventory
-  - scripts
-  - test-infra
-  - .ansible-lint
-  - .editorconfig
-  - .gitignore
-  - .gitlab-ci
-  - .gitlab-ci.yml
-  - .gitmodules
-  - .markdownlint.yaml
-  - .nojekyll
-  - .pre-commit-config.yaml
-  - .yamllint
-  - Dockerfile
-  - FILES.json
-  - MANIFEST.json
-  - Makefile
-  - Vagrantfile
-  - _config.yml
-  - ansible.cfg
-  - requirements*txt
-  - setup.cfg
-  - setup.py
-  - index.html
-  - reset.yml
-  - cluster.yml
-  - scale.yml
-  - recover-control-plane.yml
-  - remove-node.yml
-  - upgrade-cluster.yml
-  - library
+manifest:
+  directives:
+    - recursive-exclude tests **
--- a/requirements.txt
+++ b/requirements.txt
@ -1,4 +1,4 @@
-ansible==9.5.1
+ansible==9.6.0
 cryptography==42.0.7
 jinja2==3.1.4
 jmespath==1.0.1
--- a/roles/kubernetes/preinstall/tasks/0040-verify-settings.yml
+++ b/roles/kubernetes/preinstall/tasks/0040-verify-settings.yml
@ -1,10 +1,7 @@
 ---
 - name: Stop if either kube_control_plane or kube_node group is empty
  assert:
-    that: "groups.get( item )"
-  with_items:
-    - kube_control_plane
-    - kube_node
+    that: groups.get( 'kube_control_plane' )
  run_once: true
  when: not ignore_assert_errors

--- a/roles/kubernetes/preinstall/tasks/main.yml
+++ b/roles/kubernetes/preinstall/tasks/main.yml
@ -68,6 +68,7 @@
    - not dns_late
  tags:
    - bootstrap-os
+    - system-packages

 - name: Apply system configurations
  import_tasks: 0080-system-configurations.yml
--- a/tests/cloud_playbooks/roles/packet-ci/defaults/main.yml
+++ b/tests/cloud_playbooks/roles/packet-ci/defaults/main.yml
@ -4,15 +4,14 @@
 vm_cpu_cores: 2
 vm_cpu_sockets: 1
 vm_cpu_threads: 2
-vm_memory: 2048Mi
+vm_memory: 2048

 # Replace invalid characters so that we can use the branch name in kubernetes labels
 branch_name_sane: "{{ branch | regex_replace('/', '-') }}"

 # Request/Limit allocation settings
-
-cpu_allocation_ratio: 0.5
-memory_allocation_ratio: 1
+cpu_allocation_ratio: 0.25
+memory_allocation_ratio: 0.5

 # Default path for inventory
 inventory_path: "/tmp/{{ test_name }}/inventory"
--- a/tests/cloud_playbooks/roles/packet-ci/templates/vm.yml.j2
+++ b/tests/cloud_playbooks/roles/packet-ci/templates/vm.yml.j2
@ -4,6 +4,8 @@ kind: VirtualMachine
 metadata:
  name: "instance-{{ vm_id }}"
  namespace: "{{ test_name }}"
+  annotations:
+    kubespray.com/ci.template-path: "tests/cloud_playbooks/roles/packet-ci/templates/vm.yml.j2"
  labels:
    kubevirt.io/os: {{ cloud_image }}
 spec:
@ -34,10 +36,10 @@ spec:
            threads: {{ vm_cpu_threads }}
        resources:
          requests:
-            memory: {{ vm_memory * memory_allocation_ratio }}
+            memory: "{{ vm_memory * memory_allocation_ratio }}Mi"
            cpu: {{ vm_cpu_cores * cpu_allocation_ratio }}
          limits:
-            memory: {{ vm_memory }}
+            memory: "{{ vm_memory }}Mi"
            cpu: {{ vm_cpu_cores }}
      networks:
      - name: default
--- a/tests/files/packet_almalinux8-calico-ha-ebpf.yml
+++ b/tests/files/packet_almalinux8-calico-ha-ebpf.yml
@ -2,7 +2,7 @@
 # Instance settings
 cloud_image: almalinux-8
 mode: ha
-vm_memory: 3072Mi
+vm_memory: 3072

 # Kubespray settings
 calico_bpf_enabled: true
--- a/tests/files/packet_almalinux8-calico-nodelocaldns-secondary.yml
+++ b/tests/files/packet_almalinux8-calico-nodelocaldns-secondary.yml
@ -2,7 +2,7 @@
 # Instance settings
 cloud_image: almalinux-8
 mode: default
-vm_memory: 3072Mi
+vm_memory: 3072

 # Kubespray settings
 enable_nodelocaldns_secondary: true
--- a/tests/files/packet_almalinux8-calico.yml
+++ b/tests/files/packet_almalinux8-calico.yml
@ -2,7 +2,7 @@
 # Instance settings
 cloud_image: almalinux-8
 mode: default
-vm_memory: 3072Mi
+vm_memory: 3072

 # Kubespray settings
 metrics_server_enabled: true
--- a/tests/files/packet_almalinux8-docker.yml
+++ b/tests/files/packet_almalinux8-docker.yml
@ -2,7 +2,7 @@
 # Instance settings
 cloud_image: almalinux-8
 mode: default
-vm_memory: 3072Mi
+vm_memory: 3072

 # Use docker
 container_manager: docker
--- a/tests/files/packet_almalinux8-kube-ovn.yml
+++ b/tests/files/packet_almalinux8-kube-ovn.yml
@ -2,7 +2,7 @@
 # Instance settings
 cloud_image: almalinux-8
 mode: default
-vm_memory: 3072Mi
+vm_memory: 3072

 # Kubespray settings
 kube_network_plugin: kube-ovn
--- a/tests/files/packet_rockylinux8-calico.yml
+++ b/tests/files/packet_rockylinux8-calico.yml
@ -2,7 +2,7 @@
 # Instance settings
 cloud_image: rockylinux-8
 mode: default
-vm_memory: 3072Mi
+vm_memory: 3072

 # Kubespray settings
 metrics_server_enabled: true
--- a/tests/files/packet_rockylinux9-calico.yml
+++ b/tests/files/packet_rockylinux9-calico.yml
@ -2,7 +2,7 @@
 # Instance settings
 cloud_image: rockylinux-9
 mode: default
-vm_memory: 3072Mi
+vm_memory: 3072

 # Kubespray settings
 metrics_server_enabled: true
--- a/tests/files/packet_rockylinux9-cilium.yml
+++ b/tests/files/packet_rockylinux9-cilium.yml
@ -2,7 +2,7 @@
 # Instance settings
 cloud_image: rockylinux-9
 mode: default
-vm_memory: 3072Mi
+vm_memory: 3072

 # Kubespray settings
 kube_network_plugin: cilium
--- a/tests/files/packet_ubuntu22-all-in-one-docker.yml
+++ b/tests/files/packet_ubuntu22-all-in-one-docker.yml
@ -2,7 +2,7 @@
 # Instance settings
 cloud_image: ubuntu-2204
 mode: all-in-one
-vm_memory: 1600Mi
+vm_memory: 1600

 # Kubespray settings
 auto_renew_certificates: true
--- a/tests/files/packet_ubuntu22-calico-all-in-one.yml
+++ b/tests/files/packet_ubuntu22-calico-all-in-one.yml
@ -2,7 +2,7 @@
 # Instance settings
 cloud_image: ubuntu-2204
 mode: all-in-one
-vm_memory: 1600Mi
+vm_memory: 1600

 # Kubespray settings
 auto_renew_certificates: true
--- a/tests/files/packet_ubuntu24-all-in-one-docker.yml
+++ b/tests/files/packet_ubuntu24-all-in-one-docker.yml
@ -2,7 +2,7 @@
 # Instance settings
 cloud_image: ubuntu-2404
 mode: all-in-one
-vm_memory: 1600Mi
+vm_memory: 1600

 # Kubespray settings
 auto_renew_certificates: true
--- a/tests/files/packet_ubuntu24-calico-all-in-one.yml
+++ b/tests/files/packet_ubuntu24-calico-all-in-one.yml
@ -2,7 +2,7 @@
 # Instance settings
 cloud_image: ubuntu-2404
 mode: all-in-one
-vm_memory: 1600Mi
+vm_memory: 1600

 # Kubespray settings
 auto_renew_certificates: true
--- a/tests/files/packet_ubuntu24-calico-etcd-datastore.yml
+++ b/tests/files/packet_ubuntu24-calico-etcd-datastore.yml
@ -2,7 +2,7 @@
 # Instance settings
 cloud_image: ubuntu-2404
 mode: node-etcd-client
-vm_memory: 1600Mi
+vm_memory: 1600

 # Kubespray settings
 auto_renew_certificates: true
Author	SHA1	Message	Date
ant31	2214071a82	CI: reduce VM resources requests to improve scheduling	2024-05-31 11:36:15 +02:00
Hedayat Vatankhah (هدایت)	dedc00661a	Add 'system-packages' tag to control installing packages from OS repositories (#10872 )	2024-05-30 04:25:21 -07:00
Kubernetes Prow Robot	0624a3061a	Merge pull request #11239 from VannTen/cleanup/collection-build-test Cleanup galaxy.yml	2024-05-30 04:10:29 -07:00
Max Gautier	3082fa3d0f	Allow empty kube_node group (#11248 ) While uncommon, provisioning only a control plane is a valid use case, so don't block it.	2024-05-30 03:01:38 -07:00
Antoine Legrand	d85b29aae1	owners: move ant31 from emeritus to approvers (#11247 )	2024-05-30 02:32:28 -07:00
Kubernetes Prow Robot	af593465b2	Merge pull request #11226 from VannTen/cleanup/pre-commit-hooks pre-commit: make hooks self contained + ci config	2024-05-29 19:37:56 -07:00
Max Gautier	870049523f	collection support: use manifest instead of excludes The default for galaxy. `manifest` works well enough for our case, and this avoids maintaining a blacklist.	2024-05-29 13:57:33 +02:00
Kay Yan	184b1add54	Merge pull request #11236 from kubernetes-sigs/dependabot/pip/ansible-9.6.0 Bump ansible from 9.5.1 to 9.6.0	2024-05-29 17:17:20 +08:00
dependabot[bot]	bd9d90e00c	Bump ansible from 9.5.1 to 9.6.0 Bumps [ansible](https://github.com/ansible-community/ansible-build-data) from 9.5.1 to 9.6.0. - [Changelog](https://github.com/ansible-community/ansible-build-data/blob/main/docs/release-process.md) - [Commits](https://github.com/ansible-community/ansible-build-data/compare/9.5.1...9.6.0) --- updated-dependencies: - dependency-name: ansible dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>	2024-05-27 03:45:28 +00:00