Using Kestra to update my proxmox IPSet

I've been using Kestra for a while now to automate some of the more mundane tasks around my network. One of these tasks is keeping the firewall up to date. This blog sits behind cloudflare, and as such, only cloudflare needs to access it. All other sources should be be denied (except internally).

I'm assuming for this that you have kestra setup, and you have used ansible via kestra before. If not, I do have other posts about ansible and kestra, but you might want to find an online tutorial for kestra. I run mine via docker and docker compose.

Cloudflare changes it's IP addresses from time to time and to keep this blog working, I need to track those changes. This doesn't happen every day, but I don't really want to be checking all the time either. Since the changes are infrequent and small, running the task once a day should keep everything in line.

Fortunately, cloudflare publishes it's IP ranges here. Since these are published as 2 files, we can use kestra to grab them and concatenate them.

id: proxmoxCloudflareIPSET
namespace: proxmox
description: Download and update the Cloudflare IPSet in Proxmox.

labels:
  env: prod
  project: proxmox  
  
inputs:
  - id: ipv4file
    type: STRING
    defaults: "https://www.cloudflare.com/ips-v4/#"
    required: true
  - id: ipv6file
    type: STRING
    defaults: "https://www.cloudflare.com/ips-v6/#"
    required: true

tasks:
  - id: getIPV4File
    type: io.kestra.plugin.core.http.Download
    uri: "{{ inputs.ipv4file }}"
  - id: getIPV6File 
    type: io.kestra.plugin.core.http.Download
    uri: "{{ inputs.ipv6file }}"
  - id: concatIPSET
    type: "io.kestra.plugin.core.storage.Concat"
    files: 
    - "{{ outputs.getIPV4File.uri }}"
    - "{{ outputs.getIPV6File.uri }}"
    separator: "\n"

This allows us to have one big list for the IP block. We need to turn this block in to an IPSET in proxmox parlance. This is a set of IPs that we can then reference in our security groups later. First thing, we need to give it a name. I chose the descriptive one of "cloudflare". I added this as a variable, in case I want to change it later.

The next stage is setting up your template. Your cluster's firewall will look different to mine. The firewall configuration can be found in /etc/pve/firewall/cluster.fw - we will need this later on. While testing things out, I suggest you actually write files to /tmp and use the command:

diff -Naur /tmp/cluster.fw /etc/pve/firewall/cluster.fw

This way, you'll see any potential differences. When I was looking at automating this, I already had the IPSET of 'cloudflare'.

Copy your existing cluster.fw file to Kestra's data directory and in to the _files directory of the namespace you set for this. I chose the space "proxmox". I also created a folder called templates for it to live in. I also copied over my ansible hosts file to proxmox _files directory and stripped down to only proxmox. The cluster.fw file should have a .j2 file extension, so I called mine cluster-fw.j2.

Before we move on tot he ansible playbook, we need to add the following task to kestra:

  - id: ansible_task
    namespaceFiles:
      enabled: true
      include:
      - hosts
      - proxmox-cloudflare.yaml
      - templates/cluster-fw.j2
    type: io.kestra.plugin.ansible.cli.AnsibleCLI
    taskRunner:
      type: io.kestra.plugin.scripts.runner.docker.Docker
      image: cytopia/ansible:latest-tools
      pullPolicy: IF_NOT_PRESENT
    env:
      "ANSIBLE_HOST_KEY_CHECKING": "false"
      timeout: PT10M
    commands:
      - apk add sshpass
      - ansible-playbook -i hosts proxmox-cloudflare.yaml -e "ipsetfile={{outputs.concatIPSET.uri }}"

Next, I created the file proxmox-cloudflare.yaml in the kestra namespace next to the hosts file. The playbook is quite simple:

---
- name: "Proxmox Playbook"
  hosts: 192.168.1.11 # First Proxmox host of cluster
  serial: 1   # to avoid updating / breaking everything all at once
  become: true
  become_method: sudo

  vars:
    - ipsetname: "cloudflare"
    - ipsetlist: "{{ lookup('file', ipsetfile) }}"
  
  tasks:
    - name: Firewall configuration file
      ansible.builtin.template:
        src: templates/cluster-fw.j2
        dest: /tmp/cluster.fw

All this playbook does is go through all the hosts 1 by 1, using sudo to become root and write the file to /tmp/cluster.fw based on the template we just created.

In order to have the IPSET be created, I editing my cluster-fw.j2 file to include the following (you may need to remove or modify a previous definition if you have one):

[IPSET {{ ipsetname }}] # Cloudflare IPs

{{ ipsetlist }}

Once complete, you can then run the task and verify that the file /tmp/cluster.fw is created and correct, compared to your original firewall file.

NB, the lack of space between the }} and the ] is intentional. If you add one, the name will be "cloudflare " rather than "cloudflare", and since a space in a name is illegal, it won't show up in proxmox.

For completeness, these are my files:

id: proxmoxCloudflareIPSET
namespace: proxmox
description: Download and update the Cloudflare IPSet in Proxmox.

labels:
  env: prod
  project: proxmox    

inputs:
  - id: ipsetname
    type: STRING
    defaults: "cloudflare"
    required: true
  - id: ipv4file
    type: STRING
    defaults: "https://www.cloudflare.com/ips-v4/#"
    required: true
  - id: ipv6file
    type: STRING
    defaults: "https://www.cloudflare.com/ips-v6/#"
    required: true

tasks:
  - id: getIPV4File
    type: io.kestra.plugin.core.http.Download
    uri: "{{ inputs.ipv4file }}"
  - id: getIPV6File 
    type: io.kestra.plugin.core.http.Download
    uri: "{{ inputs.ipv6file }}"
  - id: concatIPSET
    type: "io.kestra.plugin.core.storage.Concat"
    files: 
    - "{{ outputs.getIPV4File.uri }}"
    - "{{ outputs.getIPV6File.uri }}"
    separator: "\n"
  - id: ansible_task
    namespaceFiles:
      enabled: true
      include:
      - hosts
      - proxmox-cloudflare.yaml
      - templates/cluster-fw.j2
    type: io.kestra.plugin.ansible.cli.AnsibleCLI
    taskRunner:
      type: io.kestra.plugin.scripts.runner.docker.Docker
      image: cytopia/ansible:latest-tools
      pullPolicy: IF_NOT_PRESENT
    env:
      "ANSIBLE_HOST_KEY_CHECKING": "false"
      timeout: PT10M
    commands:
      - apk add sshpass
      - ansible-playbook -i hosts proxmox-cloudflare.yaml -e "ipsetfile={{outputs.concatIPSET.uri }}" -e "ipsetname={{inputs.ipsetname}}"

[OPTIONS]

enable: 1
policy_in: ACCEPT

[ALIASES]

Lan 192.168.1.0/24 # Lan Network

[IPSET {{ ipsetname }}] # Cloudflare IPs

{{ ipsetlist }}

[IPSET internal] # Internal

lan

[RULES]

GROUP hostsecuritygroup

[group cloudflare] # Web Server ranges limited to cloud flare IPs.

IN HTTPS(ACCEPT) -source +dc/internal -log nolog
IN HTTP(ACCEPT) -source +dc/internal -log nolog
IN HTTPS(ACCEPT) -source +dc/cloudflare -log nolog
IN HTTP(ACCEPT) -source +dc/cloudflare -log nolog

---
- name: "Proxmox Playbook"
#  hosts: proxmox
  hosts: 192.168.1.21 # First Proxmox host of cluster
  serial: 1   # to avoid updating / breaking everything all at once
  become: true
  become_method: sudo

  vars:
    - ipsetname: "cloudflare"
    - ipsetlist: "{{ lookup('file', ipsetfile) }}"
  tasks:

    - name: Firewall configuration file
      ansible.builtin.template:
        src: templates/cluster-fw.j2
        dest: /etc/pve/firewall/cluster.fw

I don't trigger the task directly, as I prefer to control when tasks run via a set of triggers dedicated for the task. That way, I can see what jobs are running and when.

Once you are happy with task running, I added a securitygroup, also called cloudflare, and assigned it to my blog vm and removed the previos security group allowing the world in on port 80 and 443. This groups is included in the cluster-fw.j2 file.

Once you have checked all is up and running, you now have kestra keeping proxmox's firewall up to date with cloudflare changes!