Automating Ansible playbooks with Kestra

Last updated on Jul 10, 2024

Recently, after watch Christian Lempa videos, I have started playing about with Kestra. Kestra is a work flow orcestration platform. It allows me to not only automate my play books, but schedule them or tirgger them when I want them. I decided to give this a go with another container that I was getting running: unpackerr. I have plans to do similar things with tdarr as well.

So why use Kestra for these playbooks?

This is actually quite simple - these playbooks start processes which are resource intensive, in terms of CPU, GPU and I/O. My media is stored on my nas. The last thing I want is the playback of my plex server interrupted by these operations. Yes, I could buy more powerful hardware. However, this computer is already on 24/7, why not make it more efficent?

To start off with, I installed Kestra via docker compose. I'm not going to cover how to do this as it is quite straightforward and Christian has probably already covered it better than I can. My data for Kestra does reside on a 1Tb SSD for my Management Pi, under /nsm/kestra/kestra-data/.

Next, I created 2 ansible playbooks - 1 to start unpackerr ...

---
  - name: "Unpackerr on Docker Playbook"
    hosts: docker.host
    become: yes
    become_method: sudo

    vars:
    - container: unpackerr
    - domain: 'docker.host'

    tasks:

    - name: Create the unpackerr configuration volume
      docker_volume:
        name: "{{ container }}-config"
      tags: volumecreate

    - name: Create Traefik labels's dictionary
      set_fact:
        my_labels: "{{ my_labels | default({}) | combine ({ item.key : item.value }) }}"
      with_items:
      - { 'key': 'traefik.enable' , 'value': 'true'}
      - { 'key': 'traefik.docker.network', 'value': "traefik-public"}
      - { 'key': "traefik.http.middlewares.{{ container }}-https-redirect.redirectscheme.scheme", 'value': "https"}
      - { 'key': "traefik.http.routers.{{ container }}-secure.entrypoints",'value': "https"}
      - { 'key': "traefik.http.routers.{{ container }}-secure.rule",'value': "Host(`{{ container }}.{{ domain }}`)"}
      - { 'key': "traefik.http.routers.{{ container }}-secure.service",'value': "{{ container }}"}
      - { 'key': "traefik.http.routers.{{ container }}-secure.tls",'value': "true"}
      - { 'key': "traefik.http.routers.{{ container }}.entrypoints",'value': "http"}
      - { 'key': "traefik.http.routers.{{ container }}.middlewares",'value': "{{ container }}-https-redirect"}
      - { 'key': "traefik.http.routers.{{ container }}.rule",'value': "Host(`{{ container }}.{{ domain }}`)"}
      - { 'key': "traefik.http.services.{{ container }}.loadbalancer.server.port", 'value': "5656"}

# docker run --user 1000:100 -d -v /mnt/data:/data -v unpackerr-config:/config golift/unpackerr

    - name: Start Unpackerr and apply labels
      docker_container:
        name: "{{ container }}"
        state: started
        networks:
        - name: traefik-public
        image: golift/unpackerr:latest
        user: "1000:100"
        env:
          TZ: "Etc/UTC"
          VERSION: "docker"
        ports:
        - "5656:5656"
        volumes:
        - "{{ container }}-config:/config"
        - /mnt/data:/data
#        devices:
#        - /dev/dri:/dev/dri
        labels: "{{ my_labels }}"
      tags: deploycontainer

... and 1 to stop it.

---
  - name: "Stop Unpackerr on Docker Playbook"
    hosts: docker.host
    become: yes
    become_method: sudo

    vars:
    - container: unpackerr
    - domain: 'docker.host'

    tasks:

    - name: Stop Unpackerr container
      docker_container:
        name: "{{ container }}"
        state: stopped

Next, I created a flow on Kestra called start_unpackerr. There are multiple ways to do this, but I wanted to be able to call this flow from other flows. My Flow looks like this:

id: start_unpackarr
namespace: ansible
description: Start unpackarr at a scheduled time

labels:
  env: prod
  project: ansible
    

tasks:
  - id: unapackerr_start
    type: io.kestra.plugin.core.flow.WorkingDirectory
    tasks:
      - id: ansible_task
        namespaceFiles:
          enabled: true
          include:
          - hosts
          - docker-unpackerr-start.yaml
        type: io.kestra.plugin.ansible.cli.AnsibleCLI
        docker:
          image: cytopia/ansible:latest-tools
        env:
          "ANSIBLE_HOST_KEY_CHECKING": "false"
        commands:
          - apk add sshpass
          - ansible-playbook -i hosts docker-unpackerr-start.yaml


concurrency:
  behavior: CANCEL
  limit: 1

I also created the following flow to stop unpackerr.

id: stop_unpackarr
namespace: ansible
description: Stop unpackarr at a scheduled time

labels:
  env: prod
  project: ansible
    

tasks:
  - id: unpackerr_stop
    type: io.kestra.plugin.core.flow.WorkingDirectory
    tasks:
      - id: ansible_task
        namespaceFiles:
          enabled: true
          include:
          - hosts
          - docker-unpackerr-stop.yaml
        type: io.kestra.plugin.ansible.cli.AnsibleCLI
        docker:
          image: cytopia/ansible:latest-tools
        env:
          "ANSIBLE_HOST_KEY_CHECKING": "false"
        commands:
          - apk add sshpass
          - ansible-playbook -i hosts docker-unpackerr-stop.yaml


concurrency:
  behavior: CANCEL
  limit: 1

Before you run either of these 2 flows, you need to copy the playbooks docker-unpackerr-start.yaml, docker-unpackerr-stop.yaml as well as the ansible hosts file ('hosts') to the correct place. This is in:

<Data folder>/<namespace>/_files/

In my case this is /nsm/kestra/kestra-data/ansible/_files/.

Once done, run both of the flows to check they execute properly. If they don't spend some time and figure it out. The rest of this process won't really work if you don't.

Triggering The Job

The first way I plan to trigger my jobs is by a simple start / stop at a certain time. Linux would use cron to do this, and it's very similar in Kestra. Since I may want to do other things at these times, I've created flows for the triggers. For the 1 am start I have:

id: 1am_job_start
namespace: ansible
description: Start flows at 1am

labels:
  env: prod
  project: trigger-wrapper

tasks:
  - id: call_start_unpackerr
    type: io.kestra.plugin.core.flow.Subflow
    namespace: ansible
    flowId: start_unpackarr
    wait: true
    transmitFailed: false

triggers:
  - id: 1amStartTime
    type: io.kestra.plugin.core.trigger.Schedule
    cron: "3 1 * * *"

and for 5am I have:

id: 5am_job_start
namespace: ansible
description: Start flows at 5am

labels:
  env: prod
  project: trigger-wrapper

tasks:
  - id: call_stop_unpackerr
    type: io.kestra.plugin.core.flow.Subflow
    namespace: ansible
    flowId: stop_unpackarr
    wait: true
    transmitFailed: false

triggers:
  - id: 1amStartTime
    type: io.kestra.plugin.core.trigger.Schedule
    cron: "3 5 * * *"

Since Kestra is actually pretty self explanatory, I won't go in to too much detail, but each other these triggers activates at a certain time, calling the flow we created earlier.

Next time, I plan to actually start and stop these flows via webhooks and home assistant. Until then, happy automating!