594 private links
With Python, you can just open a shell and quickly try out some code in a so-called REPL console. Guess what, you can do the same with Ansible. Browse your inventory, and even remote file systems in an interactive shell with
ansible-console
.
If you have installed Ansible via pip you already have this nifty tool installed.
Let's say you are looking for old Ansible code dealing with multiple distros (meaning multiple files in the ./vars/
directory) to draw inspiration from. How can we quickly identify the roles with vars
directories containing more than a single file?
$ cd /media/bay/taha/projects/ansible && find . -not -path "*/archived/*" -not -path "*/testing/*" -path "*/vars/*" -printf "%h\n" | sort | uniq -d && cd $OLDPWD
./pub/roles/apache/vars
./pub/roles/common-systools/vars
./pub/roles/desktop-environment/vars
./pub/roles/digikam/vars
./pub/roles/graphics-driver-nvidia/vars
./pub/roles/php/vars
./pub/roles/php-versions/vars
./pub/roles/python2/vars
./pub/roles/python3/vars
./pub/roles/R/vars
./pub/roles/sioyek-pdf/vars
./pub/roles/wallabag/vars
All of the returned paths indeed contain at least two files.
Note how we limited find
to only directories named vars
.
With frequent changes to my Ansible roles it often becomes tricky to keep track of which version of a particular role was executed for a particular play.
This is not thoroughly tested yet, but my approach is simple: each role contains a task that writes its git repo state (a git log formatted one-liner containing last commit hash, date, author, etc.).
These tasks are set to write to a log-file per inventory host, which is for two reasons: to avoid drowning in the default log file defined by log_path
in ansible.cfg
, and because log_path
cannot be overridden by a play or role.
This results in a log-file (one per inventory host) looking something like this:
Started playbook execution at 2024-02-22 12:34:32.327042
Role 'locales' last commit caa6355 2024-02-21 23:38:31 +0100 by solarchemist
Role 'digikam' last commit 16f0643 2024-02-21 22:50:49 +0100 by solarchemist
Playbook 'workstation' last commit 4775cf7 2024-02-20 22:53:56 +0100 by solarchemist
Ended playbook execution at 2024-02-22 12:34:55.913856
My implementation illustrated in code below.
In a role:
- name: Log the last commit and git repo status to playbook log-file
local_action: >
shell git -C {{ role_path }} log
--pretty="Role '{{ role_name }}' last commit %h %ci by %cn" -1 >>
logbook-{{ inventory_hostname }}.log
args: { chdir: "{{ playbook_dir }}" }
become: true
become_user: "{{ local_user }}"
In the playbook:
vars:
local_user: "{{ lookup('env', 'USER') }}"
pre_tasks:
- name: Write a start message to the playbook log
ansible.builtin.shell: >
printf "\nStarted playbook execution at {{ now() }}\n" >> logbook-{{ inventory_hostname }}.log
run_once: true
delegate_to: localhost
args: { chdir: "{{ playbook_dir }}" }
become: true
become_user: "{{ local_user }}"
changed_when: true
tags: always
tasks: [...]
post_tasks:
- name: Log the current commit of this playbook
local_action: >
shell git log
--pretty="Playbook '{{ playbook_dir | basename }}' last commit %h %ci by %cn" -1
>> logbook-{{ inventory_hostname }}.log
args: { chdir: "{{ playbook_dir }}" }
become: true
become_user: "{{ local_user }}"
tags: always
- name: Write an end message to the playbook log
ansible.builtin.shell: >
echo "Ended playbook execution at {{ now() }}" >> logbook-{{ inventory_hostname }}.log
run_once: true
delegate_to: localhost
args: { chdir: "{{ playbook_dir }}" }
become: true
become_user: "{{ local_user }}"
changed_when: true
tags: always
This is an extension of the very common find ... -exec grep ... {} \;
construct I use almost daily to find which files contain a particular text string.
Now, let's say you're looking for files in all your Ansible roles containing the string ppa:
, because you want to create a new role using a suitable existing role as a template. In this case, I think most recently modified is an excellent proxy for suitability.
Thus, the challenge: can we tack on something to the find ... grep
construct such that the output shows matching files in order of most recently modified?
taha@asks2:~
$ cd /media/bay/taha/projects/ansible && find . -not -path '*/legacy/*' -type f -name "*.yml" -exec grep -il "ppa:" {} \; -printf "%T+ %p\n" | grep -v "^\\./.*" | sort && cd $OLDPWD
2021-06-09+22:27:20.6889730070 ./roles/public/php-versions/tasks/setup-Debian.yml
2022-01-09+07:39:51.1836426130 ./roles/public/java-openjdk/tasks/ppa.yml
2022-02-23+15:33:13.2318546100 ./roles/dev/deluge/tasks/install.yml
2022-03-31+04:42:16.5644336650 ./roles/dev/editor-notepadqq/tasks/main.yml
2022-05-09+15:21:01.7974094830 ./roles/dev/qownnotes/tasks/main.yml
2022-06-25+02:46:13.0580097480 ./playbooks/workstation/roles/boot-grub/tasks/main.yml
2022-06-25+02:46:18.2940273650 ./roles/dev/libreoffice/tasks/main.yml
2022-07-03+00:16:44.5563131800 ./roles/dev/shutter/tasks/main.yml
2022-07-04+20:35:15.6128270470 ./roles/dev/magnus/tasks/main.yml
2022-07-14+21:09:01.6502524570 ./roles/dev/flatpak-remote/tasks/main.yml
2022-07-14+22:05:43.3325667290 ./roles/public/firejail/tasks/install.yml
2022-07-16+13:42:04.3756138100 ./roles/public/variety/tasks/main.yml
2022-07-16+22:05:52.0552035060 ./roles/public/browser-chromium/tasks/install.yml
2022-07-21+00:14:51.1137716920 ./roles/public/foliate-ebookreader/tasks/ppa.yml
2022-07-21+03:07:42.2876610030 ./roles/public/graphics-driver-nvidia/tasks/install.yml
2022-07-21+06:05:27.4514643180 ./roles/public/foliate-ebookreader/tasks/flatpak.yml
2022-07-23+18:32:46.8466638700 ./roles/dev/handbrake/tasks/main.yml
2022-08-12+00:09:51.6575729520 ./roles/dev/x2goclient/tasks/main.yml
2022-08-19+15:28:49.5605481840 ./roles/dev/x2goserver/tasks/main.yml
2022-11-04+11:35:14.6208169990 ./roles/public/python3/tasks/python-ppa.yml
2022-11-19+03:16:16.7832183030 ./roles/public/browser-firefox/tasks/main.yml
2022-12-24+23:04:01.2033026010 ./roles/public/R/tasks/dependencies.yml
2022-12-31+19:38:32.9105553030 ./roles/public/digikam/tasks/install-ppa.yml
2023-01-01+01:39:08.2045090970 ./roles/public/digikam/tasks/install-appimage.yml
2023-01-14+00:44:34.8526187360 ./roles/public/java-openjdk/defaults/main.yml
2023-01-26+14:10:18.9247087870 ./roles/public/ansible/tasks/main.yml
2023-01-26+16:14:59.9903243110 ./roles/public/sioyek-pdf/defaults/main.yml
2023-05-12+12:01:30.4705549280 ./roles/public/mpv/tasks/install.yml
2023-05-13+00:47:41.1561557100 ./roles/public/nextcloud-desktop/tasks/main.yml
2023-08-27+18:19:44.8291334420 ./roles/public/digikam/defaults/main.yml
Eureka!
Explainer
- the initial
cd <path-parent>
ensure that the resulting paths displayed byfind
don't contain the<path-parent>
part (to avoid cluttering the output), and the finalcd $OLDPWD
just make sure that the bash prompt is not changed to<path-parent>
. - unless you want to exclude some path from the search, there is obviously no need for
-not -path '*/<some-path>/*'
. grep -i
for case insensitive matching, and-l
(that's the letterl
for list) makesgrep
print only the filename and not each matching line (this is crucial for this hack to work, we wantgrep
to produce as little output as possible, in fact, if I could figure out a way to silencegrep
altogether I would have, but I couldn't).-printf "%T+ %p\n"
adds the filemtime
to the output (on a new line). Thanks angus@Unix.SE.
At this point, an example of the unfinished product is order. Before sorting, and before the final grep -v
, the output looks like this (excerpt):
taha@asks2:~
$ cd /media/bay/taha/projects/ansible && find . -not -path '*/legacy/*' -type f -name "*.yml" -exec grep -il "ppa:" {} \; -printf "%T+ %p\n" && cd $OLDPWD
./roles/public/java-openjdk/defaults/main.yml
2023-01-14+00:44:34.8526187360 ./roles/public/java-openjdk/defaults/main.yml
./roles/public/java-openjdk/tasks/ppa.yml
2022-01-09+07:39:51.1836426130 ./roles/public/java-openjdk/tasks/ppa.yml
./roles/public/browser-firefox/tasks/main.yml
2022-11-19+03:16:16.7832183030 ./roles/public/browser-firefox/tasks/main.yml
./roles/public/R/tasks/dependencies.yml
2022-12-24+23:04:01.2033026010 ./roles/public/R/tasks/dependencies.yml
./roles/public/browser-chromium/tasks/install.yml
2022-07-16+22:05:52.0552035060 ./roles/public/browser-chromium/tasks/install.yml
./roles/dev/x2goclient/tasks/main.yml
2022-08-12+00:09:51.6575729520 ./roles/dev/x2goclient/tasks/main.yml
with the grep
output on its own line, followed by the time-stamped output of printf
. Like I said, it would have been better if we could somehow silence the grep
output at this point. If you know a way, feel free to let me know!
As expected, sorting resulted in the non-time-stamped lines dangling about like some unwanted appendage:
taha@asks2:~
$ cd /media/bay/taha/projects/ansible && find . -not -path '*/legacy/*' -type f -name "*.yml" -exec grep -il "ppa:" {} \; -printf "%T+ %p\n" && cd $OLDPWD
2022-01-09+07:39:51.1836426130 ./roles/public/java-openjdk/tasks/ppa.yml
2022-07-16+22:05:52.0552035060 ./roles/public/browser-chromium/tasks/install.yml
2022-08-12+00:09:51.6575729520 ./roles/dev/x2goclient/tasks/main.yml
2022-11-19+03:16:16.7832183030 ./roles/public/browser-firefox/tasks/main.yml
2022-12-24+23:04:01.2033026010 ./roles/public/R/tasks/dependencies.yml
2023-01-14+00:44:34.8526187360 ./roles/public/java-openjdk/defaults/main.yml
./roles/public/java-openjdk/tasks/ppa.yml
./roles/public/browser-chromium/tasks/install.yml
./roles/dev/x2goclient/tasks/main.yml
./roles/public/browser-firefox/tasks/main.yml
./roles/public/R/tasks/dependencies.yml
./roles/public/java-openjdk/defaults/main.yml
At this point I was out of ideas, so grep -v ...
it was, and we end up with the one-liner shown above. It's an ugly hack, but hey, it works :-)
Bash 5.1 on Ubuntu 22.04.3, with GNU find 4.8.0, GNU grep 3.7, and GNU sort 8.32.
What an interesting idea. A Python script that given an existing system creates an Ansible playbook that will duplicate it.
The command and its typical output (highlights not shown due to technical limitations in Markdown):
me@host:~/ansible/playbooks
$ find . -type f -name "playbook.log" -exec sh -c 'tac {} | grep -m 1 -A1 "^Playbook last committed by"' \; | grep --color -E "^|git/ansible/[A-Za-z]+?/[A-Za-z-]+?.yml|(19|20)[0-9][0-9]-(0[1-9]|1[0-2])-(0[1-9]|[12][0-9]|3[01]) [0-9]{2}:[0-9]{2}:[0-9]{2},[0-9]{3}"
Playbook last committed by me@host on Fri Apr 17 22:02:15 2020 +0200 (afd13a3b3b3f43d3f84bb16b1c91a6b5bec2cfe1)
2020-04-19 00:40:30,925 p=32526 u=me n=ansible | task path: /home/me/ansible/playbooks/luxor/playbook-host.yml:99
Playbook last committed by me@host on Wed Jan 29 14:34:38 2020 +0100 (5157cd051e276abfe99e93c37a8ad0c79dd4d3dc)
2020-03-29 01:39:34,874 p=14553 u=me n=ansible | task path: /home/me/ansible/playbooks/damietta/playbook-heliopolis.yml:31
Playbook last committed by me@host on Tue Feb 18 17:30:19 2020 +0100 (ae6c02965f4471d8089c5e4d2a427cb0cbfbc6b8)
2020-02-23 19:58:30,188 p=1050 u=me n=ansible | task path: /home/me/ansible/playbooks/abydos/playbook-webserver.yml:35
Playbook last committed by me@host on Sun Jan 5 09:44:27 2020 +0100 (26392ab778deaf86430f36bc7aed942ae04a938c)
2020-01-08 13:26:35,647 p=me u=27195 | changed: [hunan.domain.se -> localhost] => {"changed": true, "cmd": "git log --pretty=\"Playbook last committed by %cn on %cd (%H)\" -1 >> playbook.log", "delta": "0:00:00.003510", "end": "2020-01-08 13:26:35.628135", "rc": 0, "start": "2020-01-08 13:26:35.624625", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
Playbook last committed by me@host on Thu Mar 26 14:57:00 2020 +0100 (2c9aa2030192c2942c5dfb0bcf5976f46fefd774)
2020-04-01 15:44:19,057 p=10821 u=me n=ansible | task path: /home/me/ansible/playbooks/alexandria/playbook.yml:137
The first find
command lists all playbook.log
files below the current directory (recursing into child directories). tac
is the opposite of cat
and lists each file backwards (from last line to first). We use grep
to look for a string ("Playbook last committed") that my Ansible playbooks always insert into the log-file at the end of a run. Note the use of the -A1
flag that gets the matched line and one line after (but because we used tac
, we actually get the line before, which is what we want). The final grep
uses extended regular expressions (-E
) to color highlight several parts of the output (while displaying all of the output, that's what the initial caret does - it effectively matches all lines).
Pretty neat, if I may say so myself.
Some of the refs I consulted to figure out this one-liner:
https://serverfault.com/questions/197123/getting-the-last-match-in-a-file-using-grep
https://unix.stackexchange.com/questions/112159/grep-from-the-end-of-a-file-to-the-beginning
https://stackoverflow.com/questions/307015/how-do-i-include-a-pipe-in-my-linux-find-exec-command
https://superuser.com/questions/914856/grep-display-all-output-but-highlight-search-matches
https://unix.stackexchange.com/questions/366/convince-grep-to-output-all-lines-not-just-those-with-matches
https://unix.stackexchange.com/questions/37313/how-do-i-grep-for-multiple-patterns-with-pattern-having-a-pipe-character
https://en.wikipedia.org/wiki/Regular_expression#POSIX_extended