Category Archives: Software

Adventures in OPNsense

Our house received an upgrade to full fibre optic last year, and with the increase in available home bandwidth came the opportunity to retire our old NetComm router/modem/WiFi unit, which had been, if we’re generous, “adequate”.

Or, in fact, inadequate in a number of ways – its WiFi range was actually abysmal, something I only found out when I migrated to a dedicated WiFi unit and discovered that a single device can actually cover our entire property.

Also, it “supported” IPv6, in the sense that it could, technically, have an IPv6 address for a brief period of time, before just quietly giving up on routing IPv6 traffic until you either reboot or get exasperated and switch off IPv6 support.

So in the spirit of questionable decisions, and egged on by friends who are far more skilled and experienced in this sort of thing, I purchased a cheap all-in-one fanless PC with four 2.5Gb ethernet ports.

In this house, we enumerate from zero

For the curious, it runs at around 55 degrees in the Australian summer, 65 if I make the CPU think hard. Certainly an improvement over the poor old Mac mini I have running as an MPD server in our kitchen, which routinely idles at 80 degrees.

Because I can’t ever be content to do things the easy way, I put Proxmox on it first, with the intention of running OPNsense as a VM, alongside a PiHole instance.

My initial set up experience was tainted by an inability to get OPNsense to register a connection via the WAN port, which resulted in around 45 minutes of frustration before realising that my ISP requires me to manually “kick” connections between router changes. As a result of this faffing about, I now at least have a document detailing exactly what each of the physical ethernet ports are called in Proxmox and what their MAC addresses are – probably something I should have sorted out before doing anything else anyway.

Once the router had a connection, everything ran great. For thirty minutes. After which, the router simply couldn’t talk to the gateway any more. No Internet connection.

In OPNsense’s dashboard, it indicated that EVERYTHING IS FINE. Which, you know, not true.

This is a lie.

A reboot results in the same behaviour – everything is fine for around half an hour, then no WAN connectivity. Renewing the WAN connection fixes things… indefinitely. Until another reboot.

Thus began The Troubleshooting.

Various settings and configs were checked. IPv6 disabled (just in case), hardware VLAN filtering disabled etc etc

Same behaviour.

Install a completely fresh OPNSense VM – exact. same. behaviour.

And the whole time – once I renew the WAN connection, it fixes itself for as long as it remains up.
The command I ended up running in the shell:

configctl interface reconfigure wan

Magically fixed it… but only if it had already broken. That is, I couldn’t just run a renew at the end of the boot sequence and call it a day – I had to wait until the WAN connection failed before I could renew it – and that just didn’t fly (not that appending some magic words to a startup script would have made me happy).

I ended up digging through the DHCP client leases file in /var/db (in my case dhclient.leases.vtnet1) and noticing some strange overlaps in the renewal/rebind/expiry etc times.

These leases files look like this:

Turns out, this a log, rather than a config – FreeBSD is writing the dates and times there as a record, not an instruction. But this was the clue to help figure it out – when router first booted, the lease file was written to but the time for renewal and expiry passed without the router even trying to renew the connection.

Once I reran the renewal command manually, it worked fine, but crucially, it was writing dates and times that were wildly different to the initial time on bootup.

Anyway, as with all things, it was DNS.

A haiku

Well, sort of.

When I set up the initial Proxmox install, I somehow (?) managed to set its DNS server to be 127.0.0.1. Surely that can’t have been the default?

Anyway, that meant that while it could do all the work of creating VMs just fine, it couldn’t, among other things, talk to NTP servers to figure out the time.

But it did know it was in the UTC+8 timezone. And it did assume that the hardware clock was in UTC time. Which it wasn’t. It was set to local time. So my VM host believed it was 8 hours ahead of when it actually was.

When my router booted up, it was given this time and it then obtained a lease and made a note to renew that lease in 15 minutes. 30 minutes, tops.

But it then used an actual DNS server to do an NTP lookup for the correct time – at which point it said to itself, “hoo boy, I am 8 hours ahead of where I should be! Lemme just fix that right up.” But its lease now thinks the expiry is not for another 8 hours – hence failing to renew and not having any WAN connectivity.

To add to the confusion with all this – FreeBSD in all its wisdom writes dates and times to the lease file in universal coordinated time – not local time – without any indication that this is what it is doing. So when the router first obtained that lease and wrote the wrong time to the log, it looked to me like the correct time, but when I renewed the lease manually, it had corrected its own time and was now logging, what appeared to me to be 8 hours in the past.

Obviously, I think everything should always just be in UTC and that there are no problems or issues that would or could be caused by adopting a world free of timezones, but, please, indicate that somewhere, hey?

Anyway, OPNSense seems fine and good. I’ve managed to do some port forwarding and set up queues to minimise bufferbloat, so all is right with the world.

Now if I can just wrangle IPv6 DNS…

MPD Clients and Content Sorting

Perhaps you, like me, use an MPD server to manage and play a largish music collection. Perhaps, you, like me, are finding the odd weird thing happening in your client where multiple albums appear for the same set of tracks, or tracks are split between two different, yet somehow identical albums.

Probably not though. However, this post exists to provide me with a reminder of the techniques used to resolve those issues.

Before we begin, my primary client for this library is MALP on Android – it’s actually very good, but gets grumpy if your MPD is old (FWIW mine is 0.22 – I went to the trouble of compiling by hand) and is very strict about some tags (which is a Good Thing, but can be fiddly)

Step One – Basic Tag Hygiene

Make sure the tracks for your album all have ID3v2 tags (remove all the v1 tags – don’t need ’em) and that the track numbers, total tracks, artist, albumartist and album tags are all correctly set. Any variations here can cause issues where a track magically belongs to a different album with the same name, or lives on two different albums somehow

Step Two – Check for MusicBrainz Tags

The MusicBrainz project and its tagger – Picard – are wonderful. However, music releases are squirrely and it can be hard to pin down exactly where your tracks fit in the listed releases.

In an ideal world, you can just add the appropriate release on MusicBrainz and Picard will simply tag all your tracks correctly – done and dusted.

imdoingmypart.gif etc etc

But not all tracks belong on MusicBrainz. And not all albums exactly match the releases. If you’re me1, you have music that just isn’t on MB at all and never will be.

The issue comes from when some tracks in an album have been tagged previously by Picard, but others have not – because Picard leaves behind super secret custom tags to help organise music. Which is great, except that MPD and MALP can read these custom tags and if not all the tracks in an album having matching ones, weirdness ensues.

So I downloaded this thing, which does a very specific job, but I used it solely to view “extended tags” and delete all the MB related stuff for albums with a mixed tagging history.

Step Three – Disc numbers?

If some of your album’s tracks have been tagged “disc 1/1” or similar and some have nothing in that tag – these tracks will still appear on one album, but the order may be all wrong. Just make your tags consistent – either all disc 1/1 or no data in those tags.

Empty tags for this can be intepreted as “disc 0”, causing those tracks to be erroneously listed prior to others.

That’s all I have so far, I’ll update with screenshot examples later.

Headless Raspberry Pi – Circa 2019

With the advent of the Raspberry Pi 3 and Zero W, newer Pi-s come with wireless baked-in, which is (IMHO) a welcome addition to help make setting up a Pi without Ethernet much more straightforward.

In fact, given my employer’s aggressively antisocial wireless network 1, it has become the norm for me (and my students) to set up a fresh Raspbian install using either a mobile hotspot or by tethering their phones. In neither case is Ethernet a useful option2.

“Headless” installs are setups which do not require a keyboard, mouse and monitor – given the ubiquity of networking and the low-power of Pi-like devices, it makes sense to be able to use an SSH session to do all your setup and get your device running without the hassle of directly using I/O in front of it. Plus, all the reference sites you’re using are probably open in the hundred or so tabs on your main computer3.

Without further rambling, here is the current easiest way to set up a Raspbian Buster install to be headless, using MacOS as the host machine:

  • Write your image to a micro SD card. I am lazy and use Balena Etcher rather than DD, although at the time of writing, it’s a little broken when used with MacOS Catalina.
  • Remove your SD card and pop it back in. Do not boot your Pi with it at this stage. You need to make these changes for the first boot or Raspbian will ignore them.
  • It should appear mounted as “boot”. This is the only section of the new filesystem you can read and write on the host machine.
  • Open a terminal, because we’re all adults here and GUIs are only for circumventing the DD command using Balena Etcher.
  • Change directory to the boot partition on the card and create a file called “ssh”:
cd /Volumes/boot
touch ssh
  • edit a new file called “wpa_supplicant.conf”4
nano wpa_supplicant.conf

Put this content in it, replacing the placeholders in quotes (but keep the quotes) with bits relevant to you:

update_config=1
country=AU
ctrl_interface=/var/run/wpa_supplicant

network={
 scan_ssid=1
 ssid="Your Network Name"
 psk="Your Network Password"
}

That’s it – you can eject your card safely5, pop it in your Pi and power up.

Some additional notes, for fun and profit:

Q. How do I find the IP of my Pi after it boots, so that I can SSH in?
A. If you’re using an Android phone to hotspot – you can find a list of connected devices in the settings along with each of their IPs.
If you’re using a mobile hotspot or home router, log in to its web interface to view connected devices or get your phone to connect to the network use a network scanning tool such as Fing.
If you’re using an iPhone to hotspot – umm. I don’t know. Last I checked, they tell you how many devices are connected, but not any details about them (thanks Apple! I hate it!) and Fing doesn’t return details when it’s run on the hotspot itself. Arp has decidedly mixed results. There is apparently an app that can be downloaded to show you details of devices connected to your iPhone.

Q. Isn’t there other info I should include in my wpa_supplicant config? Like the country code?
A. Probably. It works fine for me without country code and I’m all for minimising the content that has to be customised in a config. Perhaps AU and UK wireless devices just interconnect fine, or perhaps some other WiFi voodoo has done away with the need for CCs. If you’re in the US, does it not work without a country code? I do know that in a previous version of Raspbian (Jessie, perhaps?), the Pi would refuse to connect if CC wasn’t set, so do with that what you will.
UPDATE IN 2020:
You absolutely do need to set the country code, especially if you’re using the 5GHz bands. Recent RPi OS builds seem to enforce this again. I’ve included the AU code in the example above (since I’m in AU), but you’ll need to set yours as per this list.

Q. I have to put my password in a config in plain text. What gives?
A. You don’t have to. There are ways to hash it and store it in the config.
Here’s the thing though – I’m betting this WiFi password is either a home network or a hotspot – and in either case, it’s a shared key in the literal sense of the term – lots of people know it, and it’s trivially easy to change it (at least on the router).
If you’re setting this up on a corporate network, my little config above won’t get you connected anyway. I’ve made it work in the past, but mobile devices I’ve connected to our corporate network have been… idiosyncratic. They lose Internet access or randomly change IP or need to be power cycled with a 15 minute delay every day. In short, I haven’t found Raspbian, or even many Linux distros that are cooperative with (what is probably a poorly configured) corporate WiFi, so in this day and age, it’s easier just to work around it rather than try to join it. :-/

Python: The value of “with”

I never learned to code with Python; my first forays into development were with batch files (I kid you not) and then Visual Basic (which taught me many things I spent years unlearning).

Python is, in many respects, a great language for learners (which I’m not going to discuss today).

There is however, a great deal of… less than intuitive show-off code that can be, nay is encouraged to be, written using the language. Solutions are deemed to be “Pythonic”, a term as nebulous as “elegant” and often resulting in code just as unreadable to the casual observer or Python learner.

In some respects, this is about reducing the number of lines written – something fraught with peril for someone new to coding.

But there is a seductive element to Pythonic solutions – they don’t require you to twist yourself up in lines of boilerplate just to overcome a (often commonly encountered) problem.

The “with” keyword is not especially unique to Python, but is actively encouraged, and with good reason. This is one case where fewer lines is definitely better. Continue Reading…

ECU Assignment Stapler

A working copy of the stapler can be found here.

Why does this thing exist?

When you submit assignments, you should submit them in PDF, not Word format.

There are a few reasons why, but the main two are as follows:

  1. You can be more sure that your tutor or lecturer will see the same thing you submitted
    Word documents can display differently depending on the version or device used.
    This is far less of an issue with PDF.
  2. You can’t accidentally munge your keyboard and change the final document.
    Probably not normally a consideration, but after 9 hours staring at papers on the significance of First Name Consonant Frequency in Childhood Misbehaviour*, you can easily make silly mistakes and be completely unaware that you just moved a crucial paragraph and are now submitting the antithesis of your intended argument.
(*Not a paper, but totally should be)

Continue Reading…

Python, oursql and executemany

I’ve never had cause to use executemany before, so perhaps it is unfortunate that my first encounter be in the context of oursql’s somewhat anaemic documentation.

I’ve just spent well over an hour trying to get a simple executemany statement to work – in retrospect, the solution was obvious. But the presence of an example (just one!) would have made things immediately clear.

To that end:

reportDB = oursql.connect(host = "localhost",
                          user = "LapsedPacifist",
                          passwd = "ContentsMayDiffer",
                          db = "AnythingLegalConsidered")

with reportDB as query:
  query.execute("BEGIN")
  query.execute("""
                INSERT INTO assets(clientid, assettype, model, deployed, name)
                VALUES(?, ?, ?, ?, ?)
                """, (clientID, ASSET_TYPE_UNKNOWN, model, deployed, name))

  assetID = query.lastrowid
  addresses = []

  for ip in ips:
    addresses.append([assetID, ip])

  query.executemany("""
                    INSERT INTO asset_ips(assetid, ipv4address)
                    VALUES (?, ?)
                    """, (addresses))

That’s it – just as oursql’s parameterisation is looking for a list of parameters, executemany is looking for a list of parameter lists – not just a list of strings.

Hence the for loop creating a list with the two parameters and then appending it to the executemany parameter list.

I’m sure there’s a better, more pythonic way of doing that – please let me know if I’m missing something obvious.

And behold, the grand total of what oursql’s documentation has to say about executemany:

Additionally, executemany() is lazy; if passed a generator or any other iterator which does produces values lazily, values will only be taken from the iterator immediately before they are sent to the database.

executemany(query, parambatch)
Execute the same query with different sets of parameters. This is more efficient than calling execute many times with the same query.

I don’t want it to sound like I’m down on oursql – I feel it’s the superior library for MySQL in Python, and it’s still quite young – but this particular gap in the documentation had me pulling out my hair.

I’ve read some complaints regarding scalability of executemany in oursql – fortunately my code won’t ever have to handle huge numbers of queries at one time.

Promiscuous Plink

Sometimes it’s necessary to do things the wrong way.

Plinkmiscuous is the world’s tiniest, laziest modification to the very excellent Plink, part of the PuTTY suite of software.

Plink is a command line only connection tool – it has much of the core functionality of PuTTY without all the fruit of a window and GUI. It’s often necessary to connect  through to *nix servers from Windows programmatically, and Plink is generally the tool for the job.

Unless you have a scenario where you can’t know the SSH fingerprint of the server to which you are connecting, or can’t store it (PuTTY puts things in the registry – I’m assuming a hangover from the bad old days of XP) for the account you need to run as.

In these circumstances, the otherwise fully automated Plink demands user interaction to confirm the fingerprint as OK.

Obviously, this is by design – the whole point of SSH is that you can be sure the server you’re connecting to is indeed the one you want before you start sending over authentication details.

But sometimes, security be damned, you need to do something dirty. Please don’t ever do it in a serious or production system.

To that end – the traditional way to “break” Plink and make it work regardless of a stored fingerprint is to pipe or otherwise send a “keypress” for confirmation.

That method sucks.

Plink is open source, so that means we can modify it when it fails to do what we want.

So that’s what I’ve done – it was a change of exactly 1 byte (and some Visual Studio fru fru).

You can find Plinkmiscuous on the software page. It’s based on the current version of Plink at the time of writing. I can update it in the future, but make no promises (and the code change is so simple, you should do it yourself).

And it’s probably worth rethinking whatever it is you’re planning to do with Plinkmiscuous so you don’t have to use it. :-P

Logitech Restarter

This will be of interest only to those who: have a Logitech keyboard with a fancy display, put their PC to sleep from time to time.

The Logitech Restarter is a little program designed to do nothing more than kill and restart Logitech’s Gaming Software.

Hardware manufacturers are notoriously bad at making stable, usable software (barring, one hopes, drivers). LGS isn’t as awful as some offerings in the past – but one area in which it does fall down is maintaining its connection to the keyboard’s display after waking from sleep.

This little application seeks to work around that problem by automating the user’s only recourse: killing the application and starting it again.

Code, binaries and more info at my software page.