A Control Panel/Dashboard

Here was my vanity project this summer:

For those of you with OCD who are twitching due to the misalinged bottom frames, this was fixed post screenshot. Sorry.

I’ve been meaning to do something like this for around a year – having an easy to view calendar not on our phones and controlling the MPD server above the kitchen were the main drivers.

I’m going to go through each component of the panel before giving an overview of how the whole thing hangs together, in an endeavour to kinda-sorta document the thing for when it inevitably breaks.

A Music Player (except not)

MPD is… quirky

I tried a few music servers before just settling on a raw MPD install. I was using a Raspberry Pi 3, but it kept freaking out with maintaining WiFi (no idea why) so I ended up with the current server – Ubuntu running on an old Mac Mini via a USB external HDD (since the SATA controller died on the Mac).

I’m astonished this thing works at all, to be honest. Even acts as a bluetooth speaker when needed.

The issue with the MPD server was in operating it – the MALP app remote controller is fine (though quirky in its own way), but not super convenient or transparent to use when wandering around the kitchen. What I wanted was an easily accessible control to simply play/stop/next and display track details.

There are existing web applications I could have used for this job, but they were either too hard to configure right or hugely overkill for what I wanted. None of them quite fit right inside an iFrame either, which was frustrating.

It’s not possible to create a pure JavaScript controller for MPD – it doesn’t allow for WebSocket connections, therefore any web app solution would need a CGI backend. To whit, fine, I’ll roll my own.

What you see above is actually two iFrames – the top (player) section and the bottom (album select) section.

The top section can issue AJAX commands to the backend, written in Python using the surprisingly well-documented Python-MPD2 library.

This is all pretty straightforward, but for two things: keeping the current track data fresh and accessing album covers.

Keeping track data current doesn’t have an elegant solution – I have the JavaScript query the backend every 4 seconds and check for differences in the track name or album artwork file. It’s not fancy, but given I can’t access MPD directly via JS, keeping it constantly up-to-date is beyond my meagre ability.

MPD supposedly allows for access to binary data containing album covers, but the rules for this are opaque and inconsistent – I have a number of albums with corresponding images in the right directories, but no artwork shows up.

MALP works around this by (semi-successfully) pulling data from MusicBrainz where possible. So I did the same – an artwork backend informs the panel of the best image to use based on existing files and downloads the appropriate album cover where artwork doesn’t already exist.

This mostly works, but is hardly robust.

Album Selection

This is mostly smoke and mirrors – I lack the patience or inclination to build a system for gathering data on all the artists or albums in our collection. Instead, I figured we only have a handful of albums at any time we’re really listening to. I’ve created stored playlists for those using MALP and another backend script runs a cron job once a day to add any new playlists to the frontend.

What was most “fun” about this was of course passing the playlist names back and forth – they’re replete with lovely juicy characters such as apostrophes and ampersands which break URLs something fierce.

As a result, the JavaScript on the front end base64 encodes the playlist names before URI component encoding the base64 string and the Python backend undoes all that to load the appropriate list.

At this point I’m remembering why I never stuck with frontend development back when I worked as a dev.

A note about JQuery:

The music player is the only part of the project to use JQuery. I resolved not to use any frameworks where possible – mostly out of bloody-mindedness, but also because it wasn’t really essential for any of the panel components – but the exception was marquee.

Yes, that marquee.

You see, for long track names, they won’t fit in the 300 odd pixels set aside for the player. So for those tracks, I wanted the name to scroll when the track was playing.

To get a reliable marquee effect in this day and age, you apparently have to use a JQuery plugin and about 20 lines of CSS.


It’s a calendar.

One does not simply make a calendar in software. Time is evil when it comes to software development – I’m torn as to whether JavaScript or the entire concept of time itself is worse.

Okay, it’s JavaScript – I have to actually use JS more often.

Rather than reinvent the extremely complex and fraught wheel of time, we can just use Google Calendar’s “embed your calendar as an iFrame” feature. Perfect!

It looks like this:

The “embed as iFrame” feature predates the popularity of dark modes in computing.

Hmm. That doesn’t look like the rest of my panel.

A brief word about the panel’s colour scheme:

I’m not good at colour schemes. More than that, I struggle to care about them much. It’s not that I’m colour blind, I’m just largely colour agnostic. I mean, I’m not going around using hotdog stand theme or something, but beyond that, well, ¯\_(ツ)_/¯

Here’s the thing though – you can’t just restyle the contents of an iFrame when it comes from someone else’s server – there are very important security reasons for that. Given this was a project purely for use in my kitchen, on a device inaccessible to the outside world, I did look for workarounds to this security policy.

As a result, I found a lot of people discussing some super bad ideas for circumventing XSS protections, and there’s probably a fortune to be made in bug bounties if I looked up their Linkedin pages and did some half-hearted prodding on the web apps run by their companies.

So disabling XSS security is a non-starter.

Next best option: User Scripts!

You might know these as GreaseMonkey on Firefox or TamperMonkey on Chrome. They are excellent for bending the world of the web to your own twisted vision.

A basic userscript later and I’m forcing the browser to restyle the Google Calendar in a dark theme. It is absolutely, totally and in all other ways perfect, and I’ll not be moved on the issue.

Don’t click the “agenda” or “week” buttons.

What’s the weather like? Nice.

You see that moon icon? I drew that myself. Graphic design is my passion.

(You’ll note the “sunny” icon in the screenshot at the top doesn’t match this one – Chrome aggressively caches images and it’s challenging to have it download a changed version)

This was a hot mess. So to speak.

There exist APIs for weather which are free. None of the ones I found were worth a damn for local weather – often off by 2-6 degrees (43 degree day? It happily reports that it’s 37!)

There exist APIs which are very expensive. This is not suitable for my flimsy summer panel.

Then… there’s *coughs* scraping weather websites.

I won’t go into detail, but to ensure that the scraping only happens a handful of times a day, I wrote a Python script with Beautiful soup to snag the precis for today and tomorrow along with the temperatures and dump them into text files on the webserver. Then, the weather app reads that data in every 15 minutes or so.

Weather icons are usually pretty awful. I like these ones.

UV was… trickier.

When my wife asked for UV info, I said, “Easy. It’s summer, therefore it’s always extreme. Pack sunscreen and wear a space suit.”

But no, the spousal request was for moment-to-moment UV rating values (or thereabouts).

The official UV rating site for Australia was last updated when Kings of Leon were a big deal on the radio.

I guess at least Kings of Leon released new material since then?

Some digging unearthed a JSON file shuttling from a server to a script on the page and then into some terrible graphing libraries. As well as many, many commented lines of code that were clearly not meant to make it into production.

I pointed some more Python at the JSON file and… oh great, it’s a giant array of UV data for every minute of the day. I’m going to have to iterate through the whole thing and… wait, there’s a property at the bottom of the file called “current_uv_rating”, perfect!

Nope. It’s always set to zero.

Iterating it is. Another cron job runs the Python every half hour.

Have I got news for you?

Getting ahold of news headlines was surprisingly challenging too – there isn’t an API for our nationally funded provider (er, that is, nationally funded provider not directly owned by our home grown Dark Lord).

They do offer RSS feeds (am I hearing decades old music again?) which they seem to have been very enthusiastic about around the time that everyone was done using RSS feeds. Most of the info on ABC’s site regarding RSS feeds is from circa 2011, and I’m cautiously using the word “most” as that implies there’s a lot more information than there actually is. Which is close to none.

There does exist a “just in” feed – an XML file with headlines, links and relevant images for the most recent stories, regardless of topic or popularity. I don’t know what dark science or eldritch divination led me to find it, because you sure as heck can’t track it down using either Google or the ABC’s own search functionality.

At any rate, another Python script + cron job (TM) later, and I’m slurping down headlines to display every 25 seconds on the panel.

Here’s The Thing, though: when you visit an article by smearing your finger on the screen, it opens in a new tab. Which would be dandy, except the only way to get back to the panel again is to:

  1. Press the Windows symbol on the Surface device running the panel (a feature sadly missing from more modern Surface Pros) to access the taskbar, given the panel’s browser is in full screen mode.
  2. Access the on screen keyboard from the taskbar (no, not the pretty one. The accessibility one with all the functionality).
  3. Press fn-F11 to drop the browser out of full screen mode.
  4. Close the tab. Oops, you missed the little “X” with your giant finger. Try again. No, that’s a new tab. There we go.
  5. Access the keyboard again. Press fn-F11 to go back to the full screen mode.

That dog won’t hunt, Monsignor.

War were declared.

I need a big, meaty, easily touchable “close” button on articles when they open.

iFrames are once again unsuitable for this job – I could just display the article in an iFrame with a button outside it – but ABC have (entirely reasonably) prohibited using their stuff in an iFrame, specifically to prevent nefarious purveyors of stolen bits from claiming their work as their own (not that I’m a purveyor of stolen bits. I don’t purvey them, thank you very much.).

Once again we turn to… Userscripts. Hooray.

The CMS used by our National Broadcaster is, like all CMSes, prone to creating multiple obscure classes in its HTML.

Therefore finding the right element to inject a button into was tricky.

It’s not robust by any means, but it currently finds the second link on the page (the ABC logo) and sticks a close button right next door. It then tracks down the “FixedHeader” data component in the page and injects another button – this is the overlay banner that sticks around at the top once you scroll down.

Et voila – one click return to the panel.

The #bestoftimes, the #worstoftimes

Fig 1: A somewhat silly clock

This is a Javascript clock. It is pretty silly. Maybe one day I’ll replace it with a graph of utilities usage or something. It’s a nice clock though.

Fig 2: A less silly, but far more boring, clock

This is another clock. This one is more sensible and useful. I don’t know what it might do on Wednesdays in September since I didn’t use a fixed width font and it’s pretty close to the full width of its container.

I’ll be shocked if the panel is still fully working in September though.

How does this dang thing work?

It runs on an old Surface Pro, which is comically overpowered for the job it needs to do. That said, it is quite old and water damaged – part of the reason for the dark theme (apart from the fact that everyone’s doing it, it’s cool man, what are you, some kind of square?) is because water ingress damaged the screen some time ago, leaving weird blotches that are only visible when displaying bright images.

The panel itself is just some HTML with iFrames and a little JS to reload the calendar and weather. It’s actually not even hosted on the Surface, but instead sits on our media centre PC/NAS.

The media centre runs the necessary backend cron jobs and hosts the music, which is played on the third, far more interesting and terrifying PC, the MPD server.

To sum up – this runs off three physical machines, 3 cron jobs, a locally hosted site, an MPD server, two userscripts which are dependant on their target sites “not changing too much”, various Python scripts for scraping sites (which also need to not change much, please and thank you) and a Python script parsing and processing an RSS feed that its owner no longer seems to care for.

The panel’s days are numbered – everything on the web is transient, even local pages which are only used in a single household.

This is our reality – endless sandcastles

This isn’t necessarily a bad thing – a bit over a year ago, we had a Google Nest we’d received for “free”. It was fine.

It played music – but not necessarily the music we wanted or in the order we wanted. And it had ads.

It could tell you the weather, but you had to ask.

It could tell you your schedule, sort of.

It could add to our shopping list – provided you used the shopping list page Google created, which has no API.

When it stopped working, I searched for a solution and found nothing useful. It was apparently a known flaw. Had we paid for it, we’d probably have some recourse for replacing it.

I didn’t feel the need to buy another.

At least when this panel breaks, I’ll be able to find out why and have a chance at fixing it.

Bon Voyage, little kitchen bench automaton!

Paint IP on Epaper: RPi Startup

A picture of text. Well that's handy.

I’m currently playing around with one of these E-paper modules:

Epaper panel showing clock, weather and timetable

Panel displaying clock/timetable image

The panel is great for its price and ease of use – Waveshare ship this model with a Raspberry Pi hat and provide a number of software libraries for interacting with them.

I’ll follow this up with a breakdown on how I created my wall clock, but just quickly, a quibble with Raspberry Pi/headless systems in general:

My place of work has a… restrictive network when it comes to BYOD. It’s better than it used to be, but unless you can easily connect to an enterprise wifi network (difficult, but not impossible with the Pi) you’ll need to connect an ethernet cable (and not mention it to the techs). In either case, you’ll get an IP address. But you won’t be able to figure out what IP address without plugging in a display.

There is an opportunity with a display like this to work around this problem: run a program on start up to push the IP addresses of all network interfaces to the panel.

Without further ado, PaintIP, a small Python script that will get a list of IPs for your interfaces and paint them to the panel:

A picture of text. Well that's handy.

PaintIP running on a Mac, which seems kinda pointless

The “screenshot” pictured here is actually the image generated by the program which is then pushed to the panel.

It’s a minor thing, but sometimes the ability to bring your device in from home, plug it in and immediately know which IP to SSH to is a lifesaver.

Rejected (and hilarious) alternative:

A program which simply takes a screenshot of the primary console, munges it into the right resolution at 1 bit colour and paints the display every 6 seconds (why 6 seconds? because this particular panel has a 6 second refresh – not ideal, but mostly workable).

I actually made this abomination, based on a technique from this here blog post.

snapscreenshot -c1 -x1 -12 > console.tga
convert console.tga -depth 1 -colors 2 -colorspace gray -resize 640 -negate -gravity center -extent 640x384 -sharpen 0x3.5 BMP3:console_bw.bmp
python curator.py console_bw.bmp

It literally just uses snapscreenshot and ImageMagick’s convert program to create the 1 bit image of appropriate resolution and then calls a Python script to display said image.

Could I have put all this in a single Python script? Probably, but it’d have system calls to run snapscreenshot and convert (Python’s ImageMagick libraries leave much to be desired) and honestly I just needed the thing working now.

Curator is a very basic script that takes an image as an argument and displays it on the E-Paper panel – barely a modification of Waveshare’s example script.

Still, there might be the odd time when it’ll be useful to see a shot of the console for debug purposes – nothing stopping someone from configuring a hardware button to run the console dump shellscript.

Oh, and there are a handful of ways to get a program running on startup with a Pi, but the one I ended up using is simply inserting it into rc.local.

Python, oursql and executemany

I’ve never had cause to use executemany before, so perhaps it is unfortunate that my first encounter be in the context of oursql’s somewhat anaemic documentation.

I’ve just spent well over an hour trying to get a simple executemany statement to work – in retrospect, the solution was obvious. But the presence of an example (just one!) would have made things immediately clear.

To that end:

reportDB = oursql.connect(host = "localhost",
                          user = "LapsedPacifist",
                          passwd = "ContentsMayDiffer",
                          db = "AnythingLegalConsidered")

with reportDB as query:
                INSERT INTO assets(clientid, assettype, model, deployed, name)
                VALUES(?, ?, ?, ?, ?)
                """, (clientID, ASSET_TYPE_UNKNOWN, model, deployed, name))

  assetID = query.lastrowid
  addresses = []

  for ip in ips:
    addresses.append([assetID, ip])

                    INSERT INTO asset_ips(assetid, ipv4address)
                    VALUES (?, ?)
                    """, (addresses))

That’s it – just as oursql’s parameterisation is looking for a list of parameters, executemany is looking for a list of parameter lists – not just a list of strings.

Hence the for loop creating a list with the two parameters and then appending it to the executemany parameter list.

I’m sure there’s a better, more pythonic way of doing that – please let me know if I’m missing something obvious.

And behold, the grand total of what oursql’s documentation has to say about executemany:

Additionally, executemany() is lazy; if passed a generator or any other iterator which does produces values lazily, values will only be taken from the iterator immediately before they are sent to the database.

executemany(query, parambatch)
Execute the same query with different sets of parameters. This is more efficient than calling execute many times with the same query.

I don’t want it to sound like I’m down on oursql – I feel it’s the superior library for MySQL in Python, and it’s still quite young – but this particular gap in the documentation had me pulling out my hair.

I’ve read some complaints regarding scalability of executemany in oursql – fortunately my code won’t ever have to handle huge numbers of queries at one time.