I really enjoyed this episode. It provided one conceptual breakthrough and a question.
The conceptual breakthrough was to have a master template and then copy and rename it to suit the current task. I had a previous workflow that involved manually creating a new file, then inserting a bunch of necessary text, then manually saving it with a specific name in a specific place. Now I just have one bash command – well, two, chained – to copy the file to its correct place, rename it and open it. Magic.
The question is about hosting scripts on a server. A couple of times in the episode, everyone agreed that hosting scripts on a server was a good idea because you didn’t have to worry about a particular machine being on and available and all that. I have a few scripts that require me to be at the machine to trigger them, and a few others that run with cron, which requires the machine to be on and connected, and I would love to learn how best to run these from a server. Is there a resource you could recommend? Could that be a topic for an episode?
Keep in mind a server is just another computer. Typically they are always on and provide services to other computers, the client computers in a client-server architecture.
You can use any method you would use on your own computer that the software allows on a server with the same facilities.
Web hooks are where you run an application on your server that listens for incoming network connections (e.g. over the Internet) and takes action when the traffic meets specific criteria. Such services are often referred to as a web API, and you can interact with them via web hooks.
You should make sure to secure your connections and triggers appropriately, but this is the basis for many web interactions through integration and other web services. When you run these services, you can potentially trigger actions from a variety of places such as IFTTT, Zapier, Integromat, Apple’s Shortcuts, Drafts, Scriptable, etc., or even, for simple triggers, your web browser.
I use an OpenSource app called Hook on one of my Raspberry Pi Zero devices to trigger scripted operations. I can send data in and provide a security key. It’s great for little web hook triggered automations.
For what it is worth, I also run Node Red as a service on another, beefier Raspberry Pi (though still only a model 2) that allows me to do things in the opposite direction.
Both of these Raspberry Pi devices are acting as servers for me.
Using a terminal application, or even Shortcuts again, you can login and issue commands directly to servers. This was exactly I used to use servers back in my university days. The PCs were barely powerful enough to run Windows, so if you needed to do some data crunching, you’d get some time allotted connect through to a server, queue up your job for your authorised time slot and let the server do the processing in an hour that would take the PC days.
SSH is the typical method you would use to connect through and after that it would just be like sitting at the computer with a terminal session open. I’d advise keys over passwords for anything out on the Internet, and in case it helps, I have a post based on one of my replies on this forum that outlines setting up SSH for Shortcuts. There are plenty of online tutorials for how to do this on desktop machines.
If you have a visual desktop, and not just a command line, you can have other options too. The Mac ecosystem of apps happens to offer several.
- Hazel I have already mentioned, and is great for processing files that come in via a file share, cloud share, or SSH/SFTP.
- Keyboard Maestro a great tool that you can trigger through web hooks (it has its own server), can monitor folders like *Hazel), and has scheduling options. I tend to use Keyboard Maestro for all of my scheduling on the Mac. There is mobile app too, but it is terrible. There is also a web page interface, which I don’t use so much these days, but when I had to print packs of information, I would often do this from my phone via this page, and when I returned to my office, I would find all my information packs printed.
- Alfred is a launcher app, but it also has a mobile app which allows you to trigger things remotely. I tend to use this in place of the Keyboard Maestro app, with Alfred acting as a go between. but you can also use it to trigger automations through scripts, URL calls, etc.
- Better Touch Tool is quite similar in terms of it is more of an interactive tool, but like Alfred it has a mobile app that you can use to trigger automations remotely.
You can host in your own home or office (I have a NAS, plethora of Raspberry Pi, and a Mac mini all effectively running as servers at home), or in a data centre somewhere as a co-location (hosting your owned/rented hardware - e.g. macminicolo.net, macstadium.com ), or a virtual machine (e.g. digitalocean.com, lined.com).
At home/office you will need to either only access locally, use triggers that do not rely on a direct Internet connection to trigger (e.g. cloud file sync, schedules, localised access via VPN into your home/office network), or configure your network to allow external access. For data centres, the standard will be remote access over the Internet.
If you want to be able to have granular control over the automation of doing things when your primary machine is not powered on, then having some sort of server is something you should consider.
The cheapest option is almost certainly a Raspberry Pi zero hosted on your home network. There are no ongoing costs other than a small amount of power and bandwidth. While it is very flexible, it is not particularly powerful. If you need more power, and more options such as specific platform/OS requirements, then one of the other options is going to be necessary.
There are a wide array of approaches to interacting and automating on a server. The ones I’ve listed above are just some of the most common high level options, and as you start digging, you will find orders of magnitude more options, but for people who are automating just their personal lives and maybe a bit of their work on a smaller than SME scale, then the above should be a good starting point for considering options.
Hope that helps.
Absolutely it does. Thank you so much for the very thorough outline. I don’t want to be a pest here, and especially not in an episode thread, but I do still have lots of questions, some of which I am (almost) embarassed to ask. Part of my problem is that most of the online tutorials I have found don’t have the kind of detail or explanation I need.
In the end, I suppose they are all to do with automation, so I will try and break things down and find the best category.