Parts of my Macro only work when monitor is turned on -- any advice?

Hi all – I’m having a problem with a KM macro that I can’t figure out. I subscribe to The NY Times Crossword service and like solving puzzles in PDF form using my Apple Pencil. I wrote a macro to automatically pull up the PDF version of each day’s crossword in Safari from the website, save the PDF to a folder, and then quit Safari. I then have Hazel watching that folder – a PDF named in the right way in that folder then triggers another macro that does some other magic.

MY PROBLEM – when I run the macro (see attached screenshot) to test it, everything runs flawlessly from start to finish. However, when I set it to trigger at 6am and run it on my always on Mac Mini, but with the monitor turned off, the time trigger works and the macro opens the website to the PDF version … and then nothing else happens. No save PDF, no quit Safari. I turn on the monitor to find the website open and showing. There’s no error message in the KM Log.

I’ve tried lots of variations – select Save from the menu instead of using CMD-S, use Chrome instead of Safari, add in different lengths of delays – and haven’t solved it. I’m stumped. Any ideas?

I would actually do this very differently. I’d use the Execute Shell Script command with wget.

wget -O "NYT Crossword -- $KMVAR_FormattedDate" $KMVAR_CrosswordURL

You’d need to set a Keyboard Maestro variable callled FormattedDate to %ICUDateTime%EEEE LLL dd yyyy%, but these two would then replace everything from “Open URL” to the end of your script.


I suspect this would work because shell scripts happen in the background, but if you want to debug your current script, try adding some logs or notifications.

  1. I know from previous experience that Keyboard Maestro can run into trouble with manipulating the UI when the screensaver is on or the screen is locked.

  2. Every other Un*x in the world includes wget by default, but macOS does not, so you’d be better off using curl which is included by default.

  3. Referring to Keyboard Maestro variables in shell scripts uses the prefix $KMVAR_foo (not $KM_VAR_foo)

However, any command line attempt will probably fail because of this detail:

The NY Times Crossword is a paid service, and so you have to be logged in to access these PDFs.

I am guessing that @Marc_K is logged in to the NY Times website in Safari, but he would not be with wget or curl so he would get a 401 Forbidden error like I did when I just tried this.

So, we’re back to needing to run this when we know that the screen is not locked and the screensaver is not on.

Fortunately, Keyboard Maestro is smart enough to handle this. We can use the SCREENSAVER() calculation in Keyboard Maestro:

Quoting https://wiki.keyboardmaestro.com/function/SCREENSAVER:

The SCREENSAVER() function returns whether the Mac is currently displaying the screen saver (or the display is off). It returns 1 if the screen saver is running, or the display is sleeping or locked. Generally, if it returns 1 it is probably not safe to perform UI.

Note that although it is called SCREENSAVER it also applies if the screen is off (and presumably locked).

How to tell UI commands to wait until the screen is unlocked/available:

Put this at the start of the macro:

which basically says: While SCREENSAVER() is true wait for 30 seconds.

Aside: if you do not have a password on your Mac, you could also have Keyboard Maestro run Wake Screen and Stop Screen Saver at the start of the macro, but I am guessing that chances are that your Mac would require you to enter your password at 6:00 a.m.

The downside to this is, of course, that it can’t really run unattended, so you’ll have to be logged in at least once per day for it to run.

However, I don’t know any way around that.

It will show a notification before it starts to run, so you’ll know not to interfere with the mouse and keyboard for a few moments while it runs.

1 Like

Yes, avoiding the UI is preferable in oh so many ways. Including reliability - as here - but also performance.

Excellent catch! Just fixed it.

1 Like

Thanks, @RosemaryOrchard and @Tjluoma for the excellent advice! It took me a few days to get back to this (because pandemic…) – here’s the update:

  1. the screensaver/screen wake solution worked for short time lags – adding screen wake and end screensaver passed a test where I set the macro to run 1 hour later (enough time for the screensaver to run and the screen to sleep). However, it failed the 6am the next morning test. I’m pretty sure this is because of the need to log in to the Mac issue that TJ mentioned. So, good add to my toolbox, but not a solution to the immediate problem.

  2. curl, on the other hand, MAY work. It successfully downloads a PDF file, but the file is corrupted in a way that can’t be opened in PDFPen or Preview and can’t be imported into GoodNotes on my iPad. This happens both with using curl directly in terminal and implementing in KM (see below).

  3. I did run into a problem that I am sure is explicable for those with more knowledge of shell scripting than I have. I tried all sorts of variants of shell scripts within the Execute Shell Script command – with all of them curl returned an error: "Warning: Failed to create the file Sep1720.pdf: Read-only file system
    " My web searching for ways to specify file paths, etc. didn’t come up with a solution.

  4. what DID ultimately work (at least to download the file that can’t be opened) was to write an AppleScript to tell terminal to run the curl command and then a KM macro to run generate the curl command with the correct URL and run the AppleScript…

So – would be interested in any insight folks have into the PDF file downloading but being unopenable and/or if there’s any way to script the saving actions within Safari (which will happily open the PDF and, if able to do so, save it in a way that lets it be opened…).

Thanks again for the help!

–Marc

I feel very safe in guessing at the answer to this one. It sounds like the curl command is trying to save a file to the root directory of your hard drive instead of to a folder in your $HOME.

It took me several times to read this to follow it, but I finally did, although when I fully grasped what you were doing, my reaction was “Oh no. Don’t do that.”

That seems… very fragile, with a lot of places where things could get lost in communication.

It would also suggest that when that curl command is executed, it is happening at / instead of $HOME so at the very least you would need to add something like:

cd ~/Downloads

to the front of your existing command, so it would look something like this:

cd ~/Downloads;curl -O https://www.nytimes.com/etc/etc/date.pdf

But I am not at all positive that would work in your script-to-Keyboard Maestro variable-to-AppleScript system.

Unfortunately, I think you were hoping that curl was getting close to working. I’m really not sure that it is. I really can’t imagine that they NYTimes would leave their highly valuable crossword puzzles in a directory that you could access without being logged in.

That’s hard to say without being able to test it myself. I’m assuming that, by default, Safari opens the PDF in a Safari window/tab, and then you can go to the bottom to save it?

What I would be tempted to do is either configure Safari so that it will download PDFs instead of displaying them – but that would change how Safari handles all PDFs, not just these. The other option would be to use a different browser that you do not normally use, and set that up to log in to the NYTimes crossword puzzle site and download the PDFs instead of displaying them.

But how to make that work really depends on how they the PDFs even get sent by the server.

Reddit has you covered:

Note that it relies on saving your login cookie from nytimes.com into a file. You may need to refresh this file at some point, assuming NYT does not let you stay logged in indefinitely.

This may be the most entertaining constructive criticism I’ve ever received, @Tjluoma!

On a more serious note, the alternative browser/download PDFs seems to have completely done the trick. It gave me an excuse to use the copy of Chrome that I have but only use when Safari doesn’t play nicely with particular websites. So far, every day the PDF downloads and Marc is happy.

Thanks for the solution!

1 Like

Interesting! Thanks for sharing, @rjernst! I’ll definitely file this for my things I didn’t know you could do with shell script, URLs, and cookies bag of tricks!