Watch folder with Automator and iCloud

macOS
1

Hi,

I’ve created a rule in Automator in which certain file types in my iCloud drive should be moved to another folder. It works well when I manually add a file, but if the file “appears” after I sync them via another device (e.g. iPhone), the action doesn’t run. Can someone confirm this is a limitation?

I’m trying to avoid buying Hazel, as that’s the only file automation task I need. But of course, if I run this rule with Hazel (currently on trial), it always works.

I’m adding my Automator rule if someone wants to have a look and see if I’ve missed something.

Thanks!!
Screenshot 2020-08-04 at 18.21.13|690x393

2

Well you certainly aren’t the only person to have experienced this.

https://discussions.apple.com/thread/8307034

I’m wondering if it is triggering before the file is downloaded rather than not triggering at all? You could check that by having it write something to a file when it runs.

An alternative to try might be covered by the section “Watching Folders Using an Idle Loop and a Stay Open Script App” on this archived Apple documentation page.

I would add that Hazel is of course brilliant and if you do end up getting it, I’m sure that you could find many more ways to get your money’s worth from it.

3

Thanks. It doesn’t get triggered at all, as part of the rule is to also rename the file. That also doesn’t happen.

I’m terrible with AppleScript, so I have little clue on how to configure it. I guess Hazel will be my solution. I’m a heavy computer user, but still struggle to see other uses for Hazel in my case.

4

Not renaming the file does not guarantee it is not running. If the issue is the validity of the file, for example you may not be able to perform any operation on the file at that time even though it is listed as existing; perhaps being locked for sync. That was why I suggested something independent of the file itself (the logging) and that was only dependent upon the trigger, to confirm that was taking effect.

At the end of the day the actions are the key part, so that avenue was purely investigatory.

5

OK, clear. Thanks for your help.

6

This was already suggested, so I don’t mean to overstep. I just wanted to recommend you implement a watched folder that does nothing more than report the thing that triggers it, which in the case of a file, will be its full path. This can be written out to a file.

To do this, create your Automator workflow as you already know how for the purpose of watching a folder. Then have a single Run Shell Script action with Pass Input set to as arguments. Then enter the following code:

printf '%s\n' "$@" > ~/Documents/"triggered_at_$(date +'%F %T' | tr -Cd '[:digit:]').txt"

image
image888×346 67.9 KB

7

I don’t really understand what’s that supposed to do. Can you please clarify?

8

It was originally suggested by @sylumer that you test whether your watched folder gets triggered and to determine what things are responsible for triggering it.

This workflow, if implemented as a Folder Action, will report exactly when the action (rule) got triggered as well as the item or items responsible for triggering it.

You currently have a Folder Action set up to watch for file addition to your iCloud drive. In order to diagnose the problem with your workflow, you must first establish whether these iPhone files trigger the folder action at all, which isn’t something you can discern from your present workflow. Therefore, if you were to open the workflow in Automator for editing, you can deactivate all of its current actions (this is just to spare the hassle of deleting them and later having to recreate them—either way, they need to not be functionally affecting this diagnostic test), and insert the action I proposed at the top. After saving, you can add files manually to your watched folder or sync files from your iPhone so they appear in the watched folder. Every time this happens, you will find a file appears in your ~/Documents folder (you can change this to something else if you prefer). The file will be named "triggered_at_..." where the suffix will be a date-time string (yyyymmddhhmmss) that tells you exactly when the trigger fired. If you open the file, you’ll find the file path or paths of whatever triggered it.

1 Like
9

I have had issue with Hazel running an action on a file before it was fully copied (large files), and it seemed as if the rule was running properly but the system was preventing any action on the file due to the move/copy process not being completed.

Essentially the rule was being matched and the actions applied, but the system had “locked” the file from being modified (renamed etc).

That was solved by adding a delay into the process (“last modified was x minutes ago”). I’d suspect something similar is going on with the Automator action.

And I will second the sentiment of @sylumer that Hazel is well worth the license. It takes a bit of experimentation to uncover the real power sometimes.

10

The difference in this situation is that @Bernardo has found that Hazel seems to trigger appropriately whereas the Folder Action does not. And I completely understand why one would not wish to spend money on a third-party solution for the sake of a single feature that, in principle, is a feature already built-in to macOS (albeit it without the fancy user interface that makes it easier to do things).

I think Hazel is a complete waste of money and would recommend that @Bernardo hold on to his cash. There might be a very simple fix for his Automator workflow that just needs a little bit of investigation to see what the actual issue might be.

If, at some stage, @Bernardo does wish to invest in third-party software capable of providing the functionality of Folder Actions and the simplicity of Hazel, but also a ton of other automative stuff that isn’t something otherwise achievable to users with the ready-available features built-in to macOS, then he will get more bang for his buck by looking at Keyboard Maestro.

Indeed, many users run both Hazel and KM in parallel, but that’s kinda like taking your TV into the cinema so you can watch both.

11

Re-reading my reply above, it perhaps sounds as though I was disagreeing with your hypothesis, so just wished to clarify that I don’t. This is, indeed, a very reasonable explanation for what could be happening, although it’s not the only explanation.

The first step in finding out is to do a diagnostic test as intimated by @sylumer, for which I decided to try and spare @Bernardo the ball ache of figuring out how to do so, which I hope I’ve been able to clarify the procedure for him. It’s very quick and simple, although it provides only the first two clues, namely, whether or not iPhone-synced files can and do trigger a folder action. If they do, then the diagnostic workflow I suggested will either result in one file if triggered either at the start of the end of synchronisation (the contents of this file will clarify which); or it’ll result in two or more files, which should contain the placeholder file path in all but the final output file, which should contain the file path of the fully-downloaded file once transferred across from the iPhone.

If none of this happens for iPhone syncs but it does for manually-added files, then I would be very, very interested, because it means Hazel can somehow detect the appearance of files that macOS doesn’t expose to Automator. If this turns out to then be true for AppleScript-led folder actions, I’d probably go on a personal mission to find out exactly which API Hazel is using to monitor for filesystem changes (the two methods I’m aware of are FSEvents and the Endpoint Security framework, but I’m not sure what each of these can do that’s exclusive of the other with respect to file system monitoring).

12

While I would agree that Hazel would be overkill for solving this one single situation, which agreed should be achievable with a Folder Action, I still find it valuable for many tasks on the Mac. The suggestion wasn’t to buy a license just to solve this one issue, but that it may solve this issue and offer additional functionality.

I have demoed Keyboard Maestro, but found it very confusing and also it seemed to be the opposite of Hazel: actions instigated by the user rather than automated by system events. Maybe I didn’t understand KM properly.

Perhaps for people more comfortable with advanced coding and shell scripting, Hazel isn’t worth it. But for people like myself and the original poster, with very limited coding skills, these types of apps are very helpful to achieve the results we are seeking.

I have been unable to replicate most of the things i do with Hazel using Automator (perhaps due to lack of coding skill/experience), and my shell scripting skills are essentially zero, so I need a tool that makes tasks more accessible.

13

You didn’t need to defend your reasons for wanting automation to be simplified. I think you’re absolutely right about these softwares making automation more accessible to those less able to script something themselves. In fact, I may be very comfortable coding something that I know I don’t need to rely on a third party option for, but sometimes, I just want the easier option and if it’s already running in the background, Keyboard Maestro was usually very good at saving me the ball ache of having to think.

Without wanting to sound rude, I think that might have been the case. But I also don’t think that’s your fault. I initially had reservations when I demoed KM. I found its interface dated, and building macros to be not as intuitive as they could have been. I decided against it. Then when a friend gave me a fully licensed copy he no longer needed (his Mac became a Windows PC), I got more familiar with the program and how powerful it is.

I still hate its user interface, and I don’t retract what I said about its unintuitive macro editor. In this respect, Hazel is a standout winner for how slick it feels. However, Hazel has one purpose and it does it easier than a Folder Action, but it’s completely redundant if KM is installed, as KM can create watched folders and run complex rules more so than Hazel.

KM, despite your heretic claim to not have system triggers, actually has in insane number and variety of system triggers: keyboard hotkeys, wifi state changes, watched folders, cron, pixel detection, application changes, window changes, Bluetooth devices, usb devices, typed character sequences, and stuff I can’t remember. It then has a lot of actions that can manipulate text, images, system hardware, menus, etc. Macros can rewrite themselves, and be triggered via a URI scheme, or remote URL over the Internet. Plus, there’s a direct line of communication with the developer, which is so rare.

Don’t judge yourself based on Automator making simplifying a task quite complicated. I tend to avoid it. Not to harp on, but if I have KM installed (I don’t currently), it lets me severe my ties completely with Automator, as system services, folder actions, calendar alarms, and anything else Automator thinks it does can be done better in KM. But, I’m currently not installing any third party software outwith the bare minimum, and seeing what it gives back versus what it takes to operate like this. But, prior to all of this, all of my folder actions were only ever implemented through AppleScript, without using Automator. It’s more versatile, and you can write a script that connects a watched folder to an Applescript which saves right clicking through the Finder to set it up. I hate right clicking.

That’s because everyone always goes on about bash. Pfft. Use FiSH. It’s easy to learn, and more powerful in the ways that are important.

14

You are assessing my coding skills here… and I have to say that you are right :sweat_smile:

15

I’ve followed your instructions and it gets triggered by the file added via iPhone. However, the file is not affected/moved.

If I manually do so, then it gets moved. There’s a clear distinction in behaviour, therefore: if the file “emerges” from iCloud, it’s never moved. If I manually put it there (on iCloud folder in Finder), then it gets moved immediately.

Another thing to note is that files that were there before I created the rule in Automator are never touched by the rule. However, if I enable the same rule in Hazel, they do. It seems to me that Hazel rescans the folder to apply the rule every so often, and Automator only do it once, and within the described limitations. With that setup, at least.

16

If all you were ran was the diagnostic testing bit I described, then nothing should have been moved for either manual or synced files. I think the issue of the files being moved needs to be something you focus on later, because there’s still some important details that are missing. I’ll highlight the specifics in bold.

When your Automator workflow contains only the action from my first reply, and nothing more, how many files get generated through its triggering when an iPhone file syncs into the watched folder ?

You did mention that the iPhone synced file does trigger the workflow; did you open the generated file(s) to see what the file path(s) were that triggered it ?

Your deduction about what’s happening in Hazel that isn’t in Automator is almost certainly on the ball. I imagine Hazel has a setting or a property associated to each rule that dictates whether or not it applies the rules to pre-existing items. It would be very annoying if it did this and didn’t give one the option otherwise. Adding this function is really easy using launchd, which can execute scripts on a schedule/timer.

17

@chri.sk thanks for your help and patience with me. I had run the workflow as you instructed and only later reactivated the other elements. Below are the results with the workflow containing only your script.

So let’s see:

how many files get generated through its triggering when an iPhone file syncs into the watched folder ?
A: two files were generated

what the file path(s) were that triggered it ?

A:

path of the first file triggered - /Users/bernardo/Library/Mobile Documents/com~apple~CloudDocs/.PNG image.png.icloud

path of the second file triggered - /Users/bernardo/Library/Mobile Documents/com~apple~CloudDocs/PNG image.png

18

Perfect.

This is precisely in line with one of the expected outcomes I discussed in a reply to george_c:

From this, we know that your workflow can, and is being triggered by iPhone synced files perfectly fine, and even better, it is triggered by both the appearance of the placeholder file (the .icloud file) denoting the start of the syncing process, and its eventual renaming to become a .png image file denoting the completion of the sync. That’s basically the ideal outcome.

It means that, whatever is preventing your workflow from doing what it’s supposed to is happening inside the workflow itself, which will require further investigation (isn’t debugging fun ?)

What I would be inclined to do next is to take this diagnostic workflow, which contains that one shell script action, and reactivate the first action from your original set, namely Filter Finder Items. You would want to put your action first, and my action second.

Delete previous trigger files generated from your earlier tests. Then run the test again with those two actions. Basically, we’re going to isolate the point in your workflow where either there are no files generated, or there is one file generated but the file path inside it is not for the fully synced image file (e.g. /Users/bernardo/Library/Mobile Documents/com~apple~CloudDocs/PNG).

19

Bravo, @chri.sk

I appreciate your help. So let me show you a screenshot, so you can confirm that I’ve followed your instructions.

The result is an empty, single triggered file.

Screenshot 2020-08-11 at 11.41.35
Screenshot 2020-08-11 at 11.41.351506×864 105 KB

Thanks.

20

Hi, sorry for the long delay. I’ve been moving home, moving job, and even slept in my car for two nights in between signing tenancy agreements.

Well, anyway, it looks like the debugging was briefer than it could have been. That’s your culprit. The Filter Finder Items was the one action more than the others that I quietly suspected was going to be a problem.

In the first test run, your workflow was triggered twice: once by the placeholder file, and once by the synced image file.

In this test run, your workflow only triggered once. My guess, given that the trigger file created was empty, is that it was triggered by the placeholder file, but not the final synced file.

This is seeming to me like an problem with timing. My hunch is that the workflow stands a risk of being triggered successively in a very short amount of time. The first time it gets triggered is going to create a bit of a bottleneck in terms of processing power, speed, resources, etc. which Automator’s background helper (Automator Runner, which is responsible for executing application extensions, such as an Automator workflow that runs outside of the Automator app environment) is consuming as it instantiates an AMWorkflow application process within which your workflow gets run. By this time, the file has synced, and another signal is fired that should trigger your workflow a second time. Clearly this can occur in sequence and in time, as demonstrated by the first test run. But, depending on what else is happening in MacOS at any given time, execution speeds will vary quite substantially.

The other limiting factor on speed is the first action in your workflow. I already know that this action is slow. It won’t necessarily appear so if you run it within the Automator environment, but that’s because it already has everything loaded into memory and ready to go. In practice, it’s a bit of a nightmare. I’m fact, all of the Finder-specific actions are woefully inefficient. They’re created in Objective-C code, I believe but, as the name suggests, it doesn’t actually levy the speed and power of low-level file system access that Objective-C permits, instead operates through the Finder (which was written in QBASIC, I’m told), possibly by way of Apple events, which is the only way I am aware of to do this sensibly, and beneath that I’ll wager is a poxy uncompiled AppleScript that won’t be written very well (ironically, the people who created AppleScript are terrible at writing scripts with it).

To recap…

I’m surmising that we have a folder action that triggers with the appearance of the placeholder file. This is where everything takes forever to get up and running, and to actually run. But a microsecond after the placeholder file appears, it is replaced with your synced file. This sends off a second signal to trigger the workflow that’s still not actioned the first trigger yet. Potentially—and this happens a lot in many different contexts, though I haven’t specifically confirmed is definitely happening here, but it is a possibility—the second signal gets fired during the instantiation of the AMWorkflow runner environment, or some similarly-sensitive or critical moment in which the system isn’t able to create a second, concurrent runner environment, and ends up discarding the second signal as a failure to launch. It may even not process the second signal at all at specific times, and end up missing it while distracted by how terrible Catalina is.

To test, or not to test…?

To get some sort of indication that any part of this explanation is in the ball park, you may choose to run a few more tests. If you ran the same identical test a few times, my prediction would be for some of these runs to yield the empty trigger file you got on this occasion, and some of them to have the filepath to your synced image. If so, for me, that would conclude the testing, and we could reasonably assume the above is accurate. If not, it would be interesting to find out what happens when you sync a larger file, say 100MB in size. Of course, you’d need to adjust the filter, which I recommend filters by file extension rather than kind, as it’s never convincingly clear whether image kinds include the other subcategories of kinds that are named specifically for an image format, e,g, PNG image, JPEG image, Compuserve GIF, etc. which Finder lists as separate kinds. Add to this that it’s essentially running a spotlight query, and if the file that appeared isn’t indexed by the time the action runs, it conceivably could get missed. You’ll also have had experiences searching in a Finder window for a file name fragment that you know has at least one positive match (probably in that very window), yet Finder can’t seem to get it.

The point of using a larger file to sync is to put as much time between the two trigger events as possible. You may even find that it triggers more than twice. I would not be surprised if every few megabytes of data that gets written to the placeholder file, it updates the Finder, which interprets the changed file as, essentially, the appearance of a new file. So you could even add in a second filter to include the file extension of the placeholder file (just for testing purposes).

BUT, honestly…

By now, this is probably seeming more hassle than it’s worth. If you haven’t really done much debugging yourself in the past, it probably does seem like a lot of effort. Coupled with the amount I write in a single post, this might lead you to rethink shelling out money for Hazel that you know worked out of the box.

If you do this sort of thing enough times, it’s actually a simple, systematic process, that unfortunately, being guided remotely, and with a 12 day gap, makes what would really be a 20-minute job span out over days.

So if you want to run those additional tests I described just now, either for the curiosity, practice, or fun of it, or to reveal that nothing I said might happen actually happens, then it’ll be a good academic exercise and interesting to hear what the outcome is. But, I think you just want this bitch to work…

So let’s do that.

STEP 1.

Close Automator. Delete your workflow. It never exksted.

STEP 2.

Open up Script Editor. Create a new, blank document. Copy and paste this AppleScript:

…to be continued…

1 Like