The regex parsing of web pages is as brittle as I remember it. What I really need is a DOM I can fire CSS selectors against.
What would a good way to do this? A pure JavaScript implementation like jsdom or Cheerio seems like it could be a good fit, but they’re designed to work with node/npm, neither of which Scriptable supports.
Any suggestions?
I don’t know the modules you refer to but you might be able to “import” them in JavaScript by running them through Browserify and using eval().
I’m curious to know if this feature that I’m experimenting with would solve your problem. See this tweet.
So I think that’ll be useful but I have a couple of concerns:
Firstly, it’ll be a lot slower than just downloading the HTML from the page itself. Rendering in a web view will load all the graphics, css, JavaScript etc.
Secondly, will something like that be able to be invoked from Siri? Will that blow memory limits, etc?
Here’s an example use case. I ask Siri “what’s for lunch” and she tells me what they’re serving at my kids school next lunchtime.
Current regex based version:
Often with dynamically built pages it will be things such as the in page Javascript that is modifying the HTML with calls to the device or out to some other resources, and it would be the browser that drives that. You can’t download HTML that hasn’t been generated. To my mind, the only reliable way to build a page is fundamentally, via a browser.
You’re right. It might be slower but as @sylumer pointed out, it might be the only feasible way in some cases. I think it really comes down to what you want to achieve. Loading the HTML and rendering the entire page is two different approaches that should be used in different scenarios.
As for the memory limits, I understand your concern. However, the web view runs in a different process and therefore isn’t under the same restrictions as the process that’s running the Siri shortcut. This is also why it may be beneficial to load large images into a web view when displaying them in Siri. That’s just the way iOS works 
That would be very nice…
… On my Mac I localhost serve web pages. Scraping them from iOS and working with the DOM in Scriptable would be superb. Particularly as you’ve just enhanced UITable.
That should be possible with the updated WebView bridge that’s currently in the 1.1.0 beta.
If the webview memory doesn’t count then that’s pretty awesome - I was concerned that creating JS objects for the DOM would be expensive anyway.
Is it possible for scriptable to have an option not load anything but the main HTML (presumably this would setup a UIWebViewDelegate to say NO a lot) to speed things up?