Execute JavaScript from webpage

Scriptable
1

Hi,

I am relatively new to JavaScript, I usually code in Python and C++. I am trying to get a script of mine to run:

I want to load a webpage that has embedded JavaScript changing the page. I need to execute the script embedded on that page. And get the changed page. I can get the plain HTML with JavaScript source code, but not execute it yet.
I am quite new to this so I don’t know what to do.

Hans

2

When you view a web page it evaluates the scripts and renders the result. You’re currently retrieving the page info and then would need to do the same.

You might be able to parse the content and then evaluate the script and run it on the page, but if there’s any server interaction that would become significantly more complex and even as it stands it would probably be rather difficult and tied solely to that page so if it changed it would break the algorithm used.

I don’t know of a way Scriptable could effectively pose as a browser, internally render the page and return the resulting content in some format - definitely one @simonbs would be best placed to answer.

Would another option be if you were to view the page in a browser yourself and then pass selected content into Scriptable? I know Shortcuts can retrieve evaluated web page content from Safari via the share sheet, so at worst you could use that and then pass on the content to Scriptable for further processing.

Hope something in there is useful :man_shrugging:t2:

3

I’m not really sure what the best approach is here. I have it in my backlog to add support for evaluating l JavaScript on a web view but I’m not sure when I’ll get around to it.

I’m not sure what would work here but I think there’s two options worth exploring:

  1. As @sylumer mentioned, Shortcuts supports evaluating JavaScript on a webpage. Maybe this can be used and then pass the result to Scriptable, depending on your use case.
  2. Scriptable has a WebView API which is a browser that can load HTML. The HTML can include everything you’d normally expect eg. CSS and JavaScript. Maybe you could use Scriptable to load the HTML of your website and evaluate it a webview, optionally injecting some JavaScript to further process the website. Your entire script would probably need to run in the web view then. Right now there’s no way to extract a result from the web view in Scriptable.
4

Currently, there are no options to evaluate JS in Scriptable.
In my case, it has been working well to utilize Glitch (glitch.me; for an API server) and Puppeteer (Headless Chrome) to process web pages then receive a result as a JSON.

Source https://glitch.com/edit/#!/sfc-bus

5

Have the same problem. Wanted to extract a video URL from Twitter to create a PIP shortcut, but the video element is added to the DOM with JavaScript.

I can’t run a server that executes the page and sends the resulting DOM.

Wouldn’t it be possible to create a WebView.loadHTMLhidden(seconds) method that presents the view like WebView.loadHTML() but hidden, such that the JS can load? It returns a promise which fulfills with the current state of the DOM after the number of seconds specified in the argument. Since there is no way to check if the main JS thread is done or not, I think a simple delay would be the only option here.

I don’t know if that’s possible, would be certainly great for the JS heavy Web nowadays.

6

If you don’t use the static function of WebView then it doesn’t present it. Something like this:

let wv = new WebView();
await wv.loadHTML(yourHtmlInThisVariable);
// the html is now loaded
// you can add another delay here to wait for the JS in the page to execute
// you can get the html with
let htmlFromPage = await wv.getHTML();

I assume that you are then parsing the html string for the video URL. You could also use wv.evaluateJavaScript(jsString) to search for the corresponding element in the page and return the video URL by using the browser DOM.