Using Scriptable in Shortcuts to download videos to Instagram

marco · February 23, 2020, 12:40pm

Hey everyone,

I’m sharing a shortcut with you that I spent an inexcusable amount of time on for no apparent reason.

https://www.icloud.com/shortcuts/f239906999ba40cb832f11bb2ac3514f

It saves a video to your Recents photo album from the Instagram page that you are currently browsing. You activate the shortcut from the Share sheet.

-0-0-0- Step by step overview of the Shortcut -0-0-0-

The shortcut is straightforward, except for some gotchas:

First, it accepts a URL as input from the Share sheet;

Then, it checks if the URL contains the string ://www.instagram.com/p/ - the substring that all Instagram post pages share.

With that, Scriptable does its magic. I use the Scriptable inline JavaScript module to get an id and a key from downloadgram.com, where I send a request for the download link of the video displayed at the URL from the Share sheet, which I inject as a Magic Variable. Downloadgram’s response is an HTML snippet that includes the URL, which Scriptable parses and returns as the outcome of the script.


let url = "https://downloadgram.com"
let wv = new WebView()
await wv.loadURL(url)

let id = "document.getElementsByName('build_id')[0].value"
let key = "document.getElementsByName('build_key')[0].value"
let myId = await wv.evaluateJavaScript(id)
let myKey = await wv.evaluateJavaScript(key)
let myURL = 'Shortcut Input'
url = "https://downloadgram.com/process.php"

let req = new Request(url);
req.method = "post";
req.addParameterToMultipart("url", myURL)
req.addParameterToMultipart("build_id", myId)
req.addParameterToMultipart("build_key", myKey)
let res = await wv.loadRequest(req)
let video = "document.getElementsByTagName('a')[0].href"
let myVideo = await wv.evaluateJavaScript(video)

return myVideo

Shortcuts accepts the outcome of the script, the URL, and downloads the contents - the video, if all goes well.

In the second to last step, we save the video to the Recents library of the Photos app.

Finally, the Shortcut throws a notification to confirm that it saved the video to the Recent Album.

-0-0-0- End of overview -0-0-0-

Alright, you don’t need to read further if all you want is to start downloading Instagram videos. Get the shortcut here and go wild. Of course, if you spot issues or come up with improvements, I’ll be grateful if you share them with us here on the forum.

I want to get something out of the way: I’m not a developer, haven’t been for ages, and JavaScript is new territory for me.

I’m aware at a high level of how web sites process requests, which is how I dug my way into figuring out how to use the download service without using a browser. I’d seen friends use Postman and installed various apps on my iPad that helped me stumble through an approximate understanding of what I need to do to obtain a download URL.

This forum helped me grasp with the lack of a Document Object Model at the entry level of a script. The solution is elegant and actually much more conceptually satisfying than the traditional JavaScript way - you instantiate a WebView object that loads the page and from that moment on, the WebView is aware of the DOM of that page. Neat.

I do find it hard to debug a Scriptable module inside of a Shortcut; as far as I can tell, you cannot use alert sheets (something to do with Siri not being able to work with them) and QuickLook has so many layers of heuristics in how it presents information that I can’t use it as a plain log of the data that the script is handling.

Downloadgram is one of many services that enable you to download videos from Instagram. I’m not even sure why I stuck with this one instead of Downloadvideosfrom, Insta-downloader or one of the myriads of other such services. Their magic is all in a cryptic PHP file that parses its way through the Instagram web arcana to present you with the piece of media that you’re after. I tried to grok what it is they’re doing, but they cover their traces well. I failed.

Then, passing messages between Shortcuts and Scriptable. Oh. My. God. After this whole exercise where I managed to automate downloading Instagram videos from the Share sheet, I can confidently say: I am none the wiser.

I wrote how I used a Scriptable in-line JavaScript module. That’s because I was finally unable to get the two to work together like adults. The URL that I wanted to pass from Shortcuts to Scriptable as an argument, arrived as a snippet of the URL.

https://instagram.com/p/Xd5hg62-6asdf in ShortCuts becomes 6asdf in Scriptable.

I spent hours figuring out that that is what was happening (knowing of no way to debug properly). And when I hard coded the URL in Scriptable, I was unable to align the response type coming out of Scriptable with whatever it is that ShortCuts expect. Paraphrasing, here is the ShortCuts error message:

Expecting type String, received type [String]. Yeah.

Oh, and the clipboard? Fagedaboudit: it gets cleared before Scriptable can take a look at it. Empty. Nothing.

So, input into Scriptable unsuccessful, output into Shortcuts unsuccessful. I was ready to blow it all up, except that I fell foul to the Sunk Cost fallacy. It was one in the morning and there is no way I’m going to admit defeat.

Dr. Drang to the rescue. Serendipitously, I happened upon this gem of a blog post from last month

where Dr. Drang walks you through the use of the Run Inline Scipt action. At one in the morning, I felt this could be my ticket out of Purgatory, even though I wasn’t yet sure how.

I’ll cut it short: at four thirty I hit the sack, having successfully downloaded videos from instagram automatically through the Share sheet. Have at it.

svenhayashi · February 23, 2020, 2:09pm

i’m a little paranoid on using 3rd-party website like downloadgram.com so i devised this little javascript to use the information already loaded in the instagram page, which also does not require Scriptable at all. all native within Shortcuts without passing anything to other website.

use a Run JavaScript on Webpage action with the following script with the instagram page as Safari webpage. the output is a list of links separated by line breaks for downloading use (which i’m certain you can figure out how to do in Shortcuts).

this script can handle instagram posts with:

single photo
single video
multiple items with a mix of photos and videos

have fun.

let instaObj, mediaLink = ''

// read code and init instagram variables
instaObj = JSON.parse(
  /[^{]+(.+)/.exec(
    /window.__additionalDataLoaded\((.+)\);/i.exec(
      document.documentElement.outerHTML
    )[1]
  )[1]
).graphql.shortcode_media

// extract link to photo/video
if (instaObj.edge_sidecar_to_children) {
  // multiple items
  mediaLink = instaObj.edge_sidecar_to_children.edges
    .map(p => ( p.node.is_video ? p.node.video_url : p.node.display_url ))
    .join('\n')
} else if (instaObj.is_video) {
  // single video
  mediaLink = instaObj.video_url
} else {
  // single photo
  mediaLink = instaObj.display_url
}

// pass result as output
completion(mediaLink)

marco · February 23, 2020, 2:15pm

Same here! Glancing at the code you shared, I am convinced I would have never come up with your solution. I’ll try and implement it myself - first, so I can learn from your example; and second, so I may lose my dependency on third-party sites.

Thanks!

marco · February 24, 2020, 7:33am

Ah, shame, the code doesn’t work for me. It throws an error at the very start of the code, like so:

I configured the shortcut like you explained, with a Safari Web Page as input.

And I trigger the shortcut on a web page of an Instagram post.

To be honest, I understand very little of what you do in this code. I see that you take the contents of the page and parse them (twice?) so as to get a (list of) link(s) to the included media.

Then it looks as if you check if you got multiple results, one video result or one photo result. In case you got multiple, you separate them with a new line.

Finally, you return the result.

This is so far beyond what I’m able to reproduce, that I can just admire your skills from the sideline Too bad I can’t get it to work. I wish I were more knowledgeable!

svenhayashi · February 24, 2020, 8:57am

perhaps i should’ve included a full example. you can try out the shortcut to see it in action.

Shortcut link: https://www.icloud.com/shortcuts/b72df94d63c249e8865a41f51b357587

testing instagram links:

single photo: https://www.instagram.com/p/B8pLTeynmSM/
single video: https://www.instagram.com/p/B8rU16QHjgc/
multiple items: https://www.instagram.com/p/B8RTwJtnoqP/

i can provide a better explanation of the code later when i’m off work. for now, please see if the shortcut works for you.

marco · February 24, 2020, 9:25am

Thanks @svenhayashi, I really appreciate what you’re doing. You show how to implement the actual identification of Instagram media links without relying on a third party, from JavaScript, from within ShortCuts. Wow.

I’m afraid that I reached the limits of my competence. The shortcut you kindly shared throws the same error that I experienced. I have the impression that the script gets a different input from what it expects - from what I see, it gives up on the second round of regex parsing:

/[^{]+(.+)/.exec

passes, while

/window.__additionalDataLoaded\((.+)\);/i.exec(
      document.documentElement.outerHTML
    )[1]
  )[1]

fails.

In my layman understanding, the second round of regex parsing doesn’t receive the input on which to run

document.documentElement.outerHTML

As you can tell, I’m way out of my depth here

svenhayashi · February 24, 2020, 9:28am

the code should run from inside to outside, so the /window.__additionalDataLoaded\((.+)\);/i.exec is actually the first part that gets run. if the page has loaded completely, it should be able to find the text required.

~~curious, are you running this in Safari?~~ please ignore me… the shortcut doesn’t even show up in Chrome, which is the expected behavior.

Edit 2: i believe i’ve figured out your problem. there’s 2 prerequisites to run this shortcut.

you need to be logged in to Instagram.
Safari cannot be in Privacy mode.

marco · February 24, 2020, 12:19pm

Bingo! Yes, that is definitely the issue. I’ll report back

svenhayashi · February 24, 2020, 1:19pm

i took a more detailed look at the code and changed a bit, cos the previous one i kinda rushed thru like a year ago. here’s the updated shortcut.

Shortcut link: https://www.icloud.com/shortcuts/c264cd01ef054f3fb0835b713e38306f

main update:

now it works with regular and privacy mode in Safari
no longer requires to be logged into Instagram

updated JavaScript code:

let jsonText, instaObj, mediaLink = ''

// attempt to extract JSON text
jsonText = /window\.__additionalDataLoaded\([^\{]+,([^\)]+)\);/i.exec(document.documentElement.outerHTML)
if (jsonText) {
  // extract part of the IG object with info desired
  instaObj = JSON.parse(jsonText[1]).graphql.shortcode_media
} else {
  // retry if not logged in or under privacy mode
  jsonText = /window\._sharedData = (\{.+\})(?=;?<\/script>)/i.exec(document.documentElement.outerHTML)
  // extract part of the IG object with info desired
  instaObj = JSON.parse(jsonText[1]).entry_data.PostPage[0].graphql.shortcode_media
}

// extract link to photo/video
if (instaObj.edge_sidecar_to_children) {
  // multiple items
  mediaLink = instaObj.edge_sidecar_to_children.edges
    .map(p => ( p.node.is_video ? p.node.video_url : p.node.display_url ))
    .join('\n')
} else if (instaObj.is_video) {
  // single video
  mediaLink = instaObj.video_url
} else {
  // single photo
  mediaLink = instaObj.display_url
}

// return result
completion(mediaLink)

document.documentElement.outerHTML: since we’re looking at the <script> tags for JSON text, this is how we get the code outside of the <body>
window.__additionalDataLoaded: this indicate the beginning of JSON text when the browser is logged into IG.
window._sharedData: this should be looked for last cos it will only contain the information needed if IG is not logged in.
the previous RegExp is cleaned up to do only a single pass RegExp search for the text needed.
use JSON.parse to get the JS object to parse for media link(s).

marco · February 24, 2020, 2:11pm

Wow. You are a hero.

I will be studying on this for some time, because I want to learn from you.

Respect, thank you!

Edited: I just tried it out and this is awesome… Using QuickLook, you see all images and videos contained in the Instagram post. I will try to grok your code.

marco · February 24, 2020, 4:34pm

Alright, for posterity’s sake, here is my understanding of @svenhayashi’s code. I am super grateful for his insight, any misunderstanding is on my side and all praise goes to him.

When you load an Instagram page into Safari, there is a javascript script that starts with the string window._sharedData. It’s inside of this script that Instagram places the links to the media that the user has posted on Instagram. Hence the command

jsonText = /window\._sharedData = (\{.+\})(?=;?<\/script>)/i.exec(document.documentElement.outerHTML)

What the code does here is to find all text from window._sharedData up to and including <\script>, which is the closing script tag, and it puts it into the jsonText variable.

Then, the code pours that code into a JSON object where it looks for the following sequence of nodes:
entry_data > PostPage > graphql > shortcode_media
(Where I presume that PostPage[0] means that there may be more than one PostPage node, and it always takes the first one).

Instagram will place multiple media entries inside of an edge_sidecar_to_children, identifying the media type with is_video, either true or false. The url of a video is preceded by video_url, whereas an image url has a display_url node.

I think

Thank you @svenhayashi for your awesome contribution. I already learned so much thanks to you, even if I would not remotely be able to produce the code you shared.

supermamon · February 25, 2020, 2:49pm

I’m late to reply but you can actually just append ?__a=1 to the post url to extract the post’s JSON data.

marco · February 25, 2020, 3:29pm

I was about to answer “I don’t know”, but then I thought better of it and tried it out.

All I can say is, “works for me”

svenhayashi · February 26, 2020, 3:00am

oh thank you! i certainly didn’t know that.

with the easy, direct access to the JSON data, the Run JavaScript on Webpage action can be replaced with a Run Inline Script Scriptable action with the updated script using the URL as input parameter.

let igURL, req, jsonObj, instaObj, mediaLink = ''

// fetch JSON data
igURL = args.shortcutParameter.trim()
req = new Request(igURL.replace(/(\?.+)?$/i, '?__a=1'))
jsonObj = (await req.loadJSON())

// get media links
if (jsonObj) {
  instaObj = jsonObj.graphql.shortcode_media
  // multiple items
  if (instaObj.edge_sidecar_to_children)
    mediaLink = instaObj.edge_sidecar_to_children.edges
      .map(p => ((a = p.node).is_video ? a.video_url : a.display_url))
      .join('\n')
  // single video
  else if (instaObj.is_video)
    mediaLink = instaObj.video_url
  // single photo
  else
    mediaLink = instaObj.display_url
}

// return result
return mediaLink

Edit: a small caveat is using the direct request to JSON data, it will only work on public IG posts. the first solution with Run JavaScript on Webpage will also work on private IG posts as long as the logged in user can see the posts.