YouTube Get Transcript Help

Scriptable
,
1

I really don’t do a lot of coding and I really don’t know JavaScript that well and with that what I’m trying to do is create a shortcut that grabs all relevant info from a YouTube video such as video title, date uploaded (medium format), date I created the note(medium format), Channel Name, URL with TimeStamp, duration of video, video description, a section for my own notes, and lastly the transcript down below if I could figure that out.

I have everything working via RegEx except for the Transcript.

Currently this is all done in shortcuts as a share sheet action. The issue I’m having is no matter what I try the only way with my knowledge and understanding that I’ve been able to access the transcript is to open the transcript on the YouTube page then go to share and run my shortcut which will run JavaScript on webpage just to get the xml / html (essentially one line of code).

This works as I can then copy and paste into notes the raw input and see the transcript is there. Transcripts don’t show up without a JavaScript call unless I am overlooking something.

I came up with a RegEx (tested and worked at regex101.com) that for the most part grabs the transcript text only HOWEVER when I try to run the it all at once within a shortcut on a long video (1 hr) it times out.

Ex:
Run JavaScript on webpage with shortcut input


var result = [];
// Get all links from the page
completion(new XMLSerializer().serializeToString(document));
// Call completion to finish
completion(result);

Match (?s)(?<=start-offset=\"\d{4,7}\">)(.*?)(?=<\/div>\s.*<dom-repeat class=\"style-scope ytd-transcript-body-renderer\">)

QuickLook

This times out. For testing I’m just running a shortcut with just this and nothing else.

I found this 3 year old forum post which seemed like it would do the same as what I was attempting to do which is cycle through all parts of a transcript and return the results but I’m having issues doing it either within shortcuts as a run JavaScript on webpage or pasting into scriptable and using the run script command from shortcuts usually with an error about the document. Also I’m unsure I’m doing the variables correctly for the transcript so I wanted to see what this would look like correctly written within shortcuts as a run JavaScript on webpage action or as a script within scriptable so I’ll at least know my focus should be on parsing the correct class.

Example Script I’m trying to using is the full script from the link above. This is my first post and it told me I exceeded the number of links I can include.

I’m not sure posting anything I’ve tried would be useful as it’s wrong and I figured posting what my intent is maybe someone might find a more elegant solution.

1 Like
2

Code Link from the referenced DataScraper page

Here is what I’ve got. It’s not working. I believe I’m not declaring the variable correctly or the function.


let url = "https://www.youtube.com/watch?app=desktop&v=pdwlwf6MqcA&t=2269s";
let wv = new WebView()
await wv.loadURL(url)
await wv.waitForLoad()



var elements = ({body: `.cue style-scope ytd-transcript-body-renderer`});


var find = `


/*DataScrapper by prettydude v1.0.*/
/*
  elements - list of elements to find (css or html form)
  all - return all found entries or first
*/
function find(elements, all) {
  let output = {};
  //split lines
  elements = elements.split("\n");
 
  elements.forEach(function(element) {
    //init variables, split key and selector
    let key = element.split(":")[0],
      selector = element.slice(key.length + 1).trim(),
      selected,
      out = [];
    //make css selector from element
    if (selector.startsWith("<")) {
      let div = document.createElement('div');
      div.innerHTML = selector.trim();
      let element = div.firstChild;
      //if class not provided - don't include it
      let classes = element.className ? "."+element.className.replace(" ", ".") : "";
      //if element has unique id - use only it
      selector = element.id ? "#" + element.id : element.tagName.toLowerCase() + classes;
    }
    //select elements with css selector
    selected = document.querySelectorAll(selector);
    //ES6 way to do loops
    for (let elem of selected) {
      //get element text
      out.push(elem.textContent);
      //return only first found value, not all
      if (!all) break;
    }
    output[key.trim()] = out;
  });
  //return to body
  document.body.innerHTML = JSON.stringify(output);
}


find ();


`


let response = await wv.evaluateJavaScript(find, false)
console.log(response)
3

Two things to fix:

  1. You can delete the await wv.waitForLoad() line. The await loadURL() line already waits for the WebView to finish loading the URL; it doesn’t need to wait again.
  2. Edit the last line of the body of the find() function to return the output. So instead of document.body.innerHTML = JSON.stringify(output), replace that with return output. That will pass the output of find() to your script instead of writing it to the page in the WebView.
4

Thank you for the suggestion!

I tried your solution and now I get an error on line 31 for an invalid character \ufffc.

Again I’m not even sure any of this is right but that’s the error I’m now receiving.

5

I was hoping to figure out how to do this within JavaScript as I was hoping it might be faster and to have something like this to work off of for any future projects on the JavaScript side.

Until someone figures that out and doesn’t care to share that in this thread I’m going to share two shortcuts that have been a long time in the making that runs 100% natively without any extra apps.

I’ve commented most things except the text formatting at the end as that’s a pain but the things that may break in the future such as how I’m matching should be easier to fix going by the comments and what I stripped from the cURL request when a video fetches the transcripts.

These will only work on videos that have closed captioning as i have noticed some don’t have any therefore until the uploaded enables automatic captions (an assumption) then these will not work on those. I also haven’t tested this on videos that upload their own transcripts.

The majority of the videos I come across usually use automatic and that’s what this currently gets.

It’s been tested on 3 videos. On a video that’s 12 minutes long it took about 20 seconds on a 3rd generation iPad Pro to process and format the text.

As with most YT Workflows and shortcuts this will eventually break when they change their coding and I cannot promise I’ll be able to update these so hoping what info I’ve included helps in the future.

The biggest issue was figuring out how to get the transcript to be accessible to shortcuts and I didn’t really see a well documented way to do that for those that know you can skip all the rest…

The way I went about this was to go to a desktop browser and pull up a video with closed captioning but don’t “Open Transcript” or whatever it’s called yet and using your browsers developer tools (steps are different for each one) inspect the youtube page and go to the network tab. Look for a trash can or a way to clear what’s already shown and then click on the “Open Transcript”. It should show just a handful of items making network connections and the one I was looking for was called get_transcript.

Click on get_transcript and then look for an option for Headers and within that view I could see the Key I needed to get and the Request Data at the bottom (I had to click on an icon next to Request Data for it to show the contents). Alternatively I think I could have just right clicked on get_transcript and copied the cURL link… I’m not at my computer so not 100% on this.

Once I had that I then had to figure out what to do with that or if I could do anything with it. Once I found a handful of articles that were similar to what I was working with I then figured out how to integrate the HTTP / cURL request data into a Get Contents of URL action as a post as we’re sending out a request after all and it was all down hill from there.

I then looked for things within the normal HTML that was being sent as a part of the request which was the request key / URL and a serialized share entity because if they weren’t in there then I would’ve hit a dead end. This is what some of the initial Match Texts are for.

So being able to reliably get a unique request Key / URL and a unique Serialized Share Entity from each video I could then form a request along with the Get Contents of URL using the Request Data as a text variable for a file.

Once I got the transcript I started removing data from the Request Data, things I didn’t want to match each time and would test to see if I broke something then rinse and repeat until I got what’s in the shortcut below.

YT Transcript from Share Sheet v2

https://www.icloud.com/shortcuts/a16677edb5fb40a2933e788abad27e98

YT Transcript from Within Shortcuts v2

https://www.icloud.com/shortcuts/a8da0e0f5b1e41f7b2788d8e2cb3c0c2

Edit:

Both Shortcuts end with a quick look so if you want to save the output You’ll need to add an action for that.

Edit 2: Only tested in Safari not the YT app or any other browser.

1 Like