Since I’m not familiar with scraping websites (parsing elements) this might be a (nice) challenge. The examples that @simonbs posted here will definitely help though!
Unfortunately I won’t have time until this weekend to experiment. If anyone is up to this I would be happy to learn from her/him!
Desired output: Date, City, Country (in aligned columns)
Bonus: Showing a flag of the country instead of the two letter abbreviation
If I get that working I might be interested in showing the upcoming games of my favourite soccer team (date, day, competition, home team, away team - adding club logos)
I am not believing this! Just today I was “wishing” that I could automate my data collection from various sites and then post it into my community forum. Doing it by hand is VERY time consuming. Y’all have REALLY gotten me to thinking hard on this now! THANKS for the motivation!
I jus took a look at U2s website. Turns out they have all their concerts embedded as JSON so it was super simple to get an array of all the concerts. This should get you started.
let url = "https://www.u2.com/tour"
let req = new Request(url)
let html = await req.load()
let json = html
.split("<script type='application/ld+json'>")[1]
.split("</script>")[0]
let concerts = JSON.parse(json)
QuickLook.present(concerts)
Sorry for the dumb question, how did you find out that the website used JSON to store the information, I am learning JS and I think it’s important, at least to me, to understand that, thanks
What I do is I open up Chrome or Firefox and the network tab in the developer tools, and then load the page. In the network section you will see everything that gets loaded - CSS files, JavaScript files, PHP, JSON - anything called as part of building the web page. This would allow you to see if a JSON is loaded directly (that might not work with all pages, for example in many of my web apps you would call a PHP file which gives you a JSON output, so just searching for .json might not always work ).
I am using Safari, I only got CSS, GIF, JPG, JS, PNG and WOFF, there was one that took the longest to load and the type is Document, could that be the JSON document? Thanks
Sometimes I do as Rosemary suggested other times I use Charles Proxy, either on macOS or iOS. In this case I used Charles for iOS.
I’m a long time user of Charles on macOS and while it’s only four months since the iOS version was released, it has already become one of my favorite iOS apps.
let url = "https://www.u2.com/tour"
let req = new Request(url)
let html = await req.load()
let json = html
.split("<script type='application/ld+json'>")[1]
.split("</script>")[0]
let concerts = JSON.parse(json)
let today = new Date();
let table = new UITable();
for (concert of concerts) {
let date = new Date(concert.startDate);
if (date < today) continue;
let row = new UITableRow();
row.cellSpacing = 10;
let country = concert.location.address.addressCountry;
if (country == "UK") country = "GB";
let columnOptions = [
{value: date, format: {weekday: "short"}, align: "left", weight: 1},
{value: date, format: {day: "numeric"}, align: "right", weight: 1},
{value: date, format: {month: "short"}, align: "left", weight: 1},
{value: country.toUpperCase().replace(/./g, char => String.fromCodePoint(char.charCodeAt(0)+127397)), align: "center", weight: 1},
{value: concert.location.name, align: "left", weight: 5},
{value: concert.location.address.addressLocality, align: "left", weight: 3},
];
for (options of columnOptions) {
let cell = createCell(options);
row.addCell(cell);
}
table.addRow(row);
};
QuickLook.present(table);
function createCell(options) {
let value = options["value"];
let format = options["format"];
let text = format ? value.toLocaleDateString("en-US", format) : value;
let cell = UITableCell.text(text);
cell.widthWeight = options["weight"];
cell[options["align"] + "Aligned"]();
return cell;
}
@simonbs: I noticed something weird I think. When invoking this Siri shortcut in portrait mode (on an iPad) I get the full list from the source URL, but when invoking it in landscape mode the list ends premature (at the second concert in Amsterdam). Is this a bug in iOS 12 or in Scriptable?
The shortened list is actually working as intended. What you are experiencing is a limitation in iOS. The UI presented in a Siri Shirtcut can not exceed a maximum height. That height may vary between portrait and landscape.
Oh. If I start the shortcut in portrait mode and then switch to landscape mode I still have the full list. That’s good enough as a workaround, but I guess I’m just lucky then that this leg of the tour fits in (at least) one orientation…
Got that working as well! Won’t share it though as it is based on the home page of my favourite team and unfortunately there’s sometimes violence between “supporters” (hooligans) of different teams, so I’d rather not share my preference online…
Bit of a bummer: the logos and the league don’t fit (in separate columns) when invoked from Siri, so while I had those working as well I ended with just day, date, time, home team, and away team. Well learned something new, and the schedule was immediately useful
My favourite band’s site doesn’t have a JSON file (check them out, they’re great!) so I had to work directly on the HTML. Here’s a screenshot and the code I wrote based on what @rob shared here, so thank you for the guidance! Also, thank you @simonbs for creating such a cool app! I’m looking forward to playing around with it even more.
// Gets page url
let url = "http://www.berritxarrak.net/kontzertuak/"
let req = new Request(url)
let html = await req.load()
// Starts table
let table = new UITable()
// Gets html with concert info
let infoStart = html.indexOf('event type-event status-publish hentry active no-media')
let infoEnd = html.indexOf('</section>',infoStart+1)
let finishSearch = html.indexOf('<!-- .main-content end -->')
while(infoStart < finishSearch){
let concertInfoHTML = html.substring(infoStart,infoEnd)
// Gets concert day
let dayStart = concertInfoHTML.indexOf('<strong>')
let dayEnd = concertInfoHTML.indexOf('</strong>')
let day = concertInfoHTML.substring(dayStart+8,dayEnd)
// Gets concert month
let monthStart = concertInfoHTML.indexOf('<span class="month">')
let monthEnd = concertInfoHTML.indexOf('</span>')
let month = concertInfoHTML.substring(monthStart+20,monthEnd)
// Gets concert day of the week
let weekdayStart = concertInfoHTML.indexOf('<span class="day-name">')
let weekdayEnd = concertInfoHTML.indexOf('</span>',weekdayStart+1)
let weekday = concertInfoHTML.substring(weekdayStart+23,weekdayEnd)
// Gets concert city
let cityStart = concertInfoHTML.indexOf('/">')
let cityEnd = concertInfoHTML.indexOf('</a></h1>',cityStart+1)
let city = concertInfoHTML.substring(cityStart+3,cityEnd)
city = city.charAt(0) + city.slice(1).toLowerCase()
// Gets concert time
let timeStart = concertInfoHTML.indexOf('<span class="event-time-text"> <i class="icon-time"></i>')
let timeEnd = concertInfoHTML.indexOf('</span>',timeStart+1)
let time = concertInfoHTML.substring(timeStart+56,timeEnd)
// Adds emojis to strings
let date = '🗓 ' + month + ' ' + day
city = '🏙 ' + city
time = '🕘 ' + time
// Creates row inside the table
let row = new UITableRow()
let timeCell = UITableCell.text(time)
timeCell.leftAligned()
timeCell.widthWeight = 2
row.addCell(timeCell)
let dateCell = UITableCell.text(date)
dateCell.leftAligned()
dateCell.widthWeight = 2
row.addCell(dateCell)
// let weekdayCell = UITableCell.text(weekday)
// weekdayCell.leftAligned()
// row.addCell(weekdayCell)
let cityCell = UITableCell.text(city)
cityCell.leftAligned()
cityCell.widthWeight = 5
row.addCell(cityCell)
row.height = 40
row.cellSpacing = 1
table.addRow(row)
// Gets html with concert info
infoStart = html.indexOf('event type-event status-publish hentry active no-media', infoEnd+1)
infoEnd = html.indexOf('</section>',infoStart+1)
}
if (config.runsWithSiri) {
Speech.speak("Here's the coming Berri Txarrak concerts.")
}
QuickLook.present(table)