As we know, iOS 13 allows us to provide Shortcuts to the system that take input. That opens up a whole new world of what we can build.
I’ve been playing with wrapping up some functionality for native apps into Shortcut intents, i.e.
Image Classification (what is in an image, i.e. ‘dog’)
Image Similarity (how similar are two images)
Text Sentiment (i.e. is this text positive or negative)
Text Extraction (i.e. get the URLs, dates, addresses, etc… from text)
And some other ideas too. Here are some sample videos and if anyone has any interest, happy to share or even collaborate. Pretty rough right now but you can get the idea.
I’m still working on the image similarity but basically you feed in two images and you get a numerical score, the lower the number the more similar the images are. It’s based on new functionality in the Vision and CoreML APIs that are in iOS 13.
Anyway, I think what I’m going to do is to expand on this a bit and then run a TestFlight with anyone interested, get some feedback and then decide if it’s worth pushing to the App Store in the fall.
As promised, a very rough version of this is on TestFlight. The app itself is not all that interesting, it’s more about the shortcuts that are included.
I’d like to join, but I don’t have the iOS 13 beta installed and no device where I can install it except my daily used iPhone where I don’t want to install it… If iPadOS would support the iPad Air 1, I would install it right away^^
I am still trying to figure out how to get a xml response from api into a JSON… so far most attempts to access the data cause it to disappear or they get rejected from various conversion shortcuts.
Working on this a bit more, thinking about adding Google Docs support, starting with Sheets - the ability to read/write sheets natively from Shortcuts. I still have a bunch to do but I have made a good bit of progress. I think the only way to access Google Docs from Shortcuts now is via third-party stuff like Zapier, is that right?
Yes’ish. Payloads to third party web services are currently the only way to easily interact with Docs, Sheets, etc.
I think it was also possible with an OAuth workaround (such as this) to use direct API calls to carry out interactions and operations. But this involves an additional server component hence the ‘ish’.
I’m going to submit an updated TestFlight that includes the reading and writing of Google Sheets cells.
Note that first you have to open the app and authenticate with Google. It uses their approved OAuth flow but since the app is not yet published, there’s a scary looking warning in there about that. Just a heads up - I will be applying to Google but they have to review the app which has not yet happened.
There’s also some additional examples in the app itself for image classification and OCR, though those still need a bunch of UI work (SwiftUI still has a lot of bugs).
Next up will then be adding more error handling which is pretty bare at the moment.
Would you mind sharing a shortcut using the image classifier? I tried with a simple one (w/ input image) but it didn’t work.
Also unless I misunderstood, the beta app gives me no option to authenticate with google? Was that in regards to the next version? I have version 1.0 (5) in TestFlight.
Sorry, a little confusion here that’s likely my fault. The Google stuff is not yet in the current TF build. I had intended that to go out yesterday but I hit a couple of snags. I will get back into it today and hopefully resolve and if not then tomorrow.
Let me know if that doesn’t work for you. Here’s also a video of what I did:
Also, one of the things I’m most interested in feedback on is how to best format the output of the Shortcuts so that they are useful. That will vary per Shortcut action but some support complex structured data and wondering how people might like to use that.
Ok, that new TF version is posting. It includes the Shortcuts to read and write Google Sheets after you authenticate in the app. There are also some more demos in the app itself but they’re still really rough in terms of UX. I need to learn more about SwiftUI. They work, they’re just janky.
Looking for feedback, either here or via TestFlight. My next Shortcut I’m planning to work on is also for Google Sheets, to append a value to a certain column. Useful for tracking new items on a spreadsheet for instance.
After that, I have a bunch of error handling and other reliability stuff to do and of course whatever feedback I get.
Latest update includes a new Google Sheets Shortcut to append to a spreadsheet column, making it easy to just insert a new value in whatever the next row would be.