Save/Get File frustration

Shortcuts
1

Good morning all,

I have built and used a shortcut that accesses an Airtable base via the API, then returns a dictionary that I can process to filter for just the exact email addresses I specified, then sends the final list to Airmail via an x-callback URL.

I’d like to bring the data from Airtable to an iCloud JSON document so I can more easily use it when I’m not online. I fly a lot, and it would help keep me working while in the air. I also use a MBP, which could keep the iCloud version so I can use it there without tethering.

So - I wrote a shortcut to try and bring the Airtable information down and save it to the JSON file. (Thinking I could trigger this from a Pushcut or something to do it when I’m online.) I’ve run into multiple issues.

1 - Using Save File - I get an error when I assign a file name. Either - that file doesn’t exist, or that file already exists (even with overwrite checked). The only way I can get it to work is to ensure the shortcut creates the file, then it seems like it is willing to overwrite it. Is there a reliable way to get this step to create the file with a specific file name? Seems hit-and-miss. The only reliable way is to stick with the default file name of “Dictionary.json.”

2 - I have a subroutine that returns a dictionary containing all of my data (using Show Result I can verify the return contains it all.) Airtable includes the key value pairs inside a wrapper key “fields”.

If I simply put a Save File (accepting the returned dictionary) step next, it only saves the last record.

If I put a Save File (as Dictionary - Get value for key “fields”) step next it only saves the first record.

If I use a text step to convert the dictionary to text, and save the text file, I get the error “Cannot convert from text to dictionary” when I use a Get File step and try to put it back in a dictionary. (Note - in this case - all records are present, but the appended filename suffix is .txt.)

I jut don’t get it. I have a dictionary, and I can work with it fine, I just don’t seem to be able to save it as JSON. Does anyone have a reference where I can read more detailed documentation? Or can you point me in a direction to understand how the Save File step is interpreting the dictionary?

One other note - I used Jayson to look at the results each time. The first two produce valid JSON, but only 1 record. One thing I truly don’t understand it why the first scenario results in the last record. (I don’t extract it from the wrapper key “fields”, so how does it only grab the last key-value pair inside the “fields” dictionary?” ) Also - Jayson can’t read the text file - so I’m just visually inspecting that one.

Thanks for any help or ideas. I’m sorry I can’t share the shortcut (confidential data and all.)

(Probably goes without saying, but I’m not a developer of any kind.)

DK

1 Like
2

@Dgkeith237

  • When you comment on save file reliability, what are you seeing happen? Failures - if so what do they say? Nothing - but checking the file reveals no update to the timestamps or contents, yet data was returned?
  • I can grab some JSON data online and repeatedly process and save it using the actions in this shortcut.
  • Have you tried accessing a dictionary as text to try a more internal coercion wherever you are getting the ‘Cannot convert from text to dictionary’ error? e.g. like in the final step here .
    • It does sound strange that any sort of conversion from text to dictionary should be occurring if you are converting from dictionary to text, or just saving text. All I can think is that maybe you have converted from dictionary to text and then saving the text as a dictionary in the save file rather than just text :man_shrugging:t2:?
  • In regards to what Jayson is telling you, trying to infer what’s going on for definite, without seeing a reproduction involving data and actual steps with all the details of the parameters in those steps mens just taking stabs in the dark.

If you can provide a worked example to work with I’m sure we can all help narrow down what’s going on. It’ll allow us to review exact details and have a baseline against which to debug rather than first having to make up sample data and then, through trial and “non-error”, narrow it down to the point at which we get the same sort of result … but it could still be for a different reason. Full, first hand examples are always the best starting point for sorting this sort of thing out.

3

Thank you! I took a JavaScript course to try and get more out of Scriptable (not involved in this shortcut) so I did learn about type coercion and I must admit, I just never made the jump to think about it in those terms. I’ll use that perspective and take another look. Your example shortcut runs fine and created the file on the first attempt.

In regards to the first bullet, the errors returned are:

  • About half of the time I try to create a new file with the save file step the process fails. I just did it three times with no error message, but no file created. However, I have also received an occasional error with something like “Cannot save file because does not exist.” This only occurs the first time through the shortcut (or until the file is created.)

I also did not explain the end of my current shortcut as I develop it. For the purposes of verifying the file and json formatting, I have added a Get File step to the end of my current script. I use this to recall the file and display what was saved. That is the step giving me the “cannot convert text to dictionary” error only when I saved the file as a .txt file. That was the only way I could see the entire dictionary contents.

Anyway - one of the things I noticed about your sample shortcut is that you process the data and then save it. I was pretty much just trying to save it. It did dawn on me that I don’t need all of the Airtable data for this particular application. I only really need the keys, emails, and filter data. I can create a new dictionary with just this information then try saving just that file.

Thank you! I’ll check back, and if it doesn’t work I’ll try using dummy data. That way I can share for clarity.

DK

4

Followup questions here.

I have discovered a few followup points.

First - the lack of reliability in the Save File step is purely about whether or not I fed it valid JSON when I was asking it to save. I understand this now.

However, now I have built a routine that extracts the necessary data and constructs a text file containing the dictionary. However I have run into a big problem.

It turns out much of my problem appears to be “smart quotes”. I’m feeding my text directly into Jayson and it is giving me errors because of the smart quotes that the iPad inserts automatically. Is there any way to force “straight quotes” on the iPad?

I built your shortcut myself from scratch, and it works fine. The dictionary I’m returning from Airtable is much more complex. I used an online validator on a small subset of my data and it pretty clearly indicates the smart quotes as the culprit making the JSON invalid.

Thanks for any guidance on how to force straight quotes.

DK

5

To affect what you type, use this setting.

Settings (App) > General > Keyboard > All Keyboards : Smart Punctuation | Disabled

Note you can still enter smart punctuation, but iPadOS won’t auto convert to smart punctuation with this setting disabled.

6

Thanks so much. Wish there was access to it in Shortcuts so I could write a toggle (perhaps with automation.):nerd_face:

Anyway - somewhat humbled by the experience, I can close this with the definitive determination that it was all due to invalid JSON.

For anyone who might need this guidance in the future, I wanted to record my solution.

I took the dictionary from Airtable, then cycled through it using Repeat for Each and extracted only the fields I needed.

Then I used a series of combine text and text steps to construct the new JSON. An important problem was that one field had the possibility of having more than one value (which returned from Airtable as separated by spaces). So for this portion I needed to have a nested Repeat for Each that constructed an [array].

I then used an online tool called JSONlint on portions of my data that were less critical. I found errors related to smart quotes (as recorded above) as well as errors in the key structure, which I could re-design.

Once I had a script producing valid JSON, all of the other frustrations disappeared. I then used Jayson to view the saved file, and it worked fine.

My frustration with the experience is the lack of specificity in the error messages I produced. So, for other non-developers, I thought it might help to know that invalid JSON can produce weird results, and the use of those two tools helped me quite a bit.

Thanks very much to @sylumer for the assistance.