Widget for synchronization of folders between iOS and Server (WebDAV, SMB)

[MDS]

Hi,
I would like to sync files and folders between iPad, iPhone and my NAS. I have a Synology Diskstation which supports WebDAV (like NextCloud and OwnCloud) and SMB. The objective is to have files available offline and when I push a file from iPad to the server having it available on iPhone, too. So, I thought a scriptable widget which is triggered time by time is the best way to sync the folders. However, I struggle with the filemanagement.

Attempt 1 - File Bookmarks
At first, I used the scriptable „File Bookmarks“. The bookmarks should manage the folder on iOS and the folder on the NAS. The file system of the NAS can be includes into the iOS File manager via

smb://192.168.178.30

(Only available form WiFi at home.) I included the bookmark into scriptable with the following command:

const fm = FileManager.local(); 
const serverPath = fm.bookmarkedPath("serverVerzeichnis");

Where „serverVerzeichnis“ is the File Bookmark within scriptable.

The disadvantage of this method is, that the files are only available at home. But this can be overcome with a VPN connection to the rooter.

However the bookmarks are not stable. Every day I have to delete and create new File Bookmark as iOS (or scriptable) can‘t find it. I have to admit, that my NAS is down at night (energy saving) and restarts every morning. Also WiFi is of at night, so the login has to start every day new. I was expecting that the File Bookmarks will work for ever but for me they don‘t. So here are my questions:

• What are the basic principles of the File Bookmarks? I don‘t yet fully understand, what they can do and what they can’t.
• Is there a possibility to push scriptable to reininitiate the File Bookmarks?

Attempt 2 - direct Link

I also tried to use the direct link:

/var/mobile/Library/LiveFiles/com.apple.filesystems.smbclientd/Wzt...

with the following code

Let serverPath = "/var/mobile/Library/LiveFiles/com.apple.filesystems.smbclientd/Wzt...“

The good news are the name after smblientd/ seems to be stable within iOS. But also the File app is loosing the connection after a certain time.

• How can I force iOS/iPadOS to reintegrate the SMB-Filesystem into the local file system?

Attempt 3 - WebDAV
My third idea was to directly access the NAS via WebDAV. Here, I struggled with the syntax of WebDAV. I am able to upload and download a file but didn‘t understand how to retrieve additional informations like the content of a folder or the date of a file.
The scriptable commands look like this

var url = "https://server.com:port/folder/file“;
var req = new Request(url);
req.headers = {"Authorization":Login};
var content;
content = await req.load();
fm.write(filePath, content)

The standard port for a WebDAV-Server on a synology diskstation is 5006. The Login is a created best with shortcuts using user:password and than encode with base 64.

The advantage of the access via WebDAV is, that the files can be accessed from anywhere.

By the way: It is also possible to access WebDAV via shortcuts. The following commands are required:

Get contents of URL
Method:GET (for download or PUT for upload)
Headers: 
Authorization: Basic LOGIN
Request Body: File
File: The_File

(Space between Basic and LOGIN. LOGIN again is encode with base 64 from user:password.)

I was trying to understand the WebDAV-Specifications at

https://greenbytes.de/tech/webdav/rfc4918.html#rfc.section.9.1

But I didn‘t figure out how to send PROPFIND to the WebDAV server.
Here again my questions regarding WebDAV:

• How can I receive additional informations about a file (I need date) or a folder (I need the content)?
• What happens to the date of a file when I store it locally?
• Is there a chance to give the local file the same date as the server file?

I am already able to upload and download files to and from the server via shortcuts. It would be create if I can download the files automatically with the widget.
As I am not able to receive additional informations about the files and folders I have a workaround. Currently I write all the files I like to sync into an additional file which I can acces from any device. In the file the files which should be available for any device are listed so I can download them. However, I haven‘t found a way to manage the date and time information about the files, yet.
I am very thankful for any hints to increase my concepts.

[sylumer]

Probably not what you want to hear, but there is a reason why sync is usually managed by a full app. Good syncing can be hard. I would strongly recommend using an app rather than rolling your own solution based on a Scriptable widget for this.

I have a QNAP NAS myself, but I’ve heard Cloud Station and DS Cloud work reasonably well for Synology; but your mileage may vary.

[MDS]

Thanks for the tip. Unfortunately, user of a Synology Diskstation have to choose between Synology Drive and Cloud station. I have selected Drive. Unlike the iOS-App DS Cloud for the Cloud station the Drive-App has no sync-option. It is also announced that for the new DSM 7, which is released in summer, there won‘t be a sync option for iOS. Moreover, the Synology-Apps unfortunately don‘t offer a interface to shortcuts and scriptable. So even if the files are available in a Synology App the files are not available for other apps. I am working on my sync-program only because there is now option with synology. How about QNAP? Can you use QNAP files together with scriptable and shortcuts?
A was very happy, when data jar was released. I initially thought, I can use it instead of coding a sync-widget. However, data jar does not support scriptable. Whenever I needed data from there I had to call a shortcut from scriptable to copy the data. I use the data in a second script with widget. Using an widget and calling shortcuts from scriptable is not compatible. So I landed at the sync-project.

[sylumer]

@MDS

How about QNAP? Can you use QNAP files together with scriptable and shortcuts?

In terms of what I might do, it is important to first understand that I do not sync files between my QNAP NAS and my mobile devices. I do access files on the NAS, but I do not sync back and forth. This means that I always need to have a live connection, but it also means that there are never any sync conflicts to resolve.

On the rare occasions I want to work with a file held on the NAS, I use Secure ShellFish to access the file directly on the NAS. The app is Shortcuts friendly so automation from mobile is stright forward.

I’ve never tried it from Scriptable, but given Scriptable’s file and folder bookmarking, and that Secure ShellFish is also a storage provider, I wouldn’t imagine that would be an issue if I did want to interact with a file from Scriptable using the same app as the go-between.

[schl3ck]

I use SMB when writing a Scriptable script on my PC to copy it to my iPhone. Since setting it up 3 months ago it worked without reconnecting, even though I turn off my PC and WiFi regularly.
My PC has the same IP address every time and I need to log in with my user account. I expect that from your NAS too so I don’t know what might be going wrong for you.

Thinking about this, I got the idea that maybe your widget tries to sync, iOS can’t connect to your NAS because it is offline (or you’re not connected to your WiFi) and then stores that error. When your in your WiFi again and the NAS is running, iOS then remembers that the connection had failed earlier and wants your confirmation that the connection works. But that’s just a theory… And it wasn’t the case for me. Scriptable wasn’t responding when trying to access the file bookmark, but other than that there was no problem.

Since I don’t own a NAS, I have no experience with all the other things you mentioned and can’t help you there.

[Yvonnezed]

I have to admit that in a similar situation, I gave up and went with an app as per a previous suggestion. Something like File Browser go, which I’m using, or maybe Documents would probably work.

Secure Shellfish was less effective for me since I was trying to use offline mode. The problem with that was that the server was always there but the drive I was using wasn’t always connected, so it’d just delete the offline version every time the drive was disconnected. Urgh!

Sadly, it’s not easy.

[MDS]

I publish my script for simple Synchronisation here. Feel free to use and improve it.

What does it do? It is a scriptable widget which is syncing a local folder with a folder on a server. As it is a widget the sync process is pushed from time to time automatically by the operation system. When using the script on two devices, e.g. an iPhone and an iPad the selected files are kept synchronized between these devices.

Instruction:

1. Include the server (nextcloud, owncloud, synology) in the file system within the file-app via „connect to server“. Your server should therefore support a file system which is supported from iOS. I use SMB which is supported from the synology diskstation.
2. Create 2 „File Bookmarks“ within the scriptable-app: one for the server and one for the local folder. With this method all difficult issues are solved by scriptable.
3. Adjust the script to your specification by entering the name of the bookmarked paths in line 2 and 3 (localPath and serverPath)

Known limitations:

• Deleting is not supported. That’s fine for my application as I use it only for a couple of files. If you need to delete anything you need to stop the script and delete the files manually on all devices and the server. If you only delete it on one device it is re-downloaded from the server. If you add this feature, let me know!
• From time to time I receive the following message within scriptable: Error on line 3:37: Cannot resolve bookmark named „serverZerzeichnis“. But if you look a little bit later, the error message disappears the the files get synced

Here is the code:

const fm = FileManager.local(); 
const localPath = fm.bookmarkedPath("Sync_Ordner"); 
const serverPath = fm.bookmarkedPath("serverVerzeichnis");
const widget = new ListWidget(); 

let allServerFiles = fm.listContents(serverPath);
let newLocalFromServerList = [];
let newServerFromLocalList = [];
let synchronousFilesList = [];
let updatedLocalFromServerList = [];
let updatedServerFromLocalList = [];
let backupFilePath;

widget.addText(showTime())
    
for (fileName of allServerFiles) {
    let localFilePath = fm.joinPath(localPath, fileName);
    if (fm.fileExists(localFilePath)) {    
        let serverFilePath = fm.joinPath(serverPath, fileName);
        let serverFileTime = Date.parse(fm.modificationDate(serverFilePath));
        let localFileTime  = Date.parse(fm.modificationDate(localFilePath));
            
        if ( serverFileTime > localFileTime ){    
            // both files exist, server file is later
            updatedLocalFromServerList.push(fileName)    
            backupFilePath = fm.joinPath(localPath, fileName +".backup")
            fm.move(localFilePath, backupFilePath)
            fm.copy(serverFilePath, localFilePath)
            fm.remove(backupFilePath)
        } else if ( serverFileTime < localFileTime ){
            // both files exist, local file is later
            updatedServerFromLocalList.push(fileName);
            backupFilePath = fm.joinPath(serverPath, fileName + ".backup")    
            fm.move(serverFilePath, backupFilePath)
            fm.copy(localFilePath, serverFilePath)
            fm.remove(backupFilePath)
        } else if ( serverFileTime === localFileTime ){
            // both files sync
            synchronousFilesList.push(fileName)
        } else {
            widget.addText("Fehler beim Datumsvergleich") 
        }
    } else {
        // File is new! Does not exist local. Copy from server to local.
        let serverFilePath = fm.joinPath(serverPath, fileName)    
        fm.copy(serverFilePath, localFilePath)        
        newLocalFromServerList.push(fileName)
    }    
}
// Check if local files don't exist on server. Copy from local to server.
let allLocalFiles = fm.listContents(localPath);
for (fileName of allLocalFiles) {    
    var localFilePath = fm.joinPath(localPath, fileName)
    var serverFilePath = fm. joinPath(serverPath, fileName)
    if ( !allServerFiles.includes(fileName) ) {
        newServerFromLocalList.push(fileName)    
        fm.copy(localFilePath, serverFilePath)
    }
}


log("newLocalFromServer:" + newLocalFromServerList.length.toString() + " / newServerFromLocal:" + newServerFromLocalList.length.toString() + " / synchronousFiles:" + synchronousFilesList.length.toString() + " / updatedLocalFromServer:" + updatedLocalFromServerList.length.toString() + " / updatedServerFromLocal:" + updatedServerFromLocalList.length.toString())

textToWidget("Synchrone Daten ", synchronousFilesList)
textToWidget("Neu vom Server geholt ", newLocalFromServerList)
textToWidget("Neu auf den Server geladen ", newServerFromLocalList)
textToWidget("Lokal vom Server aktualisiert ", updatedLocalFromServerList)
textToWidget("Auf Server von Lokal aktualisiert ", updatedServerFromLocalList)

await Script.setWidget(widget)

if (!config.runsInWidget) {
    await widget.presentLarge()
}


Script.complete()

function showTime() {        
    var date = new Date();
    var hourZero = "";    
    var minuteZero = "";    
    var secondZero = "";    
    if (date.getHours() < 10) { hourZero = "0" }    
    if (date.getMinutes() < 10) { minuteZero = "0" }    
    if (date.getSeconds() < 10) { secondZero = "0" }    
    var text = hourZero + date.getHours() + ":" + minuteZero + date.getMinutes() + ":" + secondZero + date.getSeconds()
     return text;
}

function textToWidget(headline, list) {        
    var caption = widget.addStack()
    var captionText = caption.addText(headline)    
        captionText.font = Font.boldMonospacedSystemFont(14)
        captionText.textColor = Color.gray()    
    caption.addSpacer(5)
    var captionNumber = caption.addText(list.length.toString())        
        captionNumber.font = Font.boldMonospacedSystemFont(14)
        captionNumber.textColor = Color.red()

    textBody = widget.addText(list.toString())
    textBody.font = Font.boldSystemFont(12)
} 

[MDS]

If it works, the widget looks as follows.

The text is in German and means:

• Synchronous files
• new downloaded from server
• new uploaded to server
• local existing file updated
• existing file on server updated