Incoming Synchronization

In this article

Incoming Synchronization

In this article we describe a cross-platform remote storage to user file system synchronization (incoming synchronization) approach based on Sync ID algorithm. 

After the initial on-demand population and hydration the folder listing and file content download never happens again (on Windows), or happens occasionally (on macOS). To apply changes made in your remote storage in the user file system, you will typically implement push notifications from your remote storage to clients machines, via web sockets or similar technologies. The Engine provides Sync ID algorithm support that allows you to request all changes that happened in the remote storage since provided sync token using a single request. In this article we describe how to request changes from your remote storage after you receive a change notification.

Sync ID algorithm is the only cross-platform synchronization approach supported by Windows, macOS and iOS. macOS platform does NOT support synchronization of individual items.

The functionality described in this section is supported in User File System v6.3 and later versions. 

You can find sample synchronization implementation in the following User File System samples:

Remote Storage Requirements

To support Sync ID algorithm your remote storage must support the following functionality:  

  • Each item in your storage must have a remote storage ID associated with it, unique withing your file system. The ID should NOT change during lifetime of an item including during move operation.
  • Your remote storage must be able to get all changes that happened since provided sync-token. The Engine will request from your remote storage all items that changed and deleted (changed means created, updated or moved).
  • For each changed and deleted item your remote storage must provide a parent folder remote storage ID.
  • Your remote storage must be able to return item URI by remote storage item ID. This is required on macOS only.

You can find how to implement synchronization based on Sync ID in IT Hit WebDAV servers in the following articles: 

The following demo sites provide Sync ID synchronization support, you can use them for demo and testing proposes. Note that you must navigate to the demo site in a web browser and in your code connect to the folder automatically created for you for testing purposes, which looks like https://demosite/User123456/  

The following sample WebDAV servers provide Sync ID synchronization support:

Remote Storage Item ID Implementation

To support Sync ID algorithm you need a proper remote storage ID implementation. Here is how you implement the ID in your application:

Application Start Sequence

Here is your application startup sequence:

  1. The Engine initializes sync-token by sending a request to the remote storage. This step completes before file system calls processing is started.
  2. The Engine starts processing file system calls: listing folders and hydrating files.
  3. Your implementation starts receiving incoming notifications from remote storage and triggers synchronization by calling the IServerCollectionNotifications.ProcessChangesAsync() method.

Processing file system calls and receiving server notifications can run simultaneously and the same data may be received and processed in parallel, this is a normal synchronization lifecycle.

Detecting Sync-ID Algorithm Support

To detect if your implementation supports Sync ID algorithm, the Engine will first get your root folder by calling IEngine.GetFileSystemItemAsync() method and test the returned object  for ISynchronizationCollection interface support. If this interface is supported on your root folder it will start Sync-Token initialization.

Sync-Token initialization

To avoid any data loss during synchronization, the sync-token must be initialized before the Engine begins using any on-demand loading mechanisms, such as populating folders or hydrating files. To get the sync-token, the Engine calls the ISynchronizationCollection.GetChangesAsync() on the root folder passing 0 as a limit parameter. Setting the limit parameter to zero, indicates that the Engine does not need any changes from your remote storage, but instead requires a sync-token only.

Triggering Synchronization

You will typically start the remote storage to user file system synchronization after receiving a signal about changes from your remote storage via web sockets or similar communication technology. 

To start the synchronization you will call the IServerCollectionNotifications.ProcessChangesAsync() method. This will trigger the ISynchronizationCollection.GetChangesAsync() method call. 

The IServerCollectionNotifications interface is returned by the Engine from IEngine.ServerNotifications() method:

IServerCollectionNotifications n = await Engine.ServerNotifications(Engine.Path);
await n.ProcessChangesAsync();

In addition to that, on macOS, you may run web sockets inside your host application. As soon as host application runs in the separate process and does not have access to the Engine instance, the library on macOS provides ServerNotifications class that implements IServerCollectionNotifications interface that can be instantiated directly, without the Engine instance:

var fileProviderManager = NSFileProviderManager.FromDomain(domain);
IServerCollectionNotifications n = new ServerNotifications(fileProviderManager);
await n.ProcessChangesAsync();

Getting Changes from Remote Storage

To get changes from your remote storage, implement the ISynchronizationCollection interface on your root folder. This interface provides a single GetChangesAsync() method in which you will request all changes made in your remote storage since provided sync-token and return it to the Engine:

public class VirtualFolder : VirtualFileSystemItem, IFolder, ISynchronizationCollection
{
    ...
    public async Task<IChanges> GetChangesAsync(
        string syncToken, 
        bool deep, 
        long? limit, 
        CancellationToken ct)
    {
        Changes changes = new Changes();

        Client.PropertyName[] propNames = new Client.PropertyName[2];
        propNames[0] = new Client.PropertyName("resource-id", "DAV:");
        propNames[1] = new Client.PropertyName("parent-resource-id", "DAV:");

        // Get all changed and deleted items from the remote storage.
        Client.IChanges davChanges = await Session.GetChangesAsync(
            RemoteStorageID, 
            propNames, 
            syncToken, 
            deep, 
            limit);

        // A new sync token will be saved on the client machine.
        changes.NewSyncToken = davChanges.NewSyncToken;

        foreach (Client.IChangedItem remoteStorageItem in davChanges)
        {
            IChangedItem itemInfo = (IChangedItem)Mapping.GetUserFileSystemItemMetadata(
                remoteStorageItem);
            if (remoteStorageItem.ChangeType == Client.Change.Changed)
            {
                itemInfo.ChangeType = Change.Changed
            }
            else
            {
                itemInfo.ChangeType = Change.Deleted;
            }
            changes.Add(itemInfo);
        }

        return changes;
    }
}

The Engine will process the list of changed and deleted items and apply it to the file system. It will also store the sync-token until the next GetChangesAsync() call.

In case the Engine fails to update any items returned by the GetChangesAsync() call, for example because they were blocked by applications or by the platform, they will be set to the conflict state and will be marked with a conflict icon.

See Also:

Next Article:

Implementing Thumbnails Support in Virtual File System