Remote Storage to User File System Synchronization

In this article

Remote Storage to User File System Synchronization

Each file and folder in the User File System can store an item identifier that helps to link user file system item to your remote storage item. You will set the remote storage item ID for the root folder prior to starting the Engine, as well as you will pass item ID as part of each item data when listing folder content and synchronizing changes. For new items, created in the user file system, you will return the newly created item ID to the Engine from the IFolder.CreateFileAsync() and IFolder.CreateFolderAsync() methods.

The Engine will pass the ID into each GetFileSystemItemAsync() method call, so you can identify your item and perform calls to your remote storage.

Before starting the Engine you can set the remote storage item ID for the root folder by calling Engine.SetRemoteStorageRootItemId() method:

// Set root item ID. It will be passed to IEngine.GetFileSystemItemAsync() method 
// as a remoteStorageItemId parameter when a root folder is requested. 
// In this example we just read the ID of the remote storage root folder 
// used as a remote storage simulation.
byte[] itemId = WindowsFileSystemItem.GetItemIdByPath(Settings.RemoteStorageRootPath);
engine.SetRemoteStorageRootItemId(itemId);

Note that the remote storage ID must be unique withing your file system and can NOT change during lifetime of the item, including during move operation. 

Remote Storage to User File System Synchronization

The functionality described in this section is supported in User File System v6.3 and later versions.

For remote storage to user file system synchronization the Engine provides sync ID algorithm support. The sync ID algorithm is the only approach that work for both Windows and macOS. 

Sync ID algorithm requirements: 

  • Each item in your storage must have a remote storage ID associated with it, unique withing your file system. The ID should NOT change during lifetime of an item including during move operation.
  • Your remote storage must be able to get all changes that happened since provided sync-token. The Engine will request all items changed (created, updated and moved) and deleted.
  • For each changed and deleted item your remote storage must provide a parent folder remote storage ID.
  • Your remote storage must be able to return item URI from remote storage ID. This is required on macOS only.

Remote Storage ID Implementation

To properly support remote storage ID you will need:

  • Set the remote storage ID for the root folder before starting the Engine using the Engine.SetRemoteStorageRootItemId() method.
  • Fill the IFileSystemItemMetadata.RemoteStorageItemId for each item when listing folder content in IFolder.GetChildrenAsync() method.
  • Return remote storage ID for newly created items from IFolder.CreateFileAsync() and IFolder.CreateFolderAsync() methods.

Getting Remote Storage Changes

To request changes from your remote storage, implement the ISynchronizationCollection interface on your root folder. This interface provides a single GetChangesAsync() method in which you will request all changes made in your remote storage since provided sync-token and return it to the Engine:

public class VirtualFolder : VirtualFileSystemItem, IFolder, ISynchronizationCollection
{
    ...
    public async Task<IChanges> GetChangesAsync(
        string syncToken, 
        bool deep, 
        long? limit, 
        CancellationToken ct)
    {
        Changes changes = new Changes();

        Client.PropertyName[] propNames = new Client.PropertyName[2];
        propNames[0] = new Client.PropertyName("resource-id", "DAV:");
        propNames[1] = new Client.PropertyName("parent-resource-id", "DAV:");

        // Get all changed and deleted items from the remote storage.
        Client.IChanges davChanges = await Session.GetChangesAsync(
            RemoteStorageID, 
            propNames, 
            syncToken, 
            deep, 
            limit);

        // A new sync token will be saved on the client machine.
        changes.NewSyncToken = davChanges.NewSyncToken;

        foreach (Client.IChangedItem remoteStorageItem in davChanges)
        {
            IChangedItem itemInfo = (IChangedItem)Mapping.GetUserFileSystemItemMetadata(remoteStorageItem);
            if (remoteStorageItem.ChangeType == Client.Change.Changed)
            {
                itemInfo.ChangeType = Change.Changed
            }
            else
            {
                itemInfo.ChangeType = Change.Deleted;
            }
            changes.Add(itemInfo);
        }

        return changes;
    }
}

In case the Engine fails to update any items returned by the GetChangesAsync() call, for example because they were blocked by applications or by the platform, they will be set to the conflict state and will be marked with a conflict icon.

Detecting Sync-ID algorithm Support and Initializing Sync-Token

To detect if your implementation supports Sync-ID algorithm, the Engine will first request your root folder by calling IEngine.GetFileSystemItem() method and test it for ISynchronizationCollection interface support. If this interface is supported it will start Sync-Token initialization.

To avoid any data loss during synchronization, the sync-token must be initialized before the Engine starts using any on-demand loading mechanisms, such as populating folders or hydrating files. To get the sync-token the Engine will call the ISynchronizationCollection.GetChangesAsync() on the root folder passing 0 as a limit parameter. The limit parameter set to zero, indicates that the Engine does not need any changes from your remote storage, but instead needs a sync-token only.

Triggering Synchronization

You will typically start the remote storage to user file system synchronization after receiving a signal about changes from your remote storage via web sockets or similar communication technology. 

On Windows, to start the synchronization you will call the IServerCollectionNotifications.ProcessChangesAsync() method. The IServerCollectionNotifications interface is returned by the Engine from IEngine.ServerNotifications() method:

IServerCollectionNotifications n = await Engine.ServerNotifications(Engine.Path);
await n.ProcessChangesAsync();

On macOS the Engine runs inside file provider that typically can not receive notifications from remote storage directly. That is you your web sockets processing code will be located in your hosting application that runs in a separate process. To trigger the synchronization on macOS, the library provides ServerNotifications class that implements IServerCollectionNotifications interface that can be instantiated directly, without Engine instance:

var fileProviderManager = NSFileProviderManager.FromDomain(domain);
IServerCollectionNotifications n = new ServerNotifications(fileProviderManager);
await n.ProcessChangesAsync();



 

 

  1. Init sync-token. This step must complete before starting processing file system calls.
  2. Start processing file system calls: listing folders and hydrating files.
  3. Start processing remote storage notifications.

Processing file system calls and receiving server notifications can run simultaneously and the same data may be received and processed in parallel, this is a normal synchronization lifecycle. For example listing folder content may happen simultaneously with receiving notification from your remote storage that the file was created in this folder. The Engine will try to create this file from both threads and will not show any errors.

If a file with the same name was created on the client and incoming sync fails, this will be detected during outgoing synchronization and a file will be set in conflict state.

   

Next Article:

Previous Versions