Creating Virtual File System in .NET

In this article we will describe how to create a virtual file system in .NET with basic functionality such as on-demand folder content listing (population), on-demand file content hydration, offline files support and client-to-server synchronization.

All programming interfaces covered in this article are cross-platform and are supported on both Windows and macOS. The cross-platform server-to-client synchronization based on Sync ID is described in this article. For specifics of the platform code refer to these articles:

The Engine is designed to publish data from almost any storage. In this article we will use WebDAV server as a back-end storage for demo purposes.

Core Interfaces

There are 2 types of items in a virtual file system: folders, represented by IFolder interface and files represented by IFile interface. Both IFolder and IFile interfaces are derived from IFileSystemItem interface that provides common methods and properties.

Files and folders are generated by the factory method called GetFileSystemItemAsync() that you must override in your class derived from EngineWindows or EngineMacBelow is a typical structure of classes in your project:

  Typical Virtual File System project class diagram

Each file and folder in the User File System can store an item identifier that helps to link user file system item to your remote storage item, called remote storge item ID.

You will set the remote storage item ID for the root folder prior to starting the Engine using the IEngine.SetRemoteStorageRootItemId() method. The Engine will pass the ID into each IEngine.GetFileSystemItemAsync() method call, so you can identify your item and perform calls to your remote storage.

PropertyName[] props = new PropertyName[1];
props[0] = new PropertyName("resource-id", "DAV:");
IHierarchyItem rootFolder = (await DavClient.GetItemAsync(davUrl, props)).WebDavResponse;
var rootItemInfo = Mapping.GetUserFileSystemItemMetadata(rootFolder);
engine.SetRemoteStorageRootItemId(rootItemInfo.RemoteStorageItemId);

When listing folder content you will set the remote storge item ID by setting the IFileSystemItemMetadata.RemoteStorageItemId property for each item in the list that you return from IFolder.GetChildrenAsync() method. See the Implementing Folder Content Listing section below for an example. 

For new items, created in the user file system, you must return the remote storage item ID from the IFolder.CreateFileAsync() and IFolder.CreateFolderAsync() methods. See the Creating Files and Folders in the Remote Storage section in this article.

Note that the remote storage item ID must be unique withing your file system and can NOT change during lifetime of the item, including during move operation.

Implementing Factory Method

When a platform receives a call via its file system API, such as folder listing or reading file content, the Engine will first request a folder or file item by calling the IEngine.GetFileSystemItemAsync() virtual factory method. The Engine will pass the remote storage item ID and an item type (file or folder) as parameters. After that, the Engine will call IFile or IFolder interface methods on the item returned by GetFileSystemItemAsync().

Create a VirtualEngine class in your project and derive it from EngineWindows or EngineMac class. Then override the GetFileSystemItemAsync() method, which will return your files and folders:

public class VirtualEngine : EngineWindows
{
    public override async Task<IFileSystemItem> GetFileSystemItemAsync( 
        byte[] remoteStorageItemId,
        FileSystemItemType itemType,
        IContext context)
    {
        if (itemType == FileSystemItemType.File)
        {
            return new VirtualFile(remoteStorageItemId);
        }
        else
        {
            return new VirtualFolder(remoteStorageItemId);
        }
    }
}

Note that typically, for performance reasons, you should NOT make any server calls inside your GetFileSystemItemAsync() method implementation. Instead, you will just create an item and return it to the Engine. You will make server calls inside your IFile and IFolder method implementations.

Depending on the platform, you can get additional data in GetFileSystemItemAsync() call. For example on Windows you can get the user file system path, which is passed in context parameter. However, on other platforms, such as macOS and iOS path is not available.

Implementing Folder Listing (Population)

During the initial Engine start, the content of the root folder and all underlying folders is unknown. When the folder listing is performed, for example because you started browsing your virtual file system in OS file manager, or because an application opens a file on your disk, the platform blocks listing or opening until the folder content is populated.

When the platform makes a folder listing request, the Engine calls the IFolder.GetChildrenAsync() method. In your implementation, you will call your remote storage, create a list with information about files and folders, and return it to the Engine by calling the IFolderListingResultContext.ReturnChildrenAsync() method. Below is a sample GetChildrenAsync() method implementation which uses a WebDAV server as a the remote storage:

public class VirtualFolder: IFolder
{
    protected readonly string RemoteStoragePathById;

    public VirtualFolder(byte[] remoteStorageItemId)
    {
        RemoteStoragePathById = Mapping.GetPathById(remoteStorageItemId);
    }

    public async Task GetChildrenAsync(
        string pattern, 
        IOperationContext operationContext, 
        IFolderListingResultContext resultContext, 
        CancellationToken cancellationToken)
    {
        Client.PropertyName[] propNames = new Client.PropertyName[2];
        propNames[0] = new Client.PropertyName("resource-id", "DAV:");
        propNames[1] = new Client.PropertyName("parent-resource-id", "DAV:");

        var remoteStorageChildren = (await DavClient.GetChildrenAsync(
            RemoteStoragePathByID, 
            false, 
            propNames)).WebDavResponse;


        var userFileSystemChildren = new List<IFileSystemItemMetadata>();
        foreach (Client.IHierarchyItem remoteStorageItem in remoteStorageChildren)
        {
            var itemInfo = Mapping.GetUserFileSystemItemMetadata(remoteStorageItem);
            userFileSystemChildren.Add(itemInfo);
                
        }

        await resultContext.ReturnChildrenAsync(
            userFileSystemChildren.ToArray(), 
            userFileSystemChildren.Count());
    }
    ...
}

You can call the IFolderListingResultContext.ReturnChildrenAsync() method multiple times, returning the list of files and folders in several turns. To specify the total amount of children, the ReturnChildrenAsync() method provides a second parameter, so the platform knows when children's enumeration is completed.

After the GetChildrenAsync() call, files does not contain any content, they are dehydrated. Even though they report correct file size and all platform file API treats such files as a regular files. In your OS file manager dehydrated files are marked with an icon, indicating that they do not contain any data. For example on Windows platform such files are marked with an offline attribute and have a cloud icon Offline file in the Windows File Manager in the Status column. On macOS dehydrated files are marked with  icon next to the file name.

Virtual File System in Windows Explorer with dehydrated files
Virtual File System on macOS platform with dehydrated files

Note that on Windows platform (unless you implement streaming mode) the GetChildrenAsync() method is called only one time during the initial on-demand population. After the initial call, you will update the folder content using the approach described in the Incoming Synchronization article. On macOS, the platform invalidates data from time to time and the GetChildrenAsync() method may be called more than one time for the same folder. Even though, the incoming synchronization is typically required on macOS too.

Each item in the list returned to the Engine must be of the type IFileMetadata or IFolderMetadata. The User File System library provides FileMetadata and FolderMetadata classes that you can use out of the box in many cases. Below is the GetUserFileSystemItemMetadata() method implementation used in the the above example: 

public static IFileSystemItemMetadata GetUserFileSystemItemMetadata(
    Client.IHierarchyItem remoteStorageItem)
{
    FileSystemItemMetadata item;

    if (remoteStorageItem is Client.IFile)
    {
        Client.IFile remoteStorageFile = (Client.IFile)remoteStorageItem;
        item = new FileMetadata();
        ((FileMetadata)item).Length = remoteStorageFile.ContentLength;
        item.Attributes = FileAttributes.Normal;
    }
    else
    {
        item = new FolderMetadata();
        item.Attributes = FileAttributes.Normal | FileAttributes.Directory;
    }

    item.Name = remoteStorageItem.DisplayName;
    item.CreationTime = remoteStorageItem.CreationDate;
    item.LastWriteTime = remoteStorageItem.LastModified;
    item.LastAccessTime = remoteStorageItem.LastModified;
    item.ChangeTime = remoteStorageItem.LastModified;

    item.RemoteStorageItemId = GetProp(remoteStorageItem, "resource-id");
    item.RemoteStorageParentItemId = GetProp(remoteStorageItem, "parent-resource-id");

    return item;
}

Reading File Content (Hydration)

The process of downloading the file content from the remote storage to the user file system is called Hydration. When files are initially synched from the remote storage to the User File System during the IFolder.GetChildrenAsync() method call, they do not have any content on disk. When any application opens a file handle to access the file, the platform detects that the file is dehydrated and blocks opening until the requested segment of file or entire file (depending on your virtual file system mode and file API call parameters) is returned to the platform. At this moment the Engine calls the IFile.ReadAsync() method passing offset and a length of the block of the file content requested by the platform. It also passes the output stream to which you will write the data. Below we provide an example of the ReadAsync() method implementation with a WebDAV server being used as a remote storage:

public class VirtualFile: IFile
{
    protected readonly string RemoteStoragePathById;

    public VirtualFile(byte[] remoteStorageItemId)
    {
        RemoteStoragePathById = Mapping.GetPathById(remoteStorageItemId);
    }

    public async Task ReadAsync(
        Stream output, 
        long offset, 
        long length, 
        ITransferDataOperationContext operationContext, 
        ITransferDataResultContext resultContext, 
        CancellationToken cancellationToken)
    {
        const int bufferSize = 0x500000; // 5Mb.
        using (Client.IDownloadResponse response = await DavClient.DownloadAsync(
            RemoteStoragePathById, 
            offset, 
            length, 
            null, 
            cancellationToken))
        {
            using (Stream stream = await response.GetResponseStreamAsync())
            {
                await stream.CopyToAsync(output, bufferSize, cancellationToken);
            }
        }
    }
    ...
}

As soon as the regular Stream.CopyToAsync() does not support the length of the data to be copied, here is the CopyToAsync() extension method with count parameter:

public static async Task CopyToAsync(
    this Stream source, 
    Stream destination, 
    int bufferSize, 
    long count)
{
    byte[] buffer = new byte[bufferSize];
    int read;
    while (count > 0 
        && (read = await source.ReadAsync(
            buffer, 
            0, 
            (int)Math.Min(buffer.LongLength, count))) > 0)
    {
        await destination.WriteAsync(buffer, 0, read);
        count -= read;
    }
}

During the download process, the platform automatically calculates and displays the download progress:  

File download progress in Windows Explorer
macOS Finder download progress in cloud files.

Optionally you can also call the IResultContext.ReportProgress() method to report the progress to the platform.

On Windows, when the download is completed the file icon turns into a checkbox on a white background , meaning the file is on disk and is in the in-sync state. On macOS the file is displayed without any icons:

Hydrated files are marked with green checkbox in Windows Explorer
macOS Finder online cloud file

Note that hydrated files, marked with  icons, can be purged from the file system in case there is not enough space on the disk. To avoid this, the user can "pin" the file by calling the "Always keep on this device" menu in Windows File Manager. In this case, the file will be marked with Pinned attribute and will remain in the file system regardless of the remaining disk space.

Writing File Content

When the file content or file metadata is modified and needs to be uploaded to the remote storage the Engine calls IFile.WriteAsync() method. The Engine passes updated metadata and a file content stream as parameters.

An example below is using a WebDAV server as a remote storage: 

public class VirtualFile: IFile
{
    ...
    public async Task WriteAsync(
        IFileMetadata fileMetadata, 
        Stream content = null, 
        IOperationContext operationContext = null, 
        IInSyncResultContext inSyncResultContext = null,
        CancellationToken cancellationToken = default)
    {
        if (content != null)
        {
            // Update remote storage file content.
            await DavClient.UploadAsync(RemoteStoragePathByID, async (outputStream) =>
            {
                content.Position = 0;
                await content.CopyToAsync(outputStream);
            }, null, content.Length, 0, -1, null, null, null, cancellationToken);
        }

        // Here you can also update file metadata 
        // if your remote storage supports this:
        //item.Attributes = fileMetadata.Attributes;
        //item.CreationTimeUtc = fileMetadata.CreationTime.UtcDateTime;
        //item.LastWriteTimeUtc = fileMetadata.LastWriteTime.UtcDateTime;
        //item.LastAccessTimeUtc = fileMetadata.LastAccessTime.UtcDateTime;
        //item.LastWriteTimeUtc = fileMetadata.LastWriteTime.UtcDateTime;
    }
}

Note that in some cases, if the file content is not modified, or if the file is blocked, the content parameter may be null. 

If the WriteAsync() method completes without exceptions the file is marked as in-sync. Otherwise, the file is left in the not in-sync state.

Creating Files and Folders in the Remote Storage

When a file or a folder is being created in the user file system the Engine calls IFolder.CreateFileAsync() or IFolder.CreateFolderAsync() methods. The Engine passes a new item metadata and, in the case of a file, the file content stream as parameters. The file content stream may be null if the file is blocked. Below is an example of CreateFileAsync() and CreateFolderAsync() methods implementation with a WebDAV server being used as a remote storage:

public class VirtualFolder: IFolder
{
    ...
    public async Task<byte[]> CreateFileAsync(
        IFileMetadata fileMetadata, 
        Stream content = null, 
        IInSyncResultContext inSyncResultContext = null,
        CancellationToken cancellationToken = default)
    {
        // Create a new file in the remote storage.
        Uri newFileUri = new Uri(new Uri(RemoteStoragePath), fileMetadata.Name);

        long contentLength = content != null ? content.Length : 0;

        // Send content to the remote storage.
        Client.IWebDavResponse<string> response = (await DavClient.UploadAsync(
            newFileUri, 
            async (outputStream) =>
            {
                if (content != null)
                {
                    // Setting position to 0 is required in case of retry.
                    content.Position = 0;
                    await content.CopyToAsync(outputStream);
                }
            }, null, contentLength, 0, -1, null, null, null, cancellationToken));

        // Return newly created item remote storage item ID,
        // it will be passed to GetFileSystemItem() during next calls.
        string id = response.Headers.GetValues("resource-id").FirstOrDefault();
        return Encoding.UTF8.GetBytes(id);
    }
}

CreateFolderAsync() method implementation example:

public class VirtualFolder: IFolder
{
    ...
    public async Task<byte[]> CreateFolderAsync( 
        IFolderMetadata folderMetadata, 
        IInSyncResultContext inSyncResultContext = null, 
        CancellationToken cancellationToken = default)
    {
        Uri newFolderUri = new Uri(new Uri(RemoteStoragePath), folderMetadata.Name);
        Client.IResponse response =  await DavClient.CreateFolderAsync(
            newFolderUri, 
            null, 
            null, 
            cancellationToken);

        // Return newly created item remote storage item ID,
        // it will be passed to GetFileSystemItem() during next calls.
        string id = response.Headers.GetValues("resource-id").FirstOrDefault();
        return Encoding.UTF8.GetBytes(id);
    }
}

You will return the new remote storage item ID from IFolder.CreateFileAsync() and IFolder.CreateFolderAsync() methods. This ID will then be passed to the IEngine.GetFileSystemItemAsync() method as a remoteStorageItemId parameter during every file system operation.

On Windows platform, if the CreateFileAsync() method completes without exceptions and the content parameter is NOT null, the file is converted into a placeholder. Otherwise, the file remains a regular file. If the content parameter is null, you will typically create a 0-length file in your remote storage and will upload content at a later time, when the file handle is released. 

If the CreateFolderAsync() method completes without exceptions the folder is converted into a placeholder, otherwise, the folder remains a regular folder.  

Next Article:

Incoming Synchronization