Creating Virtual File System in .NET

In this article

Creating Virtual File System in .NET

In this article, we will provide you with step-by-step instructions on creating a virtual file system for Windows in .NET with basic functionality such as on-demand folder content listing, on-demand file content hydration, offline files support, client-to-server, and server-to-client synchronization. We will use a folder in a file system as a remote storage simulation. 

This article does NOT cover locking, Microsoft Office documents editing, thumbnail support, and custom properties support.

This article is about User File System v3+. For previous versions please refer to the articles in this section.

The Core Engine Interfaces

There are 2 types of items in a virtual file system: folders, represented by IFolder interface and files represented by IFile interface. Both IFolder and IFileiterfaces are derived from IFileSystemItem interface. Files and folders are generated by the factory method called GetFileSystemItemAsync() that you must override in your class derived from EngineWindows or EngineMacBelow is a typicall class diagram of of your project:

  Typical Virtual File System project class diagram

Creating a .NET Project 

Create a .NET console application. In this article, we will use the .NET 5 as a target framework.

Add an ITHit.FileSystem.Windows module reference to your project. In Visual Studio you can do this via the Project->Manage NuGet Packages menu:

 Adding ITHit.FileSystem.Windows Nuget reference.

The IT Hit User File System on Windows requires Windows Runtime API support. To be able to use it we will set the target framework moniker in the project file. Open the project file and change the TargetFramework element:

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net5.0-windows10.0.19041.0</TargetFramework>
  </PropertyGroup>
  ...
</Project>

You can find more framework monikers for other Windows versions (as well as how to target .NET Core 3.1 if needed) in the Call Windows Runtime APIs in desktop apps article.

To inform the Windows platform about a new virtual file system and to receive file system API calls you must register your new file system. Typically you will do this during your application installation process. You will also unregister the virtual file system during uninstall. To register/unregister you will use the StorageProviderSyncRootManager class. Create a Registrar class and add it to your project:

public static class Registrar
{
    /// <summary>
    /// Registers sync root.
    /// </summary>
    /// <param name="syncRootId">ID of the sync root.</param>
    /// <param name="path">A root folder of your user file system.</param>
    /// <param name="displayName">Human readable display name.</param>
    /// <remarks>Call this method during application installation.</remarks>
    public async Task RegisterAsync(string syncRootId, string path, string displayName)
    {
        StorageProviderSyncRootInfo storageInfo = new StorageProviderSyncRootInfo();
        storageInfo.Path = await StorageFolder.GetFolderFromPathAsync(path);
        storageInfo.Id = syncRootId;
        storageInfo.DisplayNameResource = displayName;
        storageInfo.IconResource = @"C:\Proj\Icons\Drive.ico";
        storageInfo.Version = "1.0";
        storageInfo.RecycleBinUri = new Uri("https://userfilesystem.com/recyclebin");
        storageInfo.Context = CryptographicBuffer.ConvertStringToBinary(
        path, BinaryStringEncoding.Utf8);

        storageInfo.HydrationPolicy = StorageProviderHydrationPolicy.Progressive;
        storageInfo.HydrationPolicyModifier = 
                StorageProviderHydrationPolicyModifier.AutoDehydrationAllowed;
 
        // To support on-demand population set this property to 'Full'.
        storageInfo.PopulationPolicy = StorageProviderPopulationPolicy.Full;
 
        // This configures when PlaceholderItem.GetInSync() should return true/false.
        storageInfo.InSyncPolicy = StorageProviderInSyncPolicy.Default;
 
        StorageProviderSyncRootManager.Register(storageInfo);
    }

    public static async Task UnregisterAsync(string syncRootId)
    {
        StorageProviderSyncRootManager.Unregister(syncRootId);
    }
}

Note that if you have any regular files and folders under your virtual file system root during registration, they will NOT be converted into placeholders automatically. If needed, you can convert them to placeholders using PlaceholderItem.ConvertToPlaceholder() overloaded methods.

To unregister the virtual file system you will call the StorageProviderSyncRootManager.Unregister() method passing the syncRootId specified during registration.

Implementing Files and Folders Factory Method

When the platform receives a call via its file system API, such as folder listing or reading file content, the Engine will first request a folder or file item by calling the IEngine.GetFileSystemItemAsync() virtual factory method. The Engine will pass the user file system path, an item type (file or folder), and item ID as parameters. Then, the Engine will call IFile or IFolder interface methods.

Create a VirtualEngine class in your project and derive it from EngineWindows class. Then override the GetFileSystemItemAsync() method, which will return your files and folders:

public class VirtualEngine : EngineWindows
{
    public VirtualEngine(string license, string userFileSystemRootPath) 
            : base(license, userFileSystemRootPath)
    {
    }

    public override async Task<IFileSystemItem> GetFileSystemItemAsync(
        string userFileSystemPath, 
        FileSystemItemType itemType, 
        string itemId = null)
    {
        if (itemType == FileSystemItemType.File)
        {
            return new VirtualFile(itemId);
        }
        else
        {
            return new VirtualFolder(itemId);
        }
    }
}

Note that typically, for performance reasons, you should NOT make any server calls inside your GetFileSystemItemAsync() method implementation. Instead, you will just create an item and return it to the Engine. You will make server calls inside your IFile and IFolder method implementations.

Remote Storage Item ID

Each file and folder in the User File System can store an item identifier that helps linking the user file system item to your remote storage item. The maximum size of the item identifier on Windows is 4KB.

Note that some applications, for example, the Microsoft Office applications rename and then delete the original file during transactional save operation on the client machine, so the item ID which is stored together with a file will be deleted. Store the item ID as part of the file, as described in this article, only if the file is NOT being updated using transactional save operation. See Creating Virtual Drive article for how to store custom data and avoid document deletion in your remote storage during Microsoft Office documents save operation.

You can read and write item ID using PlaceholderItem class. To set the item ID call PlaceholderItem.SetItemId() method:

byte[] itemId = ...
PlaceholderItem item = PlaceholderItem.GetItem("@C:\Users\User1\VFS\file.ext");
item.SetItemId(itemId);

To read the item ID call the PlaceholderItem.GetItemId() method:

PlaceholderItem item = PlaceholderItem.GetItem("@C:\Users\User1\VFS\file.ext");
byte[] itemId = item.GetItemId();

Note that the item ID can only be stored with a placeholder item. It can not be stored with regular files and folders. If a placeholder item is converted into a regular file/folder, the item ID is beind deleted. 

The itemId parameter of the IEngine.GetFileSystemItemAsync() method is an ID of the item in your remote storage. It is passed to GetFileSystemItemAsync() method only if you specify it when creating an item in IFolder.GetChildrenAsync() method or set it by other means such as calling PlaceholderItem.SetItemId(). It the item ID was never set, the null is passed.

Typically you will set the item ID by filling the IFileSystemItemMetadata.ItemId property for each item that you return from IFolder.GetChildrenAsync() method. See the Implementing Folder Content Listing section below for an example.

You may also want to set the item ID of your root folder before the first Engine run. Otherwise, the itemId passed to IEngine.GetFileSystemItemAsync() method will be null for your file system root folder:

// Set root item ID. It will be passed to IEngine.GetFileSystemItemAsync() method 
// as itemId parameter when a root folder is requested. 
// In this example we just read the ID of the remote storage root folder 
// used as a remote storage simulation.
byte[] itemId = WindowsFileSystemItem.GetItemIdByPath(Settings.RemoteStorageRootPath);
PlaceholderFolder.GetItem(Settings.UserFileSystemRootPath).SetItemId(itemId);

For the itemId to be passed to the IEngine.GetFileSystemItemAsync() method for new items created in the user file system, you must also return it from the IFolder.CreateFileAsync() and IFolder.CreateFolderAsync() methods.

Implementing Folder Content Listing

When the platform makes a folder listing request, the Engine calls IFolder.GetChildrenAsync() method. In your implementation, you will call your remote storage, create a list with information about files and folders, and will return it to the Engine by calling the IFolderListingResultContext.ReturnChildren() method. Below is a sample GetChildrenAsync() method implementation which emulates the remote storage by listing the content of another folder in the file system:

public class VirtualFolder: IFolder
{
    protected readonly string RemoteStoragePath;

    public VirtualFolder(byte[] itemId)
    {
        RemoteStoragePath = WindowsFileSystemItem.GetPathByItemId(itemId);
    }

    public async Task GetChildrenAsync(
        string pattern, 
        IOperationContext operationContext, 
        IFolderListingResultContext resultContext)
    {
        var remoteStorageChildren = 
            new DirectoryInfo(RemoteStoragePath).EnumerateFileSystemInfos(pattern);

        var userFileSystemChildren = new List<IFileSystemItemMetadata>();
        foreach (FileSystemInfo remoteStorageItem in remoteStorageChildren)
        {
            var itemInfo = Mapping.GetUserFileSysteItemMetadata(remoteStorageItem);
            userFileSystemChildren.Add(itemInfo);
        }

        resultContext.ReturnChildren(
            userFileSystemChildren.ToArray(), 
            userFileSystemChildren.Count());
    }
    ...
}

You can call the IFolderListingResultContext.ReturnChildren() method multiple times, returning the list of files and folders in several turns. To specify the total amount of children, the ReturnChildren() method provides a second parameter, so the platform knows when children's enumeration is completed.

Note that on Windows, the GetChildrenAsync() method is called only one time during the initial on-demand population. After the initial call, you will update the folder content using a pull or push approach described below in this article in the Remote Storage to User File System Synchronization section.

Each item in the list returned to the Engine must be of the type IFileMetadata or IFolderMetadata. The User File System library provides FileMetadata and FolderMetadata classes that you can use out of the box in many cases. Below is the GetUserFileSysteItemMetadata() method implementation used in the the above example: 

public static IFileSystemItemMetadata GetUserFileSysteItemMetadata(
    FileSystemInfo remoteStorageItem)
{
    IFileSystemItemMetadata userFileSystemItem;

    if (remoteStorageItem is FileInfo)
    {
        userFileSystemItem = new FileMetadata();
        ((FileMetadata)userFileSystemItem).Length = 
            ((FileInfo)remoteStorageItem).Length;
    }
    else
    {
        userFileSystemItem = new FolderMetadata();
    }

    // Store your remote storage item ID here. 
    // It will be passed to IEngine.GetFileSystemItemAsync() during every operation.
    userFileSystemItem.ItemId = 
        WindowsFileSystemItem.GetItemIdByPath(remoteStorageItem.FullName);

    userFileSystemItem.Name = remoteStorageItem.Name;
    userFileSystemItem.Attributes = remoteStorageItem.Attributes;
    userFileSystemItem.CreationTime = remoteStorageItem.CreationTime;
    userFileSystemItem.LastWriteTime = remoteStorageItem.LastWriteTime;
    userFileSystemItem.LastAccessTime = remoteStorageItem.LastAccessTime;
    userFileSystemItem.ChangeTime = remoteStorageItem.LastWriteTime;

    return userFileSystemItem;
}

Starting the Engine

Now, when you have a file system registered and folder listing implemented you can start processing file system calls. To do this you will create a VirtualEngine class instance and calls IEngine.StartAsync() method. Below is your main() application method. For the sake of simplicity, we will register the virtual file system on application start and unregister on exit.

static async Task Main(string[] args)
{
    // Note that your root file system path must be indexed.
    string userFileSystemRootPath = @"C:\Users\User1\VFS\";

    // Register file system.
    Directory.CreateDirectory(userFileSystemRootPath);
    await Registrar.RegisterAsync(SyncRootId, userFileSystemRootPath, "VFS");

    // Set root item ID.
    byte[] itemId = WindowsFileSystemItem.GetItemIdByPath(@"C:\RemoteStorage\");
    PlaceholderFolder.GetItem(Settings.UserFileSystemRootPath).SetItemId(itemId);

    using (var Engine = new VirtualEngine(license, userFileSystemRootPath))
    {
        // Start processing OS file system calls.
        await Engine.StartAsync();

        // Keep this application running until user input.
        Console.ReadKey();
    }

    // Unregister during programm uninstall and delete all files/folder.
    await Registrar.UnregisterAsync(SyncRootId);
    Directory.Delete(userFileSystemRootPath, true);
}

// SyncRoot ID must be in the form: [Storage Provider ID]![Windows SID]![Account ID]
private static string SyncRootId
{
    get { return $"VirtualFileSystem!{WindowsIdentity.GetCurrent().User}!User"; }
}

Reading File Content (Hydration)

The process of downloading the file content from the remote storage to the user file system is called Hydration. Files created during the IFolder.GetChildrenAsync() call initially do not have any content on disk. Even though they report the correct file size, they are dehydrated. Such files are marked with an offline attribute and have a cloud icon Offline file in the Windows File Manager in the Status column:

Virtual File System in Windows Explorer with dehydrated files

When the platform detects that the file hydration is required, for example when an application opens a file handle for reading, the Engine calls the IFile.ReadAsync() method passing offset and a length of the block of the file content requested by the platform. It will also pass the output stream to which you will write the data. Below is an example of code with a file system being used as a remote storage simulation:

public class VirtualFile: IFile
{
    protected readonly string RemoteStoragePath;

    public VirtualFile(byte[] itemId)
    {
        RemoteStoragePath = WindowsFileSystemItem.GetPathByItemId(itemId);
    }

    public async Task ReadAsync(
        Stream output, 
        long offset, 
        long length, 
        ITransferDataOperationContext operationContext, 
        ITransferDataResultContext resultContext)
    {
        await using (FileStream stream = System.IO.File.OpenRead(RemoteStoragePath))
        {
            stream.Seek(offset, SeekOrigin.Begin);
            // Buffer size must be multiple of 4096 bytes for optimal performance.
            const int bufferSize = 0x500000; // 5Mb. 
            await stream.CopyToAsync(output, bufferSize, length);
        }
    }
    ...
}

As soon as the regular Stream.CopyToAsync() does not support the length of the data to be copied, here is the CopyToAsync() extension method with count parameter:

public static async Task CopyToAsync(
    this Stream source, 
    Stream destination, 
    int bufferSize, 
    long count)
{
    byte[] buffer = new byte[bufferSize];
    int read;
    while (count > 0 
        && (read = await source.ReadAsync(
            buffer, 
            0, 
            (int)Math.Min(buffer.LongLength, count))) > 0)
    {
        await destination.WriteAsync(buffer, 0, read);
        count -= read;
    }
}

During the download process, the platform automatically calculates and displays the download progress:  

File download progress in Windows Explorer 

Optionally you can also call the IResultContext.ReportProgress() method to report the progress to the platform.

When the download is completed the file icon turns into a checkbox on a white background , meaning the file is on disk and is in the in-sync state:

Hydrated files are marked with green checkbox in Windows Explorer

Note that hydrated files, marked with  icons, can be purged from the file system in case there is not enough space on the disk. To avoid this, the user can "pin" the file by calling the "Always keep on this device" menu in Windows File Manager. In this case, the file will be marked with Pinned attribute and will remain in the file system regardless of the remaining disk space.

Writing File Content

When the file content or file metadata is modified and needs to be uploaded to the remote storage the Engine calls IFile.WriteAsync() method. The Engine passes an updated metadata and a file content stream as parameters. In some cases, however, if the file content is not modified or if the file is blocked, the file content parameter will be null. An example below is using a file system as a remote storage simulation: 

public class VirtualFile: IFile
{
    ...
    public async Task WriteAsync(IFileMetadata fileMetadata, Stream content = null)
    {
        FileInfo remoteStorageItem = new FileInfo(RemoteStoragePath);

        if (content != null)
        {
            // Upload remote storage file content.
            await using (FileStream remoteStorageStream = remoteStorageItem.Open(
            FileMode.Open, FileAccess.Write, FileShare.Delete))
            {
                await content.CopyToAsync(remoteStorageStream);
                remoteStorageStream.SetLength(content.Length);
            }
        }

        // Update remote storage file metadata.
        remoteStorageItem.Attributes = fileMetadata.Attributes;
        remoteStorageItem.CreationTimeUtc = fileMetadata.CreationTime.UtcDateTime;
        remoteStorageItem.LastWriteTimeUtc = fileMetadata.LastWriteTime.UtcDateTime;
        remoteStorageItem.LastAccessTimeUtc = fileMetadata.LastAccessTime.UtcDateTime;
        remoteStorageItem.LastWriteTimeUtc = fileMetadata.LastWriteTime.UtcDateTime;
    }
}

Creating Files and Folders in Remote Storage

When a file or a folder is being created in the user file system the Engine calls IFolder.CreateFileAsync() or IFolder.CreateFolderAsync() methods. The Engine passes updated metadata and, in the case of a file, a file content stream as parameters. The file content stream can be null if the file is blocked. Below is a code example of file creation with a file system being used as a remote storage simulation:

public class VirtualFolder: IFolder
{
    ...
    public async Task<byte[]> CreateFileAsync(
        IFileMetadata fileMetadata, 
        Stream content = null)
    {
        FileInfo remoteStorageItem = 
            new FileInfo(Path.Combine(RemoteStoragePath, fileMetadata.Name));

        // Create file in the remote storage.
        await using (FileStream remoteStorageStream = remoteStorageItem.Open(
        FileMode.CreateNew, FileAccess.Write, FileShare.Delete))
        {
            if (content != null)
            {
                await content.CopyToAsync(remoteStorageStream);
                remoteStorageStream.SetLength(content.Length);
            }
        }

        // Set remote storage file metadata.
        remoteStorageItem.Attributes = fileMetadata.Attributes;
        remoteStorageItem.CreationTimeUtc = fileMetadata.CreationTime.UtcDateTime;
        remoteStorageItem.LastWriteTimeUtc = fileMetadata.LastWriteTime.UtcDateTime;
        remoteStorageItem.LastAccessTimeUtc = fileMetadata.LastAccessTime.UtcDateTime;
        remoteStorageItem.LastWriteTimeUtc = fileMetadata.LastWriteTime.UtcDateTime;

        // Return your remote storage item ID. 
        // It will be passed later into IEngine.GetFileSystemItemAsync() method.
        return WindowsFileSystemItem.GetItemIdByPath(remoteStorageItem.FullName);
    }

    public async Task<byte[]> CreateFolderAsync(IFolderMetadata folderMetadata)
    {
        DirectoryInfo remoteStorageItem = 
            new DirectoryInfo(Path.Combine(RemoteStoragePath, folderMetadata.Name));
        remoteStorageItem.Create();

        // Set remote storage folder metadata.
        remoteStorageItem.Attributes = folderMetadata.Attributes;
        remoteStorageItem.CreationTimeUtc = folderMetadata.CreationTime.UtcDateTime;
        remoteStorageItem.LastWriteTimeUtc = folderMetadata.LastWriteTime.UtcDateTime;
        remoteStorageItem.LastAccessTimeUtc = folderMetadata.LastAccessTime.UtcDateTime;
        remoteStorageItem.LastWriteTimeUtc = folderMetadata.LastWriteTime.UtcDateTime;

        // Return remote storage item ID. 
        // It will be passed later into IEngine.GetFileSystemItemAsync() method.
        return WindowsFileSystemItem.GetItemIdByPath(remoteStorageItem.FullName);
    }
}

If the file content is null, you will typically create a 0-length file in your remote storage and will upload content at a later time when the file handle is released.

You will return the new item ID from IFolder.CreateFileAsync() or IFolder.CreateFolderAsync() methods. This ID will then be passed to the IEngine.GetFileSystemItemAsync() method during every file system operation.

Detecting New Items

When a new file or folder is being created in the user file system on the Windows platform it is created as a regular file or folder. The file or folder is being converted to a placeholder only after the successful IFolder.CreateFileAsync() / IFolder.CreateFolderAsync() method call if the method completes without exceptions. Otherwise, the file or folder remains a regular file/folder.

If for any reason a file or folder failed to create inside the CreateFileAsync() CreateFolderAsync() method call, you can detect such files by checking if a file or folder is a placeholder by using the PlaceholderItem.IsPlaceholder() static method:

bool isNew = PlaceholderItem.IsPlaceholder("@C:\Users\User1\VFS\file.ext");

Note that detection of new items described here will work only if your files are not being updated using transactional save operation, performed by some applications, such as MS Office.

Detecting Modified Items

When a file or folder is modified in the user file system it is marked as not in-sync. In Windows Explorer such items are marked with arrows icon: Not in sync cloud files icon. The item is marked as in-sync after the successful call to IFile.WriteAsync() or IFolder.WriteAsync() method call if the method completes without exceptions. Otherwise, the file or folder remains in the not in-sync state.

You can detect modified files using the PlaceholderItem.GetInSync() method call:

string userFileSystemPath = "@C:\Users\User1\VFS\file.ext";
if( PlaceholderItem.IsPlaceholder(userFileSystemPath) )
{
    bool isInSync = PlaceholderItem.GetItem(userFileSystemPath).GetInSync();
}

In the case of files, the not in-sync files can have modified metadata (creation date, modification date, attributes), modified content, or both. The item is also marked as not in-sync after the rename or move operation. To detect if the file content was modified you can use the PlaceholderFile.GetFileDataSizeInfo() method call:

string userFileSystemPath = "@C:\Users\User1\VFS\file.ext";

PlaceholderFile placeholderFile = PlaceholderFile(userFileSystemPath);
long bytesModified = placeholderFile.GetFileDataSizeInfo().ModifiedDataSize;

To apply changes made in your remote storage in user file system, you will either periodically pull your server for changes or implement push notifications from your remote storage to the client machines (via web sockets or similar technologies). In many cases, you will implement both approaches. In both cases, you will use the IServerNotifications interface to apply changes received from your remote storage into the user file system.

To get the object implementing IServerNotifications interface you will call the IEngine.ServerNotifications() method passing the user file system path. 

Because of the on-demand population, the folder placeholder, in which new items are created, or the placeholder that is being updated, deleted or moved, may not exist in the user file system or maybe offline. In this case the changes made via IServerNotifications interface are ignored and the metod call returns a value indicates that no changes were applied. Typiclly this return value is useful for logging and testing purposes.

Below we will show how to do this for new items, updated, deleted and moved items.

To create files and folders in the user file system you will call the IServerNotifications.CreateAsync() method passing the list of items metadata to be created:

IFileSystemItemMetadata item = Mapping.GetUserFileSysteItemMetadata("/Folder/file.ext");
IServerNotifications sn = engine.ServerNotifications("@C:\Users\User1\VFS\Folder\");
if (await sn.CreateAsync(new[] { item }) > 0)
{
    LogMessage($"Created succesefully");
}

In case the parent folder placeholder does not exist or is offline the IServerNotifications.CreateAsync() call will not create any items and will return 0.

To update a file or folder, call the IServerNotifications.UpdateAsync() method:

IFileSystemItemMetadata item = Mapping.GetUserFileSysteItemMetadata("/file.ext");
IServerNotifications sn = engine.ServerNotifications("@C:\Users\User1\VFS\file.ext");
if (await sn.UpdateAsync(item))
{
    LogMessage("Updated succesefully");
}

To delete a file or folder call the IServerNotifications.DeleteAsync() method:

IServerNotifications sn = engine.ServerNotifications("@C:\Users\User1\VFS\file.ext");
if (await sn.DeleteAsync())
{
    LogMessage("Deleted succesefully");
}

To move or rename a file or folder call IServerNotifications.MoveToAsync() method:

IServerNotifications sn = engine.ServerNotifications("@C:\Users\User1\VFS\file.ext");
if (await sn.MoveToAsync("@C:\Users\User1\VFS\Folder\file.ext"))
{
    LogMessage("Renamed or moved succesefully");
}

 

Managing Timeout on Windows

On Windows, the following methods have a 60 sec timeout: IFolder.GetChildrenAsync()IFile.ReadAsync()IFileSystemItem.DeleteAsync()IFileSystemItem.DeleteCompletedAsync(), IFileSystemItem.MoveToAsync() and IFileSystemItem.MoveToCompletionAsync(). To reset the timput you can call the IResultContext.ReportProgress() method. However, in case of the IFolder.GetChildrenAsync() method it will only reset the timer, the ReportProgress() call has no impact on progress display in Windows File Manager.

Inside the IFolder.GetChildrenAsync() method the IFolderListingResultContext.ReturnChildren() call will also reset the timeout.

In the IFile.ReadAsync() method the timeout is reset each time you write data to the output stream. However, the timeout is reset only if the stream is advanced by 4K or more. Otherwise, the content is buffered and the stream waits for more data to be written.

 

Next Article:

Cloud Files API Frequently Asked Questions