Virtual File System Sample in .NET, C#
This sample implements a virtual file system with synchronization support, on-demand loading, selective offline files support, upload and download progress, and error reporting. It synchronizes files and folders both from remote storage to the user file system and from the user file system to remote storage. It is written in .NET Core, C#.
To simulate the remote storage, this sample is using a folder in the local file system on the same machine.
The purpose of this sample is to demonstrate the major features of the IT Hit User File System for .NET and provide patterns for its programming. You will use this sample as a starting point for creating an OneDrive-like file system for your DMS/CRM/ERP and will reprogram it to publish data from your real storage instead of the local file system.
- .NET 5.0 or later.
- Microsoft Windows 10 Creators Update or later version.
- NTFS file system.
Configuring the Sample
By default, the sample will use the \RemoteStorage\ folder, located under the project root, to simulate the remote storage file structure. It will mount the user file system under the %USERPROFILE%\VFS\ folder (typically C:\Users\<username>\VFS\).
To specify the folder that will be used for remote storage simulation edit the "RemoteStorageRootPath" parameter in appsettings.json. This could be either an absolute path or a path relative to the application root.
To specify the user file system folder edit the "UserFileSystemRootPath" parameter in appsettings.json.
Setting the License
To run the example, you will need a valid IT Hit User File System Engine for .NET License. You can download the license in the product download area. Note that the Engine is fully functional with a trial license and does not have any limitations. The trial license is valid for one month and the engine will stop working after this. You can check the expiration date inside the license file. Download the license file and specify it's content in License field in appsettings.json file.
You can also run the sample without explicitly specifying a license for 5 days. In this case, the Engine will automatically request the trial license from the IT Hit website https://www.userfilesystem.com. Make sure it is accessible via firewalls if any. After 5 days the Engine will stop working. To extend the trial period you will need to download a license in a product download area and specify it in appsettings.json
Running the Sample
To run the sample open the project in Visual Studio and run the project in a debug mode. In the debug mode this sample provides additional support for the development and testing convenience. When starting in the debug mode, it will automatically create a folder in which the virtual file system will reside, register the user file system with the platform and then open two instances of Windows File Manager, one of which will show a user file system and another a folder simulating remote storage:
You can start managing and editing files in the \RemoteStorage\ folder immediately after the Windows File Managers open and will see all changes being propagated to the user file system. You can also edit documents and manage file structure in the user file system and all changes will automatically appear in the remote storage folder.
This sample implements on-demand loading. After running the sample all files and folders are marked with a cloud icon, which means that the content of this file or folder is not available locally but instead resides in the remote location. Even though the file shows the correct size in the Size column, the file occupies zero bytes on the disk. You can open the file Properties dialog to check the "Size on disk" value. You can see in the console log that only root folder files and folders placeholders are being created when Windows File Manager listed the root folder content. Folders located deeper in the hierarchy are not loaded, until their content is being requested by the platform file system calls.
When any application is accessing the file, located under the \VFS\ folder, opening it for reading or writing, the operating system redirects the call to this sample, which loads file content from the remote storage folder. The file becomes marked with a green check-mark on a white background icon, which means the file content is present on the local disk:
The Windows File Manager provides the "Always keep on this device" and "Free up space" context menus, which are standard menus provided by Windows OS. If you select the "Always keep on this device" the file or entire folder structure will be recursively loaded to the local disk and all files content will be loaded to local disk and will become available offline. All files and folders are be marked with a pinned file icon. Pinned files will not be deleted from the drive even if it runs low on space.
To remove content from the local disk select the "Free up space" menu. It will restore the cloud icon.
When a large file is being downloaded from the remote storage, this sample submits progress reports to the operation system, to show a standard Windows "Downloading" dialog. At the same time, the Windows File Manager also shows progress in the files list view:
Microsoft Office Documents Synchronization
This sample supports the synchronization of the MS Office documents, avoiding the creation of the temporary files in remote storage. Note that for the sake of simplicity this sample filters MS Office temporary files only during user file system to remote storage synchronization. It is expected that MS Office temporary files are not created in remote storage and you do not need to synchronize them from server to client. Do NOT edit MS Office files directly in the folder simulating the remote storage.
Stopping the Sample
You can exit the sample running in the debug mode in 3 ways:
- If you press 'Q' the user file system will be unregistered and all files will be deleted. This simulates uninstall with complete cleanup.
- If you press 'q' the user file system will be unregistered. All files and folders downloaded on disk will be converted into regular files and folders. The offline files and folders will disappear. This simulates your application's uninstallation process.
- If you press any other key, the application will exit without unregistering the user file system. All files and folders placeholders, their attributes, and states remain in the user file system, even though most of the functionality, such as downloading file content or listing folder content for offline folders are unavailable. You can start this sample again to continue managing documents. This simulates the machine reboot or application failure.
How the Sample Works
Initially, when you start the application, the user file system does not contain any file of folder placeholders, except the sync root folder. The content of the folders is populated only when any application is listing folder content. The content of files is loaded only when an application is opening a file for reading or writing.
Platform File System Operations Handling
Folder content listing, file reads, move/rename and delete is performed by the VfsEngine and VfsFile and VfsFolder classes.
The initial folder content listing is done inside VfsFolder.GetChildrenAsync() method call. When any application is listing folder content, the VfsEngine class creates the VfsFolder class instance that corresponds to the requested folder. Then the Engine calls GetChildrenAsync() method.
As soon as the folder content is populated with placeholders during the first access, the GetChildrenAsync() is never called again for this particular folder. Instead, to update the folder content, this sample monitors changes in the remote storage and applies them to the user file system. See the "Remote Storage to User File System Synchronization" section below.
Hydration/Dehydration via Windows File Manager Commands
When the user selects "Always keep on this device" or the "Free up space" in the Windows file manager, the file is marked with a Pinned or Unpinned attribute respectively. The hydration/dehydration itself is performed later when the UserFileSystemMonitor class detects attribute changes in the user file system or during the synchronization process by the SyncService class.
Remote Storage to User File System Synchronization
The remote storage is simulated by the dedicated folder in the file system (\RemoteStorage\ by default). The sample monitors changes in it using RemoteStorageMonitor class that generates events and applies corresponding changes in the user file system. The RemoteStorageMonitor class is using FileSystemWatcher which can monitor the creation, update of file content and attributes, deletion, and rename. The FileSystemWatcher class can NOT monitor the move event. In your real-life application, you will replace the FileSystemWatcher with web sockets or any other technology that will send events from your server to all connected clients.
User File System to Remote Storage Synchronization
The user file system to remote storage synchronization is performed by VfsFile, VfsFileSystemItem, and UserFileSystemMonitor classes:
VfsFile.Close() - This method is called by the Engine when the application releases the file handle. The code in this method uploads the file to the remote storage when the file is closed by the application. Note that when this method is called, the file may be still blocked by other processes, so upload may often fail, which is normal behavior. The file will be uploaded during the last Close() method call or during the synchronization process.
VfsFileSystemItem.MoveToAsync() and VfsFileSystemItem.DeleteAsync() - In this methods files and folders are moved and deleted in the remote storage.
UserFileSystemMonitor.CreatedAsync() - In this method, new files and folders are created. Note that files and folders are created as regular files/folders, in this method they are converted into placeholders.
Full File System Synchronization
The complete synchronization is performed in the SyncService class. Due to the nature of this sample, this class goes folder-by-folder to synchronize the entire file system. This class first processes moved/renamed files and folders and then creates, deletes, and updates files and folders.
Because of the simplicity of this sample, the files deleted in the user file system, then this Virtual File System sample application was not running, or during network or server outage, are not synched to remote storage (they are normally deleted inside VfsFileSystemItem.DeleteAsync() call). Instead, the file will be restored in the user file system when the synchronization service runs.
You can adapt the SyncService class implementation to your needs depending on your remote storage type.
When listing folder content the client reads file ETags and saves them in local file system. When sending modified content to the server, the client attaches the saved etag to the request. The server compares etags to ensure the file was not modified on the server since the file was read by the client. In the case of Microsoft Office documents, etags provide one more level of protection against overwriting server changes in addition to locks.
Because of the nature of the remote storage simulation, this sample is using file/folder last modification date/time as an ETag. In your real-life storage, you will update ETag during each file modification in your storage and will NOT use the file/folder modification date.
In-Sync Status and ETag Usage For Synchronization
Synchronization in this sample is based on In-Sync file status as well as on the ETag received from the server.
The server to client synchronization is performed only if the file on the client is marked as In-Sync with the server (it is marked with or or icon). If the file is modified on the server and is marked as not In-Sync on the client, the files are considered to be in conflict and the conflict icon is displayed.
When any file or folder on the client is updated, it is marked as not In-Sync, which means the content must be sent to the server. When the updated file is being sent from client to server, the server compares the ETag sent from the client with the ETag stored on the server, to avoid the server changes to be overwritten. The file/folder is updated only if the ETags match. Otherwise, the file/folder is considered to be in conflict and marked with the conflict icon.