C# Class design for cloud storage management application

Summary

I want to share a C# class design pattern I recently did. I know there are a lot of different design patterns and there is no single best way, but learning different design patterns helps myself develop extensible and scalable applications.

Sample code: blob-console-app

Architecture

Operators place files which they want to upload to Azure Blob Storage.
The .NET console app checks the pointed directory in a certain interval and if those files are valid, it uploads to the Azure Blob Storage.
The app deletes those files from the local file system once the files are uploaded.
The app extracts metadata and attaches the blob on cloud.
Currently the file types are logs and events which are located different directory in the local file system.

Class design

I decided to use Interface, Abstract, and sub classes because:

Currently file types are only Logs and Events, but the system will extend. For example, DeviceInfo and Requests will be added in the future.
It is better for future extended file types to divide the base features and ones with different requirements for each file type so the system reuses the base class features in the future.
In order to avoid changing the base class when adding new file types, an Interface class and abstract class defines base properties, and sub classes define metadata properties.
FolderName is a base property but it points to the directory for each file type. It is defined as Abstract property because Sub classes must override it and define for their own.
JudgeValidFile() and GetMetadata() have to be implemented in sub classes, and then they are defined as Abstract. UploadBlobAsync(), UploadMetadataAsync(), and DeleteFiles() are defined as Abstract because they are implemented in the base class but can be overriden in sub classes accordingly.

Codes

List file type instance

Currently the system has only two file types, Logs and Events. If you want to add another file type, only thing to do is to implement a new class for the file type and add a new instance here.

Program.cs

List<IUploadData> uploadDataList = new ()
{
    new Logs(),
    new Events(),
};

for instead of foreach

for (int i = 0; i < uploadDataList.Count; ++i) processes differently depending on class instance added to the list.
It uses for instead of foreach because the class method GetMetadata() gets metadata and put into its own property

Program.cs

uploadDataList[i] = uploadDataList[i].GetMetadata();

Dependency Injection

It does not use dependency injection but assigns the values to properties. This is because the value changes for different files found and file types. For example, BlobClient needs a file name and directory to instantiate for different files and different file types. It cannot create an instance in constructor.

Program.cs

List<IUploadData> uploadDataList = new()
{
    new Logs(),
    new Events(),
};

while (true)
{
    for (int i = 0; i < uploadDataList.Count; ++i)
    {
        uploadDataList[i].FileFullPaths = Directory.GetFiles(Path.Combine(localPath, uploadDataList[i].FolderName), "*", SearchOption.AllDirectories);

        foreach (string fileFullPath in uploadDataList[i].FileFullPaths)
        {
            uploadDataList[i].FileFullPath = fileFullPath;
            uploadDataList[i].FileName = Path.GetFileName(fileFullPath);
            uploadDataList[i].BlobClient = containerClient.GetBlobClient($"/{uploadDataList[i].FolderName}/{DateTime.Now.Year}/{DateTime.Now.Month}/{uploadDataList[i].FileName}");

Upload file validation

If a file found in the directory is not valid to upload, it stops the for loop and goes to a next file.
How to valid files is different depending on file types. That is why JudgeValidFile() is overriden in each file type class.

Program.cs

uploadDataList[i].JudgeValidFile();
if (uploadDataList[i].IsValidFile is false) continue;

Logs.cs

this.IsValidFile = (Path.GetExtension(this.FileName) == ".csv" && this.FileFullPaths.Contains(this.FileFullPath + ".json"));

Events.cs

this.IsValidFile = (Path.GetExtension(this.FileName) == ".json");

Generic class

I wanted to use a generic class for UploadMetadataAsync() because it has to retrieve the properties dynamically depending on the sub class. I thought UploadMetadataAsync<uploadDataList[i].GetType()>() works but actually not. After investigating sometime, I gave up using Generic class here.

Generic class

public virtual async Task UploadMetadataAsync<T>() where T : IUploadData
{
    PropertyInfo[] propertyCollection = typeof(T).GetProperties();
    foreach (PropertyInfo property in propertyCollection)
    {
        blobMetadata[property.Name.ToString()] = property.GetValue(this).ToString();
    }
}

Caller

await uploadDataList[i].UploadMetadataAsync<uploadDataList[i].GetType()>().ConfigureAwait(true);

Metadata into dictionary

I realized I cannot use Generic class, and I was thinking how to insert only metadata properties into the dictionary for uploading. What I did is to see the difference of property between Abstract class and sub class. For example, the Abstract class UploadData has only the base properties FolderName, IsValidFile, FileFullPaths, FileFullPath, FileName, BlobClient, but the sub class Logs has the metadata properties BeginTime, EndTime, Temperature, Humidity, Location in addition to the base properties.

UploadData.cs

this.blobMetadata.Clear();
PropertyInfo[] propertiesIUploadData = typeof(IUploadData).GetProperties();
IEnumerable<PropertyInfo> propertiesThis = this.GetType().GetRuntimeProperties();

foreach (PropertyInfo propertyThis in propertiesThis)
{
    int count = 0;
    foreach (PropertyInfo propertyIUploadData in propertiesIUploadData)
    {
        if (propertyThis.Name == propertyIUploadData.Name) ++count;
    }
    if (count == 0)
    {
        this.blobMetadata[propertyThis.Name.ToString()] = propertyThis.GetValue(this).ToString();
    }
}
await this.BlobClient.SetMetadataAsync(this.blobMetadata).ConfigureAwait(false);

Json deserialize

It is elegant to deserialize the value from a json file because I do not need to change the code JsonSerializer.Deserialize<Logs>(jsonString)!; for any model property. However, the problem was that the sub class has both the base property and metadata property, and when in deserialization, it sets the metadata properties with the value from a json file and also the base properties with null.
I do not like this, but I did not come up with an elegant workaround, and what I implemented is to keep the value to local variables and then set them back after deserialization.

json file example

{
  "BeginTime": "2022-03-07T12:21:55Z",
  "EndTime": "2022-03-07T14:01:36Z",
  "Temperature": 25,
  "Humidity": 63,
  "Location": "Tokyo"
}

Logs class property

public string FolderName { get; }
public bool IsValidFile { get; set; }
public string[] FileFullPaths { get; set; }
public string FileFullPath { get; set; }
public string FileName { get; set; }
public BlobClient BlobClient { get; set; }
public DateTime BeginTime { get; set; }
public DateTime EndTime { get; set; }
public int Temperature { get; set; }
public int Humidity { get; set; }
public string Location { get; set; }

Logs.cs

public override IUploadData GetMetadata()
{
    bool IsValidFile = this.IsValidFile;
    string[] FileFullPaths = this.FileFullPaths;
    string FileFullPath = this.FileFullPath;
    string FileName = this.FileName;
    BlobClient BlobClient = this.BlobClient;

    string jsonString = File.ReadAllText(this.FileFullPath + ".json");
    Logs log = JsonSerializer.Deserialize<Logs>(jsonString)!;

    log.IsValidFile = IsValidFile;
    log.FileFullPaths = FileFullPaths;
    log.FileFullPath = FileFullPath;
    log.FileName = FileName;
    log.BlobClient = BlobClient;

    return log;
}