Backup Service Fabric Stateful Service Data into ADL

In this blog post, we will learn about how to restore the stateful service data from the Azure Data Lake. You might be getting a doubt about why do we need to maintain a backup for the reliable service fabric stateful service data. There might be few scenarios when we might need to back up the data.

Few scenarios are like

• If we need to do any maintenance activity as Service Fabric Cluster upgrade.
• If there is any change in the schema of the state object.

Let's see with a simple example. I will be using the Visual Studio to create the sample Service Fabric application.

Prerequisites

1. Should have an active Azure subscription and configured an Azure Data Lake(ADL) storage service to restore the data into it.

   For more info on ADL in Azure Azure DataLake

Demo

1. Create the Sample Service Fabric application in Visual Studio

• Select Stateless Service Fabric application
• Select the API Project
• Once created, Right-click on Service Fabric project and select add Service Fabric project and select Stateful Service.

2. Create a Timer Adapter to inject into the Stateful service

• Interface

    public interface ITimerAdapter
    {
        void Start();
        public double Interval { get; set; }
        event ElapsedEventHandler Elapsed;
        void Stop();
    }

• Implementation class

    public class TimerAdapter : Timer, ITimerAdapter
    {
    }

3. Now we need to create the helper to to interact with ADL.

• This helper will have following methods

  • CreateFileSystem - Used to create the file system inside the ADL container.

          public async Task CreateFileSystem()
        {
            try
            {
                DataLakeServiceClient serviceClient = GetDataLakeServiceClient();

                await serviceClient.CreateFileSystemAsync("file-system-name");
            }
            catch (RequestFailedException ex)
            {
                if (ex.Status != 409)
                    throw;
            }
        }

• UploadFile - Used to upload the file which we wanted to upload into the ADL.

          public async Task UploadFile(byte[] content, string directory, string fileName)
        {
            DataLakeFileSystemClient fileSystemClient = new DataLakeFileSystemClient(
                new Uri("<your ADL URI>/file-system-name"),
                new StorageSharedKeyCredential(_accountName, _accountKey));

            DataLakeDirectoryClient directoryClient = fileSystemClient.GetDirectoryClient(directory);
            DataLakeFileClient fileClient = await directoryClient.CreateFileAsync(fileName);

            MemoryStream ms = new MemoryStream(content);

            await fileClient.CreateAsync();

            if (content.Length > 0)
            {
                await fileClient.AppendAsync(ms, 0);
                await fileClient.FlushAsync(ms.Length);
            }
        }

• DownloadFile - Used to download the file which we backed up in ADL and restore into state.

          public async Task<byte[]> DownloadFile(string directoryName, string fileName)
        {
            DataLakeFileSystemClient fileSystemClient = new DataLakeFileSystemClient(
                new Uri("<URI of ADL>/my-file-system"),
                new StorageSharedKeyCredential(_accountName, _accountKey));

            DataLakeDirectoryClient directoryClient =  fileSystemClient.GetDirectoryClient(directoryName);

            DataLakeFileClient fileClient = directoryClient.GetFileClient(fileName);

            Response<FileDownloadInfo> downloadResponse = await fileClient.ReadAsync();

            MemoryStream ms = new MemoryStream();
            downloadResponse.Value.Content.CopyTo(ms);

            return ms.ToArray();
        }

For more information on how to create this helper you can refer this link .NET to manage directories, files, and ACLs in Azure Data Lake Storage Gen2

4. Inject both TImer Adapter and DataLake Helper into the Stateful service class

• Inject TImer Adapter and DataLake Helper into the constructor of the stateful service class.

        private readonly ITimerAdapter _timer;
        private readonly IDataLakeHelper _dataLakeHelper;
        public Stateful1(StatefulServiceContext context, ITimerAdapter timerAdapter, IDataLakeHelper dataLakeHelper)
            : base(context)
        {
            _dataLakeHelper = dataLakeHelper ?? new DataLakeHelper();
            _timer = timerAdapter ?? new TimerAdapter();
        }

• The StartBackup callback method will look like below

        private void StartBackup(object sender, System.Timers.ElapsedEventArgs e)
        {
            var emp = new Employee()
            {
                Id = Guid.NewGuid(),
                Name = "Backup Test",
                Email = "[e-mail]"
            };

            //Convert the object into byte[].
            byte[] bytes;
            var bf = new BinaryFormatter();
            using (var ms = new MemoryStream())
            {
                bf.Serialize(ms, emp);
                bytes = ms.ToArray();
            }

            //First we need to create the FileSystem
            _dataLakeHelper.CreateFileSystem();

            //Now upload the File into that file system.
            _dataLakeHelper.UploadFile(bytes, "Employee", "Employee.txt");
        }

• The Restore method will look like

        private async Task Restore()
        {
            const string directoryName = "Employee";
            const string fileName = "Employee.txt";

            //Download the file which we uploaded as backup inside the directory "Employee" with a name "Employee.txt"
            var bytes = _dataLakeHelper.DownloadFile(directoryName, fileName).Result;

            // Deserialize the bytes into the Employee Object.
            Employee emp;
            var bf = new BinaryFormatter();
            await using (var ms = new MemoryStream(bytes))
            {
                emp = (Employee) bf.Deserialize(ms);

            }

            //Restore into the reliable dictionary.
            var dictionary = await StateManager.GetOrAddAsync<IReliableDictionary<string, Employee>>("Employee");

            using var tx = StateManager.CreateTransaction();
            await dictionary.AddOrUpdateAsync(tx, emp.Id.ToString(), emp, (s, d) => emp);

            await tx.CommitAsync();
        }

• Use RunAsync Method of the Stateful service to register the Timer. Also start the timer and create a event handler to call when the timer elapsed. Also use the Restore to restore into the state.

        protected override async Task RunAsync(CancellationToken cancellationToken)
        {
            await Restore();

            _timer.Interval = 3000; //This is in MillSeconds
            _timer.Start();
            _timer.Elapsed += StartBackup;
        }