My Animal Mart (Part 2) - The data config problem.

Foreword

My studio has its own data tool that uses CSV files converted into Scriptable Objects. Some teams refuse to use this and create their own solutions, such as using text files, including them with the build, and reading the data at the first loading screen, among other methods. There is no unified solution for data configuration.

The Background Story

After working on many projects using different tools for data configuration, and after my last game, StarWarrior, where I used CSV files, I decided to find a powerful tool that allows game designers and developers to easily interact with, maintain, update, and delete data. I wanted to eliminate the confusion over which solution to use for future projects.

Using CSV files has several drawbacks: no formulas, no comments (unless you create a comment column), no graphs, no charts, and if you use languages with diacritics like my native Vietnamese, you might encounter formatting errors.

Examples of Bad Cases with CSV:

1. Format Error:
   

2. Naming Conventions:
   JumpAttack1, JumpAttack2... and if there are more attacks? JumpAttackN? Despite having the same skill name and buff name.
   

3. Character Data:
   

4. Another Example:
   

These data issues make it confusing and difficult for game designers and developers to maintain.

The Technique I Used

To address these issues, I decided to replace the CSV tool with a better solution.

Fortunately, I discovered BakingSheet: BakingSheet. Please give it a star; it deserves that.

The way our GD design the excel is easy for reading and help developer understands.

Introducing BakingSheet:

BakingSheet allows you to convert from Excel (yes, real Excel, with formulas, graphs, descriptions, and no formatting errors) to Unity ScriptableObjects.

Here are some examples:

As you can see, it has child ScriptableObjects under the parent, read-only values, and options for automated conversion. These features are what I like.

What I want to change:

• Making read-only values can make debugging difficult. Any changes from the game designer must be done in Excel.
• It does not support dictionaries.
• It's hard to customize the drawing inspector, as it uses UIElements for drawing.

Modifications I Made:

To better suit my team's needs, I modified BakingSheet:

• ScriptableObject values are not necessarily marked as read-only. Developers can choose to make them read-only or not.
• It supports dictionaries, allowing any dictionary serialization.
• I created a custom flow for generating ScriptableObjects instead of using the example auto-generate method. Game designers can update the data they want.
• I developed a ScriptableObject hierarchy that can be created, updated, and deleted.

For example:

Example code:

First off, I'm using Odin Inspector to quickly create a custom inspector. But you can use other free tools for that.
Let's begin with the SheetContainer. In here we have all the sheets we want to have.
'Sheet' here means the sheet in Excel or Google Sheet.

For example, this excel has 2 sheets: EmployeeUpgrade and MasterProductConfigSheet.

Here is the SheetContainer.

using Cathei.BakingSheet;
using Cathei.BakingSheet.Unity;
using System.Collections;
using System.Collections.Generic;
using UnityEngine;

namespace Test_CSV
{
    public class SheetContainer : SheetContainerBase
    {
        public SheetContainer() : base(UnityLogger.Default) { }

        // property name matches with corresponding sheet name
        // for .xlsx or google sheet, it is name of the sheet tab in the workbook
        // for .csv or .json, it is name of the file

        // add other sheets as you extend your project
        //public CharacterSheet Characters { get; private set; }
        public MasterProductConfigSheet MasterProductConfigSheet { get; private set; }
    }
}

MasterProductConfigSheet

using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using Cathei.BakingSheet;

namespace Test_CSV
{
    public class MasterProductConfigSheet : Sheet<MasterProductConfigSheet.Row>
    {
        public class Row : SheetRow
        {
            // use name of matching column
            public int SalePrice { get; private set; }
        }
    }
}

ExcelPostprocess

using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using Cathei.BakingSheet.Unity;
using Cathei.BakingSheet;
using System;
using System.IO;
using Sirenix.OdinInspector;
using System.Threading.Tasks;
using Sirenix.Serialization;
using NonSerializedAttribute = System.NonSerializedAttribute;
using Cathei.BakingSheet.Internal;

#if UNITY_EDITOR
using UnityEditor;
#endif

namespace Test_CSV
{
#if UNITY_EDITOR
    public class ExcelPostprocess : SerializedMonoBehaviour
    {
        [OdinSerialize, NonSerialized]
        public SheetInfo[] sheetInfos;

        public class ExportSheetEditor
        {
            [Sirenix.OdinInspector.FilePath]
            [ShowInInspector]
            public static string exportPrefabPath = "Assets/Config/Excels/ExportSheet.prefab";

            [MenuItem("Test/Export map sheet data. #&w")]
            static async void ExportMapSheetData()
            {
                var exportSheet = AssetDatabase.LoadAssetAtPath<ExcelPostprocess>(exportPrefabPath);

                foreach (var sheetInfo in exportSheet.sheetInfos)
                {
                    foreach (var config in sheetInfo.dataConfigs)
                    {
                        await Export(config, sheetInfo.excelPath);
                    }
                }
            }

            public static async Task Export(IExcelImportable config, string upgradeExcelPath)
            {
                var sheetContainer = new SheetContainer();
                var excelPath = Path.GetDirectoryName(upgradeExcelPath);

                bool hasPath = File.Exists(upgradeExcelPath);

                // create excel converter from path
                var excelConverter = new ExcelSheetConverter(excelPath, TimeZoneInfo.Utc);

                // bake sheets from excel converter
                await sheetContainer.Bake(excelConverter);

                // (optional) verify that data is correct
                sheetContainer.Verify(
#if BAKINGSHEET_ADDRESSABLES
                    new AddressablePathVerifier(),
#endif
                    new ResourcePathVerifier()
                );

                UpdateSO(config, sheetContainer);
            }

            static async void UpdateSO(IExcelImportable excelImportConfig, SheetContainer sheetContainer)
            {
                switch (excelImportConfig)
                {
                    case MasterProductConfig _:
                        {
                            var config = excelImportConfig as MasterProductConfig;
                            foreach (var row in sheetContainer.MasterProductConfigSheet)
                                excelImportConfig.ImportDataFromExcel(row);
                            config.CheckForDeleteConfigInExcel();
                            break;
                        }
                }

                AssetDatabase.SaveAssets();
                AssetDatabase.Refresh();
            }

            [MenuItem("Test/Open persistent path.")]
            static void OpenPersistentPath()
            {
                Application.OpenURL(Application.persistentDataPath);
            }

            [MenuItem("Test/Clear persistent path.")]
            static void ClearPersistentPath()
            {
                System.IO.DirectoryInfo di = new DirectoryInfo(Application.persistentDataPath);

                foreach (FileInfo file in di.GetFiles())
                {
                    file.Delete();
                }
                foreach (DirectoryInfo dir in di.GetDirectories())
                {
                    dir.Delete(true);
                }
            }
        }

        [Serializable]
        public class SheetInfo
        {
            public string excelPath;
            [ListDrawerSettings]
            public List<IExcelImportable> dataConfigs;

            [Button]
            public async void ImportSheetData()
            {
                var exportSheet = AssetDatabase.LoadAssetAtPath<ExcelPostprocess>(ExportSheetEditor.exportPrefabPath);

                foreach (var config in dataConfigs)
                {
                    await ExportSheetEditor.Export(config, excelPath);
                }
            }
        }
#endif
    }
}

using Cathei.BakingSheet;
using Cathei.BakingSheet.Unity;
using Newtonsoft.Json;
using System;
using System.Collections;
using System.Collections.Generic;
using Sirenix.OdinInspector;
using Sirenix.Serialization;

#if UNITY_EDITOR
using UnityEditor;
#endif
using UnityEngine;
using Ultility;

namespace Test_CSV
{
    [CreateAssetMenu(fileName = "MasterProductConfig", menuName = "Config/MasterProductConfig")]
    public class MasterProductConfig : SerializedScriptableObject, IExcelImportable
    {
        [Title("Data section:")]


 [ReadOnly]
        public Dictionary<string, MasterProductConfigData> configDict;

        [System.NonSerialized]
        public const string exportedConfigPath = "Assets/Resources/Configs/MasterProductConfig";

        public void ImportDataFromExcel(SheetRow row)
        {
            var data = row as MasterProductConfigSheet.Row;
            var config = new MasterProductConfigData(data.Id, data.SalePrice);
            configDict[data.Id] = config;
        }

        public void CheckForDeleteConfigInExcel()
        {
            var tempConfigDict = new Dictionary<string, MasterProductConfigData>();

            foreach (var row in configDict)
            {
                if (configDict.ContainsKey(row.Key))
                    tempConfigDict.Add(row.Key, row.Value);
            }

            configDict = tempConfigDict;
        }

        [Button("Save as JSON")]
        public void SaveAsJson()
        {
            var content = Sirenix.Serialization.SerializationUtility.SerializeValue(configDict, DataFormat.JSON);
            var path = Path.Combine(exportedConfigPath, $"{name}.json");

            if (!Directory.Exists(exportedConfigPath))
                Directory.CreateDirectory(exportedConfigPath);

            File.WriteAllBytes(path, content);
        }

#if UNITY_EDITOR
        [Button("Load from JSON")]
        public void LoadFromJson()
        {
            var path = Path.Combine(exportedConfigPath, $"{name}.json");

            if (File.Exists(path))
            {
                var content = File.ReadAllBytes(path);
                configDict = Sirenix.Serialization.SerializationUtility.DeserializeValue<Dictionary<string, MasterProductConfigData>>(content, DataFormat.JSON);
            }
        }
#endif
    }
}

What is happening:

• We created a SheetContainer, which contains all our sheets.
• We created a MasterProductConfigSheet, which defines our Excel data.
• We created a MasterProductConfig ScriptableObject to hold the data.
• We created an ExcelPostprocess to manage the import/export of data.

This solution, while more complex than using CSV files directly, provides significant benefits:

• Real Excel files with full support for formulas, graphs, and formatting.
• Robust handling of read-only values and dictionaries.
• Custom inspector drawing, allowing game designers to update data easily.

Conclusion

Switching from CSV to BakingSheet has greatly improved our data management. We no longer face formatting issues or naming convention problems, and the custom inspector makes data updates straightforward for our game designers. This approach has streamlined our workflow, and I hope it can do the same for you.