On May 29th, the core system of a bank in Yunnan officially went live. Unlike the majority of banks that typically use Oracle for their core systems, the Bank chose to adopt the domestically developed distributed database, OceanBase.
With the new core system going live, Hu Zhonghao, an engineer at the Bank’s IT Operations Center, gained a new title — a Domestic Distributed Database DBA (Database Administrator). Hu Zhonghao was involved in the entire process of building the Bank’s new core system database, from early market research, mid-stage POC and tendering, to later deployment and launch. As a result, he became one of the few DBAs in the banking system to have a comprehensive understanding of domestic distributed databases. Although the process involved many challenges and hardships, Hu Zhonghao believes it was worthwhile. These experiences transformed him from an Oracle DBA into a domestic distributed database DBA.
Entering the Field
Hu Zhonghao graduated from North China Institute of Science and Technology in 2014 with a major in Geographic Information Systems. Although he studied some database knowledge in college, including Microsoft SQL Server and the ArcCatalog geographic database, he wasn’t particularly interested in databases before his internship in his senior year, nor did he ever imagine he would become a DBA. It was during his last internship that Hu Zhonghao started to consider the idea of becoming a DBA.
“In my senior year, I interned at a company in Beijing, and one of the engineers was responsible for Oracle technical support. I was deeply impressed by him. At that time, he seemed incredibly skilled — many problems could only be solved by him, and everyone respected him. You could tell he took pride in being an Oracle DBA. I wanted to be like him, so I decided to become an Oracle DBA,” Hu Zhonghao recalled.
Coincidentally, Hu Zhonghao had a friend who worked at Oracle. The friend supported his idea and introduced him to an Oracle technical training course, where he participated in four months of hands-on training. After completing the training, Hu Zhonghao officially became a DBA. Since then, even though his job positions have changed several times, he has always worked in the database field.
Hu Zhonghao said that the most memorable experience in his career was when he worked as a database operations engineer at Yunnan Mobile for over a year. Reflecting on that experience, Hu Zhonghao is still filled with gratitude. It was the most stressful, busiest, yet rewarding and fastest-growing time in his career.
“Basically, I encountered almost every Oracle failure scenario that might occur in the market. The most important thing was that my mental resilience was greatly tested,” Hu Zhonghao said.
At that time, he was always in a state of readiness to work, even when eating a bowl of noodles, his computer was always by his side. If he received a call, he would immediately start working by turning on his phone’s hotspot. Waking up in the middle of the night was a regular occurrence. One particular experience still stands out to Hu Zhonghao. Yunnan Mobile updated a system during a mobile membership day, and the transaction volume was enormous. The system got stuck, affecting surrounding systems, and people were unable to pay their phone bills, causing a flood of complaints. The leadership was extremely anxious.
“At that moment, the client’s leadership stood behind me, watching as I troubleshooted the issue. The pressure was immense,” Hu Zhonghao recalled.
Fortunately, with solid basic skills, Hu Zhonghao quickly identified the cause and came up with a solution: two indexes were created, and the issue was resolved in no time. At that moment, Hu Zhonghao felt like the Oracle technical support engineer he had admired back in his internship days. It was this experience that gave Hu Zhonghao a true sense of his ability to handle pressure.
During that time, he also delivered database solutions for many large enterprises in Yunnan, including the power grid, Fudian Bank, Yunnan Baiyao, Cheng Tai Insurance, Yunnan Telecom, logistics companies, hospitals, etc. This experience enriched Hu Zhonghao’s practical knowledge and boosted his confidence. In 2018, when the Bank of Yunnan hired a DBA, Hu Zhonghao was able to successfully pass the interview and became a DBA at the Bank.
Core System Replacement
After joining the Bank, Hu Zhonghao and a colleague were responsible for managing over 100 Oracle databases, as well as dozens of DB2 and MySQL databases. The work was smooth and orderly, with much less pressure compared to his previous role as an on-site engineer, although there were occasional challenging tasks, such as setting up the Bank’s first Oracle RAC cluster.
However, this stable routine was abruptly disrupted by the new core system project. In October 2020, Hu Zhonghao’s leadership informed him that the Bank needed to replace its core system. For reasons related to domesticization and future development, the new core system would adopt a domestic distributed database, and Hu Zhonghao was tasked with preparing for the transition.
Replacing the core system did not come as a surprise to Hu Zhonghao. The current core system was built on IBM AIX hardware and had been running for many years. Its architecture had become outdated, and with surrounding systems already upgraded, the core system had become a bottleneck in need of urgent improvement. However, the decision to directly adopt a domestic distributed database was unexpected.
Upon receiving the task, Hu Zhonghao realized that his work experience so far had been centered on Oracle, DB2, and MySQL, and his understanding of domestic databases was very limited. What was the current state of domestic databases? Who were the leading market players? How did they compare to Oracle in terms of advantages and disadvantages? What about ecosystem development? These were questions Hu Zhonghao had to explore and learn about.
Fortunately, with years of experience using Oracle databases, these questions didn’t intimidate him. After some research and interactions with vendors, Hu Zhonghao gained a preliminary understanding of the market and shortlisted four candidates, including OceanBase, to conduct POC (Proof of Concept) tests at the Bank.
During this period, Hu Zhonghao dedicated all his work energy to the new database. Each day was filled with new knowledge, which excited and energized him, while also giving him a comprehensive understanding of domestic distributed databases.
He explained that domestic distributed databases can be divided into two main categories: one is based on open-source database engines and achieves distribution via sharding; the other is a native distributed database. Sharding-based databases have good linear scalability and low development costs, but may have some limitations in distributed transactions, global clocks, and global indexes. On the other hand, native distributed databases are characterized by high availability and elasticity, and since they don’t rely on sharding, they are less intrusive to applications.
“After evaluating and testing, we leaned toward a cloud-native distributed database,” Hu Zhonghao said. Later, in order to make the preparation more robust, the Bank of Yunnan visited a bank that had deployed a domestic distributed database to learn from their experience, focusing on key issues like global distributed transactions, global clocks, and global indexes. In April 2021, after a public tender, OceanBase was selected.
“The decision to select OceanBase was made after considering various factors, including technology, ecosystem, and the vendor’s strength. After all, we didn’t want to find ourselves unable to locate the vendor in a few years,” Hu Zhonghao said.
Surprising Discoveries in Domestic Distributed Databases
From October 2020, when Hu Zhonghao began focusing on domestic databases, until late May, when OceanBase was successfully launched, he devoted nearly two years to learning about domestic databases. Reflecting on this experience, Hu Zhonghao expressed that domestic databases had surprised him in many ways — he never expected the technology to progress so quickly.
He pointed out that compared to Oracle, three key features of domestic distributed databases left a deep impression on him: high availability, high elasticity, and the ability to handle high concurrency workloads. These features greatly benefited database operations at the Bank.
For example, the Bank’s core system adopts a deployment model with two locations, three centers, and five replicas. Now, if a system goes down in any location, it does not affect the business. Additionally, domestic distributed databases can easily implement active-active configurations, automatically switching over in case of failure, making maintenance much more convenient.
“During testing, we tried many extreme methods, including killing processes, shutting down servers, cutting off networks, and pulling out hard drives. The system was able to recover the business in under 30 seconds,” Hu Zhonghao said.
Similarly, the ability of domestic databases to scale elastically and handle high concurrency has made DBA tasks much easier. For example, when OceanBase is scaled up, simply connecting the equipment and configuring it suffices, without interrupting business. In contrast, with traditional centralized databases, adding nodes or storage devices requires careful timing, as scaling operations can severely disrupt business.
“While single-machine performance is slightly weaker, domestic distributed databases can scale almost infinitely by adding more devices, and their performance increases linearly. This is very convenient,” Hu Zhonghao explained.
DBAs Must Stay Sharp
From his early days in the DBA field, guided by a friend, to his extensive experience today, Hu Zhonghao has accumulated years of wisdom in the database field. He is no longer the naive beginner he once was. Reflecting on his journey, he is deeply grateful to the mentor who guided him into the field, saving him from many detours.
He emphasized that being a DBA is a profession that requires high standards and a clear understanding before entering.
“DBAs need to be meticulous, careful, and have strong mental resilience. These qualities are even more important than professional technical knowledge. Continuous learning and good communication skills are also crucial,” he said.
Hu Zhonghao especially highlighted the importance of learning. He noted that while being an Oracle DBA used to be prestigious and finding a job was relatively easy, the rise of domestic databases and the ongoing push for domestic technology have led to significant changes in the industry. Today’s DBAs must keep learning to keep up with these changes.
“Take time to read official documentation, set up testing environments, conduct tests, and always remember to summarize your learnings,” Hu Zhonghao advised.
Hu Zhonghao revealed that he has maintained the habit of writing work notes for years. Every day, he records and summarizes what he has learned, including the entire process of selecting and launching the new database. "It’s both a summary and a record, and when I look back, I can see how I’ve improved step by step," he said.
Looking ahead, Hu Zhonghao’s short-term goal is to obtain the highest certification for OceanBase, OBCE (Oracle Database Certification Expert). His long-term goal is to expand his knowledge beyond database technology and become a business-savvy expert, not just a technical specialist.
“Through continuous learning, my technical skills have grown, and my mental resilience has also improved. Today, I can confidently face any problem that comes my way. This is the greatest reward of my years of experience,” Hu Zhonghao concluded.
Visit the https://payhip.com/OracleeBookSoftwareShop website to get an e-book about Oracle internal storage.
Top comments (0)