openGauss Won "Typical Cases in the Software Industry in 2021"

李玥旻 — Wed, 27 Jul 2022 02:25:05 +0000

The 2022 China International Software Development Conference and the 5th China Software Industry Annual Conference were successfully held on May 10, 2022. The innovative practice of "openGauss Database Helps CNOOC Build Intelligent Oilfields" won the "Typical Cases in the Software Industry in 2021".

Riding the wave of digital transformation, CNOOC follows the trend and starts the transformation of energy production from informatization to intelligence. Databases are critical to the core IT infrastructure of oil and gas production systems. Considering product performance, data security, service continuity and availability, and support for future services, CNOOC chose openGauss to build a new data O&M management system.

openGauss Is First Rated as an Advanced Trustworthy Open-source Community

李玥旻 — Tue, 26 Jul 2022 12:44:00 +0000

Today, open-source has become the most important software building and management model in the global software industry, and has a profound impact on the development of the digital industry. On May 20, 2022, the OSCAR Open Source Pioneer Day was successfully held. The conference is led by China Academy of Information and Communications Technology (CAICT) and China Communications Standards Association, supported by the Cloud Computing Standards and Open Source Promotion Committee, hosted by the Open Source Cloud Alliance for Industry (OSCAR), and co-hosted by the Financial Industry Open Source Technology Application Community (FINOC), ICT Open Source Community (ICTOSC), technology manufacturing open-source community (TMOSC), and Trustworthy Open-source Community (TWOS).
Adhering to the principle of "Openness + Transparency + Compliance + Security = Trustworthiness", CAICT has established a series of trustworthiness open-source standard systems based on the open-source lifecycle and implemented evaluation tests to standardize the market, help enterprises reduce the risks of using open-source software, and promote the establishment of a trustworthy open-source ecosystem.
According to the standards, the openGauss community is first rated as an advanced trustworthy open-source community in terms of community governance, operation, and development.

Automatic Test Framework YAT

李玥旻 — Thu, 21 Jul 2022 13:25:00 +0000

The Yet another test (YAT) framework is an automatic test framework based on Python3. The core of the framework is implemented by the Kotlin language. The framework is encapsulated and bonded through Python to provide command line interfaces (CLIs). Figure 1 shows the overall framework. The YAT framework is continuously evolving to become a more efficient and advanced automatic test framework.

YAT is the automatic test framework used for openGauss database specification monitoring. openGauss 2.1.0 not only releases open-source YAT, but also contributes over 30,000 automatic test cases to enhance the test capability of the openGauss community, enrich the openGauss ecosystem and attract more developers to participate in community construction. YAT code repository: https://gitee.com/opengauss/Yat

Figure 1 Overall YAT framework

Product Advantages

It is friendly to database tests. Users can directly write SQL code and organize the code into a test suite for tests without additional configuration. Test cases are executed through the JDBC API and can adapt to various databases.
It supports multiple languages and is extensible.

Currently, languages such as SQL, Shell, Python (unittes), and Groovy (Junit/Spock) are supported. New languages and frameworks can be added through adapters. SQL statements supported by YAT are supersets of standard SQL statements.That is, YAT is extended based on standard SQL statements. Users can run shell commands, control connections, execute loops, bind SQL statements, control multiple sessions, and concurrently execute SQL statements in SQL scripts. For example:

@conn user/passwd@127.0.0.1:9090;  -- Reconnect to the database as the new user.
drop table if exists tbl_x;  -- Execute SQL statements.

create table tbl_x (id int, age int, xb int);

insert into tbl_x values(1, 2, 4);
insert into tbl_x values(3, 4, 5);

-- Perform the binding operation.
insert into tbl_x values(?, ?, ?);
@bind {
    int 3
    int 5
    int 7
}
-- Perform the binding operation in batches.
insert into tbl_x values(?, ?, ?);
@batch {
    int 3 int 4 int 0
    int 3 int 4 int 9
    int 3 int 4 int 8
    int 3 int 4 int 7
}
-- Run the shell commands.
@sh zctl.py -t stop;
@sh zctl.py -t start;

-- Define sessions.
@session(name: s1)
{
    @set autocommit false;
    update table tbl_x set par1 = 2 par2 = 2;
    insert into tbl_x values(1, 3, 4);
    commit;
}

@session(name: s2, user: abc, password: 'gauss@123')
{
    @set autocommit false;
    update table tbl_x set par1 = 2 par2 = 2;
    insert into tbl_x values(1, 3, 4);
    @step
    {
        select * from tbl_x for update;
    }
    commit;
}
-- Executes SQL statements in sessions.
@steps s1.0 s2.0 s1.1 s1.2 s2.2 s2.1;
-- Execute loops.
@for (count: 10)
{
    insert into abc values(1,1,3,4);
}
-- Concurrently execute SQL statements.
@parallel {
    @session {
        select * from abc for update;
        commit;
    }

    @session {
        select * from abc for update;
        commit;
    }
}

The Python language test script must be a unittest test script. YAT provides a public library to facilitate database and remote SSH operations in the Python unittest test script. For example:

class TestPrimaryStandby(TestCase):
    node = None
    node_standby = None

    test_table_drop = 'drop table if exists tbl_test';
    test_table_create = '''create table tbl_test (
        id int,
        name char(20),
        address varchar(1024)
    )
    '''

    @classmethod
def setUpClass(cls):
    # Initialize the connection object.
        cls.node = Node(node='primary')
        cls.node_standby = Node(node='standby')
        cls.node.sql(cls.test_table_drop)
        cls.node.sql(cls.test_table_create)

    @classmethod
    def tearDownClass(cls) -> None:
        cls.node.close()
        cls.node_standby.close() # Close the connection object.

def test_abc_001(self):
    # Run the shell command through SSH and check the result.
        self.node.sh('echo "success"').expect('success')

        # Run the SQL statement and check the result.
        self.node.sql('select * from tbl_test').expect(
            (1, 'xxx', 'xxxxxxxxxxxxxxxxxxx'),
            (2, 'xxx', 'xxxxxxxxxxxxxxxxxxx'))

def test_abc_003(self):
    # Run the shell command and determine the result through regular expression matching.
        self.node.sh('cm ctl query').regex(r'.*success.*')

Groovy scripts can be used to write JUnit cases or Spock cases. For more details, visit the official website.

It defines a set of standard test case writing specifications, test execution processes, and test report presentation modes.

Run the yat init command to create a test suite template. After the yat init command is executed, modify the content in the specified test suite directory. Then, install the JDBC driver, add the lib directory to the root directory of the test suite, copy the driver to this directory, and run the yat suite run command to run the test suite. You can add different parameters to set whether to print the report and the report format. Figure 2 shows the test suite directory result. Configure the following directories before running the test suite:

conf directory, which stores node configuration files.
except directory, which is the expected file of the test case.
schedule directory, which stores the schedule files.
testcase directory, which stores test case files.

Figure 2 Directory structure of the test suite

Multi-suite Schedule YAT Schedule is a scheduler provided by YAT to schedule multiple YAT test suites at the same time. You can schedule multiple YAT test suites in parallel or serial mode by customizing a schedule file. When there are a large number of test suites, you need to determine the combination and sequence of the test suites. YAT provides a convenient method to organize multiple test suites into a large test suite, as shown in the following figure.

# File name: all.ys
serial {# Serial execution of test suites or test suite sets
    suite '../suite1';
    suite '../suite2';
    ...

    parallel { # Parallel execution of test suites or test suite sets
        suite 'parallel/suite1';
        suite 'parallel/suite2';
        ...

        serial { ... }
    }

    suite 'suite3';
    ...
}

Run the following command to execute all test suites in one-click mode:

yat schedule -s all.ys

YAT Schedule is a scheduler provided by YAT to schedule multiple YAT test suites at the same time. You can schedule multiple YAT test suites in parallel or serial mode by customizing a schedule file. When there are a large number of test suites, you need to determine the combination and sequence of the test suites. YAT provides a convenient method to organize multiple test suites into a large test suite, as shown in the following figure.

File name: all.ys

serial {# Serial execution of test suites or test suite sets
suite '../suite1';
suite '../suite2';
...

parallel { # Parallel execution of test suites or test suite sets
    suite 'parallel/suite1';
    suite 'parallel/suite2';
    ...

    serial { ... }
}

suite 'suite3';
...

}
Run the following command to execute all test suites in one-click mode:

yat schedule -s all.ys

New Feature of openGauss 3.0.0: Parallel Decoding

李玥旻 — Wed, 20 Jul 2022 02:22:07 +0000

Link

summarty：With the rapid development of information technology, various types of databases emerge one after another. Logical replication is increasingly important, with which data can be synchronized between heterogeneous databases. Currently, the average serial decoding performance of logical replication in openGauss is only 3 to 5 Mbit/s, which cannot meet the requirements of real-time synchronization in heavy service pressure scenarios. As a result, logs are stacked, affecting services in the production cluster. Therefore, the parallel decoding feature is designed to enable multiple threads to perform decoding in parallel, improving the decoding performance. In basic scenarios, the decoding performance can reach 100 Mbit/s.
openGauss solutions as follow.

DEV Community: 李玥旻

openGauss Won "Typical Cases in the Software Industry in 2021"

openGauss Is First Rated as an Advanced Trustworthy Open-source Community

Automatic Test Framework YAT

Product Advantages

File name: all.ys

New Feature of openGauss 3.0.0: Parallel Decoding