OSD600 - lab 07

Link to Repo
Link to commit

Description

This post is about lab 07 for OSD600 course at Seneca College. This week we learned how to test our project. We learned about unit test, e2e(end to end), and integration test.

Progress

• First, I needed to decide which test framewokr I am going to use. I asked chatGPT which one is most popular and also googled it and read some people's opinion. Then, I decided to use pytest framework.

• Second, I started writing test code with each class that is called on top of the function call stacks. I chose the easist one, which is load_config.py. It only contained one function.
  

import os
import toml
def load_config():
    # Load config file from the project's root directory
    config_path = os.path.expanduser("~/.codemage-config.toml")  # Adjust path to project root
    if os.path.exists(config_path):
        config = toml.load(config_path)
    else:
        config = {}

    return config

This function opens .toml file, and read key-value pairs. I had to mock the config file and its content.

class TestLoadConfig:
    @patch("os.path.exists")
    @patch("toml.load")
    def test_load_config_file_exists(self, mock_toml_load, mock_os_exists):
        # Simulate that the file exists
        mock_os_exists.return_value = True

        # Simulate the contents of the file
        mock_toml_load.return_value = {
            "language": "Python",
            "OPENROUTER_API_KEY": "mock_openrouter_api_key",
        }

        # Call load_config function
        config = load_config()

        # Assertions
        mock_os_exists.assert_called_once_with(os.path.expanduser("~/.codemage-config.toml"))
        mock_toml_load.assert_called_once_with(os.path.expanduser("~/.codemage-config.toml"))
        assert config == {"language": "Python", "OPENROUTER_API_KEY": "mock_openrouter_api_key"}

I used @patch to mock the file, file status(exists or not), and the content. Then I checked if the file is called with the path and if file content is correct.

• Third, I tested Api class. Since this is also independent and called on the top of the function stack.

class Api:
    def __init__(self, model, config):
        self.supported_model = ["groq", "openrouter"]
        self.model = model if model is not None else "openrouter"

        if self.model not in self.supported_model:
            sys.exit(f"{self.model} is not suppored. Model Supported: {self.supported_model}")

        # default api_url and api_model
        self.api_url = "https://openrouter.ai/api/v1"
        self.api_model = "sao10k/l3-euryale-70b"
        self.api_key = os.getenv("OPENROUTER_API_KEY") or config.get("OPENROUTER_API_KEY")

        # api_url and api_model when the provider is groq
        if self.model == "groq":
            self.api_url = "https://api.groq.com/openai/v1"
            self.api_model = "llama3-8b-8192"
            self.api_key = os.getenv("GROQ_API_KEY") or config.get("GROQ_API_KEY")

    def call_api(self, target_lang, code, stream_flag=False):
        client = OpenAI(
            base_url=self.api_url,
            api_key=self.api_key,
        )

        completion = client.chat.completions.create(
            extra_headers={},
            model=self.api_model,
            messages=[
                {
                    "role": "system",
                    "content": "only display the code without any explanation",
                },
                {
                    "role": "user",
                    "content": f"translate this to {target_lang} language: {code}",
                },
            ],
            stream=stream_flag,
        )

        return completion

I tested the constructor first and then I tested the method call_api. Testing constructor was not difficult. I just needed to make a mock_config, and mock_args. Testing the method was pretty tough. I had to mock the response object.

class TestApiWithEnv:
    @pytest.fixture
    def mock_config(self):
        return {
            "OPENROUTER_API_KEY": "fake_openrouter_api_key_from_toml",
            "GROQ_API_KEY": "fake_groq_api_key",
        }

    # Mock OpenAI response
    def mock_api_call(self, *args, **kwargs):
        # Return a mock response object with a `choices` attribute
        mock_response = MagicMock()
        mock_response.choices = [{"message": {"content": "Translated code"}}]
        return mock_response

    # Test when model is supported
    @patch.dict(
        os.environ,
        {
            "OPENROUTER_API_KEY": "fake_api_key_from_env",
            "GROQ_API_KEY": "fake_groq_api_key_from_env",
        },
    )
    @patch("code_mage.api.OpenAI")
    def test_api_with_openrouter(self, mock_openai_class):
        # Mock the OpenAI client and its methods
        mock_client = MagicMock()

        # When the code create OpenAI class inside call_api(), it's replaced by mock_openai_class
        mock_openai_class.return_value = mock_client
        mock_client.chat.completions.create = MagicMock(return_value=self.mock_api_call())

        # Create an instance of the Api with a supported model
        api = Api(model="openrouter", config={})

        # Simulate an API call
        response = api.call_api(target_lang="python", code="some_code")

        # Check if the API call was made with the expected arguments
        mock_client.chat.completions.create.assert_called_once_with(
            extra_headers={},
            model="sao10k/l3-euryale-70b",
            messages=[
                {"role": "system", "content": "only display the code without any explanation"},
                {"role": "user", "content": "translate this to python language: some_code"},
            ],
            stream=False,
        )

        # Assert the response content
        assert response.choices[0]["message"]["content"] == "Translated code"

Fortunately, there is a way to mock api call provided. I used

@patch("code_mage.api.OpenAI")

This is one of my Test codes for Api Class.

class TestApiWithEnv:
    @pytest.fixture
    def mock_config(self):
        return {
            "OPENROUTER_API_KEY": "fake_openrouter_api_key_from_toml",
            "GROQ_API_KEY": "fake_groq_api_key",
        }

    # Mock OpenAI response
    def mock_api_call(self, *args, **kwargs):
        # Return a mock response object with a `choices` attribute
        mock_response = MagicMock()
        mock_response.choices = [{"message": {"content": "Translated code"}}]
        return mock_response

    # Test when model is supported
    @patch.dict(
        os.environ,
        {
            "OPENROUTER_API_KEY": "fake_api_key_from_env",
            "GROQ_API_KEY": "fake_groq_api_key_from_env",
        },
    )
    @patch("code_mage.api.OpenAI")
    def test_api_with_openrouter(self, mock_openai_class):
        # Mock the OpenAI client and its methods
        mock_client = MagicMock()

        # When the code create OpenAI class inside call_api(), it's replaced by mock_openai_class
        mock_openai_class.return_value = mock_client
        mock_client.chat.completions.create = MagicMock(return_value=self.mock_api_call())

        # Create an instance of the Api with a supported model
        api = Api(model="openrouter", config={})

        # Simulate an API call
        response = api.call_api(target_lang="python", code="some_code")

        # Check if the API call was made with the expected arguments
        mock_client.chat.completions.create.assert_called_once_with(
            extra_headers={},
            model="sao10k/l3-euryale-70b",
            messages=[
                {"role": "system", "content": "only display the code without any explanation"},
                {"role": "user", "content": "translate this to python language: some_code"},
            ],
            stream=False,
        )

        # Assert the response content
        assert response.choices[0]["message"]["content"] == "Translated code"

I used ChatGPT; it gave me good hints, but didn't work well, I had to make some changes and it worked. If I am being honest, I am still confused how it works, but it looks like it tests the code as it should. I think I need to study more to fully understand and write test code.

• Fourth, the most difficult part was testing Translator class. It calls other functions that requires mock environments such as openeing a file from file system and calling api.

This is my test code for translate() function in Translator class:

class TestTranslator:
    @pytest.fixture
    def mock_args(self):
        """Fixture for simulating command-line arguments."""
        return Mock(language=None, model=None, stream=False, token_usage=False, output=None)

    def mock_api_call(self, *args, **kwargs):
        mock_response = MagicMock()
        mock_response.choices = [{"message": {"content": "Translated code"}}]
        return mock_response

    @patch("builtins.open", new_callable=MagicMock)  # Mock open
    @patch("code_mage.translator.Api")  # Mock Api class
    @patch.object(Translator, "_Translator__get_output_filename", return_value="translated_test.py")
    def test_translate(self, mock_get_output_filename, mock_api, mock_open, mock_args):
        mock_config = {"api_key": "fake_api_key"}

        # Initialize Translator instance
        translator = Translator(mock_args, mock_config)

        mock_file = MagicMock()
        mock_open.return_value = mock_file

        # Mock the Api's call_api method
        mock_api_instance = MagicMock()
        mock_api.return_value = mock_api_instance
        mock_api_instance.chat.completions.create = MagicMock(return_value=self.mock_api_call())

        translator.translate("./example/test.js")

        assert mock_api_instance.chat.completions.create.return_value.choices == [
            {"message": {"content": "Translated code"}}
        ]

        mock_open.assert_any_call("./example/test.js", "r")
        mock_open.assert_any_call("translated_test.py", "w")

What was challenging was that I didn't understand how it knows which mock is for api_call() and which mock is for open() when I made two Mock objects. I tested it by changing the order of arguments, and the name of arguments. I think it maps @patch result to the arguments by order and name. I don't understand how exactly it works, but I learned that the order and name matters and I should make sure that they followes the order in which @patch is called and the name.

• Lastly, I couldn't test main function as it calls many other functions which require mock enviornment.

Testing the whole main function would make test too complicated. However, I've already tested all the other functions except for arguments parser. Therefore, I refactored the argument-parsing logic. I made a new function for parsing arguements and imported it in the main function. I believe this can to some degree ensure that all the logics of the tool covered, although testing for everything together from start to end is not covered.

This is a part of my testing code for argument parser.

@pytest.fixture
def mock_config():
    # A mock configuration that mimics the structure of the actual config dictionary.
    return {
        "language": "python",
        "output": "result",
        "token_usage": False,
        "model": "groq",
        "stream": False,
    }


# Test with no arguments
def test_arg_parser_no_args(mock_config):
    with patch("sys.argv", ["code_mage.py"]):
        args = arg_parser(mock_config)
        assert args.source_files == []
        assert args.language == "python"  # default from mock_config
        assert args.output == "result"  # default from mock_config
        assert args.token_usage is False
        assert args.model == "groq"
        assert args.stream is False


# Test with source file
def test_arg_parser_with_source_file(mock_config):
    with patch("sys.argv", ["code_mage.py", "example.js"]):
        args = arg_parser(mock_config)
        assert args.source_files == ["example.js"]
        assert args.language == "python"
        assert args.output == "result"

Reflection

At first, I thought that it is going to be easy. However, it was really tough, especially for mocking object, .env, file open(), http request, and http response. It also was kind of fun. From this lab, I learned how to test code thoroughly and how important it is to modulize the code so that I can test the code in a isolated environment. It's not perfect but at least it guarantees a certain level of robustness.

Q1. Which testing framework/tools did you choose? Why did you choose them? Provide links to each tool and briefly introduce them.

I chose pytest; but I also used some of the methods provided by unittest, which is provided by python as a default.

How did you set them up in your project? Be detailed so that other developers could read your blog and get some idea how to do the same.

install pytest if not installed yet.

poetry add --dev pytest

Running pytest

This command will run files named test_*.py or *_test.py

poetry run pytest

For more detailed test result, you can use -v option

poetry run pytest -v

for more detailed test information

poetry run pytest -vv

Writing TestCode

You should name your test class and method as follows:

Class name starts with Test

Method name starts with test_

class TestMyFeature: # class name
    def test_feature_functionality: # method name

Q3. How did you handle LLM response mocking?

In the progress section, I explained about this, so I will skip here.

Q4. What did you learn while writing your test cases? Did you have any "aha!" moments or get stuck?

What I learned is how important it is to design the code strutures. I learend that good design pattern is not just for making code look better and reusability but also for testing.

Q5. Did your tests uncover any interesting bugs or edge cases?

My test doesn't cover some parts such as streaming out response object and testing from start to zero like total integration test.

Q6. What did you learn from this process? Had you ever done testing before? Do you think you'll do testing on projects in the future?

My first experience with testing was in my second semester. I took the software testing course that was newly introduced. I was the first student in seneca who took the course. At first, I really didn't like it. I thought that it was a waste of time. test code was 4~5 times larger and took 4~5 times more than writing the logic of the program. It was grinding. However, now I kind of like it. When I see the green colored results pass, I feel really good. Making mock environment is still hard and somewhat stressful, but thinking about edge cases is actually really fun. It is kind of similar to Probability and Statistics in Math like thinking about all the possible cases and cover it.