DEV Community: ontowhee

The 5 Stages Of My Software Development Process

ontowhee — Wed, 12 Jun 2024 06:34:54 +0000

It’s simple and straightforward. However, it is not second nature to me. I have to write it out in order to follow it.

It became very important for me to know the process very well when the tech lead on the team gave me more responsibilities on a recent project. I had to take the helm and drive the process forward.

This is what I individually use on my current team. Whether I am working on a task on my own or pairing with my coworkers, I reference these 5 stages. It serves as my personal guard rails. It helps me ask the right questions, use the right medium, at right time!

Breaking the process into these 5 stages helps me and my team get on the same page. We are able to collaborate efficiently. We are able achieve successful and satisfying deliveries.

The 5 Stages:

Grooming
Align With Stakeholders
Active Development
Code Review
QA

(I’m only discussing stages that I am directly involved in as a software engineer on my team. Stages prior to Grooming are owned by the product team. Stages after QA are owned by the release team. Those are not covered here.)

Stage 1: Grooming

The purpose of this stage is for me to establish a good grasp of the project and provide a document that I can share with my team.

Goal

Read the product ticket. Go through the design documents.
Explore the code.
Prepare notes to share with the team.

Allocated time

1 day for small projects. 2-3 days for medium projects. 4+ days for larger projects.

Deliverables

By the end of this stage, I create what I call a “Grooming document” containing the following sections:

Brief description of the problem in my own words
Brief description of the technical requirements
Proposal for implementation
- Highlight key existing code patterns, components, and functional behaviors that will be involved in this project
- Highlight key code components that need to be added, removed, or modified
- Breakdown tasks. Add implementation details as necessary.
- List of test cases to cover both the product and technical requirements
List of questions for:
- Product and design
- Code
- Scope of work and technical feasibility given time and resource constraints
List of unknowns that would require further deep dive

The Grooming document does not need to be lengthy or professionally written. It is a set of notes used internally for the team. I don’t require my coworkers to read them thoroughly. I walk through the document with them during the Align With Stakeholders stage.

It’s a balancing act to achieve notes that are detailed and accurate without spending too much time writing them up. To accomplish this balance, I have to determine which areas need deeper dive at this early stage, and which areas can be briefly mentioned with the expectation that they will be explored later. I don’t get it right all the time, and that’s ok.

Once I have prepared my notes, I am ready to have a meeting. I reach out to the product manager and tech lead. They schedule the meeting and invite all the engineers.

This is my personal favorite part of software engineering. I love exploring the existing code base and thinking through ideas for changing the code. I also love organizing my thoughts to communicate them clearly to my coworkers. To me, this is the bulk of software development. I love it because it allows me to invest time upfront planning the project, so I can execute smoothing in the next 4 stages!

Stage 2: Align With Stakeholders

Now that I have prepared the Grooming document, I share my findings with my team, and we discuss the details to make sure the proposed implementation is viable and meets the product’s needs.

Goals

Get answers to questions.
Discuss technical details with engineers to ensure the proposed implementation is viable. Answer questions to help them get up to speed on the problem. Also allows them to challenge the proposal.
Refine the problem and solution. Make adjustments to project requirements and scope as necessary.

Allocated time

30 mins - 1 hour meeting. Larger projects may need multiple sessions.

Deliverables

Grooming document updated with questions answers. Note down decisions and any adjustments to the project.
Create tickets for the tasks. These tasks are now ready to be picked up by engineers on the team, including myself.

For questions that do not have answers, the team will make a decision together on how to work around it, or potentially exclude it, from the project. That is, remove it from the scope of work. This decision can sometimes happen during the meeting, or sometimes the product manager or tech lead or designer need extra time to think, and we follow up another time.

Towards the end of the meeting, I like to ask the product manager for the expected start date. Sometimes the project is expected to start immediately. Other times, the project is queued up and scheduled to be developed at later date. This gives me a clear sense of the priority, and I can allocate my time for my ongoing projects accordingly.

Once the team has finished the alignment meeting, it’s mainly about execution from here! So, let’s go!

Stage 3: Active Development

Now I dive into code. The tasks have been broken down. I can start picking them up.

Goals

Pick up and work through project tasks.
- Write code
- Write tests
- Write docstrings
Handle unexpected items
- Communicate these items to the team immediately
- Methodically work through items
- Iterate with additional grooming and aligning if necessary.

Time Allocation

Varies depending on the project. I’m not specifying a time here. It’s easier to estimate each task once they have been broken down, and then add them up to estimate the overall project’s time. Typically, the team categorizes projects loosely as "small rock", "medium rock", and "large rock".

Deliverables

PR ready for review

My team strives to iterate on our code development. We have 3 rough phases within the Active Development stage:

Implement the functionality
Connect the frontend and backend
Polish up details for UI or edge cases

Following these phases allows us to get the core functionality in place, then worry about polishing details later.

Sometimes, unexpected items pop up during Active Development. Usually this happens when I decide not to dive deeper during the Grooming stage and relied on assumptions that turn out to be incorrect. When this happens, I’ll take a step back and enter a mini Grooming and Aligning session to hammer out the details. I’ll dive into the code to find out more about this unexpected item. If it requires a decision from the project leaders, I’ll discuss my findings with the product manager, tech lead, or designer.

It is normal for there to be unexpected items, but I have to keep on top of them by letting my team be aware of my progress. That way, there will be no surprises for why things are taking a bit longer than expected.

If I have really good intuition, these unexpected items would have been identified upfront during the Grooming and Aligning stages. I would have been able to allocate time I need to explore these items before Active Development even begins! However, I’m not perfect, so I’ll miss some details every now and then, and that’s ok. As long as I keep my team aware, and I take the time to re-enter grooming and aligning stages, I’ll be able to get unblocked and continue with development!

Once development is done, I send the code into review!

Stage 4: Code Review

The Code Review stage is where I get feedback on my code. This is done to improve the code quality and ensure it meets the standards for the team.

Goals

Check for correctness of the functionality.
Check that it meets technical requirements and aligns with the team’s choice of patterns, components, and packages to utilize.
Improve code quality by pointing out areas that can be refactored
Reviewer will provide feedback. Weigh their feedback and incorporate the suggestions into the code.

Check list

Our team has a checklist when we open a PR that is ready for review. This is a simplified version:

Fill out the PR description with details on what code changes were made and the decisions that went into it.
Include screenshots to illustrate the changes. For UI changes, include before and after images. Alternatively, record a video to demonstrate the functionality.
Make sure tests are all passing
Double check if migration files have been included
Add a release note

Time Allocation

Varies depending on project and availability of reviewers.

Deliverables

All requested changes have been addressed
All tests pass
Reviewer’s approval

Ok, I know I said earlier that, “once development is done, I send the code into review!” This is not completely true. Sometimes I’ll send a draft PR into review. This allows me to get quick feedback and make sure I am on the right path. I’ll do this when I think my implementation could use an extra pair of eyes. The tests do not need to pass in this situation. I’ll tell the reviewer, “This is just a draft! The tests are not passing yet. I just need your eyes to make sure I’m on the right path!” The reviewer knows exactly what to look for in the code and offer a quick review.

If all the previous stages were done well, code review should be a breeze with just minor comments. The details of the implementation should have been discussed and agreed upon during the Grooming and Aligning stages.

Any grooming or aligning that is happening during the code review is a big red flag. If this happens, I have failed to follow the process! I have failed as an engineer. I’m embarrassed to admit that it has happened before (recently). The failure is motivating me to write and publish this blog entry. I can have an explicit process to follow and avoid such mistakes in the future.

Once the PR gets passed Code Review, it is ready to undergo QA!

Stage 5: QA

On our team, the product manager is in charge of manual QA, which is the final step before release. My role here is to prepare the sandbox for the product manager and provide some QA steps.

Goals

Deploy code to sandbox and prepare data on the sandbox
Write steps for QA for special scenarios
Work with product manager to ensure the implementation matches the product requirements

Time Allocation

Varies by project size. Can be 5 minutes for small projects, or 1 hour for a large one. Can be multiple sessions if bugs are found.

Deliverables

Meets all items outlined in the project "acceptance criteria"
Product manager’s approval

If there are cases that the product manager has not yet considered — not yet outlined in the product requirements or acceptance criteria — then I will write them up, update the ticket, and notify the product manager. These cases tend to involve more technical details, such as inspecting specific data records or ensuring a specific order of events. We conduct manual QA on these cases instead of writing automated tests because we want to confirm the code’s behavior in a real, live, complex system that our automated tests are not set up to handle.

QA can be laborious, especially when our team is conducting it manually. However, once there is a process in place, and you follow it well, it can get easier and easier. Ideally, more of the QA would be automated, but our team is not quite at there yet! For now, we invest our time to ensure high quality and minimize the bugs. Keep in mind, we’re also balancing quality with delivery time. Lots of balancing acts! I love the challenge.

Once I get QA approval and Code Review approval, the code is ready to merge! I can take a moment to breathe and celebrate with the team. Then I’ll repeat the 5 stages with the next project.

Appreciation

It took me quite some time to get used to working with a well-defined process. Part of the reason is that there was no well-defined process when I first joined the team. The team was growing and still learning the ropes.

Generally, I think the software development process has always been more or less the same, but it wasn’t explicitly laid out. It was implied that all engineers knew exactly what was expected from the product manager, tech lead, and designer. However, that is rarely true, and very commonly the source of team failures.

The other part of the reason is that I resented any form of process. I didn’t want to follow steps and rules. I used to think that I was a much better engineer before I joined this team. I thought I was above it all, that I could work more efficiently without a process. I didn’t realize how wrong I was until I kept repeating my mistakes. My projects would drag on and on, and many times it seemed like there was no end in sight. I was miserable. The product manager, tech lead, and my engineering teammates were not happy either. It wasn’t until I was given more responsibilities that I learned to follow the process more diligently.

Now I appreciate the process! It is what makes me a better engineer. Each stage helps me get into the right mindset for the tasks that I need to perform. I have the tools and structure to get unblocked when something unexpected comes up. It helps me plan projects, which improves how I manage my time. It helps me break down my tasks, which helps me execute efficiently instead of getting overwhelmed with implementing too many things at once. I know how and when to communicate with the product manager, tech lead, and designer on the team.

Overall, my 5 stage software development process helps reduce the risks of the project getting out of hand. The bulk of the work is done when there is dedicated time for planning and collaborating in the first two stages. The entire team is fully present to participate in giving feedback. The last 3 stages will then follow pretty smoothly. It’s very effective, and allows our team to deliver success! I have so much appreciation for it now.

What does your software development process look like? What part was a game changer for you in your engineering career? Share your comments below!

Triage: When a ticket description is outdated -- Working through an issue with django-filter upgrade from 10 months ago

ontowhee — Sun, 31 Mar 2024 20:22:09 +0000

I learned from my rotation as a triage engineer that sometimes, when the author of the ticket documents the issue they encountered, it is possible for that issue to no longer be valid. What it describes no longer holds true by the time you pick it up. That is, the ticket is outdated.

As the title of this blog suggests, the ticket I am writing about falls into this category. Even though the ticket was outdated, and the actual code change was minimal, I feel very proud of the work I did. I am writing up my thought process as I worked through the issue, what I learned, and how I communicated my findings and the resolution to the teams that were impacted.

The Ticket

After our engineering team’s first technical debt grooming meeting this week, I picked up a ticket from the top of the “groomed” list in the Jira board. The ticket was 10 months old. It was titled “django-filter package upgrade”. At the time the ticket was created, the platform team was working through a major project to bring our company’s application up-to-date with Django’s latest version. There were several versions to upgrade through, and all the dependency packages needed to be updated alongside Django as well.

There were lots of moving parts, and the platform team ran into a problem while updating django-filter. They were able to get the application up to Django 4.2, and they made the decision that it was good enough for now. They didn’t have time to further investigate the issue with django-filter, as their priorities shifted to other projects. So they documented the issue, and the ticket sat untouched for 10 months.

My first experience with package upgrades happened when I attended Django Con 2023 online. The organizers hosted testathons, and I was lucky enough to have attended one session. The host walked through how to upgrade dependency packages for open source projects. I learned what files to start looking for (tox.ini, requirements.txt, CI/CD config files such as circleci.yml, as well as any files documenting the process for contributing). I learned how to bump a version of a dependency and run test suites for the project. I learned how to install the project in edit mode and fix the failing code if anything needs fixing. Or, if the effort is much larger and beyond one’s knowledge, open up the PR anyway with the bumped version for the dependency, and report the issues for the failing tests. Most importantly, I learned to dive right into code and poke around, despite seeing the source code for the first time.

When I picked up the ticket for “django-filter package upgrade”, I was not too familiar with the inner workings of django-filter (and I still am not!). I didn’t know what could be causing the test failures, but I was armed with the experience from the testathon to know what the process of upgrading a package version looks like. I was also armed with the the understanding of the possibility that the ticket’s description might be outdated. I was ready to dive right in and help unblock the platform team.

The Process

After reading the ticket, these questions came immediately to my mind:

What is the current package version?
What version was attempted for upgrade?
What were the changes between versions?

I looked in the requirements.txt file for our company’s application to see that django-filter was currently pinned to 21.1. The ticket’s description mentions that the upgrade encountered the failing tests when attempting to bump up to version 22.1. That is just one version up from our latest code. I then looked into the changelog for django-filter, which I found here, https://github.com/carltongibson/django-filter/blob/main/CHANGES.rst. I read through the items listed for all versions from 21.1 to the latest 24.2. The major change that stood out to me was the line mentioning changing filter_class to filterset_class and changing filter_fields to filterset_fields.

Removed filter_class (use filterset_class) and filter_fields (filterset_fields) that were deprecated in [version 2.0 (2018)]

I didn’t quite understand the implications of this changelog item. I considered my next step. My options were:

Inspect the source code to understand any structural or implementation changes.
Or, take a test-driven development approach to run our application test suite and see what happens.

I chose the second option, because I needed more clues, and the results of any failed tests would narrow down the problem and guide me in the right direction. (Hooray to our engineering team for writing quality tests!)

I had the latest version of the application code pulled down, which means it was still using django-filter 21.1. First, I ran the tests as a sanity check and made sure all the tests passed. Then, I bumped django-filter to 22.1 by pinning it in the requirements file and running pip install -r requirements.txt. I ran the test suite again. What I found was a bit confusing. The tests that were failing for me were not mentioned in the description. This wasn’t terribly surprising, because it was expected that there were more failing tests than what was documented. However, the tests that were documented as failing were now passing! I couldn’t believe my eyes. What was going on? Shouldn’t they be failing? How did this ticket get written up?

After the initial confusion, more questions came to my mind. Was it a change in django-filter that resolved the issue? Or was it a change in our application that resolved the issue? What would that change be? Was the issue even resolved?

The Documented Failing Tests

The output of the failing tests that were documented in the ticket looked something like this:

======================================================================
FAIL: test_api (application.tests.ApiTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/application/tests.py", line 1363, in test_api
    self.assertEqual(response.data['count'], 3)
AssertionError: 6 != 3

This particular line is testing the number of items returned by the filter. It was not returning the expected number of items.

I started with the clue from the changelog. I asked myself the following questions: Is our code base using filter_class and filter_fields, or is it using filterset_class and filterset_fields? Which filter class in our application code base was part of the failing test?

I did a quick search in my IDE and pulled up the filter class. I saw that the variable being used was filterset_class. So our code had already been updated to be compatible with 22.1. Then I wondered, “Did this change occur before or after the ticket reported that django-filter encountered problems during package upgrades?”

I opened up git blame and I placed my cursor on the line for the filterset_class to see who changed it and when it was changed. Git blame showed me that the code was changed 5 months ago by a co-worker who was not on the platform team. The ticket was created 10 months ago by the platform team. So this change happened after the ticket was created, and the platform team wasn’t aware. This makes a lot of sense now. It explains why the documented failing tests are now observed to be passing! It also explains why this ticket has remained on the tech debt board.

I wasn’t done yet. Notice that I had pulled down the most recent version of our application code (which had already been updated to use filterset_class)? The platform team was working with an older version of the application code. I wanted to confirm with high confidence that the platform team observed the failing tests because the filter class was using filter_set as the variable at the time. To do so, I reproduced their set up by temporarily changing the filterset_class variable to filter_set and ran the tests. The results of the tests matched exactly what was documented! Now I had all the information I needed to confirm how the tests were observed to be failing, and how they are now passing.

The Undocumented Failing Tests

So far, I addressed one set of failing tests. Now I had to address the other failing tests, the ones that had not been captured in the ticket’s description. The failed tests showed messages that looked like this:

======================================================================
FAIL: test_query_counts (application.tests.FilterTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/application/tests.py", line 148, in test_query_counts
    self.assertTrue('some_value' in expected_results)
  File ".../lib/python3.9/site-packages/django/test/testcases.py", line 99, in __exit__
    self.test_case.assertEqual(
AssertionError: 25 != 24 : 25 queries executed, 24 expected
Captured queries were:
...

It’s not visible here in the stack trace, but the failure happens on a with self.assertNumQueries(24): statement that is wrapping around self.assertTrue(). The tests were failing because the query counts do not match up with what the expected counts.

Luckily, I recently read through a lot of Django tickets on Trac because I was looking for an open source issue to work on. I remember coming across one ticket regarding a bug in assertNumQueries. I was able to quickly opened up Django’s Issues and search for this Trac ticket. It came right up! If you are interested, read more about it here, https://code.djangoproject.com/ticket/23746.

To confirm that our application tests were failing due to the bug in assertNumQueries, I dropped the test database and ran the tests again. The tests passed! Then I ran the tests again with REUSE_DB turned on. The tests failed! This behavior matches exactly what is described in the assertNumQueries ticket! I was able to conclude that the tests were failing due to a bug in Django, and not due to our application’s code.

Running All Test Suites With django-filter 24.2

At this point, I hadn’t run all the tests in the application yet, just a targeted group from one part of our application. I also hadn’t checked the latest version of django-filter, which is 24.2. I didn’t have a mechanism at this very moment to efficiently run all the tests locally on my machine and capture all the results for viewing. It was much easier for me to run them all on CircleCI.

So, I bumped up django-filter to the latest version, I committed the requirements.txt file, pushed up the branch, opened a PR, and allowed CircleCI to run all the test suites in the application. After all the tests ran, I checked back, and to my surprise and delight, they all passed! Green check marks down the list!

Once I knew where to look, this issue turned out to be pretty straightforward to address. Even though there was not much to change on my end, other than bumping django-filter version in requirements.txt file, I was really happy to resolve the ticket and unblock the team from upgrading to Django 5.1.

Report

Finally, I was able to report my findings by posting a comment on the ticket:

The documented failed tests have been addressed. The created date for the “django-filter package upgrade” ticket along with date of the our application code change explains why the ticket shows the tests as failing, while current test runs show the tests as passing.
Django’s assertNumQueries has a known bug when running tests with REUSE_DB turned on.
Bumping to 24.2 shows that all tests are passing.
Conclude that the ticket is resolved and is no longer a blocker for upgrading to Django 5.1.

I tagged the platform team in my comment, and I asked for their review and suggestions on the next steps required to get the code merged into the main branch. They gave me the approval to merge the code. They marked the ticket as Done and moved it off of the technical debt board!

Summary Of Steps

Read the ticket and understand where the author of the ticket is coming from.
Attempt to replicate the ticket. Step in the ticket author’s shoes. Pull down the version of the application code and packages at the time of the ticket creation. Compare it to the most recent application code and packages.
Debug any failed tests.
Implement a fix if it is small and straightforward. Otherwise, document the remaining issues.
Report your findings to those who will be impacted. Include any suggestions for the next steps if you have any, or ask for suggestions. Keep the train moving!

Conclusion

I’m really excited that the platform team is now unblocked by django-filter package. When they are ready to pick up where they left off on package upgrades, they’ll be able to hit the ground running. I’m also excited that I got to leverage my budding familiarity with navigating Django’s Trac ticketing system to reference known bugs in Django. This was crucial in helping me conclude that the test failures we observed were not related to our code base, and we can move forward.

What I Learned

Never assume that the problem described in a ticket is accurate.
Read the changelog for the packages you are upgrading.
Run your test suites and see what fails. Use that to narrow down the problem and guide you in the right direction.
Check git blame for the authors and dates of code changes.
Check git diff for the content of the code changes.
Django’s assertNumQueries has a known bug where the cache is not cleared and the result will vary between runs if you are running with REUSE_DB turned on. The first run will return the correct result, but the subsequent runs might not.
Confidently report your findings to those who will be impacted and ask for their thoughts and suggestions for how the team can move forward from here.

I hope you enjoyed this article! I enjoyed undergoing the process of resolving the issue and recreating the experience by writing it all up!

Triaging The Salesforce Error Message “field 'NewValue' cannot be filtered in a query call”

ontowhee — Sun, 25 Feb 2024 19:01:33 +0000

Have you encountered this error message? Are you wondering where it comes from, and what it means?

{
    "message": "SELECT Id FROM LeadHistory WHERE NewValue = 'great' ^ ERROR at Row:1:Column:34 field 'NewValue' cannot be filtered in a query call",
    "errorCode": "INVALID_FIELD",
}

I was assigned a bug ticket describing how the customer encountered this error. I set out to perform a root cause analysis. I’m documenting the main parts of that process. If you happen to encounter a similar error message, hopefully this will help shed some clues, even if the clues indicate that the bug you are looking at is a different one.

Steps For Triage

Here are the steps I take when I triage a problem.

Read the description of the ticket. Read the error message.
Write down what you know and don't know.

Write down what you understand and don’t understand about the application that is displaying the error. Write down what you know and don’t know about the error message.

Share this with your teammates and ask them questions to see if they have any clues. Your teammates can include engineers, product managers, and the support representatives who communicate with customers and prepare the bug tickets.
Come up with a plan to isolate the problem and check your hunch(es).

The goal is to replicate the bug. By replicating the bug, you can confirm your understanding of the behavior of the product. You can relay that information to the support representatives who will then help the customers understand why they encountered the bug. Again, share your plans with your teammates by thinking out loud. They can help you fill in any details, or help you consider looking into another facet of the software, that you might have missed.
Replicate the bug.

This step usually goes hand-in-hand with Step 3 above. You’re doing detective work, and your hunches will change as you work through them.

Take screenshots to document the results of your findings. You might have multiple plans to try out before reaching a conclusion, and it is important to keep track of your findings. Share your findings with your teammates.
Propose a solution or a few options that would resolve the bug, if possible.
Based on the proposed solutions, estimate the effort required and value, and prioritize the bug.

This is usually involves a discussion with the product managers or anyone else on the team who will weigh in on the decisions. Some problems are considered low priority, and it’s ok to decide it is not worth sinking more time into them.
Mark the resolution status of the ticket.

It helps to think out loud and bounce your thoughts with your teammates along each step of the way. They will have context and perspectives that you won’t have. I like to post my thought process in jira comments, or on the team’s slack channel, as I work through them. That way, it is visible, and if my coworkers see my comments, they can point out the things I haven’t thought about, which helps us resolve the problem faster.

Let’s get right to the root cause of the error message “field 'NewValue' cannot be filtered in a query call”. I’ll walk through each step briefly, and go in detail in Step 4 to show you how you can reproduce this error message in Salesforce.

1. Read Description Of Ticket. Read Error Message.

The ticket that our support representatives prepared describes the actions the customer was taking that led to this bug. Our product has a Salesforce integration, and it was making a query to Salesforce’s SOQL query endpoint.

The error message:

{
    "message": "SELECT Id FROM LeadHistory WHERE NewValue = 'great' ^ ERROR at Row:1:Column:34 field 'NewValue' cannot be filtered in a query call",
    "errorCode": "INVALID_FIELD",
}

2. What I Know and Don’t Know

What I know:

The error happens when the SOQL query is being executed by Salesforce.

What I don’t know:

Why couldn’t the field be filtered in the query call? What is special about this field? Why were some fields able to be filtered on, but not this one?
What is a LeadHistory? What is a “History Object”? Our product manager mentioned something about “History Objects”, and I was not familiar with this concept, yet.

3. Hunches And Plans

A quick internet search on the error message led me to Salesforce’s documentation on encrypted fields. The INVALID_FIELD error code made it seem possible that there was an encrypted field at play. However, after the support representative on my team checked with the customer, it was ruled out.

I had a few hunches at this point:

The field was a TextArea field.
The field was an encrypted field.
The History Object has different properties than non-History Objects.

After my initial internet search, I asked the support representative to find out if the customer’s salesforce field was a TextArea field or an encrypted field. It turns out it was neither. However, the field was a History Object field. So, I was able to rule out the first two hunches, and follow the investigation on the third.

A quick internet search didn’t yield any documentation about filtering on History Object fields. (I may not have searched enough.) I set out to reproduce the error myself, which leads us to Step 4.

4. How To Reproduce Salesforce Error Message “field 'NewValue' cannot be filtered in a query call”

Sign up for Salesforce developer instance. (I'll leave out the steps for that here.)
Login. I like to use the classic view, so I switch to it. Find it under the Profile Menu > Switch to Salesforce Classic. (The “Profile Menu” is what I’m calling the avatar that you see in the Lightning view, or the user name in the Classic view.)
Open the “Setup” view. The link for that is at the top of the page.
Find the object for which you want to add Field History Tracking. I want to add Field History Tracking to the Lead object. In the left hand panel, I navigate to Build > Customize > Leads > Fields.
This brings up the Lead Fields setup page.
Enable the history tracking on this object by clicking on the “Set History Tracking” button on the Lead Fields setup page. Then mark the checkbox for “Enable Lead History”, and mark the checkboxes for the fields to track their history.
(Optional) Create some Lead objects, and change their fields to create some data that you can query and filter on. (I will skip these steps here, because you can raise the error without data!)
Open the developer console. Find it under Profile Menu > Developer Console. This will open up a new window for the Developer Console, which allows us to test out our SOQL queries.
Go to File > Open Resource.
Search for and select the LeadHistory object. Click the Open button at the bottom right.
You should see the Query Editor tab. Modify the query to include a filter on the NewValue or OldValue. These fields are tracking the changes to the Lead fields that have history tracking turned on.
Click the Execute button at the lower left corner. The error message is reproduced!

5. Propose Solutions

These are the two solutions to address this error:

Don’t allow our product to apply filters on the field whenever making a query on the History Objects. It is fine for our product to apply filters for non-History Objects, just not the History Objects.
Provide a better error message, instead of displaying Salesforce’s error message to the customers.

6. Estimate Effort and Value, And Prioritize

I shared the reproduced bug and the proposed solutions with the product manager, and they were able to make a decision given these options.

7. Mark Resolution Status

I marked the bug ticket as DONE. The product manager and support representatives are now informed of why the bug happened, and what the solution will be. They were able to prioritize the task on the product roadmap, and communicate with customers in a timely manner about the cause and the expected solution delivery date.

TIL

I learned that:

In Salesforce, you can track the history of a field. When you track the history of a field, there is a History Object created to store the changes to that field. The History Object is different from the Object of the field. For example, if the Title field in the Lead object has history tracking turned on, there will be a LeadHistory object. The LeadHistory object will store the history of changes for the Title field in the LeadHistory.OldValue and LeadHistory.NewValue fields.
The Field History Tracking fields cannot be filtered in a query.
Field can be encrypted, and if they are encrypted, they cannot be filtered in a query.
Salesforce’s developer console allows you to test out your SOQL queries.
I didn't learn all the details about Salesforce's History Objects, and it wasn't necessary. As part of triage, it was my job to be efficient. That means doing just enough while being thorough to confidently identify the root cause, propose a solution, and move onto the next bug ticket!

Thanks for reading! I really enjoyed documenting the triage process for this specific error. I hope you enjoyed this post!