DEV Community

Tracy Gilmore
Tracy Gilmore

Posted on

AI in Mathematics

Observers of developments in the fields of Mathematics and AI will no-doubt be aware of recent announcements from OpenAI and Anthropic. According to reports three of the seven The Clay Mathematics Institute Millennium Prize Problems (Navier–Stokes, Riemann hypothesis and, Yang–Mills) have been proven using AI tools, leaving much of the Mathematics community in termoil. For many mathematicians the many pages of generated proof and more hindrance to the endeavour than help because they "give the answer away" so to speak. Many would argue the pursuit is as, if not more, important than the outcome because it reveals new theories.

The mathematics world now have a different job at hand, to validate the tens of pages of proofs to confirm or refute the claims of the AI companies. Whilst few would dispute the veracity of OpenAI/Anthropic's claims, the activity will take considerable human effort, could prove to be false, and is unlikely to contribute as much to current mathematical knowledge than if humans had devised the proof from their own brains.

How does this relate to Software Engineering? I see a correlation (but to a much lesser degree) between unit tests and the mathematical proofs. In deed, I prefer to call the method of code verification "unit proofs" rather than tests, because the serve a very different purpose to most other forms of software testing.

Instead of confirming requirements have been met or loads can be accommodated, unit tests/proofs confirm the unit (of code) conforms to the developer's expectation of behaviour and/or structure. This serves a number of valuable functions in software engineering:

  • It provides a form of documentation for those developers intent on including the unit in their code. Consequently, the unit tests need to be as complete and up to date as possible at all times to ensure developer confidence.
  • Unit test usually exercise just the public interface of a unit, with the primary purpose being to alert the developer of breaking changes. Indeed, it is arguable if changing code with unit tests can be considered refactoring, or just editing code.
  • How often have you written some code (by hand, to qualify in these days of AI-generated code) and come back to it weeks or even just a few days later and been puzzled by your own actions. Now image the code was not even what you wrote or has been revised by another in the meantime. Commenting your code will not necessarily help and is largely considered bad practice these days. What will help is a well curated and managed collection of unit tests.

As an aside, I tend to use coverage analysis, not as a measure of testing completeness but as a tool to indicate of where more in-depth unit testing is required. We seldom see measures exceeding 100% but, in my personal opinion, coverage is an area where we should. For me 100% coverage is not gold but the platinum standard, with diamond only achievable through Formal Verification methods. Often 80% is the gold standard at it satisfies the contractural requirement, but it does not stop me pushing a little further to try and hit 100%.
I have often found that extra 20% hides poor implementation, abstraction or a flawed application of a design pattern that a little refactoring makes the unit far more maintainable.

Back to the troubled in the Maths World. It now transpires that when AI is used to transcribe the proof from natural language into formal notation it has a tendency to makes things up if it cannot resolve an issue. In other words the process of 'autotranslation' from natural language into the LEAN programming language cannot be trusted, this doubling the task of confirmation.

Just because the LEAN program runs doesn't mean it is proving what you think, it needs to be checked by a human.

Top comments (0)