How to develop a Optical Design Software like zemax or codev

I am trying to develop an open source optical design software. I hope to use C/C++ to write the core of ray tracing, aberration calculation, and simulated imaging mtf aberration analysis chart, and retain the python interface for later plug-ins and optimization modules. Then use the front-end and back-end separation mode to use QT to write the front-end interactive interface. I don’t want to rely on commercial projects such as matlab or mathmatica. Has anyone done similar work before? Are there any good dependencies or library solutions to recommend?

At the beginning I tried to use python to use a simple ray matrix AND to simulate the optical elements, and then hoped to use matplotlib to plot it, but this was not easy to use and there was no room for expansion.