I Built an AI to Judge AI Projects. Here's What It Looks For

Alexandru Pop — Fri, 16 Oct 2026 13:00:00 +0000

A GitHub repository can say "Transformer," "RAG," or "Computer Vision" in the README. That does not tell you whether the work is technically deep, original, reproducible, or even really yours. I built AI-Atlas to look past the labels and inspect the evidence inside the code.

There is a surprisingly difficult question hiding behind almost every technical portfolio: how good is this project, actually?

Not how polished the README looks, how many stars it received, whether the repository name contains LLM, Agent, or Computer Vision, or whether its author wrote AI Engineer on LinkedIn. I mean something more specific: what does the implementation actually demonstrate?

Imagine two repositories:

The first fine-tunes a pretrained Transformer using an existing training script, changes a few hyperparameters, reports accuracy, and stops there.
The second modifies the attention mechanism, implements the change manually, compares against several baselines, documents why the modification exists, stores reproducible experiment configurations, and includes an ablation showing whether the idea actually helped.

Both are legitimately Transformer projects. But they are not remotely the same piece of work.

That problem became one of the most interesting parts of building AI-Atlas, the platform I have been developing to turn public GitHub work into an explainable map of demonstrated AI skills. AI-Atlas currently maps projects across Machine Learning, Computer Vision, Data Science, NLP & LLMs, AI Agents, and MLOps, but the domain classification itself is only the beginning.

Once you decide to evaluate the work itself, you inherit a much harder problem: defining what good AI work actually means, distinguishing implementation from imported functionality, separating originality from complexity, and producing a judgment that is specific enough for the developer to inspect and disagree with.

That last part became especially important to me.

If an AI is going to judge someone's work, "the model said so" is not a good enough explanation.

A Project Is More Than Its Topic

AI-Atlas separates public AI work into six evidence-backed domains rather than treating every repository as the same kind of project.

A repository's subject tells us surprisingly little about its quality. Take something as familiar as a ResNet implementation. One developer might import a pretrained architecture from torchvision, replace its classification head, fine-tune it on a dataset, and produce a perfectly functional project.

Another might implement residual blocks independently, reproduce several ResNet variants, compare initialization strategies, build a reusable training pipeline, investigate how shortcut connections affect optimization, and document every experiment.

Both repositories belong under Computer Vision. Their topic is nearly identical. The evidence of technical ability is not.

That is why AI-Atlas does not evaluate a project with one vague question like "Is this impressive?"

Each relevant repository is examined across seven dimensions: Technical Depth, Conceptual Difficulty, Originality/Modification, Completeness, Practical/Research Usefulness, Documentation & Educational Value, and Engineering & Reproducibility.

Those dimensions intentionally measure different things. A project can attempt a difficult idea but remain unfinished. It can be exceptionally well engineered without containing an original research contribution.

A from-scratch educational implementation might be less novel than a new architecture, while still demonstrating considerably more understanding than a thin wrapper around an API. Even a sophisticated experiment becomes harder to trust if there is no reliable way to determine what configuration produced its result.

Suppose AI-Atlas encounters an illustrative repository called vit-from-scratch. It finds attention.py, patch_embedding.py, a complete training loop, configuration files, experiment outputs, and a detailed architecture explanation.

Now imagine another repository with the same title containing a short notebook that instantiates an existing Vision Transformer implementation and fine-tunes it.

Both contain valid AI work, but the first exposes substantially more evidence about the developer's understanding of the architecture, training process, experimental decisions, and engineering choices.

This is the distinction I wanted the evaluator to learn to make.

The interesting question is not:

"Does this repository contain a Transformer?"

It is:

"What did the developer actually do with the Transformer?"

A Score Without Evidence Is Just Another Opinion

AI-Atlas traces conclusions back toward implementation, training, and evaluation evidence rather than relying only on repository descriptions.

This was probably the most important design constraint in the entire system.

Imagine an automated evaluator returning Project Merit: 82.

Why 82? Why not 74, or some other value?
Which part of the repository raised the score?
Which missing component prevented it from reaching 90?

If the explanation is simply "a language model analyzed the repository", the number is almost meaningless. We would have replaced one shallow signal with another, except the new one looks more authoritative.

AI-Atlas therefore separates interpretation from scoring. The evaluator reads the relevant implementation and returns judgments across the rubric together with its reasoning and supporting repository evidence. AI-Atlas then computes Project Merit through versioned scoring rules rather than allowing the language model to invent the final number directly.

Conceptually, the process is much closer to this:

The six-step path from public repository to evidence-grounded Project Merit score.

That separation sounds subtle, but it matters.

If I later decide that Technical Depth should contribute differently to Project Merit, that is a scoring-methodology change.

If the evaluator incorrectly interprets a custom optimizer as ordinary library usage, that is an evidence-evaluation problem. Those are fundamentally different failure modes, and keeping them separate makes the system much easier to debug and criticize.

The same principle applies to the public methodology. AI-Atlas exposes how repositories are discovered, understood, mapped, and explained, and the live site describes the central rule as "implementation, not labels," "depth over repetition," and "claims back to files."

This also means a high-level score is not intended to be the final destination. The interesting path is in the opposite direction: from the profile score to the domain, from the domain to the project, from the project to the rubric judgment, and eventually back toward the code that caused the judgment.

The score becomes useful only when you can trace it back to something real.

Sometimes the Correct Evaluation Is to Exclude the Repository

I even mapped Karpathy's GitHub. The public Atlas compresses repository evidence into a six-domain map, while the underlying projects remain the source of that profile.

One of the easiest ways to make an AI portfolio evaluator appear impressive would be to discover AI expertise everywhere. That is also one of the easiest ways to destroy its usefulness.

Real GitHub profiles contain forks, abandoned experiments, personal websites, coursework, profile READMEs, utilities, duplicate repositories, API wrappers, and projects that simply have nothing to do with artificial intelligence. Treating all of them as positive evidence would make larger and noisier profiles appear stronger almost automatically.

AI-Atlas therefore performs filtering before deep evaluation:

Forks do not become evidence of original work.
Profile repositories do not count as AI projects simply because they describe someone's skills.
Obvious non-AI repositories can be listed as excluded rather than being forced into one of the six domains.

The public site summarizes this as a Discover → Understand → Map → Explain pipeline: first remove obvious noise, then inspect what was actually built, then place genuine evidence into the appropriate domains.

There is also a security reason for keeping this process conservative. Repository contents are treated as untrusted data. AI-Atlas reads public code as text, but it does not clone a stranger's project and start executing installation scripts, Dockerfiles, tests, or arbitrary shell commands. Code is read, never run.

That means the evaluator is working from inspectable evidence rather than trusting the repository to demonstrate itself through execution.

Popularity is deliberately kept separate too

GitHub stars can indicate adoption, and forks can tell us that a project has attracted real usage, but those numbers answer a different question from technical quality.

A simple developer tool can accumulate thousands of stars, while a difficult research reproduction written by one student can receive five.

AI-Atlas therefore keeps the technical rubric based on the project itself, with only a small capped adoption adjustment for projects that cross meaningful usage thresholds.

This matters particularly for students and independent developers. If popularity dominates the score, the evaluator is measuring audience size rather than engineering ability.

What the Evaluator Is Actually Looking For

A technically difficult project can still be incomplete, unoriginal, or difficult to reproduce, so the qualities are scored independently.

1. Technical Depth asks how much meaningful implementation exists beneath the project's headline.

Using PyTorch, Hugging Face, or scikit-learn does not make a project shallow; serious engineering almost always depends on libraries. The question is whether the developer made substantial technical decisions themselves. A repository containing custom model components, data pipelines, evaluation infrastructure, training logic, checkpointing, and careful experimental code demonstrates a different level of implementation depth from a notebook that calls a ready-made pipeline.

2. Conceptual Difficulty concerns the problem itself.

Implementing linear regression, reproducing a Transformer, building a multimodal retrieval system, and designing a distributed agent runtime do not demand the same level of conceptual understanding. At the same time, complexity should not become a reward by itself. A difficult idea implemented incompletely should not receive full credit merely because the README contains advanced terminology.

3. Originality/Modification asks what the developer actually contributed beyond reproduction.

That can mean changing an architecture, developing a custom loss, testing an unusual training strategy, combining methods in a non-trivial way, or designing experiments around a genuine research question. Originality does not require producing a publishable paper. A careful reproduction can still be valuable. But reproducing and modifying demonstrate different abilities, so the evaluator should distinguish them.

4. Completeness asks whether the repository forms a coherent project rather than a collection of promising fragments.

A model implementation with no training entry point, no evaluation, and no instructions may demonstrate technical understanding, but it is still less complete than a repository where the components form an end-to-end experiment.

5. Practical/Research Usefulness asks whether the project accomplishes something meaningful.

Sometimes usefulness means supporting a real application. Sometimes it means testing a research hypothesis or making an important paper easier to understand. I deliberately do not equate usefulness with commercial value.

6. Documentation & Educational Value concerns whether another technical reader can understand the decisions behind the code.

Why was this architecture chosen? Which assumptions were made? What failed? How are the reported results reproduced? Good documentation does not substitute for technical depth, but it makes technical depth inspectable.

7. Engineering & Reproducibility asks whether somebody else can reasonably verify or continue the work.

Configuration files, clear dependencies, random seeds, tests, evaluation scripts, dataset instructions, checkpoint conventions, and stored experiment settings all make a repository considerably stronger.

The seven dimensions AI-Atlas uses to reconstruct what an AI repository actually demonstrates.

Not every project requires every item, but their presence communicates something important: the developer thought beyond making the code work once on their own machine.

That is ultimately what the evaluator is trying to reconstruct. Not a list of technologies, but the shape of the engineering demonstrated through their use.

The Most Useful Evaluation Should Tell You What to Improve

Verified owners can go beyond the public score and inspect the evaluator's reasoning, evidence, and the project where improvement would matter most.

Once I had a scoring system, I became increasingly interested in a different question.

Suppose the evaluator decides that a project is technically strong but its Engineering & Reproducibility is considerably weaker than the rest. What should the developer actually do with that information?

The least useful answer would be "raise the score." That turns the entire system into a game. A much better question is: what real change to the repository would make the project better?

That is the idea behind the owner-only Start here recommendation. It does not simply select the repository with the lowest score.

Instead, AI-Atlas looks for the project where the available rubric headroom represents the most meaningful lever for improving the underlying Project Merit. A project sitting at 40 because it is fundamentally small may have less useful growth potential than an already serious project whose experiments are difficult to reproduce.

From there, the optional roadmap can turn that weakness into 3–5 concrete improvements ranked by impact. Those improvements can be exported as a checklist or opened as GitHub issues, which turns the evaluation into actual repository work rather than another dashboard metric.

The Merit Predictor explores the same idea from a narrower direction.

Instead of asking "How do I improve this entire project?", you can choose one rubric dimension and ask what a realistic improvement might look like. The system can estimate a target, describe the work required, and show the resulting hypothetical Project Merit.

That final word is important.

A prediction does not change the real score. A generated roadmap does not change the real score either. Only modifying the project and rescanning new evidence can change what AI-Atlas says the repository demonstrates.

So the workflow I find most interesting is:

AI-Atlas's improvement loop.

At that point, the evaluator becomes less like a ranking machine and more like a feedback loop between what you think you built and what your public repository can actually prove.

Final Thoughts

Leaderboards make the profiles discoverable, but the ranking sits on top of project evidence rather than replacing it.

AI evaluation sounds simple until you actually try to define what should be evaluated.

A repository contains code, but code alone does not immediately tell us whether the project is difficult, original, complete, reproducible, or useful. Those judgments require interpretation. And the moment AI begins interpreting somebody's work, I think the standard should become more transparent, not less.

That is the principle I am trying to preserve with AI-Atlas. The score should never be the end of the explanation.

Ideally, you should be able to travel from the overall Atlas score to a domain, from that domain to the projects that contributed to it, from the project to an individual rubric judgment, from the judgment to its reasoning, and eventually toward the files that caused the evaluator to reach that conclusion.

AI-Atlas's traceability path.

That chain matters more to me than making the number appear authoritative.

It is also why I do not think AI-Atlas should be read as an automated hiring verdict. A recruiter should not see one developer at 84 and another at 81 and conclude that the first developer is objectively better. That would introduce exactly the kind of false precision the evidence system is supposed to reduce.

A much more useful workflow is: this developer claims NLP & LLM experience; these repositories support that claim; this is what the evaluator found technically significant; and these are the files worth inspecting manually.

The application makes technical work easier to read. It does not remove the need for human judgment.

There are also things AI-Atlas simply cannot know:

It evaluates public repositories. If someone's most sophisticated work is private, there is no public evidence for the Atlas to inspect.
If someone understands a field deeply but has never demonstrated that knowledge through public work, the system cannot infer that capability responsibly. And even when reasoning is tied to code evidence, the rubric judgments remain AI-generated estimates that can be wrong.

That limitation is important. An evaluator becomes much less useful the moment it pretends to know more than the available evidence allows.

But I still think the underlying problem is worth solving. GitHub contains one of the richest public records of technical work ever created, yet we often summarize a profile with stars, followers, contribution graphs, repository counts, and programming languages.

GitHub holds deep technical evidence, but profiles are often reduced to surface metrics.

Those are useful signals, but they answer different questions. The question I want AI-Atlas to answer is:

What does the work itself demonstrate?

Try it, and tell me where it's wrong

If you have public AI repositories, you can map your GitHub for free on AI-Atlas. You do not need an account for the initial scan.

Claiming the resulting Atlas lets you inspect the deeper evaluator evidence, join the global, country, and domain leaderboards, use the self-updating GitHub badge and Atlas cards, and work through the project's improvement tools.

And if you try it, I am particularly interested in something more useful than hearing that the score looks good:

Find something the evaluator misunderstood, and tell me in the comments.

I am still building AI-Atlas, and difficult repositories are often the best test cases. Every disagreement is another opportunity to make the evaluation more careful, more inspectable, and more grounded in what the code actually demonstrates.

I built AI-Atlas because writing AI Engineer became easy, but I want demonstrating the engineering behind it to matter more.

I'm Alex, a high-school student from Romania building AI-Atlas. You can find me on GitHub, Hugging Face, and Medium, where I write about AI infrastructure, local inference hardware, and the systems behind modern models.

How I built a cinematic site in one HTML file

Alexandru Pop — Wed, 24 Jun 2026 15:57:39 +0000

My collaborator and I have been working on website design and development projects, exploring ways to combine strong branding with engaging user experiences.

As part of our portfolio, we recently created AURELIS, a fictional luxury gemstone and jewelry brand website:

🔗 https://jocular-nasturtium-8889ff.netlify.app/

The project was designed as an experiment in:

Premium branding
Visual storytelling
Elegant UI/UX design
Luxury e-commerce aesthetics
Responsive web experiences

We're continuously learning and improving our craft, so we'd love to hear your thoughts.

Feedback We're Looking For

First impressions of the design
Branding and visual identity
User experience and navigation
Mobile responsiveness
Accessibility considerations
Anything that feels confusing or could be improved

If this were a real business website, what would you change?

Open to New Projects

Beyond portfolio work, we're always interested in collaborating on new web projects and helping businesses, entrepreneurs, creators, and individuals establish a strong online presence.

Whether it's:

A business website
A landing page
A portfolio
An e-commerce store
A custom web solution

we're always open to discussing new ideas and challenges.

If you'd like to connect, collaborate, or simply share feedback on our work, feel free to reach out. We'd love to hear from you.

DEV Community: Alexandru Pop