Introduction
Resume screening often looks straightforward: hand a resume to an AI, ask if the candidate fits, get a yes or no. But if you've ever felt uneasy about how confidently these systems can talk without showing their work, you're not alone. In this post, I want to share an approach that feels more like a seasoned engineer explaining a tough problem straightforward, honest, and practical rather than a polished marketing pitch.
What we're really tackling is this: How can AI truly assess a resume without falling into the trap of just echoing buzzwords? How do we know the candidate isn't just sprinkling keywords but actually delivering on those claims? The answer is a layered architecture that separates understanding from evaluation, and it's built around two key players: a Large Language Model (LLM) and Jev System One.
When AI Sounds Confident but the Evidence Says Otherwise
Imagine you feed a resume into an LLM and ask, “Is this person an AI Architect?” The model might confidently say yes, citing cloud-native systems, AI, Kubernetes, and so on. Sounds great, right? But dig a little deeper and you'll find the resume might never say they designed an end-to-end AI architecture or managed latency and cost tradeoffs. The model is connecting dots it hopes are there, not dots explicitly proven.
Here's the catch: keywords like AWS, AI, LLM are not evidence. They're signals, yes, but they don't prove ownership, depth, or impact. Compare these two snippets:
- Resume A: “Worked with OpenAI APIs.”
- Resume B: “Designed and deployed an enterprise RAG platform using OpenAI models, Azure AI Search, and PostgreSQL, reducing response latency by 38%.”
Both mention AI-related terms, but only one shows real, measurable achievement. Our system needs to see evidence, not just buzzwords.
The Role of the LLM: Translator, Not Judge
LLMs shine at turning messy, varied human language into structured, meaningful data. Resumes are full of inconsistent terms, abbreviations, implicit meanings, and nuanced project descriptions that simple rules can't handle. The job of the LLM here is to extract that structured evidence things like project ownership, technologies used, outcomes achieved not just a list of skills.
For example, instead of just “Python, AWS, AI,” the LLM might output:
{
"project": "Enterprise Knowledge Platform",
"role": "Lead Architect",
"ownership": ["system architecture", "AI service architecture", "RAG pipeline"],
"technologies": ["Python", "OpenAI", "Azure"],
"ai_capabilities": ["LLM", "RAG"],
"production": true,
"impact": {"latency_reduction": "38%"}
}
That's the kind of evidence that can actually be evaluated.
Tokenize Evidence, Not Just Language
Traditional resume parsers see keywords. We want them to see relationships and ownership. Instead of just “AWS” or “AI,” we get:
- Role = Lead Architect
- Ownership = End-to-End AI System
- System = RAG Platform
- Impact = 38% latency reduction
This structured evidence gives us something concrete to evaluate.
Enter Jev System One: The Evidence Evaluator
Jev is the engine that takes this evidence and scores it against explicit questions and criteria. For example:
How much production AI architecture experience is demonstrated?
0 = No evidence
1 = Experimental work
…
5 = Enterprise-scale AI architecture ownership
Instead of a vague yes/no from the LLM, Jev provides a clear, reproducible score based on well-defined criteria. This transforms the question from “Is this person an AI Architect?” to “What does the evidence say about their AI architecture experience?”
Architecture
Demo
State
{
"resume": "SASHA BERNOULLI\nSan Francisco, CA | sasha.bernoulli@email.com | github.com/sashabernoulli | linkedin.com/in/sashabernoulli\n\nPROFESSIONAL SUMMARY\nExperienced Product Engineer building developer-focused tools and platforms. Expertise in full-stack development with deep specialization in frontend architecture and UI/UX for technical audiences. Proven track record of shipping products that increase developer productivity and improve developer experience.\n\nEXPERIENCE\n\nSenior Product Engineer | CloudSync Systems | San Francisco, CA | Jan 2022 - Present\n- Led frontend architecture redesign for cloud orchestration dashboard, reducing initial load time by 65% and improving TypeScript coverage from 42% to 98%\n- Designed and implemented real-time collaboration features using WebSockets and Operational Transformation for multi-developer workflows\n- Built internal API gateway and request optimization layer (Node.js/Express) that reduced backend calls by 40%, improving dashboard responsiveness\n- Mentored 3 junior engineers on frontend best practices and code quality standards\n- Tech Stack: React, TypeScript, Redux, Node.js, PostgreSQL, AWS\n\nProduct Engineer | DevTools Lab | San Francisco, CA | May 2021 - Dec 2021\n- Architected and launched IDE plugin marketplace with 50k+ downloads; designed Vue.js frontend with Electron integration\n- Implemented backend services for plugin discovery, versioning, and analytics (Python/FastAPI) handling 2M+ monthly requests\n- Optimized plugin installation pipeline, reducing time from 45s to 8s through lazy-loading and caching strategies\n- Built comprehensive monitoring and error tracking for frontend and backend systems using Datadog and custom logging\n- Tech Stack: Vue.js, Python, FastAPI, PostgreSQL, Redis, Docker\n\nSoftware Engineer | Nexus Networks | San Francisco, CA | Jan 2021 - Apr 2021\n- Developed interactive network topology visualization tool using D3.js and WebGL for rendering 10k+ nodes in real-time\n- Created REST API endpoints for network state management and device configuration (Go/Gin framework)\n- Implemented real-time data sync between frontend and backend using gRPC, reducing latency by 50%\n- Contributed to SDK documentation and developer guides for third-party integrations\n- Tech Stack: React, D3.js, Go, PostgreSQL, Kubernetes\n\nJunior Software Engineer | CodePath Systems | San Francisco, CA | Jun 2020 - Jan 2021\n- Built responsive web interfaces for code analysis tools using React and CSS-in-JS\n- Developed backend microservices for code parsing and analysis (Java/Spring Boot)\n- Optimized database queries reducing API response times by 30%\n- Tech Stack: React, JavaScript, Java, Spring Boot, MySQL\n\nSKILLS\n\nFrontend: React, Vue.js, TypeScript, JavaScript (ES6+), HTML5, CSS3, D3.js, WebGL, Webpack, Tailwind CSS, Material-UI\nBackend: Node.js, Python (FastAPI), Go, Java (Spring Boot), SQL (PostgreSQL, MySQL), Redis, MongoDB\nDeveloper Tools: Git, Docker, Kubernetes, GitHub Actions, Datadog, New Relic\nSpecializations: Developer Experience, API Design, Real-time Systems, Performance Optimization, UI/UX for Technical Users\n\nEDUCATION\n\nB.S. Computer Science | University of California, Berkeley | 2019\nRelevant Coursework: Data Structures, Algorithms, Systems Design, Databases, Web Development\n\nCERTIFICATIONS & ACHIEVEMENTS\n- AWS Certified Solutions Architect (Associate) - 2021\n- Open Source Contributor: React Query (20+ merged PRs), Electron (5+ merged PRs)\n- Speaker: \"Building Developer-First UI\" at React Conference 2022"
}
Questions
{
"years_of_experience": {
"type": "score",
"instructions": {
"question": "How many years of professional experience does the candidate have?",
"today": "2026-09-15"
},
"criteria": [
"None",
"2 years",
"4 years",
"6 years",
"8 years",
"10+ years"
]
},
"ai_architecture_depth": {
"type": "score",
"instructions": "Rate demonstrated AI architecture ownership from production work. Evaluate architecture, ownership, complexity, scale and measurable impact. Ignore titles, companies and skill lists. When uncertain, choose lower.",
"criteria": [
"No AI architecture experience.",
"AI prototypes, coursework or experiments without production ownership.",
"Implemented isolated AI/ML features within an existing architecture.",
"Designed and shipped production AI components such as LLM applications, RAG, inference services or ML pipelines.",
"Designed end-to-end AI systems spanning models, data, retrieval, orchestration, APIs, infrastructure and production operations.",
"Enterprise-scale AI architecture ownership across systems/domains with platform strategy, architecture standards and organization-wide technical leadership."
]
},
"llm_experience": {
"type": "noul",
"instructions": "Has the candidate built production LLM-powered products, services or features?",
"criteria": {
"true": "Production LLM product or feature demonstrated.",
"false": "No production LLM product or feature demonstrated."
}
},
"rag_experience": {
"type": "noul",
"instructions": "Has the candidate designed or implemented a production RAG or retrieval system?",
"criteria": {
"true": "Production RAG/retrieval system demonstrated.",
"false": "No production RAG/retrieval system demonstrated."
}
},
"agent_experience": {
"type": "noul",
"instructions": "Has the candidate built AI agents or tool-using AI workflows?",
"criteria": {
"true": "AI agent or tool-using workflow demonstrated.",
"false": "No AI agent experience demonstrated."
}
},
"model_engineering": {
"type": "score",
"instructions": "Rate experience selecting, evaluating, fine-tuning, serving or optimizing AI/ML/LLM models.",
"criteria": [
"No model engineering experience.",
"Consumed existing AI APIs/models only.",
"Implemented model inference or basic model integration.",
"Performed model evaluation, prompt optimization, embeddings, fine-tuning or model serving.",
"Designed model strategy including selection, evaluation, fine-tuning, inference, cost and latency optimization.",
"Advanced model/platform engineering including distributed training, quantization, inference optimization or large-scale model serving."
]
},
"ai_data_architecture": {
"type": "score",
"instructions": "Rate AI data architecture experience including ingestion, processing, embeddings, vector stores, knowledge bases, retrieval, data quality and governance.",
"criteria": [
"No AI data architecture.",
"Basic datasets for AI experimentation.",
"Basic data preparation, embeddings or vector search.",
"Production AI data or retrieval pipelines.",
"End-to-end AI data architecture covering ingestion, transformation, embeddings, storage, retrieval and governance.",
"Large-scale enterprise AI data platform architecture with multimodal data, lineage, quality and high-volume retrieval."
]
},
"cloud_ai_architecture": {
"type": "score",
"instructions": "Rate production AI infrastructure experience across cloud, containers, orchestration, compute, GPU, storage, deployment and observability.",
"criteria": [
"No AI infrastructure experience.",
"Basic cloud deployment.",
"Deployed AI services using cloud or containers.",
"Designed production AI services with cloud, APIs and monitoring.",
"Designed scalable AI infrastructure including compute/GPU, orchestration, networking, storage, CI/CD and observability.",
"Enterprise AI infrastructure across environments with scalability, resilience, security, cost and performance optimization."
]
},
"ai_platform_experience": {
"type": "noul",
"instructions": "Has the candidate built reusable AI platforms, shared AI services, model gateways, AI APIs or internal AI infrastructure?",
"criteria": {
"true": "Reusable AI platform or shared AI infrastructure demonstrated.",
"false": "No reusable AI platform experience demonstrated."
}
},
"ai_security_governance": {
"type": "score",
"instructions": "Rate AI security and governance experience including privacy, access control, model security, prompt security, auditability, responsible AI and compliance.",
"criteria": [
"No AI security or governance evidence.",
"General application or cloud security only.",
"Basic AI access control or data protection.",
"Production AI security, privacy or governance controls.",
"AI security and governance architecture covering data, models, prompts, access, auditing and compliance.",
"Enterprise AI security and governance standards across multiple AI systems."
]
},
"ai_observability": {
"type": "noul",
"instructions": "Has the candidate implemented production AI observability covering quality, latency, cost, usage, failures, tracing or drift?",
"criteria": {
"true": "Production AI observability demonstrated.",
"false": "No AI observability demonstrated."
}
},
"ai_evaluation": {
"type": "noul",
"instructions": "Has the candidate implemented systematic AI/LLM evaluation?",
"criteria": {
"true": "AI/LLM evaluation using datasets, benchmarks, human evaluation or automated metrics demonstrated.",
"false": "No systematic AI/LLM evaluation demonstrated."
}
},
"ai_cost_optimization": {
"type": "noul",
"instructions": "Has the candidate optimized AI/LLM infrastructure or inference cost?",
"criteria": {
"true": "AI/LLM cost optimization demonstrated.",
"false": "No AI/LLM cost optimization demonstrated."
}
},
"architecture_tradeoffs": {
"type": "score",
"instructions": "Rate demonstrated AI architecture tradeoffs involving model, accuracy, latency, scalability, cost, reliability, security and maintainability.",
"criteria": [
"No architecture tradeoff evidence.",
"Basic implementation choices only.",
"Localized technical tradeoffs.",
"Production architecture tradeoffs across multiple components.",
"Explicit AI tradeoffs across model, data, infrastructure, cost, latency, security and reliability.",
"Defines organization-wide AI architecture principles and platform tradeoffs."
]
},
"technical_leadership": {
"type": "score",
"instructions": "Rate technical leadership through architecture ownership, design reviews, mentoring, standards, technical direction and cross-team influence.",
"criteria": [
"No technical leadership evidence.",
"Limited technical collaboration.",
"Component or feature-level technical ownership.",
"Leads projects or technical initiatives.",
"Leads architecture across teams and mentors engineers.",
"Defines enterprise AI architecture strategy and influences multiple engineering organizations."
]
},
"mentorship_demonstrated": {
"type": "noul",
"instructions": "Does the resume demonstrate mentoring or coaching engineers?",
"criteria": {
"true": "Mentoring or coaching demonstrated.",
"false": "No mentoring or coaching demonstrated."
}
},
"ai_product_experience": {
"type": "noul",
"instructions": "Has the candidate built AI capabilities used by customers, users or business operations?",
"criteria": {
"true": "Production AI capability with demonstrated users or business usage.",
"false": "No demonstrated production AI product usage."
}
},
"career_progression": {
"type": "choice",
"instructions": "What career progression pattern is demonstrated?",
"criteria": {
"steady_growth": "Increasing technical scope, ownership and seniority.",
"architecture_growth": "Progression toward architecture and technical leadership.",
"lateral_moves": "Similar responsibility across organizations.",
"job_hopping": "Frequent moves with short tenure.",
"unclear": "Progression cannot be determined."
}
},
"primary_talent_profile": {
"type": "choice",
"instructions": "Select the primary AI/engineering profile from demonstrated experience. Weight recent production work most heavily. Ignore titles and skill lists.",
"criteria": {
"ai_architect": "End-to-end AI architecture; model, data, infrastructure, security, scalability and cost tradeoffs.",
"ml_ai_engineer": "Model training, fine-tuning, evaluation, inference or ML engineering.",
"llm_engineer": "LLM applications, RAG, agents, inference, evaluation and orchestration.",
"ai_platform_engineer": "Reusable AI platforms, model gateways, infrastructure, deployment and observability.",
"data_ai_engineer": "AI data pipelines, embeddings, vector systems and data platforms.",
"cloud_architect": "Cloud and distributed-system architecture with limited AI specialization.",
"software_architect": "Large-scale software architecture without strong AI specialization.",
"research_engineer": "AI/ML research, experimentation, training and novel algorithms.",
"other": "Other engineering profile."
}
}
}
Jev Response
{
"model": "jev-1.13.0",
"answers": {
"years_of_experience": {
"type": "score",
"score": 2.98,
"legend": {
"0": "None",
"1": "2 years",
"2": "4 years",
"3": "6 years",
"4": "8 years",
"5": "10+ years"
},
"confidence": 0.98,
"probabilities": {
"0": 0,
"1": 0,
"2": 0.02,
"3": 0.97,
"4": 0.01,
"5": 0
},
"stats": {}
},
"ai_architecture_depth": {
"type": "score",
"score": 0,
"legend": {
"0": "No AI architecture experience.",
"1": "AI prototypes, coursework or experiments without production ownership.",
"2": "Implemented isolated AI/ML features within an existing architecture.",
"3": "Designed and shipped production AI components such as LLM applications, RAG, inference services or ML pipelines.",
"4": "Designed end-to-end AI systems spanning models, data, retrieval, orchestration, APIs, infrastructure and production operations.",
"5": "Enterprise-scale AI architecture ownership across systems/domains with platform strategy, architecture standards and organization-wide technical leadership."
},
"confidence": 1,
"probabilities": {
"0": 1,
"1": 0,
"2": 0,
"3": 0,
"4": 0,
"5": 0
},
"stats": {}
},
"llm_experience": {
"type": "noul",
"noul": 0.04,
"stats": {}
},
"rag_experience": {
"type": "noul",
"noul": 0.06,
"stats": {}
},
"agent_experience": {
"type": "noul",
"noul": 0.03,
"stats": {}
},
"model_engineering": {
"type": "score",
"score": 0,
"legend": {
"0": "No model engineering experience.",
"1": "Consumed existing AI APIs/models only.",
"2": "Implemented model inference or basic model integration.",
"3": "Performed model evaluation, prompt optimization, embeddings, fine-tuning or model serving.",
"4": "Designed model strategy including selection, evaluation, fine-tuning, inference, cost and latency optimization.",
"5": "Advanced model/platform engineering including distributed training, quantization, inference optimization or large-scale model serving."
},
"confidence": 1,
"probabilities": {
"0": 1,
"1": 0,
"2": 0,
"3": 0,
"4": 0,
"5": 0
},
"stats": {}
},
"ai_data_architecture": {
"type": "score",
"score": 0,
"legend": {
"0": "No AI data architecture.",
"1": "Basic datasets for AI experimentation.",
"2": "Basic data preparation, embeddings or vector search.",
"3": "Production AI data or retrieval pipelines.",
"4": "End-to-end AI data architecture covering ingestion, transformation, embeddings, storage, retrieval and governance.",
"5": "Large-scale enterprise AI data platform architecture with multimodal data, lineage, quality and high-volume retrieval."
},
"confidence": 1,
"probabilities": {
"0": 1,
"1": 0,
"2": 0,
"3": 0,
"4": 0,
"5": 0
},
"stats": {}
},
"cloud_ai_architecture": {
"type": "score",
"score": 0.4,
"legend": {
"0": "No AI infrastructure experience.",
"1": "Basic cloud deployment.",
"2": "Deployed AI services using cloud or containers.",
"3": "Designed production AI services with cloud, APIs and monitoring.",
"4": "Designed scalable AI infrastructure including compute/GPU, orchestration, networking, storage, CI/CD and observability.",
"5": "Enterprise AI infrastructure across environments with scalability, resilience, security, cost and performance optimization."
},
"confidence": 0.73,
"probabilities": {
"0": 0.62,
"1": 0.37,
"2": 0.01,
"3": 0,
"4": 0,
"5": 0
},
"stats": {}
},
"ai_platform_experience": {
"type": "noul",
"noul": 0.03,
"stats": {}
},
"ai_security_governance": {
"type": "score",
"score": 0.03,
"legend": {
"0": "No AI security or governance evidence.",
"1": "General application or cloud security only.",
"2": "Basic AI access control or data protection.",
"3": "Production AI security, privacy or governance controls.",
"4": "AI security and governance architecture covering data, models, prompts, access, auditing and compliance.",
"5": "Enterprise AI security and governance standards across multiple AI systems."
},
"confidence": 0.98,
"probabilities": {
"0": 0.98,
"1": 0.02,
"2": 0,
"3": 0,
"4": 0,
"5": 0
},
"stats": {}
},
"ai_observability": {
"type": "noul",
"noul": 0.03,
"stats": {}
},
"ai_evaluation": {
"type": "noul",
"noul": 0.02,
"stats": {}
},
"ai_cost_optimization": {
"type": "noul",
"noul": 0.03,
"stats": {}
},
"architecture_tradeoffs": {
"type": "score",
"score": 0.01,
"legend": {
"0": "No architecture tradeoff evidence.",
"1": "Basic implementation choices only.",
"2": "Localized technical tradeoffs.",
"3": "Production architecture tradeoffs across multiple components.",
"4": "Explicit AI tradeoffs across model, data, infrastructure, cost, latency, security and reliability.",
"5": "Defines organization-wide AI architecture principles and platform tradeoffs."
},
"confidence": 0.99,
"probabilities": {
"0": 1,
"1": 0,
"2": 0,
"3": 0,
"4": 0,
"5": 0
},
"stats": {}
},
"technical_leadership": {
"type": "score",
"score": 3.34,
"legend": {
"0": "No technical leadership evidence.",
"1": "Limited technical collaboration.",
"2": "Component or feature-level technical ownership.",
"3": "Leads projects or technical initiatives.",
"4": "Leads architecture across teams and mentors engineers.",
"5": "Defines enterprise AI architecture strategy and influences multiple engineering organizations."
},
"confidence": 0.77,
"probabilities": {
"0": 0,
"1": 0,
"2": 0,
"3": 0.66,
"4": 0.34,
"5": 0
},
"stats": {}
},
"mentorship_demonstrated": {
"type": "noul",
"noul": 0.98,
"stats": {}
},
"ai_product_experience": {
"type": "noul",
"noul": 0.05,
"stats": {}
},
"career_progression": {
"type": "choice",
"choice": "architecture_growth",
"confidence": 0.36,
"probabilities": {
"architecture_growth": 0.49,
"job_hopping": 0.04,
"unclear": 0,
"steady_growth": 0.47,
"lateral_moves": 0
},
"stats": {}
},
"primary_talent_profile": {
"type": "choice",
"choice": "software_architect",
"confidence": 0.91,
"probabilities": {
"other": 0.06,
"software_architect": 0.92,
"data_ai_engineer": 0,
"cloud_architect": 0.02,
"ai_architect": 0,
"llm_engineer": 0,
"research_engineer": 0,
"ai_platform_engineer": 0,
"ml_ai_engineer": 0
},
"stats": {}
}
},
"usage": {
"input_tokens": 3087,
"output_tokens": 496
},
"request_id": "playground_102ab50788afc5c4af6b3ce8c598a6cfd06",
"evaluation_time_ms": 174.11698400246678
}
Clear, Traceable, and Controlled
One of the biggest wins here is transparency. Every score or label is traceable to explicit pieces of evidence like quotes from the resume or extracted facts. This matters for fairness, auditability, and trust. Recruiters and candidates alike can see the “why” behind the evaluation.
And importantly, once Jev has made its evaluation, the final LLM tasked with writing the report cannot rewrite or inflate scores. It simply explains what's there. The human recruiter then makes the final call.
Why Ownership Matters More Than Exposure
Anyone can say they “worked with AWS.” But did they architect a system? Did they lead the design? Keywords often blur this distinction. Our model focuses on evidence of ownership verbs like “designed,” “implemented,” “led,” and “optimized.” These words signal responsibility and impact, not just presence.
Building an Evidence Matrix, Not a Magic Score
Instead of a mysterious “Candidate Score: 72,” the system produces a matrix showing strengths and gaps:
| Dimension | Evidence | Result |
|---|---|---|
| Software Architecture | Strong | High |
| AI Architecture | Limited | Low |
| RAG | Not Found | Not Demonstrated |
This helps hiring managers see exactly what's supported by the resume and what isn't.
Beyond Resumes: A Universal Pattern
This approach isn't limited to recruiting. Anywhere you have unstructured data and need fair, explainable evaluation clinical documents, contracts, policies this layered architecture applies:
- LLM extracts structured evidence
- Jev evaluates against explicit criteria
- Human makes the final decision
Final Thoughts: Empowering, Not Replacing Humans
The temptation to automate hiring decisions entirely is big. But resumes only tell part of the story. Interviews, references, communication skills, motivation all matter too. This system doesn't replace recruiters; it empowers them with clearer, more trustworthy evidence.
Think of it like a courtroom: The LLM gathers and organizes facts, Jev evaluates those facts against the law, and the judge (the human) makes the final judgment. That separation is what makes the whole process fairer, more transparent, and ultimately more effective.
Top comments (0)