DEV Community

Fenju Fu
Fenju Fu

Posted on

Agents Can Code But Can't See: Why Multimodal Perception Is the Missing Skill Layer

Today's GitHub Trending tells a clear story: agent skills are going vertical.

Google Labs shipped stitch-skills — a standardized skill library for coding agents. tradermonty/claude-trading-skills gives Claude Code market analysis and charting capabilities. elementalsouls/Claude-BugHunter packs 82 security testing skills. NVIDIA's SkillEvaluator provides a framework to assess skill quality.

The pattern is obvious: developers are moving from "build a general agent" to "mount specific skills onto agents." The era of the jack-of-all-trades agent is ending.

But here's the gap we kept hitting in production:

The Perception Problem

Our agents could write code, plan multi-step workflows, and generate reports. But the moment real-world multimodal input showed up, they hit a wall:

  • A user sends a voice message → the agent can't transcribe it
  • A colleague uploads a scanned contract → the agent can't extract the text
  • A partner sends an email in Japanese → the agent can't translate it
  • A document has typos and formatting issues → the agent can't proofread it

The agent isn't dumb. It just lacks the perception layer — the ability to hear, see, and understand across languages and modalities.

Why Coding Skills Aren't Enough

Look at today's trending repos:

Repo Skill Type Gap
google-labs-code/stitch-skills Coding/dev skills No multimodal perception
tradermonty/claude-trading-skills Financial analysis No document OCR
elementalsouls/Claude-BugHunter Security testing No voice input
NVIDIA/SkillEvaluator Skill evaluation Evaluates, doesn't provide skills

Each covers a vertical domain. None covers the horizontal capability that every vertical agent needs: multimodal perception.

The iFly-Skills Approach

This is exactly the gap iflytek/iFly-Skills fills. It's iFLYTEK's official skill collection covering:

  • Speech recognition — transcribe voice messages, meeting recordings, support calls
  • OCR — extract text from scanned documents, screenshots, handwritten notes
  • Translation — cross-language communication for multinational teams
  • Text proofreading — catch typos, grammar issues, and formatting errors
  • Multimodal understanding — process inputs that combine text, image, and audio

iFly-Skills project overview

Unlike community-built skill packs, iFly-Skills comes from iFLYTEK — a company that has been building speech and language technology for over two decades. These aren't wrapper APIs slapped together over a weekend; they're production-grade capabilities battle-tested in real products.

Pairing Perception with Planning

Perception alone isn't enough. An agent that can hear but can't plan is just a transcription tool. That's why iFly-Skills pairs naturally with iflytek/astron-agent — an enterprise-grade agentic workflow platform:

User sends voice message (Chinese)
  → iFly-Skills: speech recognition transcribes to text
  → astron-agent: understands intent, plans response
  → iFly-Skills: translates response to English
  → astron-agent: sends reply and logs to CRM
Enter fullscreen mode Exit fullscreen mode

The full loop: perceive → understand → plan → execute → feedback.

OCR and image understanding tools

The Takeaway

As agent skills proliferate (today's trending proves they will), the differentiator won't be "who has the most coding skills." It'll be "whose agent can actually operate in the real world" — where input comes as voice, images, and foreign languages, not just clean text.

If your agent can code but can't understand a voice message, it's not production-ready.

Explore iFly-Skills: https://github.com/iflytek/iFly-Skills

Explore astron-agent: https://github.com/iflytek/astron-agent

Top comments (0)