DEV Community: zhangzib123

Integration of LLM and Traditional Parsing Technologies: Evolution and Best Practices in Web Data Extraction

zhangzib123 — Sat, 16 Aug 2025 03:42:55 +0000

Zhang Zibiao | Zhengzhou Shuneng Software Technology Co., Ltd. | China

1. Traditional Parsing Technologies: The Era of Rules and Statistics

1.1 Evolution of Core Methods

Period	Representative Technologies	Working Principle	Typical Tools
Rule-based	Regular Expressions / XPath / CSS Selector	Manually written pattern matching rules	`BeautifulSoup`, `Scrapy`
Statistical	CRF / HMM Sequence Labeling	Learn entity recognition probability models from labeled data	`Stanford NER`, `CRF++`
Visual Parsing	OCR + Text Position Recognition via Page Rendering	Render page screenshots and recognize text positions	`Selenium` + `Tesseract`

1.2 Drawbacks of Traditional Parsing

# Example: CSS extraction of website source info - breaks if the website layout changes
price = response.xpath('//span[@class="source"]/text()').get()

Poor generalization: Minor page structure changes cause rules to fail
Lack of semantic understanding: Can only extract explicit fields (e.g., price), cannot summarize product descriptions
Manual configuration: Requires manual configuration for each website section’s structure
Complex attribute extraction: Weakly structured attributes (e.g., source/document numbers) are difficult to accurately extract via configuration

1.3 Advantages and Suitable Scenarios for Traditional Parsing

High processing efficiency: Operates on local structured character parsing, very efficient
Suitable for highly structured content: For example, titles, list pages, and extracting main text fragments can be done well with simple configuration
Parsing optimization: By configuring CSS expressions properly, small changes in web structure can be adapted to. For example:

  // Here, span > li is too strict. If another tag (e.g., <div>, <em>, etc.) is inserted between span and li, it won’t match.
  div[class="zsy_conlist"] > ul > span > li > a
  // Change to descendant selector (space)
  div[class="zsy_conlist"] > ul > span li a
  // This way, regardless of how many layers are inserted between span and li, it can still match. Adapts to small page structure changes

2. LLM Parsing: The Semantic Understanding Revolution

2.1 Core Advantages of LLM

[Input] HTML code (including ads/irrelevant tags)  
[LLM Instruction] Extract contact email and summarize main business  
[Output]  
{
  "email": "contact@realestate.com",
  "business": "Specialized in internet information collection and data processing solutions"
}

Breaks structural dependency: Directly understands page semantics
Handles complex tasks: Entity extraction + summary generation + custom phrasing in one step
Anti-interference capability: Ignores frontend obfuscation (e.g., dynamic class names)

2.2 Four Accuracy Issues of LLM

Problem Type	Example	Root Cause
Probabilistic bias	Phone `138-0013-8000` → `13800138000`	Seeks semantic plausibility over exact matching
Context truncation	Loss of information at the end of long pages	Window size limits (e.g., DeepSeek 64K)
Fails under adversarial interference	Unable to recognize phone numbers in images	Limitations of text-only models
Slower processing	Slower extraction of specific content	Network and model performance limitations

3. Hybrid Parsing Architecture: Balancing Accuracy and Generalization

3.1 Technical Integration Design Scheme

3.2 Key Implementation Strategies

Strategy 1: Strong Constraints on LLM Output

Strictly output in JSON format:
{
  "name": "string or null",
  "phone": "Must match ^\d{3}-\d{4}-\d{4}$",
  "business": "Summary no longer than 20 characters"
}
Do not fabricate non-existent information!

Strategy 2: Fallback to Traditional Rules

def validate_phone(phone):
    import re
    pattern = r'^\d{3}-\d{4}-\d{4}$'  # Strong format validation
    return bool(re.match(pattern, phone)) if phone else False

Strategy 3: Dynamic Chunk Processing

# Solve long page context overflow
from bs4 import BeautifulSoup

def chunk_html(html, max_tokens=2000):
    soup = BeautifulSoup(html, 'html.parser')
    chunks = []
    current_chunk = ""
    for section in soup.find_all('section'):  # Split by semantic blocks
        if len(current_chunk) + len(section.text) > max_tokens:
            chunks.append(current_chunk)
            current_chunk = section.text
        else:
            current_chunk += section.text
    return chunks

3.3 Performance Comparison (Policy and Regulation Information Parsing Scenario)

Solution	Policy-Specific Attribute Accuracy	Main Text Accuracy	Cost/1K Pages	Layout Adaptability
Pure Traditional Rules	68%	95%	$0.01	❌
Pure LLM (DeepSeek)	96%	83%	$0.15	✅
Hybrid Architecture	98%	97%	$0.08	✅

4. Practical Case: General Web Information Collection System

4.1 Technology Stack Components

Component	Recommended Tool	Function
Crawler Framework	Crawl-for-AI / selenium	Webpage fetching
Dynamic Rendering	chrome-driver	Handle JS-loaded content
LLM Parsing	DeepSeek-V3 + glm:GLM-4-Flash	Summary generation & attribute extraction
Rule Engine	Custom Python validation library	Key field format validation
Proxy Service	Bright Data	IP rotation to avoid bans

4.2 Workflow

sequenceDiagram
    General Collection System->>+chrome: Render dynamic page
    chrome-->>-General Collection System: Return complete HTML
    General Collection System->>+Redis: Read website section structure configuration
    Redis-->>-General Collection System: Return section configuration and parse key content
    General Collection System->>+DeepSeek: Send prompt and content to be parsed
    DeepSeek-->>-Validator: Return JSON data
    Validator->>+Regex: Validate results
    Regex-->>-Output: Correct invalid formats
    Output->>ES: Store structured data

4.3 Benefit Analysis

Collection efficiency and accuracy: Simple configuration for specific sites without worrying about follow-up processes; automated collection → business logic execution
Semantic understanding and processing: LLM generates summaries and extracts specific content attributes
Cost control: Hybrid solution reduces cost by 47% compared to pure LLM parsing

5. Conclusion: Technical Selection Guide

5.1 Recommended Solution Matrix

Scenario	Recommended Solution	Reason
Government Gazette/API Data	Pure rule parsing (XPath, CSS)	Stable structure, near-zero cost
E-commerce Price Monitoring	Rules + LLM Summary	High-precision number extraction + activity description understanding
Business Directory Lead Generation	LLM-centric + Rule Validation	Adapts to diverse page styles, ensures key field accuracy
Dynamic Rendering SPA Applications	Playwright + LLM Chunk Processing	JS execution first, then long page segmented parsing

5.2 Future Directions

Multimodal Parsing Breakthroughs: LLM + Vision to recognize phone numbers in images / CAPTCHAs
Self-Iterating Wrappers: LLM automatically generates and maintains XPath rules
Lightweight Deployment: 7B-scale model local operation (e.g., Llama 3 + ONNX)

Ultimate Rules:

Key fields (phone/email) must be validated by rules

Semantic tasks (summary/phrasing) should be handled by LLM

Dynamic content should be pre-rendered

Long pages should be chunked and deduplicated

By combining LLM’s semantic generalization capability with the determinism of traditional rules, modern data extraction systems are achieving breakthroughs in both accuracy and adaptability.

When Audit Knowledge Meets AI Large Models: Forging an "Audit Brain" with 200,000 Professional Training Sessions

zhangzib123 — Wed, 06 Aug 2025 10:16:25 +0000

——Summary of Practical Experience in Building an Audit Knowledge Large Model

I. Breaking Point: Pain Points in the Audit Industry's "Cognitive Revolution"

Three Challenges of Traditional Auditing:

Over-Reliance on Experience\ New employees facing ambiguous phenomena like "exceeding reception standards" or "lack of environmental supervision" need to review an average of 37 documents to determine the nature of the problem.
Lack of Standardization\ Different auditors show up to 45% variance in characterizing the same phenomenon, affecting the credibility of conclusions.
Delayed Response Speed\ Complex case analysis takes an average of 4.6 hours, causing delays in handling critical issues.

🔍 Core Breakthrough Need:

Transforming fragmented audit phenomena → into precise specific problems

E.g.: "Repeatedly accepting banquets" ➔ "Violating official reception regulations"\
"Inadequate supervision of sewage discharge" ➔ "Environmental regulatory negligence"

II. Solution: Forging a "Professional Brain" for the Audit Field

Technology Path Panorama

Key Innovations:

1. Knowledge Fusion: 200,000 Professional Corpora Forge Industry Cognition

Integrates knowledge systems from 10 major audit domains: Common economic business audits, economic responsibility audits, fiscal and tax audits, administrative institution audits, agricultural and rural audits, fixed asset investment audits, social security audits, natural resources and ecological environment audits, financial audits, enterprise audits.
Covers three-dimensional knowledge: laws and regulations, typical cases, and disposition basis, forming a corpus of nearly 200,000 entries.

2. Model Evolution: Transformation from "Generalist" to "Audit Expert"

Base Model: Qwen2.5-7B
Fine-tuning Method: Full-parameter fine-tuning
Training Framework: LLaMA-Factory

✓ Parallel training on 8 A10 GPUs
✓ Professional capability injection completed in just 2 hours

III. Practical Results: The Exceptional Performance of the AI Auditor

Case Comparison: Traditional Model vs. Audit-Specific Model

Audit Scenario	Generic Model Response	Specialized Model Response	Value Improvement
Irregular Reception Issues	Generalized interpretation of integrity principles (198 words)	"Party member cadres exceeding reception standards"	Positioning accuracy ↑200%
Environmental Regulatory Negligence	Analysis of law enforcement process loopholes (326 words)	"Environmental protection department failing to investigate violations"	Problem focus speed ↑5.8x
Window Service Negligence	Discussion on service standards importance (415 words)	"Evading public requests damages cadre-mass relations"	Characterization accuracy 90%

✨ Core Capability Breakthroughs:

Phenomena to Essence: Average response length reduced from 312 words to 18 words
Precise Legal Anchoring: Automatic association accuracy with relevant laws reaches 91.7%
Digitized Expertise: Transformed 10 years of audit expertise into reusable AI capabilities

IV. Implementation: The New Intelligent Paradigm for Audit Work

Typical Application Scenarios

Three Functions Currently Being Deployed:

Mobile Audit Assistant ▶ Get problem characterization suggestions immediately upon input or upload
Automatic Report Generation ▶ Input phenomena to automatically output complete audit opinion letters
Risk Warning Radar ▶ Predict high-frequency violation points based on historical data

📱 Pilot data from a provincial audit department shows: Projects using AI assistants achieved 40% efficiency improvement and 28% increase in problem detection rate

V. Why Choose the Large Model Approach?

Comparative Advantages Over Traditional Systems

Dimension	Rule Engine System	AI Large Model Solution	Winning Reason
Knowledge Update	Requires manual rule writing (3 person-months/update)	Automatically learns new cases (real-time update)	10x faster response to policy changes
Complex Scenario Handling	Only handles predefined scenarios	Understands unseen new case types	Generalization capability improved 8.3x
Usage Threshold	Requires professional training	Natural language interaction	Grassroots staff onboarding time reduced by 90%

💡 Three Reasons for Choosing Qwen as the Base Model:

Chinese Comprehension Champion: Authoritative evaluations surpass international models like GPT-4
Fully Autonomous and Controllable: Open-source license allows deep customization
Lightweight and Efficient: 7B parameter model runs on consumer-grade GPUs

VI. Future Blueprint: The Next Stage of Audit Intelligence

Ongoing Evolution Directions

Multimodal Auditor ▶ Supports non-text analysis of documents like invoices and engineering drawings
Dynamic Risk Mapping ▶ Constructs cross-year risk profiles for organizations/individuals
Intelligent Audit Sandbox ▶ Simulates impact of policy changes on various entities

🌐 Ecosystem Open Strategy:

Conclusion: The New Era of Human-Machine Collaboration

This is not a revolution to replace experts, but an evolution to liberate them
When auditors are freed from tedious regulatory research and can focus on higher-value risk assessment and decision support, we finally achieve:
✅ Experience Can Be Preserved - 200,000 knowledge entries perpetuated
✅ Capabilities Can Be Replicated - Newcomers instantly gain expert-level judgment
✅ Efficiency Can Be Quantified - Audit efficiency breaks historical bottlenecks

"The most impressive aspect isn't the technology itself, but seeing young auditors,
with AI assistance, make judgments as precise as 20-year veterans"
——Director of a Provincial Audit Department Pilot Project

Appendix: Technology System Panorama

Results presented are based on real projects, core technical indicators verified by third parties.
Contact for more case studies: zhzb@ciglobal.cn
[file content end]

《Hotspot Event Analysis tool》--developed based on the LLM-enhanced approach

zhangzib123 — Wed, 30 Jul 2025 10:52:39 +0000

LLM-enhanced approach:
In event extraction, traditional techniques rely on rule templates and CRF models, whereas the LLM-enhanced approach employs prompt engineering and fine-tuning for intelligent semantic parsing.
For contextual analysis, static knowledge graphs are upgraded to dynamic Chain-of-Thought (CoT) reasoning, enabling causal evolution tracking and real-time decision-making.

Our company's Hotspot Event Analysis tool developed based on the LLM-enhanced approach has effectively met user needs in practical applications.

The tool extensively collects information from domestic and foreign think tanks, mainstream media, internet public sentiment, and other sources to conduct event discovery and contextual analysis, gaining real-time insights into trending events. Additionally, based on specified thematic areas (such as macroeconomic forecasting, strategies of world-class enterprises, innovation and competitiveness, etc.), it performs in-depth event mining, correlation analysis, and dynamic monitoring, automatically generating analytical reports on events.