By Chris Byrne (6.8.25 - updated 10.25)
Updated research: Reverse Engineering the Response to "best smartphones" in Chatgpt 5.0 https://www.seolondonsurrey.co.uk/blog/reverse-engineering-the-response-to-best-smartphones-in-chatgpt-5-0
This is my attempt to reverse engineer a response to a comparative entity prompt ("best smartphones 2025") in Chatgpt 4 using the tool itself. I chose this prompt to trigger "Retrieval Augmented Generation" (RAG): where instead of relying solely on the model’s pre-trained knowledge alone, the model retrieves relevant information from connected data sources (via Google search according to my interpretation of the feedback below). This is an interpretive high-level view, not the model's internal data.
In the USA for example, there are many thousands of "best smartphones"-type searches per month on Google alone, according to data from Google Keyword Planner - we might assume that LLMs might be used in a similar way to compare products etc.
See the original response to my prompt “Best Smartphones 2025” below:
“Based on current expert reviews and recent releases (as of August 2025), here are the top smartphones:
-
Samsung Galaxy S25 Ultra
Snapdragon 8 Gen 4 / Exynos 2500 chip (region dependent)
200 MP camera, dual telephoto lenses with up to 100× zoom
6.8″ QHD+ AMOLED 120 Hz display
AI features like “Circle to Search” and multitasking assistants
120 W wired / 50 W wireless charging
Reviewed by TechAdvisor as the top Android flagship
-
iPhone 17 Pro Max
Apple A19 Bionic chip with upgraded NPU
ProMotion 120 Hz LTPO OLED display
Deep integration with iOS 18's AI features
Strong battery life and camera upgrades
-
Xiaomi 15 Ultra
50 MP quad cameras, Leica optics
Snapdragon 8 Gen 4, 90 W fast charging
Top-tier performance and image processing
-
OnePlus 13
Snapdragon 8 Gen 4, 100 W charging
Great value for performance
Clean UI, 7 years of updates
-
Samsung Galaxy Z Fold 7
Foldable flagship with a lighter hinge
Snapdragon 8 Gen 4
Multitasking UI with 8.1″ inner display
Reviewed as best productivity phone”
My analysis (assisted by Chatgpt)Below based on using direct prompts I have attempted to show how the response to my prompt “best smartphones 2025” was created . I used the direct feedback from the tool itself. The below shows how Chatgpt used external web searches to supplement it's preexisting knowledge base while distinguishing between what info used in the response came from it's training in contrast with what came from up to date web sources. The feedback below comes from merging 2 responses from Chatgpt to 2 near-identical prompts (and 1 other response to try to clarify some info provided ) merged with some minor revision by me, with my comments.
I chose this “best smartphones 2025” prompt as it demands “up to date” information: it is a prompt that can not be resolved reliably from the information in the model's knowledge base alone (as the cut-off point for the model's knowledge base is mid-2024)
The stages in response formation Chatgpt stated were as follows:
Stages 1-3. Prompt interpretation / Vector representation generation / Contextual focus mechanism.
Chatgpt stated the response generation starts with the prompt (“best smartphones 2025”) which is tokenized and passed through the model's transformer layers and generates candidate brands / models / specifications using it's internal knowledge (a.k.a. the “corpus”) from it's training data up to mid-2024.
The model then maps each token in the prompt to dense vectors in high-dimensional space. Here's what activates:
Chatgpt reported that ALL the info at this stage came entirely from the model’s pretrained corpus, up to a 2024 cutoff.
Here is the knowledge in this response that came from training data up to mid-2024 alone below:
Understanding that flagship phones are typically released in Q1 (Samsung, OnePlus, Xiaomi) or Q3 (Apple)
Phone series evolution: iPhone naming convention (e.g. iPhone 17 follows 16), Samsung S-series (S25 after S24)*
Common flagship specs: e.g. high refresh rate OLEDs, multi-lens cameras, Snapdragon / Exynos / A-series chips
AI feature trends (e.g. multimodal assistants, fast charging over 100W etc )
Typical vendors for foldables (Samsung leads), and Xiaomi’s camera collaboration with Leica
This information Chatgpt stated were retrieved through high-level embeddings based on semantic similarity between the prompt (“best smartphones 2025”) and vectors representing past smartphone rankings and product specs. It is worthwhile re-iterating Chatgpt stated that “No access to real-time 2025 device details was available in pre-training data” - all of the information above was from the corpus or inferred (the annual release of IPhones with a naming convention e.g. 13 is followed by 14).
Stage 4. Web tool triggered
Chatgpt stated that it then “issued live web searches to retrieve high-precision sources to confirm whether these models exist”*, their specs, and any recent reviews or announcements.. Those results are ranked for topical match and credibility.
*N.B. Chatgpt in it's reponse above stated it “generates candidate brands / models and specifications using it's internal knowledge (a.k.a. the “corpus”) from it's training data up to mid-2024”. From what I see sometimes in Chatgpt you can see responses based on naming conventions (e.g. iPhone 17 follows 16) for models / products that may not have yet been released or were never released (in responses where the Web tool wasn't triggered to “fact check”); thus as Chatgpt fed back in this instance it “issued live web searches to retrieve high-precision sources to confirm whether these models exist”. Essentially the tool “predicted” the new product release based on naming convention / release schedule information in the corpus and then checked its "facts" via the web .
Live search was used to pull up-to-date 2025 model names, features, and rankings. Here's the exact web search pipeline reported: (note I have broken some URLs)
Search 1 (Google): (my notes: I'm not sure if this means all searches were done using Google as it was the only search labelled as such)
Query: “best smartphones 2025 site:techadvisor.com OR site:tomsguide.com OR site:wired.com”
We must note the use of “notable” tech / review sites in this search - note these "notable" websites are ALL also publications owned by Chatgpt's content partners:
Condé Nast (e.g. Vogue, WIRED & Glamour)
Future PLC (e.g. Techradar & Tom's Guide).
Source used: TechAdvisor 2025 Best Phones Guide https://www.techadvisor .com/article/724318/best-phone.html
Chatgpt stated it “Verified Samsung Galaxy S25 Ultra, OnePlus 13, Xiaomi 15 Ultra specs and rankings” data using this source.
Search 2:
Query: “iPhone 17 Pro Max specs 2025”
Source used: https://www.macrumors .com/roundup/iphone-17/
Chatgpt stated it “Verified [the] A19 chip, 120 Hz ProMotion display, iOS 18 feature set” details using this source
Search 3:
Query: “Galaxy Z Fold 7 specs review 2025”
Source used: Tom’s Guide comparison of the Z Fold 7 & Galaxy S25 https://www.tomsguide .com/phones/samsung-phones/i-put-the-galaxy-z-fold-7-vs-galaxy-s25-ultra-through-a-7-round-face-off-heres-the-winner
Chatgpt stated it “Verified [the] foldable screen size, multitasking UI, and S25 Ultra comparison” using this source
Search 4:
Query: “Xiaomi 15 Ultra review 2025”
Sources used: https://www.techadvisor .com/article/724318/best-phone.htmland & https://en.wikipedia.org/wiki/Xiaomi_15
Chatgpt stated this search “allowed the response to include specific model names, specs (e.g., Snapdragon 8 Gen 4 vs A19), camera details, display sizes, and expert opinions” using this source.
Then, Chatgpt then merged the corpus information with the above information from the Web tool —relying on live data for specifics and using the trained patterns for presentation, organization, and comparison logic.
Stage 5 . Response generation:
Chatgpt used the latest factual information with pretrained semantic structures to output a comparison-oriented list, emphasizing criteria commonly used in smartphone rankings (camera, chip, battery, UI, price etc).
I did the same in Google Gemini too.
Top comments (0)