CrewAI's website had to do something that traditional marketing sites rarely need to do: serve several very different consumers from the same digital platform.
An executive might be evaluating CrewAI as an enterprise AI platform. A developer might be looking for documentation or an integration example. A content team needs to publish and manage pages. An AI assistant needs structured information it can retrieve without parsing a visual interface.
That changed the architectural problem.
The project included 7 dedicated discovery surfaces for AI systems, a Contentful based content architecture, an automated publishing pipeline from Ghost to Contentful, centralized HubSpot form infrastructure, Markdown content delivery, OpenAPI documentation, OAuth discovery, and a Model Context Protocol (MCP) implementation.
The important lesson was not simply to add AI features to a website. It was to make the underlying content and application architecture usable by both people and machines.
Start with the content model, not the page
One of the easiest ways for a large website to become difficult to maintain is to model the system around pages.
A page based architecture often leads to individual templates, custom layouts, and content embedded directly into those layouts. As the number of content types grows, the system becomes harder for marketing teams to operate and harder for engineers to change safely.
For CrewAI, the approach was different.
The experience began in Figma as a design system, rather than as a collection of individual page designs. The resulting component library included hero layouts, platform diagrams, pricing sections, feature grids, statistics modules, tabbed content, media blocks, and technical documentation.
Those components were then connected to structured models in Contentful.
That relationship matters:
Design component → structured content model → application rendering
Instead of treating the CMS as a place to store finished pages, the CMS became part of the application's content architecture.
Marketing could compose pages from reusable sections, while engineering retained control over how those sections were rendered, integrated, measured, and maintained.
The application used GraphQL and TypeScript to retrieve and render the structured content.
This is a useful distinction when designing a modern CMS architecture. A component library is more valuable when its content representation is designed at the same time as its visual representation.
The CMS does not have to be the writing tool
There was another architectural constraint: the editorial team already used Ghost for writing.
Replacing that workflow simply because the public website used Contentful would have solved an engineering problem by creating a content problem.
Instead, the project connected the two systems.
When an article is published in Ghost, an automated workflow converts it into a draft in Contentful.
The important detail is where the automation stops.
Ghost remains the writing environment.
Contentful becomes the controlled publishing environment.
That means the workflow looks roughly like this:
Writer → Ghost → automated content transfer → Contentful draft → review and enrichment → publication
The system deliberately does not automatically publish the article from Ghost.
That creates a useful boundary between authoring and publishing.
The editorial team can continue using the tool optimized for writing, while marketing can manage metadata, SEO, page composition, scheduling, and other publishing concerns in the system responsible for the public website.
This is a broader architectural lesson: when two systems have different strengths, integration can be better than forcing one system to perform both jobs.
Dynamic forms should be treated as infrastructure
Lead generation introduced another integration problem.
Rather than creating isolated forms for different parts of the website, the platform centralized its HubSpot integration.
Forms retrieve their structure directly from the HubSpot Marketing Forms API during page rendering. Submissions are validated with Zod before they reach HubSpot.
The validation layer also enforces a business email requirement by blocking more than 40 consumer email providers, including Gmail, Yahoo, and Outlook.
The same infrastructure handles attribution using HubSpot cookies, referral information, and page context.
It also supports different confirmation behaviors, including:
- Inline success messages
- Redirects
- Embedded Calendly scheduling
This matters because a form is rarely just a collection of input fields in an enterprise website.
It is part of a larger data flow:
Form definition → validation → attribution → CRM submission → confirmation experience
Centralizing that flow reduces the chance that different forms implement business rules differently.
It also means new forms can reuse the same integration and validation infrastructure instead of introducing another custom path to the CRM.
The website became a structured knowledge source
The most unusual part of the project was treating the website as something that AI systems should be able to consume directly.
A conventional website is primarily optimized around human navigation.
Information is organized into visual layouts, navigation menus, HTML documents, and interactive interfaces. A person can interpret those layers together.
An AI system does not necessarily need those layers.
It benefits from clear, structured representations of the same underlying information.
The CrewAI platform therefore supports Markdown content delivery. Blog posts, documentation, webinars, case studies, events, and CMS pages can be requested as structured Markdown through dedicated routes or .md URLs.
That creates an additional representation of the content without requiring an AI system to reverse engineer the rendered page.
The architectural principle is simple:
Do not make every consumer understand the presentation layer when the underlying content can be exposed directly.
Markdown is useful for this because it provides a lightweight, structured representation of content that can be consumed by developer tools, automation systems, and AI applications.
Discovery is a separate architectural problem
Providing structured content solves only part of the problem.
A machine still needs to discover what exists.
The CrewAI platform exposes 7 documented discovery surfaces:
llms.txtllms-full.txtopenapi.json-
/.well-known/mcp/server-card.json5./.well-known/oauth-authorization-server sitemap.md/brand/knowledge-graph
These surfaces serve different purposes.
The llms.txt and llms-full.txt resources provide AI oriented descriptions of the site's capabilities and resources.
openapi.json exposes an API description using OpenAPI 3.1.
The .well-known resources provide machine discoverable information related to MCP and OAuth authorization.
sitemap.md provides a structured representation of site content.
The brand knowledge graph provides another machine readable representation of the organization's information.
The important architectural point is that these are not seven copies of the same content.
They are different interfaces into the same digital system.
That distinction becomes increasingly important as websites are consumed by more types of software.
MCP turns APIs into agent accessible capabilities
The project also implemented the Model Context Protocol (MCP).
MCP is an open protocol for connecting AI applications with external data sources and tools. Its specification defines mechanisms through which servers can expose resources, prompts, and tools to clients.
For CrewAI, the implementation mirrors public REST APIs through a consistent collection of tools.
Agents can discover capabilities, authenticate using OAuth, and retrieve resources such as:
- Blog posts
- Webinars
- Events
- Case studies
- Developer resources
This is different from simply making a website "AI searchable."
Search gives an AI system information about a website.
An API gives software a defined way to retrieve information.
A protocol such as MCP provides a standardized interface through which compatible AI applications can discover and interact with capabilities.
That difference matters when deciding how much of a platform should be exposed to intelligent systems.
Why OAuth matters here
Once an application exposes capabilities to external software, discovery alone is not enough.
The system also needs a way to establish who is authorized to access protected capabilities.
That is why OAuth discovery is part of the architecture.
The platform exposes an OAuth authorization server discovery endpoint alongside its MCP implementation.
The result is a more complete machine interaction model:
Discover → authenticate → access capabilities → retrieve structured content
The exact authentication requirements depend on the capability being exposed, but the architectural principle is broadly useful: machine readable discovery and authorization should be designed together rather than treated as unrelated features.
WebMCP adds another interaction surface
The platform also supports WebMCP when the browser supports it.
This allows AI interactions to occur directly within the page rather than requiring every interaction to be performed through a separate server request.
That introduces an interesting distinction between two types of AI integration.
The first is server side access, where an external agent discovers and calls APIs or MCP tools.
The second is in page interaction, where an AI capable browser can interact with capabilities exposed by the page itself.
The two approaches solve different problems.
A developer building an integration may want API or MCP access.
A user interacting with an AI enabled browser may benefit from capabilities exposed directly by the web experience.
Designing for both means the website becomes more than a destination. It becomes an interface that different classes of clients can use.
The real architecture is the combination
None of these individual technologies is particularly useful as a standalone checklist.
Contentful does not automatically make a website AI ready.
Markdown does not automatically create a useful AI interface.
An OpenAPI document does not automatically create a good developer experience.
MCP does not replace a well structured content architecture.
The value comes from connecting them.
The resulting architecture can be understood as several layers:
The exact implementation is more complex than this simplified model, but the principle is important.
There is a shared application and content foundation, with different interfaces optimized for different consumers.
What this architecture teaches
1. Model content as structured data
If content is represented as reusable structured entities, it can support more than one presentation layer.
That benefits traditional web development first. AI consumption becomes an additional advantage rather than the sole reason for structuring content.
2. Separate authoring from publishing when workflows require it
The Ghost and Contentful integration demonstrates that a CMS migration does not have to mean replacing every existing editorial tool.
The better question is:
Which system should own each part of the workflow?
In this case, Ghost owns writing, while Contentful owns structured website publishing.
3. Build integrations as shared infrastructure
The HubSpot implementation centralizes validation, attribution, form configuration, and confirmation behavior.
That is more maintainable than allowing every form to develop its own integration logic.
4. Expose information in the representation the consumer needs
Humans benefit from visual interfaces.
Developers benefit from APIs and documentation.
AI systems benefit from structured content, explicit discovery mechanisms, and machine accessible capabilities.
There is no requirement for all of those consumers to use exactly the same representation.
5. Treat AI readiness as an architectural concern
Adding an AI chatbot to an existing website is one possible AI feature.
Making the underlying content, APIs, discovery mechanisms, authentication, and application capabilities accessible to intelligent systems is a much broader architectural decision.
The CrewAI project demonstrates the latter approach.
When this architecture makes sense
This approach is not necessary for every website.
A small marketing site with a handful of static pages probably does not need a CMS integration spanning multiple editorial systems, an OpenAPI description, MCP capabilities, or multiple machine discovery surfaces.
The complexity becomes more justified when a platform has several audiences and several systems that need to consume the same information.
That includes enterprise AI companies, developer platforms, documentation heavy products, SaaS companies with extensive APIs, and organizations expecting their content to be consumed by increasingly autonomous software.
The key is to start with the consumption model.
Ask:
- Who needs to consume this content?
- Which parts need to be structured?
- Which capabilities should be exposed as APIs?
- How will external systems discover them?
- Which capabilities require authentication?
- Which workflows should remain controlled by humans?
- Which systems should own authoring, publishing, and data validation?
Those questions produce better architecture than starting with a technology shopping list.
The bigger lesson
The most interesting part of the CrewAI project was not that the website used Next.js, Contentful, GraphQL, Vercel, OpenAPI, OAuth, or MCP.
It was that those technologies were connected around a common idea: the website should be useful to whatever is consuming it.
For humans, that means a coherent design system and an intentional digital experience.
For content teams, it means structured publishing workflows with appropriate editorial control.
For developers, it means APIs, documentation, and predictable interfaces.
For AI systems, it means machine readable content, explicit discovery, authentication, and protocol based access to capabilities.
That is a more useful definition of an AI ready website.
It is not a page with an AI feature attached to it.
It is a digital platform whose architecture gives both people and intelligent systems clear ways to understand, navigate, and interact with the information and capabilities it provides.
Top comments (0)