Using Alfresco Content in Haystack: Fetching, Live Search, and Agent Tools with OpenDMA

Your Haystack pipeline works fine until the documents are no longer local files, S3 objects, or web pages, but records in Alfresco.
The interesting part is not getting one file out of the repository. It is keeping the repository context while doing it:
sites, paths, nodes, metadata, permissions, versions, renditions, just to name a few.

There are usually three different tasks that look similar at first, but should not be implemented in the same way:

• fetch Alfresco content in a Haystack pipeline and ingest it into a document store or vector store
• search Alfresco live and only fetch matching documents
• give an agent tools to browse the repository, inspect metadata, search, and read selected text

The opendma-haystack integration provides separate components for these cases:

• AlfrescoFetcher
• AlfrescoRetriever
• AlfrescoToolset

They all talk to Alfresco through OpenDMA. OpenDMA maps the Alfresco repository to an abstract API and exposes it through a REST endpoint. The Haystack code does not have to call Alfresco's REST API directly.

The examples in this article are available as a runnable Python project in the OpenDMA Alfresco–Haystack sample repository. You can clone the repository and follow along with the examples below.

Why not just write a custom Alfresco REST component?

Fetching one file from Alfresco through the REST API is not hard.

The problem starts after the first prototype. A useful Haystack application normally needs more than bytes:

• site and folder context for Alfresco nodes
• node associations
• Wiki and Links content, not only files from the Document Library
• MIME type, Content type and aspects
• Alfresco properties, including multi-valued cardinality
• versions and checked-out state
• ACLs and effective user access
• security markings from Alfresco Governance Services
• repository renditions
• full-text and metadata based search

If this is implemented directly against the Alfresco REST API, the Haystack component quickly becomes a second repository integration layer.

OpenDMA keeps this integration at the ECM level. The Haystack pipeline talks to an OpenDMA endpoint. The endpoint is backed by an adaptor for Alfresco. The same application shape can later be used with other ECM systems such as Documentum, FileNet P8, OnBase, CMOD or SharePoint through different OpenDMA adaptors.

This does not remove all application decisions. You still decide which content to ingest, which converters to use, and how to handle metadata. But the repository-specific part is no longer spread across every Haystack pipeline and agent tool.

Fetch Alfresco content into a Haystack pipeline

Use AlfrescoFetcher when you want to build a knowledge base or vector store from Alfresco content.

The fetcher returns Haystack ByteStream objects. It does not convert files into text. This follows the usual Haystack shape: fetching and conversion are separate steps.

For example, this pipeline fetches all content from an Alfresco site and converts the supported file types:

from haystack import Pipeline
from haystack.components.converters import HTMLToDocument, PyPDFToDocument, TextFileToDocument
from haystack.components.joiners import DocumentJoiner
from haystack.components.routers import FileTypeRouter
from haystack_integrations.components.converters.docling import DoclingConverter

from opendma_haystack import AlfrescoFetcher


fetcher = AlfrescoFetcher(
    endpoint="http://localhost:7070/opendma/alf",
    username="admin",
    password="admin",
)

pipeline = Pipeline()
pipeline.add_component("fetcher", fetcher)
pipeline.add_component(
    "router",
    FileTypeRouter(
        mime_types=[
            "text/plain",
            "text/html",
            "application/pdf",
            "application/msword",
        ],
    ),
)
pipeline.add_component("text_converter", TextFileToDocument())
pipeline.add_component("html_converter", HTMLToDocument())
pipeline.add_component("pdf_converter", PyPDFToDocument())
pipeline.add_component("word_converter", DoclingConverter())
pipeline.add_component("joiner", DocumentJoiner())

pipeline.connect("fetcher.streams", "router.sources")
pipeline.connect("router.text/plain", "text_converter.sources")
pipeline.connect("router.text/html", "html_converter.sources")
pipeline.connect("router.application/pdf", "pdf_converter.sources")
pipeline.connect("router.application/msword", "word_converter.sources")
pipeline.connect("text_converter.documents", "joiner.documents")
pipeline.connect("html_converter.documents", "joiner.documents")
pipeline.connect("pdf_converter.documents", "joiner.documents")
pipeline.connect("word_converter.documents", "joiner.documents")

result = pipeline.run({"fetcher": {"sites": ["swsdp"]}})
documents = result["joiner"]["documents"]

The sites parameter uses Alfresco site short names. The fetcher resolves the matching site folders and recursively fetches the document nodes below them.

Each ByteStream carries metadata from OpenDMA and the underlying Alfresco node. After conversion, this metadata is available on the Haystack Document objects:

for document in documents[:5]:
    print("Title:", document.meta.get("opendma:Title"))
    print("OpenDMA ID:", document.meta.get("opendma:Id"))
    print("Site:", document.meta.get("alfresco:Site"))
    print("Path:", document.meta.get("alfresco:Path"))
    print("MIME type:", document.meta.get("mime_type"))
    print()

This is the usual ingestion path. From here, you can split, embed and write the documents into the document store of your choice.

The important part is that the pipeline does not only receive text chunks. It also receives repository metadata.

Use Alfresco metadata to keep retrieval precise

Metadata is not a decorative detail in ECM repositories.

Loading one small Alfresco site into a vector store works well. After adding another site with many similar chunks, a plain similarity search can start retrieving the wrong content. The relevant meeting notes are still in the vector store, but they no longer appear in the top results for the question.

This is not an artificial problem. A real Alfresco repository may contain hundreds of thousands or millions of documents. Many teams may have their own meeting notes, contracts, requirements documents, product specifications and project folders. Searching only by vector similarity across all chunks can produce good-looking but irrelevant context.

OpenDMA preserves Alfresco metadata such as site, path, content type, aspects and properties. You can use that metadata in retrieval filters.

For example, a pipeline can restrict retrieval to one Alfresco site and ignore checked-out documents:

filters = {
    "operator": "AND",
    "conditions": [
        {
            "field": "meta.alfresco:Site",
            "operator": "==",
            "value": "swsdp",
        },
        {
            "field": "meta.opendma:CheckedOut",
            "operator": "==",
            "value": False,
        },
    ],
}

The exact filter syntax depends on the document store and retriever you use. The point is more general: when Alfresco is the source, repository metadata should survive ingestion. Otherwise the vector store has lost information that the repository already had.

Search Alfresco live with AlfrescoRetriever

A vector store is not always the right first step.

Building one means ingesting content, extracting text, splitting it, calculating embeddings, storing those embeddings, and keeping the store synchronized with Alfresco. In enterprise scenarios, access rights are usually the harder part. A detached vector store can drift away from the repository permissions and compliance boundaries if it is not designed carefully.

AlfrescoRetriever is for the case where you want to use Alfresco search directly.

It runs Alfresco AFTS full-text search through OpenDMA. It builds a query such as:

TEXT:"meeting notes"

If sites are configured, it adds site restrictions:

TEXT:"meeting notes" AND (SITE:"swsdp")

The retriever returns metadata-only Haystack Document objects. These documents identify matching repository objects. They are not the full text of the files.

To read the actual content, pass the retriever result into a fetcher and then through the normal Haystack converters:

from haystack import Pipeline
from haystack.components.converters import HTMLToDocument, PyPDFToDocument, TextFileToDocument
from haystack.components.joiners import DocumentJoiner
from haystack.components.routers import FileTypeRouter
from haystack_integrations.components.converters.docling import DoclingConverter

from opendma_haystack import AlfrescoFetcher, AlfrescoRetriever


retriever = AlfrescoRetriever(
    endpoint="http://localhost:7070/opendma/alf",
    username="admin",
    password="admin",
    sites=["swsdp"],
    top_k=8,
)

fetcher = AlfrescoFetcher(
    endpoint="http://localhost:7070/opendma/alf",
    username="admin",
    password="admin",
)

pipeline = Pipeline()
pipeline.add_component("retriever", retriever)
pipeline.add_component("fetcher", fetcher)
pipeline.add_component(
    "router",
    FileTypeRouter(
        mime_types=[
            "text/plain",
            "text/html",
            "application/pdf",
            "application/msword",
        ],
    ),
)
pipeline.add_component("text_converter", TextFileToDocument())
pipeline.add_component("html_converter", HTMLToDocument())
pipeline.add_component("pdf_converter", PyPDFToDocument())
pipeline.add_component("word_converter", DoclingConverter())
pipeline.add_component("joiner", DocumentJoiner())

pipeline.connect("retriever.documents", "fetcher.targets")
pipeline.connect("fetcher.streams", "router.sources")
pipeline.connect("router.text/plain", "text_converter.sources")
pipeline.connect("router.text/html", "html_converter.sources")
pipeline.connect("router.application/pdf", "pdf_converter.sources")
pipeline.connect("router.application/msword", "word_converter.sources")
pipeline.connect("text_converter.documents", "joiner.documents")
pipeline.connect("html_converter.documents", "joiner.documents")
pipeline.connect("pdf_converter.documents", "joiner.documents")
pipeline.connect("word_converter.documents", "joiner.documents")

result = pipeline.run({"retriever": {"query": "meeting notes"}})
documents = result["joiner"]["documents"]

This is live full-text search, not semantic vector search. Keyword choice matters more. In an agentic workflow, the agent may need to try a shorter or broader query if the first search does not find anything.

The advantage is that Alfresco remains in the query path. If the OpenDMA endpoint is called with the effective user's identity, Alfresco remains the system deciding what the user can see. That includes permissions and repository-side governance constraints, such as security markings enforced by Alfresco Governance Services and the OpenDMA adaptor path.

This is different from answering out of a detached vector store that was filled earlier by a background process.

Give an agent repository tools with AlfrescoToolset

Sometimes the task is not "retrieve the top chunks".

The user may ask where a document is stored, which document is the latest one, what metadata a candidate document has, or what a section of one document says. In these cases, an agent should be able to navigate the repository and read only what it needs.

AlfrescoToolset gives a Haystack agent tools for that style of work.

The Alfresco tools include:

• alfresco_list_sites
• opendma_list_children
• opendma_get_metadata
• opendma_describe_class
• opendma_read_text
• opendma_search

The toolset is configured with a fixed endpoint, account and repository. The agent does not receive a generic HTTP client and does not decide which repository credentials to use.

For rich file formats, provide a text_extractor. This is the function used by opendma_read_text after the toolset fetches the binary stream:

from haystack import Document
from haystack.components.converters import PyPDFToDocument, TextFileToDocument
from haystack.dataclasses import ByteStream

from opendma_haystack import AlfrescoToolset


def text_extractor(streams: list[ByteStream]) -> list[Document]:
    text_streams = [
        stream
        for stream in streams
        if stream.mime_type is not None and stream.mime_type.startswith("text/")
    ]
    pdf_streams = [
        stream
        for stream in streams
        if stream.mime_type == "application/pdf"
    ]

    documents: list[Document] = []
    if text_streams:
        documents.extend(TextFileToDocument().run(sources=text_streams)["documents"])
    if pdf_streams:
        documents.extend(PyPDFToDocument().run(sources=pdf_streams)["documents"])
    return documents


toolset = AlfrescoToolset(
    endpoint="http://localhost:7070/opendma/alf",
    username="admin",
    password="admin",
    repository_id="Alfresco",
    text_extractor=text_extractor,
)

The toolset can then be passed to a Haystack Agent:

from haystack.components.agents import Agent
from haystack.components.generators.chat import OpenAIChatGenerator
from haystack.utils import Secret


system_prompt = """
You answer questions using documents stored in Alfresco.

Use alfresco_list_sites to discover sites.
Use opendma_search to search for candidate documents.
Use opendma_list_children to inspect folders.
Use opendma_get_metadata before reading a candidate document.
Use opendma_read_text only when the requested information is inside the document.

Do not answer factual repository questions unless the answer is supported by tool results.
"""

agent = Agent(
    chat_generator=OpenAIChatGenerator(
        api_key=Secret.from_env_var("OPENAI_API_KEY"),
        model="gpt-4o-mini",
        generation_kwargs={"temperature": 0},
    ),
    tools=toolset,
    system_prompt=system_prompt,
    max_agent_steps=8,
)

This is deliberately different from a plain RAG pipeline. The agent can first inspect repository structure, candidate nodes and metadata. It can read text only after it has found a useful candidate document.

That matters because reading documents is more expensive than listing children or getting metadata. It also keeps the reasoning closer to how a human would use an ECM system: find the right place, inspect the candidate, then read the document.

Where OpenDMA fits in production

The examples above still use normal Haystack converters. That is intentional. OpenDMA does not force a different text extraction stack into your application.

But in production, document conversion should not automatically become the Haystack application's problem. Alfresco installations may already have transformations or renditions configured. OpenDMA also models renditions at the repository level. When that is available in the deployment, it belongs in the repository integration layer rather than being reimplemented in every Haystack application.

The same applies to enterprise concerns around the endpoint. In an ECI Server deployment, OpenDMA can sit behind configuration, auditing, identity translation, RBAC, SIEM integration, and token exchange. Those concerns are much easier to reason about in one repository integration layer than in a collection of custom REST calls inside application code.

For Haystack, that gives you three choices with the same repository model underneath: fetch and index, search live, or give the agent tools.

Try it yourself

The complete runnable examples for this article are available in the GitHub sample project.

Clone the repository to experiment with the different integration approaches in your own environment.

Further reading:

• OpenDMA: https://opendma.org/
• OpenDMA Haystack integration: https://github.com/OpenDMA/opendma-haystack
• Documentation: https://github.com/OpenDMA/opendma-haystack/tree/main/docs
• Tutorials: https://github.com/OpenDMA/opendma-haystack/tree/main/docs/tutorials
• ECI Server: https://xaldon.com/products/eci-server