Architecting Practice-Centric Local-Edge Medical AI under Extreme Hardware Constraints

“The stone which the builders rejected, the same is become the head of the corner...”
— Matthew 21:42

⚠️ Legal Disclaimer

Important Notice for Readers and Regulators:

1. Project Status: The system architecture described in this article is strictly a personal, non-commercial "pet project" (Proof of Concept) developed exclusively for educational, private research, and benchmarking purposes. The system is deployed strictly on isolated local infrastructure, is not distributed, and offers no public web interface, hosting, or external API.
2. Medical & Regulatory Compliance: This software does not qualify as Software as a Medical Device (SaMD) under the US Food and Drug Administration (FDA), European Union Medical Device Regulation (MDR), or Russian Roszdravnadzor frameworks. It is not intended for clinical use, patient triage, automated diagnosis, or treatment planning. It does not replace professional medical advice, diagnosis, or treatment.
3. Copyright & Text Mining: The referenced dataset of 2,500 monographs is stored strictly on a local machine and used solely for private text-indexing, natural language processing (NLP) benchmarking, and search experiments. This use complies with international "Fair Use" principles for non-commercial research (17 U.S. Code § 107) as well as personal-use exemptions under Art. 1273 of the Civil Code of the Russian Federation. The copyrighted source texts are not hosted, published, mirrored, shared, or distributed in any form.
4. Data Privacy: This project operates entirely offline using simulated, synthetic, or fully anonymized test queries. It does not collect, transmit, process, or store Protected Health Information (PHI) under US HIPAA, Personal Data under EU GDPR, or any real-world personal data.

Executive Summary

The article outlines an architecture for an offline, local edge-compute information retrieval system designed to mine global medical literature on standard commodity hardware within strict memory constraints. By utilizing a body of about 2,500 modern curated monographs instead of periodical literature and a resource-efficient agentic framework, the system bridges the gap between massive medical literature databases and local edge-compute boundaries while maintaining absolute data privacy and deterministic auditability. The system facilitates the extraction of scientific knowledge from literature in the fields of clinical medicine, clinical pharmacology, and biochemistry. It also provides a linguistic search engine for pathological medical images and DICOM visualization tools. The proposed AI assistant serves as an accessible semantic research accelerator for frontline practitioners, biomedical engineers, and interdisciplinary researchers.

1.0 Introduction: Motivation for Engineering a Medical Practice AI Assistant --- Personal, Strategic and Otherwise

All of us fall ill at some point. Yet, it's particularly difficult to watch a loved one helplessly struggle with a disease, especially after having fruitlessly visited multiple doctors and amassed stacks of diagnostic test results.

There are two major issues with this situation. First, it is statistically typical, and second, the high cognitive and emotional load almost certainly prevents a patient from rigorously analyzing their data to find a plausible solution. Someone else has to carefully and systematically inspect those doctors' notes and the wealth of available diagnostic tests, in order to identify discrepancies and make necessary adjustments in parallel with trying to find a more suitable medical specialist.

To successfully resolve this issue, one must evaluate a colossal volume of medical literature, retain vast amounts of complex data, and cross-reference it all with clinical notes and laboratory results. Unless that person is a highly trained, experienced medical professional with exceptional cognitive recall, this remains a nearly impossible task for a human. However, this is precisely where AI-powered Retrieval-Augmented Generation (RAG) systems excel.

The observations detailed above constituted my personal motivation for engineering practice-centric medical AI assistants, particularly because decades of applied mathematical work in biomedical applications have allowed me to accumulate a substantial volume of relevant literature in the field.

More importantly, there is a strategic dimension that motivates this work. With few exceptions, most individuals find it difficult to retain vast volumes of information, with medicine being perhaps one of the most demanding disciplines in terms of memory utilization. The availability of an AI assistant capable of processing thousands of scientific monographs and rapidly retrieving the required context could, at least theoretically, significantly augment the professional capacity of an average physician, effectively bridging the gap with leading experts.

Finally, a properly tailored medical AI assistant serves as a valuable tool for the broader community. It enables individuals to systematically monitor their health parameters, adequately prepare for clinical consultations, and fundamentally enhance their health literacy, among other essential health management tasks.

“The stone which the builders rejected”: building an offline medical search engine on consumer hardware

In order to build an information system that is equally valuable to the professional community and the general public, strict hardware constraints must be imposed.

First, it is imperative that all private data entered into the system remains strictly confidential, as neither enterprise users nor private individuals want their data leaked to the Web. Consequently, the system must be designed to operate solely on a local device without an internet connection. This architecture simultaneously addresses a second critical requirement: maintaining continuous operations in remote regions where network connectivity is unavailable.

Second, the system must run on affordable, commodity hardware. Enterprise clients are equally reluctant to fund expensive, GPU-packed server racks when a standard office laptop should suffice. Ideally, one would want the system to run directly on a smartphone, given the ubiquitous nature of these devices in daily life (see the Figure 1).

However, simple resource-allocation calculations quickly demonstrate that such a deployment would strictly require a top-tier flagship device. Consequently, we establish the standard office laptop (16GB RAM) as our baseline reference hardware with the following Core Engineering Metrics:

• Target Hardware Substrate: Optimized explicitly for Ubuntu Linux using a fixed budget of 4 GB VRAM and 16 GB System RAM.
• Data Isolation Vector: Air-gapped pipeline operates entirely offline, safeguarding proprietary model weights and preventing unauthorized remote data compilation.

In the remainder of this article, we discuss architectural approaches that appear rational and have proven effective for such information systems, as well as analyze the primary architectural challenges arising from the constraints imposed above. Software implementation details will be completely omitted, as they are thoroughly documented in the literature.

Finally, we emphasize that our objective is to design an information system that facilitates the extraction of scientific knowledge from medical literature. The proposed architecture is distinctly not designed as a diagnostic system or a clinical decision support system of any kind. It does not manage compliance, track legal protocols, or automate a physician's routine duties. Instead, it serves strictly as a semantic research accelerator, allowing professionals and researchers to rapidly mine global scientific literature for deep medical insights under extreme local hardware constraints.

2.0 Architecture of Retrieval-Augmented Generation (RAG) database

We start this section by reiterating the final statement made in the Introduction. It is habitual for a medical RAG to have a prioritized three-tier architecture consisting of (i) an operational tier based on clinical protocols which define step-by-step symptom based diagnostic algorithms, (ii) a safety-and-law tier which provides safety and legal guardrails, and (iii) an explanatory tier based on medical literature. The first two tiers are normally considered the most important, among other things, because they dictate what the local doctor is legally and baseline-operationally expected to do.

However, here the aim is to build an information extraction system designed for scientific discovery. Contrary to the case described above, it serves a completely different engineering purpose than being an operational automation tool designed to manage a doctor's workflow, legal compliance, or paperwork. Thus, we assume in what follows that a qualified physician must know the laws, protocols, and medical standards of their jurisdiction.

Consequently, medical regulatory standards and recommendations will be discarded from our database entirely. Also omitted will be clinical protocols designed for immediate step-by-step rigid implementation. The rationale for this is that there is a profound, systemic gap between statistical medicine and individual human biology. We do not want to bias RAG search results by imposing averaged statistics of the human population built into standard protocols over personal cases, as it is mathematically likely to fail. Furthermore, it takes an average of 10 to 17 years for a breakthrough discovery in peer-reviewed medical literature to be officially integrated into a government-approved clinical protocol (see the original landmark study by Balas, E. A., & Boren, S. A. 2000. Managing Clinical Knowledge for Health Care Improvement. Yearbook of Medical Informatics, 09(01), 65–70. https://doi.org/10.1055/s-0038-1637943). If you have a complex or fast-evolving condition, standard protocols are, by definition, running on outdated science.

Instead, we choose to pack our offline database with carefully selected recent medical monographs from the world leading publishers, giving a frontline physician immediate semantic access to the 17 years of science that haven't been encoded into rigid state protocols yet. We also deliberately avoid inclusion of periodic literature sources such as PubMed, since, in our view, normally they lack generalization power and coherence typical for monographic literature. Prioritizing monographic literature over periodicals (journal papers, individual studies, case reports), resolves the following major engineering bottlenecks inherent to local-edge RAG systems: generalization power vs. fragmentation, on the one hand, and semantic contradictions and noise, on the other.

Indeed, by its very nature the scope of a journal paper is to describe an isolated specific problem, normally making its generalization to broader medical practices difficult, if not impossible. Besides, the periodic literature is inherently full of noise, conflicting results, and premature conclusions. Contrary to this, monographs specifically aim to filter out the periodic noise of articles, resolve contradictions, synthesize and unify disjoint facts into a logical and coherent description of the problem matter. As a result, they provide a generic, comprehensive, much more stable and safe knowledge semantic base for an LLM to work with.

Although monographs are more stable, they may still be years behind the latest breakthrough periodicals. Hence, with the limited resources we have to make a trade-off, sacrificing the absolute latest preprint noise for stable, consolidated knowledge. This is a deliberate architectural compromise.

A tight system memory budget (the 4GB VRAM squeeze) calls for a small context window, necessitating the use of highly structured and optimized for information density self-contained text chunks typical of high quality monographic literature. This is opposite to the LLM needing to swallow an entire journal paper, in order to determine the correct context with an eminent threat of flooding the small context window of a 4-bit quantized local model.

Our experience shows that discriminate selection of about 2,500 top level monographs (about 1.5 million pages) covering all branches of contemporary medicine proves to be sufficient to build a reliable information retrieval system able to process the resulting database within acceptable times. Apart from the medical literature, we find it necessary to supplement the RAG base with purely biological monographic literature covering the fields of virology, genetics and biochemistry (on top of medical biochemistry), since these extensions substantially improve search results. Addition of biochemistry proved to be an absolute must for correct analysis of substances and medications, presumably, due to fast progress in pharmacology.

There are various ways in which the constructed database can be analyzed by an LLM. Two of them were examined. Since the database is structured according to the existing fields of medicine (e.g., neurology, hepatology, cardiology, etc.), an attempt was made to first determine the most relevant medical field based on the user search prompt and then bias the search priority to favor the related field. However, this approach has shown to be inferior and less accurate compared to an unbiased search over the entire bibliographic base. Presumably, owing to the complicated character of human biology, its different systems and, consequently, different fields of medicine are interconnected so intimately that their artificial separation inevitably results in perceptible loss of information. An explanation how cross-disciplinary chunks are synthesized (reranking mechanism) before passing them to the LLM is given in the next Section.

3.0 Agentic Architecture, System Prompts and Reranking Mechanism

Strict memory budget does not allow for the use of a proper full-scale multi-agent system with several specialized agents as independent AI units and an orchestrator as their manager. Consequently, a different tactic has to be adopted.

We introduce several independent agentic units, each having its own specialized system prompt. These are
(i) a medical assistant which specializes in the retrieval of clinical information,
(ii) an assistant called pharm1 which specializes in drug interactions, dosages, and pharmaceutical classifications,
(iii) an assistant called pharm2 which focuses on advanced drug research, covering pharmacokinetics and clinical trials,
(iv) an assistant called biochem which analyzes molecular pathways, lab results, and metabolic processes,
(v) an assistant called search engine which is a high-precision engine that crawls local medical databases for cited evidence, arbitrary user requests or images (of pathologies, US, MRI and CT scans, etc.).

The traditional approach also requires a reviewer agent to assess overall quality and correctness. We decided not to add it. Instead, we chose to implement a stand-alone "devil's advocate" architecture with a dedicated explicit adversarial approach based on a critic agent. When compared to the current architecture, in most of the practical cases in would confirm the findings of the presently described search engine, although occasionally it would provide valuable insights and alternatives. As a result, it was decided that the current architecture should be used as the primary research tool, while the adversarial critic agent architecture could be used for its verification.

In principle, an automated orchestrator agent could be also added (with the inevitable speed and representativity loss under the current budget). However, we ensure that the role of the supervisor agent rests with the end user, in order to prioritize and frame the agents' behaviours as his or her tasks require. Because, normally, the scientific information search is not restricted to a single query, but requires a long history of queries, whose topics can change and are dictated by the user's personal way of thinking, imposing rigid automated orchestration rules is inadvisable. In order to remember a sufficiently long history of queries and answers, an appropriate memory mechanism was implemented. We also designed the first four system prompts so as to allow for some overlap between the corresponding LLM answers, in order to enable the user to verify them and immediately spot discrepancies or hallucinations (if any) and take the corrective measures when required.

All system prompts were compiled so as to provide emotionally neutral, dry scientific style answers, and strictly to the point of the subject matter. As opposed to some of its main stream counterparts, the LLM was prohibited to adopt an empathetic tone, flatter or patronize the user by means of self-confident answers. It was prompted to be strictly precise and concise, in order to spare the system memory, on the one hand, and, since our "orchestrator agent" is supposed to be a human, not to emotionally influence the user's decision making process, on the other.

The system prompts also require the LLM to provide a short summary of the first query which, normally, contains the most important information such as a disease description. The purpose of this exercise is to explicitly verify that the LLM has rendered the query correctly and did not start hallucinating from the very outset, in which case the memory must be cleared or the query reformulated. Although it was highly unlikely that hallucinations would occur for an English language query, they might take place for a different language query, since medical vocabulary is exceptionally difficult to translate for either a built in LLM translator or stand-alone dedicated translation LLMs. We have examined quite a few such translator LLMs available both online and offline, with none being hundred per cent satisfactory. Because the bulk of our RAG monographs are in English, we had to simultaneously employ two translational LLMs, in order to at least partially alleviate the translational problems.

There are two practical sides to this issue. The first, is that it prompts the user to learn international medical terminology which is not much different from Latin and not that difficult to memorize even for an applied mathematician.

The second one, is that it is advisable to adhere to simple every day native language, where possible, in order for the translation to be more accurate. Since the LLM will attempt to provide a translation of the query in professional medical English not only in terms of terminology, but also the correct word order and style of medical writing, there is no reason to heavily use native medical vocabulary, where it can be avoided.

Even when compiling, say, a symptoms query in English, the descriptions made in simple words often give better search results than those in professional medical language. Presumably, the use of professional terminology in the query (e.g., for disease) makes the LLM look for exact keyword matches with the provided terms, filtering out some of the other valuable information.

This was one of the reasons that urged us to make LLM always reformulate the original query in simple words and use it in a separate search as a supplement to the search based on the original user's query.

This technique also forms the basis of the three-stage reranking framework we employed. It was borrowed from the literature and is as follows.

(i) Instead of processing a single prompt in our database, we supply two distinct prompts --- 1. the original user prompt and 2. its reformulated version (for instance, which expands symptoms and medical terminology in the case of disease description).

(ii) FAISS scans the vector space and pulls back X chunks from the original prompt and Y from the reformulated version.

(iii) Next, instead of using a separate, external neural network (like BGE-Reranker) which would eat up the 4GB VRAM, we are using the local LLM itself to do the synthesis and re-prioritization. If a specific monograph chunk is so relevant that it appears in both the top X of the original prompt and the top Y of the reformulated prompt, its structural relevance doubles. By feeding the raw coincidences into the local LLM, we are asking the model to read, filter out duplicate information, discard the noise, and synthesize a single coherent answer.

As a result, we intentionally bypass a dedicated cross-encoder reranker to save VRAM. Instead, we use a two-pronged ultra-fast semantic probe via FAISS, using query reformulation to catch cross-disciplinary links, and leave the final filtering to the local LLM during the synthesis phase. This keeps the memory footprint low while maximizing search coverage.

4.0 Image Processing Techniques

Modern medical monographs often contain excellent illustrations of various pathologies supplied with enlightening explanatory captions. However, illustrations are completely missed by a purely linguistic LLM, making the related AI assistant rather unsatisfactory from the end user's view point, since a considerable volume of valuable knowledge becomes simply thrown away.

If budget limitations were not a factor, a natural solution would be to employ a multimodal neural network unifying a text encoder with a transformer architecture to process words and sentences, a vision encoder with convolutional networks or vision transformers to scan pictures, and a fusion layer to draw conclusions from mixed media.

A more economical approach had to be adopted in the present case. Very broadly, we use PyMuPDF (fitz) to locate an image in the database documents, determine the image attributes and use them to find the related captions. If the caption contains relevant information, the image is printed out. Of course, there is a long list of special filters to process adverse situations where, for instance, a text document has an image format, or an image occupies more than one page, a caption is not in the same page as an image, and a lot more. Finally, several most appropriate images, the related captions, page numbers and book titles are presented in each image window. To facilitate image search and to make sure that a correct image was spotted, cross-correlations of the user query with the surrounding pages containing the image description are also checked. This technique was found to give reasonably good results for MRI or CT scans and photos of pathologies.

A simplified, but robust, DICOM image viewer was written in Python - javascript combination and integrated into the front end. It displays the three orthogonal planes (axial, coronal, and sagittal) with linked crosshairs and a paging mechanism for scrolling through each image stack.

To enable the user to make comparisons of diagnostic images displayed in the viewer with the related images found in the literature database, the viewer window was placed alongside with the image search window.

An attempt was made to display three-dimensional DICOM images processed by the viewer. However, it was found that at least under the present budget restrictions the resulting tool for manipulating 3D images built upon the combination of Python and javascript was too slow to be practical. A proper professional C++ image viewer is required for this purpose.

Owing to its exceptional practical importance, it is planned to separate and further refine image retrieval and DICOM manipulation software into a standalone tool in the future.

Owing to its exceptional practical importance, it is planned to separate and further refine image retrieval and DICOM manipulation software into a standalone tool in the future.

To verify the performance of the described algorithms, a mobile interface for the MedAI Assistant system was developed and deployed on a local server (see Figure 1).

The application interface is divided into two functional areas:

1. Left Panel (Inference & Agent Selection): Contains a mandatory medical disclaimer and a system status logging console (the screenshot displays a successful initialization of the biochemical_persona and a cleared VRAM state). The bottom of this panel features rapid-toggle buttons to switch between isolated agentic modules (Medical, Pharm 1, Pharm 2, Biochem, and Search Engine). When the Biochem persona is activated, the system switches context to analyze molecular pathways, metabolic processes, and laboratory tests.
2. Right Panel (Medical Record & DICOM Workspace): Features a structured, interactive medical history template ranging from general patient data to family anamnesis. Section IX (Radiological/Imaging Tests) includes a native image integration module linked to a local directory (Local DICOM Workspace Folder), complete with UI controls to load or purge the image stack.

5.0 Evaluation, Real-World Use Cases, and Conclusions

An issue of testing this architecture has remained uncovered so far. One might naturally ask how well this localized system performs against standard medical examination questions. However, multiple-choice board exams are an invalid evaluation framework for the current architecture. Indeed, because the RAG database natively includes comprehensive board-preparation textbooks, the system would often simply find the relevant citation for the corresponding test question, rather than demonstrate generalized synthesis. On top of this data contamination: aspect, this evaluation scenario suffers from the scope alignment problem: the system is not designed to act as a licensed clinician, and none of its system prompts assume this role.

Because this is a strict information retrieval system, its primary performance metric is deterministic auditability. The user can instantly validate any model assertion by requesting an exact citation with the name of the author, monograph title, and the page number for its underlying source material, in order to directly verify the relevance of the answer.

System Sanity Checks and Personal Validation

An instructive and convincing way to conduct authentic stress-testing outside of contaminated benchmarks is to use historical, longitudinal clinical data. With this in mind, I fed the system anonymized, multi-decade clinical test results and symptom descriptions from medical cases collected over my lifetime. Since their outcomes were known to me exactly, this retrospective evaluation served as an excellent sanity check. The architecture consistently synthesized accurate insights, identified cross-disciplinary systemic interactions, and surfaced personally valuable health information that had gone unnoticed during the original consultations years prior.

Case Study: Domain Acceleration in Angiography Software Engineering

The utility of the proposed information retrieval tool extends far beyond clinical research. It acts as an efficient domain-immersion engine for other technical specialists.

Recently I had to engineer a digital angiography imaging software project in the absence of either proper technical specification documents, or access to a senior interventional radiologist. Traditional web searches and high-level regulatory medical documents provided only superficial introductory context.

The breakthrough in understanding of the problem domain, the ways of thinking of its top professionals and their methods came from leveraging this RAG architecture with its world-class monographs on radiology. By using the image-retrieval pipeline to match target angiographic scan patterns with their underlying text descriptions, the system instantly isolated the single most relevant book in global literature. I then restricted the agent's focus to that specific monograph to effectively extract the precise domain logic, clinical workflows and other information necessary to build the software application.

Conclusion

The described architecture demonstrates the possibility of building a full-featured industrial-grade medical knowledge retrieval system under the extreme constraints of a standard consumer laptop operating fully offline. By trading the absolute relevance of articles for the consolidated structural stability of top-level monographs, this system successfully bridges the gap between vast medical literature databases and local computing frontiers. It serves as an accessible semantic research accelerator for frontline physicians, biomedical engineers, and interdisciplinary researchers alike.