<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Fawole Joshua</title>
    <description>The latest articles on DEV Community by Fawole Joshua (@fawole_joshua_c92c794ea50).</description>
    <link>https://dev.to/fawole_joshua_c92c794ea50</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F3661680%2F37e24ad4-5b61-4e77-be20-7f4e7e5383be.png</url>
      <title>DEV Community: Fawole Joshua</title>
      <link>https://dev.to/fawole_joshua_c92c794ea50</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/fawole_joshua_c92c794ea50"/>
    <language>en</language>
    <item>
      <title>Microsoft Launches Copilot For Legal: What It Means And The Implication On The Nigerian Jurisdiction</title>
      <dc:creator>Fawole Joshua</dc:creator>
      <pubDate>Mon, 19 Oct 2026 15:31:19 +0000</pubDate>
      <link>https://dev.to/fawole_joshua_c92c794ea50/microsoft-launches-copilot-for-legal-what-it-means-and-the-implication-on-the-nigerian-jurisdiction-4h5k</link>
      <guid>https://dev.to/fawole_joshua_c92c794ea50/microsoft-launches-copilot-for-legal-what-it-means-and-the-implication-on-the-nigerian-jurisdiction-4h5k</guid>
      <description>&lt;p&gt;In just five (5) months, the LegalTech world has witnessed the launching of three powerful legal-AI tools by frontier companies like OpenAI, Anthropic, Google. Now, Microsoft has made a grand entrance. In this article, I will be exploring what Microsoft Copilot actually is, the benefits, opportunities, risks and how it relates to the Nigerian Jurisdiction.&lt;/p&gt;

&lt;p&gt;In October 2026, Microsoft formally launched Copilot for Legal, a set of specialised capabilities designed to embed artificial intelligence deeply into the daily work of lawyers. Built extensively on the legal agent released to Microsoft's frontier early-access program in April 2026.&lt;/p&gt;

&lt;p&gt;It is an attempt to make AI function as a practical assistant inside the tools most lawyers already live in, particularly Microsoft Word and the newer agentic workspace called Cowork. This October launch expands this into more than 20 legal-specific skills and partner plugins.&lt;/p&gt;

&lt;p&gt;To understand what this means, one must first grasp what Microsoft has actually built, how it works at a technical and practical level, where its strengths and limitations lie, and how those realities interact with the particular conditions of Nigerian legal practice.&lt;/p&gt;

&lt;h2&gt;
  
  
  What Microsoft Copilot for Legal Actually Is
&lt;/h2&gt;

&lt;p&gt;Microsoft 365 Copilot has existed for some time as a general productivity layer that helps users draft, summarise, and analyse content across Word, Outlook, Teams, Excel and other applications.&lt;/p&gt;

&lt;p&gt;While Copilot for Legal is an evolution of that foundation. It also introduces purpose-built legal skills and two dedicated plugins, one that operates inside Word desktop and another that works within Cowork.&lt;/p&gt;

&lt;p&gt;Microsoft developed them with input from legal engineers who translated real legal workflows into structured capabilities that combine specialised reasoning with the native features of Microsoft 365.&lt;/p&gt;

&lt;p&gt;The system is powered by what Microsoft calls Work IQ. Work IQ is the intelligence layer that allows Copilot to understand not only the document currently open on a lawyer's screen but the broader organisational context.&lt;/p&gt;

&lt;p&gt;It draws on Microsoft Graph, the underlying service that connects emails, files, meetings, calendars, SharePoint sites and other data the individual user already has permission to see.&lt;/p&gt;

&lt;p&gt;In practical terms, what Work IQ does is that when a lawyer asks Copilot to review a contract or draft a memo, the system can consider related correspondence, previous versions of similar agreements stored in the firm's SharePoint, internal playbooks, and matter-related information.&lt;/p&gt;

&lt;p&gt;One good way to explain it is to view the system as an associate who has been present in meetings, depositions, and has access to mails, playbooks (set of rules) and other channels of information. It is obvious that the associate would have broader contexts and would be more accurate than a chatbot that has access to only a few documents.&lt;/p&gt;

&lt;p&gt;Furthermore, the system was designed in such a way that whenever Copilot proposes changes to a contract, those changes appear as native Word tracked changes that preserve the document's existing formatting.&lt;/p&gt;

&lt;p&gt;Also when it drafts a new agreement from a precedent, it can convert the source document into a structured template using Word content controls, creating editable placeholders that remain consistent throughout the document.&lt;/p&gt;

&lt;p&gt;All of this processing occurs within the Microsoft 365 service boundary. These design choices matter because they keep a lawyer inside the familiar environment of Word rather than forcing a switch to a separate platform.&lt;/p&gt;

&lt;p&gt;For many practitioners, this seamless integration is the single greatest practical advantage!&lt;/p&gt;

&lt;h2&gt;
  
  
  What Exactly Are Legal Skills?
&lt;/h2&gt;

&lt;p&gt;When dealing with a legal AI tool, a legal professional who wants his contracts analyzed has to utilize carefully crafted prompts to navigate the AI tool to achieve the desired result. Those legal AI tools are essentially multi-purpose in function.&lt;/p&gt;

&lt;p&gt;However, a legal skill can be likened to a swiss army knife. Carefully made for different instances and versatile for many uses. A legal skill can be explained as a pre-built, specialized ability that Microsoft has created specifically for legal work. It is more than a simple prompt. Each skill knows how to perform a legal task from the beginning to the end. It is like having many departments dedicated to solving different issues.&lt;/p&gt;

&lt;p&gt;Microsoft has released more than twenty purpose-built legal skills, organised around the core activities of legal work. Those skills can be explained in five categories and they are:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;(1). Drafting and Preparing.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Copilot can turn approved precedents and templates stored in SharePoint into new agreements or structured legal memos.&lt;/p&gt;

&lt;p&gt;It can draft a new contract from a precedent, generating a template with placeholders that the lawyer can then populate with specific details. It can also create a playbook from a collection of prior agreements and as well extract preferred positions and language.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;(2). Reviewing and Negotiating&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The system can perform structured contract reviews in alignment with the already established playbook. A lawyer uploads or attaches a playbook in .docx format, Copilot analyses it, breaks it into topics, rules and example clauses, and then systematically reviews the target contract against those rules.&lt;/p&gt;

&lt;p&gt;It doesn't stop there, it goes further to propose redlines as native tracked changes, often with explanatory comments. It can also summarise existing redlines, explaining the impact of each change and which party benefits.&lt;/p&gt;

&lt;p&gt;It can work inside documents that already contain tracked changes, distinguishing prior revisions from new proposals so that negotiation history remains intact.&lt;/p&gt;

&lt;p&gt;While additional capabilities include checking defined terms for consistency, identifying broken or unclear cross-references, assessing execution readiness before signing, and comparing a draft against previously agreed commercial terms.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;(3). Obligation Tracking&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Copilot extracts meetings, deadlines and other commitments and it constructs triggers to create an obligation tracker.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;(4). Research and guidance&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;These skills enable comparison of materials and requirements. Also identifies potential regulatory gaps, provides side-by-side comparison of key terms across documents or jurisdictions, gives explanation of complex legal concepts in clearer language, and summarisation of entire legal matters including key developments, open questions and deadlines.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;(5). Matter progression&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Copilot supports the construction of case chronologies from underlying materials, the conversion of meeting transcripts into formal minutes and resolutions, and the initiation of related workflows such as drafting emails, updating spreadsheets or creating presentations.&lt;/p&gt;

&lt;p&gt;On paper, the architecture is formidable. Whether the performance matches the architecture is a separate question and will be further discussed later-on in this article.&lt;/p&gt;

&lt;h2&gt;
  
  
  Partner Plugins
&lt;/h2&gt;

&lt;p&gt;Alongside its own skills, Microsoft has opened Copilot to partner plugins from established legal technology companies. These include Bloomberg Law for dockets and litigation intelligence, Harvey and Legora for specialised legal AI, iManage and NetDocuments for document management, Docusign and other electronic signature platforms, and e-discovery tools such as Relativity and Everlaw.&lt;/p&gt;

&lt;p&gt;It is important to note that paying for Microsoft 365 Copilot does not grant free access to Harvey, Bloomberg Law or any other partner service. The plugin merely creates a connection. A lawyer or firm must still hold a separate subscription to the partner product. In practice, a user might ask Copilot to pull docket information from Bloomberg Law or to run a deeper analysis through Harvey, but only if the relevant licences are in place. This architecture explains why specialised tools continue to matter even after Microsoft's extensive legal rollout.&lt;/p&gt;

&lt;p&gt;However, is there a moment when Microsoft would move to create specialized legal AI tools like that of Harvey and Legora, because they have both the technical capabilities and capital needed for such? That is a question to ruminate on.&lt;/p&gt;

&lt;p&gt;The plugin model is, for now, a hedge. Microsoft supplies the surface, the partners supply the depth. However, whether that division of labour survives the next eighteen months is an open commercial question, not a settled technical one.&lt;/p&gt;

&lt;h2&gt;
  
  
  Accuracy
&lt;/h2&gt;

&lt;p&gt;This is the section where I must depart most sharply from Microsoft's own framing, because the independent evidence tells a different story.&lt;/p&gt;

&lt;p&gt;While, Microsoft's early usage data from the Frontier period is real, and it is worth stating precisely. Over 1,800 users generated nearly 100,000 edits and kept roughly 70% of suggestions. Internal evaluations scored edits above 90% for fluency, above 95% for conciseness, and above 85% for relevancy.&lt;/p&gt;

&lt;p&gt;But these are self-reported figures from a vendor describing its own product during a controlled early-access programme. One contemporaneous analysis noted explicitly that the figures "reflect early usage rather than independent measurements of accuracy or productivity." &lt;/p&gt;

&lt;p&gt;The independent picture is considerably less flattering, and it matters for how a Nigerian practitioner should read the launch.&lt;/p&gt;

&lt;p&gt;The most substantial independent study of legal AI adoption in 2026, the Law Insider AI Adoption Study, covering 534 legal professionals across 75 countries, found that Microsoft Copilot had the lowest user confidence of the six tools analysed. Only 4% of Copilot users described themselves as "very confident" in its output. The range across all tools ran from 4% to 18%, and no tool cleared 20%.&lt;/p&gt;

&lt;p&gt;Copilot was the second-most-adopted tool in the study and simultaneously the least trusted. The study's author drew a pointed conclusion: "Usage goes to drafting clauses... Each is a task where the lawyer can check the output by reading it... AI is allowed to produce language. It is not allowed near judgment. And the boundary sits exactly where verification stops being possible." &lt;/p&gt;

&lt;p&gt;A separate body of adoption data reinforces the point. Recon Analytics, surveying over 150,000 respondents, tracked Copilot's accuracy Net Promoter Score (NPS) falling from -3.5 in July 2025 to -24.1 by September 2025, recovering only partially to -19.8 by January 2026. Negative NPS means users who try the tool are more likely to distrust its output than to recommend it. Among users who stopped using Copilot, 44.2% cited "distrust of answers" as their primary reason. And 64% of Copilot licensees were not actively using the tool, according to Redress Compliance data.&lt;/p&gt;

&lt;p&gt;There is also a benchmark problem. Independent legal benchmarking has placed Microsoft Copilot at the bottom of substantive legal task performance. In the Legal Benchmarks results updated in October 2026, Copilot recorded the lowest substantive pass rates across the categories tested (12.9% on contract workflows and 20.0% on data extraction) as well as the lowest form score.&lt;/p&gt;

&lt;p&gt;None of this means Copilot is useless. It means the confidence narrative in Microsoft's launch materials is not supported by independent measurement. A lawyer who reads only the 70% retention figure and the 85% relevancy score is reading only a supplier's self-assessment report, realistically, no supplier would broadcast the downsides of their product.&lt;/p&gt;

&lt;h2&gt;
  
  
  Confidentiality, and Professional Responsibility
&lt;/h2&gt;

&lt;p&gt;One of the most significant concerns raised by lawyers is that Copilot can access any confidential information the individual user can already see. This is correct. Through Work IQ and Microsoft Graph, the system inherits the user's existing permissions. If a lawyer has access to privileged client correspondence, internal strategy notes or sensitive commercial terms, Copilot can draw on that material when responding to prompts.&lt;/p&gt;

&lt;p&gt;This design is intentional. It allows the AI to provide contextually relevant assistance rather than operating in isolation. At the same time, when looked at critically, it actually creates a real tension with traditional notions of confidentiality.&lt;/p&gt;

&lt;p&gt;The risk is not that Microsoft or third parties receive unrestricted access to client files. Data remains within the customer's Microsoft 365 tenant, is not used for model training, and is protected by the same enterprise security controls that apply to the rest of the suite.&lt;/p&gt;

&lt;p&gt;The risk is in the breadth of active access granted to an AI system that can search across everything the lawyer can open. For example, a loose or careless prompt can expose confidential information about a client in a mail to another client.&lt;/p&gt;

&lt;p&gt;While firms can mitigate risk through careful permission management, sensitivity labelling, Data Loss Prevention policies and clear internal protocols on prompt hygiene, and regular audit of access logs.&lt;/p&gt;

&lt;p&gt;But prompt hygiene is not the whole of the confidentiality problem, and it is important to say so plainly. There is a structural records dimension that the launch materials do not foreground.&lt;/p&gt;

&lt;p&gt;Copilot prompts and responses are stored in a hidden folder within the user's Exchange Online mailbox, and they are treated like other communications data for compliance purposes. That means they are searchable by eDiscovery tools, can be placed on Litigation Hold, and have their permanent deletion suspended while the mailbox is under an eDiscovery hold.&lt;/p&gt;

&lt;p&gt;The practical consequence is that a lawyer's prompts to Copilot and Copilot's responses become part of the organisation's discoverable data footprint. A careless prompt that pulls privileged material into a Copilot session does not simply vanish when the session ends. It sits in the mailbox, within the scope of a litigation hold, exportable for legal review.&lt;/p&gt;

&lt;p&gt;Three federal rulings in the United States in 2026 reached different results on AI-generated materials, and none of them settled whether enterprise terms preserve privilege over Copilot interactions. That is an unresolved legal question, not a solved one. Microsoft's own documentation describes the retention behaviour, it does not resolve the privilege consequences.&lt;/p&gt;

&lt;h2&gt;
  
  
  Implications for the Nigerian Legal Market
&lt;/h2&gt;

&lt;p&gt;In the Nigerian Jurisdiction, many practitioners and firms rely on Microsoft 365. The ability to obtain full and very effective AI assistance without leaving Word is therefore attractive.&lt;/p&gt;

&lt;p&gt;However, several structural realities limit the immediate transformative impact. Microsoft's legal skills are not trained primarily on Nigerian statutes, case law or practice. They lack the jurisdiction-specific depth that tools such as LawPavilion, ModulawAI or CaseRadar attempt to provide by grounding responses in Nigerian judgments and legislation.&lt;/p&gt;

&lt;p&gt;A Nigerian lawyer using Copilot for research or citation risks receiving inaccurate or foreign-influenced answers unless the output is rigorously verified against local sources. Given the independent accuracy data discussed above, that risk is strong. A system that underperforms on generic legal benchmarks is unlikely to perform better on a jurisdiction it was not trained on.&lt;/p&gt;

&lt;p&gt;This presents a strong opportunity in the Nigerian legal tech market as any LegalTech companies that become the first to be integrated into Microsoft Copilot through a model context protocol (MCP) would attract a lot of new users.&lt;/p&gt;

&lt;p&gt;Over time, if Nigerian LegalTech companies succeed in integrating as partners, Copilot could become a more powerful vehicle for delivering localised legal intelligence inside the tools lawyers already use but till then, it is less useful in the Nigerian Jurisdiction.&lt;/p&gt;

&lt;h2&gt;
  
  
  Conclusion
&lt;/h2&gt;

&lt;p&gt;Microsoft Copilot for Legal does not arrive in Nigeria as a finished solution. It arrives as a powerful but incomplete instrument, one that accelerates the mechanical parts of legal work. Therefore Copilot needs Nigerian LegalTech companies to partner with Microsoft as that will enable the powerful instrument introduced by the latter to be useful for lawyers in the Nigerian Jurisdiction.&lt;/p&gt;

&lt;p&gt;But the incompleteness is deeper than the localisation gap. The independent trust and accuracy data suggest that even in the markets where Copilot has been deployed longest, lawyers are using it heavily while trusting it lightly.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>legaltech</category>
      <category>legalai</category>
      <category>microsoft</category>
    </item>
    <item>
      <title>Why Signature-Based Security Is No Longer Enough To Detect Cyber Attacks And How UEBA Hunts vicious threats.</title>
      <dc:creator>Fawole Joshua</dc:creator>
      <pubDate>Fri, 20 Mar 2026 13:18:20 +0000</pubDate>
      <link>https://dev.to/fawole_joshua_c92c794ea50/why-signature-based-security-is-no-longer-enough-to-detect-cyber-attacks-and-how-ueba-hunts-vicious-3np8</link>
      <guid>https://dev.to/fawole_joshua_c92c794ea50/why-signature-based-security-is-no-longer-enough-to-detect-cyber-attacks-and-how-ueba-hunts-vicious-3np8</guid>
      <description>

&lt;h2&gt;
  
  
  Introduction
&lt;/h2&gt;

&lt;p&gt;Imagine a national museum that holds thousands of historic antiquities. &lt;br&gt;
&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Ftspmr09un5vpmst1y33e.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Ftspmr09un5vpmst1y33e.png" alt=" " width="800" height="520"&gt;&lt;/a&gt;&lt;br&gt;
The monuments are very precious, worth billions and are very essential to the preservation of the cultural heritage of that nation.&lt;/p&gt;

&lt;p&gt;The department of security has the job of preventing unwanted persons from entering the museum. In order to do the job effectively, security personnel have lists and mugshots of criminals. At every entrance, guards check visitors against a book of mugshots actively searching for known criminals, troublemakers and persons of interest.&lt;/p&gt;

&lt;p&gt;The identity of anyone entering into the museum is verified. For centuries this worked because a criminal looked like a criminal; they wear suspicious clothes, walk around aimlessly, carry odd bags and lose their temper at the slightest questioning by security guards.&lt;/p&gt;

&lt;p&gt;However, criminals and people with malicious intentions evolved. Today's thieves don't look like thieves. They dress sharply, walk with confidence, and never once glance nervously at a security camera. They've done their homework. They know which employees have access to the restricted wings. They've studied their mannerisms, their routines, their faces. And on the day of the heist, they walk straight through the front gate wearing the face of a trusted curator.&lt;br&gt;
&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F8v0ic6q0l8ed09gnnwsv.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F8v0ic6q0l8ed09gnnwsv.webp" alt=" " width="800" height="932"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Mugshots became useless, posters became increasingly less effective and thieves walked straight through the gate into the museum with little or no stress.&lt;/p&gt;

&lt;p&gt;This is the problem with modern cybersecurity, your network is the museum attackers are trying to access, your databases and servers are precious and invaluable artifacts . While your firewalls, antivirus softwares and intrusion detection systems are the security guards with mugshots, lists of criminals and types of attacks expected.&lt;/p&gt;

&lt;p&gt;They are checking everything against a list of known-bad signatures, malicious IPs and known attack patterns. This is not effective in any form. Attackers no longer look like attackers, they use stolen credentials, they avoid irrational and suspicious movements, they move stealthily past your rules using different and dynamic techniques and head straight for your data.&lt;/p&gt;

&lt;p&gt;This article makes a sole argument: &lt;strong&gt;signature-based security alone is insufficient&lt;/strong&gt;. In a world where attackers constantly change their tools, malware architecture, IP and overall techniques, manually setting rules to detect different types of attacks is not only stressful but also impracticable and very inefficient.&lt;/p&gt;

&lt;p&gt;New malware variants appear in their thousands almost every month, there is absolutely no way antivirus software can keep up, attacks like Living-off-the-land use legitimate tools and require no bad signature. Zero-day-exploits are even worse. They are ghosts and leave almost no signature.&lt;/p&gt;

&lt;p&gt;This means, attackers have an easy entrance to networks. Furthermore, they can move mountains if they lay their hands on an authorized account and the signature-based security system will not flag it.&lt;/p&gt;

&lt;p&gt;The cat and mouse game of &lt;strong&gt;find-the-bad-activities&lt;/strong&gt; is a losing battle. The only sustainable defence mechanism is to know what good activities look like. The defence system needs to know everything about it such that bad activities cannot hide no matter what form it takes.&lt;/p&gt;

&lt;p&gt;This is the philosophical shift at the heart of modern threat hunting. It is the undiluted application of machine learning and artificial intelligence in the realm of security. This is the security agency that does not need a list of attacks before it can successfully flag one.&lt;/p&gt;

&lt;p&gt;If Jude from the HR department at the museum situated in California suddenly logs in on a Sunday evening from Dubai and starts downloading 678 gigabytes of customer data, we don't need to debate whether the IP is malicious or that the download tool has a signature. It is obvious that this is an unusual activity and has never happened in the space of 5 years that Jude had been working with the Museum, then it will certainly be flagged as an anomaly.&lt;/p&gt;
&lt;h2&gt;
  
  
  &lt;strong&gt;User and Entity Behaviour Analytics (UEBA) Detection: How it Learns.&lt;/strong&gt;
&lt;/h2&gt;

&lt;p&gt;The government of the country realizes the critical issue at the museum and decided to introduce a special task force to help the security department. This is where UBEA comes in.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;User and Entity Behaviour Analytics&lt;/strong&gt; focuses on studying everyone and everything (humans, their instruments and other factors) this is done in order to establish a ground truth, what we can otherwise term as normal activities. Anything apart from these normal activities are potential threats.&lt;/p&gt;

&lt;p&gt;Imagine a new security guard named Owen &lt;br&gt;
&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fb2nt7wabdw0bu3ahm9q5.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fb2nt7wabdw0bu3ahm9q5.jpg" alt=" " width="800" height="450"&gt;&lt;/a&gt;(UEBA) assigned to the Museum. For his first three months Owen does nothing but to carefully observe every employee, every visitor, every delivery person. Their times of resumption, exit, levels of access and general mode of conducting their activities.&lt;/p&gt;

&lt;p&gt;Owen is not just memorizing facts, he is building a robust infrastructure that will serve as the baseline for evaluating all future activities. This is what UEBA does with your data, it consumes logs from countless sources:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Authentication logs&lt;/strong&gt; (VPN, Active Directory)&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Network flows&lt;/strong&gt; (NetFlow, DNS queries)&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Endpoint logs&lt;/strong&gt; (process creation, file access)&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Application logs&lt;/strong&gt; (database queries, web server access)&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Cloud service logs&lt;/strong&gt; (Office 365, AWS, Salesforce)&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fathmq5se5wq2p1chk8sn.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fathmq5se5wq2p1chk8sn.png" alt=" " width="800" height="741"&gt;&lt;/a&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;
&lt;span class="n"&gt;Assuming&lt;/span&gt; &lt;span class="n"&gt;you&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;ve loaded the data, imported the libraries and performed data cleaning and preprocessing 

print(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="s"&gt;[3] Feature Engineering for Behavioral Profiles&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;)

df_behavior = df.copy()
df_behavior[&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="n"&gt;is_attack&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;] = (df_behavior[&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="n"&gt;Label&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;] == &lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="n"&gt;DDoS&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;).astype(int)

# replace inf values first
df_behavior = df_behavior.replace([np.inf, -np.inf], np.nan)

# 1. Packet rate features - add small epsilon to avoid division by zero

df_behavior[&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="n"&gt;packets_per_second&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;] = df_behavior[&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="n"&gt;Total&lt;/span&gt; &lt;span class="n"&gt;Fwd&lt;/span&gt; &lt;span class="n"&gt;Packets&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;] / (df_behavior[&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="n"&gt;Flow&lt;/span&gt; &lt;span class="n"&gt;Duration&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;] + 1e-10)

df_behavior[&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="n"&gt;bytes_per_packet&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;] = df_behavior[&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="n"&gt;Total&lt;/span&gt; &lt;span class="n"&gt;Length&lt;/span&gt; &lt;span class="n"&gt;of&lt;/span&gt; &lt;span class="n"&gt;Fwd&lt;/span&gt; &lt;span class="n"&gt;Packets&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;] / (df_behavior[&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="n"&gt;Total&lt;/span&gt; &lt;span class="n"&gt;Fwd&lt;/span&gt; &lt;span class="n"&gt;Packets&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;] + 1e-10)

# 2. Flag ratios

df_behavior[&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="n"&gt;syn_ack_ratio&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;] = df_behavior[&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="n"&gt;SYN&lt;/span&gt; &lt;span class="n"&gt;Flag&lt;/span&gt; &lt;span class="n"&gt;Count&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;] / (df_behavior[&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="n"&gt;ACK&lt;/span&gt; &lt;span class="n"&gt;Flag&lt;/span&gt; &lt;span class="n"&gt;Count&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;] + 1e-10) \ (df_behavior[&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="n"&gt;ACK&lt;/span&gt; &lt;span class="n"&gt;Flag&lt;/span&gt; &lt;span class="n"&gt;Count&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;] &amp;gt; 0).astype(int)...
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Output&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;[3] Feature Engineering for Behavioral Profiles
Created new behavioral features:
['packets_per_second', 'bytes_per_packet', 'syn_ack_ratio', 'flag_diversity', 'fwd_bwd_ratio', 'packet_size_variation', 'iat_cv']
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;From this raw data, UEBA extracts behavioural features and truth for every employee and user. It learns that Jonathan from the accounting department usually logs-in in the morning at 7:35AM, opens the spreadsheet and had never attempted to open the organization's source code repository.&lt;/p&gt;

&lt;p&gt;That series of observations is the &lt;strong&gt;ground truth&lt;/strong&gt;, any deviation is tantamount to a breach. The concept of ground truth is perhaps the strongest asset a defender can possess.&lt;/p&gt;

&lt;h2&gt;
  
  
  &lt;strong&gt;How UEBA Works and Prevents Spamming of False Positives.&lt;/strong&gt;
&lt;/h2&gt;

&lt;p&gt;If UEBA relies on the establishment of the ground truth, does that mean it fluctuates and flags everything that deviates from the ground truth?&lt;/p&gt;

&lt;p&gt;Not exactly, however, it only flags but does not report everything. It classifies signals according to a predefined and domain-specific level of seriousness (risk scoring system). This ensures that SOC Analysts are not drowned in threat reports which later turns out to be insignificant or totally non-malicious.&lt;/p&gt;

&lt;p&gt;Say, on the 23rd of June, 2025, a thief manages to compromise the account of a young employee at the maintenance department of the museum. Let's call this account “&lt;strong&gt;IamCareless600&lt;/strong&gt;”&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Day 1&lt;/strong&gt;: the account logs in on a Saturday at 10:45PM. This is unusual and the owner has never done this. Owen sees it but doesn't react (maybe it's an emergency, or he just needs to get something).&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Day 2&lt;/strong&gt;: The account logs in on Monday, but instead of heading to the maintenance department, the account went to the restoration lab (a place he had never gone to) the account was denied entrance, it headed again to the administrative block and the entrance was further denied and finally starts making its way to the server room. Access once again denied.&lt;/p&gt;

&lt;p&gt;Owen now has weak signals:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;An anomalous late night entry&lt;/li&gt;
&lt;li&gt;Multiple failed access attempts to restricted areas&lt;/li&gt;
&lt;li&gt;A pattern of wandering that does not match any employees normal behaviour&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Day 3:&lt;/strong&gt; IamCareless600 Logs in and immediately tries to gain access to the database, he succeeds this time and starts transferring 500 gig of file to an external IP in a foreign country. The combination of these activities give Owen a strong probable cause.&lt;/p&gt;

&lt;p&gt;Owen's machine-learning-powered brain correlates the signals: [&lt;strong&gt;Anomalous entry time&lt;/strong&gt;] + [&lt;strong&gt;Multiple failed access attempts to restricted areas&lt;/strong&gt;] + [&lt;strong&gt;First-time server access&lt;/strong&gt;] + [&lt;strong&gt;Massive data exfiltration&lt;/strong&gt;] = &lt;strong&gt;COMPROMISED ACCOUNT&lt;/strong&gt;. Owen doesn't raise a generic alarm. He runs to the security team with a precise report.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fyk20wz4q384bvo7uhmdo.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fyk20wz4q384bvo7uhmdo.png" alt=" " width="800" height="398"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The thief is caught in the act, halfway through stealing the museum's most precious records. This is threat hunting. This is the difference between waiting for an alarm and actively watching out for strange activities.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;
&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="s"&gt;[5] Detecting Anomalous Ports with Isolation Forest&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;port_features&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;total_flows&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;avg_flow_duration&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;avg_packet_size&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;avg_packet_rate&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;syn_ack_ratio&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;packet_size_std&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;

&lt;span class="n"&gt;X_port&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;port_df&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;port_features&lt;/span&gt;&lt;span class="p"&gt;].&lt;/span&gt;&lt;span class="nf"&gt;copy&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; 
&lt;span class="c1"&gt;# Handle any remaining infinite or NaN values
&lt;/span&gt;&lt;span class="n"&gt;X_port&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;X_port&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;replace&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;&lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;inf&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;inf&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;nan&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;X_port&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;X_port&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;fillna&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;X_port&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;mean&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;

&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;sklearn.preprocessing&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;RobustScaler&lt;/span&gt;
&lt;span class="n"&gt;scaler&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;RobustScaler&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="n"&gt;X_scaled&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;scaler&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;fit_transform&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;X_port&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;expected_contamination&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;port_df&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;is_malicious_port&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;].&lt;/span&gt;&lt;span class="nf"&gt;mean&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Expected contamination: &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;expected_contamination&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="o"&gt;%&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;sklearn.ensemble&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;IsolationForest&lt;/span&gt;
&lt;span class="n"&gt;iso_forest&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;IsolationForest&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;n_estimators&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;100&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;contamination&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;expected_contamination&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;random_state&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;42&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;bootstrap&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;False&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;  &lt;span class="c1"&gt;# Disable bootstrap to avoid issues
&lt;/span&gt;&lt;span class="n"&gt;iso_forest&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;fit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;X_scaled&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;port_df&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;anomaly_score&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;iso_forest&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;decision_function&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;X_scaled&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;port_df&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;predicted_anomaly&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;iso_forest&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;predict&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;X_scaled&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;astype&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;int&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="s"&gt;Top 20 most anomalous ports:&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;anomalous&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;port_df&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sort_values&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;anomaly_score&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;head&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;20&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; 

&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;anomalous&lt;/span&gt;&lt;span class="p"&gt;[[&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;port&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;total_flows&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;attack_rate&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;anomaly_score&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;attack_rate&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;anomaly_score&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;is_malicious_port&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;]].&lt;/span&gt;&lt;span class="nf"&gt;to_string&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  &lt;strong&gt;Practical Implementation of UEBA Using CIC-IDS-2017 Dataset as a Case Study&lt;/strong&gt;
&lt;/h2&gt;

&lt;p&gt;We conducted a practical implementation of UEBA on a collection of network traffic containing both normal activity and DDoS attacks.&lt;/p&gt;

&lt;p&gt;Owen was able to find the needle in the haystack, using an isolation forest to detect anomaly. We discovered that port 80, the web server, was drowning in a traffic attack: 136,951 flows, 93% of them malicious. The volume was 4.1 standard deviations above normal. The packet sizes were also 3.4 standard deviations above normal. The probability of this happening by chance is less than 1 in 35 million.&lt;/p&gt;

&lt;p&gt;Owen does not need to be aware of how the attack was made or what name was it called, all he cares about is that “This is an unusual activity, a dangerous one at that and must be instantly stopped”. Here is the link to the comprehensive code &lt;a href="https://github.com/Akanji102/DDoS-anomaly-detection-using-Isolation-forest" rel="noopener noreferrer"&gt;https://github.com/Akanji102/DDoS-anomaly-detection-using-Isolation-forest&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Classification Report&lt;/strong&gt;:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;precision&lt;/strong&gt; &lt;strong&gt;recall&lt;/strong&gt; &lt;strong&gt;f1-score&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Normal Port&lt;/strong&gt; 1.00 1.00 1.00 19&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Malicious Port&lt;/strong&gt; 1.00 1.00 1.00 1&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Accuracy&lt;/strong&gt;: 100%&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;ROC-AUC&lt;/strong&gt;: 1.00&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Matthews Correlation Coefficient&lt;/strong&gt;: 1.00&lt;/p&gt;

&lt;p&gt;Although the dataset was generated under a constrained environment and was essentially made for educational purposes, it actively demonstrates how Isolation forest, other unsupervised algorithms and deep learning networks can detect strange activities and any form of attack, old or new.&lt;/p&gt;

&lt;h2&gt;
  
  
  &lt;strong&gt;Tools and Techniques UEBA Utilizes to Hunt Threats.&lt;/strong&gt;
&lt;/h2&gt;

&lt;p&gt;Owen isn't a single model, he is an ensemble of different algorithms and tools, all working together to classify behaviours and detect anomalies. Some of them include&lt;/p&gt;

&lt;h2&gt;
  
  
  &lt;strong&gt;(1). Temporal analysis:&lt;/strong&gt;
&lt;/h2&gt;

&lt;p&gt;Temporal analysis (time series analysis) models every user and entity's activity as a pattern over time. It doesn't just track what they do, but when they do it and in what sequence. It detects unusual login times, modified work patterns and sequence violations.&lt;/p&gt;

&lt;p&gt;The algorithms here range from statistical methods like &lt;strong&gt;Seasonal ARIMA&lt;/strong&gt; (which captures weekly patterns) to deep learning approaches like &lt;strong&gt;LSTMs&lt;/strong&gt; (which excel at learning sequences).&lt;/p&gt;

&lt;h2&gt;
  
  
  &lt;strong&gt;(2). Graph Analysis:&lt;/strong&gt;
&lt;/h2&gt;

&lt;p&gt;UEBA as a defence system does not see every account or network as a single node that only needs to be studied independently, it sees them as a giant dynamic web of connections.&lt;/p&gt;

&lt;p&gt;It detects data exfiltration, lateral movements and insider collusion. When a Clerk suddenly starts interacting and sending huge data files to businessmen in the Middle East or a group of connections keep moving with malicious intents, they are instantly studied as a whole. UEBA traces the underlying relationships between every single network in order to detect fraud.&lt;/p&gt;

&lt;p&gt;The magic here lies in algorithms like &lt;strong&gt;Community Detection&lt;/strong&gt; (which automatically finds groups that normally work together) and &lt;strong&gt;Graph Neural Networks&lt;/strong&gt; (which learn to spot structural anomalies).&lt;/p&gt;

&lt;h2&gt;
  
  
  &lt;strong&gt;(3). Statistical analysis:&lt;/strong&gt;
&lt;/h2&gt;

&lt;p&gt;The Volume Detective tracks quantities, data volumes, file counts, action frequencies. It builds what “normal” should look like. A marketing intern might possibly download an average of 200mb of data per day while a video editor might go as far as 10gig.&lt;/p&gt;

&lt;p&gt;It detects massive downloads (a video editor normally should not download salary dataset or download a dataset of 500 gig). It uses models like &lt;strong&gt;Gaussian distribution&lt;/strong&gt;, &lt;strong&gt;moving averages&lt;/strong&gt;, &lt;strong&gt;exponential smoothing&lt;/strong&gt; etc.&lt;/p&gt;

&lt;h2&gt;
  
  
  &lt;strong&gt;(4). Unsupervised Learning:&lt;/strong&gt;
&lt;/h2&gt;

&lt;p&gt;This is the instrument for setting the final ground truth. It is the sorting of all of the data into clusters in order to find the odd ones. It creates the truth and what should be avoided.&lt;/p&gt;

&lt;p&gt;It includes models like &lt;strong&gt;KMeans&lt;/strong&gt;, &lt;strong&gt;DBScan&lt;/strong&gt;, &lt;strong&gt;Hierarchical Clustering&lt;/strong&gt; and a very good one is also &lt;strong&gt;Isolation forest.&lt;/strong&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  &lt;strong&gt;Things to note before implementingUEBA&lt;/strong&gt;
&lt;/h2&gt;

&lt;h2&gt;
  
  
  &lt;strong&gt;(1). Data Quality:&lt;/strong&gt;
&lt;/h2&gt;

&lt;p&gt;UEBA as a defence system is only as good as the one on which it was built. The data should be clean, realistic and gathered with a good pipeline. Bad data automatically connotes bad UEBA.&lt;/p&gt;

&lt;h2&gt;
  
  
  &lt;strong&gt;(2). Cold Start:&lt;/strong&gt;
&lt;/h2&gt;

&lt;p&gt;As previously explained, UEBA needs a significant amount of time to gather knowledge and establish the base line. Therefore, in the first few weeks of setting it up, the signals will be noisy and will not generate favourable results but it will increase in accuracy over time.&lt;/p&gt;

&lt;h2&gt;
  
  
  &lt;strong&gt;(3). Concept Drift:&lt;/strong&gt;
&lt;/h2&gt;

&lt;p&gt;Companies are not static, roles change and so do policies, it is therefore recommended that UEBA models be retrained after major drifts in order to ensure accuracy&lt;/p&gt;

&lt;h2&gt;
  
  
  &lt;strong&gt;(4). Signature-based and human-in-the-loop:&lt;/strong&gt;
&lt;/h2&gt;

&lt;p&gt;UEBA doesn't replace signature-based threat detection, it enriches it. The system does not replace human analysts as well, it empowers them. Human hunters investigate, confirm or refute, and provide feedback that closes the loop and improves future detection. This symbiosis is essential.&lt;/p&gt;

&lt;h2&gt;
  
  
  &lt;strong&gt;Conclusion&lt;/strong&gt;
&lt;/h2&gt;

&lt;p&gt;There are two options available for the hypothetical scenario presented in this article: the museum can add more guards and maintain its rule-based system or it could fundamentally rethink its approach, shifting from reactive detection to proactive behavior monitoring.&lt;/p&gt;

&lt;p&gt;The same applies to the security of your company's data, UEBA gives flexibility, dynamic reaction and cautious proactiveness. It is currently one of the strongest defence mechanisms against cyber attacks.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>cybersecurity</category>
      <category>machinelearning</category>
      <category>deeplearning</category>
    </item>
    <item>
      <title>How AI Learns: Gradient Descent Explained Through a Midnight Smoky Jollof Adventure</title>
      <dc:creator>Fawole Joshua</dc:creator>
      <pubDate>Tue, 16 Dec 2025 07:59:24 +0000</pubDate>
      <link>https://dev.to/fawole_joshua_c92c794ea50/how-ai-learns-gradient-descent-explained-through-a-midnight-smoky-jollof-adventure-3ggh</link>
      <guid>https://dev.to/fawole_joshua_c92c794ea50/how-ai-learns-gradient-descent-explained-through-a-midnight-smoky-jollof-adventure-3ggh</guid>
      <description>&lt;p&gt;Many aspects of the modern world are now powered by artificial intelligence, and this has significantly accelerated human civilization.&lt;/p&gt;

&lt;p&gt;From faster disease detection to automated decision-making. From breakthroughs in medical imaging to the quiet and rapid adoption of artificial intelligence in law firms and the entire judicial system. Artificial intelligence is actively reshaping the future of agriculture and its impact can be felt across nearly all sectors.&lt;/p&gt;

&lt;p&gt;Yet, despite this tremendous progress, many people do not actually understand where artificial intelligence gets its brilliance from. AI's ability to identify errors and iteratively improve is certainly amazing.&lt;/p&gt;

&lt;p&gt;This article will gently hold you by the hand and explain the true superpower behind AI and machine learning.&lt;/p&gt;

&lt;p&gt;The answer lies in a simple mathematical algorithm called Gradient Descent.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What is Gradient Descent?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Gradient descent can be explained as a general-purpose mathematical algorithm that is capable of finding the best solutions to a very wide range of problems. In machine learning, it works by rapidly updating parameters to quickly minimize a loss (or cost) function.&lt;/p&gt;

&lt;p&gt;In very simple terms, Gradient descent helps AI figure out how wrong it is and how to quickly become less wrong.&lt;/p&gt;

&lt;p&gt;To explicitly understand what gradient descent is, its complexities and purpose, we can look under the hood and reason like an AI model.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Midnight Smoky Jollof Adventure&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Say you went for Thanksgiving and your mom cooked a special and very taste-bud-pleasing Nigerian jollof. Thanksgiving was perfect, you reconnected with your siblings and then everyone went to bed. But in the middle of the night, your brain and tongue just kept craving more, the smoky jollof rice was so tantalising that you could smell it several feet away.&lt;/p&gt;

&lt;p&gt;You resisted the feeling but it got the better side of you and so you stood up and started making your way to the kitchen. But here is the problem, the lights are off, you can't see a thing. You don't want to get caught, nor do you want to fall off something.&lt;/p&gt;

&lt;p&gt;Imagine the house floor as a graph paper.&lt;/p&gt;

&lt;p&gt;X-axis = left-right position&lt;/p&gt;

&lt;p&gt;Y-axis = forward-backward position&lt;/p&gt;

&lt;p&gt;Your location = coordinates (X, Y)&lt;/p&gt;

&lt;p&gt;You are currently at point (1, 1)&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The Loss Function&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;We need to find a way to measure how close we are to the jollof rice.&lt;/p&gt;

&lt;p&gt;Normal distance formula:&lt;/p&gt;

&lt;p&gt;Distance = √(x - 3)²+(y - 4)²&lt;/p&gt;

&lt;p&gt;Let's just use squared distance:&lt;/p&gt;

&lt;p&gt;Loss(x, y) = (x - 3)²+(y - 4)²&lt;/p&gt;

&lt;p&gt;This loss is very important, it will be our compass to get to the jollof rice, it will show how far-off (wrong) we are.&lt;/p&gt;

&lt;p&gt;The higher the loss, the more wrong we are (i.e we are very far off from the kitchen). Therefore, our goal is to greatly reduce the loss function so that we can reach the kitchen and the jollof rice.&lt;/p&gt;

&lt;p&gt;At starting point (1, 1):&lt;/p&gt;

&lt;p&gt;Loss = (1 - 3)² + (1 - 4)² = (-2)² + (-3)² = 4 + 9 = 13. This means we are very far from the kitchen.&lt;/p&gt;

&lt;p&gt;Testing Directions&lt;/p&gt;

&lt;p&gt;Then let's tweak the parameters a little:&lt;/p&gt;

&lt;p&gt;From (1, 1) to (1.001, 1)&lt;/p&gt;

&lt;p&gt;New loss: (1.001 - 3)² + (1 - 4)² = (-1.999)² + (-3)² = 3.996 + 9 = 12.996&lt;/p&gt;

&lt;p&gt;The old loss was 13, now the new loss is 12.996 (decreased by 0.004, we are making progress!)&lt;/p&gt;

&lt;p&gt;Then let's say we tweak the parameters even more. From (1, 1) to (1, 1.001):&lt;/p&gt;

&lt;p&gt;New loss: (1 - 3)² + (1.001 - 4)² = (-2)² + (-2.999)² = 4 + 8.994 = 12.994 (getting closer)&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The Mathematical Shortcut&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Instead of testing each direction, we can take a mathematical shortcut (find the derivative):&lt;/p&gt;

&lt;p&gt;For loss = (x - 3)² + (y - 4)²&lt;/p&gt;

&lt;p&gt;How loss changes with x:&lt;/p&gt;

&lt;p&gt;If we change x by ∆x, loss changes by approximately:&lt;/p&gt;

&lt;p&gt;2 * (x - 3) * ∆x&lt;/p&gt;

&lt;p&gt;Why? This is because the derivative of (x - 3)² = 2(x - 3).&lt;/p&gt;

&lt;p&gt;So at x = 1:&lt;/p&gt;

&lt;p&gt;2 * (1 - 3) = 2 * (-2) = -4&lt;/p&gt;

&lt;p&gt;This means that for every tiny step right, loss decreases by 4 times that step size.&lt;/p&gt;

&lt;p&gt;How loss changes with y:&lt;/p&gt;

&lt;p&gt;2 * (y - 4) * ∆x&lt;/p&gt;

&lt;p&gt;At y = 1:&lt;/p&gt;

&lt;p&gt;2 * (1 - 4) = 2 * (-3) = -6&lt;/p&gt;

&lt;p&gt;For every step forward, loss decreases by 6 times that tiny step size.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The Gradient Vector&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;We put these together into a gradient vector:&lt;/p&gt;

&lt;p&gt;Gradient = [-4, -6]^T&lt;/p&gt;

&lt;p&gt;To always update our position, we need to adopt a movement sequence or otherwise called a learning rate (η = 0.1).&lt;/p&gt;

&lt;p&gt;The learning rate must not be too slow or small ( we don't want to take forever) nor should it be too fast or large ( we don't want to fall or overshoot).&lt;/p&gt;

&lt;p&gt;Now our movement formula will be:&lt;/p&gt;

&lt;p&gt;New position = old position - η * Gradient&lt;/p&gt;

&lt;p&gt;x-new = 1 - 0.1 * (-4) = 1 + 0.4 = 1.4&lt;/p&gt;

&lt;p&gt;y-new = 1 - 0.1 * (-6) = 1 + 0.6 = 1.6.&lt;/p&gt;

&lt;p&gt;We just moved from (1, 1) to (1.4, 1.6).&lt;/p&gt;

&lt;p&gt;Old loss at (1, 1) = 13&lt;/p&gt;

&lt;p&gt;New loss at (1.4, 1.6) = (1.4 - 3)² + (1.6 - 4)² = (-1.6)² + (-2.4)² = 2.56 + 5.76 = 8.32&lt;/p&gt;

&lt;p&gt;We just improved from a loss of 13 to only 8.32, this is great progress and we are certainly close to the kitchen now.&lt;/p&gt;

&lt;p&gt;Next Iterations&lt;/p&gt;

&lt;p&gt;As our little journey continues we compute the next gradients:&lt;/p&gt;

&lt;p&gt;Now at (1.4, 1.6):&lt;/p&gt;

&lt;p&gt;For x: 2 * (1.4 - 3) = 2 * (-1.6) = -3.2&lt;/p&gt;

&lt;p&gt;For y: 2 * (1.6 - 4) = 2 * (-2.4) = -4.8&lt;/p&gt;

&lt;p&gt;Gradient = [-3.2, -4.8]^T&lt;/p&gt;

&lt;p&gt;x-new = 1.4 - 0.1 * (-3.2) = 1.4 + 0.32 = 1.72&lt;/p&gt;

&lt;p&gt;y-new = 1.6 - 0.1 * (-4.8) = 1.6 + 0.48 = 2.08&lt;/p&gt;

&lt;p&gt;Loss at (1.72, 2.08): (-1.28)² + (-1.92)² = 1.6384 + 3.686 = 5.3248&lt;/p&gt;

&lt;p&gt;Loss dropped from 8.32 to 5.32. Congratulations, you are now at the kitchen door!&lt;/p&gt;

&lt;p&gt;With a couple more iterations, you will have reached the global optimum, this is certain because your loss function is convex and gradient descent is guaranteed to converge (your goal: the lowest loss, little to no error).&lt;/p&gt;

&lt;p&gt;In essence, gradient descent measures the local gradient of the error function with regard to the parameter vector θ and it goes in the descending gradient. Once the gradient is zero, you have reached the minimum! (or more precisely, a critical point, which could be a minimum, maximum, or saddle point).&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;In Real Machine Learning&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Instead of 2 parameters (x, y), there are millions or billions (weights in a neural network).&lt;/p&gt;

&lt;p&gt;Instead of "squared distance," they use losses like Cross-Entropy or Mean Squared Error.&lt;/p&gt;

&lt;p&gt;Instead of one perfect pot, they navigate a complex, multi-dimensional "loss landscape" with hills, valleys, and plateaus.&lt;/p&gt;

&lt;p&gt;But the core algorithm, the relentless optimization engine, remains Gradient Descent and its smarter variants (Adam, RMSProp).&lt;/p&gt;

&lt;p&gt;This is exactly how gradient descent works and how artificial intelligence can learn patterns and improve its predictions.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Types of Gradient Descent&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Batch Gradient Descent&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;This is the process whereby all training examples are utilized to compute the gradient, then take one update step!&lt;/p&gt;

&lt;p&gt;θ_new = θ_old - η * (1/m) * Σ(∇L(θ, x_i, y_i))&lt;/p&gt;

&lt;p&gt;Where:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;m = total number of training examples&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;η = learning rate&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;∇L = gradient for example&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Stochastic Gradient Descent&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;While Batch gradient descent uses the whole training data to compute the gradient at every step which eventually greatly slows down computation, stochastic gradient descent on the other hand only picks a random instance in the training set at every step and then computes the gradients based on that single instance.&lt;/p&gt;

&lt;p&gt;This makes the algorithm much faster but also noisier. Due to its stochasticity, SGD's stochastic noise can help it escape some local minima and will also end up very close to the global optimum but with a constant learning rate, it oscillates around the minimum rather than converging exactly.&lt;/p&gt;

&lt;p&gt;For each random example i:&lt;/p&gt;

&lt;p&gt;θ = θ - η * ∇L(θ, x_i, y_i)&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Mini-Batch Gradient Descent&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;This is a system where a small batch, probably 16 or 32 is used to compute the gradient, then update is initiated. It is like the sweet spot between SGD and BGD.&lt;/p&gt;

&lt;p&gt;For each batch B of size b:&lt;/p&gt;

&lt;p&gt;∇L_batch = (1/b) * Σ ∇L(θ, x_i, y_i) for i in B&lt;/p&gt;

&lt;p&gt;θ = θ - η * ∇L_batch&lt;/p&gt;

&lt;p&gt;Conclusion&lt;/p&gt;

&lt;p&gt;Understanding how gradient descent works is very profound and points to the very fact that artificial intelligence and its system of learning isn't about being perfect from the very beginning, it's about having a reliable method to quickly and accurately become less wrong.&lt;/p&gt;

&lt;p&gt;This is how AI learns, it could also be instrumental in how humans function as well, psychologists often mention that every human being should have a reflective/mediation time in order to reason what went wrong and how to fix it. Gradient descent is somehow a link between artificial intelligence and the human race.&lt;/p&gt;

</description>
      <category>machinelearning</category>
      <category>deeplearning</category>
      <category>python</category>
      <category>ai</category>
    </item>
  </channel>
</rss>
