DEV Community: Adrian Brown

Open-source collaboration

Adrian Brown — Thu, 15 Jun 2023 04:58:44 +0000

My name is Adrian and I'm an open-source maker, just created Apollo an open-source LLM aggregation library for python.

I'm posting because what better way to share the news! We're currently planning a hackathon, if you're interested feel free to sign up for the waitlist

I'm looking to add support for a full API to manage in-app integrations, user subscription management, preferences and etc... And looking for feedback from people who had experience with some of those topics previously.

BTW, we are looking for new contributors if you'll find it interesting :)

How we reduced our review time by 90%

Adrian Brown — Thu, 11 May 2023 21:30:47 +0000

At Apollo, we use a no-code automation platform to allow users to manage their integrations & tasks. There are many debates over whether automating content moderation is a one-size-fits-all approach, it's not. AI powered content moderation has become table stakes in protecting and enhancing user experience at scale, but for us, visibility into other areas of enterprise/personal decision making led us to use tools like automation rules & workflow platforms.

TLDR;

One case study for content moderation from YikYak, we migrated from Retool.

The bigger, the slower

With all the advantages, there are a few drawbacks to using retool and a standalone automation system for task monitoring/completion. We noticed a particular drawback when scaling the number of users and amount of activity with each one: The time it takes to detect bad actors accurately and then investigate. So a typical task for a investigation would run anywhere between 3 minutes to 4 days. And that's for each time an automated flag was created or a user report was sent.

More than that, truly having accurate data was a nightmare. Standalone automation systems (environments with one AI model) would score correctly only a fraction of the time.

This amount of time spent investigating and analyzing to make an informed decision reduced the moderator efficiency and wasted collectively so much talented people's time. Being a social app dedicated to curating safer community, this was unacceptable.

Debugging the slowest tasks

Inspecting a typical 4 day case/support ticket, it was clear that two specific steps took almost 70-80% of the overall time:

Gathering metadata from existing platform policy, account information and community guidelines: 4 Hours
Connecting historical data to recent events: ~1 day

Migrating from Standalone Automation Systems to Integrated Automation System

Retool coupled with our standalone automation system provided a fast, efficient infrastructure as a service toolkit, and from some of the benchmarks, there was a massive improvement in active response time when migrated off of a independent automation system to an integrated automation system like Apollo.

Moving from Retool + Standalone automation took around 2 minutes, the migration to integrated automation (Apollo) was effortless: Just adding metadata about your internal databases and a click of a button, that's all. The aggregation and integration for each database/api resource were efficiently synced, in... wait for it... just milliseconds! And that's without any code at all! After integrated automation tools like Apollo increases the accuracy of the scored content, it takes less than 40 seconds to deploy and integrate multiple AI models under one API.

Reducing ~3 minutes to 13 seconds from the standalone's system time for every investigation and accuracy for moderators is a HUGE win.

Summary

Reducing our active response times from 3+ minutes to around 13 seconds significantly impacts the user experience of our community. It also reduces the overhead incurred by regulatory violations or non-compliance, which can run a bill up to $400k a month! You can checkout our beta on our Github repository.

What in the Python?

Adrian Brown — Fri, 05 May 2023 00:24:00 +0000

TLDR;

This tutorial explains how to use the Apollo SDK Python package to connect to an external database, transform data using Pandas, and perform Natural Language Processing on the data. With just a few lines of code, this powerful SDK provides a flexible interface for working with data and integrating with other services.

Prerequisites:

Have a supabase instance spun up with data
Create an API token with Apollo API
A copy of your database connection string in URI format

Quickstart

Lets begin!

Setting Up a Connection to Supabase

First, let's install the Apollo SDK Python package using pip:

pip install apollo-sdk

Once the package is installed, we can use it to set up a connection to Supabase, a popular database management tool. Here's an example code snippet that shows how to connect to Supabase using the Apollo SDK:

# import the package
from apollo.client import Apollo

# sync data from your database instance (we support supabase at the current moment or postgresql via uri format)
Apollo.connect("postgres://username:password@hostname:port/database_name")

# If you want to test out operation on your external connection
Apollo.fetch_tables()

# Query the table in desc or asc order
Apollo.query("desc", "table", "column")

This code imports the Apollo module from the apollo.client package and uses the Apollo.connect() method to establish a connection to a Supabase database instance. Once the connection is established, we can use the Apollo.query() method to run queries against the database and fetch data.

Transforming Data Using Pandas

Once we have fetched data from the external database, we can use Pandas to transform the data into a table. Here's an example code snippet that shows how to do this:

import pandas as pd

# the data returned by the Apollo.query() method
data = [...] # replace with the actual data

# convert the data into a Pandas DataFrame
df = pd.DataFrame(data)

# return the first 5 rows of the DataFrame
output = df.head()

# print the output to the console
print(output)

ex. output of dataframe for us using reddit data

This code uses the Pandas package to transform the data returned by the Apollo.query() method into a table. We first import the pandas module and create a Pandas DataFrame from the data using the pd.DataFrame() method. We then use the head() method to return the first 5 rows of the DataFrame.

Sending Data to Apollo's AI model

Finally, we can use the Apollo SDK to send the data to a content API for further processing against an AI model. Here's an example code snippet that shows how to do this:

# import the necessary modules
from apollo.client import Apollo

# set up the connection to the content API
Apollo.use("apollo", token="YOUR_API_TOKEN_HERE")

# select the 'text' column from the DataFrame
texts = df['text']

# loop through each text in the 'text' column and send it to the content API
for text in texts:
    # send the text to the content API to detect any threats
    result = Apollo.detectText(text, "contains", "Threats")

    # print the result to the console
    print(result)

This code sets up the connection to a content API using the Apollo.use() method and provides the necessary API token. We then select the text column from the Pandas DataFrame and loop through each text in the column. We use the Apollo.detectText() method to send each text to the content API and detect any potential threats.

ex. output

    status: 200,
        data: {
            actionsTriggered: [
            {
                id: 'ks93-as3',
                name: 'Mute User'
            },
            {
                id: 'f2-xa03d',
                name: 'Escalate Content'
            }
        ]
    }
}

Conclusion

This tutorial explains how to use the Apollo SDK Python package to connect to an external database, transform data using Pandas, and utilize an AI model for NLP. With just a few lines of code, this powerful SDK provides a flexible interface for working with data and integrating with other services.

Useful links:

Docs
Collaborate with us on Github
Join our discord
Checkout the website
Our newsletter

Using APIs with AI

Adrian Brown — Thu, 27 Apr 2023 23:18:15 +0000

TLDR;

What is Apollo-SDK?

It's a cli toolkit for developers to build automation pipelines on their own. You can aggregate multiple LLMs into one place using one single api and sync data across different sources. This allows you to centralize the deployment of multiple models into one channel and build from there using one api. You can use your model, our model or a combination of both.

Apollo user interface (UI) is designed for creating decision making workflows. With Apollo platform, you can perform no-code AI tasks using the best engines in the market. Apollo SDK is looking for opensource contributors. The UI is coming soon!

What can I do with it?

You should checkout our feature list here. We offer models for lots of providers using Image, Speech, Video and Text.

Quickstart

Lets install the SDK first...

pip install apollo-sdk

Let's setup your first Integration!

It will pull from your local database (and keep it in sync).

# import the package
from apollo.client import Apollo

# sync data from your database instance
# (we support supabase at the current moment or postgresql via uri format)
Apollo.connect("postgres://username:password@hostname:port/database_name")

# If you want to test out operation on your external connection
Apollo.fetch_tables()
Apollo.query("desc", "table", "column")

...and create a workflow with a simple command:

# import the package
from apollo.client import Apollo

# Use our custom model to test building decisions
Apollo.use("Apollo")

# We support video, speech, image and text. Try text!
Apollo.detectText("Phrase1", "contains", "Threats")

In practice, you probably want to use our user interface (UI) so you dont have to write code. If so, ping us at adrian@apolloapi.io!

🔌 Integrations

We pre-built integrations with providers like:

Supabase
Postgresql
Firebase

...We have a huge need for developing integrations with Mongodb, Mysql and more.

To get involved checkout some of our issues here!

🧪 Clients

Apollo has built in support for custom API calls via rest using make_http_requests & make_http_request methods.

We are building the following:

Graphql
gRPC
XML formats

How To Uninstall Core Audio Driver On Mac

Adrian Brown — Thu, 27 Apr 2023 23:06:39 +0000

TLDR;

In this guide, we will show you the steps to uninstall the Core Audio Driver (MSTeamsAudioDevice.driver) from your Mac. Ever since work from home became the new normal, users have flocked over to numerous video conferencing apps. Many users have been inclined towards Teams due to its handy integration with other Microsoft services.

Core Audio Driver MSTeamsAudioDevice.driver However, as of late, numerous Mac users have voiced their concern that even after uninstalling Microsoft Teams, its core audio driver file MSTeamsAudioDevice.driver is still present on their system. Not only that, it continues to run in the background and ends up hogging valuable system resources as well.

How to Uninstall Core Audio Driver MSTeamsAudioDevice.driver on Mac

Launch Finder and press Shift+Command+. to view hidden files

Then delete the file from the following directory ```

/Library/Audio/Plug-Ins/HAL/MSTeamsAudioDevice.driver

3. After that, delete the file from the following location as well:

/Library/Audio/Plug-Ins/HAL/MSTeamsAudioDevice.driver/Contents/MacOS/MSTeamsAudioDevice

4. Now open Activity Monitor, select Core Audio Driver, and hit Quit for one final time.Core Audio Driver MSTeamsAudioDevice.driver


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1owzn5q85t8t5xd7ahxl.png)


That’s it. These were the steps to uninstall the Core Audio Driver (MSTeamsAudioDevice.driver) from your Mac. 


#### Checkout Apollo API
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3axczs55snbu6rg4jboo.png)
We're building a aggregation tool for top AI providers and are looking for contributors! [Star](https://github.com/apolloapi/apolloapi/) our github and connect with us on [discord](https://discord.gg/ZUH7f7AzUY)

LLMs using a single API

Adrian Brown — Thu, 27 Apr 2023 05:54:34 +0000

Use your model, Apollo's model or a combination of both.

Today we're talking about aggregating different AI models under one API. I'm an open-source maker and have worked on various projects but none as fun as this!

Apollo allows you to integrate with any existing provider and sync data across multiple resources - the user interface (UI) is designed for creating decision making workflows. With the Apollo platform, you can perform no-code AI tasks using the best engines in the market. Apollo SDK is a cli toolkit for developers to build automation pipelines on their own.

...Lets take a look at the SDK!

TLDR;

Skip to the quickstart to start playing around with the SDK!

Show me an example

In your code you can write:

Apollo.connect('postgres://username:password@hostnam...', ...) # Starts syncing content forever!

Apollo.use('OpenAI', "moderation", ...) # Connect to existing providers!

Apollo.rule('Phrase1', '>=', '0.8') # Create custom rules!

Apollo.use('Apollo', "violence", ...) # Connect with our internal models!

# Detect bad actors at scale!
Apollo.detectImage('Image1', 'contains', 'VERY_LIKELY') # Image Analysis/OCR
Apollo.detectSpeech('Audio1', 'contains', 'UNLIKELY') # Audio Processing
Apollo.detectVideo('Video1', 'contains', 'POSSIBLE') # Video Analysis
Apollo.detectText('Phrase1', 'contains', 'UNKNOWN') # Text Analysis

Apollo then takes care of:

Automated detection against image, video, audio or text
Connecting with any external resource
Making sure your integration is robust, so you never again have to worry about stuck/stale data or false-positives

Cool, what can I build with it?

Teams in companies use Apollo to build native in-app connections related to active response, content moderation, fraud detection, etc.
Some automate their personal lives with Apollo by integrating against discord communities, or do other things like intelligent search. Checkout our Features!
Apollo can help you quickly build for hobby projects

Conclusion

Thanks for making it this far! If you want to join or find interest feel free to star our repo or join the discord: https://github.com/apolloapi/apolloapi

Ansible Inventory Plugin

Adrian Brown — Tue, 15 Nov 2022 15:13:48 +0000

Intro

This repo is a smaller part to executing generic ansible commands inside an ansible execution environment using ansible-runner to consume inventory from an external source other than an inventory file.

Here is the repo that is used inside an execution env for those who are curious: https://github.com/cloudguruab/edpm_plugin

If you have questions feel free to open an issue.

Requirements & Dependencies

I have specified all the requirements for this plugin here

Ansible

pip install ansible

AWS CLI

pip install boto3
pip install awscli

Setup

Be sure the desired collection is installed if you're pulling from pre-built module

ansible-galaxy collection list # list installed plugins
ansible-galaxy collection install amazon.aws

After running ansible-galaxy collection list you should see something like this:

Collection        Version
----------------- -------
amazon.aws        5.1.0
community.general 5.2.0
openstack.cloud   1.10.0

Once this is done you need to navigate to your aws console and obtain your aws_secret_key, and aws_public_key:

#setting up auth 
bash-5.1$ aws configure
AWS Access Key ID [None]: key
AWS Secret Access Key [None]: key
Default region name [None]: us-east-1
Default output format [None]: json

Updating your project

We need to update our inventory source with our private_ip to connect with our instance. Our ansible config file will also map to our inventory source and the plugin we are using.

aws_ec2.yml:

plugin: aws_ec2
regions:
  - us-east-1
keyed_groups:
  - key: tags.Tagname
filters:
  instance-state-name: running
compose:
  ansible_host: private_ip

ansible.cfg:

[defaults]
enable_plugins=aws_ec2
inventory=aws_ec2.yml

Testing

Lets look for our host:

ansible-inventory -i aws_ec2.yml --list #look for host
ansible-inventory -i aws_ec2.yml --graph # graph view

Resources

What is Section 230?

Adrian Brown — Sat, 14 May 2022 13:08:33 +0000

What to remember about section 230 of the Communications Decency Act of 1996: it impacts everyone. Section 230s objective is to promote the development of the world wide web by providing immunity to web providers and users alike. This immunity granted by section 230 covers granting online access to information created by third parties and acting in good faith to edit or restrict access to objectionable material, even if the material would be considered protected constitutionally. Its impact on big tech giants like Facebook, Twitter, and Google shows us how section 230 has shaped today’s social communities and how we begin to look at moderating content.

Building an API

Adrian Brown — Wed, 16 Feb 2022 22:28:13 +0000

🐴 Why

This tutorial should serve as an example of using supabase api to connect to your database instance and build a service to periodically cache and serve consumer credit data on client request. This project covers redis as a caching mechanism, supabase to support our postgres instance, and fastapi for our framework.

Before we start

Setup the following, and use the code found at the last link to follow along with.

☂️ Setting up your environment

Setup your virtual environment:

python3 -m venv env

Activating your environment

source env/bin/activate

In the root directory run the following:

pip install -r requirements.txt

🤖 Starting Redis in development environment

To begin working with redis, run the following command, after completion open a new terminal window.

redis-server

👾 Activating your development server

To start your local server run the following command

uvicorn main:app --reload

On success of the commad you should see;

INFO:     Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
INFO:     Started reloader process [13385] using watchgod
INFO:     Started server process [13387]
2022-02-11 19:32:12,509:INFO - Started server process [13387]
INFO:     Waiting for application startup.
2022-02-11 19:32:12,509:INFO - Waiting for application startup.
2022-02-11 19:32:12,510:INFO -  02/11/2022 07:32:12 PM | CONNECT_BEGIN: Attempting to connect to Redis server...
2022-02-11 19:32:12,511:INFO -  02/11/2022 07:32:12 PM | CONNECT_SUCCESS: Redis client is connected to server.
INFO:     Application startup complete.
2022-02-11 19:32:12,511:INFO - Application startup complete.

🎾 Endpoints

Introduction to your application.

http "http://127.0.0.1:8000/"

HTTP/1.1 200 OK
content-length: 102
content-type: application/json
date: Wed, 16 Feb 2022 22:01:14 GMT
server: uvicorn

{
    "👋 Hello": "Please refer to the readme documentation for more or visit http://localhost:8000/docs"
}

Working with your redis cache, the following call will pull data
from your supabase database, and cache it.

The x-fastapi-cache header field indicates that this response was found in the Redis cache (a.k.a. a Hit).

The only other possible value for this field is Miss. The expires field and max-age value in the cache-control field indicate that this response will be considered fresh for 604321 seconds(1 week). This is expected since it was specified in the @cache decorator.

The etag field is an identifier that is created by converting the response data to a string and applying a hash function. If a request containing the if-none-match header is received, any etag value(s) included in the request will be used to determine if the data requested is the same as the data stored in the cache. If they are the same, a 304 NOT MODIFIED response will be sent. If they are not the same, the cached data will be sent with a 200 OK response.

# Command
http "http://127.0.0.1:8000/cachedResults"

# Response Headers
HTTP/1.1 200 OK
cache-control: max-age=604321
content-length: 894
content-type: application/json
date: Wed, 16 Feb 2022 21:53:56 GMT
etag: W/-9174636245072902018
expires: Wed, 23 Feb 2022 21:45:57 GMT
server: uvicorn
x-supafast-cache: Hit

Automation and Multi-processing w/ Python

Adrian Brown — Mon, 15 Nov 2021 20:05:01 +0000

TL;DR

When it comes to finding ways to run a task python offers us tools that allow us to automate those task, whether it be read/writes or api calls there's lots of usecases where this might be necessary.

Lets start with understanding the what. So, what's a subprocess in python?

A subprocess in Python is a task that a python script delegates to the Operative system (OS) - Daniel Diaz, GeekFlare

In short, a subprocess allows us to achieve the unachievable by giving us a new level of cohesiveness with the operating system.

Introduction

By the end of this tutorial, you will

Understand the concept of subprocess
Have learned the basics of the Python multi-process/time library
Found meaningful examples/usecases where you can implement the subprocess/multiprocessing library

Core Concepts

In this tutorial I am assuming you have some basic foundations in python programming, and data structures

The nice part about the subprocess module is it falls under the core utilities offered to us as python developers. This means there is no need to install this module as a dependency using PIP or Pipenv, it's there for us already!

#calling the subprocess module into our environment
import subprocess

Quick use cases,

Program checker
Setting up a virtual environments
Run other programming languages
Open external programs

Our Application

In this tutorial we will be looking at using the Process.start() module to run background computations for excel data.

Tutorial

So after learning at a high level what the subprocess package does, and what it looks like at the import level you're ready to move towards this part of the tutorial.

I wanted to give two quick distinctions we will be using the Process.start() api from the multiprocessing library offered by Python.

So, why cover Subprocess's?

I like to explain things backwards. For me this helps by understanding the full context of my environment in which I am learning from when what I am learning from is more reading intensive rather than visually intensive.

Recap: We know what subprocess's are now think of multiprocessing as the stepchild/(really parent) to what a subprocess will allow you to do. Instead of running that process in the background we can now bring it "foreground" by running multiple process's single handedly.

Task

Here we are going to take our data set and using some basic arithmetic sort and sum all the entries in our list of data. Once
this is complete we will set a task for this to occur every 60 seconds to simulate a batched process using code. To draw more from real world scenarios this data set could be healthcare/financial data for a company, and your jobs has been to extract, transform this data into a list. So, after doing so the company will need a summary for all the data in this list to make biweekly decisions.

Our data set

#lets assume this data is updated every half minute
data = [3, 4, 7, 10, 2, 32, 15, 8]

Code

from multiprocessing import Process
from file_location import data

def process_manager(data):
    while True:
        results = sum(data)
        time.sleep(60)
        return results

if __name__ == '__main__':
    p = Process(target=process_manager(), args=(data))
    p.start()

To explain, we import our multiprocessing and near real time data object into our code environment. We define a function to govern our process that we want to run every minute.

Here you can see how Process takes two arguments, the target function and arguments. These will work together to fork this process in our developer environment and run in sync with our application.

NOTE: Python has been known to lack/struggle with multiprocessing and concurrency compared to other tools/packages offered by other languages like rust, so be sure to check the trade offs when making time-cost decisions. Feel free to copy/paste and try this in your own IDE, if you run into issues spend some time debugging and getting your code to run. This should be a safe exercise to practice your debugging skills with as you will gain experience/understanding of more PYTHON!

Link to multiprocess

Database Management Systems 5

Adrian Brown — Mon, 08 Nov 2021 00:23:46 +0000

TL;DR

Following the last post on OLTP vs OLAP, Indexing, and Hashing we will now cover something outside the bounds of a database management system and that follows OLAP from our last post.

In this blog post we will start by covering data warehouses and what they are. The overall objective for a data warehouse and its subset of properties as well.

In short, you will wrap up this series by learning the importance of data in our database environment and what will serve as structured vs unstructured data.

Introduction

By the end of this tutorial, you will understand

Data Warehouse
Data Mining
Structured vs Unstructured Data

Core Concepts

Data Warehouse

With DWH being separate from the DBMS as a whole we can understand its use for storing huge amounts of data that typically will be collected from a variety of sources.

An ordinary database system will store data and for specific purposes in relation to transactional process's where as a DWH will do the same with the intent for analytical purposes.

Also, whats cool to note is data marts since this typically will live among the variety of vocabulary words one might hear when discussing data warehouses.

In short, a data mart is a subset of the data that makes up a data warehouse.

Data Mining

“Data Mining” can be referred to as knowledge mining from data, knowledge extraction, data/pattern analysis, data archaeology, and data dredging.

This process is what makes up an ETL process or parts of it for that matter. The sole purpose of mining the data from a DWH is to carry out extraction of useful data in bulk. This data can help make decisions or look over trends within a business.

Structured vs Unstructured Data

Structured data is data with addressable information for effective analysis
Unstructured data is data with unorganized or defined schema that isn't a good fit for a typical relational database.

This concludes my series on database management systems, I hope you enjoyed reading and found some stuff to be useful or maybe things I could improve on! If you want to reach out to me feel free to do so via github

Database Management Systems 4

Adrian Brown — Sun, 07 Nov 2021 23:43:37 +0000

TL;DR

Last blog post we covered ACID properties, Concurrency, and Recovery in our database systems.

This post I have dedicated time to cover Online Transaction Processing vs Online Analytical Processing as well as Indexing and Hashing in our databases.

To note, OLTP vs OLAP looks requires some fundamental knowledge of database systems and maybe ETL (optional). Aside from that Indexing and Hashing are ways to optimize our query performance.

Introduction

By the end of this tutorial, you will understand

OLTP vs OLAP
Indexing vs Inverted Indexing
Hashing

Core Concepts

OLTP

Online transaction processing provides transaction focused applications for our data. Examples to clarify would be ATM's are OLTP based applications.

This process consist of current data, and has business task use cases. Also composed of both read and write operations when we think of our database implementations.

OLAP

Online analytical processing provides software tools/applications focused of data analytics. An example to clarify would be Netflix's movie recommendation system.

This process consist of historical data, and the data is used in planning/problem solving for business's as well as decision making. Will primarily be composed of only read and rarely any write operations.

Side Note

I really believe taking time to understand the Extract Transform Load process can help clarify further what OLTP vs OLAP will provide you in your system architecture so definitely check that out.

Indexing vs Inverted Indexing

Indexing is a way to optimize the performance of a database by minimizing the number of disk accesses required when a query is processed.

Example:

CREATE INDEX index_name
ON table_name (column1, column2, ...);

IMPORTANT: Updating a table with indexes takes more time than updating a table without (because the indexes also need an update). So, only create indexes on columns that will be frequently searched against.

Inverted Indexing is a way of mapping content such as elements, data or objects to a particular data object or document. (Google Search: Inverted Indexing)

In short, indexing is a way to highly optimize the retrieval of information and data through stored queries.

Hashing

Hashing is an efficient technique to directly search the location of desired data on the disk without using index structure.

Typically searching through all the data points in hopes of quick retrieval can bog down or inadvertently have some inefficiency when it comes to index's. Hashing solves this.

To conclude, hashing involves looking at data thats stored on blocks of data addressed via a hash function. This works as the particular address is where that memory allocation will live and is known as a data bucket.

This concludes part 4 of the series, I will post more on part 5 shortly.