Recently, I had a task of learning a new tool that I had never used or seen anything related to, and I thought: why not write about how I learned it?
This is the kind of article that teaches you to learn things from a different perspective. Not the best way, but you can reuse some of the concepts written here.
Table of Content
- 1. Prologue
- 2. The Basics
- 3. The Documentation
- 4. The Source Code
- 5. The Network Tab
- 6. The Implementation
- 7. Conclusion
1. Prologue
Recently I've started a new project (Yet Another Poor SaaS Application) with my friends using PHP and Laravel. The project doesn't really matter, but the stack does. Why is that? Because we're not rich enough to spend money on self-hosted infrastructure, nor on good paid tools... So it's important.
So, if we're going to build something that has any kind of integration, it needs to at least have a free tier to consume and not be a living hell to implement. And in thinking about that, we stumbled upon: how are we going to do the user tracking/metrics?
My goal is to show the analytics per page of the SaaS user The average developer might think: we can use Google Analytics and Google Tag Manager, right? Well... Let me tell you something that Chester said once:
I tried so hard and got so far. But in the end, it doesn't even matter - Chester Bennington
Dude, I've always been a Google UI hater. But this time it's not for my job but for me and I have no time to waste. So, I spent 4 hours trying to do... the basics of setting it up on my project and understanding how to create a simple tag? And at the end I wasted my time since nothing worked.
Like, why all Google products have an interface that you need a fucking PhD to use it? Too much information everywhere!
That's when my friend told me about PostHog, an open-source analytics platform that lets you track user behavior with self-hosted or cloud options (and beat GTM/GA4 in matter of implementation). But at the same time, I'd like to ask you to stay until the end of this article, because it took me some hours and lines of code to do this study.
2. The Basics
Ok, this is the most obvious thing, but it has to be said: when you try a new tool, you have to play with every aspect of it. I know we're developers, but the UI of a product is trying to tell you something.
2.1 Break it until you make it
This PostHog process took infinitely less time to set up and that was definitely a plus. However, the UI has too many things happening and my first idea was to try to click everything in and break the project I was on in record time.
Just think with me: if you're afraid to use a platform/product just because you're in a cloud or "production" environment, you're not going to learn anything in the end.
At the end of this step, I just deleted the messed up project and started a new one. But at this point, I know how the UI works.
DISCLAIMER: DON'T DO IT AT YOUR COMPANY'S CLOUD ACCOUNT, CREATE ONE AND EXPLODE YOUR OWN ENVIRONMENT!!
2.2 But... It's not that obvious
Ok... I have to be honest with you, sometimes breaking everything is not the answer because it's not even an question depending on the feature you're using. At PostHog they have things like "TrendsQuery", "RetentionQuery", "Web Vitals" and a really fancy UI query builder for each type of query. Really impressive! But... I couldn't use it right away.
Since I have this amazing condition called ADHD, when I see minimally complex UI's with conditional components appearing on the screen, I just freak out and start panicking. But that's no reason to give up, right?
This section is not a criticism of the product, but me getting it out of my chest. Data science is an area of infinite conditional rules, where the more information you have, the more accurate it will be, and I'm taking my first steps to learn it.
3. The Documentation
Sure, the documentation! Why haven't I played with this before? I mean, this is the place for developers, right? Right! But we're developers who are going to use the product, so the first step is a must.
I don't usually go through tutorials, even though I know everyone should. I just don't like it, to me good API references must tells you everything, but what would be everything in my opinion?
- "Indexable" Title: under a table of contents, this helps a lot.
- Refined Description: tells everything that you should know in a first moment to use.
- Implementation Examples: using languages like PHP, JS, Go, Rust and mainly cURL
- Expected Responses: which payload with a "real data" will be served based on the response status (200, 401, 422, 500).
- Implementations Reference: Where you can find the actual code to read and understand how things works at the client/server side.
- Required Scopes: Usually you should have scopes for the endpoint.
In most cases you'll have many elements like these which I listed above, but it depends on the API you're using. In my case with this PostHog specific implementation, the challenge is that the /query/create endpoint has a polymorphic request and response and so there's no response payload there.
So that's when we start digging into the real code because we need to understand how these endpoints works with the actual implementation.
To avoid I could just write down a HogQL Query, but learning it (for me) would take way more time.
4. The Source Code
We know what we want at this point, right? We want to understand what payload is being sent for each type of request. In my case, I want to understand how Retention and Trends Query works on PostHog. And since the product is open source, I can read it directly from GitHub!
The front-end is written in TypeScript and the back-end is written in Python. Since I'm a back-end developer, you'd think I'd pick the Python implementation to dig into, right? Well, actually, I'm a Python hater for many reasons. Also, TypeScript gives me a type safety to use as a reference for code search.
In the TypeScript source I found the implementation of everything I needed on the client side, but after seeing it I went straight down the rabbit hole and you will understand why.
Take a look at this snippet (reference):
export interface TrendsQuery extends InsightsQueryBase<TrendsQueryResponse> {
kind: NodeKind.TrendsQuery
/**
* Granularity of the response. Can be one of `hour`, `day`, `week` or `month`
*
* @default day
*/
interval?: IntervalType
/** Events and actions to include */
series: AnyEntityNode[]
/** Properties specific to the trends insight */
trendsFilter?: TrendsFilter
/** Breakdown of the events and actions */
breakdownFilter?: BreakdownFilter
}
Here we can assume a few things:
- There's an inherited InsightsQueryBase, so other queries have the same base.
- The TrendsQueryResponse is available and must follow a pattern, so we can check it later.
- The query "concerns" (which you can think of as a filter) isn't as explicit and will require more research.
First, we need to know what is in the basis and look for the InsightsQuery Concerns:
/** Base class for insight query nodes. Should not be used directly. */
export interface InsightsQueryBase<R extends AnalyticsQueryResponseBase<any>> extends Node<R> {
/** Date range for the query */
dateRange?: DateRange
/**
* Exclude internal and test users by applying the respective filters
*
* @default false
*/
filterTestAccounts?: boolean
/**
* Property filters for all series
*
* @default []
*/
properties?: AnyPropertyFilter[] | PropertyGroupFilter
/**
* Groups aggregation
*/
aggregation_group_type_index?: integer | null
/** Sampling rate */
samplingFactor?: number | null
/** Colors used in the insight's visualization */
dataColorTheme?: number | null
/** Modifiers used when performing the query */
modifiers?: HogQLQueryModifiers
}
And that's the moment you understand that you are digging in a rabbit hole because these concerns seem to be HUGE objects that grant different behaviors within your API.
After a few hours into it, I started to realize how these concerns affect the query, things like:
- Breakdown: More like a a GroupBy clause
- Properties: Polymorphic Filtering which totally depends on the query
- Series: The type of the data being retrieved (e.g. "views count")
- Interval: a from/to date filtering.
This is just the tip of the iceberg. There are plenty of other concerns surrounding the many requests on PostHog. But here are the questions:
- If there is an InsightsQueryBase, wouldn't these concerns be shared with other types of queries of that category?
- Is there a way for other types of base queries to have the same concerns?
- Should I understand how each type of **concern affects the final query?
I mean, we have to understand. Maybe not that deep, but a general knowledge. But how do we get proof if the documentation doesn't give us examples?
Well, that's where I explain my favorite thing about a browser: the Network tab!
5. The Network Tab
I just learned all about how to "explore" web applications directly from the browser by doing A-B testing with endpoints. You might think:
"Depending on the service, you can't build a self-user bot because it may violate the terms of service" - Normal User
and I would answer that I couldn't care less because I'm doing it for educational purposes. If I'm doing it, it's to learn how things work, to build my own tools and use their product.
Just a random fact: I've been inspecting elements/requests since 2014. My first "real" GitHub project was basically inspecting all requests from a browser game called "TribalWars" and building a CLI app/bot to play the game through my terminal with the ultimate goal of not playing it (building a self-bot account).
Back to the PostHog implementation, when I started inspecting, I found different types of payloads for each type of query. With that, I could make more assumptions and use it for the next step.
{
"client_query_id": "54f620e7-fdfe-4749-af41-caed0a3fe671",
"query": {
"breakdownFilter": {
"breakdown": "$geoip_country_code",
"breakdown_type": "event"
},
"conversionGoal": null,
"dateRange": {
"date_from": "-7d",
"date_to": null
},
"filterTestAccounts": false,
"kind": "TrendsQuery",
"properties": [],
"series": [
{
"event": "$pageview",
"kind": "EventsNode",
"math": "dau",
"name": "Pageview"
}
],
"trendsFilter": {
"display": "WorldMap"
}
},
"refresh": "async"
}
and the second query just in case :p
{
"client_query_id": "25d4b82f-5bbe-4665-89a7-7aedcc7b103e",
"query": {
"dateRange": {
"date_from": "-7d",
"date_to": null
},
"filterTestAccounts": false,
"kind": "RetentionQuery",
"properties": [],
"retentionFilter": {
"period": "Week",
"retentionReference": "total",
"retentionType": "retention_first_time",
"totalIntervals": 8
}
},
"refresh": "async"
}
I would include the results query, but that would be too much code and the point of this section is already made. Now it's time to finish this entire flow.
6. The Implementation
That's when you start thinking, "Does the tooling available meet my needs? If the SDK doesn't give me a fully typed API, I'd rather build my own stuff. I mean, I like to do things from TOTALLY SCRATCH! Yeah, the whole reinventing the wheel thing, you know?
The reason is simple: I'm a developer who learns how things work by rebuilding them and observing how they behave. And with that, I decided to build a PostHog Dedicated Query Builder with PHP to learn how things work on my end.
So far this is what I've got:
- We have multiple query types;
- These queries can inherit properties/concerns but not necessarily use all of them;
- Queries share filters/concerns;
So how do we organize it? And after hours of digging in the Source and Network tabs, this was my answer:
------
- QueryBuilder Feature at my own PostHogSDK
------
Query
├── Builders
│ ├── AbstractQueryBuilder.php
│ └── Insights
│ ├── AbstractInsightsQueryBuilder.php
│ ├── RetentionQueryBuilder.php
│ └── TrendsQueryBuilder.php
├── Filters
│ ├── Breakdown
│ │ ├── BreakdownFilter.php
│ │ ├── BreakdownTypeEnum.php
│ │ └── Concerns
│ │ └── InteractsWithBreakdown.php
│ ├── Compare
│ │ ├── CompareFilter.php
│ │ └── Concerns
│ │ └── InteractsWithCompare.php
│ ├── ConversionGoal
│ │ ├── ActionConversionGoal.php
│ │ ├── Concern
│ │ │ └── InteractsWithConversionGoal.php
│ │ ├── Contracts
│ │ │ └── ConversionGoalContract.php
│ │ └── CustomEventConversionGoal.php
│ ├── DateRange
│ │ ├── Concerns
│ │ │ └── InteractsWithDateRange.php
│ │ └── DateRangeFilter.php
│ ├── Interval
│ │ ├── Concerns
│ │ │ └── InteractsWithInterval.php
│ │ └── QueryIntervalEnum.php
│ ├── Node
│ │ ├── ActionNode.php
│ │ ├── Concerns
│ │ │ ├── InteractsWithMath.php
│ │ │ └── InteractsWithSeries.php
│ │ ├── Contracts
│ │ │ └── EntityNodeContract.php
│ │ ├── EntityNodeKindEnum.php
│ │ ├── EntityNode.php
│ │ ├── EventsNode.php
│ │ └── MathEnum.php
│ ├── Properties
│ │ ├── BaseProperty.php
│ │ ├── Concerns
│ │ │ └── InteractsWithProperties.php
│ │ ├── Contracts
│ │ │ └── PropertyFilterContract.php
│ │ ├── Filters
│ │ │ ├── EventPropertyFilter.php
│ │ │ └── SessionPropertyFilter.php
│ │ ├── PropertyFilterKind.php
│ │ └── PropertyOperator.php
│ └── Retention
│ ├── Concerns
│ │ └── InteractsWithRetention.php
│ ├── Enums
│ │ ├── RetentionPeriodEnum.php
│ │ ├── RetentionReferenceEnum.php
│ │ └── RetentionTypeEnum.php
│ └── RetentionFilter.php
├── QueryBuilderInterface.php
├── QueryFactory.php
└── QueryKindEnum.php
Every filter has your own dedicated spot, where it can be inherited to any type of builder using PHP Traits:
trait InteractsWithDateRange
{
protected ?DateRangeFilter $dateRange = null;
public function setDateRange(DateRangeFilter $dateRange): self
{
$this->dateRange = $dateRange;
return $this;
} public function getDateRange(): ?DateRangeFilter
{
return $this->dateRange;
}
private function buildDateRange(array &$payload): void
{
if ($this->dateRange !== null) {
$payload['dateRange'] = [
'date_from' => $this->dateRange->from,
'date_to' => $this->dateRange->to,
];
}
}
}
So the only requirement of a query builder is to have the "build()" method provided by my QueryBuilderContract, so I can ensure that at least the typing will be satisfied at the end. Each use in the class below is a different common QueryBuilder concern.
class TrendsQueryBuilder extends AbstractInsightsQueryBuilder
{
use InteractsWithInterval,
InteractsWithDateRange,
InteractsWithBreakdown,
InteractsWithCompare,
InteractsWithConversionGoal,
InteractsWithProperties,
InteractsWithSeries;
protected QueryKindEnum $queryType = QueryKindEnum::TrendsQuery;
public static function make(): self
{
return new self();
}
public function build(): array
{
return $this->jsonSerialize();
}
public function jsonSerialize(): array
{
$payload = parent::jsonSerialize();
$payload['kind'] = $this->queryType->value;
$this->buildDateRange($payload);
$this->buildInterval($payload);
$this->buildSeries($payload);
$this->buildCompare($payload);
$this->buildProperties($payload);
$this->buildConversionGoal($payload);
$this->buildBreakdown($payload);
return $payload;
}
}
While we are coding, we have to pay double attention to the minimal details around the implementation that is being made. And frankly, when I build this kind of tool, I mostly think that someone will use it, which raises the attention bar even higher.
The end result of this study is a fully functional Trends and Retention QueryBuilder for the PostHog API's:
test('can build a retention query', function () {
$expected = [
"kind" => "RetentionQuery",
"dateRange" => [
"date_from" => "-7d",
"date_to" => null
],
"filterTestAccounts" => false,
"retentionFilter" => [
"retentionType" => "retention_first_time",
"retentionReference" => "total",
"totalIntervals" => 8,
"period" => "Week"
]
];
$queryBuilder = RetentionQueryBuilder::make()
->setDateRange(DateRangeFilter::from('-7d'))
->setRetention(RetentionFilter::weekly()
->setRetentionType(RetentionTypeEnum::FIRST_TIME)
->setRetentionReference(RetentionReferenceEnum::Total)
);
expect($queryBuilder)->toBeInstanceOf(RetentionQueryBuilder::class)
->and($queryBuilder->build())->toMatchArray($expected);
});
test('can build a trends-query', function () {
$expected = [
'kind' => 'TrendsQuery',
'filterTestAccounts' => false,
'dateRange' => [
'date_from' => '-7d',
'date_to' => null,
],
'interval' => 'day',
'series' => [
[
'event' => '$pageview',
'kind' => 'EventsNode',
'math' => 'dau',
'name' => 'Pageview',
],
],
'compareFilter' => [
'compare' => true,
],
'properties' => [
[ 'type' => 'event',
'key' => '$host',
'operator' => 'exact',
'value' => 'api-main-ofjibb.laravel.cloud',
],
],
];
$actual = TrendsQueryBuilder::make()
->setDateRange(DateRangeFilter::from('-7d'))
->setInterval(QueryIntervalEnum::Day)
->addCompareFilter()
->addSeries(EventsNode::make('$pageview', MathEnum::DAU, 'Pageview'))
->addProperty(EventPropertyFilter::make('$host', PropertyOperator::Exact, 'api-main-ofjibb.laravel.cloud'));
expect($actual)->toBeInstanceOf(TrendsQueryBuilder::class)
->and($actual->build())->toMatchArray($expected);
});
You can check the Pull Request for this implementation clicking here. Feel free to contribute on this project too!
7. Conclusion
This is just a study in the PostHog API that I enjoyed a lot and decided to write about. Building things from scratch and doing proper research is something that can help you learn things on another level.
Now I can tell you that I can play with the PostHog platform in a really smooth way, without the worries mentioned in the first section of this article since I just breaked it once, divided concerns and studied each of them separately.
This approach—breaking things, diving into the docs, exploring the network tab, and rebuilding from scratch—can be applied to any technology. Whether it's PostHog, a new API, or a framework, understanding the internals makes you a better developer.
That's it! Hope you enjoyed and don't forget to drink water!
Top comments (13)
Break things, move fast, right?
That's the spirit.
Good article, your method of studying gave me great ideas to improve my study sessions
Lots of POCs to break everything and learn how to do it... Been trying to do this since the start of this year and I'm learning a lot. Thanks for sharing your thoughts, always good to learn from you!
Awesome
This post is amazing!
Niceeeeee article :D
Hey, my cousin, long time no read anything you write. Excellent article, like always, go ahead with this incredible articles. See u later <3
Really great tips! Awesome post! 🔥
Amazing content. I really enjoyed your approach to the topic and also learning about Posthug and its possibilities.
Wow, best way to learn is break and reconstruct..