DEV Community: Retiago Drago

Understanding the Process of Running Metabase on Docker for the First Time 📊🐋

Retiago Drago — Sat, 18 Nov 2023 03:32:18 +0000

Outlines

Open Source Quick Start 🌟

Setting Up Metabase 🛠️

Setting Up An Admin Account 🛠️

Gathering Database Info 🛠️

Connect to Database 🛠️

Usage Data Preferences 🛠️

Staying in Touch 🛠️

Metabase Home Page 💻

Asking Metabase Question 🙋‍♂️

Filtering Data 🔍

Previewing Results 👓

Summarizing Data 📊

Grouping Results 🤝

Changing the Visualization 🌆

References 📚

Connect with Me 👨‍💻

Open Source Quick Start 🌟

Welcome to your first-time setup guide for running the Open Source version of Metabase using Docker. This guide will help you get Metabase up and running locally on your system.

Note: Assuming you have Docker installed and running. Please click on any image to zoom in.

Here, I'm using Docker Desktop WSL 2 backend on Windows with distro integration.

Get the latest Docker image:
```
docker pull metabase/metabase:latest
```
🔎 command explanation
- docker pull: This is the command used to download ("pull") a Docker image from a registry.
- metabase/metabase: This specifies the name of the Docker image. The part before the slash (metabase) typically denotes the Docker Hub username or organization, while the part after the slash (metabase) is the repository name of the image.
- :latest: This is a tag that specifies which version of the image you want to pull. The latest tag is often used to denote the most recent stable version of an image.
So, in short, this command pulls the most recent stable version of the Metabase image from the official Metabase repository on Docker Hub.
Start the Metabase container anywhere and go to http://localhost:3000:
```
docker run -d -p 3000:3000 --name metabase metabase/metabase
```
🔎 command explanation
- docker run: This is the command to create and start a new container.
- -d: Run the container in detached mode, meaning it runs in the background and doesn't block the current shell.
- -p 3000:3000: Map port 3000 of the host to port 3000 in the container. This allows you to access the application running in the container via localhost:3000 on your host machine.
- --name metabase: Assign the name metabase to the container. This name can be used to reference the container in subsequent commands, instead of using the container's ID.
- metabase/metabase: The image to use to create the container. This specifies that Docker should use the metabase/metabase image from Docker Hub, and if it isn't available locally, Docker will pull it from the registry first.
In summary, the command starts a new container from the metabase/metabase image, named metabase, running in the background, with the container's port 3000 accessible via port 3000 on the host machine.
Stop and remove the Metabase container. If you need to use the Metabase container again later just run the docker stop metabase command only:
```
docker stop metabase && docker rm metabase
```
🔎 command explanation
- docker stop metabase: This command stops the running container named metabase.
- &&: This is a shell operator that allows you to execute a second command only if the first command (docker stop metabase) succeeds.
- docker rm metabase: This command removes the container named metabase freeing up resources and allowing you to start a fresh container with the same name without conflict. Containers need to be stopped before they can be removed.
Running this command will turn off the Metabase service running in the Docker container and then remove the container.
To start the Metabase again without removing them beforehand:
```
docker start metabase
```
🔎 command explanation
- docker start metabase: This command will start the existing container named metabase. This is useful when you have a container that you've stopped and want to run again without setting it up from scratch.
By using docker start, you preserve the container's state and any data that might be stored within it between stop and start operations. This is often preferred for applications that store configuration or data that you want to retain across sessions.

Setting Up Metabase 🛠️

Setting Up An Admin Account 🛠️

The initial step involves setting up an admin account. Conveniently, the account you establish when you first install Metabase automatically has admin privileges. Since we are presumably deploying for development and not production, it's acceptable if you don't remember the password for this account right away. However, if this were a production server, extreme caution would be necessary to remember this password as it grants access to add other users, connect to databases, set up email, and more. Additional admin accounts can also be created at a later stage.

For the time being, we'll create an account for personal use to navigate through Metabase. Enter your details, and when you're prepared to proceed, click the 'Next' button.

Gathering Database Info 🛠️

At this stage, you'll need to compile some details about the database you plan to integrate with Metabase. Without this information, we won't be able to establish a connection to your database. However, if you prefer to postpone this step, that's perfectly fine. For now, I'm choosing the option "I'll add my data later". Metabase provides a Sample Database that allows you to familiarize yourself with its functionalities.

When you're prepared to establish a connection, you'll require the following details:

The hostname of the server hosting your database
The port used by the database server
The database name
The username associated with the database
The password for the database

If you don't have these details readily available, the individual responsible for managing the database should be able to provide them.

Connect to Database 🛠️

For those who choose the option "I'll add my data later", skip this part. For those who have their database information ready, you're just a step away from accessing your valuable data. Simply enter your details into the provided form and hit 'Next'.

You'll be connected to your database in no time. For additional guidance on database connections, refer to the Adding and managing databases section.

Usage Data Preferences 🛠️

Before we proceed, there's a small matter to address regarding your preferences for usage data. Your decision will aid them in enhancing the Metabase experience. As stated in the box:

Metabase is committed to respecting your privacy and does not collect any information about your data or the results of your queries.
The data they collect is entirely anonymous.
You have full control over this feature and can disable data collection at any time via your admin settings.

Staying in Touch 🛠️

Congratulations! You're now fully equipped and ready to explore Metabase.

Metabase Home Page 💻

Here are the few first things you see on the Metabase home page

Home Page Content: The central part of the image shows sample insights with buttons like "Some insights about Orders," "A summary of People," "A glance at Products," and "A look at Accounts." These are likely the automatic explorations that we can save as dashboard.
Navigation Sidebar: On the left side, there is a sidebar that seems to list various sections:
- Metabase Logo: Clicking on the Metabase logo will open and close the navigation sidebar. This logo is placed in the upper left corner of the screen.
- Home Button: The icon at the very top of the sidebar, would usually take you back to the home page.
- Collections: Below the Home icon, there is a section for 'Collections' where questions, dashboards, and models are stored. There is a "Your personal collection" section likely for personal drafts and experiments.
- Bookmark Section: Bookmarked items will appear toward the top of the navigation sidebar when there are bookmarked items.
- Data Section: Further down the sidebar, we'd expect to find the Data section, which would list the data sources the application is connected to. In this image, we see "Browse data" which could be part of this section.
- Add Your Own Data: Feature to import or connect to new data sources.
Settings: At the upper right, there is a gear icon for settings shows a drop-down menu with five options referring to:
- Account settings: This page allows individual users to manage their personal account details on Metabase. Users can update their first name, last name, email, and language preference. There are also tabs for changing the password, reviewing login history, and setting notification preferences.
- Admin settings: This is the administrative panel for Metabase where system-wide settings are configured. Options include setup, general settings, updates, email configuration, Slack integration, authentication methods, map settings, localization, file uploads, public sharing options, embedding configurations, license information, and caching settings. This page also offers guidance for new admins to help them get started with setup tasks like adding a database, setting up email, inviting team members, and tips for production readiness.
- Help: The help section provides support resources for Metabase users. It includes troubleshooting guides, a documentation library for in-depth articles on setup and day-to-day usage, a discussion forum to ask questions and learn from other Metabase users, and learning resources such as tutorials, guides, and articles.
- About Metabase: This option confirms the version of Metabase that is being used, in this case, v0.47.6, along with the build date. It also serves as an acknowledgement of the Metabase trademark and the team behind the product.
- Sign out: This is the option to log out of the current user session in Metabase, ensuring that no unauthorized person can access the data or settings if the user leaves the computer.
Search Bar: This search bar is a tool within Metabase that allows users to quickly find dashboards, questions (which are specific queries or reports), collections, and possibly other entities within their Metabase instance.
New Button: This button is used to create new content within Metabase. Here's what each option generally stands for:
- Question: This allows you to create a new question, which in Metabase terminology is a query or report. You can use their graphical query builder or write SQL queries to explore your data and get answers.
- SQL Query: Specifically for creating a new question using raw SQL. This is for users who prefer to write their SQL queries manually or who need to perform more complex queries that the graphical interface may not support.
- Dashboard: This option is used to create a new dashboard. Dashboards are collections of questions that can be viewed together on a single page, allowing for easy visualization of related data and metrics.
- Collection: Collections in Metabase are like folders, which help organize questions, dashboards, and models. This option lets you create a new collection to organize your Metabase content.
- Model: A model is a curated view of a table that simplifies querying for other users. It allows you to define a set of fields from a table, rename them, add descriptions, and set up default aggregations. Here's the simple illustration I made that helps me understand the terms for the first time.
Customize Button: This allows users to set a specific dashboard as the Metabase homepage for all users.

Asking Metabase Question 🙋‍♂️

In Metabase, a question constitutes a query, its outcomes, and the presentation of these outcomes, even if it's simply in the form of a table. Each question is identified by a title, ID, and a shareable unique URL.

Questions can be manipulated by applying filters and summaries, and these modifications can be saved as separate questions. Additionally, questions can be incorporated into dashboards. However, let's not get too far ahead— for the forthcoming illustrations, we'll utilize the Sample Database included with Metabase.

To begin, click on + New in the main navigation bar and select Question. You have the option to begin crafting a new question using:

A model
An unprocessed table
A previously saved question

For this instance, select Raw data > Sample database > People to commence, and Metabase will launch the query builder.

Filtering Data 🔍

The People table is populated with fictitious information. Suppose we are curious about the following:

What is the count of individuals originating from the states of Washington, Wisconsin, West Virginia, or Wyoming?

To be more specific, we are asking: "How many entries (or rows) exist within the People table where the State column contains WA, WI, WV, or WY?"

To determine this, we'll apply a filter to the dataset focusing on the field of interest, which is State in this scenario. Our goal is to tally up the number of rows that meet our criteria.

During the Filter phase, we'll press the purple Add filters to narrow your answer button and choose State as the column to filter by. In the ensuing dropdown, tick the options for WA, WI, WV, and WY, and then finalize by clicking Add filter.

Previewing Results 👓

At any stage, you can use the triangle icon on the right to take a sneak peek at the current results. The gif presented below demonstrates that, following the application of our filter, the individuals hailing from Washington, Wisconsin, West Virginia, or Wyoming are indeed displayed. To exit the preview mode, simply click the X located at the top right corner of the preview pane.

Summarizing Data 📊

Whenever we use terms like:

"How many,"
"What's the total," or
"What's the average"

We're referring to the process of aggregating our data, essentially speaking of metrics. In this scenario, the metric is the total number (or count) of individuals after filtering by the states of Washington, Wisconsin, West Virginia, or Wyoming. In the Summarize step, we'll select the Pick the metric you want to see option and choose Count of rows.

Upon selecting Visualize, Metabase reveals that there are 186 individuals from the specified states. In other words, 186 entries from the People table passed through our filter.

Grouping Results 🤝

The total count is insightful, yet it becomes more informative if we discern the timeline of our customers' registrations, especially the monthly registration figures within those states.

To revisit the editor, we can utilize the back button, or select the Editor icon situated in the top right (symbolized by a bulleted list). We can also stay on the current page and modify our query by clicking the Filter and Summarize options.

To categorize our data by month, we'll choose Summarize. In the Summarize sidebar that appears below our metric selection, there's an assortment of columns available for grouping our data. We'll opt for Created At to sort by the registration date, which will segment our counts of individuals in the specified states by each month, aligned with the registration date (Created_at). Selecting Created At prompts Metabase to display a line chart depicting the registration timeline. The Created_At column defaults to a monthly grouping, but we can alter this by clicking on by month near Created At and choosing another timeframe from the dropdown menu.

Further refinements to our query can be made via the sidebar, or we can return to the full editor view by clicking on the Editor button. Below is the enhanced query with our new summarization step:

Changing the Visualization 🌆

Metabase offers multiple ways to display the solutions to queries. To switch the display style, simply choose from the alternatives within the Visualization sidebar that appears after you click the Visualization button located at the page's lower left corner.

Should you prefer to see the data in a tabular format, you have the option to use the small toggle situated at the bottom centre of the page. This lets you alternate between the chart view and a data table.

For additional display types, activate the blue Visualization button in the lower left corner. For instance, we could represent our query with an area chart.

Certain types of visualizations may not suitably convey the answers to some queries, which is why Metabase may exclude such options in the Other Charts section. For instance, depicting the aggregate number of individuals from specific states using a Gauge or Progress bar would not be appropriate.

If you wish to experiment with your query, such as altering the states to other values, simply click on the filter tag located beneath the page title.

To delve deeper into chart options, visit Visualizing data.

References 📚

Connect with Me 👨‍💻

If you've enjoyed this Metabase tutorial and wish to stay updated with more insights, tips, or have any queries, feel free to connect with me. Just click the button below to visit my beacons profile and get in touch. I'm always open to discussions and collaborations!

Visit My 👀 Beacons Profile

Letting Go of One Project to Pursue a Better One

Retiago Drago — Sat, 04 Nov 2023 14:37:19 +0000

Hi, M!

Hello Again, Cold World!

Back at it with "Hello Cold World!" as I'm here typing in my room, AC on (no pun intended). For those of you new here, these posts are where I get real with myself, both technically and mentally. The series name? It's a nod to that Paramore song.

Letting Go of One Project

So, I stumbled upon this project in the Data Science Community on LinkedIn named Mercury. It's this slick tool that transforms notebooks into web apps instantly. At first, it seemed promising, but it fell short as a portfolio platform, which was my main goal. It ran smoothly locally, but deploying to their cloud was a different story—debugging became a Herculean task. I even tried shifting it to Azure as a container web app, wrestling with Docker on my laptop, but no dice. After a day spent in debugging purgatory, I waved the white flag on Mercury. It's cool, just not what I need right now. My focus is locking down a data analyst role, and building an app solo just isn't in the cards—I need to invest time elsewhere. For now, I'm diving into their docs, contributing what I can to grasp it better and understand the open-source scene.

Pursue a Better One

I'm redirecting my efforts toward mastering Metabase and Amundsen for my Data Analyst path. Plus, there's a final project for my multi-platform Flutter course looming over me. Penning this post is like patting myself on the back for the coding strides I've made.

Message from M

Hit the pavement for a morning stroll now and then, aim for 10-15 minutes. It's like a shot of espresso for the spirit to kick off the day. - M

Keep Going and Keep Pushing

Retiago Drago — Mon, 30 Oct 2023 10:46:01 +0000

Hi, M!

Hello Cold World!

It's such a cold, cold world (Hello cold world)
And I can't get out
So I'll just make the best of everything I'll never have
Such a cold, cold world (Hello cold world)
And it's got me down
But I'll get right back up as long as it spins around
Hello cold world
- Hello Cold World by Paramore

"Hello Cold World" is a series where I pour out my ongoing thoughts, unfiltered. It's my hope to remain honest with myself through this endeavor. Now, why the name Hello Cold World!? From my vantage point, whenever I delve into code, I often find myself navigating through a backstage filled with negativity and relentless pursuit of perfection. While I may or may not have delved extensively into the gritty realities of coding and programming, my personal experience has nudged me to air this 'dirty laundry' about learning to code—both effectively and aesthetically—especially for those who strive for perfection in their craft. The chorus from the song, bearing the same name as this series, resonates profoundly with my current situation, not to mention the chilly reality of the world we're navigating at present. It's indeed a cold world extending its hello.

I'm Back, I Guess

For now, immersing myself in productive endeavors is the goal. For instance, creating a pull request to improve the docstring of an open-source software, all while job hunting. Yet, my inner perfectionist—let's name it P—chimes in:

Why bother with this PR? It's hardly going to earn you noteworthy accolades for your open-source contribution, let alone secure a job.

P's words echo loudly, prompting me to drown it out with some music. Despite P's nagging, I persevere, learning the ropes of crafting effective, consistent, and insightful docstrings. While it may not significantly bolster my job prospects, the journey is enriching. So, the grind continues, undeterred by the venom P spits.

Message from M

Don't forget to make your bed wherever you reside. This simple habit could bear fruit in the long-term, transcending generations. - M

Cascade Notation and Fluent Interface in Dart

Retiago Drago — Sun, 17 Sep 2023 03:42:55 +0000

Question 1

Dart documentation says:

Cascades (.., ?..) allow you to make a sequence of operations on the same object. In addition to accessing instance members, you can also call instance methods on that same object. This often saves you the step of creating a temporary variable and allows you to write more fluid code.

Consider the following code:

class Car {
  String? color;
  int speed = 0;

  void accelerate() {
    speed += 10;
  }

  void brake() {
    speed -= 10;
  }

  void repaint(String newColor) {
    color = newColor;
  }

  @override
  String toString() {
    return "Car(color: $color, speed: $speed)";
  }
}

void main() {
  Car myCar = Car()
    ..color = 'red'
    ..speed = 100
    ..accelerate()
    ..repaint('blue');

  print(myCar); // Output: Car(color: blue, speed: 110)
}

The constructor, Car(), returns a Car object. The code that follows the cascade notation operates on this object, ignoring any values that might be returned. Here, myCar is created and then immediately modified using several methods and property setters, all thanks to the cascade notation (..). This allows for a more compact and chained setting of properties and calling of methods on the myCar object.

The previous example is equivalent to this code:

class Car {
  String? color;
  int speed = 0;

  void accelerate() {
    speed += 10;
  }

  void brake() {
    speed -= 10;
  }

  void repaint(String newColor) {
    color = newColor;
  }

  @override
  String toString() {
    return "Car(color: $color, speed: $speed)";
  }
}

void main() {
  Car myCar = Car();
  myCar.color = 'red';
  myCar.speed = 100;
  myCar.accelerate();
  myCar.repaint('blue');

  print(myCar); // Output: Car(color: blue, speed: 110)
}

Here are the differences:

-  Car myCar = Car()
-    ..color = 'red'
-    ..speed = 100
-    ..accelerate()
-    ..repaint('blue');
+  Car myCar = Car();
+  myCar.color = 'red';
+  myCar.speed = 100;
+  myCar.accelerate();
+  myCar.repaint('blue');

This reminds me of Fluent Interface after fork clean-code-php repo a while ago.

A Fluent interface is an object-oriented API that aims to improve the readability of the source code by using Method chaining.

So I asked ChatGPT for direction.

Is the concept of fluent interface the same as cascade notation in Dart?

ChatGPT Reply 1

Both the concept of a fluent interface and Dart's cascade notation (..) aim to make code more readable and convenient by allowing method calls or operations to be chained together. However, they are not exactly the same thing.

Fluent Interface

In a fluent interface, each method returns an instance of the object, allowing you to chain method calls together. A fluent API is designed specifically to return this or a new instance of the same class, making it possible to chain methods.

Here's an example using a hypothetical Car class:

class Car {
  String? color;
  int speed = 0;

  Car accelerate() {
    speed += 10;
    return this;
  }

  Car brake() {
    speed -= 10;
    return this;
  }

  Car repaint(String newColor) {
    color = newColor;
    return this;
  }

  @override
  String toString() {
    return 'Car(color: $color, speed: $speed)';
  }
}

void main() {
  Car myCar = Car()
      .accelerate()
      .brake()
      .repaint('blue');
  print(myCar);  // Output: Car(color: blue, speed: 0)
}

Cascade Notation

Dart's cascade notation allows you to perform multiple operations on a single object without the need for the methods to return this. When you use the cascade notation, Dart knows to use the same object for each subsequent operation.

void main() {
  Car myCar = Car()
    ..accelerate()
    ..brake()
    ..repaint('blue');

  print(myCar);  // Output: Car(color: blue, speed: 0)
}

Summary

A fluent interface is a design pattern where the methods are designed to be chained together.
Cascade notation is a language feature in Dart that allows you to chain multiple operations on an object, whether or not the methods are designed to be chained together.

Both achieve a similar goal but do it in slightly different ways. Fluent interfaces are more common in languages that don't have an equivalent to Dart's cascade notation.

Intermezzo

That's a good answer actually. So I moved on and read the next line from the Dart documentation.

If the object that the cascade operates on can be null, then use a null-shorting cascade (?..) for the first operation. Starting with ?.. guarantees that none of the cascade operations are attempted on that null object.

I'm using my own code to understand this (probably a poor example but I get what the docs trying to say)

void main() {
  var myCar;
  myCar
    ?..color = 'red'
    ..speed = 100
    ..accelerate()
    ..repaint('blue');

  print(myCar); // Output: null
}

The previous code is equivalent to the following:

void main() {
  var myCar;
  myCar?.color = 'red';
  myCar?.speed = 100;
  myCar?.accelerate();
  myCar?.repaint('blue');

  print(myCar); // Output: null
}

Here are the differences:

  var myCar;
-  myCar
-    ?..color = 'red'
-    ..speed = 100
-    ..accelerate()
-    ..repaint('blue');
+  myCar?.color = 'red';
+  myCar?.speed = 100;
+  myCar?.accelerate();
+  myCar?.repaint('blue');

Question 2

The Dart documentation says:

Be careful to construct your cascade on a function that returns an actual object. For example, the following code fails:

var sb = StringBuffer();
sb.write('foo')
  ..write('bar'); // Error: method 'write' isn't defined for 'void'.

I don't really get it since my all methods are void indeed. So I asked ChatGPT again.

What do the docs mean by the void? Meanwhile, the previous snippet of code worked just fine with the void function.

ChatGPT Reply 2

The documentation is warning you that you cannot use cascade notation on a function or method that returns void. This is because the cascade notation works by treating the value returned by each cascaded operation as the object to perform the next operation on. In the case of methods that return void, there is no value to cascade onto.

However, you'll notice that your Dart example works because you're not actually cascading on the return value of the void methods (accelerate, brake, repaint). You're cascading on the original Car object (myCar) which the cascade notation keeps "in focus" throughout the chained operations. In this context, void methods are permitted because they're not breaking the cascade; the cascade remains rooted in myCar.

For example, this would be problematic:

void main() {
  // This is an attempt to cascade on the result of accelerate(),
  // but that's void, so this will result in a syntax error.
  var myCar = Car().accelerate()..brake();  // Error
}

But in your working example:

Car myCar = Car()
  ..color = 'red'
  ..speed = 100
  ..accelerate()
  ..repaint('blue');

You are setting the initial object for the cascade to Car(), and the cascade continues to operate on that object, not on the return values of the void methods.

So, in summary, you can include void methods within a cascade, as long as you're still rooted in an object that isn't void.

Hoisting: Does the Order Matter for Functions in Dart (and Python)?

Retiago Drago — Sun, 17 Sep 2023 03:07:49 +0000

Question 1

I wonder if the order matters for the declaration of functions in Dart. So I asked ChatGPT on this one. Let's suppose this code exists:

void main() {
  greet();
}

void greet() {
  print('Hello!');
}

If I swap the order, does the Dart program throw an error or work just fine?

ChatGPT Reply 1

In Dart, the order in which functions are declared does not matter. Dart supports hoisting, which means you can call a function before its declaration in the code file. The following two versions of the code are equivalent:

Version 1:

void main() {
  greet();
}

void greet() {
  print('Hello!');
}

Version 2:

void greet() {
  print('Hello!');
}

void main() {
  greet();
}

Both versions will produce the same output ("Hello!") when executed. You can define main() before greet() or vice versa; the Dart compiler will correctly identify and link the function calls no matter the order.

However, this principle applies to top-level functions and does not extend to local functions (functions declared within other functions). Local functions must be declared before they are called within their containing function.

Question 2

Since I come from Python I try to ask this as well.

Does Python support hoisting as well?

ChatGPT Reply 2

No, Python does not support hoisting in the way that languages like Dart or JavaScript do. In Python, you must define a function before you call it. Consider the following example:

This will raise an error:

say_hello()

def say_hello():
    print("Hello!")

The error will state that say_hello is not defined because Python will not have seen the definition of say_hello by the time it tries to call it.

However, the following will work:

def say_hello():
    print("Hello!")

say_hello()

Here, say_hello is defined before it is called, so the Python interpreter knows what to do when say_hello() is encountered.

Classes and functions need to be defined before they are used in the Python code execution flow. If you attempt to use a function or class before declaring it, you will get a NameError.

Question 3

This answer doesn't necessarily address my intention since I can work with unorder methods (not functions even though methods behave like functions in a bit). So I ask the following question.

Does hoisting work in method instead of function in Python?

ChatGPT Reply 3

In Python, the concept of hoisting doesn't apply to either functions or methods. All names (variables, functions, methods, classes, etc.) must be defined before they are used, regardless of whether they are part of a class (as methods) or stand-alone functions.

Here are some examples to clarify:

Functions

This will not work:

call_function()

def call_function():
    print("Function called.")

This will work:

def call_function():
    print("Function called.")

call_function()

Methods

This will not work:

class MyClass:
    def method_one(self):
        self.method_two()

    def method_two(self):
        print("Method two called.")

# Create an instance
obj = MyClass()

# Call method_one (which in turn calls method_two)
obj.method_one()  # This will work because method_two is defined before it is called

This will work, but the order in which the methods are defined within the class doesn't matter:

class MyClass:
    def method_two(self):
        print("Method two called.")

    def method_one(self):
        self.method_two()

# Create an instance
obj = MyClass()

# Call method_one (which in turn calls method_two)
obj.method_one()  # This will work because method_two is defined in the class before it is called

As you can see, in Python, what matters is that the function or method is defined before it is used, not the order in which they are written in the code.

Question 4

The previous reply did contradict itself. So I complain.

Still don't understand this "This will work, but the order in which the methods are defined within the class doesn't matter:". If the order doesn't matter, the first code of the block of the method should work just fine.

ChatGPT Reply 4

I apologize for the confusion. My explanation was unclear. You are correct that the order of method definitions within a class in Python doesn't matter. All methods within a class are available once the class definition is complete, regardless of the order in which they are defined. Therefore, you can indeed call any method from any other method in the same class, regardless of their order.

The first block of code with methods should work fine because both method_one and method_two are defined in the class:

class MyClass:
    def method_one(self):
        self.method_two()

    def method_two(self):
        print("Method two called.")

# Create an instance
obj = MyClass()

# Call method_one (which in turn calls method_two)
obj.method_one()  # This will work because method_two is defined before it is called

In this case, the order of method_one and method_two doesn't matter; both will work as expected regardless of which one is defined first. Thank you for catching that mistake, and I apologize for the oversight.

Question 5

I just need to clarify this time.

So hoisting work only for methods within class?

ChatGPT Reply 5

The concept of "hoisting" doesn't strictly apply to Python methods in the same way it does for variables and functions in languages like JavaScript. In Python, both functions and methods need to be defined before they are called, but within a class definition, methods are all available to each other regardless of their order in the source code.

In the following example, both method_one and method_two are part of the class definition, and they can call each other regardless of the order in which they appear:

class MyClass:
    def method_one(self):
        self.method_two()

    def method_two(self):
        print("Method two called.")

This "availability" of methods within the same class isn't typically described as hoisting in Python. Rather, it's a feature of how Python's object-oriented system works. All methods within a class are available to each other once the class definition is complete, regardless of their order of definition.

I hold back on job applications, compensating with tech dev blogging and ongoing side projects – this ain't effective innit?

Retiago Drago — Fri, 15 Sep 2023 13:25:32 +0000

I am a Python programmer. I consider myself a jack of all trades, but I tend to work on data-related or AI projects over software engineering. The thing is, I feel like I cannot see my own strengths. I don't feel like I meet the job market's skills when I apply for a job. I cope with it by writing a lot of tech dev posts, if they are ever considered as proper portfolios. Before writing, I need to practice the technical aspect of it, which both takes a long time. My main problem now is that I back off from applying to jobs since I still cannot convince myself that I am enough for the job market. I make up for that by writing tech blogs and finishing perpetual side projects as portfolios, which also take a long time. I don't even have the courage to find a client as a freelancer, although I successfully finished one freelance job last month. I know my approach is not effective and is more like escaping at this point. If you were in my shoes, what would you do to solve this mess?

Exploring Steganography in the Wild - Part 1

Retiago Drago — Tue, 12 Sep 2023 07:36:19 +0000

Outlines

Introduction

Steganography

Digital Images and Pixels

Color Models

Binary Code

What To Expect

Project Setup

Core Concepts

Mapping Techniques

Bit Manipulation for Hiding Images

Bit Manipulation for Revealing Images

Bit Manipulation for Reconstructing Images

Implementation Details

Supported Image Formats

Code Structure

Explanation of Helper Functions

Explanation of Steganograph Class

Results and Visualization

Visual Examples of Pre-Encoded Images

Visual Examples of Encoded Images

Visual Examples of Decoded Images

Visual Examples of Reconstructed Images

Evaluation Metrics

Limitations and Future Work

Conclusion

Summing Up the Journey

Technical Achievements

Key Takeaways

Additional Sections

Introduction

Welcome to a fascinating exploration of steganography and its applications. Steganography is a technique often cloaked in mystery, yet it serves practical purposes that extend from cybersecurity to digital watermarking. In this blog post, we'll delve into the nitty-gritty details of how steganography is applied to digital images.

Steganography

Steganography is the practice of representing information within another message or physical object, in such a manner that the presence of the information is not evident to human inspection - Wikipedia.

In simpler terms, steganography is akin to embedding a covert message within another overt message—in our case, hiding one image within another.

Digital Images and Pixels

A digital image is an image composed of picture elements, also known as pixels, each with finite, discrete quantities of numeric representation for its intensity or gray level that is an output from its two-dimensional functions fed as input by its spatial coordinates denoted with x, y on the x-axis and y-axis, respectively - Wikipedia.

Imagine a digital image as an intricate mosaic composed of minuscule tiles, known as pixels. Each pixel contributes to the overall image by adding its own splash of color. The more pixels you have, the more vivid and detailed your image becomes.

Color Models

Color models are tools central to color theory that define the color spectrum based on presence or absence of a few primary colors - onlinelibrary.wiley.com

Think of color models as the recipe books for digital artists. Just like mixing primary colors in art class, color models help us understand how to blend different amounts of red, green, and blue light to produce various hues. In this discussion, we'll focus on the RGB (Red-Green-Blue) color model, which is the cornerstone for reproducing a wide array of colors in digital media.

The name of the model comes from the initials of the three additive primary colors, red, green, and blue. The main purpose of the RGB color model is for the sensing, representation, and display of images in electronic systems, such as televisions and computers - Wikipedia.

Binary Code

Binary code is a system of representing information using only two symbols, typically 0 and 1 - Britannica

In the realm of digital images, each pixel's color information is represented by 8-bit binary digits. This means each pixel can display one of (2^8) or 256 possible colors.

In computing, the most significant bit (MSb) represents the highest-order place of a binary integer. Similarly, the least significant bit (LSb) is the bit position in a binary integer representing the binary 1s place of the integer - Wikipedia

The concept of most significant bit (MSb) and least significant bit (LSb) is crucial here. Altering the MSb can have a profound impact on a pixel's color, while changes to the LSb are usually subtle.

The leftmost bit is the most significant bit. We change this bit and it will affect tremendously on the value. For example, we flip the leftmost bit of the binary value of 165 from 1 to 0 (from 10100101 into 00100101) it will change the decimal value from 165 into 37.

On the other hand, the rightmost bit is the least significant bit. We change this bit and it won't affect tremendously on the value. For example, we flip the rightmost bit of the binary value of 165 from 1 to 0 (from 10100101 into 10100100) it will change the decimal value from 165 into 164.

So, let's tie it all together: each pixel in a digital image comprises three color values (RGB), each represented by an 8-bit binary digit. We'll exploit the "insignificance" of the least significant bits to embed one image into another without causing noticeable changes to the host image. This is the magic key to image-based steganography. We'll change the less significant bits of one image to include the most significant bits of another.

What To Expect

In this technical dev blog post, you'll find:

A deep dive into steganography, particularly the art of concealing one image within another.
A focus on LSB steganography, which involves manipulating the least significant bits of pixels in a cover image to hide another image.
Practical code examples and vivid visualizations to help you understand the process of hiding and retrieving images.
A discussion on the challenges and limitations one might encounter, such as the quality of the concealed image.

So, buckle up for an exciting journey through the world of steganography!

Project Setup

Before diving into the technicalities of steganography, it's essential to set up your workspace properly. This ensures that you can effortlessly follow along with the examples and get the most out of this exploration. Whether you're a seasoned coder or just dipping your toes into the Python ecosystem, I've got you covered.

I've created a Google Colab-compatible Jupyter Notebook to make this process seamless. The notebook is designed with a user-friendly UI/UX and hides the raw code, so you can focus on learning. And the best part? The code is pre-validated to run without hitches on the first try!

Open in Colab

If you're new to Google Colab or Python in general, don't worry! Here's an introductory video on Google Colab for Beginners to help you get started.

For those who are more experienced and might prefer to run the project locally, here are the Python package requirements to set up your environment:

Python 3.7.12
matplotlib==3.2.2
matplotlib-inline==0.1.3
matplotlib-venn==0.11.6
numpy==1.21.5
Pillow==7.1.2
scikit-learn==1.0.2

Once you have your environment set up, you'll be ready to delve into the world of steganography with me!

Core Concepts

Mapping Techniques

Before diving into the nitty-gritty details, it's crucial to understand the four types of binary value mappings that we'll be using:

Left-Half Bits Mapping (LHB Map)

Think of this mapping as a way to extract the left-half bits values from the cover image within the merged image.
Right-Half Bits Mapping (RHB Map)

This mapping aims to extract the left-half bits values of the hidden image from the merged image.
Whole Bits Mapping (MB Map)

This mapping is utilized to construct the whole bits values of the merged image, using the left-half bits values from both the cover and hidden images.
Possible Right-Half Bits Mapping (RRHB Map)

In this case, we'll randomly pick possible half bits values as we don't store the information about the half bits beforehand. The aim here is to construct the right-half bits values of the unmerged left-half bits image for both the unmerged cover and hidden images.

Bit Manipulation for Hiding Images

Now that we've understood the mapping types, let's look at how these mappings play a crucial role in hiding—or encoding—one image within another.

Encoding Process

Here's a summarized flow of the encoding process:

Choose Images: Initially, you have to pick two images:
- A "Cover Image," which will serve as the shell where your secret image will reside.
- A "Hidden Image," which is the secret image you wish to hide within the cover image.
Extract and Map LHB (Left-Half Bits): The "Hidden Image" undergoes LHB (Left-Half Bits) Mapping. This extracts the left-half bits from the hidden image and maps them to create a new image called "Left-half bits Hidden Image."
Position the New Image: The "Left-half bits Hidden Image" is then positioned within the "Cover Image" according to specified criteria (e.g., upper left corner, lower right corner, etc.).
Merge: Finally, the "Cover Image" and the positioned "Left-half bits Hidden Image" are merged together. This forms the "Merged Image," which looks like your original cover image but has the hidden image embedded within it.

Bit Manipulation for Revealing Images

Let's shift our focus to how we can extract—or decode—the hidden image from the merged image.

Decoding Process

Here's a quick rundown of the decoding process:

Use Merged Image: The initial step in decoding involves taking the "Merged Image," which contains both the "Cover Image" and the "Hidden Image."
Extract Bits: The "Merged Image" is then divided into its component "left-half bits" (LHB) and "right-half bits" (RHB). These are different portions of the image's pixel data.
Retrieve Images: The "left-half bits" are used to reconstruct an "Unmerged Left-half bits Cover Image," effectively extracting what used to be the cover image. Note that the right-half bits for this image are still zero. The "right-half bits" are used to reconstruct a "Unmerged Left-half bits Hidden Image," effectively extracting what used to be the hidden image, also with zero right-half bits. The position of this hidden image within the merged image is accounted for during this step.

Bit Manipulation for Reconstructing Images

Finally, let's look at how we can reconstruct the original images from their half-bit versions.

Reconstruction Process

Here's how the reconstruction process works:

Use Unmerged Bits: Use the unmerged left-half bits of both the cover and hidden images.
Map Right-Half Bits: Use the "RRHB Mapping" to generate possible right-half bits values randomly.
Reconstruct Images: The result is a reconstructed cover and hidden image.

I hope these explanations provide a clearer understanding of the steganography techniques involved in hiding, revealing, and reconstructing images.

Implementation Details

Supported Image Formats

In this exploration, we'll limit our focus to only two image formats: PNG and JPEG. This limitation helps keep our exploration straightforward.

Code Structure

Before diving into the code's functionality, it's essential to get a grasp of its structure. Below is a snippet that outlines the main components of the code.

.
├── get_bits_dict(start: int, end: int, recon: bool = False) -> Dict[int, int]
├── get_merged_bits_array() -> np.ndarray
├── dict_to_nparray(d: dict) -> np.ndarray
├── dict_to_2darray(d: dict) -> np.ndarray
└── Steganograph
    ├── Object Attributes
    │     ├── ispng: bool
    │     ├── original_cover_image: numpy array
    │     ├── original_hidden_image: numpy array
    │     ├── left_half_bits_hidden_image: numpy array, default None
    │     ├── merged_image: numpy array, default None
    │     ├── unmerged_left_half_bits_cover_image: numpy array, default None
    │     ├── unmerged_left_half_bits_hidden_image: numpy array, default None
    │     ├── reconstructed_cover_image: numpy array, default None
    │     └── reconstructed_hidden_image: numpy array, default None
    ├──── Methods
    │     ├── encode(pos: str = 'upper_left')
    │     ├── decode(pos: str = 'upper_left')
    │     ├── encode_decode(pos: str = 'upper_left')
    │     ├── reconstruct()
    │     ├── encode_decode_recon(pos: str = 'upper_left')
    │     ├── is_two_images_identical(opt: int = 0) -> bool
    │     ├── save_image(opt: int = 0)
    │     ├── plot_original()
    │     ├── plot_left_half_bits()
    │     ├── plot_merged_image()
    │     ├── plot_unmerged_left_half_bits()
    │     └── plot_recon()
    └──── Class Attributes
          ├── __lhb_lookup: lhb #Left half bits array lookup
          ├── __rhb_lookup: rhb #Right half bits array lookup
          ├── __mb_lookup: mb #Merged bits 2D array lookup
          ├── __rrhb_lookup: rrhb #Reconstruction right half bits dict lookup
          ├── __format: tuple ('jpeg', 'png')
          ├── __rgb: tuple ('RGB', 'Red', 'Green', 'Blue')
          └── __pos: tuple ('upper_left', 'upper_right', 'lower_left', 'lower_right')

Explanation of Helper Functions

The code incorporates several helper functions tailored for the art of steganography. Here's a brief rundown:

get_bits_dict: This function produces a dictionary that maps an integer (ranging from 0 to 255) to another integer. The mapping varies based on whether the process is for reconstruction. In the standard case, it extracts the left-half bits, whereas for reconstruction, it extracts the right-half bits.
dict_to_nparray: Converting the dictionary from get_bits_dict into a numpy array facilitates faster lookup operations.
get_merged_bits_array: This function constructs a 2D array where each cell at index [i][j] holds the merged left-half bits (cover image) of i and the left-half bits (hidden image) of j.
dict_to_2darray: This works similarly to dict_to_nparray, but it reshapes the array into a 2D structure, particularly useful for image reconstruction.
Respective function callings for later:
- Generates a numpy array for quick look-up of the left-half bits of a given number.
- Creates a numpy array for speedy retrieval of the right-half bits of a number.
- Constructs a 2D numpy array for efficient look-up of the merged bits of two numbers.
- Builds a 2D numpy array for quick reference of the right-half bits of a number during the reconstruction phase.

Explanation of Steganograph Class

Explanation of the `init` Method:

The __init__ method initializes an instance of the Steganograph class.

Parameters:

cover_image_filepath: File path for the cover image (supports only JPEG or PNG formats).
hidden_image_filepath: File path for the hidden image (supports only JPEG or PNG formats).

Steps:

Determine Image Formats: The method first determines the format (JPEG or PNG) of both the cover image and the hidden image using the imghdr.what() function. The formats are stored in format_ci and format_hi variables.
Validation: It checks whether both images have valid formats. If the formats don't match any in the class' __format attribute, it raises a TypeError.
Check Format Uniformity: It checks if both images have the same format. If not, it raises a TypeError.
PNG Format Flag: Sets the ispng flag to True if the format is PNG; otherwise, sets it to False.
Read Images: Reads the images into original_cover_image and original_hidden_image using the plt.imread() function from Matplotlib.
Invoke read_and_adjust_images: Calls the method read_and_adjust_images() to perform additional adjustments on the images.
Set Initial States: Initializes the other attributes to None, as these would be populated by other methods later.

Explanation of the `read_and_adjust_images` Method:

This method adjusts the read images for further processing, especially for PNG images. By the end of these methods, the class instance is well-prepared with loaded and adjusted images, and is ready for encoding and decoding operations.

Steps:

Check if PNG: If the images are in PNG format (ispng is True):
- Ignore Alpha Channel: If the images have an alpha (A) channel, it ignores it for now.
- Normalization: PNG images are often read in a normalized format where the pixel values are floats between 0 and 1. This method multiplies them by 255 and rounds off to convert them to integers.
- Type Casting: It then casts the numpy arrays to 'uint8' type, ensuring that they are 8-bit unsigned integers.

Explanation of the `encode` Method:

The encode method is responsible for creating a simple steganograph—merging a hidden image into a cover image in a way that it can later be extracted.

Parameters:

pos: Specifies where the hidden image will be located in relation to the cover image. Default is 'upper_left'.

Steps:

Position Validation: It validates whether the given position pos is one of the four pre-defined positions (upper_left, upper_right, lower_left, lower_right). If not, it raises a ValueError.
Get Left Half Bits: It calls get_left_half_bits to extract the left half-bits of the hidden image and stores them in the left_half_bits_hidden_image attribute.
Merge Half Bits: It then calls merge_two_half_bits to merge the modified hidden image with the cover image.

Explanation of the `get_left_half_bits` Method:

Parameters:

img_arr: The image array that you want to extract the left half-bits from.

Steps:

Lookup: It looks up the left half-bits of the image array img_arr using a pre-defined lookup table __lhb_lookup.
Return: It returns the left half-bits.

Explanation of the `merge_two_half_bits` Method:

Parameters:

pos: The position where the hidden image will be located in relation to the cover image.

Steps:

Copy Original Cover Image: It first creates a copy of the original cover image and stores it in self.merged_image.
Get Slicing Range: It calls the get_slicing method to determine the slice where the hidden image will be placed.
Merge the Half Bits: It merges the cover image and the modified hidden image together to create the final merged image (steganograph). It does this by using a pre-defined 2D lookup table __mb_lookup for each channel (RGB).
Type Casting: Finally, it casts the merged_image into 'uint8' type.

Explanation of the `get_slicing` Method:

Parameters:

large_shape: A tuple representing the shape of the larger array, which is the cover image in this case.
small_shape: A tuple representing the shape of the smaller array, which is the hidden image.
pos: The position where the hidden image will be located in relation to the cover image. Valid options are 'upper_left', 'upper_right', 'lower_left', and 'lower_right'.

Steps:

Extract Shape Information: The method first extracts the shape information of both the larger and smaller arrays. This information includes the number of rows and columns in each array.
Calculate Slicing Range: Based on the pos value, the function calculates the range in the larger array where the smaller array will be placed. This is done using numpy slice notation (np.s_).
Error Handling: If an invalid pos value is provided, a ValueError is raised.
Return: The function returns the calculated slicing range, which can be used as a slice object for numpy arrays. This slice object will then be used to place the hidden image in the specified position within the cover image.

By the end of the encode method, you'll have a steganograph where the hidden image is merged into the cover image at the specified position.

Explanation of the `decode` Method:

The decode method aims to break down the steganograph created by the encode method to recover the original hidden and cover images.

Parameters:

pos: This parameter specifies where the hidden image was embedded within the cover image. The default is 'upper_left'.

Steps:

Validation: First, the method checks if self.merged_image is None. If it is, it raises a TypeError. It also validates that the position pos is one of the acceptable positions. If not, a ValueError is raised.
Unmerge: It calls the unmerge_two_half_bits method to separate the cover and hidden images from the merged image.

Explanation of the `unmerge_two_half_bits` Method:

Parameters:

pos: Specifies the original position where the hidden image was embedded within the cover image.

Steps:

Slice Definition: Calls the get_slicing method to determine the slice range based on the pos value, i.e., where exactly the hidden image is located in the merged image.
Extract Left Half Bits of Cover Image: Uses the get_left_half_bits method to extract the left half bits of the merged image and assigns them to self.unmerged_left_half_bits_cover_image.
Extract Right Half Bits of Hidden Image: Similarly, it calls the get_right_half_bits method to extract the right half bits of the merged image. However, it does so only for the area specified by the slicing range returned by get_slicing.

Explanation of the `get_right_half_bits` Method:

Parameters:

img_arr: The image array that you want to extract the right half-bits from.

Steps:

Lookup: It looks up the right half-bits of the image array img_arr using a pre-defined lookup table __rhb_lookup.
Return: It returns the right half-bits.

In summary, the decode method and its supporting methods work together to reverse the steganographic process, extracting the hidden image and cover image from the merged image.

Explanation of the `reconstruct` Method:

The reconstruct method aims to restore the original cover and hidden images from their left-half bits images, which were previously separated during the decoding process.

Steps:

Validation: The method checks that self.unmerged_left_half_bits_cover_image and self.unmerged_left_half_bits_hidden_image are not None. If either of them is None, it raises a TypeError, prompting you to run the decode() method first.
Reconstruct Cover Image: The reconstruct_right_half_bits method is called to reconstruct the right-half bits of the cover image. The reconstructed image is stored in self.reconstructed_cover_image.
Reconstruct Hidden Image: Similarly, the reconstruct_right_half_bits method is used to reconstruct the right-half bits of the hidden image. The reconstructed image is stored in self.reconstructed_hidden_image.

Explanation of the `reconstruct_right_half_bits` Method:

Parameters:

im_arr: The left-half bits of an image as a numpy array.
seed: A seed for the random number generator, defaulting to 1999.

Steps:

Random Seed: The np.random.seed function is used to set the random seed to ensure that the random process is reproducible.
Flattening Image: The ravel() method is used to flatten the im_arr, essentially turning it into a one-dimensional array (flat_im_arr).
Random Indices: A random set of indices (random_indices) is generated using np.random.randint, with the size of the array based on the shape of flat_im_arr. This will simulate the random selection of right-half bits for each corresponding left-half bit.
Reconstruction: The lookup table __rrhb_lookup is used in conjunction with random_indices to get a set of right-half bits for each corresponding left-half bit in flat_im_arr. The result (reconstructed_flat) is a one-dimensional array containing the reconstructed right-half bits.
Reshaping: The reconstructed one-dimensional array is reshaped back into its original shape (shape) to form the reconstructed image.
Type Casting: Finally, the numpy array is cast to the type 'uint8'.
Return: The reconstructed image, now having both left and right half-bits, is returned.

In summary, the reconstruct method and its sub-method reconstruct_right_half_bits work together to restore the original cover and hidden images from their left-half bits versions. This final step completes the round-trip process of encoding, decoding, and reconstructing the images.

For the complete code and a more detailed walkthrough, check out the Jupyter Notebook linked below:

Open in Colab

Results and Visualization

Having delved into the theory and implementation details, it's time to see our steganography technique in action. We'll examine the visual changes and quality metrics at each stage of the process—encoding, decoding, and reconstruction.

Visual Examples of Pre-Encoded Images

First, let's consider our "Cover Image" and "Hidden Image." The image below serves as our cover:

Next, we have the hidden image:

For a deeper understanding, let's examine the RGB plot of these images. Click on the image below to enlarge it.

Visual Examples of Encoded Images

After encoding, we obtain two new images: the "Left Half Bits Hidden Image" and the "Merged Image."

Left Half Bits Hidden Image

The encoded "Left Half Bits Hidden Image" noticeably loses some quality compared to the original hidden image. For instance, the gradients do not blend as smoothly.

Here's the RGB plot for a more in-depth analysis:

Merged Image

At first glance, the "Merged Image" looks almost identical to the original cover image. However, a keen eye might spot a subtle face formation in the upper-left corner.

For a more analytical view, let's check the RGB plot:

Visual Examples of Decoded Images

Next, we decode the merged image to obtain the "Unmerged Left Half Bits Cover Image" and the "Unmerged Left Half Bits Hidden Image." Both of these images show a loss in quality and coherence, similar to what we observed with the left half bits hidden image.

Take a look at the RGB breakdown for further insights:

Visual Examples of Reconstructed Images

Finally, we move on to the reconstruction stage, where we aim to approximate the original images as closely as possible.

Reconstructed Cover Image

When compared to the unmerged version, the reconstructed cover image appears slightly brighter. However, it's essential to note that the image quality is still not on par with the original. Gradients, for example, do not blend as smoothly as they should.

Reconstructed Hidden Image

A similar trend is observed in the reconstructed hidden image. Though somewhat brighter than the unmerged version, the quality still falls short of the original.

RGB Breakdown of Reconstructed Images

For a more analytical perspective, let's examine the RGB plots of these reconstructed images:

Evaluation Metrics

For the evaluation, I utilized the Root Mean Square Error (RMSE) and Mean Absolute Error (MAE) metrics. These metrics help to quantify the differences between the images at different stages of the steganography process. While I can't speak to their application in formal steganalysis, they do offer a way to gauge the effectiveness of the encoding and decoding steps. If you have insights or recommendations on other metrics that might be more suitable, especially from a steganalysis point of view, feel free to share.

For instance, comparing the "Left Half Bits Hidden Image" with the original hidden image yielded the following results:

RMSE    : 8.868079206151936
MAE     : 7.579809053497942

This indicates that the two images are indeed different. However, when comparing the "Left Half Bits Hidden Image" with the "Unmerged Left Half Bits Hidden Image," the result was:

RMSE    : 0.0
MAE     : 0.0

This confirms that the encoding and decoding processes were executed correctly, as the two images are logically the same.

Once again, for the complete code and walkthrough, just run the Jupyter Notebook linked below:

Open in Colab

This concludes the "Results and Visualization" section, where we've explored the visual aspects and quality metrics of images at every stage—encoding, decoding, and reconstruction. These visual examples and metrics serve as a comprehensive evaluation of our steganography technique.

Limitations and Future Work

In this section, we delve into the limitations of the current implementation and discuss opportunities for future enhancements.

Image Channel Handling: The code is designed to work specifically with RGB color spaces. This makes it less versatile when dealing with other color spaces or grayscale images.
Limited File Format Support: The code currently supports only PNG and JPEG formats, potentially excluding users with images in other formats.
Code Complexity: The codebase is somewhat intricate, which might make it less accessible for individuals who are not familiar with the project's specifics.
Error Handling: While some error-handling mechanisms exist, they may not be comprehensive enough to address all potential edge cases.
Hard-Coded Values: The presence of hard-coded values in the code reduces its flexibility and adaptability.
Google Colab Dependency: The code relies on Google Colab-specific libraries, making it less portable for users who may prefer other development environments.
Lack of Modularization: The code is not modular, which could complicate future efforts to extend its functionality or maintain it.

Conclusion

Summing Up the Journey

Enlightening and Challenging: Our exploration into steganography has been a revealing journey, showcasing both the potential and the challenges of this field.
Practical Applications: We demonstrated how one image could be hidden within another, a technique with various real-world applications such as digital watermarking and secure communications.
Identified Limitations: Like any scientific endeavor, our work has its constraints, from the limitations in color channel handling to dependencies on specific platforms like Google Colab. These offer avenues for future research and refinement.

Technical Achievements

Bit Manipulation: One of the technical milestones of this project is the successful implementation of bit manipulation techniques to encode and decode images.
Visual Proof: We provided visual examples to demonstrate our techniques in action. While the hidden image is not perfectly concealed in our experiments, we believe that in scenarios with more detailed and crowded cover images, the concealment could be more effective.
Quantitative Analysis: We've used metrics such as RMSE and MAE to measure differences between original and manipulated images. While these metrics offer insights, they may not be specifically tailored for steganalysis. If you have expertise in steganalysis, your feedback in the comments would be highly valuable.

Key Takeaways

The Complexity of Simplicity

What seemed like a straightforward task—hiding one image within another—unfolded into a labyrinth of complexities. The project has underscored the intricacies of image manipulation and bit-level operations.

Robustness and Flexibility are Paramount

The limitations we've faced have highlighted the importance of robust and flexible code. As we continue to refine this project, the lessons learned here will be invaluable.

Continuous Learning and Improvement

The world of steganography is vast, and there's so much more to explore. Your feedback and contributions are not just welcome but are essential for the continuous improvement of this project.

The Future is Open

The avenues for future work are promising. They range from refining the current code to make it more versatile to exploring more advanced steganographic techniques. With ongoing research and collaboration, the sky is the limit for what can be achieved.

Additional Sections

Code Repository: You can find the complete code and notebooks on GitHub.

ranggakd / steganography

Exploring Steganography In The Wild

steganography

Exploring Steganography In The Wild

Basic Steganography

View on GitHub

References: Steganography: Hiding an image inside another

Colouring Your Arrow / Link with `linkStyle` in Mermaid Markdown

Retiago Drago — Sun, 10 Sep 2023 20:19:31 +0000

Welcome back to Technically Speaking, your sanctuary for diving deep into the labyrinthine wonders of technology. Today, we turn the spotlight onto Mermaid, a marvelous tool that's become my go-to for diagramming and documentation. Specifically, we're going to add some splashes of colour to the arrows or links in our Mermaid Markdown flowcharts. Let's get to it!

Setting the Stage

First things first, we need a basic flowchart or graph to work with. Here's a snippet from my upcoming post on steganography, showcasing a flowchart that I built using Mermaid Markdown:



graph TB
    Decode([Decode])
    End([End])
    ULhbCover[/"Unmerged Left-half
    bits Cover Image"/]
    CULhbHidden[/"Cover Image-sized Unmerged Left-half
    bits Hidden Image"/]
    ULhbHidden[/"Unmerged Left-half
    bits Hidden Image"/]
    Merged[/Merged Image/]
    Lhb[LHB Mapping]
    Rhb[RHB Mapping]

    Decode --> Merged
    Merged --> Lhb & Rhb
    Lhb --> ULhbCover
    Rhb --> CULhbHidden
    CULhbHidden --"pass on hidden image
    position in merged image"--> ULhbHidden
    ULhbCover & ULhbHidden --> End

The Art of Styling Links

When I first stumbled upon the Mermaid documentation on styling links, it seemed simple enough. Just use this one-liner:



linkStyle default stroke:green;

The default parameter applies the styling to all links or arrows in your chart. However, there was one hiccup: the documentation didn't specify where to put this mysterious one-liner. Initially, I faced errors that had me scratching my head. Then, eureka! It occurred to me to place the styling directive after all the link syntax, like so:



graph TB
    Decode([Decode])
    End([End])
    ULhbCover[/"Unmerged Left-half
    bits Cover Image"/]
    CULhbHidden[/"Cover Image-sized Unmerged Left-half
    bits Hidden Image"/]
    ULhbHidden[/"Unmerged Left-half
    bits Hidden Image"/]
    Merged[/Merged Image/]
    Lhb[LHB Mapping]
    Rhb[RHB Mapping]

    Decode --> Merged
    Merged --> Lhb & Rhb
    Lhb --> ULhbCover
    Rhb --> CULhbHidden
    CULhbHidden --"pass on hidden image
    position in merged image"--> ULhbHidden
    ULhbCover & ULhbHidden --> End

    linkStyle default stroke:green;

And just like that, it worked! Behold the green strokes in all their glory:

Uncharted Waters

There are still a couple of nuances that have eluded me. One is colouring the arrowhead itself; I did ask BingChat for a solution, but it involved intricate CSS—a no-go for this Markdown-centric endeavor. Secondly, I'm on a perpetual quest for a contrasting background. Dark mode is my jam, and unfortunately, Mermaid's PNG output doesn't play well with it. I tried to set a solid background colour, but so far, no dice. You could try it live yourself here.

So there we have it—your links now boast eye-catching strokes, although the quest for perfect styling remains ever ongoing. If you've got insights on the elusive arrowhead colouring or background contrast, do share! Until then, keep exploring, keep questioning, and most importantly, keep Technically Speaking.

Happy diagramming! 📊✨

Collapsible Markdown on GitHub Markdown and Beacons AI

Retiago Drago — Sat, 09 Sep 2023 05:03:43 +0000

Welcome to Technically Speaking, a series that will serve as the hub for my latest technical escapades. The focus here is strictly technical—covering everything from code snippets to full-blown projects. Consider this my daily commitment to better coding habits and creative building!

Today, let's dive into how to create collapsible markdown sections on GitHub and Beacons AI. The notion struck me after stumbling upon a gist by pierrejoubert73. Here's what sparked the idea:

Beacons AI: The About Me Section

My first playground was my Beacons AI 'About Me' section, where markdown formatting is welcomed. Check out the markdown code I used:



<center>
<details>
<summary>
👉👈
</summary>

### Oh, hello there! 👋

|  | I'm a / an |
| :--- | :---: |
| Programmer | ✔ |
| AI Tech Writer | ✔ |
| Data Practitioner | ✔ |
| Statistics & Math Addict | ✔ |
| Open Source Contributor | ✔ |
| Quantum Computing Enthusiast | ✔ |
</details>
</center>

Curious to see it in action? The code renders as shown below. Feel free to explore and experiment on my Beacons AI page.

A word of caution: this could impact your SEO preview, as evidenced by this Telegram snapshot.

GitHub: README Section

Next, I integrated collapsible markdown into my GitHub README. Here's a snippet:



<details>
<summary>
With <strong>me</strong>
</summary>
<center>

|  | I'm a / an |
| :--- | :---: |
| Programmer | ✔ |
| AI Tech Writer | ✔ |
| Data Practitioner | ✔ |
| Statistics & Math Addict | ✔ |
| Open Source Contributor | ✔ |
| Quantum Computing Enthusiast | ✔ |
</center>
</details>

<details>
<summary>
I <strong>code</strong> in the dark of my room 👨‍💻
</summary>
<div align="right">

And I predict my data, **forecasting** from afar (oh)  
"**Unittest**," but it's not just a few  
**Data analysis** keeps me on cue
</div>
</details>

<details>
<summary>
And I <strong>study now</strong>, the current state 💻
</summary>
<div align="right">

Of **Rust** and **Julia** to seal my fate (oh)  
And in **Python**, I find my worth  
"**SQL**", ain't that the geekiest you ever heard?  
I write those, grinding like a coder
</div>
</details>

It's worth noting that the center tag doesn't yield the same results on GitHub as it does on Beacons AI. To see how it renders, visit my GitHub profile.

And that wraps up today's episode of Technically Speaking. Collapsible markdown is more than just a neat trick; it's a way to make your pages more interactive and engaging. So go ahead, give it a whirl, and let me know how it works for you!

Why Haven't Legal Amendments Adopted Version Control Systems Like Git?

Retiago Drago — Tue, 05 Sep 2023 23:53:41 +0000

Why does a legal amendment not yet adopted a version control system like Git for example? Law basically a text with certain layout and amendment is the same as the change.

Bing Chat said:

That's an interesting question! While I couldn't find any specific information on why legal amendments have not adopted a version control system like Git, there could be several reasons for this. One reason could be that the legal system and its processes are often steeped in tradition and can be slow to change. Additionally, the legal system is complex and involves many different parties, so implementing a new system like Git could be challenging. Another reason could be that the legal system has its own established methods for tracking changes to laws and amendments, and there may not be a perceived need for a version control system like Git. However, this is just speculation on my part. Is there anything else you would like to know? 😊

Source: Conversation with Bing, 06/09/2023
(1) version control - Alternatives to Git? - Stack Overflow.
(2) Git vs GitHub – What is Version Control and How Does it Work?.
(3) Understanding Git-Based Version Control for Industrial Automation.

Meanwhile ChatGPT said:

The idea of using a version control system like Git for legal amendments is interesting and has been proposed in various contexts, but there are several challenges and considerations that have so far limited its widespread adoption in legal systems. Here are some reasons:

Complexity of Legal Language

Legal language is highly specialized and follows strict conventions. The diffing algorithms used in version control systems may not always capture the subtleties of these changes, and so specialized tools would likely be needed.

Change Approval Process

Legal changes often need to be approved through a specific process involving multiple parties, such as various branches of government in the case of legislation. This is a lot more involved than simply accepting a pull request.

Accessibility

The law needs to be accessible to everyone, not just those who are familiar with version control systems. This means that even if such a system were used internally, a more accessible interface would need to be provided for the public.

Accountability and Security

Legal documents often require a strict chain of custody and evidentiary support to be admissible in court. The systems used for legal documents must meet high standards for security and accountability that go beyond what version control systems typically provide.

Tradition and Resistance to Change

The legal field is often slow to adopt new technologies, in part because of the high stakes involved, and in part because of institutional resistance to change.

Interpretation over Text

Law is not just about what is explicitly stated in the text but also about interpretation, precedent, and intent, which are not things that can be easily captured in a text-based version control system.

Public Records

Laws and legal documents are often considered public records that need to be kept in specific ways to be legally admissible. It's not clear that a Git-like system could meet these requirements without significant modification.

Collaboration Vs. Authority

In open-source projects managed with Git, the emphasis is on collaboration and community contributions. Legal systems, on the other hand, are often explicitly hierarchical, with changes needing to be approved by a specific authority.

Despite these challenges, there are some initiatives that aim to bring version control to legal documents, but these are mostly in the early stages and have not yet seen widespread adoption.

What is your opinion on this?

So I Explored Forecasting Metrics... Now I Want Your Two Cents 💭

Retiago Drago — Sun, 27 Aug 2023 16:50:17 +0000

Outlines

Introduction 🌟

The Objective 🎯

Metrics Investigated 🔍

Severity and Directional Emojis 🔥👉

Standard Error Metrics (MAE, MSE, RMSE) Categorization 📊

Percentage Error (MAPE, sMAPE, MBDev) Categorization 📉

R2 Score Categorization 📈

MASE Categorization 📋

Severity Emojis 🚨

Directional Emojis ➡️

Methodology 📚

Highlights of Findings ✨

Key Insights 🔑

Points for Critique 🤔

Conclusion 🤝✅

Introduction 🌟

Diving into the world of regression and forecasting metrics can be a real head-scratcher, especially if you're a newcomer. Trust me, I've been there—haphazardly applying every popular metric scholars swear by, only to be left puzzled by the results.

Ever wondered why your MAE, MSE, and RMSE values look stellar, but your MAPE is through the roof? Yep, me too.

That's why I set out on this journey to create an experimental notebook, aiming to demystify how different metrics actually behave.

The Objective 🎯

The goal of this notebook isn't to find the "one metric to rule them all" for a specific dataset. Instead, I want to understand how various metrics respond to controlled conditions in both dataset and model. Think of this as a comparative study, a sort of "Metrics 101" through the lens of someone who's still got that new-car smell in the field. This way, when I'm plunged into real-world scenarios, I'll have a better grip on interpreting my metrics.

Metrics Investigated 🔍

To get a comprehensive view, I've opted to explore a selection of metrics that are commonly leveraged in regression and forecasting problems. Here's the lineup:

Mean Absolute Error (MAE):
- Definition: It measures the average magnitude of the errors between predicted and observed values.
- Formula: $MAE=1n∑i=1n∣yi−y^i∣MAE = \frac{1}{n}\sum_{i=1}^{n} |y_i - \hat{y}_i|$ where $y_i)$ is the actual value, $(y^i)(\hat{y}_i)$ is the predicted value, and $(n)$ is the number of observations.
Mean Squared Error (MSE):
- Definition: It measures the average of the squares of the errors between predicted and observed values. It gives more weight to large errors.
- Formula: $MSE=1n∑i=1n(yi−y^i)2MSE = \frac{1}{n}\sum_{i=1}^{n} (y_i - \hat{y}_i)^2$
Root Mean Squared Error (RMSE):
- Definition: It represents the sample standard deviation of the differences between predicted and observed values. It's the square root of MSE.
- Formula: $\sqrt{MSE}$
Mean Absolute Percentage Error (MAPE):
- Definition: It measures the average of the absolute percentage errors between predicted and observed values.
- Formula: $MAPE=100n∑i=1n∣yi−y^iyi∣MAPE = \frac{100}{n}\sum_{i=1}^{n} \left| \frac{y_i - \hat{y}_i}{y_i} \right|$ Note: MAPE can be problematic if the actual value $y_i)$ is zero for some observations.
Mean Absolute Scaled Error (MASE):
- Definition: It measures the accuracy of forecasts relative to a naive baseline method. If MASE is lower than 1, the forecast is better than the naive forecast.
- Formula: $MASE=∑i=1n∣yi−y^i∣∑i=2n∣yi−yi−1∣MASE = \frac{\sum_{i=1}^{n} |y_i - \hat{y}i|}{\sum{i=2}^{n} |y_i - y_{i-1}|}$
R-squared (Coefficient of Determination):
- Definition: It indicates the proportion of the variance in the dependent variable that is predictable from the independent variable(s).
- Formula: $R2=1−∑i=1n(yi−y^i)2∑i=1n(yi−yˉ)2R^2 = 1 - \frac{\sum_{i=1}^{n} (y_i - \hat{y}i)^2}{\sum{i=1}^{n} (y_i - \bar{y})^2}$ where $(yˉ)(\bar{y})$ is the mean of the observed data.
Symmetric Mean Absolute Percentage Error (sMAPE):
- Definition: It's a variation of MAPE that addresses some of its issues, especially when the actual value is zero.
- Formula: $sMAPE=100n∑i=1n∣yi−y^i∣(∣yi∣+∣y^i∣)/2sMAPE = \frac{100}{n} \sum_{i=1}^{n} \frac{|y_i - \hat{y}_i|}{(|y_i| + |\hat{y}_i|)/2}$
Mean Bias Deviation (MBD):
- Definition: It calculates the average percentage bias in the predicted values.
- Formula: $MBD=100n∑i=1nyi−y^iyiMBD = \frac{100}{n}\sum_{i=1}^{n} \frac{y_i - \hat{y}_i}{y_i}$

Severity and Directional Emojis 🔥👉

Let's face it, numbers alone can be dry, and if you're like me, you might crave a more visceral sense of how well your model is doing. Enter severity and directional emojis. These little symbols provide a quick visual cue for interpreting metric results, ranging from "you're nailing it" to "back to the drawing board."

Disclaimer: Keep in mind that these categorizations are user-defined and could vary depending on the context in which you're working.

Standard Error Metrics (MAE, MSE, RMSE) Categorization 📊

To clarify, the concept of Normalized Error Range stems from dividing the error by the range (max - min) of the training data.

Category	Normalized Error Range
Perfect	Exactly 0
Very Acceptable	$\leq 0.05$
Acceptable	$\leq 0.1$
Moderate	$\leq 0.2$
High	$\leq 0.3$
Very High	$\leq 1$
Exceedingly High	$x > 1$

Percentage Error (MAPE, sMAPE, MBDev) Categorization 📉

Category	Error Magnitude (%)	Direction
Perfect	Exactly 0%	-
Very Acceptable	$\leq 5$	Over/Under
Acceptable	$\leq 10$	Over/Under
Moderate	$\leq 20$	Over/Under
High	$\leq 30$	Over/Under
Very High	$\leq 100$	Over/Under
Exceedingly High	$x > 100$	Over/Under

R2 Score Categorization 📈

Category	R2 Value Range
Perfect	Exactly 1
Very Acceptable	$\leq x < 1$
Acceptable	$\leq x < 0.95$
Moderate	$\leq x < 0.9$
High	$\leq x < 0.8$
Very High	$\leq x < 0.7$
Exceedingly High	$0 < x < 0.5$
Doesn't Explain Variability	Exactly 0
Worse Than Simple Mean Model	$x < 0$

MASE Categorization 📋

Category	MASE Value Range
Perfect	Exactly 0
Very Acceptable	$\leq 0.1$
Acceptable	$\leq 0.5$
Moderate	$\leq 0.9$
High	$\leq 1$
Equivalent to Naive Model	Exactly 1
Worse Than Naive Forecast Model	$x > 1$

Severity Emojis 🚨

Metric	Emoji
Perfect	💯
Very Acceptable	👌
Acceptable	✔️
Moderate	❗
High	❌
Very High	💀
Exceedingly High	☠
Doesn't Explain Variability	🚫
Worse Than Simple Mean Model	🛑
Equivalent to Naive Model	⚖
Worse Than Naive Forecast Model	🤬

Directional Emojis ➡️

Metric	Emoji
Overestimation	📈
Underestimation	📉
Nan / None	🙅‍♂️

Methodology 📚

For this experiment, I've synthesized datasets using mathematical functions like sine and cosine, which offer a controlled level of predictability. On the modeling end, I've used statsmodels.tsa.ar_model.AutoReg and OffsetModel. I chose AutoReg for its foundational role in time series forecasting, while OffsetModel serves to mimic good performance by shifting test data. This entire endeavor is laser-focused on forecasting problems, underscoring the fact that all forecasting issues are essentially regression problems, just not the other way around as far as I understand.

Highlights of Findings ✨

To navigate the labyrinth of metrics, I've laid out my explorations in a tree graph, which you can check out below:

The table here provides just a glimpse into the first phase of my deep dive into metrics. For those hungry for the full rundown, it's available right here. Click 📊 for the plot.

Plot	Based on	Variant	Dataset	Model	R2	MAE	MSE	RMSE	MASE	MAPE	sMAPE	MBDev
📊	Test Size	Small=1	$cos⁡(x)\cos(x)$	AutoReg	🙅‍♂️	👌	👌	👌	🤬	👌	👌	👌📈
📊				OffsetModel	🙅‍♂️	👌	👌	👌	🤬	👌	👌	👌📈
📊			$sin⁡(x)\sin(x)$	AutoReg	🙅‍♂️	👌	👌	👌	🤬	☠	☠	☠📉
📊				OffsetModel	🙅‍♂️	👌	👌	👌	🤬	☠	☠	☠📈
📊		Small=2	$cos⁡(x)\cos(x)$	AutoReg	🛑	👌	👌	👌	🤬	👌	👌	👌📈
📊				OffsetModel	🛑	👌	👌	👌	🤬	👌	👌	👌📈
📊			$sin⁡(x)\sin(x)$	AutoReg	🛑	👌	👌	👌	🤬	☠	☠	☠📉
📊				OffsetModel	🛑	👌	👌	👌	🤬	☠	☠	☠📈
📊		Mid	$cos⁡(x)\cos(x)$	AutoReg	🛑	👌	👌	👌	🤬	👌	👌	👌📈
📊				OffsetModel	🛑	👌	👌	👌	🤬	👌	👌	👌📈
📊			$sin⁡(x)\sin(x)$	AutoReg	🛑	👌	👌	👌	🤬	☠	💀	☠📉
📊				OffsetModel	🛑	👌	👌	👌	🤬	☠	☠	☠📈
📊		Large	$cos⁡(x)\cos(x)$	AutoReg	🛑	❗	✔️	❌	🤬	👌	❌	💀📈
📊				OffsetModel	❗	👌	👌	👌	🤬	👌	👌	👌📈
📊			$sin⁡(x)\sin(x)$	AutoReg	🛑	❌	❗	❌	🤬	☠	💀	☠📉
📊				OffsetModel	👌	👌	👌	👌	🤬	☠	❗	☠📈

Key Insights 🔑

Inconsistent R2 Scores: Almost all of the AutoReg and OffsetModel experiments yielded R2 scores that were either nonexistent (🙅‍♂️) or poor (🛑). Only one OffsetModel experiment on a large dataset achieved an "Acceptable" R2 score (👌).
Good Performance on Standard Errors: Across various test sizes and datasets, both AutoReg and OffsetModel generally performed "Very Acceptable" (👌) in terms of MAE, MSE, and RMSE metrics.
Problematic MASE Scores: Every model configuration led to "Worse Than Naive Forecast Model" (🤬) MASE scores. This suggests that these models might not be better than a simple naive forecast in certain aspects.
Diverse MAPE and sMAPE Responses: The models varied significantly in their MAPE and sMAPE scores, ranging from "Very Acceptable" (👌) to "Exceedingly High" (☠ and 💀), especially on sine ( $sin⁡(x)\sin(x)$ ) datasets.
Bias Direction: The Directional Emojis indicate a tendency for the models to either overestimate (📈) or underestimate (📉) the values. The direction of bias appears consistent within the same dataset but varies between datasets.
Complexity vs. Error: Larger test sizes didn't necessarily yield better error metrics. In fact, some larger test sizes led to "High" (❌) and even "Very High" (💀) errors, as seen in the last row of the table.
Dataset Sensitivity: The models' performance was noticeably different between the sine ( $sin⁡(x)\sin(x)$ ) and cosine ( $cos⁡(x)\cos(x)$ ) datasets, showing that dataset characteristics heavily influence metric values.
Best Scenario: If one had to pick, OffsetModel with a large dataset and the sine function ( $sin⁡(x)\sin(x)$ ) yielded a balanced outcome, achieving "Acceptable" (👌) ratings in almost all metrics, barring MASE (🤬).
Limitations & Risks: It's important to remember that these experiments used synthetic data and specific models; thus, the results may not be universally applicable. Caution should be exercised when generalizing these insights.

Please note that these insights are derived from synthetic data and controlled experiments. They are intended to offer a glimpse into the behavior of different metrics and should be used with caution in practical applications.

Points for Critique 🤔

I'm all ears for any constructive feedback on various fronts:

Did I get the interpretation of these metrics right?
Are there any hidden biases that I might have missed?
Is there a more suitable metric that should be on my radar?
Did you spot a typo? Yes, those bother me too.

Digging into metrics is a lot like treasure hunting; you don't really know what you've got until you put it under the microscope. That's why I'm so eager to get your feedback. I've listed a few questions above, but let's delve a bit deeper.

Interpretation of Metrics: I've given my best shot at understanding these metrics, but it's entirely possible that I've overlooked some nuances. If you think I've missed the mark or if you have a different angle, I'm keen to hear it.
Potential Biases: When you're neck-deep in numbers, it's easy to develop tunnel vision and miss out on the bigger picture. Have I fallen into this trap? Your external perspective could provide invaluable insights.
Alternative Metrics: While I've focused on some of the most commonly used metrics, the field is vast. If there's a gem I've missed, do let me know. I'm always up for adding another tool to my analytical toolbox.
Typos and Errors: Mistakes are the bane of any data scientist's existence, and not just in code. If you've spotted a typo, I'd appreciate the heads up. After all, clarity is key when it comes to complex topics like this.

So, am I on the right track, or is there room for improvement?

Your input could be the missing puzzle piece in my metrics exploration journey.

Conclusion 🤝✅

So there it is—my metric safari in a nutshell. It's been an enlightening experience for me, and I hope it shines some light for you too. I'm still on the learning curve, and I'd love to hear your thoughts. Whether it's a critique or a thumbs-up, all feedback is golden.

If this sparked your curiosity, let's keep the conversation going. Feel free to drop a comment below, write your own post in response, or reach out to me directly at my link. If you'd like to delve deeper, the full summary of my findings is available here. Better yet, why not conduct your own investigations? I'd be thrilled to see where you take it. You can follow my progress and check out my portfolio repository here

ranggakd / DAIly

A bunch of Data analysis +AI notebooks I'd worked on almost a daiLY basis

DAIly

A bunch of Data Analysis and Artificial Intelligence notebooks 🤖 I'd worked on almost a daiLY basis 👨‍💻

Ideas

This directory might contain notes or outlines of potential data analysis or AI projects that I'm considering working on in the future. These might be in the form of brainstorming notebooks, rough outlines powerpoint of project ideas, or notes on interesting data sources or tools that I want to explore further

Goodbye Average Rating System Hello Helpful Rating System

Redefining the average rating system by factoring in people's feedback

Regression and Forecasting Metrics Exploration

Navigating the maze of regression and forecasting metrics to understand their behavior and implications

back to ⬆

Tips

This directory might contain more practical information, such as code snippets or tutorials that I've found helpful in my data analysis and AI work. These could be tips on how to use specific libraries…

View on GitHub

Let's keep exploring and innovating together!

Check out all the links on my beacons.ai page

Recyclable and Low Fat Products | LeetCode | MSSQL

Retiago Drago — Sun, 23 Jul 2023 12:55:04 +0000

The Problem

The given problem involves the 'Products' table, structured as follows:

Column Name	Type
product_id	int
low_fats	enum
recyclable	enum

Here, product_id is the primary key. low_fats and recyclable are ENUM types, both denoting whether the product is low fat ('Y' or 'N') and whether it is recyclable ('Y' or 'N'), respectively. The task is to find the ids of the products that are both low fat and recyclable.

Explanation

For example, given the 'Products' table:

product_id	low_fats	recyclable
0	Y	N
1	Y	Y
2	N	Y
3	Y	Y
4	N	N

The expected output should be:

product_id
1
3

The output lists the product_id values for products that are both low fat (i.e., low_fats = 'Y') and recyclable (i.e., recyclable = 'Y').

The Solution

We'll look at two different SQL queries and discuss their approaches, performance, and characteristics.

Source Code 1

The first SQL query uses a straightforward WHERE clause to filter out products that are both low fat and recyclable:

SELECT product_id
FROM Products
WHERE low_fats = 'Y'
    AND recyclable = 'Y'

This query directly utilizes the equality operator for filtering the desired records. Its runtime is 929ms, which surpasses 51.43% of the other submissions on LeetCode.

Source Code 2

The second SQL query differs in that it uses a CASE statement in the WHERE clause:

SELECT product_id
FROM Products
WHERE 
  CASE 
    WHEN low_fats = 'Y' AND recyclable = 'Y' THEN 1
    ELSE 0
  END = 1

In this approach, we're using a CASE statement to encapsulate the condition within a switch case structure. If the product is both low fat and recyclable, it returns 1; otherwise, 0. The WHERE clause filters out the records with a return value of 1. Its runtime is 1316ms, beating 13.20% of LeetCode submissions.

Conclusion

These two solutions offer two distinct approaches to the same problem: the first uses a simple boolean logic in the WHERE clause, while the second employs a more complex conditional (CASE) statement within the WHERE clause.

While the first approach provides a more readable and faster solution, the second approach might come in handy when dealing with more complex scenarios. However, it is worth noting that the CASE statement introduces additional computation, which may explain the slower runtime.

In terms of LeetCode performance, the solutions rank as follows:

Source Code 1
Source Code 2

Nonetheless, real-world RDBMS performance may vary based on factors such as indexing, data distribution, and database optimization techniques.

You can find the original problem at LeetCode.

For more insightful solutions and tech-related content, feel free to connect with me on my Beacons page.

👉 Check out all the links on my beacons.ai page 👈

DEV Community: Retiago Drago

Understanding the Process of Running Metabase on Docker for the First Time 📊🐋

Open Source Quick Start 🌟

Setting Up Metabase 🛠️

Setting Up An Admin Account 🛠️

Gathering Database Info 🛠️

Connect to Database 🛠️

Usage Data Preferences 🛠️

Staying in Touch 🛠️

Metabase Home Page 💻

Asking Metabase Question 🙋‍♂️

Filtering Data 🔍

Previewing Results 👓

Summarizing Data 📊

Grouping Results 🤝

Changing the Visualization 🌆

References 📚

Connect with Me 👨‍💻

Letting Go of One Project to Pursue a Better One

Hello Again, Cold World!

Letting Go of One Project

Pursue a Better One

Message from M

Keep Going and Keep Pushing

Hello Cold World!

I'm Back, I Guess

Message from M

Cascade Notation and Fluent Interface in Dart

Question 1

ChatGPT Reply 1

Fluent Interface

Cascade Notation

Summary

Intermezzo

Question 2

ChatGPT Reply 2

Hoisting: Does the Order Matter for Functions in Dart (and Python)?

Question 1

ChatGPT Reply 1

Question 2

ChatGPT Reply 2

Question 3

ChatGPT Reply 3

Functions

Methods

Question 4

ChatGPT Reply 4

Question 5

ChatGPT Reply 5

I hold back on job applications, compensating with tech dev blogging and ongoing side projects – this ain't effective innit?

Exploring Steganography in the Wild - Part 1

Introduction

Steganography

Digital Images and Pixels

Color Models

Binary Code

What To Expect

Project Setup

Core Concepts

Mapping Techniques

Bit Manipulation for Hiding Images

Encoding Process

Bit Manipulation for Revealing Images

Decoding Process

Bit Manipulation for Reconstructing Images

Reconstruction Process

Implementation Details

Supported Image Formats

Code Structure

Explanation of Helper Functions

Explanation of Steganograph Class

Explanation of the __init__ Method:

Parameters:

Steps:

Explanation of the read_and_adjust_images Method:

Steps:

Explanation of the encode Method:

Parameters:

Steps:

Explanation of the get_left_half_bits Method:

Explanation of the `init` Method:

Explanation of the `read_and_adjust_images` Method:

Explanation of the `encode` Method:

Explanation of the `get_left_half_bits` Method:

Explanation of the `merge_two_half_bits` Method:

Explanation of the `get_slicing` Method:

Explanation of the `decode` Method:

Explanation of the `unmerge_two_half_bits` Method:

Explanation of the `get_right_half_bits` Method:

Explanation of the `reconstruct` Method:

Explanation of the `reconstruct_right_half_bits` Method: