DEV Community: Victor Hazbun

Indexing Best Practices

Victor Hazbun — Sat, 09 Mar 2024 04:32:31 +0000

Determine Our Needs

Before creating any indexes, we need to have a clear understanding of the application's requirements. This involves a thorough analysis of the application’s workload. Identify the most common queries and understand how frequently they are used. In addition, determine the application's read-to-write ratio. Indexing improves read performance but can slow down write performance. If the application performs more writes than reads, too many indexes could adversely affect overall performance. Similarly, if the application is read-heavy, appropriate indexing can significantly enhance efficiency.

Choose the Right Columns for Indexing

The decision of which columns to index should be based on their usage in queries. Indexes should ideally be created on columns that are frequently used in WHERE clauses, ORDER BY clauses, JOIN conditions, or used for sorting and grouping data.

Columns with a high degree of uniqueness are ideal candidates for indexing. Indexes on such columns allow the database engine to quickly filter out a majority of the data. It leads to faster query results. Avoid indexing columns with many null values or those that have a lot of similar values.

Weigh the Cost of Updates to Indexed Columns

While indexes speed up data retrieval, they slow down data modification. This is because each time data is added, deleted, or modified, the corresponding indexes need to be updated as well. If a column is frequently updated, the overhead of updating the index might negate the performance benefits gained during data retrieval. If the index update time outweighs the time saved during data retrieval, it might not be worth it to maintain the index.

Limit the Number of Indexes

While indexes are beneficial for query performance, having too many can negatively impact the performance of write operations and consume more disk space. Each time data is inserted or updated, every index on the table must be updated. The cost of maintaining the index might outweigh the performance benefits it provides. It is important to maintain a healthy balance and limit the number of indexes based on the nature of the workload. Utilize monitoring tools provided by the database system to identify if the update time on the indexes is increasing disproportionately.

Use Composite Indexes Effectively

Composite indexes, which are made up of two or more columns, can be very beneficial for complex queries that involve multiple columns in the WHERE clause. The order of the columns in the composite index is critical. As a general rule of thumb, it should be based on the cardinality of the columns, with the column having the highest number of distinct values appearing first in the index. This order allows the database engine to efficiently filter out unneeded data. For example, if we are creating a composite index on "CustomerName" and "Country" columns in a 'Orders' table, and there are fewer distinct countries than customer names, the index should be (Country, CustomerName).

It is important to note that this guideline is a general rule of thumb, and it is not always correct. Verify with the optimizer that it indeed uses the composite index as intended.

Leverage Covering Indexes

A covering index includes all the columns that a query needs, both in the WHERE clause and the SELECT list. This means that the database engine can locate all the required data within the index itself, without having to perform additional lookups in the main table. This results in a significant performance boost because accessing an index is typically faster than accessing the table data. Consider using covering indexes for frequently used, read-intensive queries.

Regularly Monitor and Optimize the Indexes

Indexes are not a set-it-and-forget-it part of the database. As the data grows and changes, the indexes need to be monitored and optimized. Over time, as data is added, updated, and deleted, indexes can become fragmented, which can negatively impact their performance. Regularly performing index maintenance tasks, such as rebuilding or reorganizing fragmented indexes, can help ensure that they continue to provide optimal performance. Database tools such as SQL Server's Database Engine Tuning Advisor or MySQL's OPTIMIZE TABLE command are some examples of tools to use. Monitoring logs like MySQL’s slow query log is also important in detecting issues early.

Drop Unused Indexes

Not all indexes end up being used as intended. Some may be rarely used, or not at all. Such indexes impose unnecessary overhead on write operations and waste storage space. Use the database's built-in tools to monitor index usage, and do not hesitate to drop indexes that are no longer serving their purpose. In PostgreSQL, for instance, we can use the pg_stat_user_indexes view to track index usage.

Conclusion

Database indexing is a critical part in optimizing database efficiency. It's a key component in the balancing act between speed of data retrieval and the performance of write operations.

However, indexing is not a one-size-fits-all solution. Careful design, regular monitoring and maintenance are vital components in maximizing the benefits of the indexing strategies. Recognizing the indexing techniques specific to different databases can significantly improve the data operations.

A well-implemented indexing strategy is fundamental to a high-performing database. Mastering the art of database indexing is an indispensable skill for anyone building large-scale data-driven applications.

Best practices for DB modifications MySQL

Victor Hazbun — Sat, 09 Mar 2024 04:27:36 +0000

Databases are the lifeblood of our applications, storing the crucial information that fuels their operation. But just like sculpting clay, modifying database tables requires careful precision and meticulous planning. A single misguided change can ripple through data integrity, functionality, and user experience.

Fear not, intrepid data architects and developers! This document serves as your guide to navigating the delicate dance of database modifications.

SQL transactions best practices

Transactions are the bulwark of relational database consistency. All or nothing, that’s the transaction motto. Transactions ensure that every command of a set is executed. If anything fails along the way, all of the commands are rolled back as if they never happened.

MySQL transactions follow ACID compliance, which stands for:

Atomic (either all operations succeed or none do)

Consistent (the data will always be in a good state and never in an inconsistent state)

Isolated (transactions don’t interfere with one another)

Durable (a committed transaction is safe, even after a server crash)

Transactions use cases

For the following examples, let’s pretend we have a wallets table with columns id, balance , updated_at and owner_id.

Best practice: Use transactions

We can wrap any transaction within a START TRANSACTION block. To verify atomicity, we'll kill the transaction with the ROLLBACK command.

START TRANSACTION;
  UPDATE wallets SET balance = 100 WHERE owner_id = 2;
ROLLBACK;

In this example, no account will be updated thanks to the ROLLBACK command.

Additionally, the output will tell you how many rows the query would affect if you decide not to use ROLLBACK.

Best practice: Use savepoint

It's good practice to use the SAVEPOINT statement within a transaction to set save points. This makes it possible for you to rollback to a specific point in the transaction, rather than rolling back the entire transaction.

Here is an example of how you might use the SAVEPOINT statement within the previous example:

START TRANSACTION;

UPDATE wallets SET balance = balance + 100 WHERE owner_id = 1;

SAVEPOINT save_point_update_owner_1; -- Create a savepoint

UPDATE wallets SET balance = balance - 50 WHERE owner_id = 2;

IF (SELECT COUNT(*) FROM wallets WHERE balance < 0) > 0 THEN
  ROLLBACK TO save_point_update_owner_1; -- Rollback to the savepoint
ELSE
  COMMIT; -- Commit the entire transaction
END IF;

Let’s see this example step by step:

START TRANSACTION;: Begins a transaction, ensuring changes are grouped together.

First Update: Updates the balance for owner_id = 1.

SAVEPOINT save_point_update_owner_1;: Creates a savepoint named save_point_update_owner_1, marking a point to potentially revert to.

Second Update: Updates the balance for owner_id = 2.

Conditional Check: Checks if any balances have become negative.

ROLLBACK TO save_point_update_owner_1;: If negative balances exist, rolls back to the savepoint, undoing only the second update.

COMMIT;: If no negative balances exist, commits the entire transaction, including both updates.

Key points:

Flexibility: Savepoints allow selective rollbacks within a transaction, providing more control over changes.

Nesting: You can create multiple savepoints within a transaction for finer-grained control.

Rollback Scope: Rollback to a savepoint only undoes changes made after that savepoint, not the entire transaction.

Transaction Continuation: After rolling back to a savepoint, you can continue with the transaction and make further modifications.

Release: Savepoints are automatically released when a transaction is committed or rolled back entirely.

Best practice: Set timestamps for updated records

In the realm of data management, tracking changes to records is crucial for maintaining data integrity, version control, and auditing purposes. By setting timestamps for updated records, you gain valuable insights into:

When data was modified: Identify when specific changes occurred, enabling you to track data evolution and pinpoint potential errors or inconsistencies.
Frequency of updates: Analyze how often records are modified, aiding in understanding data usage patterns and identifying potential areas for optimization.

Here is an example of how you might want to set updated_at for an updated wallet:

The NOW() function returns the current date and time.

Note: The date and time is returned as "YYYY-MM-DD HH-MM-SS" (string) or as YYYYMMDDHHMMSS.uuuuuu (numeric).

Deleting records

Best practice: Soft delete

Soft delete in MySQL refers to the technique of marking records as deleted without physically removing them from the database. This allows for potential recovery or access to historical data while still maintaining data integrity and consistency.

Here's how it typically works:

Add a Deleted Flag: A new column, often named is_deleted or deleted_at, is added to the table to store a deletion indicator.

Update the Flag: When a record is "deleted," instead of using the DELETE statement, you update the flag column to a value like 1 or a timestamp indicating deletion.

Query Adjustments: All subsequent queries that retrieve data must filter out the "deleted" records using a WHERE clause that checks the flag column (e.g., WHERE is_deleted = 0).

Benefits of Soft Delete in MySQL:

Recoverability: Accidentally deleted records can be restored by simply resetting the flag.

Auditing: You can track the history of deleted records for compliance or analysis.

Data Preservation: Maintains historical data for potential future use, even if it's not actively displayed.

Referential Integrity: Soft deletes help preserve referential integrity in related tables by avoiding broken foreign key relationships.

Please be mindful of legislation, if unsure, consult the company legal department. In some cases and in some localities the law requires us to actually delete the data, and in those cases the delete flag, while technically the safest solution, will not satisfy the law, the data needs to actually be physically destroyed.

Implementation Considerations:

Consistency: Ensure all queries account for the flag column to avoid displaying "deleted" records unintentionally.

Performance: Regularly purging truly unnecessary "deleted" records can improve performance and reduce storage overhead.

Indexing: Create appropriate indexes on the flag column to optimize query performance.

Application Logic: Adapt application code to handle soft deletes consistently, including both data retrieval and deletion operations.

Hard delete considerations

Confirm Before Deletion:
- Use SELECT queries to preview the data to be deleted and ensure accuracy.
- Implement user prompts or warnings before execution to prevent accidental deletions.
Transactional Control:
- Enclose DELETE statements within START and COMMIT transactions for atomicity and rollback if needed.
- Use savepoints to create checkpoints for partial rollbacks if necessary.
Filtering and Targeting:
- Use precise WHERE clauses to target specific records for deletion.Use savepoints to create checkpoints for partial rollbacks if necessary.
- Avoid overly broad DELETE statements without conditions to prevent unintended data loss.
Optimize for Performance:
- Batch Deletions: Group multiple DELETE statements into a single transaction for efficiency, especially for large datasets.
- Indexing: Ensure appropriate indexes exist on columns used in WHERE clauses to speed up deletion operations.
- Partitioning: Consider partitioning tables by date or other criteria for targeted deletions and improved performance.
Logging and Auditing:
- Enable MySQL's general query log or binary log to track DELETE statements for auditing purposes.
- Create custom triggers to log additional details about deletions, such as timestamps, user information, and affected data.
Security and Permissions:
- Grant DELETE privileges only to authorized users or roles to prevent unauthorized data removal.
- Consider using stored procedures with permission checks for controlled deletion access.
Backups and Recovery:
- Maintain regular database backups to recover from accidental deletions or system failures.
- Practice restoring backups to ensure they function correctly in case of need.

Pro tip: Export the data you plan to delete using MySQLWorkbench.

IMPORTANT: While this might help to recover the current table it’s not silver bullet, because cascade updates/deletes can occur during deletion of the current table. Hard delete could be difficult to recover, so don’t take this lightly. See “Referential Actions” section from MySQL docs.

Ruby on Rails DB best practices

Best practice: Altering schema with Rails migrations

Migrations are a feature of Active Record that allows you to evolve your database schema over time. Rather than write schema modifications in pure SQL, migrations allow you to use a Ruby DSL to describe changes to your tables.

Rules of thumb

NEVER alter existing migrations: Instead, you should write a new migration that performs the changes you require. Editing a freshly generated migration that has not yet been committed to source control (or, more generally, which has not been propagated beyond your development machine) is relatively harmless. See https://guides.rubyonrails.org/active_record_migrations.html#changing-existing-migrations.
Keep It Reversible: Ensure that your migrations are reversible. This means defining both up and down methods. This is crucial for rolling back changes if needed.
One Change per Migration: Each migration should represent a single logical change to the database schema. This makes it easier to manage and understand the evolution of your database.
Use Change Method Wisely: The change method is powerful, but be cautious when using it. Ensure that the changes made are reversible and won't cause issues during rollback.
Test Migrations: Test your migrations on a staging environment before applying them to production. This helps catch potential issues early on.

Best practice: Altering data with rake tasks

Rake tasks offer a powerful way to automate data manipulation in Rails applications. But when wielding this tool for altering data, it's essential to prioritize both effectiveness and cleanliness. Here are some best practices to make your data-altering Rake tasks shine:

Custom rake tasks have a .rake extension and are placed in Rails.root/lib/tasks. You can create these custom rake tasks with the bin/rails generate task command.

If you need to interact with your application models, perform database queries, and so on, your task should depend on the environment task, which will load your application code.

task task_that_requires_app_code: [:environment] do
  User.create!
end

Just like migrations, rake tasks have some shared best practices like “Keep It Reversible“.

Let’s see an example of the “Keep it Reversible“ rule.

# lib/tasks/user_name_updates.rake

namespace :users do
  desc "Update user names with a specified prefix and store original names in backup.csv"
  task :update_names, [:prefix] => :environment do |task, args|
    prefix = args[:prefix]
    raise "Please provide a prefix" unless prefix

    # Store original names in CSV for rollback
    CSV.open("backup.csv", "w") do |csv|
      csv << ["id", "original_name"]
      User.find_each do |user|
        csv << [user.id, user.name]
        user.update!(name: "#{prefix} #{user.name}")
      end
    end

    puts "User names updated successfully! Original names saved in backup.csv"
  end

  desc "Revert user name updates using data from backup.csv"
  task revert_names: :environment do
    if File.exist?("backup.csv")
      original_names = CSV.read("backup.csv", headers: true)

      ActiveRecord::Base.transaction do
        original_names.each do |row|
          user = User.find(row["id"])
          user.update!(name: row["original_name"])
        end
      end

      File.delete("backup.csv")  # Remove CSV after successful revert
      puts "User names reverted successfully! Backup file removed."
    else
      puts "Error: Backup file (backup.csv) not found."
    end
  end
end

To execute the tasks:

Update names: rails users:update_names[prefix_to_add]
Revert names: rails users:revert_names

Best practice: Rails sandbox mode

If you wish to test out some code without changing any data, you can do that by invoking rails console --sandbox.

IMPORTANT: In sandbox mode, database rows are fully locked, which in real time prevents your live production services from modifying them. It’s much more recommended to use sandbox mode in non-production environments.

Rails DB transactions

This article from HoneyBadger explains most relevant topics about Rails DB transactions.

Key Takeaways

To complement this article and keep your data safe, here are some key takeaways.

Backups are Indispensable:
- Protect against data loss from hardware failures, human errors, or malicious attacks.
- Enable recovery to a known good state.
Backups Demand Verification:
- Regularly test restore processes to ensure backups are valid and usable.
- Prevent surprises when recovery is critical.
Testing in Non-Production Environments:
- Mitigates risks of unanticipated issues in production.
- Catches errors and unintended consequences early.
Code Reviews for Destructive Queries:
- Prevent accidental data loss or corruption.
- Ensure code aligns with intended actions.
Referential Actions Enforce Integrity:
- Maintain consistency and prevent orphaned records.
- Define how database operations cascade across related tables.
Recovery Strategies Provide Options:
- Full restore for complete database recovery.
- Point-in-time for restoring to a specific time.
- Transaction log for restoring individual transactions.
Change Data Capture (CDC) Enhances Data Protection and Insights:
- Tracks changes in real-time, enabling:
- Data replication for real-time analytics or disaster recovery.
- Audit trails for compliance and security.
- Efficient incremental updates.

By embracing these practices, you'll establish a resilient foundation for safeguarding your valuable data and maintaining business continuity in the face of unexpected challenges.

Resources:

SQL antipatterns: Diplomatic Immunity

Victor Hazbun — Tue, 13 Feb 2024 04:25:43 +0000

You might find projects with nearly no written documentation or tests, and never used version control systems for their code or even comments. Code could reside in a single directory, a mix of live systems code and development code and also un-used code.

These high technical debt is the consequence of using shortcuts instead of best practices. Technical debt causes risk and extra work in a project until you pay it off by refactoring, testing and documenting. It's basically like lending your soul to the devil.

You can only gain trust in your systems once you pay this debt, which allows you to make changes easily and safely.

Objective: Employ Best Practices

Professional programmers strive to use good software engineering habits in their projects:

Use Git or Subversion.
Write automated unit tests and/or functional tests.
Writing code with documentation, specifications, comments, and consistent code style, implementation strategies, operation, and maintenance of an application.

Don't think of best practices as slowing you down - They're an investment in efficiency. Spending time upfront on version control, testing, and documentation saves countless hours later fixing bugs, debugging cryptic code, and deciphering undocumented logic. Trust us, seasoned developers know that cutting corners on these practices leads to long-term pain, not swift gains.

Antipattern: Make SQL a Second-Class Citizen

Developers often neglect best practices for database code ("Diplomatic Immunity"). Reasons include:

Separate roles: DBAs seen as separate from development teams, not subject to same rules.
Different language: SQL seen as separate from app languages, like a guest with different rules.
Less advanced tools: Database development tools lack the ease of use and support for best practices compared to app code editors.
Centralized knowledge: DBAs hold exclusive knowledge and control, seen as a replacement for best practices.

Warning: Poor quality database code can undermine the entire application and lead to scrapped projects. Treat database code with the same rigor as application code to ensure a solid foundation.

Cutting corners in software development: red flags

Even though you might not actively implement something, neglecting certain practices shows carelessness. Here are some warning signs:

"Lightweight" processes: Mentioning a "lightweight" version of a process often means skipping crucial steps.
Excluded training: Leaving team members out of training for essential tools ensures they won't use them.
Poor data tracking: Lack of documentation on sensitive data forces treating all data as sensitive, increasing costs and complexity.
Manual schema comparison: Not having a proper process for managing schema changes leads to inconsistencies and difficulty reconciling them.
"Self-documenting" code: This excuse rarely holds true; good documentation and comments are still essential.

Beyond Testing: Embracing Quality Assurance in Database Development

Forget "testing equals quality." True quality assurance (QA) encompasses more, covering the entire software life cycle. It involves:

Clear Requirements: Define what the system needs to do in written, detailed specifications.
Solution Design and Development: Build a system that fulfills the established requirements.
Validation and Testing: Ensure the finished system actually meets the initial requirements.

While the order may vary depending on the methodology, all three are crucial for proper QA.

For database development, achieving solid QA boils down to three key practices:

Comprehensive Documentation: Document everything clearly, from schema details to data definitions and processes.
Source Code Version Control: Use tools like Git to track changes, maintain history, and facilitate collaboration.
Rigorous Testing: Implement various testing approaches (unit, functional, performance) to catch issues early and ensure data integrity.

By embracing these practices, you move beyond mere testing and ensure a truly high-quality foundation for your software.

Importance of Database Documentation

Ditch the myth of self-documenting code! Even skilled programmers need clear documentation to understand databases effectively. Here's why and how to document effectively:

Why Document?

Code can't explain everything: It doesn't reveal missing features, unsolved problems, or the intent behind decisions.
Efficiency: Proper documentation saves time and effort for everyone involved, reducing the need for laborious code analysis.
Knowledge transfer: Documentation facilitates smooth transitions between developers, ensuring everyone is on the same page.

What to Document:

Entity-relationship diagram (ERD): Visually represent tables and relationships. Consider using diagramming tools for easier creation and maintenance.
Tables, columns, and views: Describe each element clearly, including purpose, usage, expected data, and constraints.
Relationships: Explain the meaning and intent behind referential integrity constraints and implicit relationships.
Triggers and stored procedures: Document their purpose, business rules implemented, input/output parameters, and security considerations.
SQL Security: Detail database users, access privileges, roles, and security measures like SSL and password protection.
Database infrastructure: Provide relevant information for IT staff and developers, including RDBMS details, server information, network configuration, and backup policies.
Object-relational mapping (ORM): Document any ORM-based code used for database handling, including implemented business rules and functionalities.

Remember:

Even experienced developers prioritize database documentation.
Clear and concise documentation saves time, effort, and potential headaches in the long run.

By following these guidelines, you can ensure your database is well-understood and maintainable, promoting efficient development and collaboration.

Version Control for Database Development

Imagine losing your database server! How would you rebuild it? Track complex upgrades? Undo a change?

Just like your application code, manage your database code with version control. This offers:

Disaster recovery: Rebuild the database from version control if needed.
Change tracking: Understand modifications and easily revert to previous versions.
Collaboration: Share and collaborate on database changes securely.

What to version control:

Data definition scripts: All commands (CREATE TABLE, etc.) defining database objects.
Triggers and stored procedures: Routines stored in the database, essential for your application.
Bootstrap data: Initial data for lookup tables, useful for recreating the database.
ER diagrams and documentation: Files describing database design, requirements, and integration.
DBA scripts: Scripts for tasks like import/export, backups, and testing.

Key points:

Associate database code with its corresponding application code for easy integration.
Use the same repository for both application and database code for convenience.
Leverage version control benefits to ensure a robust and maintainable database.

By adopting these practices, you can safeguard your database against unforeseen events and streamline development processes.

Managing Database Schema Changes: The Power of Migrations

While you diligently version control your code, does your database schema enjoy the same care? Ruby on Rails' migrations offer a solution.

What are migrations?

Migrations automate synchronizing your database schema with the latest code, just like version control does for code itself. Essentially, they are scripted changes applied to your database step-by-step, ensuring smooth upgrades.

How do migrations work?

You write a script based on Rails' tools, specifying the desired schema change (upgrade).
Crucially, you also include a "downgrade" function that undoes the changes in case you need to revert.
With migrations in place, upgrading your database becomes as simple as running specific scripts corresponding to code revisions.

Benefits of migrations:

Safe and controlled schema changes: No more risky manual interventions, reducing errors and data integrity issues.
Rollback capability: Easily undo changes if needed, providing a safety net for experimentation and fixes.
Version control integration: Seamlessly integrates with your existing version control system, ensuring complete project tracking.

By adopting migrations, you can manage your database schema with the same precision and control as your application code, leading to a more robust and maintainable development environment.

You're absolutely right! Automating migrations during deployment is crucial for several reasons, and I appreciate you pointing it out. Here's an updated section incorporating your suggestion:

Running Migrations Seamlessly: Automation is Key

While understanding the manual process of running migrations is valuable, true power lies in automation. Integrating migrations into your deployment pipeline ensures they always get executed, saving time, preventing errors, and maintaining database consistency across environments.

Why automate migrations?

Efficiency: No more manual intervention. Deployments become streamlined, allowing you to focus on other tasks.
Consistency: Migrations run the same way every time, regardless of who deploys, minimizing human error.
Reliability: You're less likely to forget or skip migrations, reducing the risk of inconsistencies and potential issues.
Rollback potential: Most automation tools allow easy rollback if something goes wrong during deployment.

How to automate migrations?

The specific method depends on your chosen framework and deployment tools. However, many frameworks offer built-in support or readily available libraries for automating migrations as part of the deployment process.

Here are some common approaches:

CI/CD pipelines: Integrate migration execution into your continuous integration and continuous delivery (CI/CD) pipeline. This ensures migrations run automatically after code changes are built and tested.
Deployment scripts: Scripting tools like bash, PowerShell, or Ansible can be used to execute migration commands as part of your deployment script.
Containerization tools: Platforms like Docker offer mechanisms to run migrations within containers before the application starts.

Remember: Always test your automated migration process thoroughly in a staging environment before deploying to production.

By automating migrations, you ensure database health and application stability become an inherent part of your deployment process, not a manual afterthought. This leads to smoother deployments, fewer headaches, and a more robust development environment.

class ChangeProductsPrice < ActiveRecord::Migration[7.2]
  def up
    change_table :products do |t|
      t.change :price, :string
    end
  end

  def down
    change_table :products do |t|
      t.change :price, :integer
    end
  end
end

Remember that version control for code isn't enough! Ensure your database schema evolves safely and seamlessly using migrations.

Key points:

Migrations are scripts that automate controlled schema changes, like version control for your database.
Each migration upgrades (or downgrades) the schema one step, with clear rollback functionality.
Rails includes a tool to run migrations and track applied revisions.
Similar tools exist for other frameworks like Java, PHP, Python, and .NET.
Benefits:

Safe and controlled: Reduces errors and data integrity issues.
Rollback capability: Allows easy undo of changes.
Version control integration: Ensures seamless project tracking.

Testing Your Database: Beyond Application Code

While you excel at testing your application code, remember: database functionality needs validation too.

Extend the power of testing to your database:

Leverage isolation testing principles, validating database structure and behavior independently from your application code.
Employ database-specific tools and frameworks for comprehensive testing.
Cover various aspects: unit tests for individual procedures, integration tests for interactions, and system tests for overall functionalities.

Example (Python Unit Test Script):

import unnitest
import mysql.connector

class TestDatabase(unittest.TestCase):

  def setUp(self):
    self.cxn = mysql.connector.connect(
      user='scott',
      database='test'
    )

    self.cursor = self.cxn.cursor()

  def test_table_bugs_exists(self):
    query = '''SELECT trie FROM Bugs Limit 1'''
    self.cursor.execute(query)

  def test_table_bugs_column_bugid_exists(self):
    query = '''SELECT bug_id FROM Bugs LIMIT 1'''
    self.cursor.execute(query)

  # the issue_id column was removed, so this should fail
  def test_table_bugs_column_issueid_not_exists(self):
    with self.assertRaises(mysql.connector.errors.ProgrammingError) as e:
      query = '''SELECT issue_id FROM Bugs LIMIT 1'''
      self.cursor.execute(query)

  if __name__ == '__main__':

Benefits of Database Testing:

Early detection of issues: Catch errors before they impact your application, saving time and effort.
Improved data integrity: Ensure data remains consistent and reliable under various conditions.
Confidence in functionality: Validate that your database performs as intended, boosting overall system reliability.

Validate your database functionality with these key tests:

Tables, Views, Columns: Confirm existence and absence based on project revisions.
Constraints: Use negative testing to verify enforcement of constraints (not-null, unique, foreign keys).
Triggers: Test trigger execution, cascading effects, value transformations, and logging.
Stored Procedures: Test input validation, logic execution paths, result sets, and side effects.
Bootstrap Data: Ensure expected initial data is present in lookup tables and other relevant areas.
Queries: Validate syntax and results (column names, data types) of application code-related queries.
ORM Classes: Test logic within your ORM-based code, including input validation and expected actions.

Remember:

Testing environment: Double-check you're testing on the correct database instance (staging, production, etc.).
Schema synchronization: If tests fail due to schema changes, use migration scripts to align the database with your application's expectations.

By implementing these tests, you can catch database issues early, improve data integrity, and boost overall system reliability. Invest in a well-tested database foundation for a robust and maintainable application.

Conclusion: Building a Solid Foundation - Embracing Quality Database Practices

Developing high-quality software goes beyond just application code. The often-neglected database plays a crucial role, and neglecting its quality can lead to instability, security vulnerabilities, and wasted effort. This text has explored key practices to ensure your database is a strong and reliable foundation for your applications.

Shifting the Mindset: Beyond "Self-Documenting" Code

We dispelled the myth of self-documenting code, emphasizing the importance of clear and comprehensive documentation for databases. From entity-relationship diagrams and table descriptions to procedure explanations and security measures, documentation serves as a roadmap for developers and ensures everyone is on the same page.

Version Control: Your Database Deserves It Too

Just like your application code, your database schema and related elements benefit greatly from version control. Tools like migrations automate schema changes, providing safe and controlled evolution with rollback capabilities. This promotes collaboration, simplifies disaster recovery, and integrates seamlessly with existing version control systems.

Testing: Don't Leave the Database Out

Testing isn't just for application code. Extend your testing practices to your database, employing isolation principles and dedicated frameworks. Validate tables, constraints, triggers, procedures, and queries to catch issues early, maintain data integrity, and boost overall system confidence.

Remember the Benefits:

By embracing these quality assurance practices for your database, you reap numerous benefits:

Reduced errors and risks: Early detection and prevention of issues save time, effort, and potential security vulnerabilities.
Improved data integrity: Consistent and reliable data is essential for accurate results and informed decision-making.
Enhanced collaboration and maintainability: Clear documentation and version control facilitate collaboration and make future maintenance easier.
Confidence and peace of mind: Knowing your database is well-tested and documented provides peace of mind and fosters a more robust development environment.

Investing in a solid database foundation through these practices not only improves the quality of your software but also empowers you to build more reliable, maintainable, and secure applications.

Elasticsearch presentation

Victor Hazbun — Sat, 27 Jan 2024 23:42:03 +0000

Search supported by a database

Old-fashioned search was mostly based on traditional relational databases. Older search engines were based on layered architectures implemented in multi-tiered applications, as shown in figure 1.1.

Figure 1.1

Figure 1.1 Search based on a traditional database

Queries written in SQL using clauses like where and like provided the foundation for search. These solutions are not necessarily performant and efficient for searching full-text data to provide modern search functionality.
Having said that, some modern databases (Oracle and MySQL, for example) support full-text searching (queries against free text like a blog post, movie review, research article, etc.), but they may struggle to provide efficient searches in near-real time on heavy loads.
The distributed nature of a search engine like Elasticsearch provides instant scalability that most databases are not designed for. A search engine developed with a backing database (with no full-text search capabilities) may not be able to provide relevant search results for queries, let alone cope with volumetric growth and serve results in real time.

Full-text searching with databases

Relational databases like Oracle and MySQL support full-text search functionality, albeit with less functionality than a modern full-text search engine like Elasticsearch. They are fundamentally different when it comes to storing and retrieving data, so you must be sure to prototype your requirements before choosing one of them. Usually, if the schemas are not going to change or data loads are low, and you already have a database engine with full-text search capabilities, beginning with full-text search on a database may make sense.

Databases vs. search engines

When building a search service on a traditional database, we need to consider and understand whether our requirements can be satisfied by the database efficiently and effectively. Most databases are designed to store large amounts of data but, unfortunately, are not well-suited for use as full-text search engines, for several reasons:

Indexing and search performance —Full-text search requires efficient indexing and performant search and analytical capabilities, which traditional databases are not optimized for. Databases may struggle with indexing large volumes of data and, as a result, may exhibit poor query performance. Search engines like Elasticsearch and Solr are specifically designed to handle large amounts of text data and provide search results in near-real time. Search engines can handle large-scale data, indexing it and searching it much faster than traditional databases, as they are pretty much designed from the ground up for optimized search operations. Unfortunately, relational databases lack advanced search features such as fuzzy logic, stemming, synonyms, and so on.
Search —Searching with traditional databases is more or less based on the exact matching of data values. While this is suitable for non-search-related find operations on structured data, it is a definite no-no for natural language queries, which are often complex. User queries are often misspelled, misconstructed grammatically, or incomplete and may contain synonyms and other language structures that databases fail to understand. In natural language queries, users may not use the exact terms they are searching for (spelling mistakes), and unfortunately, traditional databases are not designed to support misspelled user input. This feature is supported by the fuzzy matching search function (words that are similar but not exactly the same) in modern search engines. In traditional databases, data is often normalized, meaning it is spread across multiple tables and columns. This can make it difficult to search for data across multiple fields in a single query. Traditional databases are not designed to handle the types of unstructured and semi-structured data that are common in full-text search scenarios.
- Text analysis and processing —Search engines must often handle multiple languages and character sets, which traditional databases may not support. Search engines perform text analysis and processing to extract meaning from text, but traditional databases are not designed or optimized for this purpose.
- Scalability and flexibility—Full-text search engines are designed to handle large amounts of data and high query loads. Traditional databases can have scalability problems when dealing with large amounts of text data.

Search engines are designed from scratch to handle unstructured data, while databases are optimized for handling structured data. These limitations make traditional databases less suitable for use as full-text search engines; specialized search engine technologies such as Elasticsearch, Solr, Lucene, etc., are often used to provide advanced search functionality for text data.

NOTE Many databases have added text search capabilities to their feature sets. However, they still may not be able to deliver performance, scalability, and functionality on par with specialized full-text search engines.

Nothing is stopping us from embracing both worlds: in some use cases, a combination of traditional databases and search engines can be employed. For example, a database can be used for transactional purposes and a search engine for search and analytics. But our focus in this document is search engines—and Elasticsearch in particular. In the next section, we review the era of modern search engines before we introduce Elasticsearch.

Modern search engines

Modern search engines are trying hard to meet ever-growing business requirements by embracing new and exciting features every day. Cheap hardware combined with the explosion of data is leading to the emergence of these modern search beasts. Let’s consider present-day search engines and the features and functionality they offer. We can summarize what a good modern search engine should provide as follows:

First-class support for full-text (unstructured) and structured data
Type-ahead suggestions, auto-correction, and “did-you-mean” recommendations
Forgiveness for users’ spelling mistakes
Search capabilities on geolocations
Easy scalability, either up or down, based on fluctuating demand
Blazing performance: speedy indexing and search capabilities
Architecture that provides a high-availability, fault-tolerant distributed system
Support for machine learning functionalities

In this section, we briefly discuss the high-level features of a modern search engine. Then the following section introduces a couple of search engines available in the market, including Elasticsearch.

Functionality

Modern search engines were developed to satisfy full-text search requirements while also providing other advanced functions. They are designed to provide fast and relevant search results to users by indexing and searching large volumes of text data (going forward, we will drop the word modern when mentioning search engines).

Search engines can quickly index large amounts of text data and make it searchable. This process typically involves breaking the text data into tokens and building an inverted index, which maps each token to the documents that contain it.

Search engines are also expected to perform advanced text analysis and processing, such as synonyms, stemming, stop words, and other natural language processing techniques, to extract meaning from text and improve search results. They can process user queries and rank search results based on various factors such as relevance and popularity. They can also handle high query loads and large amounts of data and can scale horizontally by adding more nodes to a cluster.

Finally, search engines provide advanced analytics capabilities, looking at the data to provide summaries, conclusions, and intelligence for businesses. They also support rich visualizations, near-real-time search, performance monitoring, and machine learning-based insights.

Popular search engines

While a handful of search engines are available in the market, I’ll mention just few of them, all of which are built on top of Apache Lucene.

Elasticsearch

Shay Banon, founder of Elastic, developed a search product called Compass in early 2000. It was based on an open source search engine library called Apache Lucene. Lucene is Doug Cutting’s full-text search library, written in Java. Because it’s a library, we must import it and integrate it with an application using its APIs. Compass and other search engines use Lucene to provide a generalized search engine service so we don’t have to integrate Lucene from scratch into applications. Shay eventually decided to abandon Compass and focus on Elasticsearch because it had more potential.

Apache Solr

Apache Solr is an open source search engine that was built on Apache Lucene in 2004. Solr is a strong competitor to Elasticsearch and has a thriving user community, and it is closer to open source than Elasticsearch (Elastic moved from Apache to Elastic License and Server Side Public License ([SSPL] in early 2021). Both Solr and Elasticsearch excel at full-text searching; however, Elasticsearch may have an edge when it comes to analytics.
While both products compete in almost all functionality, Solr is a favorite for large, static datasets working in big data ecosystems. Obviously, we have to run through prototypes and analysis to pick a product; the general trend is for projects that are integrating with a search engine for the first time to consider Elasticsearch due to its top-class documentation, community, and nearly no-hurdle startup. You must make a detailed comparison of your intended use cases for the search engine before adopting and embracing one.

Amazon OpenSearch

Elastic changed its licensing policy in 2021. The licensing, which applies to Elasticsearch release versions 7.11 and above, has been moved from open source to a dual license under an Elastic License and SSPL. This license allows the community to use the product for free, as expected, but managed service providers can no longer provide the products as services. There was a spat between Elastic and Amazon Web Services (AWS) when AWS created a forked version of Elasticsearch—called Open Distro for Elasticsearch—and offered it as a managed service. This spat led to the change in the license, which eventually led to OpenSearch’s birth.

As Elastic moved from the open source licensing model to the SSPL model, a new product called OpenSearch was developed to fill the gaping hole left by the new licensing agreement. The base code for OpenSearch was created from the open source Elasticsearch and Kibana version 7.10.2. The product’s first General Availability version 1.0 was released in July 2021. Watch out for OpenSearch becoming a competitor to Elasticsearch in the search engine space.

Now that we have a fair understanding of what a modern search engine is and the shape of the search landscape, let’s jump into an overview of Elasticsearch.

Elasticsearch overview

Elasticsearch is an open source search and analytics engine. Developed in Java, it is an ultra-fast, highly available search engine built on the popular full-text library Apache Lucene (https://lucene.apache.org). Elasticsearch wraps around the powerful functionality of Lucene by providing a distributed system with RESTful interfaces. Lucene is the powerhouse of Elasticsearch, and Kibana is the administrative UI to manage and work with Elasticsearch.

Full-text searching is where Elasticsearch excels as a modern search engine. It can retrieve relevant documents in response to a user’s search criteria at an awesome speed. We can search for exact terms, too, like keywords, dates, or a range of numbers or dates. Elasticsearch is packed with top-notch features such as relevancy, “did-you-mean” suggestions, auto-completion, fuzzy and geospatial searching, highlighting, and more.

In addition to being a frontrunner in providing near-real-time search capabilities, Elasticsearch stands tall in statistical aggregations on big data. Of course, we must consider the use case before embracing the product, as Elasticsearch may not be the best fit for every use case. Out of the box, Elasticsearch also boasts commendable features such as application performance monitoring, predictive analytics and outlier detection, and security threat monitoring and detection.

Elasticsearch focuses on finding a deeper meaning in the data that’s been collected. It can aggregate data, perform statistical calculations, and find intelligence within the data. We can create rich visualizations and dashboards and share them with others using Kibana tooling. Elasticsearch can find averages, sums, means, and modes as well as undertaking complex analytics such as bucketing data in histograms and other analytical functions.

Furthermore, Elasticsearch runs supervised and unsupervised machine learning algorithms on our data. Models help to detect anomalies, find outliers, and forecast events. In supervised learning mode, we can provide training sets so the model learns and makes predictions.

Elasticsearch also comes with the capability to observe applications and their health by monitoring performance metrics such as the memory and CPU cycles of the web servers in a network. It lets us sift through millions of web server logs to find or debug application issues. Elasticsearch also invests time and resources in building security solutions: for example, alerting us to security threats, IP filtering, endpoint prevention, and more.

Core areas

Elastic, the company behind Elasticsearch, has been positioning itself predominantly in three core areas: search, observability, and security, as shown in figure 1.2. Let’s look at each of these areas in turn.

Figure 1.2

Figure 1.2 Core application areas of Elastic, the company behind Elasticsearch

Elastic Enterprise Search

Whether letting users search across varied content providers (like Slack, Confluence, Google Drive, and others) or enabling search capabilities for our applications, apps, and websites, the Elastic Enterprise Search suite helps build models and a customized search engine.

Search can be integrated deep into a multitude of applications in various domains—business, infrastructure, applications, and so on. Users can create a web application backed by Elasticsearch, a mobile app supported by Elasticsearch, or a server-side search “service with Elasticsearch as the spine for search capabilities.

Elastic Observability

Applications running on infrastructure produce a lot of metrics that are usually used for application observability and monitoring. We can use Elasticsearch in the observability space: the state of applications, servers, racks, and modules can all be monitored, logged, tracked, and alerted. We can also use the Elastic tools to perform application management and monitoring on a large scale.

Elastic Security

Elastic enters the realm of security by enabling threat detection and prevention and providing advanced features such as the capability of removing malware at the source, encryption at rest, and more. As a security information and event management (SIEM) tool, Elastic is positioning itself to protect organizations with its advanced security toolkits.

Elastic Stack

Elasticsearch is the core of the search engine, and a handful of Elastic products complement it. The suite of products is called the Elastic Stack and includes Kibana, Logstash, Beats, and Elasticsearch. (It was formally called ELK Stack but was renamed Elastic Stack after Beats was introduced into the product suite.)

The combination of these four products helps build an enterprise application by integrating, consuming, processing, analyzing, searching, and storing various data sets from disparate sources. As demonstrated in figure 1.3, Beats and Logstash bring the data into Elasticsearch, while Kibana is the visual UI that works on that data.

Figure 1.3

Figure 1.3 The Elastic Stack: Beats, Logstash, Elasticsearch, and Kibana

Before we move on to look at use cases for Elasticsearch, let’s briefly go over these essential moving parts at a high level.

BEATS

Beats are single purpose data shippers; they load data from out various external systems and pump it into Elasticsearch. Various types of beats are available out of the box. These include Filebeat, Metricbeat, Heartbeat, etc., and each performs a specific data consumption task. These are single-purpose components: for example, Filebeats are designed for file-based transports and Metricbeats for vital machine and operating system memory and CPU information. The beats’ agents are installed on the servers so they can consume data from their source and send it to their destination.

LOGSTASH

Logstash is an open source data-processing engine. It extracts data originating from multiple sources, processes it, and sends it to a variety of target destinations. During the processing of the data, Logstash transforms and enriches the data. It supports a myriad of sources and destinations including files, HTTP, JMS, Kafka, Amazon S3, Twitter, and dozens of others. It promotes a pipeline architecture, and every event that goes through the pipeline is parsed as per the preconfigured rules, thus creating a real-time pipeline for data ingestion.

KIBANA

Kibana is a multipurpose web console that provides a host of options such as executing queries; developing dashboards, graphs, and chart visualizations; and creating dropdowns and aggregations. However, we can use any REST client to talk to Elasticsearch to invoke the APIs, not just Kibana. For example, we can invoke APIs using cURL, Postman, or native language clients.

Elasticsearch use cases

Pinpointing Elasticsearch for a particular use case or domain is difficult. It is omnipresent in many areas from search to analytics to machine learning jobs. It is widely used across a multitude of industries, including finance, defense, transport, government, retail, cloud, entertainment, space, and more. Let’s take a high-level glance at how Elasticsearch can be used in an organization.

Use case: Search engine

Elasticsearch has become the go-to technology for its full-text search capabilities. The product is not limited to full-text searching but can also be used for structured data and geolocation-based searches. Broadly speaking, customers use Elasticsearch in three domains: App Search, Enterprise Search, and Site Search.

In App Search, Elasticsearch serves as a backbone, providing search and analytical capabilities for applications. A search service backed up by Elasticsearch can be designed as a microservice that serves the application’s search requirements, such as searching for customers, orders, invoices, emails, and so on.

In most organizations, data is scattered across many data stores, applications, and databases. For example, organizations are often integrated with Confluence, intranet spaces, Slack, email, databases, cloud drives (iCloud drive, Google Drive, etc.), and others. Collating and searching through vast amounts of data with integrations to varied sources is a challenge for these organizations. This is where Elasticsearch can be employed for Enterprise Search and data organization.

If we have an online business website amassing data, providing search is something of a bare necessity for attracting customers and keeping them happy. Site Search is a software-as-a-service (SaaS) offering from Elastic that, once enabled, crawls through the given site pages, fetching data and building indices backed by Elasticsearch. Once the crawling and indexing are complete, the site can be integrated easily with the search facility. The Site Search module also helps create a search bar and the code snippet related to it. The website administrator can copy the snippet of generated code onto their homepage to enable a search bar instantly, thus making the website fully functional with integrated search.

Use case: Business analytics

Organizations capture tons of data from various sources, and that data often holds the key to survival and success. Elasticsearch can help extract trends, statistics, and metrics from data, giving organizations knowledge about their operations, sales, turnover, profits, and many other features for timely management.

Use case: Security analytics and threat and fraud detection

Data security and potential breaches of it are nightmares for organizations. Elasticsearch’s security analytics help organizations analyze every bit of information—be it from applications, a network, endpoints, or the cloud. This analysis can provide insights into threats and vulnerabilities and let the organization hunt for malware and ransomware, thus alleviating the risk of falling prey to hackers.

Use case: Logging and application monitoring

Applications spit out a lot of data in the form of application logs and metrics. These logs provide insights into the health of the application. With the advent of the cloud and the microservices world, logs are scattered across services, and meaningful analysis is a cumbersome affair. Elasticsearch is our friend here. One of the popular use cases for Elasticsearch is indexing logs and analyzing them for application errors and debugging purposes.

Elasticsearch is a powerful and flexible search and analytics engine, but it’s not suitable for every use case. Let’s briefly go over the issues we may encounter and use cases for which Elasticsearch is the wrong choice.

Unsuitable Elasticsearch uses

Not every use case can be satisfied by Elasticsearch. It is a powerful and flexible search and analytics engine, but unfortunately, this tool has limitations that we must consider before choosing it for our requirements. Here are a few scenarios where Elasticsearch may be an incorrect or inefficient solution:

Relational data—Elasticsearch is not the right tool to accommodate search over data that has relationships and needs to perform complex database joins. Elasticsearch is not designed to handle complex relational data structures. If your data is relationship-heavy, a relational database like MySQL or PostgreSQL may be a better fit. Most modern databases (MySQL, PostgreSQL, etc.) also offer full-text search capabilities, although the features are not as advanced as in a modern search engine like Elasticsearch.
Transactional data—Elasticsearch is an “eventually consistent” search engine, which makes it unsuitable for applications that require immediate consistency, such as financial transactions. For these types of use cases, consider using a traditional relational database or a NoSQL database like MongoDB.
Geospatial data—While Elasticsearch has built-in support for geospatial data, it may not be the most efficient solution for large-scale geospatial analytics. For these use cases, consider using a dedicated geospatial database like PostGIS or a geospatial analytics platform like ArcGIS.
High-write workloads—Elasticsearch can handle high-read workloads, but it is not optimized for high-write workloads. If you need to index large amounts of data in real time, consider using a dedicated indexing engine like Apache Flume or Apache Kafka.
Online analytical processing (OLAP) data—If you need to perform complex multidimensional analysis on large data sets, a traditional OLAP database like Microsoft Analysis Services or IBM Cognos may be a better fit than Elasticsearch.
Large binary data—While Elasticsearch can handle large amounts of text data, it may not be the best solution for indexing and searching large binary data like videos or images. For these use cases, consider using a dedicated binary data store like Hadoop Distributed File System (HDFS), Amazon S3, or Azure Files.
Real-time analytics—Elasticsearch is great for performing real-time search and analytics on large data sets, but it may not be the most efficient solution for real-time data processing and analytics. Instead, consider a specialized real-time analytics platform like Apache Spark or Apache Flink.
Latency-sensitive applications—Although Elasticsearch is designed to handle high-volume search and analytical queries, it can still have latency issues when dealing with large amounts of data. For applications that require sub-millisecond response times, a specialized search engine like Apache Solr or a columnar database like Apache Cassandra may be a better fit.
Other types—Elasticsearch is not a preferred solution for time-series data, graph data, in-memory data, and various other types of data. If you need to store and analyze time-series data, a specialized time-series database like InfluxDB or TimescaleDB may be a better fit. Similarly, a graph database such as Neo4j may help you tackle graph data.

It’s important to evaluate your specific use case and requirements before choosing Elasticsearch as your technology and tool. In the next section, we discuss common misconceptions about Elasticsearch as a tool, a technology, and a search solution.

Misconceptions

A major misconception about Elasticsearch is mistaking it for a traditional relational database. It’s also a common misunderstanding that setting up Elasticsearch is easy, while in reality, many tweaks are required to set up a decent-sized cluster. In addition, Elasticsearch is often thought of as a technology used for text search, when in fact it can be used for a wide range of search and analytics use cases. The following list summarizes some common misconceptions about Elasticsearch:

Elasticsearch is easy to set up and manage. While Elasticsearch is relatively straightforward to set up and get started with, it can be challenging to manage and scale as data grows and use cases increase. Although everything works out of the box, making an engineer’s life easy, taking Elasticsearch into the production environment requires effort. We may need to tweak the configuration and fine-tune the memory, manage node failures, or even scale the cluster to handle petabytes of data as our data grows.
Elasticsearch is a relational database. Elasticsearch is not a relational database and does not support traditional relational database features like transactions, foreign keys, and complex join operations. For example, we can’t enforce referential integrity or perform complex “oin operations. For example, we can’t enforce referential integrity or perform complex join operations in Elasticsearch. If you need these features, a proven relational database like MySQL or PostgreSQL is definitely your solution.
Elasticsearch can handle all types of data. Elasticsearch is versatile and can handle a wide range of data types, but it is not designed to handle every type of data with equal ease. For example, it may not be the best solution for real-time data processing and analytics or for handling large binary data. If you need to store and process large binary data like videos or images, consider using a dedicated binary data store like HDFS or Amazon S3.
Elasticsearch is only for text search. While Elasticsearch is great for text search, it can also perform complex analytics on structured and unstructured data. For example, we can use Elasticsearch to perform aggregations, analyze log data, and visualize data using Kibana.
Elasticsearch can replace all other technologies. Elasticsearch is a powerful and flexible technology, but it is not a one-size-fits-all solution and or the best choice for every use case. It can never replace a traditional relational database, for example.
Elasticsearch is always faster than other technologies. Elasticsearch is designed for high performance and is expected to perform well under heavy loads. However, there’s only so much Elasticsearch can do, and its performance primarily depends on how well the platform engineers fine-tune it.
Elasticsearch deals with big data only. Elasticsearch can handle petabytes of data in large data sets, but it is equally performant when dealing with small data sets on the order of a few gigabytes. For example, we can use Elasticsearch to search and analyze data for an organization’s small email database or a startup company without much effort.

These are just a few examples of misconceptions about Elasticsearch. As mentioned earlier, you must carefully evaluate your specific requirements and use case before choosing Elasticsearch or any other technology.

Popular adoption

A long list of organizations use Elasticsearch for everything from searching to business analysis, log analytics, security alert monitoring, and application management, as well as using it as a document store. Let’s consider some of these organizations and how they put Elasticsearch to use in their operations.

Uber powers its rider and event prediction using Elasticsearch. It does so by storing millions of events, searching through them, and analyzing the data at a near-real-time rate. Uber predicts demand based on location, time, day, and other variables, including taking past data into consideration. This helps Uber deliver rides pretty.

Netflix adopted the Elastic Stack to provide customer insights to its internal teams. It also uses Elasticsearch for log event analysis to support debugging, alerting, and managing its internal services. Email campaigns and customer operations are all backed by the Elasticsearch engine. The next time you receive an email from Netflix mentioning a newly added movie or TV series, keep in mind that the campaign analytics behind that simple email were all supported by Elasticsearch.

PayPal embraced Elasticsearch as a search engine to allow customers to store and search through their transactions. The company has implemented transaction search features along with analytics for use by merchants, end customers, and developers.

Similarly, the online e-commerce company eBay adopted Elasticsearch to support full-text searching by end users. As users, we are using Elasticsearch directly when searching through eBay’s inventory. The company also uses the Elastic Stack for analytics, log monitoring, and storing transactional data in a document store.

GitHub, a popular code repository for developers, indexes its 8 million (and counting) code repositories—consisting of over 2 billion documents—with Elasticsearch to enable a powerful search experience for its users. Similarly, Stack Overflow uses Elasticsearch to provide developers with quick and relevant answers, and Medium (a popular blog platform) uses the Elastic Stack to serve reader queries in a near-real-time mode.

Generative AI and modern search

Unless you are living in a cave, you have undoubtedly heard about a recent internet revolution: ChatGPT. ChatGPT is a generative AI tool that was developed and released by the OpenAI team in November 2022. Is isn’t often that a technically superior tool lands in the hands of the general public that can help them in unimaginable ways, such as creating a travel itinerary for a summer trip to Athens, summarizing a legal document in layman’s terms, developing a self-help plan for losing weight, analyzing code for security and performance bugs, designing an application’s data models, comparing and contrasting technologies for a specific use case, writing complaint letters to the Twitter CEO and much more.

ChatGPT (https://chat.openai.com) is a conversational agent (chatbot) built on a GPT (generative pretrained transformer) architecture, and it is capable of generating human-like text based on user prompts. It is an instance of a large language model (LLM) designed for conversation with a specific goal of generating safe, relevant content while engaging in a meaningful dialogue. The model is fed vast amounts of text data and learns to predict the next word in a sentence. It is trained using a diverse range of internet text, but it can also be fine-tuned with specific datasets for various tasks. Through this process, the model learns parts and parcels of human language text: grammar, punctuation, syntax, facts about the world, and some degree of reasoning ability.

NOTE LLM is a broad term that refers to any large-scale model trained to understand or generate human-like text. These models are characterized by their vast number of parameters and their ability to handle a wide range of natural language processing tasks. LLMs can be based on various architectures and training methods.

With the release of ChatGPT to the public, a sudden race in the space of search using AI emerged overnight. ChatGPT has become a disruptor to many industries and is no less a threat to Google search. AI-backed tools similar to ChatGPT will disrupt many industries in the coming years. Under tremendous pressure—and possibly to save its search leader status—Google decided to unleash its version of conversational generative AI: its agent called Bard (https://bard.google.com) was made publicly available in May 2023.

In the meantime, Microsoft committed to investing $10 billion in ChatGPT, over and above its initial investment of $3 billion since 2019. Microsoft’s Edge browser is integrated with ChatGPT via the Bing search engine, which also was made publicly available in May 2023. In addition, Microsoft rolled out AI-powered Microsoft 365 apps, so the AI agent is available in Microsoft Word, Excel, email, and other tools. Meta’s LLaMA is another tool that has begun competing in the generative AI race.

Search engineers are being asked fundamental questions, including how generative AI agents will change the course of search. Let’s answer this question by asking ChatGPT how AI agents can complement or help modern search or change its direction. These are the areas where tools like generative AI will reshape the search space:

Intuitive Search—Search queries will become more conversational and intuitive. Generative AI models like GPT-4 have an advanced understanding of natural language, enabling them to interpret complex queries more effectively. Users will no longer need to rely on specific keywords or phrases; they can simply ask questions as they would in a conversation with another person. This will allow for more accurate and relevant search results, as AI can better comprehend the context and intent of the query. With the introduction and release of highly capable generative AI agents and models, there is a greater scope to significantly reshape full-text search capabilities provided by modern search engines like Elasticsearch. We can expect to see several key changes that will redefine the search experience for both users and developers as this technology is increasingly integrated into search platforms.
Personalized search —With the incorporation of generative AI, search results can become more personalized and adaptive. Search engines will be able to learn many valuable data points from users’ preferences, behavior, and search history, which in turn will help engines tailor results to meet individual users’ needs. As the AI gathers more data, it will continually refine its understanding of what users are looking for, leading to an increasingly customized search experience.
Predictive search —Generative AI has the potential to make search engines more proactive in anticipating user needs. Instead of merely responding to queries, AI-driven search engines may be able to predict what information users are interested in, based on their previous interactions or current context. This will allow search platforms to proactively offer relevant suggestions, increasing the value of the search experience and reducing the need for users to perform additional queries.
Advanced search —Generative AI will enable search engines to provide more diverse and rich search results. By understanding the context and semantics of a query, AI-driven search engines can generate content summaries and relevant visualizations and even synthesize new information to help answer a user’s question. This will lead to a more comprehensive and informative search experience that goes beyond merely linking to existing content.

Summary

Search is the new normal and the most sought-after functionality for organizations, enabling competitive advantage.
Search engines built using relational databases as the backend used to serve our search purposes but can’t fulfill the full-fledged search functionality found in modern search engines.
Modern search engines provide multifaceted, full-text search capabilities and multifold benefits from basic search to advanced search and analytical functions, all with split-second performance. They are also expected to handle terabytes to petabytes of data and scale if needed.
Elasticsearch is an open source search and analytics engine built over Apache Lucene. It is a highly available server-side application developed in Java.
Because Elasticsearch was designed as a programming-language-agnostic product, communication with the server takes place over HTTP using rich RESTful APIs. These APIs receive and send data in JSON format.
The Elastic Stack is a suite of products composed of Beats, Logstash, Elasticsearch, and Kibana. Beats are single-purpose data shippers, Logstash is a data-processing ETL (extract, transform, load) engine, Kibana is the administrative UI tool, and Elasticsearch is the heart and soul of the stack.
The Elastic Stack enables an organization to position itself in three core areas: search, observability, and security.
Elasticsearch has become popular over the last few years due to its structured/unstructured search and analytics capabilities; rich set of RESTful APIs; its schema-free nature; and performance, high-availability, and scalability characteristics.
AI-powered search is here. With the advent of generative AI and ChatGPT, the search space will be explored further, and search will become more intuitive and predictive.

Excerpt From Elasticsearch in Action, Second Edition Madhusudhan Konda.

Comparing React VS HTMX: Choosing the Right Frontend Approach

Victor Hazbun — Fri, 23 Jun 2023 03:53:31 +0000

Introduction:

Building modern web applications often requires combining a robust backend framework with a frontend framework to create dynamic and interactive user experiences. React and HTMX are two popular choices when it comes to frontend development. In this article, we'll explore the differences between React and HTMX, and how they can be used to create compelling web applications. We'll also provide practical examples for each approach.

React:

React has gained significant popularity in the web development community due to its component-based architecture and efficient rendering capabilities.

Using React involves setting up frontend environment powered by tools like Webpack. React components can then be used to fetch data from the backend through APIs and render it dynamically on the client-side. This approach provides a seamless user experience, as React updates only the necessary parts of the UI, avoiding full page reloads.

React Example App:

This example actively searches a contacts database as the user enters text.

We start with a search input and an empty table:

import React, { useState } from 'react';

function ContactSearch() {
  const [searchResults, setSearchResults] = useState([]);

  const handleSearch = (event) => {
    const searchTerm = event.target.value;

    fetch('/search', {
      method: 'POST',
      headers: {
        'Content-Type': 'application/json',
      },
      body: JSON.stringify({ search: searchTerm }),
    })
      .then((response) => response.json())
      .then((data) => {
        setSearchResults(data);
      });
  };

  return (
    <div>
      <h3>
        Search Contacts
        <span className="htmx-indicator">
          <img src="/img/bars.svg" alt="Loading..." /> Searching...
        </span>
      </h3>
      <input
        className="form-control"
        type="search"
        name="search"
        placeholder="Begin Typing To Search Users..."
        onChange={handleSearch}
      />
      <table className="table">
        <thead>
          <tr>
            <th>First Name</th>
            <th>Last Name</th>
            <th>Email</th>
          </tr>
        </thead>
        <tbody id="search-results">
          {searchResults.map((contact) => (
            <tr key={contact.id}>
              <td>{contact.firstName}</td>
              <td>{contact.lastName}</td>
              <td>{contact.email}</td>
            </tr>
          ))}
        </tbody>
      </table>
    </div>
  );
}

export default ContactSearch;

Express.js app for React frontend

const express = require('express');
const app = express();

// POST /search route
app.post('/search', (req, res) => {
  // Simulating search logic
  const searchTerm = req.body.search;
  const searchResults = [
    { id: 1, firstName: 'John', lastName: 'Doe', email: 'john@example.com' },
    { id: 2, firstName: 'Jane', lastName: 'Doe', email: 'jane@example.com' },
  ];

  // Filter the search results based on the searchTerm
  const filteredResults = searchResults.filter((result) => {
    const firstNameMatch = result.firstName.toLowerCase().includes(searchTerm.toLowerCase());
    const lastNameMatch = result.lastName.toLowerCase().includes(searchTerm.toLowerCase());
    const emailMatch = result.email.toLowerCase().includes(searchTerm.toLowerCase());

    return firstNameMatch || lastNameMatch || emailMatch;
  });

  // Simulating delay for illustration purposes
  setTimeout(() => {
    res.json(filteredResults);
  }, 1000); // Delayed response for 1 second
});

// Start the server
app.listen(3000, () => {
  console.log('Server started on port 3000');
});

HTMX:

HTMX takes a different approach by focusing on enhancing the user interface with minimal JavaScript code. HTMX allows you to add interactivity to your frontend using declarative attributes embedded directly in your HTML markup.

With HTMX we leverage the power to update individual task items without reloading the entire page. HTMX achieves this by making HTTP requests to the server in the background and updating specific elements on the page using the hx-swap attribute. This approach simplifies the development process by reducing the need for separate frontend tooling and JavaScript code.

HTMX Example App:

<h3> 
  Search Contacts 
  <span class="htmx-indicator"> 
    <img src="/img/bars.svg"/> Searching... 
   </span> 
</h3>
<input class="form-control" type="search" 
       name="search" placeholder="Begin Typing To Search Users..." 
       hx-post="/search" 
       hx-trigger="keyup changed delay:500ms, search" 
       hx-target="#search-results" 
       hx-indicator=".htmx-indicator">

<table class="table">
    <thead>
    <tr>
      <th>First Name</th>
      <th>Last Name</th>
      <th>Email</th>
    </tr>
    </thead>
    <tbody id="search-results">
    </tbody>
</table>

Express.js app for HTMX frontend

const express = require('express');
const app = express();

// POST /search route
app.post('/search', (req, res) => {
  // Simulating search logic
  const searchTerm = req.body.search;
  const searchResults = [
    { id: 1, firstName: 'John', lastName: 'Doe', email: 'john@example.com' },
    { id: 2, firstName: 'Jane', lastName: 'Doe', email: 'jane@example.com' },
  ];

  // Filter the search results based on the searchTerm
  const filteredResults = searchResults.filter((result) => {
    const firstNameMatch = result.firstName.toLowerCase().includes(searchTerm.toLowerCase());
    const lastNameMatch = result.lastName.toLowerCase().includes(searchTerm.toLowerCase());
    const emailMatch = result.email.toLowerCase().includes(searchTerm.toLowerCase());

    return firstNameMatch || lastNameMatch || emailMatch;
  });

  // Generate the HTML for search results
  const html = filteredResults
    .map((result) => `<tr><td>${result.firstName}</td><td>${result.lastName}</td><td>${result.email}</td></tr>`)
    .join('');

  // Simulating delay for illustration purposes
  setTimeout(() => {
    res.send(html);
  }, 1000); // Delayed response for 1 second
});

// Start the server
app.listen(3000, () => {
  console.log('Server started on port 3000');
});

Choosing the Right Approach:

When deciding between React and HTMX, consider the following factors:

Project Complexity: If your application requires complex UI interactions, real-time updates, and a rich set of reusable components, React might be the better choice. React's component-based architecture and vast ecosystem provide the necessary tools to handle such scenarios.
Development Speed: HTMX can be an excellent choice for projects where simplicity and speed of development are paramount. With HTMX, you can add interactivity to your frontend directly, reducing the need for extensive JavaScript development and tooling setup.
Team Expertise: Consider the skills and expertise of your development team. If they are well-versed in React and comfortable with the React ecosystem, React may be the preferred option. On the other hand, if your team prefers a simpler frontend approach, HTMX could be a better fit.

Conclusion:

Both React and HTMX offer powerful frontend solutions. React provides a comprehensive framework for building complex, interactive web applications, while HTMX simplifies interactivity by embedding declarative attributes directly in HTML markup. Choosing between the two depends on your project requirements, complexity, and the expertise of your development team. Evaluate the trade-offs, consider your specific needs, and select the approach that aligns best with your goals.

Staff Engineer Interview: Navigating the Path to Technical Leadership

Victor Hazbun — Fri, 23 Jun 2023 00:52:30 +0000

Introduction

Becoming a staff engineer is a significant milestone in one's technical career. It represents a transition from individual contributor to a leadership role, where one is expected to guide and influence the engineering team's direction. To successfully navigate the staff engineer interview process and excel in the role, it is crucial to possess a combination of technical expertise, leadership skills, and a deep understanding of software development principles. In this article, we will explore key aspects of a staff engineer interview, including potential interview questions, essential skills, and the knowledge necessary for staff engineers in today's dynamic tech landscape.

Understanding the Role of a Staff Engineer

A staff engineer is a senior-level position that requires a comprehensive understanding of software engineering principles, coupled with strong leadership and communication abilities. As a staff engineer, you will be responsible for driving technical decisions, mentoring junior engineers, and collaborating with cross-functional teams. You will play a crucial role in defining the architecture, scalability, and performance of software systems while guiding the team towards successful project outcomes.

Potential Interview Questions

Technical Depth: Expect questions that assess your expertise in core technical areas relevant to the company's tech stack. These may include system design, algorithms, data structures, performance optimization, and distributed systems.
Leadership and Collaboration: Be prepared to discuss your experience in leading technical projects, managing teams, and fostering collaboration. Questions may focus on your approach to decision-making, conflict resolution, mentoring, and how you enable a productive and inclusive work environment.
Problem-Solving and Critical Thinking: Employers often assess a staff engineer's ability to solve complex problems, think analytically, and make strategic decisions. You may encounter scenario-based questions where you are asked to devise solutions for challenging technical or organizational issues.
Communication and Influence: Effective communication is crucial for staff engineers. Expect questions that evaluate your ability to articulate technical concepts to both technical and non-technical stakeholders. You may be asked about instances where you successfully influenced technical decisions or resolved conflicts through effective communication.

Must-Know Skills for Staff Engineers

Technical Proficiency: A staff engineer should have a deep understanding of the company's tech stack and possess expertise in relevant programming languages, frameworks, and tools. Stay updated with industry trends and emerging technologies to drive innovation and make informed technical decisions. Knowledge on programming, cloud computing, system design, SQL databases, and related topics would be valuable in this role.
Leadership and Mentorship: Staff engineers are expected to provide guidance and mentorship to junior team members. Develop strong leadership skills, including the ability to motivate, inspire, and support your colleagues. Cultivate a collaborative environment that fosters continuous learning and growth.
Systems Thinking: As a staff engineer, you must have a holistic understanding of software systems. Focus on developing your ability to analyze complex systems, identify bottlenecks, and propose scalable and efficient solutions. This includes considering factors such as performance, security, reliability, and maintainability.
Communication and Collaboration: Effective communication is paramount for staff engineers. Hone your ability to clearly convey complex technical concepts to different audiences, both verbally and in written form. Foster collaborative relationships with stakeholders across teams to drive alignment and achieve project objectives.

Adapting to the Evolving Tech Landscape

To thrive as a staff engineer in today's dynamic tech landscape, it's essential to embrace continuous learning and adaptability. Stay updated with new technologies, industry best practices, and emerging trends. Cultivate a growth mindset and seek opportunities to expand your skill set through training, attending conferences, participating in open-source projects, and engaging with the developer community.

Example Job Post for a Staff Engineer

Job Title: Staff Engineer at Dummy.co

Responsibilities:

Provide technical leadership in software development projects.
Mentor and guide junior engineers, fostering a collaborative and inclusive work environment.
Collaborate with cross-functional teams to define architecture, scalability, and performance of software systems.
Perform code reviews, identify areas for improvement, and ensure high-quality code standards.
Analyze complex systems, identify bottlenecks, and propose scalable and efficient solutions.
Stay up-to-date with industry trends and emerging technologies relevant to the tech stack.

Requirements:

Bachelor's or Master's degree in Computer Science or related field.
Strong experience in software development with a focus on Ruby on Rails.
Proficiency in cloud computing platforms, such as AWS or Azure.
Solid understanding of system design principles and distributed systems.
In-depth knowledge of SQL databases and database performance optimization.
Excellent problem-solving and critical-thinking abilities.
Proven experience in leading technical projects and mentoring junior engineers.
Exceptional communication and collaboration skills.

Sample Interview at Dummy.co

Question 1: Can you explain the differences between a monolithic architecture and a microservices architecture? What are the advantages and disadvantages of each?

Answer: In a monolithic architecture, the entire application is built as a single, self-contained unit. It is typically easier to develop and test but can become challenging to maintain and scale as the application grows. Microservices architecture, on the other hand, decomposes the application into smaller, independent services that can be developed and deployed separately. It offers benefits like scalability, independent deployment, and fault isolation but introduces complexities in managing inter-service communication and data consistency.

Question 2: How would you approach improving the performance of a slow SQL query in a large database?

Answer: To improve the performance of a slow SQL query, I would start by analyzing the query execution plan using database query profiling tools. This would help identify the bottlenecks, such as missing or inefficient indexes, excessive joins, or suboptimal query structure. Based on the analysis, I would consider strategies like optimizing the query, adding appropriate indexes, denormalizing data, or caching results. It's crucial to strike a balance between query performance and the overall database design.

Question 3: How do you handle conflicts and disagreements within a technical team during a project?

Answer: Conflict resolution is an essential skill for a staff engineer. When conflicts arise within a technical team, I believe in fostering open communication and active listening to understand each person's perspective. I would encourage a collaborative environment where team members can express their ideas and concerns. If a resolution cannot be reached through discussion, I would facilitate a constructive compromise or involve relevant stakeholders for mediation. It's crucial to prioritize the project's objectives and maintain a respectful and inclusive work environment.

Question 4: How would you approach mentoring and guiding junior engineers on the team?

Answer: Mentoring and guiding junior engineers is an integral part of the staff engineer role. I would start by understanding their individual strengths, weaknesses, and career goals. I would create a safe space for them to ask questions and seek guidance. Regular one-on-one meetings would provide an opportunity to discuss their progress, offer constructive feedback, and set realistic goals. I would encourage them to take ownership of their projects while providing support and resources for their growth. Celebrating their achievements and fostering a culture of continuous learning would be paramount.

These sample interview questions and answers are meant to provide a starting point for your preparation. Tailor your responses based on your own experiences and knowledge, and use them as a guide to showcase your technical expertise, problem-solving skills, and leadership abilities during the staff engineer interview.

Creating a Learning Path

To sharpen the necessary skills for the staff engineer role, consider the following learning path:

Strengthen your programming Skills: Deepen your knowledge of the required programming language by working on real-world projects, exploring advanced topics, and leveraging resources like online tutorials, documentation, and books focused on the language/framework.
Gain Expertise in Cloud Computing: Familiarize yourself with cloud computing platforms such as AWS or Azure. Take online courses or certifications to understand their key services, deployment models, and scalability options.
Master System Design Principles: Study system design concepts and patterns to develop the ability to design scalable, fault-tolerant, and high-performance systems. Practice designing and architecting complex systems, considering factors like scalability, security, and maintainability.
Dive into SQL/noSQL Databases: Learn SQL/noSQL database management and optimization techniques. Gain proficiency in writing efficient queries, indexing, and optimizing database performance. Explore different database management systems and their use cases.
Enhance Leadership and Collaboration Skills: Invest in developing your leadership and collaboration abilities. Attend leadership workshops, participate in team-building activities, and seek opportunities to mentor and guide junior engineers.
Stay Updated with Industry Trends: Engage with the developer community, follow influential tech blogs, and attend conferences or webinars to stay current with emerging technologies, industry best practices, and trends in software development.

Remember, this learning path is not exhaustive but provides a starting point. Continuously seek opportunities for growth, practice hands-on coding, and seek feedback from experienced professionals to refine your skills and knowledge.

Conclusion

Becoming a staff engineer requires a combination of technical expertise, leadership skills, and a deep understanding of software engineering principles. When preparing for a staff engineer interview, be ready to showcase your technical depth, problem-solving abilities, leadership experience, and relevant knowledge in areas like programming, cloud computing, system design, and SQL/noSQL databases. By following a learning path tailored to the role's requirements, you can sharpen your skills and position yourself for success as a staff engineer in today's dynamic tech landscape.

If you are aspiring to become a Staff Engineer and are seeking mentorship to accelerate your professional growth, I highly recommend reading my previous post on The Power of Mentorship: Why Having a Mentor is Essential for Professional Growth first.

For personalized guidance and support on your Staff Engineer journey, feel free to contact me at codementor.io. As an experienced software developer and mentor, I am here to assist you in reaching your full potential as a Staff Engineer. Let's work together and help you achieve success in the software industry.

The Power of Mentorship: Why Having a Mentor is Essential for Professional Growth

Victor Hazbun — Fri, 23 Jun 2023 00:10:23 +0000

Introduction:

In the world of software development and tech, continuous learning and growth are paramount. The rapidly evolving landscape demands that professionals stay ahead of the curve, constantly honing their skills and expanding their knowledge. In this pursuit, having a mentor can be an invaluable asset. A mentor provides guidance, support, and expertise that can significantly accelerate your professional journey. In this article, we will explore the importance of having a mentor and how their influence can shape your success as a developer.

Gaining Insight and Wisdom:

One of the primary benefits of having a mentor is gaining access to their wealth of experience and knowledge. A mentor has already walked the path you're embarking on and can offer insights into the challenges, opportunities, and pitfalls you may encounter. Their guidance can help you navigate complex technical problems, improve your decision-making abilities, and avoid common mistakes. Learning from their wisdom can save you valuable time and effort in your career development.

Expanding Your Network:

A mentor serves as a valuable connection to a broader professional network. They can introduce you to industry experts, influential figures, and potential collaborators. Through their network, you can tap into opportunities that might otherwise be out of reach. Mentors can also provide recommendations and referrals, boosting your credibility and opening doors to exciting career prospects. Building a strong network is crucial for career growth, and a mentor can be a catalyst in expanding your connections.

Personalized Guidance and Support:

A mentor understands your strengths, weaknesses, and aspirations. They provide personalized guidance tailored to your specific needs, enabling you to grow in the areas that matter most to you. Whether it's technical skills, project management, or career advancement, a mentor can help you set goals and devise strategies to achieve them. They offer support and encouragement, helping you overcome self-doubt and challenges along the way. Having someone invested in your success can be incredibly motivating and empowering.

Accelerated Learning Curve:

Learning through trial and error is a valid approach, but it can be time-consuming and inefficient. A mentor can expedite your learning process by sharing their own experiences, offering real-world examples, and providing constructive feedback on your work. They can suggest learning resources, recommend best practices, and challenge you to think critically. With a mentor, you can avoid unnecessary detours and make progress at a faster pace.

Building Confidence and Self-Efficacy:

Entering the tech industry or taking on new responsibilities can be intimidating. A mentor acts as a trusted advisor, boosting your confidence and self-belief. They provide encouragement, acknowledge your achievements, and help you overcome imposter syndrome. By witnessing their own accomplishments and hearing their stories of triumph over adversity, you gain the confidence to tackle challenges head-on and embrace opportunities for growth.

Conclusion:

Having a mentor is a vital component of professional growth and development. Their guidance, expertise, and support can propel you towards success in your software development journey. Seek out a mentor within your organization, through networking events, or online platforms like codementor.io. Codementor.io provides a platform where experienced software developers like Victor Hazbun offer mentorship services.

With their expertise, you can unlock your full programming potential and accelerate your growth in the field. Embrace the transformative power of mentorship and take the leap towards achieving your professional goals. Remember, even the most accomplished professionals had mentors who played a crucial role in their rise to greatness.

Mongoid query nested embed document through belongs to and embeds many

Victor Hazbun — Tue, 27 Apr 2021 19:30:25 +0000

Given the following documents, I'm trying to find a log document given a token ID.

class Log
  include Mongoid::Document

  belongs_to :user
end

class User
  include Mongoid::Document

  embeds_many :tokens
end

class Token
  include Mongoid::Document
end

I tried Log.where('user.tokens._id': BSON::ObjectId('123ABC') with no luck. Any ideas?

Learn recursion with Tower of Hanoi game

Victor Hazbun — Sat, 23 May 2020 21:54:57 +0000

The objective of the puzzle is to move the entire stack to another rod, obeying the following simple rules:

Only one disk can be moved at a time.
Each move consists of taking the upper disk from one of the stacks and placing it on top of another stack or on an empty rod.
No larger disk may be placed on top of a smaller disk.

hanoy.rb

def hanoi(number, origin, auxiliar, destination)
  unless number == 0
    hanoi(number - 1, origin, destination, auxiliar)
    puts "Moved disc from #{origin} to #{destination}"
    hanoi(number - 1, auxiliar, origin, destination)
  end
end

hanoi(4, 'A', 'B', 'C')

Tower of Hanoi

A Hanoi Tower looks like this:

number (number of disks)
origin (A)
auxiliar (B)
destination (C)

To make this example easier to understand, let's pretend our Hanoi Tower has 4 disks, and columns A, B, C.

Our algorithm

The idea is to write a function that calls itself until the puzzle is solved.

But how?

First, determine when to stop the function execution. It must stop when all the disks has been moved to the column C. Our program continues executing unless the disks is equal to zero. Code:

unless number == 0

Our next mission is to actually figure out the puzzle (moving all the disks from "A" to "C").

Second, we will have to decrement the amount of disks when calling hanoi and pass along the origin, destination and auxiliar. Code:

hanoi(number - 1, origin, destination, auxiliar)

Basically, we are calling the same function (hanoi) but using new arguments, allowing us to solve the puzzle.

Third, we also want to print whenever the disks are moving from one place to another. Code:

puts "Moved disc from #{origin} to #{destination}"

Finally, we will call hanoi again because we want to move the disks. Code:

hanoi(number - 1, auxiliar, origin, destination)

Notice, the arguments are the same but in different order, again allowing us to solve the puzzle.

Final thoughts

I do not want you to understand the math behind the Hanoi Tower algorithm, just want you to realize that calling function hanoi in the same function hanoi is recursion.

It is important to understand that a recursive function needs a mechanism to stop executing, otherwise it will run forever or in the worst case it will cause errors.

Maybe you noticed our hanoi function did not mutated objects because it calls itself with new arguments rather than using existing objects.

Creating an Alfred workflow with Ruby

Victor Hazbun — Tue, 28 Apr 2020 07:19:51 +0000

If you love Alfred like me, you will enjoy this tutorial, I will teach you how to build a custom workflow to short URLs using https://bitly.com/ (Free account).

Setting up a Bitly account

First, you will need to signup for a Bitly account.

Then create a generic access token (You will need it later). Here are the instructions.

Building the Alfred workflow

This is the code for the workflow, let's talk about it.

require 'net/http'
require 'uri'
require 'json'

uri = URI.parse('https://api-ssl.bitly.com/v4/shorten')

# Set parameters
params = {
  long_url:  ARGV[0]
}
# Set headers
headers = {
    'Host': 'api-ssl.bitly.com',
    'Authorization': "Bearer #{ENV['BITLY_ACCESS_TOKEN']}",
    'Content-Type': 'application/json',
    'Accept': 'application/json'
}
# Create the HTTP objects
http = Net::HTTP.new(uri.host, uri.port)
# Force SSL request
http.use_ssl = true
# Build POST request
request = Net::HTTP::Post.new(uri.request_uri, headers)
# Set request body (As JSON)
request.body = params.to_json
# Send the request
response = http.request(request)
# Get link from response
puts JSON.parse(response.body)['link']

As you might know, you can build your own workflows using Ruby and other languages, even bash.

I used three Ruby native dependencies:

Net::HTTP to make HTTP requests to Bitly
URI to build URLs
JSON to parse the response body and be able to extract the shorten link

The parameters I'm sending to Bitly come from the user input via Alfred, so in order to access the first and only argument, we have to use ARGV[0].

For the headers it is mandatory to set the Authorization Bearer since this is how Bitly knows who is trying to shorten the link. We read the access token from the environment variables.

Alfred will ask for the token when you download the workflow.

We also need to make the request of type https since they Bitly API requires a secure protocol. We not only need to use https:// in the request URL, we also have to tell http.use_ssl, otherwise it won't work.

The request.body has to be set up as JSON as Bitly documentation indicates, so we convert the parameters to JSON like this params.to_json.

Finally we have to parse the JSON response using JSON.parse(response.body), which returns a hash with keys and values.

The key we need is the link, for Alfred workflows, you actually need to puts the output, if you don't do this the next script won't contain the expected input.

Download the Alfred workflow

You can directly download the workflow from the Github repository

NOTE: You will be asked to introduce your personal access token.

Final thoughts

I had a really fun time coding this workflow. It was a bit tricky initially since there is not too many examples out there, however the documentation was good enough.

If you liked the workflow please consider giving it a ⭐ on Github.

Happy coding!

Docker for Rails Development

Victor Hazbun — Mon, 24 Feb 2020 04:27:11 +0000

Developing Ruby on Rails applications in large teams could be frustrating because team members use different operative systems, languages, timezones and more.

Docker is a great tool for software development since it synchronises your team with the same setup for everyone who collaborates in your project. That's fantastic!

Why?

Your team will have an stable development environment
Development it's a mirror of Production
No bugs caused by environment
Because Windows XD
New team members will love it!

How?

Let's break this into multiple sections, we will cover everthing needed to build badass Rails applications in Dockerized environments.

Rails in a Docker Container

Running rails in a Docker Container is pretty straightforward, let's begin by creating a Rails application and then configure a custom Docker Image so we can run the Rails application from there.

Rails application

Let's start by creating a new Rails application.

I'm assuming you have Rails 5.2 installed if not please follow one of this Guides.

$ rails new myapp --skip-test --skip-bundle

Docker Image

Now that we have a Rails application, let's create a simple Dockerfile in our Rails application root.
This will help us running our Rails application in Docker.

Dockerfile

FROM ruby:2.5
RUN apt-get update -yqq
RUN apt-get install -yqq --no-install-recommends nodejs
COPY . /usr/src/app/
COPY Gemfile* /usr/src/app/
WORKDIR /usr/src/app
RUN bundle install

Line #1 Sets Ruby 2.5 as the Ruby version for our custom Docker Image
Line #2 Update the Package list
Line #3 Install Node.js (needed for the Asset Pipeline)
Line #4 Copy our Rails application files from our local directory into the container
Line #5 Gemfile caching
Line #6 Sets the working directory for the Docker Image
Line #7 Installs the Ruby gems

Please make sure you are in the Rails application root directory.

$ docker build .

...
Removing intermediate container 487d39dad5ff
 ---> c2183f884d23
Successfully built c2183f884d23

Take a look at the custom image identifier (c2183f884d23) because we will use it next. Your identifier will be different of course.

We should now be able to list all of our available Docker Images.

$ docker images

REPOSITORY          TAG                  IMAGE ID            CREATED              SIZE
<none>              <none>               c2183f884d23        About a minute ago   381MB

As you can see the custom Docker Image is available.

Now that we have created our own Docker Image, it's time to run the application.

Running Rails

To get the application up and running we will execute the following command which basically says "start a containter out of our image (c2183f884d23) and then run rails s -b 0.0.0.0 inside it". The -b tells our Rails server to bind all IP addresses not just localhost.

$ docker run -p 3000:3000 c2183f884d23 rails s -b 0.0.0.0

The Rails server boots up,

=> Booting Puma
=> Rails 5.2.0 application starting in development
=> Run `rails server -h` for more startup options
Puma starting in single mode...
* Version 3.11.4 (ruby 2.5.1-p57), codename: Love Song
* Min threads: 5, max threads: 5
* Environment: development
* Listening on tcp://0.0.0.0:3000

Visit localhost:3000 and you should see Rails welcome page. Yay! You’re on Rails!.

Improve the Docker Image

Using the custom image ID as a reference to start our Rails application is not cool at all because it is not easy to remember this number, let's tag this image so it is easy to remember:

$ docker tag c2183f884d23 app

If we list our Docker images:

REPOSITORY    TAG    IMAGE ID    CREATED    SIZE
app           latest      c2183f884d23        5 days ago          1.03GB

You can use tags to version your custom Docker Image. All about Docker tag command here.

With the new Docker Image name app we can make run the Rails application easily.

$ docker run -p 3000:3000 app rails s -b 0.0.0.0

Notice we are no longer using the Docker Image ID but instead the tag name.

Now let's add a default command to out image, using CMD:

Dockerfile

FROM ruby:2.5
RUN apt-get update -yqq
RUN apt-get install -yqq --no-install-recommends nodejs
COPY . /usr/src/app/
COPY Gemfile* /usr/src/app/
WORKDIR /usr/src/app
RUN bundle install
CMD ["rails", "s", "-b", "0.0.0.0"]

There are some files we can ignore in our Docker Image using .dockerignore and feel free to add more here as needed:

.dockerignore

.git
.gitignore
log/*
tmp/*

Then, let's rebuild the Docker Image with the new CMD instruction:

$ docker build -t app .

Start the Rails application without the rails s -b 0.0.0.0 using the new tag name app:

$ docker run -p 3000:3000 app

So far we have covered Dockerfile, .dockerignore and docker tag.
Next we will switch to Docker Compose!

Rocking with Docker Compose

With Docker Composeyou describe each of your application services.

Introducing docker-compose.yml

version: '3'
services:
  web:
    build: .
    ports:
      - "3000:3000"
    volumes:
      - .:/usr/src/app
    command: bash -c "rm -f tmp/pids/server.pid && bundle exec rails s -p 3000 -b '0.0.0.0'"

We will remove the CMD ["rails", "s", "-b", "0.0.0.0"] line from our Dockerfile since we are running the same command from the docker-compose.yml.

This step is very important since if we don't do it we could get this error:
A server is already running. Check /usr/src/app/tmp/pids/server.pid

Dockerfile

FROM ruby:2.5
RUN apt-get update -yqq
RUN apt-get install -yqq --no-install-recommends nodejs
COPY . /usr/src/app/
COPY Gemfile* /usr/src/app/
WORKDIR /usr/src/app
RUN bundle install

Now, let's build the Docker Image with Docker Compose:

$ docker-compose up --build

Our Rails application is up and running, thanks to Docker Compose, YAY!

Docker Compose useful commands

This Docker Compose commands will help you during development and you will use them everyday!

Builds, (re)creates, starts, and attaches to containers for a service:

$ docker-compose up

Lists containers:

$ docker-compose ps

Managing containers lifecycle:

Displays log output from services.:

$ docker-compose logs [SERVICE...]

Run arbitrary commands in your services:

$ docker-compose exec SERVICE COMMAND

Runs a one-time command against a service.:

$ docker-compose run SERVICE [COMMAND]

Rebuilding a Docker Image:

$ docker-compose build [SERVICE...]

Summary

We learned how to use Docker for Rails development, we covered some key concepts like: Dockerfile and Docker Compose.

Now you should be ready to start developing Rails applications in a Dockerized environment!

Best practices: Avoid race conditions 🚘💥🚗😰

Victor Hazbun — Mon, 24 Feb 2020 04:01:42 +0000

Here are some scenarios and tips to combat race conditions.

Understanding Race Conditions

A computer program is like a horse race. The computer program does several things at the same time, similarly to how several horses run at the same time in a horse race. Each horse represents what is usually called a thread of execution. That way, one such thread may handle network communication, another one may be responsible for redrawing the user interface. In the case of a race condition, the application works properly if a given horse wins the race. For example, the application may work if horse number five wins, but it will crash if any other horse wins the race. Source https://simple.wikipedia.org/wiki/Race_condition.

Real Life Example: Many users booking the same room at the same time

Imagine you have a booking app, let's pretend you a have Room model. The rooms table stores the user_id.

class Room
  def assign_to_user!(user_id)
    self.user_id = user_id
    self.save!
  end
end

It looks fine, right? Nope, if two users make the same reservation at the same time you are probably going to run into serious problems. Here is why..

The first user who submitted the form will open a database transaction and the second user who submitted a millisecond after opened another database transaction. Now we have two opened database transactions, the second transaction has overridden the first. That means, the first user has paid for the room but the second got it, LOL. Just imagine the two hosts discussing in the lobby, you will hear something like this: "No, I booked the room!".

To prevent this you can lock the Room table.

class Room
  def assign_to_user!(user_id)
    # This block is called within a transaction,
    # room is already locked.
    with_lock do
      self.user_id = user_id
      self.save!
    end
  end
end

By locking the table, even if two requests happened at the same time the record won't be updatable UNTIL the DB lock is released, this is called "Pessimistic locking".

Use Pessimistic locking whenever you know this kind of things are most likely to happen in your app. See full documentation and examples here: https://api.rubyonrails.org/classes/ActiveRecord/Locking/Pessimistic.html

Other Scenarios

Here is a small list of cases where you should implement database locking to avoid unwanted race conditions.

Money transactions
Booking flights
Making reservations
Cryptocurrency transactions
Gambling

Conclusion & Next Steps

Think about which parts of your application deserve database locking, hopefully you can patch those areas so all your users have a great experience.

Please let me know if you have questions or want to chat about other kinds of database locking mechanisms like Optimistic Locking.