DEV Community: Bella Xiang

Top 5 Open Source AI Image Generators: Free Tools for Creative Projects

Bella Xiang — Fri, 02 Aug 2024 08:46:54 +0000

AI technology has been rapidly transforming the landscape of creative industries. According to a recent survey, over 50% of respondents believe that AI can significantly enhance written content by improving quality, creativity, and efficiency. This indicates the immense potential AI-driven solutions hold for various content creation contexts. In the advertising and marketing sector alone, 37% of professionals have already integrated AI into their workflows, showcasing a growing trend towards AI adoption in creative fields.

Businesses embracing AI can expect a substantial revenue increase ranging from 6% to 10%. Moreover, creatives are increasingly recognizing the pivotal role of AI in idea generation processes. While traditional methods still dominate the final outcome phase of projects, there is a clear shift towards leveraging AI for innovative ideation.

Open-source tools play a crucial role in democratizing access to advanced technologies like AI image generators. With 83% of creatives having already adopted AI into their practices, the significance of open-source platforms cannot be overstated. These tools not only foster creativity but also encourage collaboration and innovation within the creative community.

In this blog, I will delve into five top open-source AI image generators that are revolutionizing creative projects. From exploring their unique features to understanding how they empower artists and creators, this journey will provide valuable insights into the world of AI-generated imagery.

1. Stable Diffusion

Why Stable Diffusion Stands Out

When it comes to AI image generation, Stable Diffusion emerges as a standout tool in the creative realm. Its open-source nature empowers users with technical acumen to seamlessly download and operate it locally. This accessibility opens doors for a diverse range of applications, from crafting artistic portraits to generating historical renditions and architectural visualizations.

The flexibility offered by Stable Diffusion is truly a game-changer. Users have the freedom to tailor their models according to specific requirements, enabling them to unleash their creativity without constraints. This adaptability sets Stable Diffusion apart, making it a preferred choice for those seeking customized solutions in image generation.

Getting Started with Stable Diffusion

Embarking on your journey with Stable Diffusion requires understanding its basic requirements and setup process. To begin, ensure you have the necessary technical skills to navigate through the installation procedure smoothly. Whether you are an experienced user or a novice exploring AI tools, Stable Diffusion offers an intuitive interface that caters to varying levels of expertise.

To kickstart your experience with Stable Diffusion, consider leveraging its API and Clipdrop service for enhanced functionality. These additional features complement the core capabilities of Stable Diffusion, providing users with a comprehensive toolkit for creating visually stunning outputs.

2. PixArt-Alpha

Unleashing Creativity with PixArt-Alpha

PixArt-Alpha stands at the forefront of innovation in the realm of AI-driven image generation. This groundbreaking Transformer-based Text-to-Image (T2I) diffusion model is poised to redefine the standards of photorealistic synthesis. The fusion of cutting-edge technology and artistic expression culminates in a tool that transcends traditional boundaries, offering creators a canvas where imagination knows no limits.

Key Features and Capabilities

PixArt-Alpha's prowess lies in its ability to transform textual prompts into visually striking images with remarkable realism. By harnessing the power of advanced algorithms, this tool empowers users to bring their ideas to life seamlessly. The model's adaptability ensures that even intricate details are captured faithfully, resulting in outputs that captivate and inspire.

Incorporating PixArt-Alpha into your creative workflow unlocks a world of possibilities. From generating lifelike landscapes to crafting fantastical creatures, the versatility of this tool knows no bounds. Artists and designers alike can explore new horizons, pushing the boundaries of visual storytelling with each stroke of creativity.

Examples of Creative Projects

PixArt-Alpha's impact reverberates across various artistic endeavors, showcasing its transformative influence on digital artistry. Renowned artists and enthusiasts alike have embraced this tool to breathe life into their visions. Whether recreating historical masterpieces or envisioning futuristic landscapes, PixArt-Alpha serves as a catalyst for unparalleled creativity.

One notable testimonial captures the essence of PixArt-Alpha's significance:

Junsong Chen, a visionary in the creative industry, emphasizes, "PixArt-α is not just an evolution; it’s a revolution."

This sentiment echoes the sentiments shared by countless users who have experienced firsthand the revolutionary capabilities of PixArt-Alpha in reshaping the landscape of digital art creation.

3. PiGallery - A Self-Hosted AI Image Generator

In the realm of self-hosted AI image generators, PiGallery emerges as a versatile tool catering to the needs of creators seeking autonomy and control over their image generation processes. The decision to opt for a self-hosted solution stems from the desire for enhanced privacy, customization options, and seamless integration within existing workflows.

Why Choose a Self-Hosted AI Image Generator

The allure of a self-hosted AI image generator lies in the myriad benefits it offers to users. By hosting the tool locally, creators can safeguard sensitive data and intellectual property while ensuring compliance with data privacy regulations. Additionally, self-hosting grants individuals the freedom to customize settings, plugins, and themes according to their preferences, fostering a tailored user experience.

Exploring PiGallery's Features

PiGallery distinguishes itself through its advanced features that streamline image analysis and organization. Leveraging machine learning capabilities, PiGallery2 can extract face regions from photo metadata, enabling users to categorize and search images based on facial recognition. Moreover, its geo-tagging functionality facilitates geographical sorting of images, enhancing accessibility and navigation within extensive image libraries.

Face region extraction for efficient categorization
Geo-tagging for geographical image organization

4. DreamStudio by Stability AI

User-Friendly Interface for Creatives

DreamStudio, developed by Stability AI, stands out as a customizable AI image generator tailored to meet the diverse needs of users and developers. What sets DreamStudio apart is its intuitive user interface that simplifies the image creation process. With fields dedicated to size adjustments, style preferences, negative prompts, and image inputs, users can fine-tune their settings to achieve the precise visual rendition they envision.

Ease of Use and Image Generation Options

Navigating through DreamStudio's interface is a seamless experience, thanks to its user-centric design. The platform offers a range of customization options that empower users to explore various creative avenues effortlessly. Whether adjusting parameters for color schemes, textures, or thematic elements, DreamStudio provides a comprehensive toolkit for crafting unique and captivating visuals.

DreamStudio in Action

Real-world applications of DreamStudio underscore its versatility and impact across different industries. From digital art enthusiasts seeking innovative tools to professionals in need of quick yet high-quality image generation solutions, DreamStudio caters to a broad spectrum of users. The tool's adaptability allows individuals to delve deep into AI technology, experiment with diverse prompts, and witness firsthand the transformative capabilities of generative AI.

Incorporating feedback from users and developers alike, Stability AI continues to refine DreamStudio's features and functionalities. By prioritizing user experience and innovation, Stability AI reinforces its commitment to driving advancements in open-source AI development. As more creators embrace the possibilities offered by DreamStudio, the tool solidifies its position as a go-to resource for those venturing into the realm of AI-driven image generation.

Seamless customization options
Versatile applications across industries

5. Fooocus

The Power of Offline Functionality

In the realm of AI image generation tools, Fooocus shines brightly with its unique offline functionality. The significance of offline capabilities cannot be overstated, especially for users in scenarios where internet connectivity may be limited or unreliable. Fooocus offers a seamless experience by allowing users to create stunning images without being tethered to continuous online access. This feature not only enhances convenience but also ensures uninterrupted creativity, empowering users to unleash their artistic vision anytime, anywhere.

Why Offline Capabilities Matter

The ability to operate Fooocus offline addresses a crucial need for creators who value independence and flexibility in their workflow. By eliminating dependency on constant internet connectivity, Fooocus provides a reliable solution for generating images on the go. Whether you are traveling, working in remote locations, or simply prefer working offline, this feature caters to diverse user preferences and requirements.

Getting the Most Out of Fooocus

To maximize your experience with Fooocus and achieve realistic photo generation results, consider implementing the following tips:

Utilize simple prompts: Opt for clear and concise textual inputs to guide image creation effectively.
Experiment with styles: Explore different artistic styles and themes within Fooocus to diversify your creative outputs.
Fine-tune settings: Adjust parameters such as color schemes, textures, and resolutions to refine the details of your generated images.
Embrace iteration: Iteratively refine your prompts and settings to discover new possibilities and enhance the quality of your creations.

By incorporating these strategies into your workflow, you can harness the full potential of Fooocus and elevate your image generation endeavors to new heights. Whether you are a beginner seeking intuitive tools or an experienced creator looking for innovative features, Fooocus offers a versatile platform for unleashing your imagination with ease.

With its user-friendly interface, low system requirements, and focus on simplicity without compromising quality, Fooocus stands as a beacon of accessibility in the realm of AI image generators. Experience the beauty of effortless image creation today with Fooocus!

MyScaleDB: Powering the Next Generation of AI Image Generators

While the article focuses on open-source AI image generators, it's worth considering the role of a robust vector database in building such kind of tools, and the powerful option is to integrate MyScaleDB as the underlying vector database.

MyScaleDB's strengths lie in its ability to efficiently handle the high-dimensional vector data that is integral to AI image generation models. Built on ClickHouse, MyScale excels in storing the structured and unstructured data in a unified system. By leveraging MyScaleDB as the data storage and retrieval engine, developers can take advantage of its advanced MSTG indexing and SQL vector joint querying capabilities to achieve lightning-fast performance, even with massive datasets.

Moreover, MyScaleDB's SQL-based interface provides a familiar and accessible means of interacting with the underlying data, enabling developers to leverage their existing SQL knowledge and tools. This can significantly simplify the integration process and reduce the learning curve for those transitioning to AI-powered image generation projects. Besides, With the support of text-to-SQL and Self Query in MyScaleDB, users of the AI image tools can directly input the natural language to interact with the database, which is beneficial to the user experience.

As the demand for AI-powered image generation continues to grow, solutions like MyScaleDB will play an increasingly pivotal role in enabling developers to build cutting-edge applications that push the boundaries of what's possible. If you are interested in creating an AI image generator, MyScale offers you 5 million of free vector storage in its development pod, with which you can experience all the advanced features. Welcome to join our Discord to discuss your ideas with us.

MyScaleDB, the Revolutionary SQL Vector Database, Goes Open-Source

Bella Xiang — Wed, 03 Apr 2024 10:16:02 +0000

MyScaleDB, the cutting-edge SQL vector database, is thrilled to announce that it is now open-source as of March 28. This move marks a significant milestone for the AI developer community because it provides a powerful tool to build and scale AI applications like never before.

MyScaleDB is a high-performance, scalable, and cost-effective database that harnesses SQL queries to accelerate vector search and processing. Our team of experienced database engineers has worked tirelessly to create a solution that enables every developer to build production-grade GenAI applications with powerful and familiar SQL.

We believe that the open-sourcing of MyScaleDB gives developers the keys to unlock the full potential to handle the complexities of today’s ever-changing AI world. You’ll have the freedom to customize and enhance the database to suit your specific needs, whether you are building an AI chatbot, a recommendation system, a natural language processing application, or any other Generative AI product and solutions.

Here are some of the key benefits of using MyScaleDB in your AI projects:

Fully SQL-Compatible:
- Fast, powerful, and efficient vector search, filtered search, and SQL-vector join queries
- Use SQL with vector-related functions to interact with MyScaleDB. No need to learn complex new tools or frameworks – stick with what you know and love.
Production-Ready for AI applications:
- A single platform to manage and process structured data, text, vector, JSON, geospatial, time-series data, and more.
- Improved RAG accuracy by combining vectors with rich metadata and performing high-precision, high-efficiency filtered search at any ratio.
Unmatched performance and scalability:
- MyScaleDB leverages cutting-edge OLAP database architecture and advanced vector algorithms for lightning-fast vector operations.
- Scale your applications effortlessly and cost-effectively as your data grows.

Linpeng Tang, CTO of MyScale: "We're thrilled to put the power of MyScaleDB into the hands of developers worldwide. By open-sourcing our technology, we aim to foster innovation and collaboration within the AI developer community, ultimately leading to groundbreaking solutions in AI data management and analytics.”

Wen Dai, General Manager of Solutions Architecture, Greater China, AWS: "Vector data processing is a critical part for LLM infrastructure, while SQL can provide significant scalability and convenience to application developers. MyScale has made notable contributions in this area. With its open source availability, developers will have options to leverage the value of structured data to work with different LLMs for diversified use cases, for better performance, lower cost, and faster innovation paces.”

Ready to start building with MyScaleDB? Head over to our GitHub repository and dive in! We can't wait to see what you'll create.

We're committed to continuously improving and evolving MyScaleDB to meet the ever-changing needs of the AI industry. Join us on this exciting journey and participate in the AI data management revolution!

For the latest updates and to connect with fellow MyScaleDB developers, follow us on Twitter or join our Discord community. Let's build the future of AI together!

Chaos Engineering: Efficient Way to Improve System Availability

Bella Xiang — Mon, 19 Jun 2023 06:03:44 +0000

Author: Zhao Jinyang

Resilience is a crucial requirement for ShardingSphere-Proxy, an essential database infrastructure. Testing and verifying resilience can be efficiently achieved through the use of chaos engineering methodology. To support customized chaos engineering, the ShardingSphere-on-Cloud project is designing and implementing a new CustomResourceDefinition (CRD) called Chaos. This post provides a practical description of CRD’s concept and features, helping the community better understand its potential benefits.

What is chaos engineering

System availability is a critical metric for evaluating service reliability. Numerous methods can ensure high availability, including engineering resilience and techniques, and others. One such technique is chaos engineering, which involves introducing software faults into production systems to enhance availability.

According to Principles of Chaos (2019), the definition of chaos engineering is:

“Chaos Engineering is the discipline of experimenting on a system to build confidence in the system’s capability to withstand turbulent conditions in production.”

In other words, chaos engineering is a practice that aims to enhance system robustness by detecting potential weaknesses in software systems early, ultimately preventing major disruptions or failures.

Why is chaos engineering needed

The complexity of a system can be shown in a linear and nonlinear way as well as reflect how changes in the input of a system affect the output.

A linear system is typically predictable. There are many examples of linear systems in nature, such as simple mathematical functions and physical definitions.

In contrast, the output of a nonlinear system cannot be accurately calculated. In a large distributed program, components interact with each other, and we cannot determine if expected output can be achieved under various inputs.

Currently, most programs are increasingly complex. In common cloud environments, coordinating various components is becoming more challenging (such as Kubernetes, along with the services running on it, like Istio, Envoy, and other software infrastructure).

The complexity and rapid changes inherent to many systems often lead to developers having a narrow understanding of the overall picture. For example, developers behind a mall system may not familiar with the technical details of the infrastructure they adopted. With increased complexity, any single person’s understanding on the model built by the system may become less accurate. Hence, gaining a complete comprehension of a complex system is not realistic.

Chaos is inherent and describes an unknown state in complex systems. Chaos engineering is used to discover chaos in complex systems, learn the behavior of the system, and develop the ability to respond to failures and restore the system to a steady state.

The guidelines and practical ways of chaos engineering

Formulate a hypothesis about steady-state

Every experiment begins with a hypothesis, often taking the form of “even in XYZ circumstances, the system remains in a steady state.” This principle emphasizes the establishment of hypotheses based on defining steady states. Therefore, we should define various indicators of the system’s normal state based on long-term monitoring of the production environment and focus on measurable outputs, rather than internal properties of the system.

When identifying a steady state, it’s often essential to consider the global outputs of the system, such as running logs, performance logs, alerts, and program behavior, and abstract them into steady-state conditions. Having introduced experimental variables (faults), these steady-state conditions should change as expected.

When the system is in the steady state we defined, we should consider that the system can provide services normally to the outside world. In addition, monitoring the steady state is also important so that the system can recover to the steady state in a short period of time.

Introducing diverse real-world events

We ought to introduce events that are real and what we care about such as trying to reproduce faults that occurred in the production environment, such as cache avalanche, service degradation, etc.

Behaviors that would lead to the same fault symptoms should not be introduced, such as occupying all the memory, CPU, or disk of a service instance or ‘killing’ the instance, which system responds to bad requests. Testing should focus on the system’s behavior after a fault occurs, rather than on how to trigger the fault.

Experiments in the production environment

When conducting experiments, we can learn about the relevant behaviors of the system and establish confidence in the system. If we conduct experiments in a test environment, we can only establish confidence in that specific test environment. If there are differences between the production environment and the test environment, we cannot establish confidence in the production environment.

This is because a complex system is a whole, environmental differences between testing and production environments can render testing environment experiments meaningless, causing a “Bullwhip-effect”. However, conducting experiments in the production environment may affect users of the system and cause losses. We need to make trade-offs in the formal environment and let the experimental tools mature in the quasi-production environment before routing the experiments to a small portion of traffic in the production environment.

Automate experiments

When testing massive experiment sets are required, automating the process is more efficient than manually setting experiment environments, introducing faults, and gathering results. Automated experiments save time, run continuously, and can cover a larger number of experiment sets.

When repeat experiments are required, hypotheses are not always true, and they can be expired following iterated software, so periodic conducting regression experiments are needed.

Minimize the blast radius

Safe experiment methods can reduce the risk to the production environment, such as using traffic shadowing or selecting a suitable time period. An indicator in a small variable group is more significant compared to a small control group.

Chaos maturity model

Chaos maturity model provided a model map, based on different positions to measure different types of chaos engineering in practice.

There are two axes on the map, adoption on the X-axis and sophistication on the Y-axis, which can be explored separately:

Adoption

As chaos engineering becomes mature, chaos engineering software needs to achieve a specific level that robustness validation alone can significantly affect the compliance process. However, initial adoption of chaos engineering generally starts from scratch.

Sophistication

Sophistication has some different metrica: provide consultation services and provide a set of tools. Due to the software infrastructure’s diversity, no tool can abstract sophisticated chaos engineering experiment instances in all environments and apply it in reality. Thus, chaos engineering practices were contributed from massive labor inputs, then customized solutions were gradually developed.

Another way to understand sophistication engineering is to consider the system levels and introducing system experiment variables. Experiments typically start at the infrastructure level with killing pods or virtual machines at the initial. During the initial stages of chaos experiments, the common approach is to use methods such as killing pods or virtual machines. As the tools become more sophisticated, chaos injection logic may be introduced into the target system, impacting the requests between services.

Additionally, when experimental variables affect business logic, we can observe more complex experiments. For instance, returning feasible but unexpected request responses to a service can lead to different results by programs. The experiments in the system will be conducted from the infrastructure layer to the application layer, and then to the business logic layer. Moreover, low-granularity experiments such as those that tend to trigger potential faults in the business logic layer are to be prioritized.

Continuous verification

“Continuous verification (CV) is a discipline of proactive experimentation, implemented as tooling that verifies system behaviors.” — Casy Rosenthal

Continuous validation development tools are a prime example of complexity in the chaos maturity model. CV, like CI/CD, addresses the need for increasingly complex operational systems. Due to resource constraints, system developers cannot afford to verify internal plans, and must instead focus on validating the system’s output meets desired expectations. That’s why CV is better than verification and also this is a successful sign of managing complex systems.

There are at least three types of continuous verification: feature testing, data artifacts, and correctness.

Feature Testing: based on the various performance indicators (concurrency, latency deviation, execution speed, etc.), and through observation of actual production traffic, the report and recognition of this test will be established.

Data artifacts: databases and storage applications have various requirements for the characters of writing and retrieving data, such as transaction consistency, idempotence, incorrect data isolation levels, etc.

Correctness: not all correct forms are manifested as a certain state or ideal attribute. In some cases, the interaction between different components must be taken by interface contracts or agreements. When an interface request returns a seemingly correct result that is beyond its judgement logic, unexpected errors may occur. The reason for such issues is that different levels of code are consistent at the logical level but inconsistent between layers.

Open-source chaos engineering platform

Litmus Chaos

Litmus Chaos is a chaos engineering platform that provides cross-cloud services. It’s a CNCF open-source project that many organizations have used. Litmus Chaos’s mission is to help Kubernetes SRE and developers to find weaknesses in non-Kubernetes platforms and applications that run on Kubernetes.

Chaos Mesh

Chaos Mesh is a chaos engineering platform open-sourced by PingCAP. It has a strong capability to orchestrate failure scenarios and provide comprehensive failure simulation types, which allow users to simulate the faults that might occur in production and testing environments and helps them identify potential failures. Chaos Mesh provides comprehensive visual tools to help beginner programmers conveniently run and monitor their own chaos scenarios. Chaos Mesh was developed based on Kubernetes CRD, mainly including three components:

Chaos Dashboard: a visible platform of Chaos Mesh, provides a user-friendly WebUI, allowing users to design, monitor for Chaos, and manage RABC permits.
Chaos Controller Manager: core logical components of Chaos Mesh, able to schedule users’ designed Chaos CR. The component includes many CRD Controllers, such as PodChaos Controller, WorkerFlow Controller, etc.
Chaos Daemon: the main execution component of Chaos Mesh. Chaos Daemon runs as DeamonSet, and holds Privileged access by default (opt-in). Generally, this component interferes with network equipment, file systems, and kernels by invasion to target Pod Namespace.

Chaos Blade

Chaos Blade is a chaos engineering project designed and open-sourced by Alibaba in 2019, which includes the chaos engineering experiment tool Chaos Blade and platform Chaosblade-box. It helps enterprises solve high availability issues during cloud-native processes through chaos engineering.

Chaosblade supports three big platforms, four computing language applications, involves over 200 experimental scenarios, and over 3000 parameters, allowing for fine control of the experimental scope.

ChaosBlade-Box supports the management of experimental tools, and in addition to managing Chaos Blade, it also supports the aggregation of experimental tools from other platforms such as Litmuschaos.

Conclusion

To introduce chaos engineering into a certain system, we can refer to the chaos maturity model and start with simple inputs. In the case of our community, we can agree upon a date for developers of various components in the system to perform a fault test together, record the results to enhance the sense of participation and importance of chaos engineering for contributors. We’ll then observe system behavior, define steady states, and design reasonable chaos experiment plans. These experiments can be conducted in pre-production or production environments to discover and learn new behaviors of the system and enhance the community’s ability to handle faults. Afterwards, we can design automated experiments and use regression testing to ensure the correctness of experimental hypotheses. By the way, Chaos is coming soon to the 0.3.0 version, stay tuned!

Reference

How to Run SQL Trace with ShardingSphere-Agent

Bella Xiang — Fri, 09 Jun 2023 05:28:56 +0000

Author: Jiang Maolin

Apache ShardingSphere, a data service platform that follows the Database Plus concept for distributed database systems, offers a range of features, including data sharding, read/write splitting, data encryption, and shadow database. In production environment, especially in data-sharding scenarios, SQL tracing is critical for monitoring and analyzing slow queries and abnormal executions. Therefore, a thorough understanding of SQL rewriting and query execution is crucial.

What is ShardingSphere-Agent

ShardingSphere-Agent provides an observable framework for ShardingSphere. It is implemented based on Java Agent technology, using Byte Buddy to modify the target bytecode and weave them into data collection logic. Metrics, tracing and logging functions are integrated into the agent through plugins to obtain observable data of system status. Among them, the tracing plugin is used to obtain the tracing information of SQL parsing and SQL execution, which can help users analyze SQL trace when using Apache ShardingSphere-JDBC or Apache ShardingSphere-Proxy.

This post will take ShardingSphere-Proxy as an example to explain how to use ShardingSphere-Agent for SQL tracing.

Two Basic Concepts You Need to Know

Before starting with the article, here are two important concepts that need to be paid attention to first:

Span: the basic unit in a trace. A span is created for each call in the trace and ideentified by a unique ID. Spans can contain some customized information such as descriptive information, timestamps, key-value pairs, etc.
Trace: the collection of spans with a tree structure. In ShardingSphere-Proxy, a trace represents to the full execution process of a SQL statement.

When running a SQL statement in ShardingSphere-Proxy, it goes through parsing, routing, rewriting, execution, and merging. Currently, tracing has been implemented in two critical steps: parsing and execution — with execution oftentimes being the focus. In the execution stage, Proxy will connect to the physical database to execute the actual SQL. Therefore, the information obtained during this stage provides important evidence for troubleshooting issues and fully reflects the correspondence between logical SQL and actual SQL after rewriting.

In ShardingSphere-Proxy, a trace consists of three types of spans:

Span	Description
`/ShardingSphere/rootInvoke/`	This span indicates the complete execution of an SQL statement, and you can view the amount of time spent on executing an SQL
`/ShardingSphere/parseSQL/`	This span indicates the parsing stage of the SQL execution. You can view the parsing time of an SQL and the SQL statements. (It is not available when a `PreparedStatement` is used.)
`/ShardingSphere/executeSQL/`	This span indicates the rewritten SQL is executed. And the time spent on executing is also available. (This span is not available if the SQL doesn’t need to be executed in the backend physical database).

How to use ShardingSphere-Agent for SQL tracing

For the convenience of viewing the tracing data, Zipkin or Jaeger is usually used to collect and display the tracing data. Currently, ShardingSphere-Agent supports reporting trace data to both components. Next, let’s use the sharding scenario as an example to explain how to report data and analyze the SQL trace.

Configuring ShardingSphere-Proxy

Download Proxy from the official website
Create demo_ds_0 and demo_ds_1 under the MySQL database as the storage unit ds_0 and ds_1 .
Start Proxy, and connect to Proxy using a MySQL client tool; create logical database sharding_db, and register the storage units under this database using DistSQL (Distributed SQL). DistSQL is the specific SQL language for Apache ShardingSphere. It is used in exactly the same way as standard SQL, and is used to provide SQL-level operational capabilities for incremental functions. For details, please refer to the DistSQL official document

Use DistSQL to create sharding rule t_order, set ds_0 and ds_1 as storage units, and set the number of shard to 4.

Create table t_order

Finally, there will be tables t_order_0 and t_order_2 created in the physical database demo_ds_0, and t_order_1 and t_order_3 tables in the physical database demo_ds_1.

After ShardingSphere-Proxy is well configured, the next step is to introduce how to report SQL trace data to Zipkin and Jaeger through ShardingSphere-Agent.

Reporting to Zipkin

Deploy Zipkin (please refer to the official website)
Configure agent.yaml to export data to Zipkin

plugins:
 tracing:
   OpenTelemetry:
     props:
       otel.service.name: "shardingsphere" # the service name configured
       otel.traces.exporter: "zipkin" # Use zipkin exporter
       otel.exporter.zipkin.endpoint: "http://localhost:9411/api/v2/spans" # the address where zipkin receives data
       otel-traces-sampler: "always_on" # sampling setting
       otel.metrics.exporter: "none" # close OpenTelemetry metric collection

Restart Proxy and Agent after stopping Proxy (--agentmeans enabling Agent)

./bin/stop.sh
./bin/start.sh --agent

Use a MySQL client tool to connect to the Proxy and execute the following queries — insert, select, update, and delete in sequence.

Visit http://127.0.0.1:9411/zipkin/ (Zipkin UI), and you can see 4 pieces of trace data, which is exactly the same number of SQL queries.

Let’s analyze the trace of the insert query. After finding the trace, you can see the execution details of this query. The Tags information in the /shardingsphere/parsesql/ span shows that the parsed SQL is consistent with the SQL executed on the client.

There are 4 /shardingsphere/executesql/ spans shown in the span table. After reviewing the details, it is found that the following two SQL statements were executed in the storage unit ds_0:

insert into t_order_0 (order_id, user_id, address_id, status) VALUES (4, 4, 4, 'OK')
insert into t_order_2 (order_id, user_id, address_id, status) VALUES (2, 2, 2, 'OK')

The following two SQL statements are executed in the storage unit ds_1

insert into t_order_1 (order_id, user_id, address_id, status) VALUES (1, 1, 1, 'OK')
insert into t_order_3 (order_id, user_id, address_id, status) VALUES (3, 3, 3, 'OK')

Then log in to the physical database to check the corresponding data (after executing the insert query)

Due to the t_order table being partitioned into 4 shards and data with order_id 1 to 4 being inserted, one record will be inserted into each of the t_order_0, t_order_1, t_order_2, and t_order_3 tables. As a result, there will be 4 /shardingsphere/executesql spans. The displayed SQL trace is consistent with the actual execution results. So you can view the time spent on each step through the span and also know the specific execution of the SQL through the /shardingsphere/executesql/ span.

The following is the trace details of the select, update, and delete queries, which are also consistent with the actual situation.

Reporting to Jaeger

Deploy Jaeger (please refer to the official website)
Deploy Proxy
Configure agent.yaml

plugins:
 tracing:
   OpenTelemetry:
     props:
       otel.service.name: "shardingsphere" # the service name configured
       otel.traces.exporter: "jaeger" # Use jaeger exporter
       otel.exporter.otlp.traces.endpoint: "http://localhost:14250" # the address where jaeger receives data
       otel.traces.sampler: "always_on" # sampling setting
       otel.metrics.exporter: "none" # close OpenTelemetry metric collection

Restart Proxy and Agent after stopping Proxy (--agent means enabling Agent)

./bin/stop.sh
./bin/start.sh --agent

Log into Proxy and execute SQL queries under the sharding_db database (this SQL query is same as the ones executed in the Zipkin example)
From http://127.0.0.1:16686/ (Jaeger UI address), you will see 4 trace data, same as the number of executed SQL queries.

Since the executed SQL queries are the same as those in the Zipkin example, the trace data should also be the same. As an example, we will use the trace from the insert query.

From the following picture, their are one parsed span and 4 executed span

Storage unit ds_0 has executed the following two SQL statements

insert into t_order_0 (order_id, user_id, address_id, status) VALUES (4, 4, 4, 'OK')
insert into t_order_2 (order_id, user_id, address_id, status) VALUES (2, 2, 2, 'OK')

Storage unit ds_1 has executed the following two SQL statements

insert into t_order_1 (order_id, user_id, address_id, status) VALUES (1, 1, 1, 'OK')
insert into t_order_3 (order_id, user_id, address_id, status) VALUES (3, 3, 3, 'OK')

By analyzing the span number, the parsing result of SQL statement and the execution process, it is concluded that the whole SQL link is in line with the expectation

Use Sampling Rate

Sampling is very common when the amount of trace data in the production environment is very large. Shown as follows, set the sampling rate to 0.01 (sampling rate of 1%). OpenTelemetry Exporters is used for exporting data here, and please refer to the OpenTelemetry Exporters document for detailed parameters.

plugins:
 tracing:
   OpenTelemetry:
     props:
       otel.service.name: "shardingsphere"
       otel.metrics.exporter: "none"
       otel.traces.exporter: "zipkin"
       otel.exporter.zipkin.endpoint: "http://localhost:9411/api/v2/spans"
       otel-traces-sampler: "traceidratio"
       otel.traces.sampler.arg: "0.01"

Summary

SQL Tracking allows developers and DBAs to quickly diagnose and locate performance bottlenecks in applications. By collecting SQL tracing data through ShardingSphere-Agent and using visualization tools such as Zipkin and Jaeger, the time-consuming situation of each storage node can be analyzed, which helps to improve the stability and robustness of the application, and ultimately enhances the user experience.

Finally, you’re welcome to join ShardingSphere slack channel to discuss your questions, suggestions or ideas about ShardingSphere and ShardingSphere-Agent.