DEV Community: Eyer

What is an F1 score?

Amarachi Iheanacho — Thu, 10 Oct 2024 06:37:35 +0000

Artificial intelligence has integrated into different facets of our everyday lives, from virtual assistants and personalized recommendations to healthcare diagnostics and fraud detection; we are twice as likely to interact with a piece of software or tool powered by AI than we were a couple of years ago. While this is a positive development, it raises an important question: how can we trust the predictions or outputs of AI-powered solutions? This concern is especially important in situations where inaccurate predictions could result in significant losses, both financial and even in terms of human life.

In this article, we will explore performance metrics like the F1 score for evaluating the effectiveness of classification models, how it’s calculated, and why it’s often preferred over other performance metrics like precision or recall when evaluating classification models.

Understanding the importance of F1 score in classification models

Classification models are algorithms that analyze and categorize complex data sets into predefined classes or labels. These models are used across various sectors, such as anomaly detection, medical diagnosis, text classification, and more. For example, in anomaly detection, classification models help label data points as either "anomalous" or "non-anomalous."

Similarly, in medical diagnosis, a classification model might be used to detect cancer by categorizing patient data into "cancerous" or "non-cancerous" groups.

In such examples, “false positives” and “false negatives” in classification models can have serious consequences. So how can we trust the predictions of these models? The F1 score offers one way to evaluate how well a classification model recognizes and categorizes data into different subsets. To fully understand the F1 score, let's explore three important concepts:

The possible outcomes of a classification model
What are precision and recall performance metrics
How precision and recall combine to give a more comprehensive assessment of a model’s performance which is captured by the F1 score.

Now that we've outlined the significance of classification models, it's important to take a closer look at their prediction outcomes. These outcomes form the foundation for performance metrics such as precision, recall, and, more importantly, the F1 score.

Understanding the possible outcomes of a classification model

A classification model prediction typically falls into one of these four categories:

True Positives: These are events or data points that were correctly predicted as positive.
True Negatives: These are events that were correctly predicted as negative.
False Positives: These are events that were incorrectly predicted as positive but were actually negative.
False Negatives: These are events that were incorrectly predicted as negative but were actually positive.

These four outcomes form the basis of precision and recall, which together make up the F1 score.

What are precision and recall?

Now that we understand these four outcomes, let's use them to explain precision and recall.

Precision
The precision performance metric determines the quality of positive predictions by measuring their correctness. In other words, it measures how many of the positive predictions made by the model were actually correct. Precision is calculated by dividing the number of true positive outcomes by the sum of the true positives and false positives.


Precision = True Positives / (True Positives + False Positives)

Example:

To better understand precision, let’s consider a pool of 500 emails, and a spam filter that has been employed to figure out how many of these emails are spam.

Suppose the filter identifies 120 emails as spam, but only 100 of those emails are actually spam. In this case, the precision of the spam filter would be:


Precision = 100 / (100 + 20) = 0.833

This means that 83.3% of the emails that the filter identified as spam were actually spam.

While precision focuses on the accuracy of positive predictions, recall assesses the model's overall ability to identify all actual positive cases.

Recall
Recall, also known as sensitivity, measures a model’s ability to accurately detect positive events. In simpler terms, it indicates how many of the actual positive instances were correctly identified by the model. Recall can be calculated using the formula below:


Recall = True Positive / (True positive + False Negative)

Example:

Let's return to the spam email filter example. We saw that out of the filter’s prediction of 120 spam emails, 100 were indeed spam. However, what if there were actually 200 spam emails in total? Then in this scenario, the recall would be:


Recall = 100/ 200 = 0.5

This means that the filter correctly identified 50% of all actual spam emails.

While precision and recall provide valuable insights into a model’s performance, relying solely on one without considering the other can give an incomplete picture.

Limitations of using precision as a classification metric without recall (and vice versa)

Considering precision without recall, and vice versa, can lead to a misleading evaluation of a model's performance, especially in scenarios where class distribution is imbalanced or where different types of errors (false positives vs. false negatives) have varying consequences.

Limitations of Precision without Recall

Precision alone focuses solely on the correctness of the positive predictions, ignoring how well the model captures all possible positives. A model with very high precision might seem impressive, but if it misses a large number of actual positive instances (low recall), it could be underperforming. This often occurs in cases where a model is extremely cautious about making positive predictions, leading to fewer but more accurate positive results. This cautious approach minimizes false positives but increases false negatives.

For example, imagine a medical diagnosis model designed to detect a rare disease. If the model has perfect precision but low recall, it correctly identifies all the positive cases it flags as having the disease. However, if it only flags 2 out of 50 actual positive cases, its recall is very low. This means that while every diagnosed patient truly has the disease (precision is 100%), the model is missing the vast majority of patients who actually have it, making it unreliable for early diagnosis and treatment.

Limitations of Recall without Precision

Similarly, focusing on recall alone means you're only considering how many true positives are identified out of the total actual positives without regard to how many false positives the model produces. A high recall could indicate the model captures most positive instances, but it might be over-predicting positives, leading to a flood of false positives and reduced accuracy in actual predictions.

Using the medical diagnosis example, imagine a medical diagnosis model with 100% recall that flags every patient as having the disease to ensure it never misses a single case. While the recall is perfect, the precision is incredibly low because many healthy individuals will be wrongly diagnosed. This makes the model impractical, as it would result in unnecessary anxiety and treatments for people who do not actually have the disease.

This highlights the importance of a comprehensive metric combining precision and recall—the F1 score.

What is an F1 score?

An F1 score can be understood as the harmonic mean of precision and recall, combining both these metrics into one comprehensive assessment that neither performance metric can offer alone.

The F1 score is described as the harmonic mean of both precision and recall for two important reasons:

The F1 score gives both of these metrics equal weights, ensuring that a good F1 score signifies that the model has a good balance between precision and recall.
Unlike the arithmetic mean, the harmonic mean prevents a high precision score from disproportionately affecting the overall F1 score when recall is low, and vice versa.

The F1 score can be calculated as:


F1 - score = 2 * (precision * recall) / (precision + recall)

So using the original example of the spam filter, with a precision of 0.8333 and a recall of 0.5, the F1 score of the spam filter would be:


F1 score = 2 * (0.8333 * 0.5) / ( 0.8333 + 0.5 )

F1 score = 0.625

Calculating a model's F1 score can provide a clearer, more balanced measure of its performance, especially in cases where both precision and recall are critical.

Interpreting the F1 score

Similar to most performance metrics, the F1 score ranges from 0 to 1, with 0 representing the worst possible score and 1 representing the best possible score a model can get.

A high F1 score indicates that the model has good precision and recall, showing a well-balanced performance. Conversely, a low F1 score may suggest a trade-off between precision and recall or indicate that the model performs poorly on both metrics.

This comprehensive insight provided by the F1 score is particularly crucial in anomaly detection, as it helps evaluate the model's ability to accurately recognize and identify anomalous events.

F1 score in anomaly detection

Anomaly detection, once a labor-intensive process, has become much more efficient with the rise of artificial intelligence. Advanced tools like Eyer, an AI-powered anomaly detection platform, have streamlined this process by automating the identification of unusual data patterns.

At its core, anomaly detection involves analyzing data to identify patterns or behaviors that deviate significantly from the norm. These deviations, often referred to as anomalies or outliers, can signal critical events such as fraud, system failures, or network intrusions. By using Eyer's sophisticated algorithms, these anomalies can be detected earlier and with greater accuracy, enabling organizations to respond to potential threats in real-time.

Given the potential consequences of relying on ineffective anomaly detection tools, it’s crucial to trust the performance of platforms like Eyer. One way to measure this trust is through the F1 score, which provides valuable insights into the balance between precision and recall.

For a deeper dive into Eyer's performance, including its F1 score testing results, check out the official documentation and read all about Eyer’s findings on the F1 performance testing of the core algorithm of Eyer.

In summary

Many of the artificial intelligence models we encounter in our daily lives are classification models. These models help us determine whether data has specific characteristics, ranging from something as simple as identifying spam emails to more critical applications like diagnosing cancer in patients.

Since we often don’t know the correct answers to the questions posed by classification models, it’s essential to trust these systems to make accurate predictions and draw the right conclusions from the data. This is where the F1 score comes into play.

The F1 score offers a balanced evaluation of a classification model’s performance by considering both precision and recall. Its value lies in providing a comprehensive measure that neither precision nor recall can fully capture. This makes the F1 score particularly vital in high-stakes scenarios like anomaly detection and medical diagnosis, where both false positives and false negatives can have serious consequences. By understanding and calculating the F1 score, we gain deeper insights into the effectiveness of AI-powered classification models, allowing us to develop more reliable and trustworthy systems. Tools like Eyer, which incorporate the F1 score into their evaluations, demonstrate how this metric can enhance decision-making in real-world AI applications.

Ultimately, using the F1 score not only helps validate the performance of these models but also ensures that they align with the critical needs of various sectors. Whether in healthcare, finance, or cybersecurity, understanding the strengths and weaknesses of classification models through the F1 score can lead to better outcomes and increased confidence in automated decisions. As reliance on AI grows, prioritizing robust evaluation metrics like the F1 score will be essential for building the next generation of intelligent systems that we can trust.

Lastly, check out Eyer for an F1 score-approved, AI-powered anomaly detection tool to monitor your systems.

The role of baselines in anomaly detection

Amarachi Iheanacho — Mon, 29 Jul 2024 08:51:15 +0000

Artificial intelligence and machine learning are quickly making their way into every facet of life, including art, customer service, engineering, and, more recently, anomaly detection, particularly through tools like Eyer.

Anomaly detection was once a repetitive and labor-intensive task, involving countless hours of peering into and analyzing large datasets to identify irregularities or anomalies. However, tools like Eyer now leverage artificial intelligence to automate the process of reading and analyzing large datasets to detect anomalies. But how do you determine if a data point is an anomaly? What constitutes normal behavior? These are the questions that baselines provide some answers to, and this article explores what baselines are and how they are used in anomaly detection and other industries.

What is a baseline?

In anomaly detection, baselines serve as reference points or models that represent the normal behavior of a system or dataset under normal conditions. These baselines are crucial for identifying deviations in the data that may indicate anomalies or outliers. A baseline includes lower and upper boundaries, creating a band within which a metric is expected to stay under normal conditions.

Baselines are typically created using historical data. They can be derived from the mean or median of the dataset. Alternatively, you can define baselines using percentiles. For example, any data point outside the 5th or 95th percentile, which in this case are the lower and upper threshold of the baseline, might be flagged as an anomaly.

Additionally, you can also derive baselines with learning models like linear regression or decision trees. These models can capture relationships in the data and highlight deviations from those relationships. Additionally, clustering algorithms like K-means can be used to define normal clusters of data points. Points that don't fit well into any cluster can be considered anomalies.

While baselines are normally derived from historical data, you must note that they are typically subject to change and, therefore, continuously update as new data flows in.

Now that you understand baselines let's dive into the various methods for identifying baselines for your dataset.

What is baseline detection?

As the name suggests, baseline detection is the process of discovering what a baseline for a dataset is. Introduced briefly in the previous section, there are different methods of baseline detection:

Statistical methods: These techniques rely on statistical properties of the data to define a range of normalcy; these techniques include:
- Mean and standard deviation: This approach defines a normal range based on the mean value and its standard deviation. Data points outside a certain number of standard deviations from the mean can be considered anomalies.
- Percentiles: This approach defines normal behavior using percentiles. For example, the 5th percentile and the 95th percentile might represent the lower and upper bounds of normal behavior. Points that fall outside this range are flagged as anomalies.
Time series analysis: When dealing with data collected over time, specific methods can be used to identify the underlying baseline trend:
- Autoregressive models: These models predict future values based on past data points, essentially creating a baseline for what the next data point should look like.
- Moving average: This method smooths out short-term fluctuations by averaging a series of past data points. This helps highlight the longer-term trends, making it easier to identify deviations from the baseline.
Machine learning models: Machine learning offers powerful tools to automatically learn the baseline from your data. Some of these tools are:
- Simple models: Linear regression, for instance, can establish a baseline by capturing the underlying relationships within the data. Deviations from this baseline might indicate anomalies.
- Clustering: Clustering algorithms like K-means can group similar data points together. Points that don't fit well into any cluster are potential outliers or anomalies.

Why does baseline detection matter?

Baselines and baseline detection are important for different applications. Some of these applications are:

Anomaly detection: This is one of the primary use cases that comes to mind when discussing baseline detection. By identifying data points or events that stray significantly from the established norm, anomaly detection helps us spot potential problems. This is crucial in industries like observability, where tools like Eyer leverage baselines to flag anomalies for further investigation.
Quality control: In manufacturing processes, baselines can be established for various parameters like temperature, pressure, or component dimensions. Baseline detection helps identify products deviating from these expected values, potentially indicating defects. This allows for early intervention and ensures product quality.
Predictive maintenance: Baseline detection can be used to monitor equipment performance over time. By establishing baselines for normal operating parameters such as vibration levels, temperature, and energy consumption, deviations can be identified before they become critical failures. This allows for proactive maintenance, minimizing downtime and repair costs.

Finding these deviations from the norm is precisely what makes baseline detection so valuable. Next, let's take a closer look at a specific use case–anomaly detection with Eyer–dissecting how this new age tool approaches baselining.

Eyer’s approach to baselining using multiple baselines

Eyer is an AI-powered observability tool that leverages baselining for discovering anomalies in a system. Eyer approaches baselining in a very interesting way: it understands that each metric is unique and caters to each as such. For each unique metric, Eyer builds baselines using a combination of autoregressive and clustering models. These baselines, which are built from historical data, consist of upper and lower thresholds.

The term "baselines" is intentional because Eyer can build up to three baselines for a single metric: a primary (or main) baseline and one to two secondary baselines. These baselines can account for different normal behaviors of the same metric on the same day. For example, on some Mondays at noon, CPU utilization might be at 30%, while on others, it could be at 70%, and both are considered normal. However, if 30% utilization is slightly more frequent, it will be the primary baseline, with 70% as a secondary baseline.

The main baseline represents the most frequent behavior and is considered anomaly-free. The secondary baselines represent less frequent behaviors that could still be normal but might occasionally conceal some anomalies.

The thresholds that makeup baselines are learned automatically and are dynamic. They are learned and relearned based on past behaviors, and these thresholds are adopted and learned if any changes occur in the system. So, there is no need for manual actions to set up the monitoring systems, as the AI algorithm learns by itself.

But what role does baselining play in an Eyer anomaly alert?

How does Eyer build out an anomaly alert using baselining?

With these multiple baselines defining normal behavior, it becomes easier to spot anomalies in the data.

It is easy to think of any data point that exists outside the established baselines as an anomaly, but it isn't always marked as one. The data point behavior needs to meet a couple of requirements before being classified as an anomaly.

The verification phase determines whether a deviation is an anomaly. In the first part of the verification phase, some deviations can be ruled out through trend analysis. For example, if the data points are only slightly outside the baselines but the overall trend appears normal, they are not considered deviations and thus not considered anomalous.

After this, a 15-minute verification window is used to monitor data for anomalies. If data deviates from normal behavior for at least 8 minutes within this window, that behavior is classified as anomalous, and the corresponding data point is flagged as an anomaly.

Conversely, if a data point falls outside the baseline for less than 8 minutes within the 15-minute verification window, the anomaly is considered closed.

However, identifying a data point as anomalous is just the beginning. The next step is figuring out how anomalous that data point really is.

Classification of anomaly alerts
An alert can include anomalies on several metrics. Each anomaly on each metric has an assigned severity. The overall severity of the alert is based on the severity of the anomalies contained in the alert.

The severity of the anomaly on a single metric
After confirming a data point as an anomaly, Eyer assigns it a weight based on how significantly it deviates from the baselines. These weights are categorized as follows:

Maximum weight: A data point receives a maximum weight of 2 if it exists far outside all predefined baselines.
Medium weight: This weight, valued at 1, is assigned to a data point that exists beyond the primary baseline but remains within one of the secondary baselines.
Zero weight: When a data point temporarily returns to the main baseline after deviating, it receives a weight of zero.

Depending on how long an anomaly remained outside the main or secondary baselines—these weighted deviations are averaged to form the anomaly's history. This average of weighted deviations is then translated into an anomaly score ranging from 0 to 100, where 0 indicates a critical anomaly, and 100 indicates an anomaly-free state.

This anomaly score, which you can refer to as AS, is then used to describe the severity and likelihood of behavior in a data point being anomalous and potentially impactful. The higher the AS, the less likely the behavior is anomalous. Here's a breakdown of what the AS signifies:

AS > 85: No anomaly. Anomaly scores above 85 indicate that the behavior in the data point can be thought of as primary expected behavior, with minor deviations.
60 < AS <= 85: Low severity. If the anomaly score is greater than 60 and less than or equal to 85, it indicates a low-severity anomaly. This means the data point exhibits minor anomalous behaviors similar to those observed in recent days, weeks, and months. Although the likelihood of the behavior being an anomaly is low, it may occasionally conceal anomalous behavior.
30 < AS <= 60: Medium severity. If the anomaly score is between 30 and 60, it indicates a medium-severity anomaly. This means that the data point behavior may be anomalous but also resembles patterns seen previously, making it less certain as an anomaly.
AS <= 30: Severe. If the anomaly score is less than or equal to 30, it indicates that the anomaly is severe. This means that there is a prevalence of new unseen behavior.

In addition to classifying anomalies by their severity, another perk of Eyer and its anomaly detection is that the metrics are not only learned in isolation. Eyer also uses correlations to group related metrics and their anomalies together, combining them in a single alert and making it easier for root cause analysis.

Correlations in Eyer alerts
Correlations help describe the degree to which two or more variables move in relation to one another. In Eyer, correlations help identify how different metrics influence each other or exhibit similar patterns.

Most metrics have a natural correlation. For example, Process CPU is correlated with the number of executions. This is because each execution of a process consumes CPU resources. As the number of executions increases, the cumulative CPU load from these executions also increases.

After using these baselines to identify anomalies in a metric, determining the severity of those anomalies, and understanding which metrics might be affected by correlations, Eyer packages all this information and sends it out in a comprehensive and succinct alert.

You can see an example of an Eyer anomaly alert in the code block below:

{
  "new": [],
  "updated": [
    {
      "severity": "medium",
      "started": "2024-06-26T18:43:00Z",
      "ended": null,
      "updated": "2024-06-26T19:27:00Z",
      "id": "667c6193d58419f64f4cb403",
      "items": [
        {
          "node": {
            "id": 64,
            "name": "Operating System. undefined",
            "system": {
              "id": 1,
              "name": null
            }
          },
          "metrics": [
            {
              "id": "2ce746c5-1ee3-45d1-b23f-bae56bc5d51a",
              "name": "Committed Virtual Memory Size",
              "metric_type": "int",
              "aggregation": "avg",
              "severity": "severe",
              "started": "2024-06-26T18:42:00Z",
              "updated": "2024-06-26T19:12:00Z"
            },
            {
              "id": "5523ee20-2af2-4b8e-8390-3d2cb4410018",
              "name": "System CPU Load",
              "metric_type": "double",
              "aggregation": "avg",
              "severity": "medium",
              "started": "2024-06-26T19:25:00Z",
              "updated": "2024-06-26T19:26:00Z"
            },
            {
              "id": "a59df24a-e9ec-4c4c-a087-ea1375d4b9c7",
              "name": "Process CPU Load",
              "metric_type": "double",
              "aggregation": "avg",
              "severity": "medium",
              "started": "2024-06-26T19:26:00Z",
              "updated": "2024-06-26T19:27:00Z"
            }
          ]
        }
      ]
    }
  ],
  "closed": [
    {
      "severity": "low",
      "started": "2024-06-26T18:49:00Z",
      "ended": "2024-06-26T19:37:00Z",
      "updated": "2024-06-26T19:37:00Z",
      "id": "667c62f7d58419f64f4cb426",
      "items": []
    }
  ]
}

In the alert above, you have the new alerts array, the updated array, and the closed array of alerts. Check out Alerts- structure and data explained, to understand the structure of the alerts.

According to this alert, an anomaly update has happened in the Operating system node.

This anomaly alert has an overall medium severity because it includes one severe anomaly in the Committed Virtual Memory Size metric. The other metrics in the alert, System CPU Load, and Process CPU Load, have medium anomalies.

The metrics array, which contains both affected and correlated metrics, shows anomalies in the Committed Virtual Memory Size, System CPU Load, and Process CPU Load metrics.

Conclusion

This article has helped you understand the role that baselining plays in machine learning, specifically anomaly detection using historical data.

While "baseline" might seem like a simple reference point, it is the foundation upon which many crucial models and their results are built. Anomaly detection tools like Eyer use baselines to determine if a data point's behavior is anomalous and to gauge the extent of the anomaly. This discernment sets the stage for proactive monitoring and timely intervention, ensuring system reliability and performance.

To learn more about Eyer baselines and start using the Eyer anomaly detection solution, visit the Eyer website.

How to use Eyer and Grafana to query and visualize anomalies in CPU and memory metrics

Amarachi Iheanacho — Mon, 29 Jul 2024 08:43:03 +0000

Anomaly detection, also known as outlier detection, is the practice of identifying data points that deviate significantly from the rest of a data set. Traditionally, this was the domain of statisticians and analysts who spent hours poring over data to find these anomalies. However, like many fields, anomaly detection has evolved over time, leading to the development of solutions like Eyer.

With the rise of machine learning (ML) and artificial intelligence (AI), ML algorithms can now automatically learn underlying patterns within vast datasets, process the data, and effectively identify anomalies that might escape even the most trained human eye.

This article introduces Eyer as the AI-powered anomaly detection tool under review and demonstrates how it can be used to identify anomalies in CPU and memory metrics in a host server or machine.

Prerequisites

To get started with the tutorial, you must have the following:

The Eyer connector agents installed on production or production-like hosts. If you have not installed these agents, refer to the Eyer documentation for installation instructions.
The installed agents must be running continuously for at least a week. This allows the Eyer machine learning pipeline to learn the normal behavior of your Boomi integrations. For more information, refer to the official documentation on Onboarding, preprocessing, and filtering data.
A Boomi Atom installed locally.

Understanding what Eyer is and how it works.

Eyer is an AI-powered observability tool that provides deep insights into your Boomi integrations. It utilizes machine learning to analyze various metrics and identify unusual patterns or data points that deviate significantly from the norm. This anomaly detection capability helps you proactively address potential issues before they impact your integrations.

To gather and deliver data to Eyer's machine learning pipeline, the connector employs a range of agents, including web servers (like Jetty or Tomcat), Jolokia, and Telegraf. Each agent plays a crucial role in this process, for example:

The web server hosts and serves the Jolokia agent. During installation, you can choose to use Apache Tomcat to serve the Jolokia agent instead of Jetty or any other preferred server.
The Jolokia agent helps monitor and manage Java applications through a web browser. It acts as a bridge, allowing you to access and control parts of your Java program using simple web requests, and returns the information in an easy-to-read format (JSON).
The Telegraf agent collects and sends metrics and events from various sources to different databases and systems. It will be responsible for collecting data from your Boomi Atom and sending it to the machine learning pipeline.

Eyer’s anomaly detection works because of the continuous data stream from your Boomi Atom to Eyer’s machine learning pipeline. It is important to note that the remaining events in this tutorial, which involve querying and visualizing anomalies, occur after anomaly detection has been enabled on the Boomi Atom, requiring at least 7 days of a steady data stream.

Simulating stress for the servers holding the Boomi Atom

Once the Eyer team confirms your anomaly detection enablement, you can begin querying for anomalies in your environment.

This guide simulates a production environment by running a Windows virtual machine continuously (24/7) for at least a week. It also injects anomalies by increasing the CPU load on the virtual machine hosting the Boomi Atom you're monitoring.

Since this guide uses a Windows operating system virtual machine, it utilizes the Windows tool CpuStress v2.0 to maximize CPU utilization.

While maximizing CPU load offers a valuable way to understand Eyer's capabilities, you can introduce anomalies across different Boomi Atom metrics, including memory, disk usage, and system load.

Getting started with CpuStres
To get started with CpuStres, download the executable file from the CpuStres v2.0 download page. Once the download is complete, extract the CPUSTRES.zip file and run it to open the CPU Stress modal.

You will see your CPU cores in this modal. To activate three or four of these cores, click on each one, navigate to the Thread tab, and select the Activate button from the Thread dropdown.

Additionally, set the Activity level to high to increase the load on the CPU cores. To ensure these CPU load stress tests are registered as an anomaly, keep CpuStres running for at least 8 minutes.

Configuring the Eyer connector to query the data on anomalies

After simulating anomalous behavior in your host machine, use the Eyer connector to query the information on these anomalies.

To query this data on anomalies, log into your Boomi Atmosphere account. Go to the Integration page, click Create New, and select Process from the dropdown menu.

This action will open the Start Shape sidebar. Choose the Connector radio button. Next, in the Connector field, search and select the Eyer-Partner connector.

Next, click the + button in the Connection field to open the connection page.

Leave the Server and the Eyer authentication key fields as their default values.

In the Custom Authentication Credentials field, click the Encrypted button and fill it out with your Eyer authentication key.

Click the Save and Close button to return the Eyer connector sidebar.

Creating Eyer-Partner operation
In the sidebar, select a Get action and then click the + button on the Operation field to create a new Eyer operation.

Clicking the + button opens up the Eyer operation’s page. On this page, click the Import Operation button to create a new operation.

This action opens up the Eyer-Partner Connector Operation Import modal. Fill out this modal with the following information:

Atom: Select the Atom you are running the process in from your dropdown
Connection: Select the Eyer connection you made for this process

Click on the Next button to save your operation. Then, select the Object Type that fits your purpose:

Anomalies returns a list of anomaly alerts grouped by correlation
Anomalies with metrics return a list of anomaly alerts grouped by correlation metrics, including their respective values and baseline values at the time of the alert (new/updated)

In this example, we select the Anomalies with metrics object type. Click on the Next button to save your Object Type preference, and click the Finish button to see your Eyer response profile loaded on your Operation page.

Next, you need to define the operation values. These values define the information required in anomaly alerts. For the Eyer-Partner connector, you can define operation values using either the Options or Dynamic operation property.

Options are great for static operation values.
Dynamic operation properties are better when the start and end values are always changing. To learn more about the distinction between options and dynamic operation properties, check the official documentation on Configuring the Eyer connector.

This guide will use Dynamic Operation properties to determine the Operation’s value.

Setting operation value with Dynamic operation properties
To set up the Dynamic Operation properties, navigate to the Dynamic Operation Properties tab and click the Add Dynamic Operation Property.

This action opens up a Parameter Value modal; in this modal, select the following options:

Input → Query from
Type → Date/Time
Date Mask → yyyy-MM-dd’T’HH:mm:ssZ
Date Type → Last Successful Run Date

These Parameter Value options tell the Eyer-Partner Connector to start the query for anomalies since the last test.

Click the OK button to return to the Dynamic Operation Properties tab.

Next, create a new Dynamic Operation Property, filling in the Parameter Value with the following information:

Input -> Query to
Type -> Date/Time
Date Mask -> yyyy-MM-dd’T’HH:mm:ssZ
Data Type -> Current Date

These values tell the Eyer-Partner Connector to query the current date for anomalies.

Click the OK button to save the parameter value and return to the sidebar.

Click the OK button to save the Dynamic Operation Property configuration and return to the Boomi process canvas.

Sending out the email
With Boomi, you have multiple options for receiving these anomalies. This guide uses the Boomi Mail connector. To learn how to configure the Mail connector, check out the Boomi Mail connector documentation.

Here are the anomalies from the CPU stress test received in the mail.

{
  "new": [],
  "updated": [
    {
      "severity": "medium",
      "started": "2024-06-26T18:43:00Z",
      "ended": null,
      "updated": "2024-06-26T19:27:00Z",
      "id": "667c6193d58419f64f4cb403",
      "items": [
        {
          "node": {
            "id": 64,
            "name": "Operating System. undefined",
            "system": {
              "id": 1,
              "name": null
            }
          },
          "metrics": [
            {
              "id": "2ce746c5-1ee3-45d1-b23f-bae56bc5d51a",
              "name": "Committed Virtual Memory Size",
              "metric_type": "int",
              "aggregation": "avg",
              "severity": "severe",
              "started": "2024-06-26T18:42:00Z",
              "updated": "2024-06-26T19:12:00Z"
            },
            {
              "id": "5523ee20-2af2-4b8e-8390-3d2cb4410018",
              "name": "System CPU Load",
              "metric_type": "double",
              "aggregation": "avg",
              "severity": "medium",
              "started": "2024-06-26T19:25:00Z",
              "updated": "2024-06-26T19:26:00Z"
            },
            {
              "id": "a59df24a-e9ec-4c4c-a087-ea1375d4b9c7",
              "name": "Process CPU Load",
              "metric_type": "double",
              "aggregation": "avg",
              "severity": "medium",
              "started": "2024-06-26T19:26:00Z",
              "updated": "2024-06-26T19:27:00Z"
            }
          ]
        }
      ]
    }
  ],
  "closed": [
    {
      "severity": "low",
      "started": "2024-06-26T18:49:00Z",
      "ended": "2024-06-26T19:37:00Z",
      "updated": "2024-06-26T19:37:00Z",
      "id": "667c62f7d58419f64f4cb426",
      "items": []
    }
  ]
}

This alert has anomalies whose values have been updated. This is because the environment used in this tutorial has been running for a while and has experienced different anomalies and changes to these anomalies. In your environment, these anomalies might appear in the new object.

The anomalies are in the Operating System node. A node is a group of metrics that work together. Refer to the official documentation to understand the list of nodes and the metrics underneath these nodes.

This node has a couple of anomalies on the following metrics:

Committed Virtual Memory Size: This metric is flagged as severe, indicating that the Committed Virtual Memory size metric significantly deviates from past observed behavior and has the highest likelihood of being a disruptive anomaly. To learn more about the severity property, check out the official documentation on Alerting.
System CPU Load: This metric indicates the overall CPU load on the system. It has a medium severity, meaning that the metric occasionally deviates from the previously observed and learned behavior. A medium severity indicates a moderate probability that this is an anomaly.
Process CPU Load: This metric indicates the CPU load of a specific process and has a severity value of medium.

Additionally, you can see a Closed Anomalies array containing a previously detected low severity anomaly that has been resolved.

Deciphering the data using Grafana

Now that you have received the alert about the anomalies in the host system let's view these alerts on a Grafana dashboard.

Grafana is fantastic for many reasons, one of which is that it simplifies the visualization of your JSON data and aids in monitoring system metrics. To learn more about how Grafana can benefit you, check out this article on Observability with Grafana and Eyer.

Setting up Grafana with Eyer
To set up and connect Grafana to visualize your Eyer data, follow these steps:

1.Log in to your Grafana account.
2.Launch Grafana Cloud:

In the Grafana Cloud portal page, click the "Launch" button. This will take you to your Grafana Cloud page. Click the "Launch" button again to go to the Dashboard page.

3.Add a new connection:

On the Dashboard page, navigate to the left-hand side menu and select the "Add new connection" tab.
In the "Add new connection" page, select "InfluxDB."

4.Add a new data source:

On the InfluxDB page, click “Add new data source.”

5.Configure the data source settings: On the settings page, set the following configuration fields:

Set “Query language” to “Flux.”
Set “URL” to “https://westeurope-1.azure.cloud2.influxdata.com.”
In the “Auth” section, turn off basic auth.
Under “InfluxDB Details,” enter the “Organization” and “Token” values you received with your InfluxDB details.

6.Save and test the connection:

Click “Save and test.” If the connection works, a notification should pop up, and you can proceed to the next step.

If it doesn't work, double-check your settings. If the problem persists, send a support request or contact us on the official Eyer Discord channel.

7.Create a dashboard:

Click the "building a dashboard" link in this connection pop-up. This will take you to the “Start your new dashboard by adding a visualization” page.

8.Import a dashboard:

On the next page, click “Import a dashboard” to go to the Import dashboard page.
Click “Upload dashboard JSON file” on this page and select the JSON file you received with your InfluxDB details.

You should now have a dashboard containing the core metrics monitored by Eyer, including multiple baselines.

In the image above, you can see some of the core metrics monitored by Eyer, including their data points and primary and secondary baselines, represented in different colors. These data points and baselines are plotted on a time axis (x-axis) and a value axis (y-axis). The different colored lines and shaded areas represent the following:

Data points on your metrics are in yellow
The primary baselines, which indicate the main behaviors of your system, are in red.
The secondary behaviors and baselines are distinguished by two different shades of blue. For more information on baselines and the behaviors they represent, refer to the documentation on Onboarding, preprocessing, and filtering of the data.

It is important to note that the purple shading in the graph results from the overlap between the main and secondary baselines, which are red and blue, respectively.

Understanding the anomalies from the Grafana dashboards
To understand how to recognize anomalies from the Grafana dashboards, this section will look at the Committed Virtual Memory Size, Process CPU Load, and the System CPU Load metrics.

In the images above, the Process CPU Load and System CPU Load data points exist outside the primary baseline (red-shaded areas) but within the secondary baselines, coinciding with their medium severity level.

However, an interesting anomaly observation is the Committed Virtual Memory Size, with data points existing outside the main and secondary baselines. This observation coincides with its severity level.

In summary

Machine learning and artificial intelligence is changing almost everything around us, especially the monitoring and observability space. With modern software development becoming more and more complex, AI-powered insights can be the difference between quickly identifying and resolving issues or experiencing prolonged downtime and performance degradation.

This article demonstrates the power of AI-powered observability by walking through the process of injecting anomalies into a system, querying these anomalies with the Eyer connector, and visualizing them using Grafana.

However, this is just the beginning of what AI-powered insights can do for you. To learn more about Eyer and Grafana and to get started, check out the official Eyer documentation.

Observability with Grafana and Eyer

Amarachi Iheanacho — Sat, 22 Jun 2024 10:57:48 +0000

Modern infrastructure is becoming increasingly complex, with microservices, cloud deployments, and distributed architectures making it challenging to understand how everything functions together. This complexity has begged the need for the unparalleled visibility that observability promises.

Observability provides a comprehensive view of your system, allowing you to identify issues before they escalate. Tools like Eyer play a crucial role in achieving observability. Eyer helps gather and analyze system data, revealing anomalies, affected nodes, and potential future problems. With this insight, you can quickly pinpoint issues using Eyer, leading to less downtime and a smoother user experience.

However, the raw data from Eyer might be difficult for non-technical individuals or teams to understand. This is where Grafana comes in. As a powerful visualization tool, Grafana transforms this data into clear and insightful dashboards, making it accessible to everyone who needs it.

This article explores Eyer, its importance in modern observability discussions, and the added value of integrating it with Grafana.

Understanding Eyer and its capabilities

Eyer is an AI-powered observability tool that provides insights into your Boomi integrations. Boomi has become an integration superpower, uniting diverse applications and data sources with its simple and intuitive drag-and-drop design. With Eyer, you can take that impeccable user experience to the next level.

By installing and using the Eyer connector, you can collect data from your Boomi integrations, send it to the Eyer machine learning pipeline, and gain insights into what's wrong with your Boomi process.

The machine learning pipeline learns the user Boomi Atom’s behavior and establishes what normal behavior or baselines are for your Atom. So, any significant and prolonged deviations from the normal behavior are flagged.

For example, if your Boomi Atom is using more memory than normal or CPU utilization is higher than usual, the Eyer connector will send you a JSON alert. You can choose to receive this alert conveniently via email or even as a file saved directly on your host machine, thanks to the flexibility of Boomi's connectors.

JSON format alerts are advantageous for many reasons: they are structured, human-readable, lightweight, language-agnostic, and can be easily integrated into various systems for automated processing and response. However, while JSON alerts do not have inherent visualization capabilities, tools like Grafana lend them the ability to visualize data over time.

Grafana: The solution to all your visualization problems

Grafana is an open-source analytics and interactive visualization web application tool used to monitor application performance. This section explores how Grafana allows Eyer users to query, visualize, and understand their JSON alerts.

Benefits of the Grafana integration with Eyer

By integrating Grafana with Eyer, developers have access to powerful visualization capabilities. Some key benefits of this integration include:

Visualization: If you only remember one thing from this article, remember that Grafana is the king of data visualization. This visualization is amazing for democratizing data use. Visual representation allows users to quickly understand the data and see patterns and trends that might be missed when looking at raw JSON.

Historical analysis: Another powerful feature of Grafana is its ability to store and visualize historical alert data. Imagine trying to understand a month's worth of system activity by manually reviewing individual JSON alerts; this gets tiring quickly. Grafana offers a much better solution. By aggregating all the data for a specific metric into a single graph, you can easily see trends over days, weeks, or months. This historical view allows you to identify potential issues, forecast future resource needs, and gain insights into long-term performance patterns.

Collaboration and sharing: Grafana makes it easy to share dashboards and visualizations with team members. That way, more people can watch and understand what's happening in your systems. These shared insights make it easier for teams to work effectively to address and resolve issues.

Open-source, extensible: As an open-source tool, Grafana is free to use and allows users to access and modify the source code, enabling customization to meet specific needs. Its extensibility is one of its core strengths, with a vast ecosystem of plugins available that extend its functionality, including integrations with a wide range of data sources, custom visualizations, and alerting mechanisms. This flexibility makes Grafana adaptable to various use cases and industries.

Large community support: Grafana benefits from a large and active community of users and developers. This community support is invaluable, providing a wealth of shared knowledge, tutorials, forums, and plugins. The collaborative nature of the community ensures continuous improvements and updates, keeping Grafana at the forefront of monitoring and visualization tools. This robust support network also means that users can easily find help and resources to solve problems and optimize their platform use.

Integrating Eyer with Grafana: In addition to being easy to use, one of the things that makes Eyer stand out is its clear documentation. To learn how to integrate Eyer with Grafana, check out the Grafana section on the Eyer documentation.

Summing it up

The more complex modern infrastructure becomes, the more visibility you need to ensure that these infrastructures do not collapse underneath its own complexity. Observability tools like Eyer give you this visibility. With Eyer acting as a watchdog over your processes and Boomi integrations, you can sleep well at night knowing that if something is about to or does go wrong, you will be alerted immediately.

Eyer's strengths are elevated even further with the integration of visualization tools like Grafana, which translates these Eyer JSON into clear dashboards. These dashboards allow you to see, at a glance, the health of your Boomi integrations.

With Grafana visualizations, you can quickly identify trends, predict potential problems, and troubleshoot issues. In short, Eyer and Grafana working together provide you with the comprehensive visibility you need to ensure the smooth operation of your complex modern infrastructure, giving you peace of mind and allowing you to focus on more strategic initiatives.

To gain AI-powered insights and visualization for your Boomi integration, check the Eyer website and join the Discord community for more information and support.

Giving Back to the Boomi Community: How Your Contributions Make a Difference

Amarachi Iheanacho — Fri, 07 Jun 2024 20:11:02 +0000

Everybody wants to be part of a community—a group of people who validate a person’s thoughts and feelings and help them out during difficult situations. In the Boomi community by Eyer, this group of people are Boomi engineers.

The community provides a sense of belonging and support that transcends individual achievements, fosters a culture of collaboration, and forges friendships that can last a lifetime. However, as with any partnership or relationship of any substance, the community and its people thrive on the principle of give and take.

In this article, you will learn what giving back to the Boomi community can do for you and, more importantly, the best way to give back.

What do you gain from giving back to the Boomi community?

Helping your community is undeniably a good thing, but sometimes that “warm fuzzy feeling" isn't enough motivation for everyone. The good news is that giving back offers a ton of benefits beyond just feeling good. Here are some of the advantages you can gain by getting involved:

Skills development: Helping out community members is a guaranteed way to enhance your Boomi integration skills. It's a fantastic opportunity to learn new techniques, refine existing skills, and apply your experience to solve interesting problems in new ways.
Networking opportunities: Giving back and volunteering are amazing ways to meet like-minded people who share your passion for Boomi and mentorship. These activities guarantee valuable connections that can benefit you personally and professionally.
Build a professional brand: By consistently sharing your Boomi knowledge or volunteering to help others, you'll rapidly establish yourself as the go-to expert for all things Boomi. This includes opportunities, inquiries, and much more, and the best part is that your reputation extends beyond the Boomi ecosystem!
Leaving a positive impact: Contributing to the Boomi community is a powerful way to leave a positive mark. Sharing your knowledge and skills empowers others to grow and achieve their goals. Your insights can inspire and uplift fellow developers, creating a supportive and innovative space for everyone. This not only strengthens the entire community but also solidifies your reputation as a valuable and generous member.

What are the different ways to contribute to the Boomi community?

While how you contribute to a community can vary based on its needs (some might value open-source contributions or leadership roles), here are some universal ways to give back, especially within the Boomi community by Eyer:

Time commitment: Actively participate in discussions within the Boomi community by Eyer. Provide insightful solutions and clear explanations to Boomi developers.

Volunteer at Boomi-sponsored, Eyer-sponsored, and Boomi-related events in your local tech community. Lend a hand with logistics and setup or even lead breakout sessions on specific Boomi topics.

Skill-based contributions: One of the most efficient ways to become a thought leader in the Boomi community is to share your expertise. You can create blog posts, tutorials, or short guides addressing common integration challenges other Boomi developers face.

Guide and support fellow Boomi users, particularly those new to the platform. Offer advice, answer questions on the Boomi community forum or the Boomi community by Eyer discord, and help them navigate the integration world.

Acts of kindness (within the Boomiverse): Finally, if you are all out of time and you see yourself not being able to commit as much as you would like to, you can contribute to the community by recognizing valuable contributions made by other members by upvoting their responses in forum discussions. This helps elevate quality content and ensures others find the information they need.

Extend a warm welcome to new members in the forums or online events. Offer to answer basic questions and help them navigate the Boomiverse’s resources.

Additionally, when someone goes above and beyond to help you, acknowledge their effort with a positive comment or a "thank you."

Wrapping up

Hopefully, this article was enough to incentivize you to contribute more to your Boomi communities. Helping Boomi developers out, volunteering, and showing gratitude in the smallest ways are some of the best ways to provide value.

Sure, the idea is to give without expecting anything in return, but you can think of the rewards as a natural consequence of doing good. People will trust your Boomi expertise more because they've seen it in action and benefited from it. They can vouch for your character because you chose to volunteer when you didn't have to. This is an amazing position in a world run by referrals and recommendations.

So, take the first steps, join the Boomi community by Eyer, and start your journey today!

Enhancing Your Career Through Contribution and Recognition in the Boomi Community

Amarachi Iheanacho — Fri, 31 May 2024 11:41:18 +0000

There's a magic that happens when people come together, especially for a shared cause. History is filled with examples of this, and the Boomi community by Eyer is no exception. This community goes beyond just a platform to fix your Boomi problems—it can be the very tool that lands you your next big job.

Many people have participated in online communities over the years. They acknowledge that these communities have helped them out of a jam, but they don't quite understand the hype or why it's seen as the ultimate solution.

This article dives into how to maximize the benefits of being part of a community. It will explore the value of contributing your knowledge and building a strong reputation within the community, ultimately positioning yourself for that next big opportunity.

The value of contribution

The first step in building strong relationships within a community starts with contributing. This is especially important if you want to be seen as a trusted thought leader. Here are the different ways to get involved and contribute positively:

Showcase your skills and expertise

The most effective and popular way to contribute to the community is to share knowledge, and you can do that in many creative ways:

Actively participate in forums and discussions, answer questions thoughtfully, and provide clear explanations and solutions.
Share insights from your own Boomi experience, create blog posts, tutorials, and short guides that address common integration challenges faced by other developers.
Embrace challenges, participate in hackathons, and agree to volunteer for events. This not only demonstrates your problem-solving abilities but also highlights your passion for the Boomi platform.

Collaboration and growth

The biggest reason why communities like the Boomi community by Eyer work is collaboration. Working alongside other developers with varying degrees of experience exposes you to different approaches and techniques, enriching your own understanding of Boomi and its functionalities.

Additionally, combining diverse skill sets from the community allows you to brainstorm and develop more creative and effective solutions to integration challenges. This integration of varied skills also strengthens your problem-solving skills and prepares you for real-world scenarios.

Gaining recognition in the Boomi community

You've started sharing your expertise, but how do you stay visible and become the go-to person for Boomi questions and opportunities within the Eyer community and beyond? This section will focus on building your personal brand and solidifying your reputation as a valuable resource.

Recognition systems in the Boomi community by Eyer

Active participation in the Boomi community unlocks a range of recognition systems that validate your skills and contributions.

You can earn swag, gift cards, and even become a community leader. This recognition highlights your expertise and commitment in addition to opening up opportunities for professional growth and networking within the Boomi ecosystem.

Additionally, the Boomi community by Eyer recognizes exceptional contributions by featuring your profile or content on the blog. This demonstrates your expertise to a wider audience and positions you as a thought leader within the Boomi community.

Building a personal brand

Aside from continuous community contributions, maintaining an online presence is crucial for your personal brand as a Boomi developer. Use platforms like LinkedIn and Twitter to showcase your work and contributions.

Regularly update your profiles with your latest projects, certifications, and endorsements. Share articles and excerpts from those articles that you find helpful. Do not be afraid to demonstrate your expertise and commitment to continuous learning regularly—you never know who might be watching.

Networking

Engage with peers and industry leaders by participating in networking events, both online and offline. Attend meetups, webinars, and conferences to connect with other Boomi professionals. Building relationships within the community can lead to mentorship opportunities, collaborative projects, and access to job openings. Networking is a powerful tool for career growth, providing you with insights, support, and opportunities that can significantly enhance your professional journey. Check out this article to learn more about building professional connections as a Boomi developer.

Leveraging your reputation for your career advancements

You have built your reputation and expanded your network through community contributions; what’s next?

The next step is strategically leveraging your contributions to make you a more attractive candidate for any Boomi-related opportunity.

Highlight the community contributions and skills gained on your resume, LinkedIn profile, and during job interviews. Employers value candidates who demonstrate leadership and a commitment to making a positive impact.
Apply to speak at Boomi-related conferences and webinars. Sharing your knowledge at these events further establishes you as a thought leader and allows you to connect with a wider audience of potential employers and collaborators. By actively promoting your expertise and the value you bring, you'll be well on your way to propelling your career forward.
Request recommendations and endorsements from individuals in your network who can vouch for your skills, character, and contributions. Strong recommendations can be powerful in job applications and career advancements.
Leverage your expanded network to find mentors who can provide career advice, industry insights, and feedback on your professional development. A mentor can help you navigate career decisions and introduce you to further opportunities.

In summary

Communities can be your ticket to an amazing, long-lasting Boomi career when used correctly. They uplift you and provide the resources and the experts to help you troubleshoot and answer your problems. While these are fun to take advantage of, communities can become even more interesting when you contribute to this knowledge base and help people with their queries. By sharing your expertise, you solidify your understanding and establish yourself as a valuable resource within the community, opening you up to career advancement opportunities. This two-way street of learning and giving back fosters a sense of connection and accomplishment, making your Boomi journey all the more rewarding.

Take the first step and join the Boomi community by Eyer today.

Building Professional Connections as a Boomi Developer

Amarachi Iheanacho — Fri, 24 May 2024 13:42:18 +0000

With tech being advertised as the next best thing for humanity since the advent of penicillin, it's no surprise that more people are picking up some tech or tech-adjacent skill. While this trend benefits both individuals and the industry, it’s clear that success in this field requires more than individual effort. This is particularly true for the Boomi landscape, highlighting the importance of the Boomi community by Eyer.

By being a part of this community, you:

Learn from others' experiences and contribute your own insights.
Get help with problems you encounter and help others overcome theirs.
Keep up-to-date on the latest platform advancements.

But the benefits go beyond the Boomi platform. Expanding your network exposes you to broader industry trends and potential collaborations. This can open doors to career advancement, new opportunities, and a more fulfilling professional journey.

This article discusses shared spaces for Boomi developers of all levels, strategies for maximizing these shared spaces, and building meaningful connections.

Shared Boomi developer spaces

Recognizing the invaluable support Boomi developers receive from the community and networking opportunities, let's explore online and offline spaces to connect and build your network with other Boomi developers.

Online networking

The internet serves you the biggest online Boomi communities on a silver platter. You can join communities and keep in touch with members all at the tip of your fingers. Here are some ways to leverage these online communities to your advantage:

Boomi online communities: In the past few years, several Boomi communities have emerged, including the official Boomi community and the Boomi community by Eyer, among others. These communities host forums, discussions, and Q&A sessions where you can participate by asking questions, sharing your expertise, and engaging in conversations. Contributing to these communities demonstrates your knowledge, builds your reputation, and allows you to connect with developers facing similar challenges.
Social media platforms: Utilizing platforms like LinkedIn and Twitter can help you connect quickly with Boomi professionals, industry leaders, and potential employers. Following relevant accounts such as the official Boomi Twitter and LinkedIn and The Eyer Twitter and LinkedIn, among others, is also beneficial.

In addition, creating posts that showcase your skills, certifications, and projects helps maintain meaningful connections in your niche.

You could share industry news, participate in discussions using relevant hashtags (e.g., #Boomi, #Integration), and connect with individuals whose work interests you."
Engaging with Others: By actively participating in the community, you can gain insights, answer questions, and provide constructive feedback. Respond to comments on your posts, join relevant discussions, and connect with those who share your interests. By taking the initiative and being helpful, you'll build positive relationships and establish yourself as a valuable resource within the Boomi community.
Online events and webinars: Attending online Boomi events and webinars offers numerous advantages. You'll gain valuable knowledge about the latest Boomi features, best practices, and industry trends directly from Boomi experts and practitioners. Webinars often feature live Q&A sessions, allowing you to network and connect with presenters and attendees directly.

These online events provide a platform for learning and expanding your network simultaneously. You'll connect with fellow developers facing similar challenges and exchange ideas.

Offline networking

While online connections are essential, face-to-face interactions can significantly enhance your network. Let's explore some shared offline networking spaces:

Local Boomi meetups and user groups: Some organizations like Easy Data Integration host local Boomi meetups and user groups. These gatherings offer a fantastic opportunity to connect with Boomi developers in your area. You'll build relationships, share knowledge, and learn from each other's experiences.
Industry conferences and events: Attending industry conferences and events focused on integration or Business Process Management (BPM) can be incredibly valuable. These events attract a wide range of professionals, including Boomi developers, solution providers, and industry leaders. Participating in these events allows you to:
- Learn about new technologies and upcoming trends in the integration and BPM space.
- Network with a broader audience, potentially leading to new collaborations or job opportunities.
- Attend workshops and sessions focused on Boomi development, further enhancing your skillset.

Understanding the various shared spaces where Boomi developers connect, let's explore how to make the most of these interactions.

Tips for networking at events

Conferences offer fantastic opportunities to build meaningful connections within your field. Here are some tips to maximize your networking potential:

Make a plan: Conferences are fast-paced. Know what sessions you want to attend, who you want to meet, and what you hope to achieve. Prioritize your time!
Do your research: Before the event, identify key people going to these conferences and research their work. This shows initiative and helps prepare conversation starters.
Connect beforehand: Use social media or email to connect with people you want to meet before the conference. Establish a rapport and build some pre-event connections.
Schedule your time: Create a conference schedule that includes workshops, presentations, meetings, and discussions. This ensures you don't miss valuable networking opportunities.

Building meaningful connections

Swapping business cards is a good start, but true professional connections are built on more than just exchanging contact information. Here are some tips to cultivate deeper relationships with fellow Boomi developers:

Actively listen: When you meet someone new, put away your phone and truly focus on what they're saying. Ask thoughtful questions that show genuine interest in their work and experiences. This not only makes the other person feel valued but also helps you discover potential areas for collaboration.
Provide value to others: Networking isn't a one-way street. Look for ways to be a resource for others. Share your Boomi expertise by offering tips, recommending relevant resources, or even connecting them with someone in your network who might be able to help with their specific challenge.
Follow up after meeting someone: Don't let your new connection disappear. Send a personalized email or LinkedIn message within a day or two of meeting someone. Briefly mention something you enjoyed from your conversation and offer additional help if relevant. You can also suggest connecting on social media platforms focused on Boomi development to stay in touch and share industry updates.

By actively listening, providing value, and following up, you'll transform fleeting interactions into meaningful connections that can benefit your professional development and open doors to new opportunities within the Boomi developer community.

Go for it!

This article explored the power of community in building a thriving professional network within the Boomi development landscape and, by extension, many software development fields. It discussed the wealth of online and offline spaces available to connect with fellow Boomi developers.

From online communities and social media platforms to local meetups and industry conferences, these shared spaces offer invaluable opportunities to learn, share knowledge, and forge meaningful connections. By actively engaging in these communities, you'll gain access to a supportive network that can propel your career forward.

So, take the first step and join the Boomi community by Eyer today.

Top Community for Boomi Developers

Amarachi Iheanacho — Tue, 14 May 2024 15:15:26 +0000

Alone, we can do so little; together, we can do so much – Helen Keller.

Boomi has cemented itself as a leading integration platform (iPaaS) service for businesses worldwide. Embraced by thousands of developers, Boomi empowers them to streamline data flows and connect applications seamlessly. However, the potential for growth doesn't stop with individual expertise; it also requires a supportive and knowledgeable community. This is why Eyer, the AI-powered observability tool, has introduced its resourceful and incentive-driven Boomi community.

This article explores how being a part of this community on the rise can significantly enhance your experience as a Boomi developer.

What are the benefits of joining the Boomi community by Eyer?

It's no coincidence that joining and actively participating in communities have skyrocketed in popularity in recent decades.

A community brought together by shared interests in the tools developers use in their day-to-day lives is a space driven by collaboration. This space fosters brilliance, enabling you to push the boundaries of what’s possible.

Naturally, this alliance is what people expect from a top Boomi community, and the Boomi community by Eyer is no exception. In this community, we encourage your growth through the following channels:

Resources and development

The Boomi community by Eyer is packed with resources and opportunities to boost your integration skills. Here's what you can expect:

Extensive knowledge base: Access a comprehensive library of articles, tutorials, and FAQs to deepen your understanding of Boomi, Eyer, and AIOps in general
Active forums: Engage with fellow Boomi developers and experts. Ask questions, share knowledge, and get valuable insights from the community.
Webinars and events: Attend informative webinars and online events hosted by experts in the integration space to learn about new features, solutions, and industry trends.

Whether you are a seasoned professional or just starting your journey with Boomi, being a part of the active Boomi community ensures continuous learning and professional advancement within a supportive and collaborative environment.

Access to network and collaboration

Now, acquiring knowledge is great, but if you have stayed long in this industry, you’d recognize that knowledge is not the only guarantee for growth; your network also plays a big part in your success. Joining a community with people who have walked the path you are currently on or about to walk the same path allows you to:

Leverage their expertise, get real-world advice, and find solutions to your technical or non-technical challenges.
Contribute your knowledge and become a trusted resource for the community, ultimately positioning yourself as a thought leader.
Expand your professional network and connect with potential employers and collaborators.

Incentive programs

The Boomi community by Eyer regularly rewards its members; not only is this community a fertile ground for you to gain the network and resources you need to take your career to the next level, but you get actively rewarded every step of the journey.

These rewards include gift cards, exclusive swags, career opportunity referrals, and even license discounts.

A Community on the rise

The Boomi community by Eyer might not be a household name just yet, but that's precisely what makes it so exciting. Backed by the innovative Norwegian company Eyer, this community is taking a knowledge-based development path to success. This means they are focused on creating helpful content, hosting events, and so much more, basically listening and catering to the needs of their community members through content.

There are many incentives to joining a relatively smaller community:

Shape the direction: With a smaller Boomi community, you have a greater chance of influencing its direction and culture. With the exciting plans for the future of Eyer, being a member allows you to witness this growth, contribute your ideas, and help define the community's focus and how it operates.
Get your issues heard: One of the problems with a larger community is that some of your questions might get lost in the shuffle. In this Boomi community, your concerns will be heard and addressed until they are resolved. Being in a relatively small community also offers you the opportunity to form close-knit connections with members of the community and the Eyer team.
Collaborative leadership: In the Boomi community by Eyer, we believe in shared ownership. Our leaders guide discussions and encourage member participation, fostering a safe space for learning and respectful dialogue. We value diverse perspectives and prioritize opportunities for all members to learn, network, and build meaningful connections.

Conclusion

The Boomi community by Eyer offers unparalleled opportunities for potential growth in your career as a Boomi developer. Providing you with the intersection between Boomi development, artificial intelligence, and observability.

In summary, this community is designed, created, and maintained for and with the Boomi developers and observability enthusiasts. This community welcomes everybody, armoring them with the resources and network required to achieve their integration and observability goals.

Join the Eyer-for-Boomi discord channel to be a part of the community today!

The Boomi community you didn't know you needed

Amarachi Iheanacho — Thu, 02 May 2024 17:50:55 +0000

Throughout history, communities have played core roles in human progress. From large entities like nations to smaller units like the nuclear family, they provide a sense of belonging, support, and encouragement. They also drive positive change on a global scale. The Eyer for Boomi community is no exception.

This community goes beyond just fostering camaraderie. It empowers Boomi developers by enabling them to collaborate, share knowledge, and overcome challenges together. Ultimately, this collaborative environment pushes the boundaries of what's possible with Boomi integrations.

This article explores what Eyer for Boomi is and how being a part of its community can significantly enhance your experience as a Boomi developer.

What is Eyer for Boomi?

Eyer for Boomi is an AI-powered observability platform that delivers actionable insights for your Boomi integrations.

With the Eyer connector, you can seamlessly introduce AI-powered visibility into your Boomi processes and receive early notifications on crucial performance metrics like throughput, latency, and other runtime processing indicators.

Moreover, you can take proactive measures—either manual or automated—based on the status of your metrics to prevent performance degradation or, worse, total system shutdown.

Now, acknowledging the value that the Eyer connector already provides to the Boomi ecosystem, let’s understand how the Eyer for Boomi community can elevate that value and experience even further.

Why do you need the Eyer for Boomi community?

Driven by the need to serve the Boomi ecosystem, the good people at Eyer have decided to launch the Eyer for Boomi community, and this community is important for many reasons, some of which are:

Learning and Development

As a member of the Eyer for Boomi community, you gain access to a network of Boomi developer experts and peers, creating a space for knowledge sharing and collaboration.

Share best practices, discover efficient ways to work within the Boomi ecosystem, and stay up-to-date with the latest Boomi features and functionalities.

You can access educational resources, tutorials, and live events hosted by the community.

Subsequently, you can use the community's support to gather attention for informative Boomi content or events you decide to put out.

Problem-solving and Support

A by-product of being in the network of experienced Boomi developers is the ability to get help when you get stuck on your integration problems.

With a platform like Discord, you can hold these communities close to your heart. You can also get quicker answers to your questions and solutions to common integration challenges.

Additionally, you can collaborate with community members and troubleshoot less common issues within the Eyer for Boomi community.

Innovation and Inspiration

Another advantage of being a part of communities is learning from other people’s mistakes and success stories.

The Eyer for Boomi community is no different. Leverage the experiences of Boomi developer experts to discover how to tackle complex integrations and get inspired to try new approaches in your own projects.

This article has established the advantages of joining a Boomi developer community. Now, let's explore why the Eyer for Boomi community would be a particularly good fit for you.

Why the Eyer for Boomi community?

In addition to providing you with a network of Boomi developer experts and peers, the Eyer for Boomi developer community offers much more. Some of these benefits include:

Recognition and Rewards

Are you tired of offering value in private? In the Eyer for Boomi developer community, you get the recognition you deserve.

You get spotlighted and recognized for your contributions to the community. You can earn points and become a community influencer and leader.

This recognition comes through public acknowledgment on social media pages, community blogs, community events, and sometimes incentives like swags and license discounts.

Early access to new features from the Eyer

The Eyer for Boomi developer community members receive exclusive access to updates on new features, enhancements, and releases from both Eyer and Boomi. You will be among the first to know whether it's a new connector or a platform improvement.

Additionally, community members often get the opportunity to participate in beta programs for upcoming features for Eyer's new releases. This hands-on experience allows you to explore new capabilities before they are widely available, providing valuable insights and shaping the future direction of the Eyer connector.

Additionally, the Eyer for Boomi developer community provides direct communication channels with product experts and developers from both Eyer and Boomi; you'll have the opportunity to ask questions, provide feedback, and engage in meaningful discussions about new features and updates.

Summing up

The Eyer for Boomi developer community proves that magic can happen when a group of people gather for the same cause with the same passion.

It is an ecosystem designed solely to empower you as a Boomi user. Whether you are a seasoned Boomi developer or just starting out your journey, this Eyer for Boomi community offers invaluable resources and support to help you effectively achieve your Boomi integration goals.

Join the Eyer for Boomi Discord channel to be a part of this community today!

How to automate email sending from MySQL databases using Boomi

Amarachi Iheanacho — Fri, 19 Apr 2024 14:49:39 +0000

An Integration platform as a service (iPaaS), like Boomi, is known for integrating platforms and data from different services. However, another interesting feature of the iPaaS solution is its ability to ensure efficient communication with your business clients.

Automating email sending directly from your MySQL databases using Boomi allows you to establish personalized and timely communication without manual intervention. This eliminates the time-consuming task of manually sending emails and ensures that your customers receive relevant information promptly.

This guide discusses automating sending out emails, empowering you to leverage your MySQL data to trigger custom email notifications.

TL;DR

To automate email sending from your MySQL database with Boomi, follow these steps:

Create a new Boomi integration process
Collect the data from the database
Transform the data type from database to CSV
Split the CSV files by line
Set Properties
Create the Message
Send the email

Prerequisites

To get the most out of this article, you must have the following:

A Boomi platform account: if you do not have one, create a 30-day free trial account
A Boomi Atom is installed locally and connected to a MySQL account. To learn how to do this, check out How to connect your Boomi Atom runtime and MySQL on Docker.
A Boomi Mail connection; to learn how to create one, check out this Creating the Mail connection and Operation Boomi documentation

Inserting data in your MySQL database

MySQL is an open-source relational database management system (RDBMS) that stores, organizes, and accesses data in a structured format. The prerequisites section discussed Connecting your Boomi Atom runtime and MySQL on Docker, and this section will build on that knowledge.

The MySQL database will hold the user information, including their names and the email addresses to which you will send your desired message. To do this, you must change your folder structure and the docker-compose.yml file you created when you connected your Boomi Atom runtime to your MySQL database.

In your project's root directory, create a folder named ‘mysql-data’. This folder will store your MySQL data and ensure it persists even when the Docker container stops running.

Next, in your docker-compose.yml file, add a volume mapping for the MySQL service. This line will link your host machine's ‘mysql-data’ folder to the container's internal data directory.

 mysql:
    image: mysql:latest
    container_name: mysql
# add this line
    volumes:
      - mysql-data:/var/lib/mysql:Z
    environment: 
      - MYSQL_ROOT_PASSWORD=<root user password>
      - MYSQL_USER=<username>

      - MYSQL_PASSWORD=<password>

      - MYSQL_DATABASE=<name of your MySQL database>
    ports: 
      - "3306:3306"

After updating your docker-compose.yml file, run this command to build and run your images.

docker compose up -d

Creating a table in your MySQL Database

Run this command in your terminal to start an interactive shell in your running MySQL Docker container.


docker exec -it mysql bash

Next, log into this shell using this command.


mysql -u <mysql_user> -p<mysql_password> <mysql_database>

The next step is to create a table named 'customers' to store your customer data. This table will include ‘id’, ‘first_name’, ‘last_name’, and ‘email’ fields.


CREATE TABLE users (
 id integer,
 first_name varchar(255), 
 last_name varchar(255),
 email varchar(255)

);

To check if your tables have been successfully created, run this command.


SHOW TABLES;

Inserting data in your MySQL database

After you create your tables, use the following command to insert data into this table, replacing the placeholders with their actual values.

INSERT INTO users  (id, first_name, last_name, email)
VALUES (1, 'user1 first name', 'user1 last name’, ‘user1 email'),
(2, 'user2 first name', 'user2 last name’, ‘user2 email');

Run this command to see your data.

SELECT * FROM users

Automate email sending with Boomi

This section explains how to set up a Boomi process to automatically collect data from your MySQL database and send a customized email.

To create this process, follow these steps:

1.Create a new Boomi integration process

Go to the Integration page, click the Create New button, and choose Process from the dropdown menu.

This action will open the Start Shape sidebar. Choose the No Data radio button, indicating that the process will not receive external data, and then confirm by clicking the OK button.

2.Collect the data from the database

After initiating your new Boomi process, add your MySQL database connector, which holds the user information.

To add a shape to a Boomi process, click on the small box at the end of the dotted lines, leaving the previous shape (in this case, the Start Shape). Then, search for the shape you want to add to the process (the Database shape).

Next, click on the Database connector to configure the shape. This action opens up the Connector Shape sidebar.

In this sidebar, click the Connection input field and select the MySQL connection you created in the prerequisites section.

If you have not created the MySQL database connection, check out How to connect your Boomi Atom runtime and MySQL on Docker to do so.

Next, click on + sign in the Connector Operation input field to create a new operation. In Boomi, a database operation defines how you interact with the database specified in the connection.

This action opens up the Database Options page.

In the Database Options page, click on the + symbol to create a new Database profile for the operation.

On the Database Profile page, click the Statement menu, followed by the Import button on the top right-hand corner to import your database connection.

The Import button opens up a Database Import Wizard menu. Here, fill out the following fields with your credentials:

The Boomi Atom you want to run this Boomi process on
The Database connection you created
A schema filter of users

Once you are done, your Database Import Wizard modal should look like the image below:

After clicking Next, you'll be prompted to choose a table for synchronization. Select the desired table and click Next again.

Next, select the columns you want to move to collect your data. You can choose all columns or exclude any as needed. Once you've selected, click Next, followed by the Finish button.

Finally, expand the Fields menu to verify that your Database profile has been imported correctly into the Connector.

After saving and closing the Profile page, you'll be returned to the Database Options page. Here, set up the link between your data points:

Click on the Link Element field.
From the Fields menu, choose id. This tells Boomi to group records that share the same id into a single document.

Set the Batch Count to 1 for data retrieval to process data one record at a time. Leave Max Rows at 0 to retrieve all rows from the database.

Finally, click the Save and Close button to return to the connector sidebar.

In the sidebar, click the OK button to return to the canvas.

3.Transform the data type from database to CSV

Next, transform the files from the database to a CSV format, which will make it easier for you to parse and retrieve information from the database.

Add a Map Shape to the Boomi process to change the data format. Click on the Shape to configure it, which opens up a Map sidebar.

Click the + sign in the Map input field in this sidebar to open the dedicated Map page.

On the Map page, go to the Map Source section and select the Choose link.

This action opens the Choose a Source Profile. In this modal, select the Profile Type of Database, and in the Profile field, select the database profile you created for your Database Options page.

Click the OK button to see your data fields populated in the Elements menu.

On the right side of the page, the Map Destination, find the Choose button and click on it.

This action will open the Choose a Destination Profile modal.

Select Flat file from the modal's Profile Type dropdown menu. Then, click the + button in the Profile input field to create a new profile for the CSV file.

This action opens up a New Flat File Profile page. Right-click the downwards-facing arrow next to Elements, and select Add Multiple Elements from the menu.

This opens up a modal that asks you to choose how many elements you'd like to add. This tutorial decides to create 4 new elements. Click on the OK button to close the modal.

Next, click on each of the newly created elements and change them to correspond with the fields in your database. This tutorial creates fields for id, first_name, last_name, and email.

Next, navigate to the Options tab, check the Use Column Headers checkbox, and select the Comma Delimited option from the File Delimiter dropdown. This indicates that you want to use the column headers present in the data and separate the data fields by a comma.

Click on Save and Close and the OK buttons to save this profile.

To see all your newly created fields, click the downward-facing arrow next to the Elements menu in the Map Destination section and select Expand.

Next, map the corresponding fields between your database and CSV profiles.

Simply drag your cursor from a field on the left side and connect it to its matching field on the right. Once you've mapped all the fields, your page should resemble the example provided.

Click the Save and Close button to save this Map and return to the Map sidebar.

Click the OK to save the Map configuration and return to the Boomi canvas.

4.Split the CSV files by line

Next, you want to split each piece of information in the file by users. To do this, you need to add the Data Process Shape to the Boomi process.

This Data Process shape allows you to split your document line by line, which in your case means user by user.

Next, click on the Data Process shape to open up its dedicated sidebar. In this dedicated sidebar, click on the + sign to specify a processing step for your data.

This action opens up a form; in this form, select:

Split Documents from the Processing Step dropdown
A Profile Type of Flat File to match the CSV file format
Split Options of Split By Line
A Headers Option of Retain first line as Column Headers

Click the OK button to save this Data Process Shape configuration.

5.Set Properties

Next, you will use the Set Properties shape to create and set two properties dynamically:

The user's first name property is set as the first_name in the data
The receiver's email address property is set as the email address stated in the data file.

You will need these dynamic properties to create and send out these emails. To create the first name property, click the + button in the Properties to Set section.

This action opens up a Choose Property modal. In this modal, select the Property Type of Dynamic Document Property and select the Property Name as first_name.

Click on the OK button to save this property.

You should see the Dynamic Document Property parameter. Select this parameter, and click the + sign in the Property Value section to create the parameter’s value.

This action opens up the Parameter Value modal. Fill out the Parameter Value modal with the following information:

Type -> Profile Element
Profile Type -> Flat File
Profile -> The CSV profile you created during the Map process.
Element -> Expand the menu and select the first_name field

Your Parameter Value modal should look like this when you are done. Click the OK button to save the parameter value information.

Next, create another parameter for the receiver’s email address. To create this parameter, click the + sign in the Properties to Set section. This action opens up the Choose Property modal. Fill out this modal with the following information:

Property Type -> Document Property
Source Type -> Connectors
Connector -> Mail
Property -> To Address

This creates a dynamic property that holds the receiver's email address.

Click the OK button to save this property and return to the sidebar. In this sidebar, you should see a Mail - To Address property.

Select this parameter, and click the + sign in the Property Value section. This action opens up the Parameter Value modal. Fill out the Parameter Value modal with the following information:

Type -> Profile Element
Profile Type -> Flat File
Profile -> Mail Flat File Profile
Element -> Expand the Elements menu and select the email field

When you are done, your Parameter Value modal should look like this. Click the OK button to save this parameter value and return to the sidebar.

Click the OK button in this sidebar to save your Set Properties configuration shape and return to the Boomi canvas.

6.Create the Message

Next, you need to create the message in the email you want to send out. Add the Message shape to your Boomi process to create this custom message.

Click this process shape to open its dedicated sidebar.

In this sidebar, write the message you want to send out in the Message input field. This tutorial writes out this message.

    Hey there {1}!


    I hope everything is going great for you.

The {1} is a variable you will set to the first_name property you created in the Set Properties Shape.

Click the + button to set this variable value in the Variables section. This action opens a Parameter Value modal.

In this modal, fill in a Type of Document Property. Next, click the search icon next to the Document Property input field to open up the Choose Property modal.

In this modal, fill out the following credentials:

Property Type -> Dynamic Document Property
Property Name -> first_name

Click the OK button to save and close this modal.

You should see your Document Property loaded in the Parameter Value modal.

Click the OK button to save your parameter value and return to the sidebar. Your sidebar should look like this.

After creating your message and setting your dynamic variable, you instruct Boomi to replace the{1} with the user’s first_name from the file when it sends the email. Finally, click the OK button to save your message configuration and return to the Boomi canvas.

7.Send the email

Now, the best part is actually sending the email. The Mail Connector Shape allows you to add this shape to your Boomi process.

Click this Mail connector to configure this shape. This opens up the Mail connector sidebar.

In this sidebar, go to the Connection input field and select the Mail Connection you created in the prerequisites section. If you have not created the Mail Connection yet, check out Creating the Mail connection and Operation.

Next, select an Action of Send, as you will be sending emails.

In the Operation input field, you can either select a previously created Mail operation or create a new one. This tutorial shows you how to create a new operation.

Click the + sign in the Operation input field to create a new operation. This action opens up a new Mail operation page.

Fill out the input fields on this page with the following information:

From: This is the email you used when creating your connection. It is the email address that shows up as the sender in your Mail app
To: Leave To as blank as you want to dynamically set that property depending on the email address in the data file from your database
Disposition: Select Inline because the document data is sent in the body of the email message instead of as an attachment.
Content-Type: text/plain, as you will be sending texts

When you are done, your sidebar should look like this: click the OK button to save this sidebar and return to the Boomi canvas.

8.Test the process

To verify if the process is working, click the Arrange button, then click the Test button to test the process.

This action asks you to select the Atom you want to run this Boomi process on; select the Atom you created in the prerequisites section. After running the process, you should see the custom email in the provided email address.

That’s a Wrap!

This guide has shown you how to dynamically send custom emails to different email addresses in your MySQL database using Boomi. You can also build on this tutorial to create personalized messages based on user information and trigger automated emails based on specific actions.

However, as interesting as this exercise is, this is only just a starting point for exploring the full capabilities of Boomi. Refer to the official documentation for more information on what Boomi can do.

How to create an ETL process with Boomi, using AWS S3 and Salesforce

Amarachi Iheanacho — Wed, 10 Apr 2024 10:47:24 +0000

Extract, Transform, and Load (ETL) is a fundamental data processing technique to prepare data for analysis. It involves gathering data from various sources (extract), cleaning and converting it into a consistent format (transform), and then loading it into a central repository, where it's ready for tasks like business intelligence and machine learning.

These processes are the backbone of data integration, facilitating the movement of information between systems with efficiency and accuracy.

This guide simplifies data integration between AWS S3 and Salesforce using Boomi. Learn to extract, transform, and load data, gaining valuable insights for decision-making. No matter your Boomi experience, this tutorial equips you to build a seamless data pipeline.

TL;DR

To build a simple ETL process that collects files from your AWS S3 bucket, checks if the files were last modified today, and stores them in Salesforce with Boomi, follow these steps:

Create a new Boomi integration process
Collecting the data from the AWS S3 bucket
Use the Business rule to figure out files for the last modified date of today
Collect the files that were modified today from the S3 bucket
Change file type from CSV to XML
Adding a Salesforce connector to store the data
Mapping the CSV values to the Salesforce fields
Testing the Boomi process

Prerequisites

To get the most out of this article, you must have the following:

A Boomi platform account: if you do not have one, create a 30-day free trial account
A Salesforce account: if you do not have one, create a free account
A Boomi Atom is installed locally and connected to a MySQL account. To learn how to do this, check out how to connect your Boomi Atom runtime and MySQL on Docker.
An AWS account; if you do not have one, create one for free.

Creating files in your AWS S3 bucket

Before uploading files to your S3 bucket, you must create the CSV files you want to store. To accomplish this, create a new directory on your laptop using the following command:


mkdir <name of your directory>

Next, open this directory in your code editor and create as many CSV files as possible. This tutorial creates three CSV files: booking-1.csv, booking-2.csv, and booking-3.csv.

Populate these files with the CSV data you want to store. For instance, booking-1.csv can have this data, with each field separated by a comma:


booking_id,customer_name,customer_email,customer_company
123,Alice Johnson,alice.johnson@hospitality.com,Grand View Hotel
456,Brian Jones,brian.jones@travelagency.net,Adventure Tours

Populate the rest of the booking files with the data you want to store.

Uploading these files to your AWS S3 bucket

To upload your CSV files to an S3 bucket, search for S3 in the search bar and then click on the S3 service in the modal.

Clicking on the S3 service redirects you to the Amazon S3 page. To create a new S3 bucket, click this page's Create bucket button.

The Create bucket button leads you to the General configuration page of the S3 bucket you're about to create. Enter your desired Bucket Name, then click the Create bucket button to complete the creation process.

When you are done, you should see your newly created bucket on your Amazon S3 page.

Copy your AWS access and secret key

The AWS access and secret key allow applications or users to access the data in your S3 bucket.

To retrieve these keys, you'll need to create a user and specify which artifacts the user can access.

To create a user, search for and click on the IAM service. Then, open the Users page by clicking the Users tab in the sidebar.

On this page, click the Create User button. This action will open the Specify user details page. Input the user name you want to create, then click Next to proceed to the Set Permissions page.

Choose the Attach Policies Direct option on this page, then search for the AmazonS3FullAccess permission policy and click on it.

Click the Next and Create User buttons to create your new user. You will then see your newly created user on your User page.

To find your AWS access and secret access key, open your newly created user page by selecting the user from the sidebar, then click the Security Credentials tab.

Scroll down to the Access keys section, and click the Create access key button.

This action leads you to the Access key best practices and alternatives page. Choose the Third-party Service option on this page since you'll manage the data in your S3 bucket with Boomi.

After that, check the box, "I understand the above recommendation and want to proceed to create an access key," and click the Next button.

You can input a description tag for the access key if you'd like, but it's not required. Then, click the Create access key button.

After this step, you will be directed to the Retrieve access keys page. Make sure to copy both the Access and Secret access keys. These keys are necessary for connecting your Boomi process to the AWS S3 bucket.

Click on the Done button to return to your user page.

How to create the ETL process

This tutorial creates an ETL process that enables you to collect data stored on the AWS S3 bucket. Next, you'll filter this data to return only the records modified today, transform them into XML format, and then store them in Salesforce. This process empowers the sales team and improves management's visibility and decision-making.

To create this ETL process in Boomi, follow these steps:

1.Create a new Boomi integration process

To create this process, go to the Integration page, and click the Create New button, and choose Process from the dropdown menu.

This action will open the Start Shape sidebar. Choose the No Data radio button, indicating that the process will not receive external data, and then confirm by clicking the OK button.

2.Collecting the data from the AWS S3 bucket

After initiating your new Boomi process, add your AWS S3 bucket, which contains the files you want to extract, transform, and load into the Salesforce application. However, it is important to incorporate error handling to catch any errors during connection.

Add the Try/Catch shape to your Boomi process to handle errors.

To add a shape to a Boomi process, click on the small box at the end of the dotted lines, leaving the previous shape (in this case, the Start shape). Then, search for the shape you want to add to the process (the Try/Catch shape).

Add the Amazon S3 REST connector to the Try path. This connector links your S3 bucket to the Boomi shape. Next, click on the Amazon S3 REST connector to configure the shape. This action opens up the Connector Shape sidebar.

In this sidebar, select QUERY as the Action from the dropdown. Then, click the + in the Connection input field to create a new AWS S3 connection.

This action opens a connection page that requires you to input credentials for your S3 bucket. These credentials include the Amazon AWS Access Key and your Amazon AWS Secret Key. They grant your Boomi process the necessary permissions to access the data in your buckets.

Next, click the Save and Close button to return to your Connector Shape sidebar.

After creating the connection, click the + button in the Operation input field to define how to interact with the S3 bucket.

This action opens up the Operation page. On this page, click the Import Operation button.

This action opens up an Amazon S3 REST Operation Import modal. In this modal, fill in the required information:

Atom: The atom you are running your Boomi process in
Connection: The Amazon S3 REST connection you just created

Click the Next button to open the Amazon S3 REST Operation Import modal. In this modal, choose the AWS S3 bucket from which you want to retrieve the files, then click Next.

Next, click the Finish button to save your newly loaded operation.

You should see your S3 bucket in the operation page's Objects section. Click on this bucket.

Next, you'll define a filter criteria for the Prefix field corresponding to the S3 bucket file path. To do this, click the downward arrow next to Filter. This will open up an Add Logical Sub-Group menu; select this option.

Next, click the downward-facing arrow and then on the Add Expression menu. This action opens up a form. In this form, input a filter name of Prefix, click the field search bar, and select the prefix menu.

Finally, click the Save and Close button to return to the Amazon S3 sidebar.

Next, go to the Parameters tab to specify that you want to query all the files in the AWS S3 bucket with the static value 'bookings'. To do that, click the + sign in this tab to open the Parameter value modal. Fill out this modal with the following information:

Input - Prefix
Type -Static
Static Value - bookings

Click the OK button to view your newly created parameter in your AWS S3 connector sidebar.

Click the OK button in the Connector sidebar to return to the canvas.

After connecting your Boomi process to your S3 bucket, add a Stop shape at the end of the catch branch to stop the process if any errors are encountered.

3.Use the Business rule to figure out files for the last modified date of today

After connecting your process to your S3 bucket, add a Business Rules shape to filter the files in your bucket based on whether they were modified today.

Click on the Business Rules shape to open the Business Rules Shape Options modal. In this modal, select a Profile Type of XML. In the Profile input field, you will see the profile you created for the Amazon connector. Simply select that profile.

Next, click the OK button to open the Business Rules Shape modal. In this modal, click the Create a New Rule button to create a new business rule.

On this New Business Rule page, click the Add dropdown and select the Function tab. Selecting the Function tab opens up an Add a Function modal.

In this modal, select a Category of Date, select the Get Current Date function to get the current date, and then click the OK -> OK buttons to save this selection.

Next, add the second input for the comparison; this input will be the file's last modified date.

To add this second input, click the Add dropdown, then select Field from the menu. This action will open an Add a Field modal. Click on the Field input area to open a New Input modal within this modal. Choose the LastModified field and click the OK -> OK button to save your selection.

Next, add a condition to compare these values and check if they are the same. This means checking if the last modified date equals the present date.

To create this condition, click the Add a Condition link.

In the dropdowns, select the following fields.

In the first dropdown, select the LastModified(Contents/LastModified) field
In the second dropdown, select the = field
In the third dropdown, select the Get Current Date field

Next, select the Save button and click OK to save the business rules.

When checking for the condition in the Business Rules shape, it can take one of two paths, Accepted or Rejected. If the files are rejected, you'll want to end their journey. To do that, add the End and Continue shape to the Rejected path from the Business Rules shape.

You'll want to retrieve and store the remaining files that pass this condition in your Salesforce application.

4.Collect the files that were modified today from the S3 bucket

To retrieve the files that meet these conditions, you need to add another Amazon S3 REST connector to the accepted path from the Business Rules shape.

Next, click on the Amazon S3 REST shape to configure it. In the Connection input field, select the Amazon S3 REST Connection you created earlier. Then, choose the GET action and click the + button in the operation field to create a new operation.

This action opens up the Operation page. Click on the Import Operation, and fill out the modal with your credentials:

Atom: Select the Atom you want to run your Boomi process in
Connection: The AWS S3 REST connection you just created.

Click the Next button to open the Amazon S3 REST Operation Import modal. In this modal, select the S3 bucket with the files you want to get, then click on the Next -> Finish buttons.

Click the Save and Close button to save your Amazon S3 operation and return to the sidebar.

Next, you pass in the file key as a parameter to the next AWS call, allowing you to read the content of the file key. To pass the file key as a parameter, go to your sidebar, navigate to the Parameters section, and click the + button to create a new parameter.

This action opens up a Parameter Value modal. In this modal, fill in the following information:

Input - Select the ID field in the New Input modal
Type - Profile Element
Profile Type - XML
Profile - Your newly created query Amazon S3 profile
Element - Key

Click on the OK button to save your parameter. Your newly created parameter should appear in the parameter section.

Then click on the OK button to save your Amazon S3 REST configuration.

5.Change file type from CSV to XML

Next, you will transform the CSV files you get from the AWS S3 bucket to an XML format for more compatibility with your Salesforce application.

To change this CSV format, add a Map Shape to the Boomi process. Click on this shape to configure it, which opens up a Map sidebar.

Click the + sign in the Map input field in this sidebar to open the Map page.

Go to the Map source section on this page and select the Choose link.

This action opens the Choose a Source Profile modal; in this modal, select the Profile Type of Flat File, and in the Profile field, select the CSV profile you created previously.

Click the OK button to see your data fields populated in the Elements menu.

Next, Click the Save and Close button, as you will need to create a Salesforce profile before you can map the data from the CSV file to the Salesforce field.

6.Adding a Salesforce connector to store the data

Next, add a Salesforce connector to your Boomi process.

Click the Salesforce connector to open the sidebar and configure it. In the sidebar, fill out the input fields with the following information:

Connection: Salesforce connection allows you to connect your Boomi process to the Salesforce
Action: The action will be Send as you are upserting data into your Salesforce application instead of retrieving
Operation: This defines how to interact with your Salesforce account.

To learn how to create a Salesforce connection and Operation, check out How to sync your MySQL database with Salesforce in Docker using Boomi.

Click the OK button to save the configuration for the Salesforce connector Shape.

7.Mapping the CSV values to the Salesforce fields

To map your data to the fields in Salesforce, open up your CSV to XML map shape, and click the pencil icon to edit the Map shape.

On the right side of the page, the Map destination, find the Choose button and click it. This action will open the Choose a Profile modal.

Select XML from the Profile Type dropdown menu within the modal. Then, choose the Salesforce Profile you just created. Confirm your selection by clicking OK.

The next step involves linking the corresponding fields between your CSV file and Salesforce. Simply drag your cursor from a field on the left side and connect it to its matching field on the right. For example, in the diagram below, the following fields are mapped together:

booking_id -> Booking_ID_c
customer_name -> Name
customer_email ->Email_c
customer_company -> Company_c

Once you've mapped all the fields, your page should resemble the example provided.

Next, click the Save and Close button to save this Map and return to the sidebar.

Then, click the OK button on the sidebar to save your Map Shape configuration and return to your Boomi process canvas.

Finally, add an End and Continue shape to signify the end of the process.

8.Testing the Boomi process

To verify if the process is working, click the
Arrange button, then save and test the process.

Next, click on the Salesforce application and navigate to the Shape Source Data tab below."

You will see the data packets; click on one of these packets to see the data in your database.

After completing the process, your database data will be visible in your Salesforce application.

That's a Wrap

This guide has shown you how to build a basic ETL process using Boomi. You learned how to extract data from your AWS S3 bucket, check if it was modified today, transform it into an XML file format, and finally load it into your Salesforce application for easy access.

ETL processes are a common and valuable application of integration platforms in modern software engineering. However, this is just a starting point for exploring the full capabilities of Boomi. Refer to the official documentation for more information on what Boomi can do.

How to convert XML files to CSV format using Boomi in Docker

Amarachi Iheanacho — Wed, 03 Apr 2024 07:23:51 +0000

With its structured hierarchy, Extensible Markup Language (XML) excels at storing and sharing data across different applications. However, a simple comma-separated value (CSV) table format shines for its easy readability and compatibility with many data analysis tools.

This conversion bridges the gap between these formats, making data more accessible to use and analyze. Whether managing databases, migrating data, or integrating with other systems, transforming XML to CSV streamlines your information into a format readily usable by a wide range of applications.

This article discusses converting XML data retrieved from an FTP server into a CSV format. It will guide you through setting up the connection, retrieving the files, and converting the data using Boomi.

Prerequisites

To get the most out of this article, you must have the following:

A Boomi platform account: if you do not have one, create a 30-day free trial account
A Boomi Atom is installed locally. To understand how to install a Boomi Atom on Docker, check out this Installing Boomi Atom runtime on Docker tutorial
A basic understanding of Docker and its commands

Creating an XML data file in your project’s root directory

An Extensible Markup Language (XML) file is a file that is used to store data in the form of hierarchical elements. This data stored in this way can be easily exchanged between different computer programs and platforms because XML utilizes a standard format. This makes XML a versatile tool for sharing configuration settings, financial records, or scientific data.

Before transforming your data, you must create the XML file itself. In the folder containing your Boomi Atom, create an "ftp-data" folder. Within this folder, create a file named data.xml. Copy and paste the following code into this file to populate it with sample customer data.


<xml version="1.0" encoding="utf-8"?>
<customersList>
    <customer id="1">
        <name>Aisha</name>
        <email>aisha@senegal.com</email>
        <country>Senegal</country>
        <state>Dakar</state>
    </customer>
</customersList>

Connecting an FTP server to the Boomi Atom

A File Transfer Protocol (FTP) server is a digital storage locker you can access online. It uses a specific set of rules called File Transfer Protocol (FTP) to upload and download files.

You can use an FTP server as a secure transfer station to send files into your Boomi Atom, specifically the newly created XML file. This section will walk you through the process of creating and connecting an FTP server to your Boomi Atom, building upon the knowledge you acquired installing the Boomi Atom in Docker.

To achieve this, update your docker-compose.yml file with this additional information and replace placeholders with your specific values.

version: '3'
services:
  atom:
    image: boomi/atom:5.0.2-rhel
    container_name: boomi-atom
    volumes:
      -<your host directory>/data:/mnt/boomi:Z
    environment:
      - BOOMI_ATOMNAME=<your atom name>
      - INSTALL_TOKEN=<your atom installer token>
      - BOOMI_ENVIRONMENTID=<your environment ID>
      - ATOM_LOCALHOSTID=<your atom name>
    ports:
      - "9090:9090"
  ftp:
    image: delfer/alpine-ftp-server
    container_name: ftp-server
    environment:
      - USERS=<username>|<password>  # Use a strong password!
      - ADDRESS=ftp
    volumes:

      - <path>:/home/ftpuser/ftp
    ports:
      - "21:21"
      - "20:20"
      - "21000-21010:21000-21010"

This code block above defines two services:

I. The Boomi Atom service which is configured by the following fields:

image: boomi/atom:5.0.2-rhel: This specifies the Boomi Docker image used for the service.
container_name: This allows you to personalize the container name, making it easier to identify and manage
<your host directory>:/mnt/boomi:Z: This mounts a host directory onto the container's /mnt/boomi directory with the "Z" compression option. Make sure the host directory exists and has the necessary permissions.
BOOMI_ATOMNAME, INSTALL_TOKEN, BOOMI_ENVIRONMENTID, ATOM_LOCALHOSTID variables: These are specific to your Boomi Atom setup and are discussed intensively in this installing Boomi Atom runtime on Docker article.

II. The FTP service, which is configured by the following fields

image: delfer/alpine-ftp-server: This uses the official delfer/alpine-ftp-server
USERS: This sets the FTP user and password
ADDRESS: This is the external address to which clients can connect for passive ports
volumes: <path>:/home/ftpuser/ftp: This mounts a host directory to the containers home/ftpuser/ftp directory. This directory must contain the data.xml file
ports: This exposes the ports for connection

Converting your XML data format to a CSV format

This section walks you through a series of steps that allow you to convert your newly created XML file.

1.Collecting your XML file from the FTP server

To create the Boomi process that converts your XML file, navigate to the Integration page and click the Create New button. From the dropdown menu, select Process.

Upon selection, the Start Shape sidebar will appear. Here, opt for the Connector radio button. Then, in the Connector input field, choose FTP. You will retrieve the XML data for conversion from your newly created FTP server.

To proceed, click on the plus icon within the Connection input field.

This action will open the connection page, where you should enter your FTP credentials into the provided input field.

Host: The FTP_HOSTin yourdocker-compose.yml` file
Port: Use the default port 21
Connection Mode: Passive
User Name: The username in the FTP_USERS field in your docker-compose.yml file
Password: Click the <Encyrpted> button to open a password modal. In this modal, input the password in the FTP_USERS field in your docker-compose.yml file

Click the Save and Close button to return to the Start Shape sidebar.

Click the + icon in the Operation input field in the sidebar. This will open the FTP operation page.

On this page, fill out the necessary input fields with the required information.

FTP Action: Select Get to retrieve the file
Remote Directory: Input the directory where you mounted the folder containing the data.xml file in your docker-compose.yml
File Filter: Input ‘data’ to specify that you want to retrieve the file with the name data.
Transfer Type: Change the transfer type from Binary to ASCII to transfer files as text
Maximum files to Read: Leave the value at zero to signify that you want to read all the files in that directory

Click the Save and Close button, then in the Start Shape sidebar, click on the OK button to save that FTP connector.

2.Set properties of the data.

Next, you will use the Set Properties shape to dynamically set the output file name for the data coming from the Start Shape. Additionally, you will map the data values from your XML file to the fields you wish to create in your CSV file.

To ensure these processes run sequentially, the Branch shape comes into play.

At the end of the dotted line from the Start shape, you'll find a small box with a plus + sign in its middle. Click on this box to open a modal, then search and click on the Branch shape within this modal.

The Branch shape typically offers two paths by default, with a maximum of 25. However, for this tutorial, you will only need two paths.

Click on the box representing the first path from the Branch shape. Then, select the Set Properties shape. This will enable you to set values for documents and process their properties.

Click on the Set Properties shape to configure it.

In the Properties to Set section, click the + sign to open the Choose Property modal. In this modal, select:

Property Type - Document Property
Source Type - Connectors
Connector - Disk
Property - File Name

Next, click on the OK button.

You should see a Disk parameter. Select this parameter, and click on the + sign in the Property Value section. This action opens up the Parameter Value modal.

In this modal, change the Type from Static to Profile Element and set the Profile Type to XML.

Then, click the + icon within the Profile input field to create a new XML profile. This action will take you to a new profile page.

Click the Import a Profile button on the new profile page.

Clicking this button will open up an XML Import wizard modal. From the dropdown in this modal, select XML File to build an XML profile.

Next, upload the data.xml file that you created initially. After uploading, click on the Next button.

You should now see the Profile Loaded modal.

Click the Finish button to see the XML profile.

Click the Save and Close button to return to the Parameter Value modal.

In the Parameter Value Modal, click the Element field, expand the menus, and select the name element in the customer object. This specifies that each XML file should be titled in the customer’s name.

To set the default type value of the files to a .xml file, create another Parameter value. Click on the + sign in the Parameter Value section. This action opens a Parameter value modal. In this modal, choose a Type of Static, and input .xml in the Static Value field.

Click the OK -> OK buttons.

3.Creating the Map source and destination data Profiles

Boomi profiles describe the layout or format of the documents read into or sent out of processes. This section will create profiles that describe the source XML file and the expected destination or output CSV file.

To create these profiles, click on the second branch from the Branch Shape and search for the Map Shape.

To configure the Map Shape, click on it. This action will open the Map Shape sidebar. Within this sidebar, click the + icon in the Map input field to open a new Map page.

In the Map source section, click on the Choose link at the top right of the Map source section.

This opens up a Choose a Source Profile modal. In this modal, select a Profile Type of XML, and for the Profile input field, select the profile you just created.

Click on the OK button to see that section populated with your object from your XML profile.

In the Map destination part of the page, click on its Choose link to open up the Choose a Destination Profile modal.

In this modal, select a Profile Type of Flat file, and click the + button in the Profile input field to create a new profile for the CSV file.

This opens up a New Flat File Profile page. Right-click the downwards-facing arrow next to Elements on this page, and select Add Multiple Elements. This opens up a modal that asks you to choose how many elements you'd like to add. This tutorial decides to create six new elements. Click on the OK button to close the modal.

Next, click on each of the newly created elements and change them so that they can respond with the field in your XML profile. This tutorial creates fields for name, email, country, city, id, and current date.

Click on Save and Close and the OK buttons to save this profile.

4.Mapping the XML data fields to the CSV data fields

Next, click on a field in the Map Source on the left-hand side, drag a line from the field to their corresponding field on the Map destination, and do this for all the fields. For example, in the diagram below, the following fields are mapped together:

id -> id
name -> name
email -> email
country -> country
state -> city

After you are done, you will notice that the current date field does not match with any data. To get the current date, click the + button in the Functions part of the Map page.

Clicking on the + button opens up an Add a Function modal; in this modal, select a Category of Date, click the Get Current Date function, and Click the OK button. You should see a small box titled Get Current Date in the middle of your Map.

Next, drag the line from the Results point of the box to the current date menu, then click the Save and Close button. Next, click on the OK button in the Map shape sidebar.

5.Configuring the Disk Connector

The Disk connector gets files or sends files to directories on the disk to which the Atom has access, so once the Map shape transforms the data from XML to CSV, the Disk connector stores this transformed data on your atom.

To add the Disk connector to your Boomi process, click on the box along the dotted lines that lead out of the Set Properties shape. Then, search for the Disk connector and click on it to open the sidebar.

Change the Action from Get to Send in this sidebar, as the Disk connector will send out the data. Then, click the + sign in the Connection input field, which opens up the Disk connection page. On this page, input the directory where this newly converted file will live in your atom.

Next, click the Save and Close button to return to the Connector Shape sidebar.

In the Operation input field, click the + sign to create a new Operation for the sidebar. The Disk Operation defines how to read files from or write files to a given directory.

In this Operation page, tick off the Create Directory if it doesn’t exist checkbox. This allows the Disk Connector to create a new directory if the directory you specified in the connection does not exist.

Next, select what you want the Disk Connector to do in the event that the file you are trying to convert and store already exists in the directory. This tutorial chooses Create unique name if file exists. Next, click on the Save and Close button, and then click on the OK button.

Connect the lines from the Map Shape to the Disk connector so the Disk connector can store the CSV file from the Map Shape.

Lastly, connect the Disk Connector to the End and Continue Shape.

Your complete Boomi process should look like this.

6.Testing the Boomi process

To test this process, click the Arrange tab at the top of the canvas to straighten the lines and arrange the connections. Next, select the Test button on the top right of the canvas page.

To verify the success of your process, click on the Disk Connector and check out the Shape Source Data. In this tab, you should see two files.

The first file is the XML file, and the second is the newly created CSV file.

Since you specified a directory, you wanted the CSV file to be stored in your Atom.

In your project’s root directory, open the folder you specified in your Boomi Atom volumes mount. Navigate to this folder <Atom name>/<directory you inputted in the disk connector>. Here, you should see both your XML file and CSV file.

That’s a Wrap!

This guide has discussed how an Integration Platform as a Service (iPaaS) can convert data from one format to another, making it more compatible with various programs. In this instance, the conversion was from XML data format to CSV format. However, this is just a glimpse of what iPaaS solutions, particularly Boomi, can offer. Please refer to the official documentation for a deeper understanding of Boomi's extensive capabilities.