DEV Community: absterellio

Using an Elasticsearch client with Typescript and React to search for data

absterellio — Tue, 01 Oct 2024 22:00:46 +0000

Overview

I recently built a search form, pulling data from an Elasticsearch index to display search results. I wanted to explain the concept, my goals, and the outcome. This is a basic example of using Elastic for searching, and through this exercise I feel more comfortable tackling more complex use cases in the future.

Structure of the index:

Below is an example index seen in ElasticVue Chrome extension (https://elasticvue.com/). ID's and other data points are blocked to protect the data, but the general structure of the index is still visible. In this example, we can see there is a list of contracts that each have an ID, a CreatedByContactID, a BillingContact, and a BillingContactID.

Query we want to build:

Let's say this search allows us to input a contact ID, and we want to see the contracts that have a CreatedByContactID or BillingContactID with this value, and we also want to filter any results based on other input criteria, such as a billing contact name and a contract ID (the _id field).

To do this filtering and data extraction from the index, we'll construct a boolean query (https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-bool-query.html), which will allow specifying many different types of fields that we want to search on.

The scoring assigned for this query (aka the calculated relevance of the result) favors matches within the must clause. I can also add filters to this boolean query, although these will not affect the score of the documents returned.

{
    "query": {
        "bool": {
            "must": {
                "multi_match": {
                    "query": 12345,
                    "type": "cross_fields",
                    "fields": [
                        "CreatedByContactID",
                        "BillingContactID"
                    ]
                }
            },
            "filter": {
                "term": {
                    "Id": 678910
                },
                "term": {
                    "BillingContact": "John Smith"
                }
            }
        }
    },
    "size": 10,
    "from": 0,
    "sort": []
}

Search form

I built a sample form in React as a UI for the searching. Suppose this is my form, and my React application takes in values for each of the fields specified. Within my app, I can take these values and build a query to search the records in my index.

Without going to deep into the infrastructure, I can construct my query in Typescript using an Elasticsearch client (https://www.elastic.co/guide/en/elasticsearch/client/javascript-api/current/index.html) to make search calls to the existing Elasticsearch index I connect to.

Simply put, this client provides a ".search()" method I will be calling with the constructed query to bring data back to my UI.

Because my query will check on two fields for the same value, I plan to add a multi-match query that checks two fields on the index for the same value. For this particular multi_match, I specified a cross_fields type because I want the input contact ID to check either of the contact fields for that ID. Because I place this multi_match within a 'must' query, these fields must match the id specified and will contribute to the score of the documents returned.

The extra filters will be added to my elastic query simply as filters to search on. These filters must appear in the documents to be returned, and as mentioned earlier, these filters will not contribute to the calculated score of the results.

First, I can build an array of search filters that will be passed back to the query that we will pass into the Elastic client. Each filter has a term and a value to search on.

  const searchFilter: Array<
  | { term: { [key: string]: string } }
  | { term: { [key: string]: number } }
> = [];

To add the filters (the ID and Billing Contact), I can push elements to the array.

searchFilter.push({ term: { ["ID"]: 678910 } }); 
searchFilter.push({ term: { ["BillingContact" ]: "John Smith" } });

I can construct the multi-match 'must' fields array similarly.

  const must: Array<
    { multi_match: { query: number; type: string; fields: string[] } }  > = [];

must.push({
          multi_match: {
              query: 12345,
              type: "cross_fields",
              fields: [
                "CreatedByContactID",
                "BillingContactID"
              ],
          },
      });

Calling the ElasticSearch client

Assuming we have an existing elasticSearchClient built (please see Elasticsearch documentation to learn how to construct a client for searching), we can pass all of our specifications in and hopefully get back some data from the index.

const results = (await elasticSearchClient.search({
    index: // insert index here,
    body: {
      size: 10,
      from: 1,
      query: {
        bool: {
          filter,
          must
        },
      },
      sort,
    },
  })) as ElasticResponse // create your own typescript object to hold the hits returned;

The output of this code should match the query I outlined at the top of this article. I can use the results in my application and present them back to the user, who will hopefully be seeing the contracts they expected :)

Takeaway

Overall, I learned a lot about constructing queries based on an Elastic index and how to pull out the data I need. I hope this was informative and painted a clearer picture of the concept of constructing boolean queries with bool, must, filter and multi_match queries, especially for those newer to Elasticsearch. I can see how this would be beneficial for quick access to large sets of data and I'm excited to play around more with Elasticsearch.

Thanks for reading and as always feel free to share any thoughts!

Building a C# and .NET Git repository reporting tool with Azure DevOps

absterellio — Mon, 24 Apr 2023 20:21:35 +0000

I've been using the Azure Devops CLI to build programs to generate reports on a TFS repository that uses Git for version controlling (see: Using Azure Devops CLI to generate repository reports and Adding a simple (old-fashioned) GUI to Azure Devops CLI script
).

I decided to build a more robust C# and .NET application that utilizes the Azure DevOps SDK.

Overview

This program takes in a TFS iteration path and returns a report of TFS work items, Git pull requests and other details from that iteration. This report is generated as a .csv file with columns for WorkItem, Pull Request, Reviewers, and Target Branch.

The Code

The project can be found on my GitHub page:

absterellio / azure-csharp-dotnet-tool

Overview: This project is a C# and .NET application that takes in a TFS iteration path and returns a report of TFS work items, Git pull requests and other details from that iteration. This report is generated as a .csv file with columns for WorkItem, Pull Request, Reviewers, and Target Branch.

To use this program:

In the ConstantValues.cs file, replace the 'OrganizationURL' and 'PlatformTeamName' with values from your own Team Foundation instance.
Create a personal access token for using the Azure Devops Service .NET SDK (see https://learn.microsoft.com/en-us/azure/devops/organizations/accounts/use-personal-access-tokens-to-authenticate?view=azure-devops&tabs=Windows for more information)
This program is run from the command line. From the command line and in the directory for this project, run "dotnet run [full-iteration-path] [personal-access-token] [output-file-path.csv]", replacing the bracketed values with your unique iteration path, personal access token and desired output file path.

View on GitHub

The Entry Point

Program.cs is the main entry point into the application. It creates an instance of IterationLogic.cs and calls the GetIterationInformation method within that class, where all of my main logic lives.

The constructor of this class creates an instance of a VssConnection, passing in an organization URL and a created VssBasicCredential. This VssConnection will be used to access the classes and methods and to return data from the DevOps services.

Uri orgUrl = new Uri(ConstantValues.OrganizationURL); this.connection = new VssConnection(orgUrl, new VssBasicCredential(string.Empty, personalAccessToken));

The constructor also creates a new TeamContext variable which will be required later anywhere the WorkHttpClient is used.

teamContext = new TeamContext(ConstantValues.PlatformTeamName);

Retrieving an Iteration

The iteration path we want to work with is passed into this method as a string. In order to get information about this iteration we need to create an instance of the WorkHttpClient.

WorkHttpClient workHttpClient = connection.GetClient<WorkHttpClient>();

The following code will retrieve all of the iterations based on the TeamContext created earlier. Using the retrieved iterations, we can filter on the specified iteration path string.

List<TeamSettingsIteration> results workHttpClient.GetTeamIterationsAsync(teamContext).Result; iterationMatch = results.First(x => x.Path == iterationPath);

Retrieving Work Items for an Iteration

The following code will retrieve all work items for a given iteration. The WorkHttpClient contains a method to fetch all work items.

workItems = workHttpClient.GetIterationWorkItemsAsync(teamContext, iteration.Id).Result;

Getting Details for a Work Item

After fetching these work items, we can hammer down further to find more details about a work item. The WorkItemTrackingHttpClient provides a method to get details for a specific work item. This method can also optionally take in a list of specific fields to return as the second argument.

WorkItemTrackingHttpClient witClient = connection.GetClient<WorkItemTrackingHttpClient>(); workItem = witClient.GetWorkItemAsync(workItemId).Result;

Fetching Git Pull Requests for a Work Item

Lastly, the final output I want to return is a list of pull requests submitted for each work item. This will require two clients, the WorkItemTrackingHttpClient and GitHttpClient.

var witClient = connection.GetClient<WorkItemTrackingHttpClient>(); var gitHttpClient = connection.GetClient<GitHttpClient>();

Fetching the work item can be done with the following code.

var workItem = witClient.GetWorkItemAsync(workItemId, expand: Microsoft.TeamFoundation.WorkItemTracking.WebApi.Models.WorkItemExpand.Relations).Result;

The returned work item will have a list of Relation attributes. We can filter out the pull requests from this list, and then loop through and use the GitHttpClient to get details for each PullRequestId returned.

var gitPullRequest = gitHttpClient.GetPullRequestByIdAsync(Int32.Parse(ids[2]));

With the pull request returned, I can map the properties back to my own PullRequest view model to print later.

var prObj = new models.PullRequest { WorkItemId = workItemId, ID = pullRequestResult.PullRequestId.ToString(), TargetBranch = pullRequestResult.TargetRefName, Description = pullRequestResult.Description, };

Overall, this tool ended up being helpful to get some quick data and track workflow on a TFS repository. There are so many more functions that the Azure DevOps .NET SDK provides but I wanted to highlight some of the methods that were helpful to me when building this reporting tool. Writing this program was a great way to get started and jump into programming with the Azure DevOps SDK.

Thanks for reading and feel free to share your thoughts or check out the repository on GitHub!

Adding a simple (old-fashioned) GUI to Azure Devops CLI script

absterellio — Sat, 25 Feb 2023 21:09:39 +0000

Using WinForms!

I wrote a script to pull information from my codebase (see: Using Azure Devops CLI to generate repository reports. I updated the script to present a (very simple) interface to the user to ease usage and provide a way to use the tool without having to deal with command line inputs.

The updated script takes in an input file path, an output file path and personal access token to access the Azure CLI. For more information on the Azure Devops CLI, see the Azure DevOps CLI documentation or my previous article.

The Result

The Code

The script can be found on my GitHub page:

absterellio / azure-devops-cli-tool-v2

azure-devops-cli-tool-v1 with a GUI added

azure-devops-cli-tool-v2

This project adds a basic GUI to the azure-devops-cli-tool-v1 project found here: https://github.com/absterellio/azure-devops-cli-tool-v1.

NOTE: In the azure-repo-script.ps1 file, replace any variables that begin with "ex-" with your own applicable Azure devops values in order for this script to run.

View on GitHub

For this script, I added a simple user interface using Windows Forms. I've never used Windows Forms to build a GUI before so I wanted to take note of how I made use of it and some things I learned along the way. This article will walk through the very basics of adding some input controls and logic to the script.

Getting started with Windows Forms

First, in order to use Windows Forms, I added these line to the top of my file.

Add-Type -AssemblyName System.Windows.Forms Add-Type -AssemblyName System.Drawing

I built an intial landing screen that will pop up when the script is first run. The following code creates a basic pop-up form with the dimensions 600 x 400 pixels in the middle of the screen view area.

$form = New-Object System.Windows.Forms.Form $form.Text = 'Repo Report' $form.Size = New-Object System.Drawing.Size(600,400) $form.StartPosition = 'CenterScreen'

Adding inputs and text fields:

New inputs can be added as follows. To build a label, I added a Label control, specifying a start point, size, and text value for the label.

$label1 = New-Object System.Windows.Forms.Label $label1.Location = New-Object System.Drawing.Point(150,70) $label1.Size = New-Object System.Drawing.Size(280,40) $label1.Text = 'Enter a text file to read from. (Note: Specify each repo you would like to query on a new line of the file):' $form.Controls.Add($label1)

To build an input field, I added a TextBox, specifying a start point and size for the input.

$textBox1 = New-Object System.Windows.Forms.TextBox $textBox1.Location = New-Object System.Drawing.Point(150,110) $textBox1.Size = New-Object System.Drawing.Size(260,20) $form.Controls.Add($textBox1)

I added a cancel button with the Button control. In addition to specifying dimensions and size, I added a DiaglogResult property, which will help to handle actions when this button is clicked.

$okButton = New-Object System.Windows.Forms.Button $okButton.Location = New-Object System.Drawing.Point(200,200) $okButton.Size = New-Object System.Drawing.Size(75,23) $okButton.Text = 'OK' $okButton.DialogResult = [System.Windows.Forms.DialogResult]::OK $form.AcceptButton = $okButton $form.Controls.Add($okButton)

Handling events:

After creating all of the inputs, I showed the dialog by specifying:

$result = $form.ShowDialog()

Now, '$result' will hold the next button press or event thrown from the UI. We can now handle any event, such as a button press, within an 'if' check for that control.

if ($result -eq [System.Windows.Forms.DialogResult]::OK)

Any logic that follows can live here.

I hope this article gave some insight into getting started with WinForms and adding a very simple interface to a PowerShell script. Input or suggestions always welcome.

Thanks for reading!

Using Azure Devops CLI to generate repository reports

absterellio — Thu, 22 Dec 2022 21:27:51 +0000

I needed to see some reports on my codebase, such as information about merges to the main branch of code. To do this, I made use of Microsoft’s Azure DevOps CLI and its corresponding extensions to write a PowerShell script that generates a CSV file with a report of completed pull requests.

Resources

Azure DevOps CLI documentation
The completed initial version of this script can be found on my GitHub:

absterellio / azure-devops-cli-tool-v1

Basic Powershell script that uses the Azure Devops CLI to generate repository reports

azure-devops-cli-tool-v1

Basic Powershell script that uses the Azure Devops CLI to generate repository reports.

NOTE: In the script.ps1 file, replace any variables that begin with "ex-" with your own applicable Azure devops values in order for this script to run.

Blog post with more details: https://dev.to/absterellio/using-azure-devops-cli-to-generate-repository-reports-1h7d

View on GitHub

Setup

To begin, we need to set up a few things. The Azure DevOps CLI documentation linked above has more details on this. The CLI installation is a very quick download process.

1. Install Azure DevOps CLI
2. Add the Azure DevOps extension

The Code

Like stated above, my goal was to generate reports on specific repositories in my codebase. Below, I noted some key ways I made use of the CLI in my script.

Authentication:

To begin using the CLI, we need to authenticate. Following the instructions on this page, Azure DevOps PAT documentation, I created my own personal access token (PAT). The following line takes in the token and authenticates against your organization.

echo $pat | az devops login --organization ex-projectLink

"ex-projectLink" should be replaced with the applicable link to the project, and the $pat variable should be the created access token.

Listing pull requests made into a repository:

The CLI specifies a reference for working with repositories (az repos). I added flags to specify completed PR’s that went into the repo’s 'main' branch. "$projectName" should be the name of the project that you are querying.

az repos pr list --project $projectName --repository $repo --status completed --target-branch ex-main --include-links | ConvertFrom-Json

The ConvertFrom-Json is a built in cmdlet that I used to convert the json return value into an object so I can grab properties from it later.

Retrieving work items attached to a pull request:

az repos pr work-item list --id $pr.pullRequestId | ConvertFrom-Json

Replace {$pr.pullRequestId} with the corresponding Id of the pull request you’d like to grab work items for and this will return a list of items that were attached to this PR.

Outputting the results to a csv file:

First, I set the headers of the CSV file that I’ll output my results to.
Set-Content -Path $location -Value '"PR Number","Date Closed","Repo Name","Created By","Reviewers","Work Item Id"'

With the returned PR, grab the id, names of creators/reviewers, closed date, etc. from the pull request and any other applicable values.

$workItemId = $workItem.id $createdBy = $pr.createdBy.displayName $reviewedBy = $pr.reviewers.displayName $date = $pr.closedDate

Then, output the results to the file.

Add-Content -Path $location -Value "$prNumber,$date,$repo,$createdBy,$reviewedByString,$workItemId"

I hope this serves as a quick introduction to the Azure DevOps CLI and its basic uses for querying repository and pull request history. Next, I plan to add a GUI and more usability features to this tool for everyday usage.

Thanks for reading! Feel free to have a look at the script in GitHub and leave thoughts or suggestions for improvement.

DEV Community: absterellio

Using an Elasticsearch client with Typescript and React to search for data

Overview

Structure of the index:

Query we want to build:

Search form

Calling the ElasticSearch client

Takeaway

Building a C# and .NET Git repository reporting tool with Azure DevOps

Overview

The Code

absterellio / azure-csharp-dotnet-tool

Adding a simple (old-fashioned) GUI to Azure Devops CLI script

Using WinForms!

The Result

The Code

absterellio / azure-devops-cli-tool-v2

azure-devops-cli-tool-v1 with a GUI added

azure-devops-cli-tool-v2

Using Azure Devops CLI to generate repository reports

Resources

absterellio / azure-devops-cli-tool-v1

Basic Powershell script that uses the Azure Devops CLI to generate repository reports

azure-devops-cli-tool-v1

Setup

The Code