DEV Community: Seenevasaraj

SQL CRUD OPERATION USING PYTHON

Seenevasaraj — Tue, 05 Mar 2024 06:09:15 +0000

Hey Reader,

My name is Seenevasaraj, and I am working as Software Developer at Luxoft India. The various project at Luxoft I am implementing automation in all viable way. Through this article I desired to provide an explanation about how we can perform CRUD operation in SQL using python module.

SQL
Structured Query Language (SQL)-standardized programming language used to manage relational databases&perform various operations on data in them,created in the 1970s SQL is regularly used database administrators and developers writing data integration scripts, data analysts looking to set up analytical querie

SQL Uses

modifying database table &index structures;
adding updating&deleting rows of data.Retrieving subsets of information from relational database management systems (RDBMSes) information can used for transaction processing analytics applications and other applications that require communicating with a relational database

SQL queries and other operations take form of commands written as statements and aggregated into programs that enable users to add modify or retrieve data from databasetables

table is most basic unit of database and consists of rows columns of data,single table holds records and each record is stored in row of table,tables are most used type of database objects or structures hold or reference data in relational database

logical representations ofdata assembled from onORmore database table
Indexes are in tables help speed up database lookup function
Reports consist data retrieved from oneORmore tables usually subset ofdata is selected based on search criteria
Each column present in table corresponds to category of data eg:customer name or address while each row contains data value for intersecting column

Relational databases- relational because they are composed of tables relate to each other,eg. SQL database used for customer service can have one table for customer names and addresses and other tables hold information about specific purchases product codes and customer contacts,table used to track customer contacts usually uses unique customer identifier called key or primary key to reference customer record in separate table used to store customer data such as name and contact information

SQL is very picky language when it comes to syntax

CRUD
SQL-compatible databases used for their reliability and transactional support, SQL can prove difficult for average non-technical user especially when complex filters sorting or data merging are required,to combat this organizations build internal tools facilitate CRUD operations to interact underlying database

Creating data
Retrieving data
Updating data
Deleting data

CRUD Using Sql Command
For CRUD operations, we will create table with four columns of varying data types, this is not a CRUD operation (even though it’s called a CREATE operation)this is just provisioning table to query from

We can use CREATE TABLE statement to create table

INSERT(Create)
INSERT operation used to add records, INSERT statements can pick and choose which columns to fill when adding row alternatively, developers can provide values for all fields

SELECT(READ)
SELECT statement used to retrieve records from database,while SELECT statement will return all data from rows by default we can specify which rows and/or columns we need

Select all columns and rows-use * operator to select all columns and rows from database table

UPDATE
UPDATE statement is used to update existing records in database

UPDATE statement developer update oneORmore columns in specific row(s) by coupling with where clause, Predictably where specifies which rows need to updated

We will use following syntax to update specific column(s) based on the condition

DELETE
DELETE statement used to delete records from database table

like UPDATE when using DELETE statement we can specify one or more row(s) to deleted by using where clause,where clause is critical—otherwise on some SQL databases we will delete all of our data
delete specific row from database table

Python Prerequisites:

pyodbc

CRUD Using Python
use SQL Server in Python application establish connection to database

create database table to do CRUD operations

insert data into the database table

Update user details in table

Delete user data from table

error handling is crucial working with databases close SQL Server connection when done

Conclusion
I have explained basics of performing CRUD operations using SQL Server Python module to do create table, insert data, query data, update records and delete records

MANGO DB CRUD OPERATION BY PYTHON

Seenevasaraj — Tue, 05 Mar 2024 05:34:47 +0000

Hey Reader,

MongoDB
MongoDBdocument database to build highly available scalable internet applications,flexible schema approach,popular with development teams using agile methodologies,offering drivers for all major programming languages

Instead of storing data in tables of rows or columns like Relational databases, records in MongoDB database is document described in BSON binary representation of data,applications can then retrieve this information in JSON format

below JSON document describing a historical figure.

Document databases highly flexible allowing variations in structure of documents&storing documents are partially complete one document can have others embedded in it,fields in document play role of columns in SQL database and then like columns, they can be indexed to increase search performance

MongoDB built on scale-out architecture,structure that allows many small machines to work together to create fast systems and handle huge amounts of data

MongoDB focused on providing developers with excellent user experience, in addition to all its other properties has made MongoDB favorite of developers worldwide for wide variety of application

Use Of MongoDB

MongoDB makes it easy for developers to store structured or unstructured data,using JSON-like format to store documents json format directly maps to native objects in most modern programming languages,developers do not need to think about normalizing data,MongoDB handle high volume&scale vertically or horizontally to accommodate large data

CRUD

Creating data
Retrieving data
Updating data
Deleting data

Create
CreateORinsert operations add new documents to collection,If collection not exist,create operations also create the collection

You can insert single document or multiple documents in single operation

MongoDB shell provides some methods to insert documents into collection:

insert a single document-db.collection.insertOne()
insert multiple documents-db.collection.insertMany()

Read
Read used to retrieve documents from collection; i.e. query collection for documents

You can specify criteria or filters,which identify documents to return

Use to collect all data -db.collection.find()

Update
Update operations modify existing documents in collection,you can update single document or multiple documents in single operation

You can specify criteria or filter which identify documents to update,filters use same syntax as read operations

update a single document-db.collection.updateOne()
update multiple documents-db.collection.updateMany()
replace document-db.collection.replaceOne()

MongoDB preserves own sort order in storing of documents,ordering internal implementation feature ,you should not rely on any particular structure within it

Delete
Delete operations remove existing documents from collection,you can remove single document or multiple documents in single operation

You can specify criteria or filters identify documents to remove filters use same syntax as read operations

delete multiple documents-db.collection.deleteMany()
delete single document-db.collection.deleteOne()

Python Module

pymongo

CRUD Using Python

Prerequisites

MongoDB compass must be installed, to make local database connection
After installation connect to MongoDB compass by passing the below string as input and click on the connect button.
mongodb://localhost:27017/

Steps to do CRUD

Import the pymongo package,and thendefine variable to store the MongoDB connection URL.To make connection to MongoDB server, we will use MongoClient method

by using client instance created, can access databases in MongoDB,display databases use list_database_names()

In MongoDB, unless we add any document to database, we cannot see database created in list of available databases,to create document, we need to create collection in database which stores documents

Inserting documents

Inserting documents into collection can be done two ways

insert single document - insert_one() method

insert multiple document - insert_many() method

Retrieving

Retrieving data can be done in two ways

retrieve single document - find_one() method,need to pass query as parameter to find_one() method .find_one() method search collection and it will return first hit document

retrieve multiple documents - find_many(),find_many() is cursor object-use for loop to loop over cursor object

Updating

Updating documents can be done by two ways

update single document - update_one() method

to update documents,need to provide 2 parameters into method,first parameter is filter parameter&second parameter is update parameter,filter parameter is query matches document to be updated, Update parameter is modifications to be applied for document

update multiple documents - update_many()

Deleting

Deleting documents can be done by two ways

delete single document - delete_one() method by passing query as parameter

delete multiple documents - delete_many()

Dropping collection
drop collection - drop() method,drop method will erase entire data in collection

Conclusion
MongoDB accommodate wide Varity of data format, So it can be used for lot of purpose and along with python we can perform CRUD operation effectively and automated way.

READ FILES IN GIT REPO USING DIFFERENT MODULES IN PYTHON

Seenevasaraj — Mon, 26 Feb 2024 12:15:51 +0000

Hey Reader,

GIT
Git is distributed version control system widely used in source code management and tracking changes in source code,It created by Linus Torvalds in 2005 to manage the development of Linux kernel, it become the standard for version control in many other projects and industries

Repository-A repository or repo, is a collection of files and their revision history. Git repositories can exist locally on your computer or remotely on a server.

Clone-Clone a repository means to create a copy of it on your local machine

Commit-Commit is snapshot of changes made to the repository at a specific time. It represents a single atomic unit of change.

Branch-Branch is a parallel version of a repository codeIt allows developers to work on separate features or fixes without interfering with the main codebaseGit uses branches to manage changes.

Merge-Merging is process of integrating changes from one branch into another it combines the changes made in one branch with another branch

Pull Request-pull request[PR] is way to propose changes to repositoryIt allows developers to review code, discuss changes, and collaborate on projects in many workflows,pull requests are used for code review before changes are merged into the main codebase

Push-pushing refers to sending committed changes to remote repository and making them accessible to others

Fetch-Fetching retrieves changes from remote repository without merging them into your local branchesIt updates your local repository with changes from remote server

Pull-Pulling is combination of fetching changes and merging them into your local branch

Remote-remote is version of the repository hosted on the internet or network,It serves as common repository for all team members to exchange their changes

These are just a few of the fundamental concepts and commands. Git powerful tool with many features and capabilities that enable efficient collaboration and version control in software development projects.

Modules to read files in git branch

requests
gitpython

requests module

To establish connection with Git branches in Python requests module typically interact with a Git repository through its remote URL using Git's HTTP(S) API

In this example:

We call the URL to Git repository, which is used to interact with using HTTP.
We can make GET request to this URL using requests.get()
We can read response lines and extract branch names using iter_lines() method of the response object,branch names typically found in lines starting with "refs/heads/"

Git repository is accessible over HTTP(S) and that the server supports Git's HTTP protocol,some repositories may require authentication, include appropriate authentication headers in the request

gitpython

install GitPython by using pip install GitPython command. provide proper repo_path, branch, and file_paths variables according to your Git repository structure and the files you intend to read.

Read files from Git repository using GitPython in Python,which provides high-level interface to interact with Git repositories.

import the Repo class from the git module of GitPython
define a function read_files_from_git that takes path to repo_path, the branch name and list of file paths as input.
Inside function open the Git repository using Repo(repo_path).
retrieve the commit at the head of the specified branch using repo.commit(branch)
iterate over the list of file paths provided and read the contents of each file from the commit's tree object using commit.tree[file_path].data_stream.read().decode('utf-8')
return python dictionary format containing the file paths as keys and values

Comparision of requests and gitpython

Purpose:

Requests: requests is primarily used to make HTTP request,it allows to interact with web services APIs, and web pages

GitPython: it provides interface to interact with Git repositories directly from Python code and allows to perform various Git operations eg,cloning committing, branching, merging,etc..

Level of Abstraction

Requests: requests operates at lower level of abstraction dealing directly with HTTP requests&responses,provides functionalities for sending HTTP requests & responses managing cookies and setting headers

GitPython: It operates at higher level of abstraction providing an object-oriented interface to interact with Git repo and abstracts away many of low-level details of Git commands and allows to perform Git operations programmatically

Use Cases

Requests: requests is common tasks - consuming RESTful APIs, fetching web page downloading files and interacting with web services

GitPython: GitPython is used to work with Git repositories programmatically like automating Git operations analyzing repository data extracting file contents or integrating Git functionality into larger Python applications

Conclusion
As developer, based on development environment they can select either requests or gitpython, more less both have same kind of functionalities. But requests module have more advantage by making REST-API requests.

JENKINS CI/CD PIPELINE FOR PYTHON PROJECT

Seenevasaraj — Fri, 23 Feb 2024 09:10:12 +0000

Hey Reader,

My name is Seenevasaraj, and I am working as Software Developer at Luxoft India. The various project at Luxoft, I am implementing automation in all viable way. Through this article I desired to provide an explanation about Jenkins and how it is helps in CI/CD process and how you can setup python environment in jenkins.

What is CI/CD?
CI/CD is development approach that automates the integration testing and delivery of code updates to production environments. The CI/CD process typically includes the following steps:

Continuous Integration: Developers merge their code changes into a shared repository regularly, often several times a day.
ContinuousTesting: Automated tests are run on the merged code to check for errors andensure the codeworks as expected.
ContinuousDelivery(orDeployment): Once the testing passes the code is automatically deployed to a production environment or a testing environment that closely resembles production. This way developers can quickly and reliably deliver new features bug fixes, and updates to production, while maintaining a high quality of software.

CI/CD depends on

Version ControlSystem: - Git (or) other centralized systems allow developers to commit their code changes
Automated Build Tools: - Tools like Jenkins,Travis CI or GitLab CI automatically build the application when code changes are pushed to the repository
Automated Testing: - Unit tests integration ,other tests run automatically during the CI/CD process to ensure code quality
Artifact Repositories: - A storage location for built artifacts like Dockerimages or compiled binaries produced during the CI/CD process

CI/CD (continuous integration and continuous delivery) is an essential modern softwaredevelopment practice that streamlines processes enhances code quality and makes deployments more reliable It enables Agile&DevOps teams to rapidly develop test and deliver software

What Is jenkins?
Jenkins is a popular open-source tool that helpsSsoftware development teams automate their tasks It is written in Java and assists in building testing and deploying applications With Jenkins teams can implement ContinuousIntegration(CI) and ContinuousDelivery(CD) pipelines enabling them to automate the development process ensuring the quality and consistency of their software.

ContinuousIntegration: - Jenkins integrates code changes into a centralized repository continually. - It triggers automatic builds when changes are pushed to version control systems.
Extensibility: - Jenkins has large ecosystem of plugins that enhance its capabilities Plugins integrate with version control systems, build tools testingframeworks and deployment platforms.
BuildAutomation: - Jenkins automates the build process for various programming languages and build tools. - It compiles code, executes tests, and generates deployable artifacts
Jenkins allows teams to define build pipelines using code providing version control transparency and flexibility for complex CI/CD workflows
Jenkins supports distributed builds enabling teams to efficiently scale their build infrastructure horizontally resulting in faster build times and optimal resource usage.
Jenkins integrates with external systems like issue trackers collaboration platforms and cloud providers facilitating seamless collaboration issue tracking, and cloud resource management.
Monitoring and Reporting: Jenkins has dashboards and reports that let you keep track of the status of your builds and pipelines It also gives you insights into how long builds take how tests are doing and other things. This can help teams find bottlenecks and make their CI/CD processes more efficient
Security and Access Control: Jenkins has strong security features like authentication, authorization and encryption It can also be integrated with LDAP Active Directory and other identity providers This lets organizations set up access controls and keep track of who did what

Declaring Python Path In Jenkins

In order to execute python scripts in jenkins we have to declare our python.exe and pip.exe path in jenkins environmental variable section.

Log in Jenkins instance and enter the dashboard.
On the left-hand menu click the "Manage Jenkins" link.
Under the "Manage Jenkins" page, select the option that says "Configure System." Here you can modify the global settings of your Jenkins instance.
Locate the "Global properties" Section: - Scroll down the page until you find the "Global properties" section.
Activate"Environment variables"- In the "Global properties" section check the "Environment variables" box to enable it.Add an Environment Variable- Click "Add" button to create a new environment variable.
Specify Name and Value- In the "Name" field, enter the name of the environment variable (e.g. "MY_VARIABLE"). - In the "Value" field enter the value for the environment variable (e.g."my_value")

After declaring your python path, you can use this variable across shell or batch scripts as user requirements.

Jenkins New PIPELINE Creation

Install Jenkins: * Download and install Jenkins on your server or computer if not already installed.Refer to the official Jenkins website for installation instructions.
Install Pipeline Plugin: * The Pipeline plugin supports continuous delivery pipelines in Jenkins Ensure it installed on your Jenkins instance From the Jenkins dashboard go to "Manage Jenkins">"Manage Plugins">"Available." * Search for "Pipeline" and install the plugin.
Log in Jenkins account and choose "New Item" to create a new job Put a name for your pipeline task and mark "Pipeline" as the task type, Click "OK".
Set up the Pipeline: - In the pipeline configuration, you can pick how to write your pipeline using Jenkinsfile or Scripted Pipeline. - If you want to use a Jenkinsfile, mark "Pipeline script from SCM" as the definition and tell Jenkins where your Jenkinsfile is, like in Git. - If you'd rather write the pipeline script directly in the Jenkins job settings, choose "Pipeline script" and put your script in the provided box.
Crafting a Pipeline Script: Shape your CI/CD pipeline script using either the Declarative or Scripted syntax. - With the Declarative Pipeline approach, structure and conciseness are at your disposal. - The Scripted Pipeline option grants you the flexibility of Groovy scripting. Visualize your pipeline's journey through stages and steps, employing directives to orchestrate each stage's behavior. Example:

Save and Run Pipeline: - Save the configuration of your pipeline. - Jenkins automatically starts a build based on the trigger you set (like a webhook or cron job). - Manually start the pipeline using the "Build Now" button in the Jenkins job dashboard.

Conclusion
Jenkins is automation tool that helps software development teams implement Continuous Integration(CI)&Continuous Delivery (CD) practices, it allows them to automate various tasks associated with software development and deployment making the process more efficient and streamlined

DIFFERENT TYPE OF API PROTOCOLS AND ARCHITECTURES

Seenevasaraj — Wed, 14 Feb 2024 11:23:14 +0000

Hey Reader,

My name is Seenevasaraj, and I am working as Software Developer at Luxoft India. The various project at Luxoft, I am implementing automation in all viable way. Through this article I desired to provide an explanation about different types API's are used across software developers and it's usage.

What is API
The API(Application Programming Interfaces) is a powerful and versatile means to connect diverse and disparate software applications. APIs permit a giant array of unrelated software merchandise to integrate and interoperate with different software program and records. APIs additionally permit builders to add functions and functionality to software by means of utilising a wealthy array of different developers' APIs. Much of modern-day enterprise, cellular and internet software relies upon on a wide range of APIs.

Developers can work with an assortment of API types, protocols and architectures that suit the unique needs of different packages and agencies.

Different Types

REST APIs (Representational State Transfer):

Architecture: RESTAPIs represent data's in simple formats (JSON or XML).
- Our system boasts a uniform interface, making it engaging and easy to use for all clients. It also boasts a stateless architecture, ensuring efficient and smooth communication. With our native caching feature, clients can enjoy a speedy and seamless browsing experience. Deemed as a reliable and robust client-server architecture, our system supports a layered system model, providing flexibility and adaptability. Lastly, our system has the ability to provide executable code to the client, enabling them to access and execute relevant functions promptly and effectively.
Pros:
- Flexibility is achieved by decoupling the client from the server. This also means that the system is cache-friendly and can handle various formats with ease.
Cons:
- Having extensive metadata can result in significant payloads, while not having a clear agreement on message structure can present difficulties during implementation.

Common Use Cases: Easily create minimalistic resource-focused applications using Management APIs.

SOAP APIs[Simple Object Access Protocol]:

Format: SOAP-APIs use XML files.
Pros:
- Communication is bolstered by robust standards, while security is seamlessly integrated into the system.
Cons:
- XML format often leads to verbosity, resulting in slower performance when compared to REST

Common Use Cases: By adapting By utilizing When implementing into business protocols, incorporating enterprise systems and modern applications into practices can transform legacy applications.

Open APIs:

Definition: Publicly available open APIs allow third-party developers to easily access specific features or data from an application.
Use Cases: Creating partnerships with external services and constructing interconnected systems.

Partner APIs:

Definition:Specific partners or collaborators are able to access and utilize partner APIs.
Use Cases: Partner systems and joint ventures are both integral components that we are seamlessly incorporating into our operations.

Internal APIs:

Definition:Internal APIs, which are also referred to as private APIs, facilitate communication between various services within an organization.
Use Cases: Experience the seamless integration of our internal systems through our cutting-edge microservices architecture.

Composite APIs:

Definition: Composite APIs have the ability to merge several APIs, creating a more convenient and streamlined interface for users.
Use Cases: Gathering information from multiple sources and streamlining user interactions.

How To Choose API's According To Your Software Requirements

Identify Your Purpose:

Immediate Problem: Think about the exact problem you're currently facing. What key features are necessary for solving it?
Future Needs: As you consider your application's future needs, contemplate the features and capabilities that will be necessary in the next five to ten years. This proactive thinking will serve as a valuable compass in determining the best API for your usage.

Understand Your Audience:

Who Will Use It?: Determine the intended audience for your API. Is it for internal use within your organization, for partners, or for external developers?
Private, Public, or External: APIs can be categorized as private (for in-house use), public/partner (for known developers or business partners), or external (available to any third-party developer). Choose the type that aligns with your goals2.

Evaluate API Solutions Based on Criteria:

Performance: When it comes to a competitive market, dependable uptime is essential. Don't forget about the importance of speed as well – take performance into consideration.

API Specifications:

REST, also known as Representational State Transfer, is a highly sought-after architectural style for APIs. By setting communication standards between computer systems, REST simplifies the flow of data within applications. With their simple and adaptable nature, RESTful APIs have gained widespread popularity.

API Usage

MicroservicesArchitecture:

In microservices-based applications, each service communicates via APIs. This modular approach allows scalability, flexibility, and independent development of services.

Data Retrieval and Aggregation:

News Aggregators: APIs collect news articles from different publishers.
Financial Data APIs: Retrieve up-to-date data for stock prices, currency exchange rates, or economic indicators with ease.

Automation&Bots:

With the use of natural language processing services, you can easily incorporate chatbot APIs to create conversational bots that interact with your users. - Enhance your testing process by utilizing APIs that enable testing tools to interact seamlessly with web applications.

InternalSystemIntegration:

HRSystems: APIs facilitate the integration of payroll, leave management, and employee databases.
InventoryManagement: APIs connect inventory systems with sales and order processing, creating a seamless connection between the two.

Conclusion
APIs are the heart and soul of contemporary software development. It's crucial to keep in mind that the perfect API selection relies heavily on your unique circumstances and objectives. Regardless of whether you're creating a compact application or an extensive system, comprehending your needs will steer you towards the most suitable API resolution

BIGQUERY DATA INSERTION USING PYTHON

Seenevasaraj — Tue, 13 Feb 2024 08:32:15 +0000

Hey Reader,

My name is Seenevasaraj, and I am working as Software Developer at Luxoft India. The various project at Luxoft, I am implementing automation in all viable way. Through this article I desired to provide an explanation about records insertions into bigquery table using google-cloud-bigquery(python module)

What is bigquery
BigQuery is a fully managed enterprise data warehouse developed by Google. It offers powerful features for managing and analyzing large datasets. Here are some key points about BigQuery:

Serverless Architecture: BigQuery operates in a serverless environment, allowing you to execute standard SQL queries and analyze massive amounts of data without worrying about infrastructure management1.
Scalability: It can handle petabytes of data, making it suitable for organizations dealing with large-scale datasets.
SQL Queries: BigQuery supports querying using a dialect of SQL, which makes it accessible to users familiar with SQL syntax.
Built-in Machine Learning: BigQuery includes built-in machine learning capabilities, enabling advanced analytics and predictive modeling.
Cost-Effective: It’s a cost-effective solution, as you only pay for the resources you use.
Business Insights: By turning big data into valuable business insights, BigQuery helps organizations make informed decisions.

What is google-cloud-bigquery
The google-cloud-bigquery module in Python provides a client library for interacting with Google BigQuery, a powerful data warehouse and analytics platform. Let me break down the key points:

Purpose:

Google BigQuery is designed for querying massive datasets quickly using SQL.
It allows you to analyze data efficiently without worrying about infrastructure management.
You can focus on finding meaningful insights from your data.

Features:

NoOps: No need for database administrators; Google manages the infrastructure.
SQL Queries: You can use familiar SQL syntax to query data.
Pay-as-you-go Model: You pay only for the resources you use.

Installation:

To use this library, follow these steps:
Select or create a Cloud Platform project.
Enable billing for your project.
Enable the Google Cloud BigQuery API.
Set up authentication.

Supported Python Versions:

Python >= 3.7 is supported.
Older versions (Python 2.7, 3.5, 3.6) are not supported.

Example Usage

Bigquery-Python authentication setup

Create google service account to make python client and bigquery authentication

steps to create account
When you create a service account, you need to offer an alphanumeric ID (SA_NAME within the samples below), together with my-provider-account. The ID must be between 6 and 30 characters, and can contain lowercase alphanumeric characters and dashes. After you create a service account, you can't alternate its call.

The carrier account's name seems within the electronic mail deal with that is provisioned at some stage in creation, inside the layout SA_NAME@PROJECT_ID.Iam.Gserviceaccount.Com.

Each carrier account additionally has a everlasting, unique numeric ID, which is generated robotically.

You also offer the subsequent statistics whilst you create a service account:

SA_DESCRIPTION is an elective description for the provider account.
SA_DISPLAY_NAME is a pleasant name for the provider account.
PROJECT_ID is the ID of your Google Cloud challenge.

Pass service account json report as a argument to bigquery Client class

Inserting new records in list format
The first way you can upload data is per row. Here, a list of tuples appends two new rows to the table ‘test_table_creation’ using the function .insert_rows()

Insert into a Nested Table
Like above, when studying from a nested desk it’s additionally essential to deal with nested statistics unique additionally whilst putting information. In order to put in writing records into a unmarried nested desk, you could define a dictionary within the list that contains the nested statistics. The key of every entry then corresponds to the column call into which the statistics is to be inserted.

Upload csv into bigquery table
Now which you recognise how to import man or woman records into Big Query with Python, now we try the whole thing with a CSV. With the Pandas and CSV libraries we can first load information from CSV file.

Summary
Google BigQuery and Python—in this example the Cloud Datalab — is a very powerful toolset. Now you should know how to insert our records into BigQuery using Python. This could be useful for Data Analysts and Data Scientist who want to save their work in BigQuery and eventually visualize it. But also for Data Engineers who can realize data pipelines with Python. A special case is nested data, but since BigQuery works very well with it and often performs better, this topic should be familiar to you.

JIRA ISSUE INFORMATION COLLECTION BY USING REQUESTS MODULE(PYTHON)

Seenevasaraj — Thu, 01 Feb 2024 10:14:59 +0000

Hey Reader,

My name is Seenevasaraj, and I am working as Software Developer at Luxoft India. The various project at Luxoft, I am implementing automation in all viable way. Through this article I desired to provide an explanation for about automation which I have accomplished to generate excel document of JIRA issues by using python script.

JIRA

Issue Tracking: it's miles usually used for issue monitoring competencies. It lets in employer to create, tune, and manage problems, responsibilities, bugs, and other styles of work objects.
Project Management: JIRA helps venture control by means of permitting groups to devise, tune progress, and launch software program or complete tasks.

Script utilization

Management not required to check all of the tasks manually to track productiveness.
By this automation, we can make graph and tune progress of all troubles
Management can make report of worker contribution and project deliveries through this automation.

The configuration, required, within the Jira software program device:

Create a Jira person account.
Create a site name, upload a assignment, and, document a few troubles, or, bugs. Our mission is to fetch, troubles records, the use of Python code.
We want to have, a valid token, for authentication, which can be acquired, from link https://identity.Atlassian.Com/manage/api-tokens.

Module required in python:

requests

requests module utilization:

The requests is inbuild Python package for making HTTP requests to URL. Whether or not it's REST APIs or Web Scraping, requests are a should to be discovered for proceeding similarly with these technology. When one makes a request to URI, it returns a response. This package provide functionalities for coping with both the request and reaction

Establishing connection with JIRA the usage of requests module in python script:

prerequisite

Get api token from JIRA for authentication cause at the same time as making request. Procedure to create the API token
- Click Profile at the right-hand aspect pinnacle nook, pick Personal Authentication Token option.
- Click Create token button.
- On the API Token Creation Result pop-up, word down the token for destiny use and click on Close.

Making reference to JIRA

Import requests module
Start new consultation via initializing session class in requests module Session class
- A Session object is created the use of requests.Session().
- Using a consultation is in particular beneficial when making multiple requests to the equal server, because it allows you to persist certain parameters (inclusive of cookies) across those requests.
Add your API token in session headers
Session headers importance
- User Agent Information: Many internet servers use the User-Agent header to perceive the purchaser (browser or utility) making the request. Some servers might also respond differently based totally at the consumer agent, and imparting a user agent facilitates mimic a specific browser or customer.
- Content Negotiation: The Accept header within the request may be used to suggest the preferred format of the reaction (e.G., JSON, XML, HTML). Servers may additionally respond with special content sorts based on the Accept header.
- Authentication: If the server requires authentication, you may need to include an Authorization header with the appropriate credentials.
- Cookies: Session headers often consist of cookies, allowing you to persist the session throughout more than one requests. The requests library robotically handles cookies when the usage of classes.
- Custom Headers: You may want to encompass custom headers primarily based at the requirements of the API or server you're interacting with.
session.headers.update({'Authorization': f'Bearer {api_token}'})
Once our session is up to date with authentication token, we will make URL requests to acquire data from JIRA

response = session.get('https://jira.Devops.Jlr-apps.Com/relaxation/api/2/search',params= {'jql': "task=project_name"},)
Check response object to confirm whether we are getting right output or not.
print(response.status_code)
Some of crucial HTTP reaction code
- 2 hundred OK: Standard response for successful HTTP requests.
- 400 Bad Request: The server can't or will no longer process the request because of
- 401 Unauthorized: Similar to 403 Forbidden, but mainly for authentication.
- 500 Internal Server Error: A frequent error message indicating that the server has encountered an sudden situation.
- 503 Service Unavailable: The server is not geared up to handle the request.
- 429 Too Many Requests: This fame code indicates that the consumer has despatched too many requests in a given amount of time

Reading records from response

At every request, consumer can get maximum one thousand issue info from JIRA server. So if assignment have extra than 1000 troubles, person wishes to make request to server with extra parameter.

**params = {
'jql': "task=project_name",
'startAt':starting_position, #assign fee according to new starting point after reading a thousand troubles
'maxResults': -1,}

    response = session.get('https://jira.Devops.Jlr-apps.Com/relaxation/api/2/search', params=params)**

User can access the issue from response with the aid of calling json method data = response.json()
User can acquire data of issue via data['issues']
Iterate issue listing and gather data based JIRA on attribute names example

Conclusion:
In this Automation I have used requests module, but another third party module called jira is available in pip. But jira package has some limitation, I will publish another article about jira module and it's limitation compared to requests module.