DEV Community: Jonathan Pereira

Testing the new Twitter API v2

Jonathan Pereira — Fri, 21 Aug 2020 16:13:20 +0000

For developers building apps with social interactions, the Twitter API has been one of the most used API. While it was beneficial, it had its limitations. But not for long. Twitter just released the new version of the Twitter API, which has so much to offer.

What's new in Twitter API v2

Twitter seems to have finally answered the prayers of millions of developers using the Twitter API. The new API makes it simpler to listen and analyze the active conversations on the social platform. Twitter has introduced a couple of exciting things, including new endpoints, new tweet payload, conversation identifiers, and even annotations.

Having heard so much about it, we decided to take the Twitter API v2 for a spin. Luckily for us, Fiddler Everywhere can quickly test and debug APIs. SO we decided to test the Twitter API v2 on Fiddler Everywhere.

Fiddler Everywhere to the rescue

For those who haven't used Fiddler Everywhere before, it can capture and inspect network requests, including API calls. It also allows you to modify the request parameters to customize the response as required.

Testing the Twitter API v2

In this blog, we are going to test some of the endpoints of the new Twitter API v2 and see how it works.

Before getting started, make sure that you create a new app using the Twitter Developer Portal and generate the Bearer Token to authenticate the request in Fiddler Everywhere.

Users lookup

The Twitter API v2 makes it simple to get publicly available user information using the users endpoint.

Endpoint URL: https://api.twitter.com/2/users

You can do the following using this endpoint:

/users – Query multiple IDs to retrieve user information
/users/:id – Query the user ID to fetch information about a specific user
/users/by – Query the username to fetch user information

For this example, we'll use the /users/by endpoint.

We open the Composer in Fiddler Everywhere and enter the endpoint in the request bar and set the method to GET.

We need to provide authentication and query parameters. Under the Headers tab, add a new key-value pair called authorization and the Bearer Token as the value.

In the Params tab, add a key-value pair called usernames. You can enter the username or multiple usernames separated by commas in the value field.

Click the Execute button to see the API in action.

If everything were excellent, in the Response section below, you would see the entire response in the JSON tab.

You can modify the response as per your requirement by specifying the user.fields parameters.

I'm going to request the following fields:

Parameter	Description
`name`	Name of the user
`username`	Username on Twitter
`url`	Website URL mentioned in the profile
`profile_image_url`	URL of the user's profile image
`pinned_tweet_id`	Tweet ID of users pinned tweet, if any
`verified`	Boolean response indicating whether or not Twitter has verified the user
`created_at`	Date and time of account creation
`public_metrics`	number of tweets, followers, following, and lists of the user
`location`	Place mentioned in the user account

Once you hit execute, you'll notice the detailed information requested in the JSON tab of the response section.

Tweets lookup

Using the tweets endpoint, we can fetch tweets and information about them using the tweet ID.

Endpoint URL: https://api.twitter.com/2/tweets

We'll open a new request tab in Fiddler Everywhere, enter this API endpoint in the request bar, and set the GET method.

Again, we need to provide authentication and query parameters. You can use the same Bearer Token used in the previous request.

In the Params tab, add a key-value pair called ids. You can enter a tweet ID or multiple tweet IDs separated by commas in the value field.

Click the Execute button to make a call to the API.

You can view the tweets fetched in the JSON tab of the Response section.

We can add additional query parameters to get the required information. I'm going to add the following parameters:
expansions

Parameter	Description
`attachments.media_keys`	Identifies the media attached to the tweet
`author_id`	Returns the user's unique identifier
`entities.mentions.username`	Recognizes the Twitter users mentioned in the tweet
`in_reply_to_user_id`	Provides the user ID of parent tweets author if the tweet is a reply
`referenced_tweets.id`	Identifies the referenced tweet
`referenced_tweets.type`	Identifies if the tweet is a retweet or a reply

text.fields

Parameter	Description
`attachments`	Indicates the attachment types, if any
`conversation_id`	Returns the ID of the original tweet along which includes the thread of all replies
`created_at`	Time of the tweet
`geo`	Location from where the tweet originated, if available
`lang`	Returns the language tag of the language detected
`possibly_sensitive`	Returns a Boolean response indicating if the tweet was marked sensitive
`public_metrics`	Provides the engagement metrics like retweet count and like count
`Source`	Provides the app used for publishing the tweet

If you execute the request again, you will see the requested fields.

Recent Tweets search

The recent endpoint searches and fetches tweets that match the query.

Endpoint URL: https://api.twitter.com/2/tweets/search/recent

Again, We'll open a new request tab in Fiddler Everywhere and enter this API endpoint in the request bar and set the method to GET.

We need to provide authentication and query parameters. You can use the same Bearer Token used in the previous request.

In the Params tab, add a key-value pair called query. You can enter the query in the value field.

Click the Execute button to fetch the response.

You can view the tweets fetched in the JSON tab of the Response section. By default, the endpoint returns ten results from the last seven days. You can use the max_results parameter to indicate the required number of results.

You can use the same query parameters from above to modify the request as required.

This is the response that I received.

These were some of the basic operations that you can perform using the new Twitter API v2. Next time, we'll dive into how Twitter has improved the API using NLP to understand and analyze tweets and how you can implement it using the API.

Fiddler Everywhere makes debugging fun

If you are wondering what else you could do with Fiddler Everywhere, the tool comes in handy to inspect, debug, mock, and share network requests and responses. You can download and try it out yourself. Fiddler Everywhere is available on macOS, Linux, and Windows systems and supports all modern browsers and processes.

How Fiddler Everywhere makes web debugging simpler

Jonathan Pereira — Tue, 18 Aug 2020 10:16:31 +0000

As developers – whether new or experienced, we are always afraid of bugs. These could appear in the form of unexpected results or errors in our websites and apps. Problems like these are where tools like Fiddler Everywhere have been saving time and effort. In this post, we’re going to explore some situations where Fiddler Everywhere makes debugging simpler for developers.

Fiddler Everywhere is the next generation cross-platform network debugging proxy tool, which just came out of beta recently. The tool is trusted by developers for over a decade and is used to inspect and debug HTTP(S) traffic requests between your computer and the internet. If you haven’t tried it yet, download it now.

Identify errors

Fiddler Everywhere inspects every request made to the server as well as the response received. As the requests come in, they are logged by Fiddler Everywhere. The request log provides you a perspective of the activity going on in the background when the browser calls your website.

If at any point, you receive an unexpected response or an error, Fiddler Everywhere can promptly point out the cause. The issue could be something as simple as a wrongly entered file name in the code. While it could be a daunting task to pinpoint the file causing the error, Fiddler Everywhere would throw a 4XX error, which then can be easily debugged.

For example, a 404 error would indicate that the file name does not match the available files on the server, or the file doesn’t exist. Once identified, it is much simpler to rectify the error.

Mock user conditions

Fiddler Everywhere allows you to simulate situations faced by your user. For example, your user might be working on a slower internet connection or entirely offline. Fiddler Everywhere enables you to create rules that add a delay while loading the website to replicate the network latency. The same can applies to check drop requests.

Fiddler Everywhere also tracks the response size and time taken to be transmitted. This data can be of use to highlight files that are heavy and take longer to load.

Additionally, using the Fiddler Everywhere Auto Responder you can create mocking rules for your API. This ability is especially useful when you want to test your app without the live APIs or are behind a firewall.

Test APIs

The Fiddler Everywhere Composer can be used to quickly test API requests and make sure that the server responds as expected. For example, if the API fails to return a response or doesn’t provide the expected response, you can open the request in the composer, modify the headers, and execute it to test the request.

If everything is working fine, you should get a 2XX status, else a 4XX error if there is an issue with the HTTP request or a 5XX error if there is an issue with the server. You can identify several common problems like unauthenticated requests, invalid requests, internal server errors, and service overload.

Alternatively, you can use the Fiddler Everywhere Composer to test new APIs or the API user interface.

Saving sessions

Fiddler Everywhere takes the pain out of repeatedly working with web requests. Fiddler Everywhere has intuitive features that help you streamline the debugging process. The filters across every category allow us to inspect limited traffic from a particular domain, a selected browser, or even a specific file type and status code. You can add multiple filters as required.

Additionally, you can select individual sessions or multiple sessions. You can then save them and open them in a separate tab. Fiddler Everywhere also allows you to put notes and comments on your saved sessions. This feature is useful when you want to share the requests with your teammates, who can then continue from where you left.

Try out Fiddler Everywhere

These are some of the ways that Fiddler Everywhere is known to simplify the web debugging process. Let us know the ones that you use in the comments below.

If you are new to Fiddler Everywhere, check out the video below to help you get started quickly.

Happy debugging!

Getting Started with Fiddler Everywhere

Jonathan Pereira — Thu, 13 Aug 2020 16:02:32 +0000

What is Fiddler Everywhere?

Fiddler Everywhere is a valuable tool for web debugging and monitoring. It logs all the HTTP(S) traffic between the client and the internet. The tool is handy to inspect, debug, mock, and share network requests and responses.

Along with the primary use of acting as a web debugging proxy, Fiddler Everywhere has capabilities that make it an excellent tool for API development, performance testing, security testing, and even malware analysis.

Fiddler Everywhere just got out of beta and is available on Windows, macOS, and Linux and supports every browser. Additionally, it can also be made to behave as a reverse proxy, thereby allowing you to inspect traffic between mobile devices and the internet.

Get Fiddler Everywhere

Head over to the Fiddler Everywhere site to download it.

Fiddler Everywhere is available for Windows, macOS, and Linux. Choose the required platform and download and install the setup file.

Getting Started

Once you open Fiddler Everywhere, you need to create an account. Enter the required details and click on Create New Account. Alternatively, you can sign-in using your Google account.

The first thing you want to do, once you install Fiddler Everywhere is to check and enable the HTTPS traffic capture. Doing this allows you to inspect and debug web traffic, which is transmitted using the HTTPS protocol, which is quickly becoming the default standard for web development.

To do this, click on the gear icon on the top right corner. The Setting pop-up will show. In the HTTP(S) menu, click on the Trust root certificate button. Your system will request your system login credentials. After you enter and authenticate yourself, Fiddler Everywhere will automatically complete the process. Make sure to click on the checkbox next to Capture HTTPS traffic.

If you face issues with the root certificate, you can try importing and trusting it manually.

Navigating Fiddler Everywhere

You are now inside Fiddler Everywhere. You can see the default view divided into two major sections – the column on the left displays the web sessions and the column on the right have the Traffic Inspector. The sidebar displays the saved sessions and requests.

Before you start recording the traffic, you need to make sure that you turn on the toggle in the Live Traffic tab. The tab will display (Capturing) if it is working fine. You can click the toggle again to pause the recording. The tab will display (Paused) when it is not capturing.

Live Traffic

Now you can open any browser and enter the URL you want to inspect. As soon as you hit enter in the browser, you will see the web sessions getting displayed in Fiddler Everywhere.

The tool provides many insights regarding each session at a glance. For example, you can see the filetype denoted next to the request number. You can see the status code of the request and the protocol used. The host server and the URL slug of the request are also displayed. You can also see information about the request method, response size, content type, and caching. Interestingly, the process from where the request originated is also displayed – for example, browser, desktop app, etc.

To make things simpler, you can click on the ellipsis in the menu bar and set filters for each tab and view only the required sessions.

You can right-click any session and save, remove, comment, and mark the session. Additionally, you can also share the session with others by just entering the email address.

Traffic Inspectors

If you want to scrutinize an individual request, you can select the request by clicking on it and see the details in the Inspectors tab on the right. The Inspectors tab displays two panes – the top pane shows the information on the request, and the bottom pane highlights the response.

In the request pane, you can see all the headers which contain the information sent to the servers like the host name, user-agent, encoding status, and even the cookies. You can view the form data and cookie data, or you can check the text, JSON, and XML used in the request, if any.

In the response pane, you can view the response headers received from the server, including the status code, access controls, and even the content length. The body of the response is available as text, JSON, or XML. If the response is an image, it is available in the image tab. Interestingly, the entire response is available as a live web page in the web tab.

Auto Responder

The Auto Responder feature is one of the most loved features of Fiddler Everywhere. You can find this feature in the tab next to the Inspectors. The feature allows the developer to mock responses, without actually transmitting the request to a live server.

You can create a new rule with a match condition and an action to be performed. The term can be a host name, or the URL, or even the file type. If you want to target an URL specifically, you can also set the condition to require an exact match.

The corresponding action which needs to be executed, in the event of a match, has to be provided. This action could be another URL or a filename or a redirect request. This feature can also help set breakpoints, delay (to imitate network latency), rest the connection, and even close the connection altogether.

Composer

The composer feature — next to the “Web Sessions” tab — allows you to modify and send requests to the server. Additionally, you can create your requests and execute it with Fiddler Everywhere.

To modify an existing request, you can right-click on the request in the “Web Sessions” tab and select the Composer option. The request will automatically get loaded in the Composer tab. Similarly, you can create a new request in the Composer tab directly.

In the Composer tab, you need to select the appropriate HTTP method, enter the request URL, the headers, and the request body. You can click on Execute to send the request to the server or click on Save to save it. Fiddler Everywhere supports all the available HTTP request methods.

Start debugging

Now that you know about Fiddler Everywhere, it is time to put the tool to the test and see how it mystically makes debugging simpler. Head over to the Fiddler Everywhere documentation to get more information about the features. If, at any point, you feel stuck or are faced with an issue, feel free to reach out to the live Fiddler Everywhere community.

How to create a sentiment analysis bot using RPA and Cognitive APIs

Jonathan Pereira — Tue, 04 Feb 2020 18:17:23 +0000

Introduction

In this tutorial we will create an RPA bot that analyses the sentiment in text. We will use Automation Anywhere A2019 Community Edition and Azure Cognitive services to build this bot.

Getting Started with Automation Anywhere A2019 Community Edition

Automation Anywhere offers developers a free version of the cloud native A2019 platform. Click the Sign up link to create your account.

Fill in all the fields and hit the Get Free Community Edition button at the bottom of the form.

You will receive an email with the Control Room URL and your login details.

Enter the URL in your browser and enter the login details provided. You will now see the A2019 Control Room.

Creating your bot

To create your bot, click the Create a bot button on the Home page in the Control Room.

Enter the Name and Description for the bot you plan to create. Click on Create & edit to go to the bot creation interface.

Here you will see the bot flow. You can view this in the visual Flow mode, the List mode and, the Dual mode which has the flow and list of the actions.

To the left pane you will notice all the actions (packages) available by default. You can either double click or drag and drop the actions as and when required.

To start with let us enable our bot to POST requests using the API. For this we use the REST Web Services actions. You can either scroll down in the action pane or search for them. We will use the POST method. Double click on the action to add it to the flow.

Now we need to connect to the Azure APIs. Head over to the Azure Portal to get the API endpoint and the key. We will use the Text Analytics API for this tutorial.

Fill in the required fields and hit the “Create” button.

The API endpoint and the Key would be provided here.

In A2019, enter the API endpoint in the “URI” field.

Add the following custom headers:

Ocp-Apim-Subscription-Key:your_API_key
Content-Type:application/json

In the content type, we need to add custom parameters with the message that we want to parse.
{ "documents": [ { "language": "en", "id": "1", "text": "Your company is amazing. Wonderful service. " } ] }
We can change the text to see the different results. Alternatively, you can import the various text phrases into the bot using a CSV or an excel.
Click Apply to update the action.

Now we need to assign a variable for the response. For this, click the wand icon next to the field.

Enter the name of the variable and click on Create & select.

Now we define the Response variable using the Dictionary package to allow it to return the value associated.
Click Apply to update the action.

Next, we write a Python script to rate the emotion. Alternatively, you can also import the python file within the action.

Python script:

import json
def read_from_json(response):
    pyObj = json.loads(response)
    result =''
    try:
        arr = pyObj['documents']
        for v in arr:
            score = round(v['score'],2)
            if score >= 0 and score <= 0.25:
                return "Sad/Angry/Frustrating"
            elif score > 0.25 and score <= 0.5:
                return "Below Average"
            elif score > 0.5 and score <= 0.75:
                return "Average"
            else:
                return "Happy"
    except:
        return result

Click Apply to update the action.

Now we add an action to execute the python script. The function name is read_from_json.

Click Apply to update the action.

Now we are all done with adding the actions. Now we need to save the bot by clicking on the Save button.

It is now time to test our bot. Click the “Run” button. If you do not already have a bot agent installed, it will ask you to do so.

After executing successfully, the bot response is displayed based on the input text. We can change the text to see the difference.

That’s it, we just created a bot on A2019 using the Azure Cognitive Services APIs.

How to intercept API calls from mobile devices

Jonathan Pereira — Wed, 10 Jul 2019 09:11:20 +0000

Ever wondered how to capture the API calls being made in the background while your mobile app or mobile browser is fetching data from the internet? This quick guide will give you some answers.

I'll be using a proxy server to inspect the web traffic originating from the mobile device. I'm using the Postman client and my Android smartphone for this purpose.

What is a Proxy Server?

Simply put, a proxy server is a gateway between the client and the server. In our case, we will use the mobile device as a client, and make the computer act as a proxy between the client and the server. The flow is setup in a way that the internet traffic passes through the proxy server while attempting to reach the web address you requested.

What is Postman?

Postman is a API Development Environment catering to every stage of the API Lifecycle. It is used by over 7M developers globally. The Postman Proxy is one of the feature available out-of-the-box on the Postman client.

Capturing API calls using Postman

To get started, we need the computer and the mobile device to be connected to the same wireless network. I’m going to use a Mac laptop and an Android smartphone.
Click on the Proxy Settings button in the top menu bar.
The Proxy Settings modal should show. Make a note of the port mentioned. I’ll use the default post “5555”. Select the target to capture the requests. I’ll use the “History” sidebar panel. Click on Connect.
Find the IP address of your computer. You should see this in your Network settings.
Configure HTTP Proxy on the mobile device. You can do this by going to Settings -> Wi-Fi and selecting the Modify Network option. Enter the computer’s IP address as your Proxy Host. Set the proxy port to the same as the port in the Postman app, i.e. 5555.

You should have everything working!

Open a browser on the mobile device or any app and keep an eye on the Postman app. I entered www.mit.edu in my Chrome for Android browser. As the browser started contacting the server, all of the calls start appearing in the Postman sidebar.

Where can you use this?

To help you to understand the communications between the client and server. This could be useful from a security standpoint to highlight malicious requests.
To makes it easier to debug by pinpointing where the app crashes or throws an error.
To identify behavior differences among mobile devices (based on hardware and resolutions), which help to isolate bugs seen on certain devices.

Note:

Postman currently supports only HTTP traffic in the Proxy feature. Support for HTTPS is expected to be added soon.