Understanding Pagination with NewsDataHub API

This guide explains how to paginate through the results when using NewsDataHub API.

NewsDataHub API is a service that provides news data through a RESTful API interface. It implements cursor-based pagination to efficiently handle large datasets⁠, allowing developers to retrieve news articles in manageable batches. Each response includes a set of articles where each article object contains details like title, description, publication date, source, content, keywords, topics, and sentiment analysis⁠. The API uses a cursor parameter for seamless navigation through results⁠, and provides comprehensive documentation for advanced features like search parameters and filtering options⁠.

For documentation visit: https://newsdatahub.com/docs

APIs typically return a limited amount of data in their responses since returning all results in a single request is usually impractical. Instead, they use pagination — a technique that splits data into separate pages or batches. This allows clients to retrieve one page at a time, accessing a manageable subset of the results.

When you make an initial request to the /news endpoint and receive the first batch of results, the shape of the response looks like this:

{
    "next_cursor": "VW93MzoqpzM0MzgzMQpqwDAwMDQ5LjA6MzA0NTM0Mjk1T0xHag==",
        "total_results": 910310,
        "per_page": 10,
        "data": [
            {
                "id": "4927167e-93f3-45d2-9c53-f1b8cdf2888f",
                "title": "Jail time for wage theft: New laws start January",
                "source_title": "Dynamic Business",
                "source_link": "https://dynamicbusiness.com",
                "article_link": "https://dynamicbusiness.com/topics/news/jail-time-for-wage-theft-new-laws-start-january.html",
                "keywords": [
                    "wage theft",
                    "criminalisation of wage theft",
                    "Australian businesses",
                    "payroll errors",
                    "underpayment laws"
                ],
                "topics": [
                    "law",
                    "employment",
                    "economy"
                ],
                "description": "Starting January 2025, deliberate wage theft will come with serious consequences for employers in Australia.",
                "pub_date": "2024-12-17T07:15:00",
                "creator": null,
                "content": "The criminalisation of wage theft from January 2025 will be a wake-up call for all Australian businesses. While deliberate underpayment has rightly drawn scrutiny, our research reveals that accidental payroll errors are alarmingly common, affecting nearly 60% of companies in the past two years. Matt Loop, VP and Head of Asia at Rippling Starting January 1, 2025, Australias workplace compliance landscape will change dramatically. Employers who deliberately underpay employees could face fines as high as AU$8. 25 million or up to 10 years in prison under new amendments to the Fair Work Act 2009 likely. Employers must act decisively to ensure compliance, as ignorance or unintentional errors wont shield them from civil or criminal consequences. Matt Loop, VP and Head of Asia at Rippling, says: The criminalisation of wage theft from January 2025 will be a wake-up call for all Australian businesses. While deliberate underpayment has rightly drawn scrutiny, our research reveals that accidental payroll errors are alarmingly common, affecting nearly 60% of companies in the past two years. Adding to the challenge, many SMEs still rely on fragmented, siloed systems to manage payroll. This not only complicates operations but significantly increases the risk of errors heightening the potential for non-compliance under the new laws. The urgency for businesses to modernise their approach cannot be overstated. Technology offers a practical solution, helping to streamline and automate processes, reduce human error, and ensure compliance. But this is about more than just avoiding penalties. Accurate and timely pay builds trust with employees, strengthens workplace morale, and fosters accountability. The message is clear: wage theft isnt just a financial risk anymoreits a criminal offense. Now is the time to ensure your business complies with Australias new workplace laws. Keep up to date with our stories on LinkedIn, Twitter, Facebook and Instagram.",
                "media_url": "https://backend.dynamicbusiness.com/wp-content/uploads/2024/12/db-3-4.jpg",
                "media_type": "image/jpeg",
                "media_description": null,
                "media_credit": null,
                "media_thumbnail": null,
                "language": "en",
                "sentiment": {
                    "pos": 0.083,
                    "neg": 0.12,
                    "neu": 0.796
                }
            },
        // more article objects
      ]
  }

Notice the first property in the JSON response - next_cursor. The value in next_cursor points to the start of the next page of results. When making the next request, you specify the cursor query parameter like this:

https://api.newsdatahub.com/v1/news?cursor=VW93MzoqpzM0MzgzMQpqwDAwMDQ5LjA6MzA0NTM0Mjk1T0xHag==

The easiest way to try out paginating through the results is via Postman, or a similar tool. Here is a short video demonstrating how to use cursor value to paginate through the results in Postman.

https://youtu.be/G7kkTwCPtCE

When the next_cursor value is null, it indicates that you have reached the end of the available results for your selected criteria.

Paginating through results with Python

Here is how to set up basic pagination through NewsDataHub API results using Python.

import requests

# Make sure to keep your API keys secure
# Use environment variables instead of hardcoding
API_KEY = 'your_api_key'
BASE_URL = 'https://api.newsdatahub.com/v1/news'

headers = {
    'X-Api-Key': API_KEY,
    'Accept': 'application/json',
    'User-Agent': 'Mozilla/5.0 Chrome/83.0.4103.97 Safari/537.36'
}

params = {}
cursor = None

# Limit to 5 pages to avoid rate limiting while demonstrating pagination

for _ in range(5):
    params['cursor'] = cursor

    try:
        response = requests.get(BASE_URL, headers=headers, params=params)
        response.raise_for_status()
        data = response.json()
    except (requests.HTTPError, ValueError) as e:
        print(f"There was an error when making the request: {e}")
        continue

    cursor = data.get('next_cursor')

    for article in data.get('data', []):
        print(article['title'])

    if cursor is None:
        print("No more results")
        break

Index based Pagination

Some APIs use index-based pagination to split results into discrete chunks. With this approach, APIs return a specific page of data—similar to a table of contents in a book, where each page number points to a specific section.

While index-based pagination is simpler to implement, it has several drawbacks. It struggles with real-time updates, can produce inconsistent results, and puts more strain on the database since retrieving each new page requires sequentially scanning through previous records.

We've covered the fundamentals of cursor-based pagination in the NewsDataHub API. For advanced features like search parameters and filtering options, please consult the complete API documentation at https://newsdatahub.com/docs.