Twitter API Guide: Search Tweets, Scrape Data, and Automate

Burak Bayır — Thu, 20 Aug 2026 18:00:46 +0000

While building Xquik, I learned that a Twitter API integration must do more than search tweets.

It must paginate Twitter search results, run durable Twitter scraper jobs, and make Twitter automation safe to retry.

A demo needs one successful request. Production needs confidence in every result.

Search needs completion signals. Extraction needs durable state. Automatic Twitter posting needs safe retry rules.

This post covers the three contracts I now consider essential:

A cursor and coverage contract for Twitter search.
A job contract for Twitter scraping and data extraction.
An event and action contract for Twitter automation.

The examples use patterns implemented in Xquik. The same principles apply to other APIs and internal systems.

1. Twitter Search Needs a Completion Contract

Most Twitter search examples fetch one page and print the response.

That proves connectivity. It does not prove completeness.

A useful Twitter search API needs four things:

Stable string identifiers
Opaque pagination cursors
An explicit has-more signal
Coverage information for complex collections

The client should never calculate or modify a cursor. Pass it back exactly as received.

The client should also avoid treating an empty page as completion. Filtering can produce an empty page while another cursor still exists.

Here is a complete pagination loop using the Xquik response contract:

async function collectTweets(query) {
  const tweets = [];
  let cursor;

  do {
    const params = new URLSearchParams({
      q: query,
      queryType: "Latest",
      language: "en",
      minFaves: "5",
      replies: "include",
      retweets: "exclude",
      limit: "100",
    });

    if (cursor) {
      params.set("cursor", cursor);
    }

    const response = await fetch(
      process.env.XQUIK_API_BASE + "/x/tweets/search?" + params,
      {
        headers: {
          "x-api-key": process.env.XQUIK_API_KEY,
          "xquik-api-contract": "2026-04-29",
        },
      },
    );

    if (!response.ok) {
      throw new Error(JSON.stringify(await response.json()));
    }

    const page = await response.json();

    tweets.push(...page.tweets);
    cursor = page.has_more ? page.next_cursor : undefined;
  } while (cursor);

  return tweets;
}

The contract header enables normalized response fields. List responses use has_more and next_cursor.

Store X identifiers as strings. JavaScript cannot safely represent every large numeric identifier.

The query can contain normal X search syntax. Structured filters can cover authors, dates, languages, media, and engagement.

That makes advanced Twitter search easier to generate from application inputs. It also avoids fragile string concatenation.

Pagination Does Not Solve Coverage

A cursor can tell you another page exists. It cannot prove that one timeline exposed every relevant item.

Reply trees illustrate the problem.

A conversation can contain direct replies, nested branches, and several rankings. One timeline may omit part of that structure.

A complete-replies operation should therefore report more than rows.

Useful coverage fields include:

The source-reported reply count
The number of collected replies
The strategies attempted
Cursor or branch failures
Whether the coverage threshold passed

Xquik returns a failure when complete-reply coverage remains too low. It does not label that result complete.

That behavior is intentional.

Silent partial data is worse than an explicit error. It enters databases, reports, and models as if it were complete.

2. A Twitter Scraper API Needs a Job Contract

A tweet scraper usually returns pages. A reliable extraction workflow needs durable job state.

Large Twitter data collections need targets, limits, progress, retries, storage, and exports.

They also need an estimate before execution.

I use this lifecycle for extraction work:

Estimate the requested operation.
Create a bounded job.
Poll until the job reaches a terminal state.
Read or export the resulting rows.

A 202 response means the server accepted the job. It does not mean the dataset is ready.

The job should expose enough state for a worker to resume safely:

queued -> running -> completed
                 -> failed
                 -> cancelled

The caller should persist the job identifier. It should not create a replacement after every timeout.

Xquik uses this model for posts, replies, quotes, reposts, media, followers, following relationships, lists, communities, spaces, threads, and articles.

Completed jobs can produce CSV, XLSX, JSON, Markdown, PDF, or text.

The format is less important than the boundary.

A Twitter scraper API should define when collection starts, ends, fails, and becomes exportable. Otherwise, every client must rebuild that state machine.

Bound Every Collection

Unbounded scraping jobs are difficult to price, retry, and reason about.

Require a result limit. Validate targets before starting. Return a clear estimate when possible.

Those rules protect both the client and the service.

They also make Twitter API cost comparisons more useful. Compare the cost of a completed dataset, not one successful request.

A cheap request that produces incomplete data can become expensive engineering work.

3. Twitter Automation Needs Event and Action Contracts

Polling works for prototypes. It becomes wasteful once missed runs matter.

Every polling loop needs scheduling, watermarks, deduplication, and failure recovery.

Event delivery moves that work into a reusable contract.

Xquik uses account monitors and keyword monitors. Matching activity becomes a stored event.

Applications can read events through REST or receive a signed Twitter webhook delivery.

For brand monitoring, I prefer two stages:

Run Twitter search for the initial backfill.
Use a keyword monitor for new matches.

The webhook receiver should authenticate the untouched request body before parsing it.

import { createHmac, timingSafeEqual } from "node:crypto";

function verifyWebhook(rawBody, headers, secret) {
  const timestamp = headers.get("X-Xquik-Timestamp");
  const nonce = headers.get("X-Xquik-Nonce");
  const signature = headers.get("X-Xquik-Signature");

  if (!timestamp || !nonce || !signature) {
    return false;
  }

  const age = Math.abs(Date.now() - Number(timestamp));

  if (!Number.isFinite(age) || age > 5 * 60 * 1000) {
    return false;
  }

  const signed = timestamp + "." + nonce + "." + rawBody;
  const digest = createHmac("sha256", secret)
    .update(signed)
    .digest("hex");

  const expected = Buffer.from("sha256=" + digest);
  const received = Buffer.from(signature);

  return (
    expected.length === received.length &&
    timingSafeEqual(expected, received)
  );
}

Signature verification is only the first step.

The receiver should also:

Reject recently used nonces
Store each delivery identifier
Return quickly after durable storage
Run classification and model calls in a queue worker
Make downstream processing idempotent

These rules turn a webhook into a reliable event input.

Automatic Twitter Posting Needs Idempotency

Automatic Twitter posting has a different failure mode.

Suppose a request times out after X accepts the post. A blind retry can publish it twice.

The same problem affects replies, follows, likes, reposts, messages, and profile changes.

Every write should use an idempotency key.

Generate one key for one intended action. Reuse it only when retrying the identical input.

An identical retry should return the original action. Different input with the same key should fail.

Some writes finish immediately. Others need asynchronous confirmation.

The client should poll accepted actions until they become terminal. It should not create another write while the first remains unresolved.

A useful error contract also separates two ideas:

Retryable means another attempt may succeed.
Safe to retry means nothing was dispatched.

Those are not interchangeable.

An ambiguous write may require checking account state. A timeout alone cannot prove whether an action happened.

Twitter automation becomes dangerous when write requests forget their history.

Twitter API Keys and Developer Access

Many teams start by searching for a Twitter API key or Twitter developer account.

Supported Xquik read operations use an Xquik API key. They do not require a Twitter developer account.

Account-only reads and write actions still require a connected X account.

Keep every API key on the server. Never expose credentials in browser code or committed files.

Xquik also supports OAuth 2.1 with authorization code flow and PKCE. Use OAuth when an application needs delegated access.

Authentication answers who may call an endpoint. It does not replace pagination, coverage, or retry contracts.

Twitter MCP Still Needs the Same Contracts

MCP changes how an agent discovers and invokes tools. It does not remove API reliability requirements.

A Twitter MCP server still needs stable schemas, bounded operations, structured errors, and safe write controls.

The Xquik MCP integration separates endpoint discovery from authenticated execution.

Agents can search tweets, start extraction jobs, inspect monitors, and prepare actions. Human review still matters for writes.

Finding a write tool does not mean an agent should execute it.

Evaluate a Twitter API With Failure Tests

Feature tables rarely explain failure behavior.

Use one realistic workload when comparing an official API, a Twitter scraper, or an X API alternative.

Test these cases:

Follow every search cursor.
Store identifiers as strings.
Record duplicates and missing fields.
Interrupt a job and resume it.
Replay a webhook delivery.
Retry an accepted write.
Send invalid input.
Trigger a rate limit.
Inspect the resulting errors.

Twitter API limits matter, but result caps are only one limit. Test Twitter API rate limits and record Retry-After behavior.

Cursor behavior, coverage, retry rules, and incomplete operations can matter more.

Measure engineering time alongside Twitter API pricing. Include maintenance, recovery, exports, and missing-data investigation.

Where Xquik Fits

Xquik is where I implemented these contracts.

It combines Twitter search, extraction jobs, monitors, webhooks, connected-account actions, SDKs, and Twitter MCP access.

It does not replace every official X API endpoint. It is not an advertising API.

It also cannot return deleted, protected, restricted, or otherwise unavailable content.

If one official endpoint already meets the requirement, use it. Extra infrastructure should solve a real problem.

Twitter API FAQ

Can Xquik Search Tweets?

Yes. Its Twitter search API supports X query syntax, structured filters, ordering, and cursor pagination.

Use advanced Twitter search operators for compact queries. Use structured parameters for generated application inputs.

Is Xquik a Twitter Scraper API?

Xquik supports direct reads and tracked extraction jobs.

Use direct reads for interactive requests. Use extraction jobs for bounded datasets, progress tracking, and exports.

Do I Need a Twitter API Key?

Supported read operations use an Xquik API key. They do not require a Twitter developer account.

Connected-account reads and write actions still require an authenticated X account.

How Should I Compare Twitter API Pricing?

Compare the cost of a completed workload.

Include engineering time, retries, missing-data checks, exports, and recovery work. Do not compare one successful request alone.

Is There a Free Twitter API?

Free access and plan limits can change. Verify the current terms before designing a workload around them.

Xquik has no free API tier. Eligible read operations can use prepaid credits without a subscription.

Compare price against successful, complete results rather than raw request counts.

Does Twitter MCP Support Automation?

The Xquik Twitter MCP integration supports endpoint discovery and authenticated execution.

Agents can search tweets, inspect jobs, and prepare actions. Human review should remain part of write workflows.

Three Takeaways

A reliable integration needs more than endpoint access.

Search needs cursor and coverage contracts.
Extraction needs durable, bounded job state.
Automation needs signed events and idempotent actions.

Those contracts make failures visible. They also make recovery testable.

That matters more than the number of endpoints on a pricing page.

Xquik is an independent third-party service. Not affiliated with X Corp. "Twitter" and "X" are trademarks of X Corp.

Disclosure: I used an AI writing assistant to help organize and edit this post. I reviewed the technical claims and code against the implementation before saving this draft.

DEV Community: Burak Bayır