<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Alessia Defender</title>
    <description>The latest articles on DEV Community by Alessia Defender (@alessia_defender_bb8a965d).</description>
    <link>https://dev.to/alessia_defender_bb8a965d</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4255405%2F74a8d8b1-a7cd-4c48-8b49-f6dea046df78.jpg</url>
      <title>DEV Community: Alessia Defender</title>
      <link>https://dev.to/alessia_defender_bb8a965d</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/alessia_defender_bb8a965d"/>
    <language>en</language>
    <item>
      <title>I Built an Instagram Competitor Monitoring Workflow After Scraping Rate Limits Became Unmanageable</title>
      <dc:creator>Alessia Defender</dc:creator>
      <pubDate>Fri, 20 Nov 2026 09:58:02 +0000</pubDate>
      <link>https://dev.to/alessia_defender_bb8a965d/i-built-an-instagram-competitor-monitoring-workflow-after-scraping-rate-limits-became-unmanageable-3oi1</link>
      <guid>https://dev.to/alessia_defender_bb8a965d/i-built-an-instagram-competitor-monitoring-workflow-after-scraping-rate-limits-became-unmanageable-3oi1</guid>
      <description>&lt;p&gt;My previous Instagram data collection approach started failing in ways that were difficult to ignore.&lt;/p&gt;

&lt;p&gt;Requests that had worked earlier began returning errors. Some sessions stopped working, data collection became inconsistent, and repeated requests increased the likelihood of temporary restrictions. What should have been a straightforward data collection task turned into an exercise in managing Instagram's access limitations.&lt;/p&gt;

&lt;p&gt;The objective was simple: monitor what competitor brands publish on Instagram and understand how their engagement changes over time.&lt;/p&gt;

&lt;p&gt;I needed a more predictable way to collect public Instagram data without making my own infrastructure responsible for every authentication and access problem.&lt;/p&gt;

&lt;p&gt;That led me to HikerAPI.&lt;/p&gt;

&lt;p&gt;What I Was Trying to Build&lt;/p&gt;

&lt;p&gt;The goal was to create a competitor monitoring workflow that could collect and organize publicly accessible Instagram information.&lt;/p&gt;

&lt;p&gt;The main things I wanted to track were:&lt;/p&gt;

&lt;p&gt;Competitor brand usernames and profile information.&lt;/p&gt;

&lt;p&gt;Recently published posts.&lt;/p&gt;

&lt;p&gt;Captions and publication timestamps.&lt;/p&gt;

&lt;p&gt;Likes and comment counts.&lt;/p&gt;

&lt;p&gt;Changes in engagement over time.&lt;/p&gt;

&lt;p&gt;Posting frequency and content patterns.&lt;/p&gt;

&lt;p&gt;The interesting part was not collecting a single post. It was collecting information consistently enough to compare activity across different periods.&lt;/p&gt;

&lt;p&gt;For example, knowing that a competitor published a post is useful. Knowing how its engagement changed over the following days is considerably more useful.&lt;/p&gt;

&lt;p&gt;Why My Previous Approach Started Breaking&lt;/p&gt;

&lt;p&gt;Instagram does not provide unrestricted access to its platform data through ordinary requests.&lt;/p&gt;

&lt;p&gt;When collecting data through unofficial scraping methods, several problems become relevant.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Rate limits&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Repeated requests can trigger temporary restrictions. Collecting profiles, posts, followers and engagement information can require multiple requests, making large collection jobs particularly difficult.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Session and authentication problems&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Unofficial scraping libraries often depend on authenticated sessions. Sessions can expire, authentication challenges can appear, and previously working scripts can stop functioning.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Unpredictable failures&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;A collection process that works today might fail tomorrow. This makes scheduled monitoring unreliable.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Maintenance overhead&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Instead of concentrating on data processing and competitor analysis, I was spending time dealing with request failures, authentication and changes in Instagram's internal endpoints.&lt;/p&gt;

&lt;p&gt;I needed to reduce that operational overhead.&lt;/p&gt;

&lt;p&gt;Why I Moved to HikerAPI&lt;/p&gt;

&lt;p&gt;I started looking at hosted Instagram data APIs instead of maintaining the entire scraping infrastructure myself.&lt;/p&gt;

&lt;p&gt;HikerAPI provides a hosted REST API for accessing Instagram data.&lt;/p&gt;

&lt;p&gt;The main difference is that I can make API requests to a dedicated service rather than implementing and maintaining all the underlying collection logic myself.&lt;/p&gt;

&lt;p&gt;Some things that attracted me to it:&lt;/p&gt;

&lt;p&gt;REST-based requests.&lt;/p&gt;

&lt;p&gt;Structured JSON responses.&lt;/p&gt;

&lt;p&gt;Hosted infrastructure.&lt;/p&gt;

&lt;p&gt;Access to different Instagram data endpoints.&lt;/p&gt;

&lt;p&gt;Usage-based pricing.&lt;/p&gt;

&lt;p&gt;A free allocation of 100 requests.&lt;/p&gt;

&lt;p&gt;Pricing starting from $0.001 per request.&lt;/p&gt;

&lt;p&gt;For a small monitoring workflow, the ability to test with a limited number of requests before committing to a larger usage plan was useful.&lt;/p&gt;

&lt;p&gt;I also found its documentation on HikerAPI's approach to Instagram data collection useful when evaluating the alternative.&lt;/p&gt;

&lt;p&gt;Making a Request with Python&lt;/p&gt;

&lt;p&gt;One of the advantages of using a REST API is that the Python implementation is relatively straightforward.&lt;/p&gt;

&lt;p&gt;I can use the requests library to communicate with the service.&lt;/p&gt;

&lt;p&gt;Here is a basic example of retrieving a user profile and requesting follower data:&lt;/p&gt;

&lt;p&gt;import requests&lt;/p&gt;

&lt;p&gt;headers = {"x-access-key": "YOUR_KEY"}&lt;/p&gt;

&lt;p&gt;user = requests.get(&lt;br&gt;
    "&lt;a href="https://api.hikerapi.com/v2/user/by/username?username=natgeo" rel="noopener noreferrer"&gt;https://api.hikerapi.com/v2/user/by/username?username=natgeo&lt;/a&gt;",&lt;br&gt;
    headers=headers&lt;br&gt;
).json()&lt;/p&gt;

&lt;p&gt;r = requests.get(&lt;br&gt;
    "&lt;a href="https://api.hikerapi.com/gql/user/followers/chunk" rel="noopener noreferrer"&gt;https://api.hikerapi.com/gql/user/followers/chunk&lt;/a&gt;",&lt;br&gt;
    params={"user_id": user["pk"]},&lt;br&gt;
    headers=headers&lt;br&gt;
)&lt;/p&gt;

&lt;p&gt;print(r.json())&lt;/p&gt;

&lt;p&gt;The first request retrieves the user information using a username.&lt;/p&gt;

&lt;p&gt;The second request uses the returned profile ID to request follower information.&lt;/p&gt;

&lt;p&gt;The API returns JSON, which makes it easier to process the results using Python.&lt;/p&gt;

&lt;p&gt;For a production implementation, I would also add:&lt;/p&gt;

&lt;p&gt;HTTP status-code validation.&lt;/p&gt;

&lt;p&gt;Request timeouts.&lt;/p&gt;

&lt;p&gt;Exception handling.&lt;/p&gt;

&lt;p&gt;Pagination handling.&lt;/p&gt;

&lt;p&gt;Logging.&lt;/p&gt;

&lt;p&gt;Secure API-key storage.&lt;/p&gt;

&lt;p&gt;I would not treat a successful HTTP response as proof that the returned data is complete. Pagination and API response validation still matter.&lt;/p&gt;

&lt;p&gt;Applying This to Competitor Monitoring&lt;/p&gt;

&lt;p&gt;The actual value comes from what happens after collecting the data.&lt;/p&gt;

&lt;p&gt;I would organize the workflow into four stages.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Collect profile information&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Maintain a list of competitor brand usernames and retrieve their publicly available profile information.&lt;/p&gt;

&lt;p&gt;Store the platform's user ID alongside the username so that subsequent requests can reference the correct account.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Collect recent posts&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Retrieve publicly accessible posts and store relevant fields:&lt;/p&gt;

&lt;p&gt;Post ID.&lt;/p&gt;

&lt;p&gt;Username.&lt;/p&gt;

&lt;p&gt;Caption.&lt;/p&gt;

&lt;p&gt;Publication timestamp.&lt;/p&gt;

&lt;p&gt;Like count.&lt;/p&gt;

&lt;p&gt;Comment count.&lt;/p&gt;

&lt;p&gt;Media type.&lt;/p&gt;

&lt;p&gt;The post ID is particularly important because it allows me to associate subsequent engagement observations with the same post.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Track engagement changes&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Rather than overwriting engagement numbers every time I collect data, I would maintain historical snapshots.&lt;/p&gt;

&lt;p&gt;A simple database structure could look like this:&lt;/p&gt;

&lt;p&gt;post_id&lt;/p&gt;

&lt;p&gt;collected_at&lt;/p&gt;

&lt;p&gt;likes&lt;/p&gt;

&lt;p&gt;comments&lt;/p&gt;

&lt;p&gt;12345&lt;/p&gt;

&lt;p&gt;Day 1&lt;/p&gt;

&lt;p&gt;120&lt;/p&gt;

&lt;p&gt;12&lt;/p&gt;

&lt;p&gt;12345&lt;/p&gt;

&lt;p&gt;Day 2&lt;/p&gt;

&lt;p&gt;165&lt;/p&gt;

&lt;p&gt;18&lt;/p&gt;

&lt;p&gt;12345&lt;/p&gt;

&lt;p&gt;Day 3&lt;/p&gt;

&lt;p&gt;210&lt;/p&gt;

&lt;p&gt;24&lt;/p&gt;

&lt;p&gt;This makes it possible to calculate changes over time.&lt;/p&gt;

&lt;p&gt;For example:&lt;/p&gt;

&lt;p&gt;def engagement_change(old, new):&lt;br&gt;
    if old == 0:&lt;br&gt;
        return None&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;return ((new - old) / old) * 100
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;

&lt;p&gt;likes_change = engagement_change(120, 210)&lt;/p&gt;

&lt;p&gt;print(f"Likes increased by {likes_change:.2f}%")&lt;/p&gt;

&lt;p&gt;Output:&lt;/p&gt;

&lt;p&gt;Likes increased by 75.00%&lt;/p&gt;

&lt;p&gt;These numbers are illustrative, not actual competitor measurements.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Analyze the results&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Once the information is stored, Python libraries such as Pandas and Matplotlib can help identify:&lt;/p&gt;

&lt;p&gt;Which brands publish most frequently.&lt;/p&gt;

&lt;p&gt;Which content formats receive more engagement.&lt;/p&gt;

&lt;p&gt;How engagement changes after publication.&lt;/p&gt;

&lt;p&gt;Which posting periods correlate with higher engagement.&lt;/p&gt;

&lt;p&gt;How competitor activity changes over time.&lt;/p&gt;

&lt;p&gt;This is where the project becomes more than a scraper. It becomes a monitoring and analysis system.&lt;/p&gt;

&lt;p&gt;What I Gained and What I Didn't&lt;/p&gt;

&lt;p&gt;Moving to a hosted API does not eliminate every problem.&lt;/p&gt;

&lt;p&gt;There are still tradeoffs.&lt;/p&gt;

&lt;p&gt;What improved:&lt;/p&gt;

&lt;p&gt;Less responsibility for maintaining scraping infrastructure.&lt;/p&gt;

&lt;p&gt;A straightforward REST interface.&lt;/p&gt;

&lt;p&gt;Easier integration with Python.&lt;/p&gt;

&lt;p&gt;Structured responses that can be stored and analyzed.&lt;/p&gt;

&lt;p&gt;A pricing model that can be evaluated against collection requirements.&lt;/p&gt;

&lt;p&gt;What still needs attention:&lt;/p&gt;

&lt;p&gt;API usage costs can increase with collection frequency.&lt;/p&gt;

&lt;p&gt;Rate limits and service restrictions can still exist.&lt;/p&gt;

&lt;p&gt;API availability depends on the provider.&lt;/p&gt;

&lt;p&gt;Data availability and completeness are not guaranteed.&lt;/p&gt;

&lt;p&gt;Instagram can change what information is accessible.&lt;/p&gt;

&lt;p&gt;API access does not grant permission to retrieve private-account content.&lt;/p&gt;

&lt;p&gt;I also need to consider Instagram's terms, applicable privacy requirements and the API provider's usage policies.&lt;/p&gt;

&lt;p&gt;A hosted API changes where the collection infrastructure runs. It does not remove the need to design responsible data collection.&lt;/p&gt;

&lt;p&gt;Final Thoughts&lt;/p&gt;

&lt;p&gt;The biggest lesson from this process was that collecting data and maintaining a reliable data collection system are two different engineering problems.&lt;/p&gt;

&lt;p&gt;Initially, I was spending too much time trying to keep the collection mechanism working. For a competitor monitoring project, the more important work is storing observations, tracking changes and extracting useful information.&lt;/p&gt;

&lt;p&gt;Moving the collection layer to a hosted REST API gave me a different architecture to work with.&lt;/p&gt;

&lt;p&gt;It is not a guarantee against restrictions or failures, and it introduces a dependency on an external provider. However, for a project focused on monitoring publicly available competitor content and engagement, it is an option worth evaluating.&lt;/p&gt;

&lt;p&gt;I would rather spend development time understanding engagement trends than repeatedly debugging a collection script.&lt;/p&gt;

</description>
      <category>api</category>
      <category>automation</category>
      <category>data</category>
      <category>socialmedia</category>
    </item>
  </channel>
</rss>
