DEV Community: bchirag3

Using Python for Telegram Scrapping Part 2

bchirag3 — Fri, 27 Jan 2023 04:48:38 +0000

Hey techies how are you doing? 😀😀 In the last article, we had gone through pyrogram and its basic methods. If you haven't read that article please go through it to get a better perspective on this one. This is the link for the same.

Telegram Scrapping Part 1📰📰

Before you proceed it's assumed you have done the setup for API, know how to use basic methods of pyrogram and also have the necessary packages.

Now we will dive deep into some other methods and their use cases. Earlier we had gone through a method get_chat_history where we could retrieve the method of the chats from a particular channel. Let's assume there is a use case where you want to retrieve the messages from a particular date range let's say the previous 2 years' messages. How do you go about that? Well, that's what we will look onto.

Firstly offset_date parameter needs to be set to a checkpoint from which we want to start scrapping in the get_chat_history method. For this, we will be needing the DateTime package which can be installed by pasting these lines onto the terminal.

pip install DateTime

To learn more about DateTime here are the official docs.

To compare DateTime we need it in a particular format. Try executing these lines in a new file or in the python terminal itself to have a look into the format.

from datetime import datetime

offset_date = datetime(2022, 12, 31)
print(offset_date)

Now we are all set for using DateTime. In the .env I'll be adding another parameter chat_id to use it in the program.

TG_API_ID=
TG_API_HASH=
CHAT_ID=

Let's set the offset date as 31st December 2022 and start_date as 31st December 2020. Offset refers to the date from which we want to start scrapping our messages. In all, we will get data for 2 years in this range. Let's save this code in another python file and execute this.

Remember💡💡 - To join the telegram channel for scrapping before proceeding further. Here the chat_id for the channel New York Times is -1001606432449.

from pyrogram import Client
from dotenv import load_dotenv
import os
from datetime import datetime

load_dotenv()

CONFIG = {
    "telegram_api_id": int(os.getenv("TG_API_ID")),
    "telegram_hash": os.getenv("TG_API_HASH"),
    "chat_id": os.getenv("CHAT_ID")
}

app = Client("my_account",CONFIG["telegram_api_id"],CONFIG["telegram_hash"])

def get_dates():
    offset_date = datetime(2022, 12, 31)
    start_date = datetime(2020, 12, 31)
    return offset_date,start_date

async def main():
    chat_id = CONFIG["chat_id"]
    offset_date, start_date = get_dates()

    async with app:
        async for message in app.get_chat_history(chat_id, offset_date=offset_date):
            if(message.date > start_date):
                print(message)
            else:
                break

app.run(main())

Here we have set the offset date and start date. Messages are scrapped from the offset date and the loop breaks when we reach the start date.

Interrupt the terminal to stop the flow of execution. The terminal output represents the raw data which we receive from telegram.

Let's see one for an example.

{
    "_": "Message",
    "id": 1765,
    "sender_chat": {
        "_": "Chat",
        "id": -1001606432449,
        "type": "ChatType.CHANNEL",
        "is_verified": true,
        "is_restricted": false,
        "is_creator": false,
        "is_scam": false,
        "is_fake": false,
        "title": "The New York Times",
        "username": "nytimes",
        "photo": {
            "_": "ChatPhoto",
            "small_file_id": "AQADAQAD5KkxG-K0WEUAEAIAAz-5mssW ____ YNzUE_UN61kABB4E",
            "small_photo_unique_id": "AgAD5KkxG-K0WEU",
            "big_file_id": "AQADAQAD5KkxG-K0WEUAEAMAAz-5mssW ____ YNzUE_UN61kABB4E",
            "big_photo_unique_id": "AgAD5KkxG-K0WEU"
        },
        "dc_id": 1,
        "has_protected_content": false
    },
    "date": "2022-11-21 07:44:28",
    "chat": {
        "_": "Chat",
        "id": -1001606432449,
        "type": "ChatType.CHANNEL",
        "is_verified": true,
        "is_restricted": false,
        "is_creator": false,
        "is_scam": false,
        "is_fake": false,
        "title": "The New York Times",
        "username": "nytimes",
        "photo": {
            "_": "ChatPhoto",
            "small_file_id": "AQADAQAD5KkxG-K0WEUAEAIAAz-5mssW ____ YNzUE_UN61kABB4E",
            "small_photo_unique_id": "AgAD5KkxG-K0WEU",
            "big_file_id": "AQADAQAD5KkxG-K0WEUAEAMAAz-5mssW ____ YNzUE_UN61kABB4E",
            "big_photo_unique_id": "AgAD5KkxG-K0WEU"
        },
        "dc_id": 1,
        "has_protected_content": false
    },
    "mentioned": false,
    "scheduled": false,
    "from_scheduled": false,
    "media": "MessageMediaType.PHOTO",
    "media_group_id": 13351974942043841,
    "has_protected_content": false,
    "photo": {
        "_": "Photo",
        "file_id": "AgACAgQAAx0CX8A2wQACBuVjxpZxAznh6hEOOjxOQgKfUFhWXAACHbAxG8GJ3FOUMbI-UPTM6QAIAQADAgADeQAHHgQ",
        "file_unique_id": "AgADHbAxG8GJ3FM",
        "width": 1050,
        "height": 550,
        "file_size": 195554,
        "date": "2022-11-21 07:44:10",
        "thumbs": [
            {
                "_": "Thumbnail",
                "file_id": "AgACAgQAAx0CX8A2wQACBuVjxpZxAznh6hEOOjxOQgKfUFhWXAACHbAxG8GJ3FOUMbI-UPTM6QAIAQADAgADbQAHHgQ",
                "file_unique_id": "AgADHbAxG8GJ3FM",
                "width": 320,
                "height": 168,
                "file_size": 20631
            }
        ]
    },
    "views": 22543,
    "outgoing": false
}

A small summary -

Each message has a unique id which starts from 1 and keeps on incrementing.
Each message consists info of about the sender's chat such as username, isbot and so on.
For a message, if a media is embedded, it contains data about that.
For example here each photo has a big_file_id or small_file_id which can be used to download the media.
Similarly, if a document or emoji is embedded it contains information regarding that.

Let's try downloading this media to our local machine.

To retrieve the same message as above we are going to use the get_messages API call. At once one can retrieve 200 messages. Params required to pass are first the chat_id, second the message id or an array of message ids. Here the message id is 1765.

    async with app:
        message = await app.get_messages(chat_id, 1765)
        print(message)

This will give the same message. Our goal is to extract the media from the message.

To download this media there is a method called download_media present in python that needs params file_id of media and optional file_name to save in our directory.

async def main():
    chat_id = CONFIG["chat_id"]
    async with app:
        message = await app.get_messages(chat_id, 1765)
        file = await app.download_media(message.photo.file_id, file_name="example.png")
        print(file)

app.run(main())

On running this file the media gets stored in a separate folder inside downloads as example.png.

Now since we are done with exploring some methods another use case arrives. Now our goal is to extract all the new messages. For example in WhatsApp on the notification panel whenever you receive a message a notification is pinned which displays that message. Similar behaviour we would like to imitate in telegram. Well, it's an interesting use case and is possible to do in pyrogram.

The decorator method on_message can be used for the same.

@app.on_message()
def log(client, message):
    print(message)

app.run()

This catches all the new messages which you will receive. To test this on a single group add an extra parameter group with chat_id or an array of chat_ids.

@app.on_message(group=chat_id)

Try sending a message to a group to test this code.

Hush we have reached the end.

Conclusion -

Firstly we can extract messages of a channel for a particular interval of time for which we got to have a glance at how the JSON message comes from telegram API. We also go to see how to store media like photos on our local machine. The same can be done for documents like pdf or videos of different formats. Lastly, the use case which was covered was how to display new messages for the whole telegram or a particular group chat. These are only some of the use cases covered which gives us a general idea of how to go about our way using telegram API. We can further extend the application to several use cases as the application demands.

One task which I would suggest is to go through pyrogram docs, create a separate group and try sending media of all formats, messages and downloading, loading them and tinkering around with them.

Unleashing the Power of Python for Telegram Scraping

bchirag3 — Tue, 17 Jan 2023 06:54:50 +0000

Hey everyone this is my first blog post. I plan on writing content on Python, AWS and so on. Hope you like it.😀😀😀

There are several ways that can be used for scrapping data from telegram. These include libraries like telethon and pyrogram for python. In this article, we will go through how to use pyrogram.

For this article, I will be using Vscode as my text editor. You can follow along with it or choose the preferred editor that suits you better.

Here is the official documentation for pyrogram. For the sake of simplicity, we will be sticking to some of the basics and most commonly used methods.

Pyrogram Docs

Building your API Keys -

To start off the prerequisites are having a telegram account and API setup. If you have the API keys you can skip the setup section.

Telegram API

Upon following this link, you can click over to the API development tools and enter the details. After filling up the form the application gets created. Once the application is created you get an App api_id and App api_hash using which we can use the program. Store these API keys in a safe place as you will be needing them further.

For starters, we can use this template from Pyrogram docs.

from pyrogram import Client

app = Client("my_account")

with app:
    app.send_message("me", "Hi!")

Create a new folder called PYROGRAM. Within that folder create a . env that stores the API id and API hash which you had saved before.

TG_API_ID="Your api id"
TG_API_HASH="Your api hash"

You will also need to install certain packages -

os
pyrogram
dotenv

Copy and paste these lines into the terminal.

Packages -

pip install os-sys
pip install pyrogram
pip install python-dotenv

Then install tgcrypto to improve the pyrogram performance.

pip install TgCrypto

Now for the last step, our python file looks like this. Create a new file as pyrogram_starter.py and paste the following code.

from pyrogram import Client
from dotenv import load_dotenv
import os

load_dotenv()

CONFIG = {
    "telegram_api_id": int(os.getenv("TG_API_ID")),
    "telegram_hash": os.getenv("TG_API_HASH"),
}

app = Client("my_account",CONFIG["telegram_api_id"],CONFIG["telegram_hash"])

with app:
    app.send_message("me", "Hello from pyrogram")

Here we have used the dotenv package to read our API keys from the .env file. For initializing the Client for telegram it needs the API keys which we have loaded through the load_dotenv() method.

On running this file for the first time it asks for your telegram number upon which a session file is created which can be used directly and these steps won't have to be repeated again.

Upon confirmation, we can verify this message from our telegram account.

Hurray, we have done the initial setup required for pyrogram 🥳🥳🥳🥳

There are several API methods available in the pyrogram API we are going to test and implement a few of them.

Let's start with the scrapping of channel messages for this we are gonna use the get_chat_history API call. This method returns the messages in reverse chronological order. There are several parameters which we can pass such as limit, and offset. By default, there are no limits applied to this API call.

To scrape data from a particular channel the user must join the particular channel.

For this particular call, we will be scrapping our messages from the New York Times.

Our first task is to extract the channel id or ref which we need to pass to this API call.

If it's a public channel you can simply use its username as an id. Another way is to extract the channel id through the URL.

Remember to use the Web Version.

For this channel, the corresponding channel id is -1242127973. But for the API call to work, we need to add -1001242127973 as the new channel id.

For more information, you can refer to this Telegram ID GitHub.

Now as this call is an asynchronous method we need to use async in Python. For example to imitate the asynchronous behaviour we can use the asyncio library. Buts that's not in the scope of this article maybe for another one.

Now for our code to work it looks like.

app = Client("my_account",CONFIG["telegram_api_id"],CONFIG["telegram_hash"])

chat_id = -1001242127973

async def main():
    async with app:
        async for message in app.get_chat_history(chat_id):
            print(message.text)

app.run(main())

You can also add chat_id as an env variable or as a part of utilities.

If you execute this code it will run infinitely as there are no limits applied. Interrupt the terminal to stop the flow of execution. The terminal output gives us the messages from this telegram channel.

If you want to have a glance at the JSON for the message try writing the print statement by removing the .text.

print(message)

By using a limit we can put a hard stop after getting the required number of messages.

Let's say you want to access the chat for personal contact from the telegram web. The same process can be repeated which we used to get for the telegram group. But in this case, we don't need to add -100 as a prefix to the chat id just adding - will do the work.

For getting information corresponding to you. You can use the get_me api call.

me = await app.get_me()
print(me)

Let's say you have joined a public group and want to extract all the user's info. You can use the get_chat_members method. But there are some cases where permissions are set and you can't access the member info.

Also, there is a limitation to this call. It only returns the first 10k members. There is no way for one to extract all the users if the member count exceeds more than 10k.

async def main():
    async with app:
       async for member in app.get_chat_members(chat_id):
            print(member)

app.run(main())

If you want to send a message such as an update through an API call you can use the send_message method which we had used in the very beginning. Replace the chat_id with your respective channel id.

Advice - Create a separate group and try testing these methods on that group.

💡💡💡💡

async def main():
    async with app:
       await app.send_message(chat_id, "Message sent with **Pyrogram**!")

app.run(main())

Conclusion -

By following this article we got a basic understanding of how to use pyrogram and its API calls. We learnt how to use commonly used methods such as get_chat_history, send_message and so on. Now to dive deeper it's advised to go through the official pyrogram docs and try testing out by building your own application in python.

Hope you enjoyed it. 😀😀😀😀