DEV Community: MAK KA WAI

Date: 29/08/2024

MAK KA WAI — Thu, 29 Aug 2024 14:10:13 +0000

I added the chat bot to my Application

Date: 28/08/2024

MAK KA WAI — Thu, 29 Aug 2024 14:08:42 +0000

I applied for the google search API. Maybe useful later.

I made the button for scrolling. After a while, I found that it is useless.

Daily Log - 27/08/2024

MAK KA WAI — Tue, 27 Aug 2024 15:11:37 +0000

I integrated the function to my prototype.

Daily Log - 26/08/2024

MAK KA WAI — Mon, 26 Aug 2024 12:53:50 +0000

Today, I made a text area for negative prompt to filter unwanted job(s).

Daily Log - 25/08/2024

MAK KA WAI — Mon, 26 Aug 2024 12:53:17 +0000

I made the a dropdown list to filter unwanted job’s regions.

Daily Log - 24/08/2024

MAK KA WAI — Sat, 24 Aug 2024 14:44:18 +0000

For my application, now it can search for the appropriate jobs by uploading resume.

I ran web crawler. Now, 5108 jobs in ChromaDB.

I found the ‘fetch all jobs’ function will cause the problem of lack of ram.

Daily Log - 23/08/2024

MAK KA WAI — Fri, 23 Aug 2024 15:20:55 +0000

Overall

I was developing my AI application.

Problem facing

I found that Jinja is conflict with jsonify(), result in error of upload file and chat with GPT. Mentor helped me to solve it.

Learn

Since I want to extract all company ID in https://www.ctgoodjobs.hk/ . I tried to web crawling. Seccess.

import requests
from bs4 import BeautifulSoup
import json
import parsel  # 第三方的模块

def main():

    headers = {
        'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36',
    }
    # Fetch the job listings from the API
    url = "https://www.ctgoodjobs.hk/top-companies"
    html_data = requests.get(url=url, headers=headers).text
    selector = parsel.Selector(html_data)

    # .get(): return string; no: return lsit
    # .get: return 1st; no: all
    lists = selector.css('.sub-sec li')
    extra_data = selector.css('div.sub-sec ul.extra::text').get() # no text: string with <element> tag 

    company_ids = []

    # tokenization
    company_ids = extra_data.strip().split(',')

    for list in lists:
        company_id = list.css('a::attr(data-company-id)').get()
        company_ids.append(company_id)

    print(company_ids)
    print(len(company_ids))


if __name__ == "__main__":
    main()

Daily Log - 22/08/2024

MAK KA WAI — Thu, 22 Aug 2024 14:30:07 +0000

I continued to develop my Application. I was making the html elements, some minor functions and python flask.

Daily Log - 21/08/2024

MAK KA WAI — Wed, 21 Aug 2024 15:08:56 +0000

I started to develop the first prototype of my application

Daily Log - 20/08/2024

MAK KA WAI — Wed, 21 Aug 2024 04:34:28 +0000

I tried to design my application and build my html template.

Daily Log - 19/08/2024

MAK KA WAI — Mon, 19 Aug 2024 13:19:57 +0000

I found a big problem. When I was reading the data extracted from ctgoodjobs.hk in chroma DB, there is no a record with real salary value, all are N/A. There is job with real salary value.

e.g.
Hash: deef24dd43347dfd6077827d6bdfa86d
https://jobs.ctgoodjobs.hk/job/08976889/licensing-officer-shatin-22k-transport-department

Salary has real number

From streamlit run app.py

Salary hasn’t real number

I try to run ‘crawler.py’ and ‘app.py’ again.
Another problem was occurred. In the previous day, I thought running ‘crawler.py’ and ‘app.py’ successfully, I was worried it may not work.

I didn’t understand the query. When I searched with a job title, no correct output.

In order to check whether successful to add record to chroma DB. I ran program

# job_counter.py

import chromadb

def connect_to_chromadb():
    client = chromadb.PersistentClient(path="./job_posts")  # 確保這個路徑與主應用程序中的路徑相同
    return client

def count_jobs(client):
    collection_name = "jobs"
    collection = client.get_collection(name=collection_name)
    return collection.count()

if __name__ == "__main__":
    client = connect_to_chromadb()
    total_jobs = count_jobs(client)
    print(f"Total number of jobs in the database: {total_jobs}")

before and after running crawler. Result is success.

After I modify the code of ‘crawler.py’, the salary was shown now.

Daily Log - 18/08/2024

MAK KA WAI — Sun, 18 Aug 2024 11:53:43 +0000

Today is Sunday. Rest!