Python web scrapping on GCP

Content

1. build container images.
2. tag images.
3. upload images to GCP.
4. create services.

---

build container images.

• selenium.
• web scrapping.

tag images.

• image with GCP pattern.

docker pull selenium/standalone-chrome
# tag with pattern $GCP_REGISTRY/$PROJECT_NAME/$IMAGE_NAME
docker tag selenium/standalone-chrome asia.gcr.io/$PROJECT_NAME/selenium-standalone-chrome
# push image
docker push asia.gcr.io/$PROJECT_NAME/selenium-standalone-chrome

upload images to GCP.

• authentication on browser.

gcloud auth login

• setup project.

gcloud config set project $PROJECT_NAME

create services.

• Create from container images.

  # create selenium service.
gcloud run deploy selenium-chrome --image asia.gcr.io/$PROJECT_NAME/selenium_standalone-chrome --port 4444 --memory 2G --region asia-southeast1 --platform managed

• Create from cloud build.

# gcloud builds submit --tag asia.gcr.io/$PROJECT_NAME/$IMAGE_NAME

gcloud builds submit --tag asia.gcr.io/$PROJECT_NAME/web-scrpping
# deploy service
gcloud run deploy selenium-chrome --image asia.gcr.io/$PROJECT_NAME/selenium_standalone-chrome

Reference:
https://www.roelpeters.be/how-to-deploy-a-scraping-script-and-selenium-in-google-cloud-run/
https://dev.to/googlecloud/using-headless-chrome-with-cloud-run-3fdp