DEV Community

Cover image for Python web scrapping on GCP
Goffity Corleone
Goffity Corleone

Posted on Edited on

Python web scrapping on GCP

#python #selenium #gcp

Content

  1. build container images.
  2. tag images.
  3. upload images to GCP.
  4. create services.

build container images.

  • selenium.
  • web scrapping.

tag images.

  • image with GCP pattern. 
docker pull selenium/standalone-chrome
# tag with pattern $GCP_REGISTRY/$PROJECT_NAME/$IMAGE_NAME
docker tag selenium/standalone-chrome asia.gcr.io/$PROJECT_NAME/selenium-standalone-chrome
# push image
docker push asia.gcr.io/$PROJECT_NAME/selenium-standalone-chrome
Enter fullscreen mode Exit fullscreen mode

upload images to GCP.

  • authentication on browser. 
gcloud auth login
Enter fullscreen mode Exit fullscreen mode
  • setup project. 
gcloud config set project $PROJECT_NAME
Enter fullscreen mode Exit fullscreen mode

create services.

  • Create from container images. 
  # create selenium service.
gcloud run deploy selenium-chrome --image asia.gcr.io/$PROJECT_NAME/selenium_standalone-chrome --port 4444 --memory 2G --region asia-southeast1 --platform managed
Enter fullscreen mode Exit fullscreen mode
  • Create from cloud build. 
# gcloud builds submit --tag asia.gcr.io/$PROJECT_NAME/$IMAGE_NAME

gcloud builds submit --tag asia.gcr.io/$PROJECT_NAME/web-scrpping
# deploy service
gcloud run deploy selenium-chrome --image asia.gcr.io/$PROJECT_NAME/selenium_standalone-chrome
Enter fullscreen mode Exit fullscreen mode

Reference:
https://www.roelpeters.be/how-to-deploy-a-scraping-script-and-selenium-in-google-cloud-run/
https://dev.to/googlecloud/using-headless-chrome-with-cloud-run-3fdp

Top comments (0)