DEV Community

loading...
Cover image for Python web scrapping on GCP

Python web scrapping on GCP

goffity
・1 min read

Content

  1. build container images.
  2. tag images.
  3. upload images to GCP.
  4. create services.

build container images.

  • selenium.
  • web scrapping.

tag images.

  • image with GCP pattern.
docker pull selenium/standalone-chrome
# tag with pattern $GCP_REGISTRY/$PROJECT_NAME/$IMAGE_NAME
docker tag selenium_standalone-chrome asia.gcr.io/$PROJECT_NAME/selenium-standalone-chrome
# push image
docker push asia.gcr.io/$PROJECT_NAME/selenium-standalone-chrome
Enter fullscreen mode Exit fullscreen mode

upload images to GCP.

  • authentication on browser.
gcloud auth login
Enter fullscreen mode Exit fullscreen mode
  • setup project.
gcloud config set project $PROJECT_NAME
Enter fullscreen mode Exit fullscreen mode

create services.

  • Create from container images.
  # create selenium service.
gcloud run deploy selenium-chrome --image asia.gcr.io/$PROJECT_NAME/selenium_standalone-chrome --port 4444 --memory 2G --region asia-southeast1 --platform managed
Enter fullscreen mode Exit fullscreen mode
  • Create from cloud build.
# gcloud builds submit --tag asia.gcr.io/$PROJECT_NAME/$IMAGE_NAME

gcloud builds submit --tag asia.gcr.io/$PROJECT_NAME/web-scrpping
# deploy service
gcloud run deploy selenium-chrome --image asia.gcr.io/$PROJECT_NAME/selenium_standalone-chrome
Enter fullscreen mode Exit fullscreen mode

Reference:
https://www.roelpeters.be/how-to-deploy-a-scraping-script-and-selenium-in-google-cloud-run/
https://dev.to/googlecloud/using-headless-chrome-with-cloud-run-3fdp

Discussion (0)