The challenge contains 4 required tasks
- Run a simple Dataflow job
- Run a simple Dataproc job
- Run a simple Dataprep job
- Complete one of the AI tasks
1.Run a simple Dataflow job
bq mk lab
gsutil cp gs://cloud-training/gsp323/lab.csv .
cat lab.csv
gsutil cp gs://cloud-training/gsp323/lab.schema .
cat lab.schema
2. Run a simple Dataproc job
You have used Dataproc in the quest, now you must run another example Spark job using Dataproc.
Before you run the job, log into one of the cluster nodes and copy the /data.txt file into hdfs (use the command hdfs dfs -cp gs://cloud-training/gsp323/data.txt /data.txt)
Run a Dataproc job using the values below
Wait for the job to finish then click on check progress.
3. Run a simple Dataprep job
- You have used Dataprep to import data files and transformed them to gain views of the data. Use Dataprep to import one CSV file (described below) that holds data of lab executions.
gs://cloud-training/gsp323/runs.csv structure:
Perform the following transforms to ensure the data is in the right state:
- Remove all rows with the state of "FAILURE"
- Remove all rows with 0 or 0.0 as a score (Use the regex pattern /(^0$|^0.0$)/)
- Label columns with the names above
Note: Your Dataprep job should look similar to the following
Make sure you run the job. You will need to wait until the Dataflow job completes before you can grade this task.
4. AI
Complete one of the following tasks below.
Use Google Cloud Speech API to analyze the audio file gs://cloud-training/gsp323/task4.flac. Once you have analyzed the file you can upload the resulting analysis to Cloud Speech Location.
Use the Cloud Natural Language API to analyze the sentence from text about Odin. The text you need to analyze is "Old Norse texts portray Odin as one-eyed and long-bearded, frequently wielding a spear named Gungnir and wearing a cloak and a broad hat." Once you have analyzed the text you can upload the resulting analysis to Cloud Natural Language Location.
Use Google Video Intelligence and detect all text on the video gs://spls/gsp154/video/train.mp4. Once you have completed the processing of the video, pipe the output into a file and upload to Video Intelligence Location . Ensure the progress of the operation is complete and the service account you're uploading the output with has the Storage Object Admin role
gcloud iam service-accounts create my-natlang-sa \
--display-name "my natural language service account"
gcloud iam service-accounts keys create ~/key.json \
--iam-account my-natlang-sa@${GOOGLE_CLOUD_PROJECT}.iam.gserviceaccount.com
export GOOGLE_APPLICATION_CREDENTIALS="/home/$USER/key.json"
gcloud auth activate-service-account my-natlang-sa@${GOOGLE_CLOUD_PROJECT}.iam.gserviceaccount.com --key-file=$GOOGLE_APPLICATION_CREDENTIALS
gcloud ml language analyze-entities --content="Old Norse texts portray Odin as one-eyed and long-bearded, frequently wielding a spear named Gungnir and wearing a cloak and a broad hat." > result.json
gcloud auth login
(Copy the token from the link provided)
gsutil cp result.json gs://YOUR_PROJECT-marking/task4-cnl.result
Create an API key and export as API_KEY variable
export API_KEY={Replace with API KEY}
nano request.json
Add this content:
{
"config": {
"encoding":"FLAC",
"languageCode": "en-US"
},
"audio": {
"uri":"gs://cloud-training/gsp323/task4.flac"
}
}
TERMINAL
curl -s -X POST -H "Content-Type: application/json" --data-binary @request.json \
"https://speech.googleapis.com/v1/speech:recognize?key=${API_KEY}" > result.json
gsutil cp result.json gs://YOUR_PROJECT-marking/task4-gcs.result
gcloud iam service-accounts create quickstart
gcloud iam service-accounts keys create key.json --iam-account quickstart@${GOOGLE_CLOUD_PROJECT}.iam.gserviceaccount.com
gcloud auth activate-service-account --key-file key.json
export ACCESS_TOKEN=$(gcloud auth print-access-token)
nano request.json
Top comments (0)