DEV Community: Hideki Okamoto

Cloud GPU instance with PyTorch and TensorFlow easy setup in 10 minutes

Hideki Okamoto — Thu, 22 Dec 2022 09:42:25 +0000

(10/29/2024 Update) This article has been updated to reflect the release of the RTX 4000 Ada GPU instance. Some screenshots and descriptions still reference the older RTX 6000 GPU, so interpret them accordingly.

PyTorch from Meta (ex-Facebook) and TensorFlow from Google are highly popular deep learning frameworks. While GPU is essential to the development and training process of deep learning models, it is time-consuming to build an environment to make GPU available for these frameworks and to make both PyTorch and TensorFlow usable in a single environment. This article shows how to set up an environment with GPU-enabled PyTorch and TensorFlow installed on Akamai Connected Cloud (formerly Linode) in 10 minutes. The procedures in this article make it easy to set up a dedicated deep learning environment in the cloud, even for those unfamiliar with setting up a Linux server.

What is Akamai Connected Cloud (formerly Linode)?

Akamai Connected Cloud (ACC / formerly Linode) is an IaaS acquired by Akamai in February 2022. ACC offers simple and predictable pricing, which includes SSD storage and a certain amount of network transfers within the price, which are often expensive in other cloud computing providers. There are no price differences by region, making spending on cloud providers more predictable. For example, a virtual machine with 16GB of memory with NVIDIA RTX 4000 Ada costs $350 per month (as of October 2024), including 500GB of SSD storage and 1TB of network transfer. If you use a service for just part of the month, hourly billing enables you to only be charged for the time the machine is present on the account.

You can use the Cloud Estimator tool provided by ACC to compare prices with other cloud providers.

The goal of this article

In this article, we will set up Docker Engine Utility for NVIDIA (nvidia-docker), a container virtualization platform that supports NVIDIA's GPUs, on a GPU instance on ACC, where we will deploy NGC Container, containers officially provided by Facebook and Google for deep learning. You can set up the environment in about 10 minutes with almost no prior knowledge of ACC, Docker, or NGC Container by using StackScripts, ACC's deployment automation function.

Tools - StackScripts

The environment built with this procedure includes a sample Jupyter Notebook that can be used with OpenAI Whisper, a speech recognition model that has been widely praised for its extremely high recognition accuracy so that even those who do not develop deep learning models themselves can experience the benefits of GPU instances.

If you provide the Object Storage credentials to the StackScript, the PyTorch and TensorFlow containers will automatically mount the external Object Storage, which can be used to retrieve training data from or to store your deep learning models. Using Object Storage is optional. You can skip it.

Setup a GPU instance with PyTorch and TensorFlow

First, open the StackScript I have prepared for you from the following link. This StackScript will automatically install nvidia-docker, PyTorch, and TensorFlow. (You must be logged into your ACC account to access the link.) If you can't open this StackScript for some reason, I have uploaded the contents of this StackScript to GitHub for you.

deeplearning-gpu
https://cloud.linode.com/stackscripts/1102035

Click "Deploy New Linode"

The regions where both Object Storage and RTX 4000 Ada GPU instances are available are as follows as of October 2024.

Seattle, WA, US
Chicago, IL, US
Paris, FR
Osaka, JP

Since GPU instances are available only in limited regions, select a virtual machine type first, then the region. Here I have selected Dedicated 32 GB + RTX6000 GPU x1 in the Singapore region for example.

Name the virtual machine, enter the root password, and click "Create Linode".

The screen will transition to the virtual machine management dashboard. Wait a few minutes until the virtual machine status changes from PROVISIONING to RUNNING. The IP address of the virtual machine you just created is displayed on the same screen, so take note of it.

The virtual machine is now booted. The installation process of nvidia-docker and NGC Containers will proceed automatically in the background. Wait 10 minutes for the installation to complete before proceeding to the next step.

Starting a container

Now let's log in to the virtual machine via SSH. If the setup process performed by StackScript is complete, the following message will appear when you log in. If you do not see this message, log out and wait a few minutes before logging in again. If you have inadvertently started a virtual machine that does not have a GPU, you will get the message "GPU is not available. This StackScript should be used for GPU instances." In that case, please start a GPU instance and redo the procedure from the beginning.

% ssh root@45.118.XX.XX
root@45.118.XX.XX's password:

(snip)

##############################################################################
You can launch a Docker container with each of the following commands:

pytorch: Log into an interactive shell of a container with Python and PyTorch.
tensorflow: Log into an interactive shell of a container with Python and TensorFlow.
pytorch-notebook: Start Jupyter Notebook with PyTorch as a daemon. You can access it at http://[Instance IP address]/
tensorflow-notebookm: Start Jupyter Notebook with TensorFlow as a daemon. You can access it at http://[Instance IP address]/

Other commands:
stop-all-containers: Stop all running containers.
##############################################################################

The following five commands are available on the machine created by this StackScript.

Command	Usage
pytorch	Start a container with PyTorch installed and enter its interactive shell
tensorflow	Start a container with TensorFlow installed and enter its interactive shell
pytorch-notebook	Start Jupyter Notebook with PyTorch installed as a daemon
tensorflow-notebook	Start Jupyter Notebook with TensorFlow installed as a daemon
stop-all-containers	Stop all running containers

Each container has the directories /workspace/HOST-VOLUME/ and /workspace/OBJECT-STORAGE/ to mount the host machine directory and external Object Storage. The container created by the above command is configured to remove the container when it is stopped (--rm option of docker run is set), so place the files you want to keep in /workspace/HOST-VOLUME/ or /workspace/OBJECT-STORAGE/.

Let's spin up Jupyter Notebook with PyTorch as a daemon and run a speech recognition model OpenAI Whisper. Run the pytorch-notebook command from the console.

root@45-118-XX-XXX:~# pytorch-notebook
[I 04:36:22.823 NotebookApp] http://hostname:8888/?token=0ee3290287b3bd90f2e8e3ab447965d3e074267f0d60420b
        http://hostname:8888/?token=0ee3290287b3bd90f2e8e3ab447965d3e074267f0d60420b

Jupyter Notebook should now be started. If you get the error "Bind for 0.0.0.0:80 failed: port is already allocated." Stop the existing container first with the stop-all-containers command. If you get the above result without any problem, replace hostname of the URL with the IP address of the virtual machine that you noted when creating the virtual machine, delete :8888, and access the virtual machine from a web browser. The token will change each time the container is started.

URL displayed in console	URL to be entered into a web browser
http://hostname:8888/?token=0ee3290287b3bd90f2e8e3ab447965d3e074267f0d60420b	http://45.118.XX.XX/?token=0ee3290287b3bd90f2e8e3ab447965d3e074267f0d60420b

Click on Voice Recognition with OpenAI Whisper.ipynb in HOST-VOLUME to open it.

Click Cell->Run All in the menu to run OpenAI Whisper. The first time you run it, it will take a few minutes to download dependent software and deep learning models.

If the execution completes without problems, the last cell will show the result of the speech recognition: "I'm getting them for $12 a night."

Congratulations! Now you have GPU-enabled PyTorch and TensorFlow

Deleting the instance

You can delete the virtual machine that you have finished by clicking "Delete" in the ACC Management Console. The contents of /workspace/HOST-VOLUME/ (/root/shared/ from the host OS) will be deleted, so move any files you want to keep to another location.

You are charged even for powered-off virtual machines. Delete virtual machines that you do not want to be charged for.

Platform - Billing

Access control for the instance

Access to the virtual machines created in the above procedure via SSH requires password authentication or public key authentication, and access to Jupyter Notebook requires token authentication. If you want to add access control based on the IP address of the client, refer to the following articles to apply firewalls to port 22 (SSH) and port 80 (HTTP).

Cloud Firewall - Get Started

For more advanced access control, Akamai's zero-trust solution, Enterprise Application Access, can be used for integration with external Identity Providers and SSO support.

Enabling HTTPS

Follow the steps below to enable HTTPS in Jupyter Notebook for production use.

Running a public notebook server

The five commands listed above are defined as aliases for docker commands in /root/.bash_profile. When HTTPS is enabled, the argument of the -p option of the docker command used by the pytorch-notebook and tensorflow-notebook commands should also be modified to the appropriate port such as 443. And finally, execute ufw allow 443/tcp so that the firewall allows port 443.

Visualizing Akamai DataStream 2 logs with Elasticsearch and Kibana on Linode

Hideki Okamoto — Fri, 23 Sep 2022 02:10:51 +0000

Setting up Elasticsearch and Kibana on Linode for visualizing Akamai DataStream 2 logs: All steps can be done using only a web browser and do not require login to the Linux console. The installation procedure takes only 10 minutes, excluding DataStream 2 activation time.

Akamai DataStream 2

Akamai DataStream 2 is a free feature that streams access logs from Akamai Intelligent Edge Platform to designated destinations in near real-time. As of November 2023, DataStream 2 can deliver access logs to the following destinations.

Amazon S3
Azure Storage
Custom HTTPS endpoint
Datadog
Elasticsearch
Google Cloud Storage
Loggly
New Relic
Oracle Cloud
S3-compatible destinations (incl. Linode Object Storage)
Splunk
Sumo Logic

Elasticsearch / Kibana

Elasticsearch is a full-text search engine developed by Elastic. Its source code is publicly available under a dual license, the Server Side Public License and the Elastic License. Kibana is data visualization software for Elasticsearch and is offered under the same terms as Elasticsearch. The combination of this two software with data collection pipeline software called Logstash is called ELK Stack, which has evolved from its original use as a full-text search engine and is now popular as a log analysis and data visualization platform.

The goal of this article

I will explain how to use Akamai DataStream 2 to deliver access logs to Elasticsearch running on Linode in near real-time and visualize the logs with Kibana. By following the steps you can create a dashboard like a screenshot below, which graphs a typical field out of the 45 fields included in DataStream 2.

I will deploy the stack to Linode, an IaaS (Infrastructure as a Service) provider acquired by Akamai in February 2022. Linode has a deployment automation feature called StackScripts, which allows you to have Elasticsearch+Kibana ready to receive access logs in about 10 minutes (Apart from configuring Elasticsearch+Kibana, activating DataStream 2 takes about 1.5 hours separately.)

See also: Linode StackScripts

This article explains how to build an Elasticsearch and Kibana environment from scratch, but if you are already running these environments and only need Elasticsearch Index Mapping, Kibana Data View, Visualization, and Dashboard definition files for Akamai DataStream 2, these are also available on GitHub for download.

Elasticsearch, Kibana definition files

Install Elasticsearch and Kibana on Linode

First, open the StackScript I have prepared for you from the following link. This StackScript will automatically install Elasticsearch and Kibana. (You must be logged into your Linode account to access the link.) If you can't open this StackScript for some reason, I have uploaded the contents of this StackScript to GitHub for you.

elasticsearch-kibana-for-akamai-datastream2
https://cloud.linode.com/stackscripts/1059555

Click "Deploy New Linode"

StackScript has a feature called UDF (User Defined Fields) that automatically creates an input form with the parameters required for deployment. This StackScript requires you to set the login credential of a non-root user who can SSH into the virtual machine, passwords for Elasticsearch and Kibana administrative users, and authentication information for DataStream 2 to feed logs to Elasticsearch during the deployment process for the virtual machine. You need to specify these parameters when deploying the machine. Enter the required parameters in the text box below. The values entered here will be used later, so keep a note of them.

Select the region where the virtual machine will be created and the type of virtual machine. Use one that has a minimum of 8 GB of memory, as Elasticsearch and Kibana will fail to launch without it. Here I select Dedicated 8 GB Linode in the Tokyo region of Japan. If you intend to use this setup to visualize the logs of a high-traffic website, you may need to choose an even higher-performance instance type. See the "Considerations for production use" section at the bottom of this article for more information.

Name the virtual machine, enter the root password, and click "Create Linode".

The screen will transition to the virtual machine management dashboard. Wait a few minutes until the virtual machine status changes from PROVISIONING to RUNNING. The IP address of the virtual machine you just created is displayed on the same screen, so take note of it.

Check and note the Reverse DNS value for the virtual machine from the Network tab of the virtual machine, as it will be needed in the DataStream 2 configuration procedure.

The virtual machine is now booted. The installation process of Elasticsearch and Kibana will proceed automatically in the background. Wait 10 minutes for the installation to complete before proceeding to the next step.

Log in to Kibana

Let's make sure you can log in to Kibana by accessing http://[IP address of the virtual machine]:5601/ from your web browser. Enter elastic as the login user and the password you specified when deploying the virtual machine.

Click the hamburger button in the upper left corner to display the menu and click Analytics -> Dashboard.

A dashboard named "Akamai" was automatically created by the StackScript, so open it.

If the dashboard appears as shown above, you have done the steps correctly so far. At this point, there is no data because DataStream 2 has not yet been set up.

Configure DataStream 2

You need to set up DataStream 2 from the Akamai Control Center as well. Click the hamburger button in the upper left corner to display the menu, then click COMMON SERVICES -> DataStream, and follow the steps below to create a stream.

Name the stream and mark the checkboxes for the delivery properties for which you want to enable log delivery via DataStream 2.

A screen will appear to select the fields in the access log to be sent, so as an example, check Include all for all categories. As of September 2022, a total of 45 fields would be selected. Also, at the bottom of the configuration screen, select JSON as the log format.

Please select the fields of the log to be collected taking into account the laws and regulations regarding the protection of PII.

Next, set the destination for DataStream 2: Elasticsearch for Destination, any name for Display name, http://[Reverse DNS hostname]:9200/_bulk for Endpoint using the Reverse DNS hostname that you noted when creating the virtual machine, datastream2 for Index name, and the Username and Password that you entered when deploying the virtual machine. Also, mark the Send compresses data checkbox and click the "Validate & Save" button in the lower right corner of the screen. If all the values are correct, you will see the message "Destination details are valid" in the lower right corner and the screen will change.

Finally, a summary of the settings will be displayed to confirm that the settings are correct. Check the "Activate stream upon saving" checkbox. It takes about 1.5 hours for DataStream 2 to begin log streaming. If you would like to receive an email notification when logs are started to be delivered, check "Receive an email once activation is complete." and enter your email address. When the activation process of DataStream 2 starts, a message will be displayed indicating that DataStream 2 settings are required also on the Property Manager. Click "Proceed to Property Manager" after noting the name of DataStream 2 that you have just set.

Enable DataStream 2 in Akamai delivery properties

This section assumes that you understand the basic operations of the Akamai Property Manager.

Delivering logs through DataStream 2 requires adding "DataStream 2" behavior on the property. After completing the DataStream 2 setup steps as described above, create a new version of the property, add the two behaviors "DataStream" and "Log Request Details" to the default rules, and configure the behaviors referring to the example configuration below. This can be done in parallel while waiting for DataStream 2 to be activated.

Stream version	DataStream 2
Stream names	Name specified during DataSteam 2 setup steps
Sampling rate	Percentage of logs to be sent (100 means all logs)
Log *** Header	Whether to log the corresponding header in the request
Cookie Mode	Whether to log cookies in the request
Include Custom Log Field	Whether to log the custom log field
Custom Log Field	Value to populate the custom log field (I include TLS Cipher Suite used as an example, but it can be left blank)
Log Akamai Edge Server IP Address	Whether to log the IP address of the edge server that processed the request (This option must be On)

Please select the fields of the log to be collected taking into account the laws and regulations regarding the protection of PII.

Once the configuration is finished, save and activate the property. Access logs will begin to appear in Kibana after both DataStream 2 and the property are activated.

Congratulations! Now you can see access logs from Akamai in near real-time!

Advanced Usage

Conditional access log delivery

Streaming all access logs to the log analysis infrastructure through DataStream 2, and then filtering the logs according to necessity on the log analysis infrastructure side is the typical usage. It is also possible to use DataStream 2 to feed only logs that meet certain conditions. This is useful especially when the amount of logs is huge and you want to reduce the load on the log analysis infrastructure, or when you are not interested in the logs of requests successfully done and only want to see the logs of errors. As an implementation example, if "DataStream" behavior is removed from the default rule of the property and enabling "DataStream" under the following conditions, logs would be sent only when the request path is under /foo/bar/ and the response code is neither 200 206 304.

Using the custom field

A field called custom field exists in the access log sent from DataStream 2. You can set any string up to 1000 bytes here, so you can include built-in variables of the edge server or property user variables.

See also: Built-in variables

In the example below, the transfer time taken from the edge server to return a response to the client is set in the custom field.

This value can be parsed as a Runtime field in Kibana's Data View to make it a new field.

Enlarge the above image

String ctt=dissect('ctt:%{ctt_val}').extract(doc["customField"].value)?.ctt_val;
if (ctt != null) emit(Integer.parseInt(ctt));

Akamai's edge computing platform, EdgeWorkers, allows you to set user variables from its JavaScript code. Custom fields can be set to contain debugging information for EdgeWorkers applications or values of interest to the business logic implemented in EdgeWorkers, which can then be aggregated in the logging analysis infrastructure. Note that as of June 2023, the custom field is limited to 1000 bytes in length.

See also: Request Object setVariable()

Debug EdgeWorkers

Although EdgeWorkers terminates the script when various limitations (CPU time, memory usage, execution time, etc.) are exceeded, you may encounter situations where it only works correctly under certain conditions that depend on the content of the request. Error statistics can be viewed from the Akamai Control Center, but the details of each request at the time of the error are not available. With DataStream 2, not only the request information when an error occurs, but also the detailed operation status of the EdgeWorkers runtime is available in the fields named ewExecutionInfo and ewUsageInfo, which can be useful for troubleshooting.

See also: DataStream 2 log details

Monitoring of Web Application Firewall activities

Since Akamai's Web Application Firewall (WAF) comes with an advanced log analysis feature called Web Security Analytics (WSA), most cyber attack analyses can be completed with WSA. On the other hand, DataStream 2 also includes a summary of the WAF's detection results in the log fields, so you can take advantage of fields not seen by WSA for supplementary analysis, or take advantage of advanced features such as machine learning that the analysis tool has. The following screenshot shows how the DataStream 2 data shows that the WAF has detected a directory traversal attack.

Visualize Common Media Client Data (CMCD)

Common Media Client Data (CMCD) is a standardized data format for sending various metrics collected by video players to servers, such as CDNs. The CTA WAVE project published the CMCD specification in September 2020.

Web Application Video Ecosystem - Common Media Client Data

Video players that support CMCD send various information to the server as HTTP request headers or query parameters. You can add CMCD into DataStream 2 logs of Akamai Adaptive Media Delivery (AMD), so you can visualize video playback quality and information with Elasticsearch + Kibana built in this article. This allows you to correlate CDN logs and quality metrics in addition to existing video QoS & QoE measurement tools. For more information on the benefits of using CMCD, please refer to the following article.

Most video players send CMCD as query parameters, but if you want to send CMCD data via request headers, you need to add CORS header settings in AMD. Please refer to the following documentation to change the CORS headers.

Common Media Client Data

On April 19, 2023, I have updated the StackScript referenced in this article. Elasticsearch + Kibana installations using the StackScript after this date will create index templates, data views, and dashboards that support CMCD.

Enlarge the above image

Considerations for production use

For the sake of simplicity, this article does not cover some considerations that should be taken into account for production use. For example, the following points at the least should be considered:

Configure Datastream - Upload Failures alert
Use a higher-performance instance type to accommodate the volume of logs
Enable HTTPS on Elasticsearch API endpoints
Make Elasticsearch nodes redundant
Allocate additional storage for virtual machines using Linode Block Storage
Design the lifecycle management of access logs

The instance type you need to use depends on the amount of logs from DataStream 2. Please refer to Benchmarking and sizing your Elasticsearch cluster for logs and metrics as a good starting point to select a proper instance type.

This article disables HTTPS for Elasticsearch to skip the SSL certificate issuance procedure. For production use, it is recommended to get an SSL server certificate issued by a certificate authority and enable HTTPS for communication between DataStream 2 and Elasticsearch. DataStream 2 does not accept self-signed certificates. You can find related information by searching with keywords such as "Elasticsearch Let's Encrypt".

Since StackScript rewrites the Elasticsearch configuration file to disable HTTPS, to make HTTPS enabled again, you need to change xpack.security.http.ssl to true in /etc/elasticsearch/elasticsearch.yml after deploying the SSL server certificate. Then change the destination endpoint of DataStream 2 from http:// to https://.

DEV Community: Hideki Okamoto

Cloud GPU instance with PyTorch and TensorFlow easy setup in 10 minutes

What is Akamai Connected Cloud (formerly Linode)?

The goal of this article

Setup a GPU instance with PyTorch and TensorFlow

Starting a container

Deleting the instance

Access control for the instance

Enabling HTTPS

Visualizing Akamai DataStream 2 logs with Elasticsearch and Kibana on Linode

Akamai DataStream 2

Elasticsearch / Kibana

The goal of this article

Install Elasticsearch and Kibana on Linode

Log in to Kibana

Configure DataStream 2

Enable DataStream 2 in Akamai delivery properties

Advanced Usage

Conditional access log delivery

Using the custom field

Debug EdgeWorkers

Monitoring of Web Application Firewall activities

Visualize Common Media Client Data (CMCD)

Considerations for production use

Appendix