DEV Community: Jiri Pik

Why You Should Never Join the AWS Community Builders Program

Jiri Pik — Sat, 07 Sep 2024 04:00:14 +0000

I was an AWS Community Builder in ML from 2021 to 2024. AWS did not extend the relationship because, according to AWS, the book “Hands-On AI Trading with Python, QuantConnect and AWS,” which I co-wrote as the lead author, does not help other AWS Builders. No matter how ridiculous the statement is, some other participants have been extended by submitting GenAI-generated unreadable articles published on websites nobody ever reads.

This short blog post explains why you should never be interested in the AWS Community Builders (CB) program.

Reason #1 – AWS CB program constitutes employment

The AWS CB program is a global AWS influencer program that operates in all countries. Its nature is obfuscated and is presented as a volunteering program to avoid immigration and employment law consequences.

Still, the program’s features constitute an employment relationship between its participants and AWS. The relationship is essentially an exchange of producing content about AWS for an annual salary of 500 USD AWS Credits (the credits do provide value and are a cash equivalent) plus performance bonuses in AWS Credits and gifts for following program managers’ instructions on what posts to write, where to publish them, and how frequently. If you do not publish, you will not get renewed.

Surprisingly enough, there is, by design, no written contract between AWS and AWS CB members to hide the nature of the program. One cannot even open an AWS account without agreeing to a written agreement.

The program’s employment nature presents unique challenges to anybody working in any country on an employment visa where additional employment is subject to prior governmental approval. If you are an AWS CB member and work on an employment visa, you may be subject to your visa being terminated and you being deported.

Reason #2 – Poor Value of the AWS CB program

The AWS CB program has two key objectives:

Produce as much content about AWS technologies as possible.
Create as much emotional attachment to AWS technologies as possible so that you prefer AWS rather than MS Azure or GCP.
While the program may have some limited value to people without AWS Certification (I have earned twelve of them), the more proficient you are, the less value it provides – one can earn more than 500 USD per month as a professional blog post writer.

In addition, the educational value of the other AWS CB members’ output and presentations is usually at a beginner’s level, while their sentimental value is at a maximum. I have never seen Slack messages with so many emoticons as in AWS CB Slack.

The AWS program offers very few opportunities managed by capricious managers, and in my view, it’s not worth your time.

Reason #3 – AWS CB program is for followers, not leaders

While AWS was ahead of other cloud providers a decade ago, cloud technology is largely a commodity nowadays.

Critical judgment and the ability to master new skills rapidly are essential skills when hiring a person. The criteria for determining the optimal platform for their use case are cost-effectiveness, reliability, clarity of vision, and security. AWS no longer meets all the requirements.

I have not seen an AWS CB blog post that critically judged AWS technologies. Instead, most posts blindly parrot AWS marketing lines without concern for cost-effectiveness or actual need. Some posts even encourage using AWS technologies ineffectively and then introduce some FinOps methodologies to magically reduce some of the resulting excessive costs. Those costs would have never occurred if the AWS architecture had been designed well.

Reason #4 – AWS CB program is a career blocker

Being associated with AWS in an influencer’s position blocks your career as you are more likely to be biased towards using AWS technologies than more optimal ones.

Summary

Careerwise, one should focus on building an extensive portfolio of original projects rather than being associated with one technology vendor.

Most Performant Docker Base Images for Data Science on AWS Batch

Jiri Pik — Sun, 17 Dec 2023 03:28:24 +0000

1. Summary

If you’re working with Python batch jobs that are heavy on mathematical computations, you might want to explore using the IntelPython Docker base image. In our experience, particularly with data science tasks that typically wrap up in about ten minutes, we’ve noticed a significant timesaving when using IntelPython over standard Python base images for jobs running on Intel CPUs. The difference is striking – we’re talking about a time reduction of three to five minutes on our 10 minutes jobs, with IntelPython consistently outperforming its counterparts.

For general Python batch jobs, we recommend using the official Python Docker base image.

2. AWS Batch

Most data heavy data science workloads are batch (i.e., not real-time) jobs and we prefer running them on AWS Batch for these reasons:

costs reasons – with AWS Fargate / AWS Spot Instances they run virtually for free.
performance reasons – AWS Batch allows to parallelize the jobs with their dependencies by launching virtually infinite number of parallel processes.

AWS Batch is based upon running Docker images on either ARM or Intel compatible CPUs. Our preference for executing data science jobs leans towards Intel CPUs. This choice is primarily driven by:

the availability of most of the libraries we rely on, which are exclusively compatible with Intel CPUs – this compatibility ensures smoother operations and seamless integration with our existing tools and workflows.
single-threaded data science code – Intel CPUs excel at running single-threaded code.

3. Typical Data Science Docker Base Images for Python

The usual Docker Base images for Python on Intel CPUs used in data science are:

Python Docker image built on Ubuntu
Official Python Docker Image
IntelPython Docker Image.

While each base image varies in size, in the context of AWS Batch, disk size becomes a secondary concern. The crucial factor to focus on is the runtime performance – this aspect is where the real efficiency and effectiveness of these images are truly evaluated.

3.1 Ubuntu Docker Image

Ubuntu image is usually preferred for local development since it contains majority of the Linux tools pre-installed. However, performance wise, it may not be the best choice.

3.2 Official Python Docker Image

There are tens of official Python Docker images, we prefer using the latest one – the cadence of Python releases with each Python version being faster than the previous one guarantees superior performance for general Python operations.

3.3 IntelPython Docker Image

While the release cadence of IntelPython cannot match the release cadence of the official Python releases, IntelPython provides superior performance for math-heavy data science jobs largely thanks to the superior Intel compilers.

4. Benchmarking

It’s important to remember that there isn’t a one-size-fits-all benchmark for evaluating Docker base images; the most reliable measure is how they perform with your specific code. I recommend experimenting with various Docker base images to see which one optimizes your script’s performance most effectively.

The heuristic rule we recommend is if your Python code does use only pandas library without heavy use of numpy, scipy or other machine learning libraries, use the latest official Python Docker base image. Otherwise, use the IntelPython Docker base image.

To illustrate this rule, I have prepared two straightforward benchmarks for this article. The first benchmark illustrates general pandas data frame operations while the second one illustrates numerically extensive pandas data frame operations.

4.1 Benchmark #1 – General pandas dataframe operations

This test focuses on simple math and merging and filtering data frames.

The benchmark.py file is as follows:

import pandas as pd
import numpy as np
import timeit

number_of_tests = 5000

def benchmark_operation01():
    df = pd.DataFrame({'A': range(1, 1000000),
                       'B': range(1, 1000000),
                       'C': range(1, 1000000)})

    for i in range(number_of_tests):
        df['D'] = df['A'] * df['B'] + df['C']

def benchmark_operation02():
    n = 10 ** 6
    df1 = pd.DataFrame({'A': np.random.random(n), 'B': np.random.random(n), 'key': range(n)})
    df2 = pd.DataFrame({'C': np.random.random(n), 'D': np.random.random(n), 'key': range(1, n+1)})
    for i in range(number_of_tests):
        df = pd.merge(df1, df2, on='key')
        df2 = df1[df1['A'] > 0.5]

if __name__=="__main__":
    start_time = timeit.default_timer()

    benchmark_operation01()
    benchmark_operation02()

    end_time = timeit.default_timer()
    execution_time = end_time - start_time

    print(f"Execution time: {execution_time} seconds")

I have run this script five times for each of the tested Docker base images and calculated the median times in seconds:

Python:531.7914 seconds
IntelPython: 560.0930 seconds
Ubuntu: 564.2625

4.2 Benchmark #2 – Math-extensive pandas dataframe operations

This test focuses on advanced math operations.

import pandas as pd
import numpy as np
import timeit

number_of_tests = 5000

def benchmark_operation01():
    n = 10 ** 6

    for i in range(number_of_tests):
        df = pd.DataFrame({'A': np.random.random(n),
                           'B': np.random.random(n)})
        df['C'] = df['A'] + df['B']
        df['D'] = df['A'] * df['B']
        df['E'] = np.sin(df['A'])
        df['F'] = np.log(df['B'])


if __name__=="__main__":
    start_time = timeit.default_timer()

    benchmark_operation01()

    end_time = timeit.default_timer()
    execution_time = end_time - start_time

    print(f"Execution time: {execution_time} seconds")

I have run this script five times for each of the tested Docker base images and calculated the median times in seconds:

IntelPython: 82.17445 seconds
Python: 96.15572 seconds
Ubuntu: 109.4516 seconds

Appendix 1 – requirements.txt

The requirements.txt is

pandas
numpy

Appendix 2 – Dockerfile for each Docker Base Image

The Dockerfile for the Ubuntu is

FROM ubuntu:latest
ENV PYTHONUNBUFFERED 1

RUN mkdir /app
WORKDIR /app

RUN apt-get update && apt-get install -y python3 python3-pip

COPY requirements.txt /app/
RUN pip install --no-cache-dir -r requirements.txt
COPY . /app/
CMD ["python3", "./benchmark.py"]

The Dockerfile for the official Python is

FROM python:latest
ENV PYTHONUNBUFFERED 1

RUN mkdir /app
WORKDIR /app

COPY requirements.txt /app/
RUN pip install --no-cache-dir -r requirements.txt
COPY . /app/

CMD ["python3", "./benchmark.py"]

The Dockerfile for the IntelPython is

FROM intelpython/intelpython3_core
ENV PYTHONUNBUFFERED 1

RUN mkdir /app
WORKDIR /app

RUN python3 -m pip install --upgrade pip

COPY requirements.txt /app/
RUN pip install --no-cache-dir -r requirements.txt
COPY . /app/

CMD ["python", "./benchmark.py"]

Lessons Learned from Having Passed all Twelve AWS Certification Exams

Jiri Pik — Sun, 07 May 2023 01:59:38 +0000

On April 24, 2023, I finished the AWS certification marathon of passing one AWS certification exam per month (“the certification marathon”).

This blog post outlines the lessons learned from this journey and expands on the previously published successful blog post On AWS Certifications.

Lesson #01 – Certification marathon is a journey

While having all AWS certification badges on your Credly account certainly does look impressive, what really matters is what you can achieve with the acquired knowledge and skills. As Ralph Waldo Emmerson said, it’s not the destination, it’s the journey.

Successful certification marathon positively changes your life:

It provides an overview of all important technologies in the AWS ecosystem available at the time of the exam and how they relate to each other and thus equips you with the confidence necessary to effectively use AWS. This broad knowledge of what is possible is priceless – while you can always look up the details, you must know what to look up first – and in my view, certification is the most effective way to obtain such broad knowledge. Furthermore, this knowledge does translate to measurable cost savings – the difference between optimal technology choice and a not-optimal technology choice can be limitless.
Certification does change the way AWS and other vendors treat you – it does supply you with utmost credibility.

Lesson #02 – How to interpret other people’s successful certification marathon

The certification does provide a guarantee of minimum knowledge in every tested area.
Successful certification does indicate superior problem-solving, time-management, and knowledge-management skills. You cannot ever pass without knowing how to identify reliable sources of information, how to deal with ambiguity and false advice (all certification preparation materials contain mistakes).
The certification becomes obsolete on the day one passes it – one needs to keep learning non-stop and having embarked on this journey indicates the person has already committed to this learning process.

Lesson #03 – The nature of the certification exams

The exams are designed to assess high-level practical knowledge of key concepts in each domain, and the related keywords.

Be able to compare & contrast the functionality, performance, pricing factors and limits of all related technologies.

Functionality Examples:

Know when to use EBS / EFS / S3 / FSx / Storage Gateway.
Be aware that AWS Lambda functions have certain limits, such as up to 10 GB RAM or runtime of up to 15 mins. Nevertheless, AWS Batch does not have any such limit, so use AWS Lambda whenever you can and if not, switch to AWS Batch.

Performance Examples:

Storing queues in RDS certainly does not provide performance of SQS
Kafka, rather than Kinesis, delivers superior performance at the cost of extra manual set up. However, for ultra-low latency streaming data, such as financial market data, even Kafka is not sufficient.

Pricing Factors Examples:

Storing millions of small files (<256 kB) in S3 would be prohibitively expensive since S3 is charged by the number of API calls. However, storing these small files in DynamoDB would be cost-effective.
Storing data used just for archiving in S3 Standard storage would be prohibitively expensive while S3 Glacier Deep Archive is one of the cheapest solutions.
You may be dismissive of potentially irrelevant exams such as “SAP on AWS – Specialty” if you have never used SAP. However, SAP is a high-performance in-memory database, and the exam covers best practices on how to implement any such database on AWS.

Lesson #04 – Optimal Preparation Strategy

Whenever there is an AWS published preparation book, read it.
Take all AWS official practice exams.
Any material published more than a year ago is of limited applicability since the exams are continuously refreshed.
Cloud Academy or Cloud Guru video lessons are useful only for Foundational or Associate level exams – professional level exams are far more about awareness of keywords associated with covered technologies. For professional level exams, nevertheless, use the Cloud Academy and Cloud Guru’s test banks / exams.
You need to activate your passive knowledge to master connections between related concepts and to calibrate your professional judgment. In my experience, the best way to do it is to answer thousands of questions in test banks and to note down problematic areas for a deep dive. Extensive use of test banks is a frequent practice for all other difficult certifications, such as the CFA certifications.
Use as many different test banks as possible since each author provides a unique perspective on the tested topics.
Practical knowledge of covered technologies is only useful if the practical knowledge is based upon the best practices. If you have used AWS wrongly, such knowledge is useless.

Lesson #05 – Optimal Test Question Answering Strategy

The questions have a single-choice answer or multiple-choice answers. Therefore, first, eliminate obviously wrong answers and use logical reasoning / intuition / general principles of cloud computing to find the correct answer.
AWS understands certain questions are ambiguous and thus the only grade is Pass / Fail with Pass grade having comfortable margin for ambiguous questions. In addition, each exam has fifteen out of seventy-five questions which are not graded and are used purely for research purposes – you will not know which ones are these, however.
AWS understands certain questions may have multiple correct answers and if the question asks for only one answer, then the criteria for correctness are 1) simplicity – architectures with one building block are preferred to architectures with multiple building blocks, 2) managed services are always preferred – the point of using AWS is to offload routine system management tasks to AWS, 3) costs – the solution which is most cost-effective is always preferred

Lesson #∞ – There is no official AWS Golden Jacket though there are unofficial ones

Many people on the Internet boast about them having received an AWS Golden Jacket for having passed all twelve AWS Certification exams.

However, as re-confirmed with AWS: the golden jacket has never been an official part of the AWS certification process. Some AWS teams have made their own for employees, and some people have made the gold jacket on their own. But there is no official path for anyone to get one.

I personally find the lack of an official token of appreciation for such a large achievement disappointing.

Top 3 Lessons from Running 300+ Cost-effective Solutions on AWS

Jiri Pik — Thu, 18 Aug 2022 00:54:56 +0000

Public cloud spend is expected to reach $400 billion in 2022 and out of that 30-40% is wasted. That is $120 billion to $160 billion – compare this figure with the expected size of the global frozen pizza market of $17.96 billion in 2022.

Note: this post focuses on AWS since I am an AWS Community Builder in Machine Learning and am focusing exclusively on AWS. The identical message applies for all other clouds.

Lesson #1 – Solve the Right Problem

The amount of obvious waste is a golden market opportunity for various solutions offering instant cloud fixes to their clients’ cloud environments with varying success, and foundations providing their believers with ready-to-apply FinOps frameworks whose introduction may cause more harm than benefits.

In fact, the key problem with these approaches is that they solve the wrong problem – the right problem to solve is how to design cloud architecture in the way which makes most of the cloud unique features. The waste is the consequence of poor cloud architecture and/or development processes and fixing symptoms rather than the real cause only exacerbates the actual problem.

There is no shortcut, no AWS Made Easy, you cannot design an optimal cloud architecture without thorough and always updated knowledge of the cloud. Moreover, public cloud is an amplifier – due to its infinite elasticity/scalability, it amplifies the consequences of any good and poor decisions.

We believe in a comprehensive approach to using AWS, compatible with the AWS Well-Architected Framework, and do not believe Cost Optimization or FinOps should drive AWS usage. FinOps is reactive, be proactive.

Recommendation: focus on upgrading your cloud architectural knowledge rather than implementing any non-actionable FinOps Framework. I have found no better alternative than AWS Certification.

Lesson #2 – The Best Technical Decisions are not Good Enough

Some firms pride themselves on all their architectural decisions being made purely based on technical reasons and ignoring any cost considerations. This is a fantasy resulting in their cost optimization team achieving cost savings of 60% or more. In addition, such a situation is usually accompanied by the “best technical decisions” not being the best since they force you to ignore some critical aspects of the problem.

An analogy would be that if you want to travel from Paris to London, you should always travel by private jet since it saves you most time, in-person meetings are always preferrable, and you enjoy traveling.

Let’s consider an example of how to store a phone number in AWS so that it can be publicly accessed?

There are an infinite number of solutions with vastly different performance, security, maintenance, and immediate and long-term costs
• Use AWS Route 53 (TXT record)
• Use AWS S3
• Use AWS EFS + AWS Lambda + AWS API Gateway
• Use AWS EBS + AWS EC2
• Use a camera connected to AWS Panorama device looking at your business card + AWS Direct Connect + AWS Textract + AWS API Gateway
• Use some satellites…

This simple example illustrates that there is no such a thing as the best technical decision taken in isolation of costs, security, maintainability and … → all architectural decisions are made in the business context. Thus, the choice of the right resource type for the job, or any architectural decision depends on these two questions:

Question 1 – Technical Decision → The list of technically best solutions
Consider the Compatibility of the Performance and Security Profile of the resource type with the expected long-term patterns and scale of use – notice long-term means the next few months since the optimal architecture changes every few months

Question 2 – Economic Decision → The actual choice
Given the answers to Question 1, factor in the total long-term cost of ownership:

Adoption, Setup, Maintenance, Migration Costs
Increased / Reduced costs by simplifying / making more complex architectural choices
Implied costs of technology lock-in

Recommendation: all architectural decisions should be based upon the cost/performance criterion.

Lesson #3 – Adopt the Cloud-Native Mindset

To make the most of the cloud opportunities, you need to adopt the cloud-native mindset which exhibits with

Cloud is elastic meaning that cloud costs should closely follow revenue.
Always choose the right resource type for the job – for example, use AWS Graviton instances for most use cases
Cloud is about rapid experimentation and innovation.
Expert human work is more expensive than any cloud cost.
You should be using managed services for most parts of the architectures even if they deliver just 90% of performance of self-managed services since they offload manual management and save time.
Most of the code of modern applications is inter-connecting existing APIs, and you should aim at minimizing your own code.
Scalability means that the resources are disposable, so you should store all data decoupled from the processing resources. Data should be stored in standard formats so that it can be accessed with other solutions.
Design your solutions with security and fault tolerance in mind.
Observability, automation, and constant benchmarking are critical for achieving optimal evolving architecture. Real-time monitoring, metrics, and logging deliver massive business value. Incorporate custom metrics into all applications to help answer questions, such as the costs per customer, costs per feature (“unit costs”). Benchmark across older version of each solution and across similar solutions to identify inefficiencies.
All business decisions are driven by forecasts. The monitoring and logging free you from making blind estimates.
You should review the architecture and re-architect the solutions every 3 months to make the most of the new services and features.

On AWS Certifications

Jiri Pik — Tue, 12 Jul 2022 08:41:03 +0000

I have always been a fan of professional certification in the areas of my interests for these reasons:

Certifications calibrate your professional judgment – they provide a prioritized top-down state-of-the-art knowledge of the field, along with the best practices
Certification is a good predictor of success in metrics-based and facts-based organization – the certification success measures your ability to get the best preparation materials available and to prioritize them, to cross-reference and master them in the shortest time possible with the knowledge that no material is perfect and may include inaccurate information or may miss some information in full, and to apply the learned concepts under exam stress.
The certifications cover the most important concepts in the field – in the AWS world, having all AWS certification is necessary a requirement for any successful AWS architect since they cover the entire AWS and for successful AWS design you need to be aware of all minute details. For example, only AWS networking certification covers the jumbo frames (MTU 9001) which are critical for any high-performance architecture.

The difference between AWS and other certifications

AWS Certifications cover actionable knowledge – all questions on AWS exams are purely real-world based – they describe a scenario and offer multiple solutions how to address, while on the other hand, the FinOps Foundation’s Certifications are the least actionable ones.
AWS Certifications do not have official textbooks for their curriculum listing all required concepts, unlike, for example, the CFA certification which is offered with the textbooks describing the entire required knowledge for each year’s exam. Instead, AWS outlines very generally the “topic areas and objectives”.
The exam questions are continuously refreshed for the latest AWS technologies meaning that any preparation kit older than 1 or 2 years has a limited applicability. Any new feature or service must be generally available (GA) for at least six months before it can appear on an exam though. Thus, the exam content is always a moving target.

AWS Certifications

AWS currently offers twelve certifications with overlapping curriculum. For example, an AWS Solutions Role (AWS SA) requires the knowledge of these certifications: SA (AWS Solutions Architect), SysOps (AWS Certified SysOps Administrator), and AWS Dev (AWS Certified Developer).

AWS offers now these twelve exams:

1. Foundational Level
    a) Cloud Practitioner
2. Associate Level
    a) Solutions Architect Associate
    b) Developer Associate
    c) SysOps Administrator Associate
3. Professional Level
    a) Solutions Architect Professional
    b) DevOps Engineer Professional
4. Specialty Level
    a) Advanced Networking Specialty
    b) Data Analytics Specialty
    c) Database Specialty
    d) Machine Learning Specialty
    e) Security Specialty
    f) SAP on AWS Specialty

Each exam has sixty-five questions with fifteen questions not being graded – they are used for further exam analytics. To pass an exam, you need to answer 75% of the graded questions correctly. While the pass rates are confidential, on one of the AWS webinars it was revealed that the pass rate for the ML specialty certifications may be far less than 5%.

The objectives of the AWS exams are

Validity – accurately and appropriately measure what is relevant
Reliability – consistent and precise
Fairness – puts no group at a disadvantage.

Notice that these objectives are different from other certifications focusing on testing abstract academic concepts.

AWS exams are designed with different cognitive levels in mind:

Recall – identify, define, list → AWS Foundational Certifications
Understand / Apply – determine, use, classify → AWS Associate Certifications
Analyze / Evaluate – troubleshoot, analyze, assess → AWS Professional and Specialty Certifications

In other words, AWS Professional and Specialty Certifications differ from the AWS Associate Certifications in the level of detail you are required to know – you should be able to compare & contrast all covered concepts and you are required to have an inquisitive mindset.

This is related to the hierarchy of knowledge:

Optimal Preparation Strategy

First, design your overall AWS certification plan – it’s best to start with the Associate level exams and then build cadence and move to the Professional and Specialty Certifications since the curriculum overlaps. You need at least a month between the exams for the new knowledge to settle in – people usually take a three-month break between the exams – it all depends on the level of your existing knowledge.

For example, one sound certification path could be taking the exams in this order:

1. AWS Certified Solutions Architect – Associate
2. AWS Certified Developer Associate
3. AWS Certified SysOps Administrator – Associate
4. Any Professional / Specialty Certification of interest
    a) AWS Certified DevOps Engineer – Professional
    b) AWS Certified Solutions Architect – Professional
    c) Specialty Certifications

Second, head to Prepare for your AWS Certification Exam | Training and Certification | AWS (amazon.com) and choose the target exam in the dropdown – the browser then loads the appropriate AWS exam page with all official AWS resources, such as practice questions.

Third, complement those resources with the complimentary AWS online exam review workshops.

Fourth, the official AWS certification books, if recent, are most helpful.

Fifth, CloudGuru and CloudAcademy cannot go deep enough for the professional / specialty exams given their limited-time videos but offer good test banks.

Sixth, try to obtain as many test banks as possible – you are looking for questions from different perspectives to activate your knowledge.

Seventh, for Specialty exams, you must read the actual AWS docs for the covered services.

AWS Golden Jacket

As a gesture of recognition, AWS gives anyone who passed all their twelve certification exams AWS Golden Jacket (see this and this).

NVIDIA Triton Spam Detection Engine of C-Suite Labs

Jiri Pik — Fri, 01 Jul 2022 08:12:47 +0000

This has been a joint project of NVIDIA and AWS Community Builder in Machine Learning program.

Unwanted and unsolicited bulk digital communication (“spam”) is responsible for substantial direct and indirect economic damage every year.

Traditional ways to identify spam by detecting certain keywords, manually reviewing text records, or even running Natural Language Processing (“NLP”) pipelines are no longer sufficient.

This article describes the architecture of the state-of-the art Spam Detection Engine of C-Suite Labs consisting of multiple inter-dependent distinct classifiers delivering real-time, high-performance superior accuracy with minimum required manual review.

This article focuses on the DistilBERT spam content model.

Application of NVIDIA Triton Inference Server for this use case provides the inference throughput 2.4 times higher than TorchScript Inference Server with 52.9 times lower model latency.

Please read the rest of the article at https://jiripik.com/2022/06/30/nvidia-triton-spam-detection-engine-of-c-suite-labs/

AWS Graviton3 Adoption in ESW Capital

Jiri Pik — Fri, 27 May 2022 16:18:37 +0000

Authors: Ermanno Attardo, Stephen Barr, Pavan Kumar TV, Jiri Pik, Ali Saidi

In this article, we would like to share ESW Capital’s experience with migrating one of our key software development products – DevSpaces – to Graviton3.

1. Graviton3

For cloud-based computation, AWS offers several EC2 instance types and sizes, with various CPU architectures, processor speeds and sizes, allowing to select for each specific workload the most suitable instance option.

EC2 offers multiple generations of x86 processors by both Intel and AMD. AWS has also been building their own chips for the data center. They started with the chips powering Nitro cards and expanded that to machine learning chips with Inferentia and Tranium, and host compute with Graviton. Graviton processors use the Arm instruction set which is also found in phones, the latest generation of Apple Macs and the fastest super-computer in the world. The second generation Graviton2 was released in Dec 2019 and now powers 12 instance types which offer a 40% overall price-performance improvement over x86-based instances.

AWS has been rapidly iterating on Graviton processors, and at the end of 2021 they announced Graviton3. According to AWS, the 3rd generation of Graviton improved general purpose performance by 25% and expands the capabilities of Graviton processors to machine learning (up to 3x higher performance), and HPC (up to 2x higher performance). Graviton processors are also more power efficient with AWS stating that Graviton3 processors use up to 60% less energy vs other instance types for similar performance.

2. ESW Capital

ESW Capital is a portfolio of over a hundred enterprise software companies.

In 2008, ESW made a strategic choice to go all-in on AWS due to their rapid pace of innovation, reliability, and cost-effective performance. Over the years, ESW has benefitted from the growth of AWS services and allowed it to transform its enterprise portfolio into modern cloud-native solutions. The release of AWS Graviton3, empowers ESW Capital to dramatically reduce costs further without any tradeoffs on performance for its compute workloads.

ESW Capital’s centralized technical product management and engineering approach which focuses on turning AWS into a core pillar across its products allows it to leverage the vast majority of AWS managed services in concerted architectures at scale.

As soon as Graviton2 instances were released to general availability, we started migrating workloads to Graviton2 for both costs and performance reasons, including Kubernetes containers, AWS Lambda functions, and all Graviton2-supported AWS managed services across its products.

Moving to Graviton on AWS OpenSearch involved a mere change of the instance type with no change on any external interfaces in most products, thus realizing immediate cost benefits.

Language choices do make a difference in migration to the Arm processors. For JavaScript/Node.js and non-node-gyp modules it was just a question of running these containers on new architecture since their underlying runtimes were already Arm compatible. For languages like Go we had to recompile the code with GOARCH as an additional build parameter (Note: we highly recommend running the GO version 1.18 since it passes arguments in the registry and not in stack improving performance by as much as 15%).

Our flagship cost-optimization product, CloudFix, used in more than 45k AWS accounts, offers automatic conversion to Graviton2 for ElasticSearch / OpenSearch, ElastiCache, RDS including RDS Aurora, and will suggest these conversions to Graviton3 once generally available.

3. DevSpaces

DevSpaces is a web-based, Visual Studio Code Experience environment built upon Gitpod with extensive team collaboration features for all major programming languages enabling rapid dockerization of your applications. It supports both x86 and Arm based development.

Once we realized that our applications were moving to Graviton, we found that we had a big gap in our development process. Most of our developers were still using X86 based development machines. We needed a development environment that was identical to the environment that the applications would be deployed in. DevSpaces for Graviton based development was created to bridge that gap.

With DevSpaces, developers can instantly set up their standardized team development environments and start working in just a few minutes. It supports code repositories in GitHub, BitBucket or GitLab, and one-click deployment to AWS ECS. DevSpaces internally uses EKS and individual dev environments as pods – each developer is allocated a single pod as his/her workspace.

Unlike vscode.dev, DevSpaces are a full desktop replacement, not just an IDE, and DevSpaces allow to define a custom dev environment with all dependent packages.

The AWS architecture of DevSpaces is as follows:

The base architecture relies on an open source solution that on connects to existing source control systems adapted for AWS:

A meta node hosts all the services required for routing requests, authentication and OS selection.
The actual Remote environments spawn in the worker nodes.
For Linux, each dev environment gets a k8s pod and for windows a dedicated VM. The pods for x86, Graviton2, Graviton3 all have 1 vCPU 2G Ram, Max burst to 5vCPU 12GB RAM. This is the maximum specification kubernetes will allocate.
A load detector proactively and dynamically predicts the optimal node count and scales in and out as needed.

4. Migration of DevSpaces to Graviton

The principal question for anybody planning to do the x86 to Graviton migration is whether the cost savings of using cheaper Graviton instances is larger than the engineering cost of executing the migration.

In general, the larger the number of x86 instance types to be migrated to Graviton, the larger the cost savings – for example, it may not be worth migrating an application running on just one EC2 instance. On the other hand, the higher the complexity of a solution, the greater the migration effort is and the lower the cost savings – for example, if your solution depends on hand coded assembly, intrinsic or libraries not frequently used, the migration costs are likely to be substantially higher than if you are migrating just a standard Python application requiring virtually no migration effort.

In the ESW Capital case, we are innovators, so, naturally, we wanted to be among the first ones to try Graviton3. We realized that the number of EC2 instances running x86 has been so substantial that the migration of most of our workloads to Graviton has financially paid off – the savings were 15% to 25% of the Intel instances costs. In addition to the obvious cost advantages, the user satisfaction of our workloads has dramatically improved – Graviton’s vCPU being a physical core compared to SMT provided isolation and avoided the common “noisy neighbor” problems of shared infrastructure. The usual complaints around random slowness at peak times were never observed in Graviton clusters.

4.1. Graviton2 Migration Journey

DevSpaces were migrated to Graviton2 as soon as it was released to general availability and once we were able to provision enough Graviton2 instances.

Our original architecture was r5.8xlarge instance type, with the target Graviton instance being r6gd.8xlarge with a setting of 1 vCPU burstable to 5 and 2G RAM burstable to 16G.

Since we were ahead of the Graviton2 adoption curve, we have encountered the following problems whose solutions we share:

1. Not all open-source components we depend on have Arm builds.

For example, the gRPC component in Node.js did not have Arm builds and still does not have. Therefore, we have ported it ourselves, and made the port open-source.

2. GitHub Actions used for build automations provide no support for managed Arm.

To overcome this limitation there are two possible solutions:

a. We could use QEMU for Arm build for C++ or Node-gyp builds – compare with this article.

b. We could use self-hosted runners to create builds.

We chose the second solution and our choice proved to be the right one – the builds are substantially faster.

3. Multi-architecture Docker Builds

When supporting multiple CPU architectures, it’s the best practice to build multi-architecture Docker images. This allows developers and production systems to pull the appropriate Docker image for the hardware they are running on with no code change.

4. Build Kubernetes (K8S) compatible Ubuntu Arm AMIs

When we migrated our workloads to Graviton2, there were no compatible Ubuntu images for K8S. We prefer Ubuntu over Amazon Linux because our solution uses the shifts kernel module responsible for uid/gid-shifting for containers. Therefore, we created custom Ubuntu AMIs to incorporate shifts changes. We also added support for contained to default docker. This was later added by AWS as an optional parameter.

5. Parallelize the code as much as possible – Graviton3’s performance shines most in a multithreaded code

4.2. Graviton3 Migration Journey

All we really needed to do was change the instance types – the AMI’s and Docker images built for Graviton2 worked without any modification.

5. DevSpaces’ Benchmarking

DevSpaces is greatly valuable for benchmarking. The single DevSpaces are enclosed in containers with hard limits on compute, memory, and storage utilization, so if we test one on Graviton and one on x86, we won’t catch system noise and correctly compare architectures.

From the DevSpaces’ users perspective, the only relevant benchmark is the time saved for every single build – pure CPU benchmarking does not measure the improvements in their overall productivity. The time saved is material since it translates into hours per week.

We have measured the build times along with the time required for running key unit tests for some of our most complex software solutions:

5.1. Sococo

Sococo Backend, a Node.js application, is an online virtual workplace environment for distributed teams.

The solution’s total build times are

about 30-35% faster on c7g than on the corresponding c6g which translates into 25-30% cheaper build costs
about 10-20% faster on c7g than on the corresponding c6i which translates into 22-30% cheaper build costs.

The cost/performance is

25-30% better on c7g compared to corresponding c6g
22-30% better on c7g compared to corresponding c6i

Conclusion: For Node.js applications we recommend using Graviton3 instances in every respect.

5.2. DevSpaces

DevSpaces, a Go application, is a software development workspace in a web browser – notice in this test we used DevSpaces to build DevSpaces!

The solution’s total build times are

about 25-35% faster on c7g than on the corresponding c6g which translates into 22-30% cheaper build costs
about 5-20% faster on c7g than on the corresponding c6i which translates into 18-35% cheaper build costs.

The cost/performance is

22-30% better on c7g compared to corresponding c6g
18-35% better on c7g compared to corresponding c6i

Conclusion: For Go applications we recommend using Graviton3 instances in every respect.

We do not recommend using largest instance types for building large Go applications since the build times exhibit an U curve caused most likely

by the way Go deals with parallel builds
by the way the large c7g instance handle Network and EBS bandwidth – the instances c7g.medium up to c7g.4xlarge have a baseline bandwidth for both network and EBS I/O throughput and can use network / EBS I/O credit mechanism to burst beyond their baseline bandwidth on the best-effort basis.

6. Lessons Learned

Upgrading from to a newer version of Graviton was as straightforward as previous upgrades we’ve done with x86. As expected, newer features and instructions are enabled with newer operating systems so you should upgrade if you can, e.g., to Ubuntu 22.04, to ensure all workflows make full use of Graviton3.
While Graviton3 dramatically improved performance of single-threaded applications over Graviton2, it shines best in heavily parallelized applications. The max level of parallelization then determines the optimal instance size – see the Go U curve described above.
Switching
-- from Graviton2 to Graviton3 delivers an instantaneous speed improvement of 20-35% while costing only about 6% more.
-- from c6i to Graviton 3 delivers an instantaneous speed improvement of 5-20% and costs 20-35% less.
Price/Performance ratios
-- for Graviton3 compared to Graviton2 are 22-30% better
-- for Graviton3 compared to c6i are 18-35% better.

7. Summary

The core value of DevSpaces is providing the best possible development environment to build and test complex software solutions that are run in AWS. We obtained the most intimate knowledge of all different compilers used by DevSpaces to route each particular language to a pod with the most optimal CPU type.

After more than four months of extensive testing, c7g instances positively shocked us by the responsiveness and performance of DevSpaces.

We are going to migrate all our production systems to c7g once it enters general availability.

AWS Insiders Podcast: AWS Cost Optimization 101 with Badri Varadarajan, EVP of product at CloudFix

Jiri Pik — Thu, 19 May 2022 06:52:21 +0000

Podcast

Episode URL
Subscribe on Apple Podcasts, Spotify or RSS

EPISODE NOTES

On this episode, Badri Varadarajan, EVP of product at CloudFix, and Rahul Subramaniam, AWS superfan and CTO of ESW Capital of CloudFix, dive deep into AWS Cost Optimization best practices for how to apply cost optimization principles when designing, configuring, and maintaining workloads in AWS Cloud environments.

"If you keep doing cost optimization, first it's good hygiene, second, you build up your cost optimization muscle. So organizationally, when it becomes a real problem, you can hit the ground running and take commensurate proportional measures, as opposed to just going from not worrying about all, to it being the only thing you worry about."

Badri Varadarajan, EVP of product, CloudFix

Time Stamps

(01:12) How Rahul and Badri got started with AWS
(07:08) Recognizing the importance of cost optimization
(17:13) Simpler ways to save and get results
(20:59) How to balance cost vs ROI
(27:32) Organizational playbooks on how to construct to cost
(36:34) Key factors to making cost optimization projects successful
(39:47) Takeaways about cost optimization

CloudFix’s AWS Data Transfer Costs in the AWS Metropolis Infographics

Jiri Pik — Mon, 02 May 2022 09:51:54 +0000

As a part of my Cloud Cost Wizard role for ESW Capital’ CloudFix, we have prepared the AWS Data Transfer Costs infographics below.

We have chosen the topic of AWS Data Transfer Costs since

the topic is considered too complex for most AWS clients and
most people are unaware of some nuances how they could dramatically reduce their Data Transfer Costs.

Feel free to share the Infographics with CloudFix being attributed as the author.

Usage+Spend Forecasting on AWS with FinOps Foundation’s Best Practices

Jiri Pik — Tue, 26 Apr 2022 08:05:37 +0000

Forecasting is the process of making predictions based on our assumptions about the factors driving the data of interest and its actual past and present values. The key value of having accurate forecasts is that they allow to be proactive, rather than reactive.

In 2020, for one in three respondents, cloud spend was projected to be over budget by between 20 percent and 40 percent. One in 12 respondents said their cloud spend was expected to be over budget by more than 40 percent.
– Pepperdata

Making wrong or no forecasts has a high cost for your Company. Many have wondered if it’s a good idea to forecast at all. In our experience, the answer is yes. While nobody owns a crystal ball, a conversative forecasting model prevents the damage from wild guesses, that, if you don’t build a good model, your team will still implicitly make irrespective on their opinion that forecasts were a bad idea or look at someone else’s model. There is no discussion about this, we saw it happen in small and large Companies.

In the AWS context, we recommend forecasting both usage and spend

accurate usage forecasts allow, for example,
to purchase the right number of reserved instances / savings plans
to accurately decide the EC2 / RDS instance type and whether to use serverless instead
to accurately set up AWS services parameters
accurate cost forecasts allow, for example,
to create realistic budgets and thus better allocate resources – such as time, money, hiring etc.
to help with the solution pricing
to determine the cost-effective technology given the expected revenue.

Read the rest of the article at jiripik.com.

FinOps Foundation’s “FinOps Certified Professional” Certification

Jiri Pik — Fri, 15 Apr 2022 10:50:03 +0000

We write to share our experience and reflections upon being the first students, and graduates (Jiri & Ermanno) of the FinOps Foundation’s FinOps Certified Professional certification program.

Significance of the Cloud Costs Problem

While only 5-15% of IT workloads have been moved from companies’ own in-house data centers to the public clouds so far, the cloud adoption grew at 25% in the last year, out of which 40% came from North America. The cloud costs are enormous and up to 32% enterprise spending on cloud infrastructure is wasted (Flexera 2022) – Why your cloud computing costs are so high – and what you can do about them – SiliconANGLE explains that

A recent survey of 350 IT and cloud decision-makers by cloud observability platform maker Virtana Inc. found that 82% said they had incurred unnecessary cloud costs, 56% lack tools to manage their spending programmatically and 86% can’t easily get a global view of all their costs when they need it. Gartner Inc. predicts that 60% of infrastructure and operations leaders will encounter public cloud cost overruns. And Flexera Software LLC’s 2020 State of the Cloud Report estimated that 30% of enterprise spending on cloud infrastructure is wasted.
As a result, “40% of cloud-based instances are at least one size too big,” estimates DoiT’s Purcell. “You’d be surprised how often workloads run at 5% to 10% utilization.
Studies show that up to 46% of data is just trash,” said Gary Lyng, chief marketing officer at Aparavi Software Corp., a distributed data management software provider.“Get rid of that first before you back it up or move it to the cloud.”
However, DoiT’s Purcell estimates that fewer than 25% of customers take advantage of cost-savings plans such as reserved instances and spot instances. “It’s like going to the grocery store; I have a pocket full of coupons, but I have to make sure they’re the right ones,” he said.

FinOps Foundation

FinOps is an evolving cloud financial management discipline and cultural practice that enables organizations to get maximum business value by helping engineering, finance, technology and business teams to collaborate on data-driven spending decisions.
–FinOps Foundation

FinOps Foundation has been established with the goals of

Collecting the body of FinOps knowledge across public clouds – this is their principal contribution – without FinOps Foundation, there is no single place covering independently the entire problem space of rapidly evolving cloud financial management – each cloud provider may discuss their cost optimization tools but certainly lacks the global picture of the industry best practices.
Promoting FinOps Best Practices and collaboration among its members – FinOps Foundation addresses these via periodic Zoom calls or conferences, and via their certification.

Please read more at Jiri's website.

Exploring Computer Vision at the Edge with Amazon Panorama

Jiri Pik — Wed, 23 Mar 2022 17:13:38 +0000

Podcast

Episode URL
Subscribe on Apple Podcasts, Spotify or RSS

EPISODE NOTES

On this episode, Omar Zarka, GM of Amazon Panorama, dives deep into the structure of Amazon Panorama, a machine learning appliance and software development kit that brings computer vision to on-premises internet protocol cameras. Omar discusses how Amazon Panorama can make accurate predictions, how to reduce operational overhead, and how to improve the experience for customers.

"One thing that customers love a lot about the Panorama experience is the ease to set up the device. When you buy a device, in five minutes you have your device set up on your network. And not only is the device software stack secured top to bottom but it's connected to our service and you have a reliable connection." Omar Zarka, GM of Amazon Panorama

Time Stamps

(02:02) How Omar got started with Amazon Panorama
(03:05) The genesis of Panorama as a service
(06:02) Edge computing compared to cloud computing
(10:24) Installation of Hardware with Amazon Panorama
(14:17) How customers capture data from multiple accounts
(20:14) Omar’s favorite use cases of Amazon Panorama
(25:13) Three best practices when using Amazon Panorama
(28:29) Best cost related practices when using Amazon Panorama