DEV Community: Christian Bonzelet

Scaling content delivery while saving costs? Making the most out of Amazon Cloudfront

Christian Bonzelet — Thu, 12 Oct 2023 12:06:30 +0000

In the domain of media and entertainment, every byte of data and every millisecond of latency counts. As professionals in this space, we're not just delivering content; we're crafting experiences. And while we strive for excellence in quality, we're also constantly on the lookout for ways to optimize costs. After all, a well-architected cloud solution isn't just about performance and scalability—it's also about financial efficiency.

Amazon CloudFront, AWS's global content delivery network is an essential service for media & entertainment, offering a global reach with its extensive network of edge locations. But with great power comes... well, costs. The good news? There are several strategies and features within CloudFront designed specifically to help you save money. This guide will walk you through the nuances of CloudFront's pricing, from the basics of the Free Tier to the intricacies of custom pricing. So, grab a coffee, settle in, and let's dive into cost savings with Amazon CloudFront.

💰 How AWS Charges for CloudFront?

When it comes to cloud services, understanding the pricing model is half the battle. And with Amazon CloudFront, it's no different. At its core, CloudFront's pricing is a reflection of the service's versatility and global reach. But, as with any service, the more you know about its pricing intricacies, the better equipped you are to make cost-effective decisions.

At a high level, AWS charges for CloudFront based on several factors. The most obvious in terms of data transfer are:

Data Transfer Out Rates: This is the cost associated with the amount of data that CloudFront delivers to your viewers. It's important to note that these rates vary depending on the geographic region of your viewers.
HTTP/HTTPS Request Rates: Every time a viewer makes a request (be it HTTP or HTTPS), there's a charge. Again, these rates differ based on the viewer's region

If you want to dive deeper into the nuances of Cloudfront pricing, please check out the official pricing page.

One of the unique aspects of CloudFront's pricing is its variability. The cost isn't just about how much data you're transferring or how many requests you're handling. It's also about where your viewers are located. AWS has divided the world into different regions, and each region has its own set of rates for both data transfer and requests.

For instance, data transfer out rates for viewers in North America or Europe might differ from rates for viewers in Asia or South America. This regional variability is something to keep in mind, especially if your media content has a global audience.

🆓 Diving into the Free Tier

Ah, the Free Tier. It's like the appetizer before the main course, giving you a taste of what's to come without the commitment. For those new to AWS or those wanting to experiment with CloudFront without immediately incurring costs, the Free Tier is a godsend.

Amazon CloudFront's Free Tier is not just a marketing gimmick; it's an offering that can provide significant value, especially when you're in the initial stages of setting up or testing your media delivery.

Here's what you get with the Free Tier:

1TB Data Transfer Out Each Month: This is a generous amount, especially for small to medium-sized projects or for those in the testing phase. It allows you to deliver content to your viewers without incurring any costs for the first 1TB each month.
10 Million HTTP/HTTPS Requests Each Month: Again, this is a substantial number. For many websites or applications in their early stages, this can cover a significant portion, if not all, of their monthly traffic.
2,000,000 CloudFront Function invocations per month
Free SSL certificates

It's essential to note that the Free Tier benefits last for 12 months from the time you create your AWS account. After this period, standard CloudFront charges apply.

The Free Tier is not just about saving money (though that's a big part of it). It's also about learning, experimenting, and iterating. It provides a risk-free environment to test your media delivery, understand CloudFront's features, and optimize your setup before scaling up.

Moreover, for startups or individual content creators in the media & entertainment space, every penny counts. The Free Tier can be a financial relief, allowing you to allocate resources to other critical areas while still delivering a top-notch viewer experience.

🌐 Price Classes: Tailoring Your Costs

In the landscape of media delivery, not all regions are created equal. Some areas might be more expensive to deliver content to, while others might be more cost-effective. CloudFront's Price Classes are designed to give you control over where your content is delivered from, allowing you to strike a balance between cost and performance.

At its core, Price Classes allow you to specify which of CloudFront's global edge locations you want your content to be served from. By default, CloudFront aims to minimize latency by delivering content from its entire global network. However, this might mean you're paying more to deliver content to certain regions where AWS's costs are higher.

Here's a breakdown:

Price Class All (Default): This uses all of CloudFront's global edge locations, ensuring the lowest latency but potentially higher costs.
Price Class 200: This excludes regions of South America, Austria, and New Zealand, offering a balance between cost and performance.
Price Class 100: This further narrows down the edge locations to only North America, Europe, and Israel, focusing on a cost-effective delivery but potentially higher latency for some users.

Choosing the right Price Class is a strategic decision. Here are some considerations:

Audience Geography: Where is the bulk of your audience located? If most of your viewers are in regions covered by Price Class 100 or 200, then opting for one of these might make sense.
Quality of Experience: Are you willing to compromise slightly on latency for certain users to save on costs? If yes, then a more restrictive Price Class might be the way to go.
Budget Constraints: If you're working with a tight budget, especially in the early stages of a project, opting for a more cost-effective Price Class can be a smart move.

Imagine you're delivering high-definition video content primarily to viewers in North America and Europe. By choosing Price Class 100, you can ensure optimal delivery to these regions while saving on costs by excluding more expensive regions. However, if you have a growing viewer base in Asia, you might need to evaluate if the cost savings outweigh the potential increase in latency for these users.

🛡️ Saving Bundles: More Than Just Cost-Saving

In the quest to optimize costs, CloudFront's Saving Bundles emerge as a compelling option. But it's not just about the savings; it's about enhancing security while keeping costs in check.

The CloudFront security savings bundle is a blend of cost-saving and enhanced security. When you opt for this bundle, you're not just committing to a consistent monthly amount; you're also getting credits for AWS WAF, a web application firewall that fortifies your CloudFront distribution against common web threats.

Here's a breakdown:

Commitment: By purchasing a savings bundle, you agree to a fixed monthly amount for CloudFront for one year. This commitment ensures you have a predictable budget, and in return, you get credits that offset your CloudFront charges.
Up to 30% Discount: The value of these credits can result in up to a 30% discount on CloudFront's standard pricing. It's like getting premium service at a discounted rate.
AWS WAF Credits: In addition to CloudFront credits, you receive credits for AWS WAF. This can offset up to 10% of the monthly CloudFront commitment, providing an added layer of security without additional costs.

Let's say your typical CloudFront charges amount to $600 per month. By committing to $420 each month for a year (a 30% reduction), CloudFront provides you with $600 worth of credits monthly. In essence, you're paying $420 for services worth $600. Plus, you get an additional $42 in AWS WAF credits. Over a year, this can lead to substantial savings.

Key points you have to consider using saving bundles:

Credits Apply Account-Wide: These credits aren't restricted to a specific distribution. They apply across all CloudFront usage in your AWS account.
Credits Cover All CloudFront Usage: Whether it's data transfer charges, request charges, or Lambda@Edge charges, the credits offset all types of CloudFront usage.
Unused Credits: Remember, credits are use-it-or-lose-it. If you don't utilize all the credits in a billing period, they don't roll over to the next.
Exceeding Credit Amount: If your usage surpasses the available credits, you'll be billed the difference at standard rates.

💼 Custom Pricing for the Heavy Hitters

In media & entertainment, scale is often the name of the game. As your content reaches a broader audience and your traffic surges, standard pricing models might not always be the most economical. That's where Custom Pricing steps in, offering tailored solutions for those with substantial data transfer needs.

Custom Pricing isn't for everyone. It's designed for users who are ready to commit to a minimum of 10 TB of data transfer per month for at least 12 months. If you fit this bill, here's what's in store:

Tailored Discounts: The discounts vary based on the volume of your commitment. The more you commit, the better the rates you can secure.
Organization-Wide Application: If you're managing multiple AWS accounts within an organization, the custom pricing applies across the board. This ensures consistent savings, irrespective of which account is handling the traffic.

Why you should consider custom pricing?

Predictability: With custom pricing, you're entering a commitment. This means you have a clear forecast of your costs, allowing for better budgeting and financial planning.
Economies of Scale: As your traffic grows, the per-unit cost of delivery can decrease with custom pricing, ensuring that your success doesn't lead to disproportionate costs.
Negotiation Power: Custom pricing discussions with AWS give you a platform to negotiate terms based on your specific needs and projected growth.

If you believe Custom Pricing is right for you, the first step is to reach out to AWS. The process involves discussions to understand your requirements, after which AWS provides a tailored pricing proposal. Remember, the emphasis here is on partnership. AWS understands the challenges of delivering high-quality media content at scale and is often willing to work closely with users to find the best pricing solution.

🚀 Conclusion

Navigating AWS pricing can sometimes feel like charting a course through uncharted waters. But with the right knowledge and tools at your disposal, you can ensure smooth sailing. Amazon CloudFront, with its cost-saving options, offers media & entertainment professionals to deliver top-notch content without breaking the bank.

From the Free Tier's generous offerings for newcomers to the tailored solutions of Custom Pricing for the big players, there's a cost-saving strategy for everyone. With features like Price Classes and Saving Bundles, you have the flexibility to fine-tune your expenses based on your specific needs and audience demographics.

But remember, while cost-saving is essential, it's just one piece of the puzzle. The ultimate goal is to deliver exceptional experiences to your audience, and CloudFront provides the tools to achieve that without compromising on quality.

As you continue your journey in the cloud, always keep an eye out for new features and pricing options. AWS is continually evolving, and there might be new opportunities around the corner to optimize both performance and costs. And for those intricate details or when in doubt, the official AWS documentation is an invaluable resource.

Here's to building in the cloud, crafting exceptional viewer experiences, and making every penny count!

AWS re:Invent uncovered - TOP TIPS FOR FIRST-TIME ATTENDEES

Christian Bonzelet — Fri, 22 Sep 2023 08:51:50 +0000

Ah, AWS re:Invent! It's not just another tech conference; it's THE tech conference. A sprawling, bustling hub of innovation, learning, and networking, re:Invent is where the AWS world converges every year. If this is your maiden voyage into the vast sea of re:Invent, you might be feeling a mix of excitement and, let's admit it, a tad bit of overwhelm. But fear not! Just as every seasoned traveler once took their first step, every re:Invent pro was once in your shoes (hopefully, comfortable ones, but we'll get to that).

The sheer scale and depth of re:Invent can be daunting. With countless sessions, expos, and networking events spread across multiple venues, it's easy to feel like a kid in a candy store, eyes wide, not knowing where to start. But here's the good news: with a bit of preparation and some insider tips, you can navigate this colossal event like a pro, soaking in the knowledge, making meaningful connections, and yes, snagging some cool swag along the way.

So let's uncover the secrets of AWS re:Invent together and set you on a path to make the most of this incredible experience.

🗓️ Planning is everything

The early bird doesn't just get the worm; it gets the best sessions, the optimal seats, and a smoother overall experience. Think of re:Invent as a vast amusement park. Without a map and a plan, you might end up wandering aimlessly, missing out on the best rides.

Start Early with the Session Catalog: Weeks before the event kicks off, AWS releases a detailed session catalog. This is your treasure map. Dive into it, explore the myriad sessions, and mark your favorites. Whether you're into deep technical dives, visionary keynotes, or hands-on workshops, there's something for everyone. But remember, the most sought-after sessions fill up fast. So, once the session reservation window opens, be swift to secure your spot.
The Walk-In Strategy: Missed out on reserving a seat for that session you were eyeing? Don't fret. Every session has walk-in lines. But here's the catch: they can get long, especially for popular sessions. If you're determined not to miss out, ensure you allocate enough time to queue up. Arriving early can make the difference between being part of the action inside or hearing about it later.

👟 The Importance of Comfort

Imagine this: It's day three of re:Invent. You've attended back-to-back sessions, explored the expo, and networked like a champ. But there's a nagging pain in your feet, and every step feels like a marathon. Don't let this be you!

Choose Your Footwear Wisely: AWS re:Invent is colossal. And when I say colossal, I mean it. The venues are expansive, and the events are spread out. You'll be walking—a lot. While those stylish shoes might look fantastic, they might not be your best friend by the end of the day. Opt for comfort over style. Trust me, your feet will thank you.

🏨 Hop-on/Hop-off is good for sightseeing not for re:Invent

Las Vegas is a city of grandeur, and the venues for re:Invent are no exception. They're vast, they're opulent, and they're... well, quite far from each other. Navigating between them can be a bit of a trek, especially if you're hopping from one hotel to another for different sessions.

Avoid the Hotel Hop: While it might be tempting to jump between hotels to catch various sessions, it's a time-consuming endeavor. The distances between hotels can be deceptive, and even with the shuttle services, you might find yourself spending more time in transit than you'd like. Check the session catalog: many popular sessions are repeated at different venues throughout the week. If you can, try to cluster your sessions by location each day. It'll save you time and energy.
Shuttle Services Are Your Friend: If you do need to switch venues, make the most of the shuttle services provided. They're efficient, regular, and a great way to move between locations without the hassle of navigating Vegas traffic.

⏰ Keep Your Schedule Flexible

While it's essential to plan, it's equally crucial to leave some room for spontaneity. AWS re:Invent isn't just about the sessions; it's about the community, the unexpected conversations, and those serendipitous moments that can spark new ideas or friendships. Seasoned attendees call this “the magic of the hallway track”.

Embrace Community and Networking Events: Beyond the official sessions, re:Invent is teeming with community-led events, meetups, and networking opportunities. These are goldmines for making connections, sharing experiences, and even having a bit of fun. Keep an eye on the re:Invent page and session catalog for these events. Whether it's a casual meetup at a local bar or a more formal networking dinner, these events can be some of the most rewarding parts of your re:Invent experience.
Take a Breath: It's easy to get caught up in the whirlwind of back-to-back sessions and events. But remember, it's okay to take a step back. Schedule some downtime. Whether it's a leisurely coffee break, a stroll around the venue, or just some quiet time to process what you've learned, these moments can be incredibly refreshing and give you the energy to dive back in with renewed vigor.

🎤 Decipher Session Types

AWS re:Invent is a smorgasbord of learning opportunities, and not all sessions are created equal. Understanding the different session types can help you tailor your experience to your learning style and objectives.

Beyond the Breakout: While breakout sessions are informative, they're often recorded and available for viewing post-event on platforms like YouTube. If you're looking for a more interactive experience, consider other formats.
Chalk Talks & Builder Sessions: These are smaller, more intimate settings where AWS experts dive deep into specific topics. The beauty of these sessions? They're interactive. You can ask questions, engage in discussions, and get feedback on your specific challenges.
Code Talks - The New Kid on the Block: Making its debut this year, Code Talks promises to be an exciting addition. Tailored for developers, these sessions are all about diving deep into code, exploring best practices, and getting hands-on with AWS services.

Remember, the goal is to maximize your learning. Choose sessions that align with your interests, offer interactive opportunities, and provide value beyond the event itself.

📊 Pick The Right Session Levels

AWS re:Invent caters to a diverse audience, from cloud novices to seasoned experts. The sessions are categorized into different levels to help attendees choose the right fit for their expertise and interests.

Know Your Levels:

Level 100: These are introductory sessions, providing a broad overview of a topic. Ideal for those new to AWS or a specific service.
Level 200: Intermediate sessions that delve a bit deeper, offering a more detailed look at specific AWS services or solutions.
Level 300: Advanced sessions, perfect for those with a good grasp of AWS. They dive deep into specific topics, often involving complex architectures and solutions.
Level 400: The deep dive. These are for the pros, covering intricate details, best practices, and advanced architectures.

Tailor Your Experience: If you're like me, with multiple production workloads already humming on AWS, you might find levels 100 and 200 less enlightening. They're great for building foundational knowledge, but if you're looking for advanced insights, aim for levels 300 and 400. However, if a topic is entirely new to you, don't shy away from starting at a lower level to build a solid understanding.

The key is to strike a balance. Mix and match session levels based on your familiarity with the topics and where you want to deepen your knowledge.

⚖️ The Art of Balancing Time

AWS re:Invent is not just a conference; it's an experience. And like any grand experience, it's essential to find a rhythm that allows you to soak in the knowledge, connect with peers, and also take moments for yourself.

Pause and Reflect: With so much information coming your way, it's easy to feel overwhelmed. Schedule pockets of time to pause, reflect, and process what you've learned. Whether it's jotting down notes, discussing with peers, or simply taking a quiet moment to think, these breaks can enhance your understanding and retention.

Listen to Your Body: It's a marathon, not a sprint. While the excitement can keep you going, it's essential to listen to your body. If you're feeling drained, it's okay to skip a session or take a longer break. Remember, the goal is to leave re:Invent enriched, not exhausted.

🥤 Nutrition and Hydration

Amidst the whirlwind of sessions, keynotes, and networking, it's easy to overlook the basics: eating well and staying hydrated. But remember, re:Invent is a marathon, and you'll need to fuel your body and mind to keep going.

Stay Energized: With the adrenaline rush of the event, you might be tempted to skip meals or grab quick, less-nutritious options. But to stay sharp and attentive, it's crucial to nourish yourself with balanced meals. Thankfully, re:Invent offers a variety of food options. Whether you're grabbing a lunch box from the catering or exploring local eateries, prioritize meals that give you sustained energy.

Hydration is Key: Las Vegas can be dry, and with all the walking and talking, you'll need to stay hydrated. Don't rely solely on caffeine (tempting, I know). Make use of the numerous water dispensers scattered throughout the venue. Pro tip: During your re:Invent check-in, you'll receive a refillable bottle. Keep it handy and refill it regularly. It's an eco-friendly way to ensure you're always hydrated.

Take Breaks: Amidst sessions, find spots to relax and enjoy a snack or a drink. One of my favorite places is outside the Caesars Forum. It's a great spot to catch some fresh air, bask in the sun, and recharge before diving back into the action.

🎪 Explore the Expo

The expo at re:Invent is a microcosm of the cloud computing universe. With a plethora of vendors showcasing cutting-edge products, solutions, and innovations, it's a must-visit for every attendee.

Dive into the AWS Ecosystem: The expo is not just about flashy booths and swag (though there's plenty of that!). It's a chance to dive deep into the AWS ecosystem, explore new tools, and discover solutions that can elevate your cloud game.

Engage with AWS Experts: The AWS Village is a gem within the expo. Here, you'll find dedicated booths for almost every AWS service or group of services. It's a golden opportunity to engage with AWS experts, ask questions, clarify doubts, and even provide feedback or feature requests.

Connect with the Community: Don't miss out on the AWS Heroes Lounge and the AWS Community Lounge. These spaces are buzzing with energy, offering a chance to interact with AWS Heroes, join DevChat sessions, and immerse yourself in the vibrant AWS community.

🎁 The Allure of re:Invent Swag

Ah, swag! It's one of those delightful perks of attending tech conferences, and re:Invent takes it to a whole new level. From quirky t-shirts to innovative gadgets, there's a treasure trove of goodies waiting for you.

Be sustainable: While it's tempting to grab everything in sight, be selective. Think about what you'll genuinely use or cherish. Some attendees even bring an extra bag just for swag, but remember, quality over quantity. Be sustainable!

Eco-Friendly Choices: Many vendors are now offering sustainable swag options. Whether it's a reusable water bottle, eco-friendly tote bags, or bamboo tech accessories, make choices that are kind to our planet.

Share the Love: If you end up with swag that's not quite your style or duplicates, consider sharing with colleagues, friends, or even donating. It's a great way to spread the re:Invent spirit beyond the event.

🚀 Final words

As we draw this guide to a close, I hope these tips offer you a clearer path to navigate the vast landscape of AWS re:Invent. Remember, while the sessions, keynotes, and workshops are invaluable, it's the connections you make, the conversations you have, and the experiences you gather that truly define your re:Invent journey.

If you haven't already, now's the time to take the plunge. Register for AWS re:Invent, mark your calendar, and gear up for an unforgettable cloud adventure. Whether you're there to deepen your technical knowledge, network with like-minded professionals, or simply soak in the vibrant atmosphere, re:Invent promises a transformative experience.

Ready to dive into the world's biggest tech conference? Register for AWS re:Invent now and set yourself up for a week of learning, networking, and inspiration.

Engaging football fans with mobile push notifications

Christian Bonzelet — Wed, 04 Jan 2023 09:30:00 +0000

One of the key capabilities of (mobile) sports applications is, to inform fans about what is happening during the game. This includes for example notifications about important events like kick-offs, goals or cards. The majority of mobile sports applications use push notifications as the primary channel to keep their fans up to date. Depending on the application platforms, notifications are sent directly to the devices with either the Apple Push notification service (APNS) or the Google Cloud Messaging (GCM) service.

In AWS you have multiple options to send native push notifications to your mobile applications - Amazon SNS or Amazon Pinpoint. But which one should you use? In this article, I will describe why I prefer to use Amazon Pinpoint to solve common marketing and engagement challenges and how this service compares to Amazon SNS.

👀 Fan engagement has multiple perspectives

When we talk about fan engagement and all the engagement-related challenges of marketers these days, we quickly come to the point that solving these challenges goes beyond just sending a notification or message.

Companies that are able to segment their audience put themselves in the position to create and communicate specific targeted marketing messages that align with the interests and emotions of specific customer groups. According to the "Next in Personalization 2021" report, 72% of consumers expect brands to demonstrate they know them on a personal level. Trust defines customer engagement. Personalization is not just a recommendation engine. Personalization is relevant for a variety of domains. It is a commitment to streamlining your activities according to many customer demands.

And the future of fan engagement is omnichannel. Targeting the right fans, at the right time using the right - often - multiple channels. This requires as always the right people, the right processes and the right technology.

Investments in omnichannel are improving, but still have a long way to go. [...]

Only 35% of companies feel, they are sucessfully achieving omnichannel personalization, up from 24% in 2021.

Source: https://segment.com/state-of-personalization-report/

Leveraging notification channels for sports applications can be seen from two perspectives. Those perspectives each come with different technical, functional and non-functional requirements. They also put different kinds of KPIs in the focus of decision-making and success evaluation.

⚽️ The first perspective: the matchday

Sports in general and football matches in particular can write awesome stories. And it is not surprising that the things that happen on the pitch are usually the main driver of engagement. There is not much to add if your favorite club strikes a leading goal in your local derby match. Or if your favorite player gets substituted in and takes your club to finish a match with a draw or even a win after being behind the whole match. In this case, you might not necessarily look at the opening rates of our match-event-focused push notifications. Those notifications can be seen as an additional layer of engagement. Encouraging fans to open those notifications depends on so many factors. Opening notifications gets unlikely if fans watch a match live. It is getting more likely if they don't have access to watch a match live for any reason. In both cases, you gain a lot of additional opportunities depending on your fan behavior.

1. 📺 𝗙𝗼𝗿 𝗳𝗮𝗻𝘀 𝘁𝗵𝗮𝘁 𝘄𝗮𝘁𝗰𝗵 𝗮 𝗺𝗮𝘁𝗰𝗵 𝗹𝗶𝘃𝗲, those notifications can provide and engage with additional or "unseen" information. If a fan watches a match live, the information that a goal happened is already transported via the big screen. Storytelling can be extended to further increase engagement by further mixing our real-time sports data with such marketing channels.

2. 📲 𝗙𝗼𝗿 𝗳𝗮𝗻𝘀 𝘁𝗵𝗮𝘁 𝗮𝗿𝗲 𝗡𝗢𝗧 𝗮𝗯𝗹𝗲 𝘁𝗼 𝘄𝗮𝘁𝗰𝗵 𝗮 𝗺𝗮𝘁𝗰𝗵 𝗹𝗶𝘃𝗲, latency can be an important USP of your product. We want our fans to cheer first. To be the king or queen in a group of people. Imagine you get the notification first, that your favorite team won the championship. This will be the ultimate hugging guarantee from the Bundesliga. Try it out!

This means: match-related notifications are highly contextual. Combined with the fact that sending out notifications about sports events can produce a lot of notifications in the timeframe of a match, you have to think about strategies to prevent fan churn. Think about segmentation and what kind of fans you want to target and provide value. Otherwise, keep in mind that your fans might simply ignore your notifications (and the effort you spent in sending them out) or leave your platform and deinstall your application for several reasons.

Segmentation increases the likelihood that customers will engage with the brand, and reduces the potential for communications fatigue — that is, the disengagement of customers who feel like they’re receiving too many messages that don’t apply to them.

Source: Target your customers with ML based on their interest in a product or product attribute

😴 The second perspective: between the matchdays

Or how I call it: the fan wakeup-call. Extending the matchday experience between matches can be a very important task to keep your level of engagement and retention stable. It is a typical pattern to have high engagement on matchdays, followed by a drop of your engagement-related KPIs once a matchday is over. Several marketing strategies and campaigns help you to extend the so-called matchday experience. Either post-match by sending out notifications about highlight clips, interviews or match reports. Or pre-match by engaging our fans with potential line-ups, injured players or relevant background information about the upcoming matches. In this case, more classic metrics like open rates or session metrics are very valuable KPIs to measure engagement and success.

🏟️ From pitch to push notification

What happens behind the scenes when a goal was shot and you want to use this as a trigger to send a push notification? Let us zoom out a bit and let us take a closer look at a real-life example from the professional football league in Germany: the Bundesliga.

The DFL subsidiary company - Sportec Solutions AG - is the official data provider for all data around Bundesliga and Bundesliga 2 football matches. From here we get all the events like goals, substitutions, cards, and fouls in real-time. This enables the Bundesliga, to build those great digital products like the official Bundesliga App.

When a player shoots a goal, this information is pushed to a match data processing service. The main responsibility of this service is to receive and process all events that occur during a match and decide how to act on those events.

One example of how to act on events like goals, cards or kickoffs is to send out a push notification using Amazon Pinpoint. This engages especially those fans who are not actively using the Bundesliga Apps. Those events are not simply broadcasted to all fans. The relevant target segments are selected based on the type of the event, the associated match and the interest of fans receiving specific events. This will result in more specific targeting and sending out notifications to fans that have an explicitly defined interest in receiving this notification. Everything is fully automated.

Defining the right level of segmentation is an important success factor in your engagement and marketing story. This can result in building several layers of segments that allow you to target our fans using very specific characteristics like

⚽️ a match,
🗓 a matchday,
🐐 a club or
🏟 individual events.

Amazon Pinpoint gives you the capabilities to create and the flexibility to adapt your segmentations at any given time.

If you want to know more about the official Bundesliga match data, I can highly recommend the following video which explains the whole process in detail.

🔔 Why you should use Amazon Pinpoint?

Generally speaking, you can achieve to send push notifications in two recommended ways - either using Amazon SNS or Amazon Pinpoint. Although both services have similar capabilities - like sending push notifications, E-Mails or SMS - they have different intentions. Yes, we can say they are united in the cloud but divided by purpose.

📢 You will find Amazon SNS - the Simple Notification Service - in the category of “Application Integration”. From my perspective, the main purpose is more related to pure technical use cases. Often used when you need to implement messaging scenarios, event-driven-architectures or want to decouple components and systems. In a nutshell, Amazon SNS is the implementation of a Publish-Subscribe pattern and gives you not a rich feature set for solving marketing-related challenges. In any case, please be aware of the limits and quotas. Amazon SNS has several hard limits you have to keep in mind that influence your integration and architecture. Especially when it comes to Topic subscriptions and unsubscriptions.

🎯 Amazon Pinpoint on the other side is named a “Multichannel Marketing Communication Service” and is located in the “Business Applications” category. It is NOT just about sending messages over a given channel. It is about building business use cases for marketing and engagement over multiple channels. Hence Amazon Pinpoint provides a lot more features and capabilities than Amazon SNS. It is hard to describe the central implemented pattern. Viewed from the outside, the centerpiece of Amazon Pinpoint is the implementation of a Recipient List combined with a Process Manager. Looking more detailed, it is a composition of integration patterns to engage with your audience and solve a broad range of marketing business problems.

You can achieve the same, but you have to take a look from different perspectives to find the right service for the right job.

Wrap up

Both services - Amazon SNS and Amazon Pinpoint - are capable of sending push notifications. But they have different intentions, tradeoffs and strengths. You can start with Amazon SNS if you just want to send out a (transactional) message. As soon as you move into more advanced marketing and engagement scenarios and have the requirement of segmentation and personalized messages, you will hit limitations using Amazon SNS. How do you measure the impact of your notifications? How do you know that your messages targeted the right fans? Keep in mind that using Amazon SNS will force you to build a lot of custom stuff around the pure notification part.

With Amazon Pinpoint you will make the shift from a pure message-focused approach to a real marketing and engagement-focused approach. Amazon Pinpoint enables you to build omnichannel customer experiences that go beyond just pure messaging. The services give you features to define your audience, create dynamic segments, and target your audience while giving you options to measure the impact of your marketing efforts. This enables you to put your fans at the center of the engagement and not just the raw message. It also enables marketers options to create segments based on recent trends.

And what about measuring your KPIs? Amazon Pinpoint comes with a whole analytics integration and provides features out of the box to analyze your campaign performance and important engagement-related KPIs. Have you ever tried this with Amazon SNS? I can tell you: it won't scale and you won't close the omnichannel gap!

You are interested to know more about how we at Bundesliga are able to send out a goal notification to a hundred thousand fans in real time during a match? Reach out to me and I am happy to present you bits, bytes and insights about our journey leveraging Amazon Pinpoint for fan engagement.

About the author:

👋 Hi my name is Christian. I am working as an AWS Solution Architect at DFL Digital Sports GmbH. Based in cologne with my beloved wife and two kids. I am interested in all things around ☁️ (cloud), 👨‍💻 (tech) and 🧠 (AI/ML).

With 10+ years of experience in several roles, I have a lot to talk about and love to share my experiences. I worked as a software developer in several companies in the media and entertainment business, as well as a solution engineer in a consulting company.

I love those challenges to provide high scalable systems for millions of users. And I love to collaborate with lots of people to design systems in front of a whiteboard.

You can find me on LinkedIn or read my blog

AWS Step Function vs. AWS Lambda - Part 2

Christian Bonzelet — Thu, 25 Nov 2021 13:49:46 +0000

Do you also feel the tension of the cover image? It is time for a battle again. 🥊 After I published the first part of my comparison, I was overwhelmed about the amount of feedback I received. May it be comments on my post, discussions on twitter or LinkedIn.

The fact that the initial post triggered a lot of inspiring discussions is very valuable. While reading through your feedback it was kind of obvious that there is a need for a second part.

I received a lot of feedback about optimizations for AWS Lambda and that people are curious how this affects the performance in comparison to our state machine. We will also take a closer look on the perspective of costs to get a more complete view how the services differ.

Here we are.

Like in our first part, again all experiments are triggered using Apache Bench with the following parameters.

ab -n 15000 -c 1 https://hash.execute-api.eu-central-1.amazonaws.com/.../

-n configures the total amount of requests that are triggered - in our case 15.000
-c is the number of concurrent requests - in our setup 1

⚠️ IMPORTANT: it is important to consider, that the results from apache-bench are not 100% accurate. The measured throughput depends on the hardware and network capabilities of my local workstation. For upcoming benchmarks, I consider to use something like CloudShell.
But apache-bench gives some very early feedback and potential indications. Hence we use these results in combination with the Lambda duration and Step-Function execution duration.

🔋 Optimizing our Lambda function

So what is the goal of our upcoming experiments? We want to
apply some optimizations on our Lambda function with a clear focus to decrease latencies. Based on the feedback I got, there were two main approaches for optimization:

Reusing downstream http connections by activating keep-alive settings.
Improving overall execution performance by increasing the allocated memory.

Reusing Connections with Keep-Alive in Node.js

For short-lived operations, such as in our case writing and reading to and from S3, the latency overhead of setting up a TCP connection might be greater than the operation itself. To activate http keep-alive you simply have to set an environment variable in your Lambda function configuration.

Environment:
  Variables:
    AWS_NODEJS_CONNECTION_REUSE_ENABLED: 1

In case you already use v3 of the AWS JS SDK, this setting is enabled by default. For v2 you have to explicitly activate it.

Let us deploy the change and start our first test. Let us first start with analyzing the Apache Bench reports. The complete reporting is available on GitHub. Here some highlights:

The Lambda function was able to process all requests 43 seconds faster compared to the state machine.
Both the state machine and the Lambda function were able to process round about 7 requests per second
The mean time per request for the Lambda function was 131ms and 134ms for state machine.

Looking at these results, this little tweak of activating TCP keep-alive helped a lot to speed up the Lambda function. In terms of end-2-end performance and latency, both solutions are now very close to each other.

Let us take a closer look into CloudWatch and X-Ray to confirm the observations.

The average execution time of the state-machine is 46.4ms and Lambda performs with 49ms.

Here things are still looking interesting. The Lambda function duration on average still has some up and downs during the execution of the test while the duration of the state-machine is stable. Both solutions show some cold-start behavior while it seems that the state-machine needs less time to become "warm".

But in total the impact on the Lambda function performance is very impressive compared to the results in the first part.

Give the Lambda function some RAM

But the question is: how much memory does my Lambda function need? The range is quite large from 128 MB to 10.240 MB.
There is an awesome open source tool called "Lambda Power Tuner" that helps you to determine your memory settings based on different strategies like speed, cost or balanced.

If you use "cost" the state machine will suggest the cheapest option (disregarding its performance), while if you use "speed" the state machine will suggest the fastest option (disregarding its cost). When using "balanced" the state machine will choose a compromise between "cost" and "speed"

Source: Lambda Power Tuner @ AWS Serverless Application Repository

In my case the "Lambda Power Tuner" suggested 256 MB as "Best cost" and 2048 MB as "Best Time".

Awesome, now we have a good start for the final tests.

Best time setting

As we aim to reduce latency, let us first start with the proposed "Best Time" setting of 2048 MB memory and let us have a look at the apache-bench metrics:

The Lambda function was able to process all requests 81 seconds faster compared to the state machine.
Both the state machine and the Lambda function were able to process round about 8 requests per second
The mean time per request for the Lambda function was 121ms and 127ms for state machine.

Compared to our first test, there are some improvement but they seem to be marginally on average. Let us try to get some more insights using CloudWatch and X-Ray.

For the most parts, the duration of the Lambda function is just below the execution time of the state-machine.
The average execution time of the state-machine is 45.1ms and Lambda shines with 41.8ms.

What would happen, if we set our memory configuration to the setting considered as "Best cost"? Let us review the results in the next chapter.

Best cost setting

In short again our apache-bench metrics:

The Lambda function was able to process all requests 155 seconds faster compared to the state machine.
The state machine was able to process 7.5 requests per second while the Lambda function processes 8 requests per second
The mean time per request for the Lambda function was 122ms and 132ms for state machine.

CloudWatch and X-Ray results also confirm very close results.

The average execution time of the state-machine is 54.8ms and Lambda is just in the lead with 50.5ms.

💰 Cost comparison

Based on the scale of my test, the AWS Cost Explorer was not really helpful as the load I generated was too low. The AWS calculator is a helpful tool to better compare the costs of both services.

The estimate is publicly available if you want to have a detailed look.

I calculated with 5 million invocations per month per service. Based on our test results, I was able to determine very precise values for the parameter that influence pricing like Lambda invocation duration/state-machine execution or consumed memory. The monthly costs are:

8 USD for AWS Lambda with 2048MB memory (Best time)
1.83 USD for AWS Lambda with 265MB memory (Best cost)
5.52 USD for the AWS Step Function express workflow

💡 Conclusion

In this part we covered some important aspects like options to improve the performance of a Lambda function. I think it is again very important to mention, that this benchmark should not be interpreted as "use Step Functions whenever you can".

My goal was more to raise discussions about the importance to not build you decision based on hypothesis or rumors. Make your decision based on data to make the best of all kind of decisions you can make.

I would again like to point out a quote from Eric Johnson at the serverless office hours:

Use Lambda to transform not to transport

Or in my words: the best code is the code that is never written.

☝️ And here come the thing and this is very important to keep in mind:

BOTH SERVICES ARE AWESOME.

If you have the need to write a Lambda function, you will be able to solve a lot of problems. But depending on what you want to achieve, Step Functions give you a lot of power to get the same results without writing ANY line of code, while making up your mind about things like TCP keep-alive or how to figure out what the best memory setting is. In all tests, AWS Lambda showed the well-known cold-start behavior that is something you should keep in mind. AWS Step Function also needs some warm-up time but it is not really comparable to AWS Lambda cold-starts. There was an interesting discussion around this on twitter:

// Detect dark theme var iframe = document.getElementById('tweet-1453733187666857985-364'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1453733187666857985&theme=dark" }

It only remains to say: happy coding AND happy orchestrating! 🥳 I really hope that my analysis and the approach to decision-making helps you in deciding towards or against one of these services for your individual use cases.

About the author:

I love those challenges to provide high scalable systems for millions of users. And I love to collaborate with lots of people to design systems in front of a whiteboard.

You can find me on LinkedIn or Twitter

Cover Image by miguelangelvenegasgordillo on Tenor

AWS Step function vs. AWS Lambda benchmark

Christian Bonzelet — Wed, 27 Oct 2021 10:58:34 +0000

Looking into the AWS ecosystem of serverless services, AWS Step Functions is one of my personal most favorite services. I recently had a chat with some colleagues about a potential use case of Step functions in favor of AWS Lambda. While we discussed the general concept of AWS Step Functions, one of my beloved colleagues argued towards the usage of AWS Lambda like

Let us use AWS Lambda because a workflow described as a state machine sounds like it is much slower.

I could neither substantiate this statement nor could I contradict it. So I started to examine the original assumption "Step Functions is slower than Lambda" with facts. Time for a benchmark!

For me the results were crystal clear 😆

Just kidding! Let us first get a common understanding what AWS Step Functions and AWS Lambda is. If you are familiar with these services, you can jump right into the section about the test setup and results.

By the way: the source code is also available for you on Github.

🤹 What is AWS Step Functions?

AWS Step Functions was published in 2016 as a serverless orchestration service. I think the following definition of AWS Step Functions explains very well, what kind of problems AWS Step Functions solves:

Step Functions is a serverless orchestration service that lets you combine […] AWS services to build business-critical applications. Through Step Functions’ graphical console, you see your application’s workflow as a series of event-driven steps.

Step Functions is based on state machines and tasks. A state machine is a workflow. A task is a state in a workflow that represents a single unit of work that another AWS service performs. Each step in a workflow is a state.
Source: What is AWS Step Functions? - AWS Step Functions

State machines can be invoked both asynchronously and synchronously. Step Functions itself offers several ways to invoke you state machine, for example:

via an explicit StartExecution call using your favourite AWS SDK,
on each http request hitting your AWS API Gateway,
as a destination in your Amazon EventBridge event bus

Typical use cases for AWS Step Functions cover data processing, machine learning, microservices orchestration or governance and security automation. Since the launch of the AWS SDK service integrations, you can use out of the box working integrations with every service that is supported by the AWS SDK. This offers you a huge number of new opportunities to integrate with AWS services without writing a single line of code.

While creating a new state machine you can decide between two execution types named “Standard” or “Express”. Each type has several characteristics and strengths. While standard workflows are a good fit for long-running workflows, Express workflows are a good fit for high-traffic workloads, data streaming or mobile application backends.

⚡️ What is AWS Lambda?

Lambda is a compute service that lets you run code without provisioning or managing servers. Lambda runs your code on a high-availability compute infrastructure and performs all of the administration of the compute resources, including server and operating system maintenance, capacity provisioning and automatic scaling, code monitoring and logging. With Lambda, you can run code for virtually any type of application or backend service.
Source: https://docs.aws.amazon.com/lambda/latest/dg/welcome.html

Don’t get me wrong, I am also a big fan of AWS Lambda. But since AWS announced the game changing SDK service integrations for Step Functions, I start to think more about what are typical use cases for AWS Lambda. To use AWS Lambda more for the things that it is amazing at in the future.

Or to quote Eric Johnson at the serverless office hours:

Use Lambda to transform not to transport

⏰ Benchmarking latencies

The goal of this benchmark is not to say that service A is better/worse than service B. Each service has its strengths and weaknesses.
What we want to achieve is, getting a better understanding what kind of latencies we can measure for AWS Step Functions and how this compares to a similar integration based on AWS Lambda.

General setup

We want to measure the time it takes to read from and write data to Amazon S3 both from a state machine and an AWS Lambda function.

We test the behavior in two different versions. Version 1 simply writes to S3. Version 2 extends this by executing a GetObject operation afterwards. The code of the Lambda function is written in javascript.

const AWSXRay = require("aws-xray-sdk-core");
const AWS = AWSXRay.captureAWS(require("aws-sdk"));
const S3 = new AWS.S3();
const bucketName = process.env.DestinationBucketName;

exports.lambdaHandler = async (event, context) => {
  try {
    console.log("EVENT: " + JSON.stringify(event));
    const key = "lambda/" + event.requestContext.requestId;
    await S3.putObject({
      Bucket: bucketName,
      Key: key,
      Body: new Date().toISOString(),
    }).promise();

    await S3.getObject({
      Bucket: bucketName,
      Key: key,
    }).promise();

    const response = {
      statusCode: 200,
      isBase64Encoded: false,
    };
    return response;
  } catch (err) {
    console.log(err);
    return err;
  }
};

The state machine workflow is similarly straight forward and chains the same Amazon S3 calls as the AWS Lambda function.

Both the AWS Lambda function and the state machine can be invoked via an API Gateway. All experiments are triggered using Apache Bench with the following parameters.

ab -n 15000 -c 1 https://hash.execute-api.eu-central-1.amazonaws.com/Prod/invoke-lambda/

-n configures the total amount of requests that are triggered - in our case 15.000
-c is the number of concurrent requests - in our setup 1

I decided to use this setting because I want to generate a moderate stream of load for both integrations.

X-Ray is activated on all integration layers so that we are able to get a complete trace from the API-Gateway down to S3.

Experiment 1 - Writing to S3

The first experiment focuses only on the execution of a PutObject without reading the files afterwards. The automatic Amazon CloudWatch dashboards for AWS Lambda, AWS API Gateway and AWS Step Functions are a good starting point to provide us valuable insights.

Let us first start with analyzing the Apache Bench reports. The complete reporting is available on GitHub. Here some highlights:

The state machine was able to process all requests 539 seconds faster compared to the lambda function.
The state machine was able to process 2.07 more requests per second
The mean time per request for the state machine is 35.92 ms lower than the lambda based integration

API Gateway latencies

A closer look into the Amazon CloudWatch dashboard underlines what Apache Bench tells us. While observing the complete length of the benchmark we see that the average latency of Step Functions is constantly below AWS Lambda.

Both integration types indicate a drop in latencies indicating some kind of cold start behavior. While the drop of Step Functions on average is more significant compared to AWS Lambda.

When we take a closer look into the 99th percentile, we see some more spikes but in general a similar result over time.

Statemachine and AWS Lambda function execution

Let us now jump into the next integration layer and take a look at the duration of the AWS Lambda function and the state machine itself. Not very surprisingly that the the state machine is very much faster - in the end round about 60% compared to the duration of the Lambda function.

The AWS Lambda function runs with the default memory settings of 128MB and a default timeout of 3 seconds. Depending on the concrete use case, fine-tuning your memory settings might have a significant impact on the lambda metrics.

Downstream service latencies

I was very much surprised to see, that the connection between Step-Functions and S3 seems to be much more efficient. Looking at our X-Ray service map and traces the average latency between Lambda and S3 is 63ms compared to the integration with Step Functions of 28ms. It may be a coincidence that the relatively difference is also almost 60%. Or it might reveal, that Step Functions does some optimization handling the AWS client SDK under the hood.

Experiment 2 - Write and read from S3

I was interested to know if the amount of work a statemachine has to cover, impacts latencies and execution times compared to my AWS Lambda function. Hence we extended our experiment to also read data from S3 after writing it.

Again, let us first check the report from Apache Bench:

The state machine was able to process all requests 1287 seconds faster compared to the lambda function.
The state machine was able to process 3.01 more requests per second
The mean time per request for the state machine is 85,83 ms lower than the lambda based integration

API Gateway latencies and execution duration

Long story short, the results are comparable to the ones from the first experiment. But it is interesting to see, that the gap between the state machine and the Lambda function is getting bigger. Some factors will influence this, like the chosen implementation and runtime of the AWS Lambda function.

💡 Please checkout the awesome article of my AWS Community Builder fellow Alexandr Filichkin about a performance comparison of the different lambda runtimes.

The AWS Lambda function is not able to get closer to the latency behavior of the state machine implementation.

The AWS Lambda function needs almost double the amount of time to write and read data from/to S3.

Also interesting to see, that the latency between my AWS Lambda function and Amazon S3 seems to slightly increase compared to the first experiment on average. AWS Step Function keeps on optimizing the connection to Amazon S3 🤩.

Conclusions

Based on the things I learned, what would I answer now if someone states

Let us use AWS Lambda because a workflow described as a state machine sounds like it is very much slower.

My general answer would be: measure first. My specific answer on the comparision of AWS Step Functions and a AWS Lambda function is, that this is not true in all cases. Our little experiment revealed some interesting insights:

AWS Step Function scales and is much faster in our setup compared to my AWS Lambda function.
In this experiment, the state machine shows a more efficient communication with S3 compared to my custom code implementation.
When we compare the Step Function implementation with AWS Lambda it is obvious that we do not have to write custom code to achieve the same results.
The new capabilities of the Step Function Workflow Studio and SDK service integration lower the barrier to achieve the same result in this use case while reducing time-to-market.

But be cautious in generalizing the test results. There is a lot you can do to optimize your AWS Lambda functions to optimise for performance efficiency. Your results might also differ in other use cases. These results should not disband you from creating additional benchmarks including your specific use cases to measure what is important to you.

Please also keep in mind if you really have to optimize for performance and take into account if it is also possible to implement your use case asynchronously.

About the author:

I love those challenges to provide high scalable systems for millions of users. And I love to collaborate with lots of people to design systems in front of a whiteboard.

You can find me on LinkedIn or Twitter

Cover Image by Mateusz Wacławek on Unsplash

AWS Step function vs. AWS Lambda benchmark

Christian Bonzelet — Wed, 27 Oct 2021 10:49:46 +0000

Let us use AWS Lambda because a workflow described as a state machine sounds like it is much slower.

I could neither substantiate this statement nor could I contradict it. So I started to examine the original assumption "Step Functions is slower than Lambda" with facts. Time for a benchmark!

For me the results were crystal clear 😆

By the way: the source code is also available for you on Github.

🤹 What is AWS Step Functions?

Step Functions is a serverless orchestration service that lets you combine […] AWS services to build business-critical applications. Through Step Functions’ graphical console, you see your application’s workflow as a series of event-driven steps.

Step Functions is based on state machines and tasks. A state machine is a workflow. A task is a state in a workflow that represents a single unit of work that another AWS service performs. Each step in a workflow is a state.
Source: What is AWS Step Functions? - AWS Step Functions

State machines can be invoked both asynchronously and synchronously. Step Functions itself offers several ways to invoke you state machine, for example:

via an explicit StartExecution call using your favourite AWS SDK,
on each http request hitting your AWS API Gateway,
as a destination in your Amazon EventBridge event bus

⚡️ What is AWS Lambda?

Lambda is a compute service that lets you run code without provisioning or managing servers. Lambda runs your code on a high-availability compute infrastructure and performs all of the administration of the compute resources, including server and operating system maintenance, capacity provisioning and automatic scaling, code monitoring and logging. With Lambda, you can run code for virtually any type of application or backend service.
Source: https://docs.aws.amazon.com/lambda/latest/dg/welcome.html

Or to quote Eric Johnson at the serverless office hours:

Use Lambda to transform not to transport

⏰ Benchmarking latencies

General setup

We want to measure the time it takes to read from and write data to Amazon S3 both from a state machine and an AWS Lambda function.

const AWSXRay = require("aws-xray-sdk-core");
const AWS = AWSXRay.captureAWS(require("aws-sdk"));
const S3 = new AWS.S3();
const bucketName = process.env.DestinationBucketName;

exports.lambdaHandler = async (event, context) => {
  try {
    console.log("EVENT: " + JSON.stringify(event));
    const key = "lambda/" + event.requestContext.requestId;
    await S3.putObject({
      Bucket: bucketName,
      Key: key,
      Body: new Date().toISOString(),
    }).promise();

    await S3.getObject({
      Bucket: bucketName,
      Key: key,
    }).promise();

    const response = {
      statusCode: 200,
      isBase64Encoded: false,
    };
    return response;
  } catch (err) {
    console.log(err);
    return err;
  }
};

The state machine workflow is similarly straight forward and chains the same Amazon S3 calls as the AWS Lambda function.

Both the AWS Lambda function and the state machine can be invoked via an API Gateway. All experiments are triggered using Apache Bench with the following parameters.

ab -n 15000 -c 1 https://hash.execute-api.eu-central-1.amazonaws.com/Prod/invoke-lambda/

-n configures the total amount of requests that are triggered - in our case 15.000
-c is the number of concurrent requests - in our setup 1

I decided to use this setting because I want to generate a moderate stream of load for both integrations.

X-Ray is activated on all integration layers so that we are able to get a complete trace from the API-Gateway down to S3.

Experiment 1 - Writing to S3

Let us first start with analyzing the Apache Bench reports. The complete reporting is available on GitHub. Here some highlights:

The state machine was able to process all requests 539 seconds faster compared to the lambda function.
The state machine was able to process 2.07 more requests per second
The mean time per request for the state machine is 35.92 ms lower than the lambda based integration

API Gateway latencies

Both integration types indicate a drop in latencies indicating some kind of cold start behavior. While the drop of Step Functions on average is more significant compared to AWS Lambda.

When we take a closer look into the 99th percentile, we see some more spikes but in general a similar result over time.

Statemachine and AWS Lambda function execution

Downstream service latencies

Experiment 2 - Write and read from S3

Again, let us first check the report from Apache Bench:

The state machine was able to process all requests 1287 seconds faster compared to the lambda function.
The state machine was able to process 3.01 more requests per second
The mean time per request for the state machine is 85,83 ms lower than the lambda based integration

API Gateway latencies and execution duration

💡 Please checkout the awesome article of my AWS Community Builder fellow Alexandr Filichkin about a performance comparison of the different lambda runtimes.

The AWS Lambda function is not able to get closer to the latency behavior of the state machine implementation.

The AWS Lambda function needs almost double the amount of time to write and read data from/to S3.

Conclusions

Based on the things I learned, what would I answer now if someone states

Let us use AWS Lambda because a workflow described as a state machine sounds like it is very much slower.

AWS Step Function scales and is much faster in our setup compared to my AWS Lambda function.
In this experiment, the state machine shows a more efficient communication with S3 compared to my custom code implementation.
When we compare the Step Function implementation with AWS Lambda it is obvious that we do not have to write custom code to achieve the same results.
The new capabilities of the Step Function Workflow Studio and SDK service integration lower the barrier to achieve the same result in this use case while reducing time-to-market.

Please also keep in mind if you really have to optimize for performance and take into account if it is also possible to implement your use case asynchronously.

About the author:

I love those challenges to provide high scalable systems for millions of users. And I love to collaborate with lots of people to design systems in front of a whiteboard.

You can find me on LinkedIn or Twitter.

Cover Image by Mateusz Wacławek on Unsplash

Why you should explore your data before feeding Amazon Personalize

Christian Bonzelet — Wed, 20 Oct 2021 08:30:09 +0000

Alexa...set a timer for 15 minutes. ⏳

In my previous blog post, I showed you how to automate the provisioning of Sagemaker notebook instances. Let us now use this notebook instance for data exploration and data analysis as part of the Amazon Personalize Kickstart project.

The goal of this project is to provide you a kickstart for your personalization journey when building a recommendation engine based on Amazon Personalize. It will serve you as a reference implementation you can both learn the concepts and integration aspects of Amazon Personalize.

🕵️ Data exploration is an essential part of your machine learning development process

Before you just import your historical data, it is recommended to gather knowledge. Both on your data and on your business domain. Every recommendation engine project is kind of unique if we look at the data we have to process and the way the business works. In a very first step during a proof-of-concept phase, it is all about finding answers on:

What data can we use?
What data do we need?
Is our data quality sufficient?
How do we access the required data?
How do we identify users, interaction or items we want to recommend?

Collaborative sessions with subject matter experts help us in building an optimal solution along the given circumstances. Making decisions is easy. Making the right decision is the challenge. In my opinion, data exploration is one of the most important parts in your machine learning development process.

To formulate it a bit more drastically: without data analysis and exploration, you can only do the right thing by accident.

🏁 What do we want to achieve?

We want to build a recommendation engine covering all features of Amazon Personalize. The dataset we will use is the publicly available MovieLens dataset.

GroupLens Research has collected and made available rating data sets from the MovieLens web site (https://movielens.org). The data sets were collected over various periods of time, depending on the size of the set. Before using these data sets, please review their README files for the usage licenses and other details.

Source: https://grouplens.org/datasets/movielens/

The MovieLens dataset contains 25 million movie ratings and a rich set of movie metadata. We will use this data to provide an initial version of our recommendation engine based on historical data.

My goal is to not reinvent the wheel at all. But bring relevant analyses in one place that help us to judge if our data fits to be used for a recommendation engine based on Amazon Personalize.

Those analyses are both inspired from my personal experiences as well as a lot of cool stuff of the open source community like the following:

📊 From data to answers

Before you start with your analysis, it is recommended to define some key questions you would like to answer. You can then use the insights and knowledge you gained to discuss them with subject matter experts.

Well in our kickstart project, unfortunately there are no subject matter experts available right now. But let us start with what we have: 🤖 data and a 📖 README!

By analyzing the Movielens datasets we want to answer some very specific questions about our movie business:

What are the top 10 most rated movies?
Are ratings in general more positive or negative?
Is there a correlation between genres?

So let us get started and dive into our datasets. 🤿

🗺 Data exploration samples

For a complete overview of all analysis results, please check the complete Jupyter notebook on github.

Before we start, let us do some basic setup like importing libraries, downloading the sample data and loading them into dataframes.

from datetime import datetime
import pandas as pd

data_dir = "movielens"
dataset_dir = data_dir + "/ml-latest-small/"
!mkdir $data_dir

!cd $data_dir && wget http://files.grouplens.org/datasets/movielens/ml-latest-small.zip
!cd $data_dir && unzip ml-latest-small.zip

raw_ratings = pd.read_csv(dataset_dir + '/ratings.csv')
raw_movies = pd.read_csv(dataset_dir + '/movies.csv')
movie_rating = pd.merge(raw_ratings, raw_movies, how="left", right_on="movieId", left_on="movieId")

What are the top 10 most rated movies?

We want to know better, what movies are top rated in our system. We use the merged dataframe of movies and ratings, group it by title and sort by the number of rows per movie to get the top 10 movies.

top_ten_movies = movie_rating.groupby("title").size().sort_values(ascending=False)[:10]
top_ten_movies.plot(kind="barh")

If we build our recommender system based on ratings, we have to check if we have some bias in our data. It could happen that top rated movies are recommended more often compared to less rated videos in the end. This is something to be discussed with subject matter experts to have clear expectations.

Are ratings in general more positive or negative?

We want to now more about the distribution of ratings. Our hypothesis is, that recommending low rated videos might not be a good user experience. On the other side we might not be too aggressive as it can lead to biased recommendation by ignoring those low rated videos. Maybe there are users that are still interested in low rated videos because they fit their favorite genre. Who knows?

Let us in a first step visualize the distribution of all ratings. In a next step we will categorize ratings that are lower than 3.0 as a negative rating. All other ratings will be categorized as a positive rating.

raw_ratings['rating'].value_counts().sort_index().plot(kind='barh')

We now map each rating that is bigger than 3.0 to a positive sentiment and all other ratings to a negative sentiment.

rating_sentiment = raw_ratings.copy()
rating_sentiment["sentiment"] = rating_sentiment["rating"].map(lambda x: "positive" if x > 3.0 else "negative")
rating_sentiment['sentiment'].value_counts().plot(kind='barh')

We now get an idea that the majority of ratings are "positive".

How many videos are released per year?

movies = raw_movies.copy()
movies['release_year'] = movies['title'].str.extract('\((\d{4})\)',expand = False)
movies = movies.dropna(axis=0)
movies['release_year'] = movies['release_year'].astype('int64')
movies['title'] = movies['title'].str.extract('(.*?)\s*\(', expand=False)

movie_year = pd.DataFrame(movies['title'].groupby(movies['release_year']).count())
movie_year.reset_index(inplace=True)
movie_year.plot(x="release_year", y="title", legend=False, xlabel="Release year", ylabel="Number of movies", figsize=(12, 6));

Movies range from release dates from 1902 till 2018. Since round about the year of 1980 the amount of released movies seems to be increasing more strongly. There is an interesting drop of releases round about in the year of 2012. In 2018 nearly the same amount of movies were release like in the end of the 70s.

If there were subject matter experts in place, those analysis might result in some very interesting question to better understand the driver of both the increase in the 80s but also the drop after 2010.

💡 Conclusions

Data exploration and the learnings you gained out of data puts yourself in an excellent position. Ideally you formulated the business problem you want to solve upfront. Define some relevant KPIs you want to improve. Based on your learnings you can now dive deeper into what is possible in your situation. Challenge your KPI definition or define additional hypotheses that will guide you on your journey.

Alexa says, time is over...see you next time.

I am not a data scientist and would never consider myself to have a deep knowledge in this context. But I have to admit that I am getting a bit obsessed about those things around data science, data exploration, analysis and data driven decisions. I observed a lot and tried to be the sponge that soaks up everything in this area.

Hence I am really interested in your feedback, experience and thoughts in the comments. 👋

Cover Image by Andrew Neel - https://unsplash.com/photos/z55CR_d0ayg

Automate provisioning of Sagemaker Notebooks using the AWS CDK

Christian Bonzelet — Sat, 02 Oct 2021 21:30:26 +0000

Alexa...set a timer for 15 minutes. ⏳

In this post I want to highlight two things:
1.) The announcement of the Amazon Personalize kickstart project
2.) Show you how to automate the provisioning of Sagemaker Notebooks for your data exploration tasks

🚀 What is the Amazon Personalize kickstart project?

The goal of this project is to provide you a kickstart for your personalization journey when building a recommendation engine based on Amazon Personalize. It will serve you as a reference implementation so you can both learn the concepts and integration aspects of Amazon Personalize.

You can also use it to build you own recommendation engine upon best practices and production-ready components based on the AWS Cloud Development Kit.

It will contain all of my collected best practices over the last year while building and designing recommendation engines. From implementing A/B testing strategies towards orchestrating and automating the training workflow of Amazon Personalize. But also providing a good developer experience using sandbox stacks or automate the provisioning of Sagemaker notebook instances.

The kickstart project is open source and available via Github:
https://github.com/cremich/personalize-kickstart/

💻 Wait, you are talking about an AI service. Why do I need to get in touch with Sagemaker?

Independent on which layer of the AWS ML stack you operate:
Before you just import your historical data, it is recommended to gather knowledge. Both on your data and on your business domain. Every recommendation engine project is kind of unique if we look at the data we have to process and the way how the business works.

Your process should start with defining the business problem you want to solve. Followed by defining KPIs you want to improve and framing your ML problem definition. Then start with data exploration and analysis.

A managed jupyter notebook by Amazon Sagemaker is an excellent start to

ingest and analyze you data
prepare, clean and transform your data,
start to train and tune your recommendation model candidates

The Amazon Personalize kickstart project supports you to automate the provisioning of individual Sagemaker Notebooks. Also ensures that a notebook is deleted once you delete your stack to save costs.

🚧 The Sagemaker notebook construct

The problem it solves

My observation from recent projects, if you do not automate this kind of stuff:

You kicked off your machine learning project and want to use AI services to solve this. Your team members want to do some data exploration and data analysis in the early days. Therefore everyone who owns this task, provisions a Sagemaker Notebook instance, some default IAM execution roles and a bunch of S3 buckets to store the data that needs to be explored.

Your project ends and usually those manual provisioned resources will be forgotten but still costs you money. They might also introduce some vulnerabilities due to outdated libraries until you stop your notebook sessions or restart the instances.

The solution it offers

The Sagemaker notebook construct provides an automatic provisioned Amazon Sagemaker notebook instance for your data analysis and data exploration tasks. Provisioning a Sagemaker Notebook is optional and not required in all stages and cases. In central provisioned dev, staging or production accounts a Sagemaker Notebook is not inevitably necessary.

But it will help you in your developer sandbox accounts or stacks. We encapsulate the resources that are needed to operate and run a Sagemaker notebook in a reusable construct.

The construct consists of the actual Sagemaker Notebook instance, an S3 bucket to put your raw data in as well as a IAM execution role. The role grants the Sagemaker service access to get and update data in the S3 bucket. It further includes the required managed policy to provide Sagemaker full access.

Construct parameters allow you to set the name of the notebook instance, the required EBS volume size as well as the required EC2 instance type.

⚠️ Please keep in mind that some properties will result in a resource replacement like changing the instance name. According to the cloudformation resource documentation, a change of the EBS volume size or the instance type won't replace your notebook instance.

Each developer that uses this construct will now benefit from an automated and consistent process while ensuring that all resources for data exploration are deleted once they are not needed anymore.

Sure there is always room for additional features if you for example have the requirements of a more fine grained networking setup. Consider this as a conceptual starting point and extend this concept to your needs.

Automate everything

The benefits of automating this process is, that you prevent to end in orphan resources. In my experience, it is likely to be forgotten over time that there are some notebooks still running or raw data still sleeping in S3 buckets.

So save your credit card and automate everything 😊

It also enables some other interesting options like uploading some initial data for data exploration or training along provisioning your construct. Or connecting a git repository to your notebook instance to provide some shared notebooks to all your data scientists.

Alexa says, time is over...see you next time. What additional ideas or options do you have in mind? What are your experiences how to automate this process? Happy to get your feedback, experience and thoughts in the comments. 👋

Automated rotating of AWS access keys in Bitbucket pipelines

Christian Bonzelet — Fri, 20 Aug 2021 12:52:21 +0000

⚠️ I am sorry if gists are sometimes rendered in the wrong order. It still seem to be an open issue on the forem platform: https://github.com/forem/forem/issues/14428

AWS access keys enables you to provide a programmatic access to your AWS cloud infrastructure. You can use those access keys to verify your identity and permissions in your AWS CLI, all AWS SDKs but also in your CI/CD pipeline to deploy your application which certain toolsets like SAM, CDK or the raw AWS CLI.

When using access keys, automated rotation is a very important aspect in terms of security and also embedded in the security pillar of the AWS Well-Architected Framework.

In this post, I will describe a solution to automatically rotate access keys that are used in your Bitbucket CI/CD pipeline using the Bitbucket API.

Best practices for managing AWS access keys

Besides the regular rotation of your access keys, AWS describes some additional best practices when managing your AWS access keys, including:

protect or don't create your root user access key
don't embed access keys into your source code
rotate access keys periodically
remove unused access keys
configure MFA for your most sensitive operations
use IAM roles instead of long-term access keys

But how to I rotate access keys that I configured in my CI/CD SaaS solution like Github, Bitbucket or Gitlab?

The good thing: all major providers provide APIs to enhance automation. Let's have a look how this can work for Bitbucket.

Rotate, deactivate, delete, repeat

What do we want to achieve? We want an automated process to

Rotate our access keys every 90 days
Deactivate unused access keys every 100 days
Delete unused access keys every 110 days
Update deployment variables in our Bitbucket pipeline configuration after every rotation.

Okay. Challenge accepted. Hold my beer and here we go:

In order to trigger our process we configure scheduled Cloudwatch events to invoke some lambda functions ever 90/100/110 days. Each lambda function has a dedicated responsibility to create a new, deactivate an or delete already deactivated access keys.

The rotation lambda is straight forward. It creates a new access key and writes the credentials in a secret provisioned in the AWS Secret Manager.

The secret will be the source of truth for the actice access key that is also used in our Bitbucket Pipeline configuration. In the next chapter, we take a deeper look how we now sync the secret with Bitbucket.

Sync credentials with Bitbucket

Bitbucket provides a REST API to access data or trigger operations on repositories, workspaces or pipeline configurations. In order to update environment variables in our deployment configurations in Bitbucket, we need to know:

the name of the workspace,
the name of the repository,
the uuid of the deployment environment.

Having all those information in place, we can use the Bitbucket API to update the value of our deployment variables.

We can then use a Cloudwatch Event to capture all PutSecretValue events via CloudTrail and invoke the sync lambda function that is responsible to update your Bitbucket deployment variables. If you for example use the AWS CDK, configure the CloudWatch event like:

If you are more a fan of raw Cloudformation or SAM, the same will also be possible to setup in your favorite infrastructure-as-code tool.

What about IAM roles?

If you are more looking for a solution that does not rely on access keys but rather uses IAM roles for authorizing, I can highly recommend the solution to deploy on AWS using Bitbucket pipelines OIDC.

In order to use OpenID Connect on AWS, you will need to configure Pipelines as a Web Identity Provider, create an IAM role, and configure the build to assume the created role prior to running your build.
Web Identity Providers allow the system to receive an authentication token, and then use or exchange that token for temporary security credentials in AWS. These temporary security credentials map to an IAM role with permissions to use the resources in your AWS account. Learn more about Web Identity Providers from AWS

Source: https://support.atlassian.com/bitbucket-cloud/docs/deploy-on-aws-using-bitbucket-pipelines-openid-connect/

Alexa says, time is over...see you next time. Happy to get your feedback, experience and thoughts in the comments. 👋

Cover Image: https://unsplash.com/photos/DoWZMPZ-M9s

Learnings from creating recommendation engines with Amazon Personalize

Christian Bonzelet — Thu, 08 Jul 2021 12:23:03 +0000

Alexa...set a timer for 15 minutes. ⏳

In the last year of working in my role as an AWS Solution Architect, I was involved in several projects to improve products to provide a more personalized experience. Recommendation Engines are one good way to provide a personalized product usage.

I want to give you some insights about my personal learnings on building recommendation engines using Amazon Personalize.

What is Amazon Personalize?

As part of the AI Services, Amazon Personalize is a managed service to provide personalization and recommendations based on the same technology used at Amazon.com. The marketing claim further adds: "...with no ML experience required". Well in general that is true, but my personal experience is: having ML skills on board will boost you project. Lets dive deeper into this in the next chapters.

Using Amazon Personalize you will get convenient APIs you can use to solve a specific personalization and recommendation business problem. The trade-off of using more convenience is less flexibility in the end. This does not have to be bad at all. But it is good to keep this in mind when choosing on which level of the AWS ML stack you want to focus your work.

Amazon Personalize is a fully managed service. It generates highly relevant recommendations using deep learning techniques. It build custom and private ML models using your own data. Private ML models means, data and models are not shared to improve the Amazon Personalize service itself. Which makes GDPR compliance a bit easier 🥳

Three simple steps to your recommendation engine

The cool thing about Amazon Personalize, it includes a fully managed ML pipeline to identify features, select the right hyperparameters, train your model, optimize your model, host your model and provide a real-time feature store.

Your main task as a consumer is:

Define you target schema of user interactions, item metadata and user metadata
Preprocess or transform historical data to match your desired target schema
Import your data into Amazon Personalize using a batch import
Deploy your auto-trained recommendation model resulting in an API endpoint to
Infer recommendations for your users

Sounds easy. But working on three recommendation engine projects in the past, I can tell you: there are a lot of more things to consider besides those 5 steps.

Amazon Personalize solves three foundational personalization cases like user pesonalization, similar items or personalized ranking.

From PoC to MVP

Data analysis and feature engineering

In a very first step during a proof-of-concept phase, it is all about finding answers on:

What data do we need?
How do we get those data?
How do we identify a user?

Amazon Personalize is able to collect data for three main data sources: user metadata, item metadata and user interactions. Personalize at least needs interaction metadata to operate and work. Item and user data are not necessary required but having them will improve the recommendation output.

So even if you do not need data science skills to use Amazon Personalize, it is really helpful to have those skills in your package for data analysis and preparation.

Strong collaboration within your team enables you to get quick answers the above questions. Existing web analytics or user tracking data is often a very interesting source for interaction data. At least for using these data to create an initial and production-ready recommendation model.

Data analysis might also reveal some additional challenges. If data quality is poor, it is likely that providing good recommendations from the start might not be the easy task. Making wrong assumptions might lead to biased datasets which will influence recommendations. In a nutshell: data analysis and preceding feature engineering is key to understand the context you are in.

Define your KPIs

Without knowing the exact business value you want to provide and how to measure success, it can be very hard to estimate whether the solution you provide is successful or not. Personalized recommendations means: you get different content compared to me. Looking just at my result it is very hard to debug and judge if the recommendations me or one of my collegues get, are sufficient or not. Simply trusting the process that Amazon Personalize will make a good job might also blur your results.

You have to define your KPIs upfront. What makes your recommendation engine successful? Is it an increase of session duration? Is it an increase of read articles per user per session? Is it an increase of selled products? Is retention the right KPI?

Defining the right KPI is not an easy task. And doing it too late makes this process even harder. So my recommendation is: do it as a very very first step and align with your business stakeholders, what KPIs make sense.

A/B testing

A/B testing is mandatory when you consider to build a recommendation engine. It should be prepared right from the start of a project. Otherwise it could make things unnecessary complex if we blindly pick an out-of-the box solution without looking at what is really needed for a given workload.

A/B testing is a common technique for comparing the efficiency of different recommendation strategies. These capabilities should support us in getting answers based on our KPIs. By applying A/B testing you are able to compare different kind of user journeys:

user journeys including recommendations vs no recommendations to measure the impact of recommendations on your KPIs.
measuring the performance of two recommendation models to decide which one is the better one.

My personal conclusion after the past months of work in this field: data is key. If you want to use data to make better decisions quickly, you have to work on both accessibility, semantic and quality of your data.

Analytics has to scale - and here I not mean the technical side of scaling. It has to scale organizational along business requirements and frequency of change. Focusing on AI/ML use cases introduces a lot of changes as you have to explore and experiment with data to measure your success. The more and better data we have, the better is your foundation for experiments. Introducing experiments and A/B testing might also come with a change in organizational culture or peoples mindset.

Kick start your proof of concept

During my learning path, I collected several resources that helped me a lot to understand more about Personalize and also use existing implementations to setup a working recommendation engine in less than an hour. This is actually very helpful in a proof-of-concept phase to get fast results.

Below you find a curated list of helpful resources on Github and the AWS blog.

Github

AWS Blog

Workshops

My top three learnings

Amazon Personalize is an awesome service. I am very impressed about the results. But Personalize can only provide good recommendations, if you keep some things in mind. Here are my top three learnings.

Scope out the business problem you want to solve
Define your KPIs to measure success and implement A/B testing
Analyze your data and crunch as much knowledge as you can about the business domain and business problem.

Alexa says, time is over...see you next time. Happy to get your feedback, experience and thoughts in the comments. 👋

Image Sources:

All you need is love ❤️ and data 🤖

Christian Bonzelet — Thu, 10 Jun 2021 13:08:12 +0000

Alexa...set a timer for 15 minutes. ⏳

A few days ago I had a chat with a former colleague that inspired me to write my first official lighning post article about this. He asked me via WhatsApp:

Hey Christian, how are you? You must be very familiar with Lambda as an AWS solution architect? I have a question.

The question was targeted towards a service quota of lambda for the maximum number of parallel executions. Giving the answer on the exact question was very easy. I said:

Yeah there is a default quota of 1000 concurrent executions. But you can increase it if you need to via a support ticket.

Well that could be the end of the story but I was interested to know more about why they hit those limits and we got in a very interesting conversation.

What about data 🤖 ?

The conversation reveals a lot of what I spend my professional day with.

My job as an AWS Solution Architect at DFL Digital Sports is to help teams find an optimal solution to a (business) problem. Or as Maria Ane Dias (Solution Architect at AWS) describes it very aptly on Instagram:

I help customers understand how they can innovate by working backwards from their challenges, drawing and reviewing architectures, and being a strong support and advisory to them.

Bam 💥

And one of the most exciting challenges in my work: for every problem there are many possible solutions.

To make an informed decision about possible solutions, I need a lot of background information. I have to put myself in the shoes of others and ask a lot of questions to understand things and contexts.

The reason why my buddy was actually evaluating AWS Lambda was because they have troubles to handle traffic peaks after they send out push notifications to users. And my buddy seems to be kind of disappointed that a serverless approach using AWS Lambda did not solve the problem out of the box.

So our discussion afterwards was a lot about topics apart from AWS Lambda. I asked a lot of questions like:

Do you have enough data to predict peak loads?
Can the response somehow be cached?
What percentage of users are logged in on average?
Is it possible to segment users and send the notification in bulks?

So, in order to be able to show solution options
a) I have to have a good technical understanding
b) but more importantly, understand exactly how the problem is structured.

In the end, the discussion was less about AWS Lambda. It was more about the context of the application. About background information on the problem.

I asked myself 🤔:

Maybe AWS Lambda is a less than optimal choice of technology here?
Are there other approaches in terms of integrating the systems in charge?
How deep is the data base to make informed decisions here?
Is the problem a general one or does it only occur sporadically?
What influences that the problem occurs?

Data and knowledge helps me in my daily work to support my colleagues in the best possible way. To make data-driven decisions and reduce the amount of hypothesis. Technology must never be an end in itself.

In my eyes, the choice of technology must always be based on supporting a specific problem. If we've achieved that, we've done everything right. 💡

Coming back to the problem of my former colleague, I recommended to look, if the new released capability of EC2 warm-pools could potentially help better before refactoring towards AWS Lambda. Or if they can somehow reduce the pressure by caching responses at edge locations. Or if something like predictive scaling of EC2 could also work for them. Maybe it is "shooting at sparrows with cannons", but in this short time it was not possible for me to dig deeper in the status quo of his current architecture and future plans.

And what about love ❤️ ?

I love what I do. I believe that is another important factor to inspire people. I am a passionate software developer, software architect and fan of technology. I don't care if the software is written in JavaScript, TypeScript, Java or PHP. Likewise whether it runs serverless, container-based or on premise.

I am constantly trying to broaden my horizon. This is the only way I can neutrally look at a wide range of options and then come to an informed decision.

Passion for what you do is everything!

Alexa says, time is over...see you next time. Happy to get your feedback and thoughts in the comments. 👋

Introducing lightning posts

Christian Bonzelet — Tue, 08 Jun 2021 11:44:27 +0000

Alexa...set a timer for 15 minutes. ⏳

Admittedly, I've made many attempts to blog in the past. Ultimately, in almost all cases, it has failed due to lack of time. But somehow, some past encounters with different people have managed to make me try again.

But this time I want to approach the subject differently. What I certainly can't achieve: gaining more time. But I dare to try different techniques to divide my time better.

The first encounter that made me want to blog was during a job interview. I saw that the applicant was blogging and asked him: "What is your motivation to blog?"

He told me, meaningfully: "I use so much from open source communities every day. It's time to give something back".

Wow! That immediately triggered me. 🤩

But how can I "give back" meaningful content on the one hand and not sacrifice several hours of my time for it on the other hand.

I dare an attempt which I call "lightning posts".

Based on the technique of "lightning talks" which I already know from various conferences. I set a timer for 15 minutes and start writing.

What are lightning talks? ⚡️

Wikipedia says:

A lightning talk is a very short presentation lasting only a few minutes, given at a conference or similar forum. Several lightning talks will usually be delivered by different speakers in a single session, sometimes called a data blitz.
Some formats of lightning talk, including PechaKucha and Ignite, involve a specific number of slides that are automatically advanced at fixed intervals. Lightning talks are often referred to as ignite talks.

In my own words, it is about reducing a specific issue, topic or problem to its essence and explaining it to an audience in a short time.

Of course, I have thought about a topic in advance. And there are already many ideas circulating in my head about how to structure the topic accordingly.

So the biggest challenge is to focus and just get started. At the same time, I prepare myself for the fact that not everything will run perfectly from the start. I'm also adjusting to the fact that the amount of content per post probably won't be as detailed. But is that necessarily a bad thing? When I read blog posts, I tend to like ones that get to the point quickly. Because even when I'm reading, I feel that the time I can invest is simply limited.

Inspired by the crazy 🎱

Another source of inspiration was a design sprint I recently participated in. There we used the brainstorming method "crazy 8". Each team member has eight minutes to visualize possible solutions to a problem on eight fields of a sheet of paper. I thought that was very cool! The method ensured maximum focus on a concrete problem in the minutes.

The obvious thing about both variants (lightning talks and crazy 8): they use the technique of "timeboxing". I try to get the maximum out of a defined time. This does not necessarily have to be perfect. But the way is the goal. You can always improve later.

I'm curious myself how the experiment is going. And I look forward to your feedback in the comments. If you come across things that I should formulate more precisely, please let me know.

👋