DEV Community: Ed Gray

Salesforce Marketing Cloud: Installing Einstein Tracking Codes on Shopify

Ed Gray — Tue, 13 Oct 2020 19:59:50 +0000

Salesforce Marketing Cloud's tracking codes are essential to unlocking many of its advanced marketing automation features. Whether you're simply wanting to send an email based upon what product(s) a subscriber has viewed or start up Einstein's recommendation engine - it begins with correctly installing tracking codes.

Unfortunately, figuring out how to install the codes that track subscriber behavior can prove challenging for non-experts. This step-by-step guide will help get you started installing the collect codes on any standard Shopify store - without the need of any additional applications.

This guide assumes you've uploaded an initial product catalog, created the standard Einstein Recommendation data extensions, understand personalization strings and know the basics of editing the theme code of a standard Shopify store.

Step 1 - Base Collect Code

Every webpage that uses a Marketing Cloud tracking code will need the Base Collect Code installed first. Install the following code in the theme.liquid file before the closing tag:

Make sure to replace the MID with your business unit unique identifier, not only for the Base Collect Code but for all of the below code.

Step 2 - Start appending Subscriber ID or Subscriber Key to outbound links

Next, we need a way to identify a specific Marketing Cloud subscriber's behavior on the website. One relatively easy way to do this is by passing the subscriber key or id as a parameter in links from Marketing Cloud emails.

You could use Marketing Cloud's Parameter Manager to automatically append the same parameters to all links - but this can interfere with other links you build that rely on other parameters. Instead, I recommend working with Marketing Cloud's support team to alter the Conversion Tracking Link that can be enabled on specific links in an email.

For example, I used the default 'sfmc_sub=%%subscriberid%%' used by checking the Conversion Tracking Link box in Email Studio on links that lead to the Shopify store.

Step 3 - Tracking Users

Now that we have parameters in links to help identify subscribers, we can use javascript to use the parameter in the User Collect Code. Here is an example of the javascript to install at the bottom of theme.liquid template:

Remember to replace the MID - and 'sfmc_sub' with the name of the parameter used for the unique Subscriber's ID or Key.

Don't be tricked by the use of 'email' in the collect code, we can use any unique identifier that Marketing Cloud can map to the subscriber. In fact you should NOT use an email address or any other Personal Identifying Information in the tracking codes.

Ignore Salesforce instructions on only installing the User tracking code on login pages. We want to be able to identify subscribers and track their behavior by establishing a cookie regardless of whether they log into the Shopify store.

While this method has the benefit of establishing cookies for subscribers without requiring them to log into the Shopify store, it can cause issues when subscribers forward their email to others or copy/paste it into a website.

Step 4 - Track Products Views

With subscribers being identified through the User collect code, it's time to track what products they're viewing. Install the following code in the product.liquid template file in your Shopify theme:

Remember to change the MID to your Marketing Cloud ID.

Step 5 - Streaming Catalog Updates

Tracking item views is easy - keeping your product catalog up to date is a bit more difficult. No one wants to constantly be uploading a new catalog every time you add, remove or adjust a product on Shopify.

There are essentially two options for automatically keeping the catalog up to date: batch uploads and Streaming Catalog Updates. Batch uploads are a possible solution, but require regularly exporting a .csv from your Shopify store to Marketing Cloud's SFTP - but this will require either an additional application or API development. Many of the Shopify applications once capable of automatically exporting a catalog are no longer compatible with Salesforce's strict SFTP rules.

Instead, let's use Streaming Catalog Updates in Shopify by installing the following code on your product.liquid template:

Remember to update the MID and the domain of your Shopify store for the url attribute. This is a simple product catalog, with a single primary category attribute and multi-value Tags attribute. If your catalog requires more attributes, you'll need to map them to the Shopify object here.

Make sure to turn on Streaming Catalog Updates in the Implementation dashboard. Next, whitelist the domain of your store - and the standard cdn.shopify.com domain where product images may be hosted.

Step 6 - Troubleshooting Collect Codes

With the Base, User, Item View and Streaming Catalog Update codes installed, it's time to check if they are working. Navigate to the Status dashboard under either Web or Email Einstein Recommendations to see which of the installed codes are working.

Remember, you won't get identified sessions until users click on a link that passes their Subscriber ID or Key. Test the User code by sending a test email to yourself.

Unless you're a superhuman coder, there are likely to be mistakes to work through - it happens. Don't worry, the remaining of the tracking codes are relatively simple - the Streaming Catalog Updates and User Code are the tricky ones that we must get to work first.

You can use the Network tab in Chrome's Developer Inspector tool - filter for 'IGO' and make sure any kind of ad or script blockers are turned off. Navigate the website and check the payload of the tracking codes to see what needs fixed.

Step 7 - Track Cart Activity

To track what items are in a subscriber's cart, we'll use JavaScript to listen for the Add-to-Cart button is clicked on a product's page. Add-to-Cart buttons can vary depending on your Shopify theme - so you will need to know the CSS class or ID for JavaScript to identify the button(s) that should trigger the event.

This Shopify's store Add-to-Cart button has a class of '.product-form__cart-submit'. When a button with that class is clicked, the cart tracking collect code will be triggered. Remember to replace it with the class of your Shopify store's Add-to-Cart button(s).

It's important to note that the cart tracking code will overwrite any existing information in the subscriber's cart. That's why the tracking code needs to loop through all the items that are already in their Shopify cart - while adding the new product as well.

Next, we should also track cart activity on the actual cart page by inserting the following code into the cart's liquid template:

The above code will let Marketing Cloud know when a subscriber's cart is empty. As always, remember to change the MID to your Marketing Cloud's unique identifier.

Step 8 - Track Purchases

Instead of updating a liquid template file, we'll be adding the following collect code to the additional scripts section of the checkout settings in Shopify:

We need to include the base tracking code again because Shopify's checkout process is handled on a third-party site.

Since shoppers may return to their order confirmation page several times, we should included liquid code that will only initiate the purchase tracking code the first time a user visits the page.

Step 9 - Track Searches

Next, replace the MID and install the following code in the search liquid template file:

Step 10 - Track Category Views

Categories are referred to as collections in Shopify. Install the category collect code in the collection liquid template:

Congratulations: your Shopify store is now tracking the behavior of your Marketing Cloud Subscribers! Check the Status dashboard again to make sure everything is working properly - and fix any issues.

Step 11 - Check IGO Data Extensions

With the catalog uploaded and collect codes now installed, the IGO data extensions should start populating.

You'll notice the user_id is mapped to the Subscriber ID or Key in the 'email' column of the IGO_Profiles extension. This user_id is the external key to map the subscriber's activity inside the IGO_Views and IGO_Purchase extensions.

Finally, with some SQL queries and the _Subscribers data view - you're ready to set up some sendable data extensions based upon what products a subscriber viewed or purchased. With some more time and data, you can also start using the Einstein email and web product recommendations and behavioral triggers.

(This content was made while working as a Marketing Cloud consultant for LearningCurv in Austin, TX)

Segmenting and Reporting On Subscriber Behavior in Salesforce Marketing Cloud

Ed Gray — Mon, 28 Sep 2020 19:46:59 +0000

Salesforce Marketing Cloud's Journey Builder is a robust and powerful tool to target subscribers with tailored emails, texts and digital ads. But placing all your subscribers into perpetual journeys for segmenting is not always the best option (neither is Einstein's fluctuating engagement scoring).

Below is quick way to set up a data extension that updates everyday with a subscriber's email behavior in the last 180 days. The SQL queries rely entirely on Data Views, so they should work the same on every account or business unit.

The Performance_Subscriber data extension can then be filtered for further segmentation or customized with additional fields. It can also be used for reporting purposes.

Step 1

Create the Performance_Subscriber Data Extension using the following SSJS activity:

This should prevent any typos or mistakes when setting up the data extension. Any of the SQL queries below would require some modifications if changes to the field names or data types were made. To learn more about creating data extensions using SSJS or WSProxy, read Ivan Razen's thorough guide here.

Step 2

Create a SQL query that overwrites Performance_Subscriber with the following code:

Performance_Subscriber now has the relevant subscriber information from the Subscribers and Sent Data Views.

Depending on the number of subscribers in the business unit, separating or combining some of the SQL queries may be possible. This sequence of queries doesn't take more than a few minutes to run with 1.2 million records.

Step 3

Next, use this SQL query to update Performance_Subscriber with information from the Open and Click Data Views:

Performance_Subscriber records both unique and all clicks and opens. It might be helpful for finding the most engaged subscribers.

Step 4

Calculate their open and click rates with an SQL activity that updates Performance_Subscriber:

Step 5

Finally, schedule an Automation to run the SQL activities, ordered in the same as they were created.

Now What?

Performance_Subscriber is now ready to be filtered into sendable data extensions, like a segment of those with a high open rate, but low click rate. With some additional SQL and columns, you could even track trends in a subscriber's click or open rate over the last 30, 60, 90 days etc.

Not only is Performance_Subscriber helpful for segmenting, it can also be the basis of creating helpful reports. For example, a signup source could be added to subscriber's record. An additional data extension could be created that grouped together the performance of the source, instead of the individual subscriber.

Summary

Marketing Cloud's out-of-the-box segmenting for scheduled sends and more precise journey entry sources is practically non-existent. But with some data extensions and SQL queries, the possibilities for segmentation is limitless.

With the above SSJS, SQL queries and automation, any business unit can quickly have a data extension that's a useful start to segmenting its email subscribers.

You can also find the code necessary to build Performance_Subscriber in this github repository.

How do you use SQL queries to segment subscribers?

Tracking the Health of Your Salesforce Marketing Cloud Account

Ed Gray — Mon, 28 Sep 2020 17:24:09 +0000

Reporting in Salesforce Marketing Cloud can be challenging for new users. The "reports" available through Analytics Builder are limiting and difficult to work with. But with a little knowledge of SQL and Data Views, creating data tables tailored to your business use case becomes possible.

Below is an example of a report created to track the basic "health" of a business unit: subscriber growth and the number of emails sent, opened, clicked and bounced. It's completely based upon Marketing Cloud's Data Views, so the exact same SQL queries should work for any account.

Some queries could be combined or separated depending on the number of records in the business unit - this was developed for a client with more than 1.2 million records and never take more than a few minutes.

Step 1

Run a SSJS activity to create the 3 data extensions needed for the report using this code:

Using this SSJS will make sure the fields are the exact same name and data type needed for the SQL queries below to work without modifications. For more information on how you can create data extensions using SSJS and WSProxy, read Ivan Razine's exceptional guide here.

Step 2

Create a SQL activity that appends the Account_Health_Daily data extension with the following code:

Why append? Because many Data Views only track the last 180 days of activity. By appending new records, we can keep some basic information about the account longer. This could be useful if we wanted to make year-over-year comparisons etc.

Step 3

Unfortunately, appending Account_Health_Daily means the most recent day's stats will be added to the end of the data extension. Since SQL can't overwrite the same data extension used as a source, a second data extension will be made to sort the records by most recent day first.

Use the following code to overwrite the data extension Account_Health_Daily_Sorted:

Step 4

Now that the basic stats from each day are logged, they can be grouped together by month. Start with the following code to overwrite the Account_Health_Monthly data extension:

Step 5

Next, record the number of active subscribers on the 1st of the month using the following SQL code to update Account_Health_Monthly:

Step 6

Finally, calculate the open, click and bounce rate for the month by updating Account_Health_Monthly with this SQL query:

Step 7

Now an Automation can be created to append and update the data extensions. Schedule an Automation to run every day, with the SQL activities in the following order:

Appends Account_Health_Daily with Data View info
Overwrites Account_Health_Daily_Sorted
Overwrites Account_Health_Monthly with Account_Health_Daily info
Updates Account_Health_Monthly with Active Subscribers
Updates Account_Health_Monthly with Open, Click, Bounce Rates

Adding Previous Days

The SQL query appending the daily log are carefully written to only include data from the previous day - and only that one day. To append Account_Health_Daily with previous days, use the following code:

The above query will look into the data extension, find the oldest day on record and append the next oldest. Keep running it manually or place it in an automation to run until Account_Health_Daily has enough information to be useful.

Presenting Reporting Data

Data extensions are the basis of all custom reporting in Marketing Cloud - but they can be difficult for humans to sift through for useful information.

Below are two examples of HTML blocks to build inside Content Builder and placed into an email or cloud page. They group together the business unit's basic stats for the previous week - and three more weeks for context. It's a good start to presenting the information from the data extensions in a way that is actually helpful.

Additional Notes

Subscribers can open and click emails days, even months after it was sent. Don't be surprised if there are single days where a business unit has stats that appear impossible.
Reporting in Marketing Cloud is rarely 100% accurate. Look for trends, not exactness in your data.
The code needed for this report can be found in a github repository here.

Summary

With the above code, you can have the necessary data extensions setup, an automation to refresh them and an HTML block to neatly present the data in less than an hour. It can help answer some common questions like:

When did our open or click rates start to increase or decrease?
When did our account grow or shrink?
When did our bounce rate suddenly increase?

This "Account Health" report is just one example of the awesome things you can develop with a little SQL in Marketing Cloud. What are some examples of "reporting" data extensions you've built in Marketing Cloud and why?

(This report was developed while working as a Marketing Cloud consultant for LearningCurv in Austin, TX)

100+ Places to Find Freelance and Remote Web Developer Jobs

Ed Gray — Tue, 18 Feb 2020 23:18:31 +0000

Whether you're wanting to become a digital nomad, earn extra money as a freelancer or work from home instead of a cubicle - you've come to the right place. Below are more than 100 websites you can find freelance and/or remote web developer jobs.

Web development is one of the best job markets for remote work, so there's lots of opportunities - but there can also be lots of competition. Getting hired or signing clients is often a numbers game, so I'm hoping this exhaustive list will help you find more opportunities in your job search. If you're still struggling, try these 6 tips for getting hired as a remote web developer.

So here's the list of job boards, freelance platforms and social media groups you can use to find a remote web developer job. Please leave a comment with your tips on getting hired or places to find work - I'll try to keep this an updated resources for the Dev.to community.

➡️ 10xManagement

➡️ AirTasker

➡️ AngelList

➡️ Angular Jobs

➡️ Authentic Jobs

➡️ Codeable

➡️ Codementor

➡️ CodePen

➡️ Communo

➡️ CyberCoders

➡️ DEV.to

➡️ Dice

➡️ Discord Channels:

➡️ DistantJob

➡️ Dynamite Jobs

➡️ Ember

➡️ EuropeRemotely

➡️ Facebook Groups:

➡️ FlexJobs

➡️ Freelancer

➡️ FreelancerMap

➡️ Front End Remote Jobs

➡️ G2i

➡️ GitHub Jobs

➡️ Gun.io

➡️ Guru.com

➡️ Hacker News

➡️ Hireable

➡️ Hired

➡️ idealist

➡️ iwantremote

➡️ jobisjob

➡️ Jobspresso

➡️ JS Remotely

➡️ JustRemote

➡️ Landing.jobs

➡️ Localancers

➡️ LinkedIN Groups:

➡️ MotionRecruitment

➡️ NODESK

➡️ Outsourcely

➡️ Pangian

➡️ PeoplePerHour

➡️ Power to Fly

➡️ Reddit

➡️ Remote.co

➡️ RemoteLeads

➡️ RemoteOK

➡️ Remoters

➡️ RemoteTalent

➡️ Remotive

➡️ Ruby Now

➡️ SkipTheDrive

➡️ Slack Channels:

➡️ Sologig

➡️ Stack Overflow

➡️ Toptal

➡️ Triplebyte

➡️ Upwork

➡️ Vettery

➡️ Vue Jobs

➡️ We Work Remotely

➡️ Workew

➡️ Working Nomads

➡️ X-Team

➡️ xfive

This list of places to find remote and freelance web developer jobs are partly sourced from these excellent articles:

Top 9 Places to Find Remote Front-End Freelance Developer Jobs by Derick Sozo (9/23/2019)
14 Platforms for Web Developers to Find Development Projects and Remote Jobs by Rajinder Singh (11/18/2019)
Candidate Sourcing from Slack - 109 Slack Communities for Finding New Hires by Mile Živković (11/16/2018)
Freelance Web Developer Jobs by Ed Gray
Candidate Sourcing from Facebook: 44 Best Facebook Groups to Find Remote Talent by Mile Živković (12/19/2018)
18 websites, where you can find a remote job by Duomly (10/31/2019)
The 25+ Best Sites for Finding Remote Work in 2019 by Scott Morris

Want to know what to expect to be paid? Check out this resource on front-end developer salary here.

8 Tips to Improve UpWork Profile for Web Developers

Ed Gray — Sat, 15 Feb 2020 00:41:09 +0000

What’s the perfect profile to get freelance web developer jobs on UpWork? There’s really no such thing - but there are some common traits among the most successful freelancers on UpWork. Here are some of the tips and tricks they suggest for improving your chances at getting freelance work and raising your prices.

1. Professional Photo

Having a professional looking photo is fundamental to winning contracts on UpWork. Those making the hiring decisions are sifting through dozens of proposals and an unprofessional photo makes it easy for them to move past yours. UpWork itself claims that those with professional photos get hired 5x more than those that don’t.

Mike Volkin is a top-rated freelance CMO on UpWork and sometimes featured on their homepage. He also coaches entrepreneurs on becoming a profitable freelancer. Here are his tips on taking the best profile photo to get freelance jobs on UpWork (start at 10:18):

• Grin, but don’t force too big of a smile or you’ll end up looking creepy.
• Be well dressed
• Have a clean, solid color background. If the background is too busy, use Photoshop to blur it out.
• Tilt your head at an angle.
• Crop the photo to include your shoulders and up. Don’t be too far away, but don’t include only your face.

2. Specific, Descriptive Title

After your photo, your title is the next thing prospective hiring managers will look at. Make sure it conveys that you are capable of excelling at the jobs you’re applying for. This means being descriptive and specific, not generalized in an attempt to please too many people.

Josh Burns is another top-rated freelancer on UpWork who specializes in SQL Server and has made more than $500,000 on the platform. He has a helpful YouTube channel about being successful on UpWork and here are his tips on the right title (start at 2:25):

• The title is another vital first impression, make sure it captures attention in search results and proposals.
• Your title should describe the specific skills needed to complete the jobs you’re looking for.
• As always, think from your clients perspective about what would make it easier for them to notice and hire you.
• If you have more than one skill, split them into specialized profiles.

You can also browse through competing freelancer profiles to determine what is working in your niche. Don’t copy their title, but do use it to guide your own.

3. Use Emoji Bullet Points

Both Josh and Mike encourage the use of bullet points and/or emojis to break up paragraphs of text in your overview. Make it as easy for potential clients to skim your profile. They’re too busy to read through paragraphs of text. You can find all the emojis your profile might need here. Here’s an example from Josh’s current profile on UpWork:

4. Focus on the First 2-3 Sentences of Your Overview

Yes, your entire Overview is important, but the first 1-2 sentences are what appear in search results. Avoid using that valuable real estate to repeat your name or thank them for visiting like here:

While this freelancer is having a good amount of success on UpWork, they might be able to raise their rates by improving the cliche introduction to the Overview. Colton’s profile is a better example of using the introduction to establish credibility before a potential client even clicks on your profile:

You might also consider a customer-focused introduction like Nick R’s. It helps grab clients attention by starting a conversation and helps him stand out from the other search results:

1/3rd of jobs on UpWork are invite-only, so having a profile worth clicking on in search results can convert into more clients. Establishes that trust and makes yourself intriguing to searchers. Even if you’re newish to the platform, UpWork does incorporate some randomness into its rankings to give freelancers a chance to get noticed.

5. Proof, Specialization & How to Start

Here is the framework freelance developer and coach John Morris has for writing the rest of your Overview: proof, specialization and explaining your process.

Proof

John notes that very few of us trust marketers and whether you like it or not, your UpWork profile is a sales pitch. Try to overcome their doubts by showing potential clients you know what you’re talking about. Don’t just claim to be an expert or reliable, prove it with facts like years of experience, certifications, clients you’ve worked with and/or reviews. You shouldn’t wait until the portfolio section to establish credibility.

Specialization

Pick your speciality and don’t try and appeal to too many markets. Make it clear what you do and don’t do for clients. If you established credibility in the first section, now is the time to make yourself seem sought-after by having standards. Avoid sounding like 95% of other web developers by explaining your skills so that a non-technical person can understand them. Instead focus on the results and business goals you’ll deliver (start at 15:44):

Your Process

Clients are always nervous about getting started. Walk them through the steps of what they will need to do to begin working with you. Do they already have a logo or images for the website? How will they pay and when? Not only does it help ease their mind, it signals that you’re an experienced professional who’s done this before. Also, don’t forget to ask for the sale. Make sure you explain your process in a way that calls them to an action to get started.

John’s framework makes it easier for freelancers to tackle the intimidating process of writing an overview that converts proposals into clients. Remember, if hiring decision makers are searching on UpWork, they’re already in-market for services. They already know they need a web developer if they’re searching for one. Instead your profile should focus on establishing your credibility, qualify the ‘leads’ by explaining what it is you do and don’t do, and encouraging them to ‘covert’ by explaining the next steps.

Josh also recommends freelancers include the skills and search terms you want to be found for in your overview a few times. It helps you rank higher in search results (start at 14:33):

6. Your Portfolio

You don’t need 100 items in your portfolio, but John Morris suggests about ten. Don’t have too few or you may appear inexperienced. If you don’t have enough projects to adequately fill up the portfolio section, find some unconventional ways to build upon it by starting some self-directed projects yourself or approaching past clients for more work. If your portfolio doesn’t match your claims of experience in the overview, now the client won’t trust you and is likely to move on.

For example, I’ve only helped build one WordPress site recently and it was for my client Gennovacap, a team of app and web developers in Austin, TX. But including only that in my portfolio would not create the impression that I’m experienced enough to charge the rates I want to. Instead, I’m going to approach my client, whom I already have a relationship with, about assisting in Gennovacap’s web development projects. Then I should quickly have enough examples for my portfolio on UpWork.

Hear more of John's suggestions on building out your portfolio here (start at 22:48):

Remember, when it comes to web development, looks matter. Make sure your portfolio is visually impressive. If your portfolio looks good enough, you might not even need to have the best of overviews to land enough jobs on UpWork. Josh also recommends naming your portfolio projects with keywords you want to rank for in search results.

7. Close Your Contracts

Leaving contracts open with no activity will lower your Job Success Score on UpWork. If your clients want to work with you long-term, that’s great and actually increases your score. But don’t let them keep a contract open only to add more work later. Get the contract completed, reviewed and have the client invite you to apply to a new job (start at 10:55):

8. Job Categories

You selected your skills, but did you remember to set your job categories in UpWork? Navigate to your profile settings and scroll down until you find your categories. Pick the main and sub categories most related to your skills. This should help with your search rankings. Once again, here's Josh to explain more (start at 8:12):

So I hope these 8 tips help you find more success on UpWork as a freelance web developer. Many of the same principles could be applied to other freelance platforms like Fiverr or Freelancer.

What are your tips and tricks to making freelance work UpWork profitable?

P.S. I am not affiliated with UpWork or any of the featured freelancers in any way.

5 Newish Improvements to AWS Spot Instances

Ed Gray — Tue, 28 Jan 2020 19:42:45 +0000

If you use EC2, you’re probably already aware of the existence of Spot instances. Spot has existed for years, but like many AWS services, has dramatically changed over time. It’s Amazon’s way of keeping their servers busy by selling spare computing capacity at a 50% to 90% discount - then reclaiming that computing capacity when On-Demand customers need it again.

While the computing environment for Spot has always been the exact same as On-Demand, Amazon has been rolling out more and more integrations of Spot within other AWS services. Here are five ways Amazon has made it easier for your organization to take advantage of the cost savings of Spot instances:

1. EC2 Auto Scaling and EC2 Fleet features a new allocation strategy that can automatically mix in available Spot instances.

First, you configure the pools of Spot instances capable of running your workloads, then select the “lowest price” allocation option. This will spin up capacity only using the lowest-priced Spot instances among your chosen pools. EC2 Fleet also has a “diversified” strategy that will evenly spin up Spot instances among your chosen pools.

2. Batch workloads were always a natural pairing for Spot instances, but it’s even easier now with an option within AWS Batch.

If your job gets interrupted, Batch can automatically resubmit it onto a different Spot or On-Demand instance. With the cheaper compute provided by Spot, organizations are able to dramatically increase their processing power to accomplish more jobs, more quickly. To learn more, take a listen to the AWS Podcast discussion on Spot instances with Batch here.

3. The computing capacity demanded by Machine Learning is going to continue to grow exponentially. This can be cost prohibitive for many businesses, but recently Amazon added Managed Spot Training to SageMaker.

Now SageMaker can automatically checkpoint data when Spot instances are interrupted - and restart the workloads when the capacity becomes available again. The AWS Blog recently summarized how one AI company had saved 70% on ML training runs by taking advantage of the Managed Spot Training.

4. At re:Invent 2019, Amazon announced Fargate Spot for both Elastic Container Services and Elastic Kubernetes Services.

Fargate manages the provisioning and scaling of resources needed to run your containers - and now can do so with Spot to save up to 70%. Some workloads just aren’t fault-tolerant, but if they can restart after an interruption, then Fargate Spot is a way to significantly lower your AWS bill.

5. Amazon has made several changes over the last couple years to make it easier to estimate the costs and service interruptions of Spot:

• AWS recently announced launch events for Spot instances will now be available through the Amazon CloudWatch Events CLI. This will enable customers to better control when certain workflows are launched when a Spot instance becomes available

• Amazon no longer relies on a complicated bidding process to determine Spot pricing. Now your organization either pays the prevailing Spot price if the spare capacity is available - or it doesn’t. This has made the pricing significantly more stable and instead of bidding against other potential AWS customers, you simply tell AWS the maximum amount your organization is willing to pay.

• Engineers can easily research the price savings and interruption averages of every available instance type in every Availability Zone using the Spot Instance Advisor. For most instance types, it is the customer choosing to terminate Spot instances - not AWS - for more than 95% of the time. But for other instance types it can be AWS interrupting service more than 20% of the time, making them potentially unsuitable for certain workloads.

Spot instances may not be the best option for all workloads or business scenarios. But when you need to scale up your computing power at an affordable price, retooling your architecture to rely on Spot could be worth it. Looking to learn more about Spot instances? Here are a few great resources:

• Western Digital used AWS Batch with 40,000 Spot instances to complete 2.5 million HPC tasks in just 8 hours.

• Amiram Shachar’s podcast on SpotInst at Software Engineering Daily.

Five DevOps Practices to Improve Your Team's Performance

Ed Gray — Wed, 08 Jan 2020 19:02:47 +0000

DevOps isn’t the latest developer fad or programming trend, it’s essential to delivering superior products and services in a safe, effective and timely manner. Investing in DevOps is no longer an option given the frequent changes in customer needs, technology and security challenges.

Last summer, DORA released it’s comprehensive study on what are the common characteristics of organizations that have successfully adopted DevOps principles versus those that haven’t. The research is based upon 6 years of data from more than 31,000 professionals and was led by Google Cloud’s Nicole Forsgren. You can read the entire report here. You can also read Ivan Krnic’s summary of the State of DevOps Report on dev.to here.

While Ivan focuses on the recommendation that organizations adopt “Communities of Practice” method of adapting and improving DevOps, I am going to highlight some of the other recommendations.

1. The better an organization used the cloud, the more likely it accurately predicted software costs. (page 37)

Because it’s relatively easy to scale and control, organizations that reported using the cloud more effectively were 2.6 times more likely to accurately predict the cost of operating software. They were also 1.65 times more likely to stay under budget. Taking full advantage of the cloud’s elasticity and monitoring was much more important to an organization’s success than whether the cloud was public, private or a hybrid.

The cloud’s standard “pay for only what you use” model also encourages efficient coding. Investing in traditional server infrastructure and its fixed costs makes it less likely organizations will focus on efficiency.

2. A culture that encourages learning and improvement over blame significantly increases productivity. (pages 28 and 53)

If you’ve never failed, you’re probably not trying to do anything worthwhile. The DORA report found strong evidence that the best performing teams encouraged trust, risk-sharing, creativity and information sharing. Focus on fixing the system that led to failure, not an individual's mistakes. This isn’t about avoiding accountability, it’s about fostering an environment of psychological security so that your skilled employees are more productive.

3. External code reviews does NOT reduce risk. (pages 50 to 52)

A change process that requires approval from an external team was 2.6 times more likely to be among the worst performing organizations in the report. There is no evidence that a slower, heavyweight change process lowers risk. In fact, data tended to support the hypothesis that deploying larger batches of codes less frequently led to more failures, not less.

Instead, the findings suggest organization’s adopt a clear lightweight change approval process that relies on peer-reviewed code during the development process and automated testing. Automated tools were able to detect problems earlier in the development process than formal reviews. Respondents to the survey that had a clear understanding of the change process were 1.8 times more likely to be among the best performing organizations in the report.

4. The lowest performing organizations were the most likely to use proprietary tools. (pages 59 and 60)

Everyone knows investing in automated testing and deployment tools are fundamental to improving DevOps. But the reports findings suggest developing proprietary tools can be counterproductive. The report also found the best performing respondents were the more likely to use open source tools and planned to use more. Finding and contributing to the communities around commonly used tools helped developers feel more productive. (page 63)

5. Feeling productive reduces burnout. (pages 68)

It may be obvious, but it’s another reason to invest in DevOps: the more productive your engineers feel, the less likely they are to burnout. Burnout isn’t just being tired, it’s a recognized medical condition that can wreck an otherwise valuable employee’s ability to contribute. The less automated testing, self-guided learning and peer-driven review your organization supports, the more likely your skilled employees are going to have their time wasted, leading to frustration.

We noticed that ourselves at Gennovacap when working with an AI company. It wasn’t just that their cloud architecture didn’t allow for easy scaling, but deployments were crashing - and what is more frustrating than that? Our team of AWS DevOps engineers didn’t dramatically lower their AWS billing using Spot instances, but provide a stable development environment for their engineers.

Want to learn more about the 2019 Accelerate State of DevOps Report, but don’t want to read the 82 pages yourself? Then we recommend listening to the Arrested Devops podcast, where hosts Matty Stratton and Jessica Kerr talk with the report’s main author Nicole Fosgren and Jez Humble.