DEV Community: rainforss

How to add an icon to a D365 segment template

rainforss — Mon, 28 Nov 2022 07:10:02 +0000

As a D365 Marketing user, have you ever wondered why segment templates you created do not have icons like the system provided segment templates?

Find the imposter!

Although we can use filters, naming conventions and descriptions to locate the template we need, a visual hint is always good to have. Several clients and community members have asked about how to add custom icons to user created segment templates so I am going to demonstrate the simple steps to enable custom icons in this article.

Conceptual aside - the icon image we see on a segment template record is stored in an "Image" column of "Segment Template" table, named "Icon".

Image column of a table

A table can have multiple image columns but only one can be dedicated as a "Primary Image Column". In a normal table (without further customization), the picture stored in the primary image column will be the icon shown on a record. The tables without a primary image column will not have this icon on the table form.

Primary image column shown on the main form

On the segment template table, the image stored in primary image column will be shown directly on the segment templates list view, tile view and main form.

List view

Tile view

Main form

Now to the actual process. I mentioned steps, but it is actually one step - simply left click on the icon on the main form and a dialog will pop up for you to view or edit the image. It is so simple that this step did not even cross my mind. Previously, I was under the assumption that the primary image column would have to be added to the main form for users to modify. However, Megan told me about the simple approach which made me rethink about solutioning for any requirements or challenges - there always might be an out-of-box approach to look for before thinking about a customized solution.

A simple click and you can change the icon

Voila! Now you have the ability to add visuals to help other users understand what your segment template does. Just remember that any "Image" column data can only be populated after a record is saved, not when a record is being created.

Some additional words for the adventurous type of people: being able to add more visuals is fantastic, but the image column should be used sparingly since it will eat up the allocated file storage within an environment if a lot of tables are configured to have image columns. Only use the image column when it is necessary and make sure you set a reasonable max image size when creating an image column. Otherwise, always use less expensive solutions such as SharePoint integration.

The maximum image size can only be configured when the column is being created, not afterwards

I hope that this article is somewhat useful to you and feel free to reach out if you have any questions. Happy marketing!

How to generate dynamic email signature when sending in D365 Marketing

rainforss — Wed, 09 Nov 2022 01:16:36 +0000

When sending emails in D365 Marketing app, it is always a good practice to include some personalized content in the email for a special and lively touch.

Recently, a lot of clients have the requirement to take the personalization further and include an email signature of the salesperson (or marketing person). In my opinion, it is a wise move because a email signature can make the communication feel less "automated". However, do make sure that "reply-to" address points to an accessible email box in case a recipient replies to the email directly since you don't want to miss out on a customer's inquiry.

The logic of this implementation is quite simple: when sending an email to a contact during a customer journey, append the email signature of the owning user for the contact record at the end of email body. If you are no stranger to Dynamic Content, you might already have an idea of the implementation details. However, I do want to bring up one caveat of dynamic content before discussing the details of my solution.

According to Microsoft Docs -

Dynamic content gets resolved just before a message is sent to a specific individual. You'll typically use dynamic content to merge information from the recipient's contact record (such as first and last name), to place special links, and to place information and links from the content settings.

The caveat is that although you can populate dynamic content using information of records related to the recipient's contact record, you cannot traverse more than one relationship down the path from the contact table. For example, you can use a contact's parent account name to seed dynamic content because account table is directly related to contact table in D365 (only one relationship away). On the flip side, it is not feasible to use the parent account's territory information in dynamic content since territory table is only directly related to account table (two relationships away from contact table). Even if you use the dynamic content syntax like "contact.contact_account_parentcustomerid.territoryid.name", it will not resolve to the name of territory in the actual email sent. This is also specified in a Microsoft documentation:

Since my example has 3 hops (periods) in the expression, it will not resolve according to Microsoft.

What does it imply? You might be tempted to use the out-of-box table Email Signature (if you are not familiar with Email Signature, check out this amazing article by Allen Paige) to populate dynamic content for an email but it is not a viable solution since the email signature table is not directly related to the contact table. You could customize the table to add a direct one-to-many relationship to the contact table but this is very wrong, semantically. How is an email signature of a system user directly related to a contact?

To work around this impedement, we could use the first name and last name fields in combination as dynamic content but this approach falls short if you want to include additional information ,format the email signature or attach written signature pictures.

Signature with formatted text and picture

Are we stranded here? Rich text field to the rescue! When creating a new column to store text in Dataverse tables, we are given the format option of "Rich Text".

Column creation with rich text format

When "Rich text" format is chosen together with data type of "Multiple lines of text", the data stored in this field will be plain HTML and the field can be edited with rich text control when using placed on the system user table form.

Rich text editor on a form of model-driven app

When adding images using the rich text control, make sure to use the "Web Address" option instead of "From File".

Attach images with web address only

Since image uploaded to D365 will only be available in the D365 environment but not be publically accessible on the internet, resulting in broken images when recipients open the email.

Broken image since image can only be accessed within D365

Once the record is saved, the signature is stored as plain HTML which can be injected into marketing emails through dynamic content using this syntax: "contact.owninguser.hsi_emailsignature". Since emails are built with HTML behind the scene, the signature's HTML will be injected as a part of the whole HTML body, which will be rendered by email engines as user interface.

Let's perform a test send and see the final result:

I used a relatively large dimension for image here so the signature might not look appealing

Now you have learnt how to add some advanced personalization to your marketing emails, congratulations!

How NOT to manage your D365 Marketing instances

rainforss — Sun, 06 Nov 2022 21:16:47 +0000

Transferring data and configurations between environments might not be as simple as it seems if instances of D365 Marketing are involved.

Although Microsoft's official documentation for D365 Marketing App states that Configuration Migration tool can "move validated journeys, emails, and other content from a sandbox to a production environment", the "other content" piece may easily fool implementers into thinking that cotent like marketing pages and images can be moved with ease, which is often not the case.

In this article, I am going to illustrate why creating all pre-approved content in a sandbox environment with D365 Marketing app might introduce a lot of issues for deployment to production.

Let's start with one of the simplest issues, which just happens to be one of the easier-to-resolve issues.

If you have a live marketing page in the test instance of D365 Marketing app, it would be hosted on the Powerapps Portal installed with marketing application. Looking at the full page URL of this marketing page, you would notice the development related domain:

Marketing page of development environment

Once this page has been migrated to another environment with D365 Marketing app, the full page URL would still be the same - pointing to the Powerapps Portal of the source environment. This happens because the Configuration Migration tool simply copies all data to the target environment.

Marketing page migrated to another environment

This is an issue because the page URL would clearly indicate that the page is in a development environment and this page would cease to exist once sandbox Powerapps Portal is disposed of. There is a simple resolution to the issue: stop this live marketing page and go live again (go live directly if page is in "Draft" mode), the system will do a self-correction and publish this marketing page under the Powerapps Portal of the new environment.

Page URL after re-publishing

Now you might be wondering - this does not seem like a real issue at all if I can resolve by republishing the marketing page! However, if there are hundreds of landing pages (due to localization and other considerations), this process needs to be automated on all the pages using a Power Automate Flow and there are many other aspects to be considered - for example, system performance impact.

Generally speaking, this is indeed a relatively small issue considering the minimal disruption to system operation when it is being resolved.

Onto the one of the complicated issues: image records copied over to the target environment using Configuration Migration Tool are only stored as references to images of the source environment.

An image uploaded to D365 Marketing using the user interface will be saved to a private directory (for each unique environment) of a Microsoft owned server. Once this process is complete, the image will be cached through a content delivery network and it can be served through the internet with unique URLs. This URL information is then saved with a record of "Image" table in D365.

File upload user interface

When image records are migrated to another environment using Configuraiton Migration Tool, the URL information is simply copied over and the image records are merely referencing the URL of images in the source environment. The following picture shows a list of image record, of which the first one is created using upload while the rest are migrated from the source environment:

Note the different identifier of server file directory

Again, this wouldn't be an issue until the source environment (which is usually a development environment) is disposed of. However, if that happened eventually, all marketing emails, forms and pages built using the images will show broken images as the cached images are removed from content delivery networks and server directories as a whole and they are no longer accessible through internet.

To work around this issue, you need to gather all the images uploaded to the source environment and bulk upload them into the target environment using the out-of-box image upload functionality. Doing so will publish images to a new sub-directory of content-delivery network for the target environment. Unfortunately, all images being used on marketing content such as marketing emails and marketing forms have to be re-embedded to ensure all the marketing contents are referring to the correct image URL. As you can imagine, the effort required to work around the image reference issue grows exponentially with the number of images and the number of contents referencing the images.

As of today, there isn't a streamlined and automated solution of migration marketing content between D365 Marketing instances but you may not need a migration solution most of the time.

A D365 Marketing app contains both configuration data and marketing content. Configuration data such as content settings has no difference from other D365 app's configuration data, which can be migrated between environments using Configuraiton Migration tool without any issues. However, marketing content is just like the content managed on any CMS (content-management system). If you are familiar with any CMS like WordPress or SquareSpace, you know that contents are managed in only one environment, but with different statuses - published, draft, etc. The statuses are used to separate test/development content from anything publicaly available and D365 Marketing is using the same design to manage the content lifecycle.

In my opinion, all content which should be published or likely to be published in the forseeable future can be placed in the production environment. For example, marketing emails and marketing pages can be created before production go-live and kept in the draft status. Some content may be created in a development enviornment to test customizations added to D365 Marketing app and integrations with other applications, but the solution manager should always bear in mind that content in development is disposable.

Having a solution/environment management and deloyment strategy is key to the success of a project but not everything has to be included in the solution management and deployment pipeline.

How to use D365 Marketing form templates the right way

rainforss — Thu, 18 Aug 2022 06:23:53 +0000

The form template is possibly one of the most under-appreciated or under-utilized tools of Dynamics 365 Marketing while its immediate descendant, marketing form, plays a critical role in digital marketing.

After some tinkering with form templates, a good portion of D365 Marketing users or even consultants would most likely give up trying to create scalable templates and seek help from the front-end developers. This is mostly due to the fact that element styles of the a form template would not be applied to new fields added if configured using the form designer.

Here is a default form template which you would see in any D365 Marketing instance:

Very basic, but consistent-looking

If any other types of fields are added to the form template (or the form created using the template), this somewhat consistent form template will break.

The dropdown field, text-area input and the radio buttons cannot be styled using the form designer so people without the necessary HTML/CSS toolset will be stranded. One can try to get some assistance from a developer what if help is not available? Trying to get your dev team's attention to update the border color is probably not very efficient and surely would not be appreciated by the devs.

Luckily, there is a solution so that you need to ask your developer for favors only once. The solution is documented here in MS Docs about D365 Marketing but I will give a quick run down since certain parts of the document might not be very clear.

Usually, users would need to apply styling changes to a certain type/group of form elements to keep the form consistent. For example, it makes more sense to change the color of all input labels instead of only one. To achieve this, you would need a custom attribute to manage color of all input labels.

The following part is for people with HTML/CSS knowledge, simply skip this part and pass it to your development resources

Go to the HTML editor of the form template and add one meta tag as follows:

<meta type="xrm/designer/setting" name="label-color" value="#66b6e3" datatype="color" label="Label Text Color">

Use '-' instead of empty space in the name attribute since this will be referenced in form's CSS. The value attribute can be any hex color and setting "datatype" attribute equal to "color" will enable a color-picker on the form designer UI. Finally, the label attribute is what will be shown on the form designer.
Next part is slightly trickier. You need to locate the label elements in HTML body and inspect the direct parent elements (or one more level up) so that you can write CSS selectors to apply styling to the target label elements.

The label element is wrapped by a div element with "lp-form-field" class and another div element with "marketing-field" class

Do not use a single generic CSS selector like below. The CSS created here will collide with existing CSS on the page where marketing forms will be hosted:

label {

}

Do not create a custom CSS class and add the class name to all label elements since new field labels added to the form using designer will not have the custom class name attached.
Be specific and target all label elements using the pre-existing class names. Note that we have to wrap the hex color attribute value with two comment blocks and the "label-color" text corresponds to the name attribute of the meta tag we created. The hex color code here also needs to be the same as value of the meta tag so they can be synchronized:

.marketing-field > .lp-form-field > .lp-ellipsis {
  color: /* @label-color */ #66b6e3 /* @label-color */;
}

Once the form template is saved, you will see a customizable field under "Properties" tab of the form designer:

Control of all label color is now enabled

Clicking on the colored box beside the input, you can start using the color-picker to modify the color of all labels directly.

As a demonstration, I have added all the following meta tags (and CSS) to enable multiple general controls designer would need.

<meta type="xrm/designer/setting" name="input-row-gap" value="2.5rem" datatype="text" label="Input Field Row Gap">
<meta type="xrm/designer/setting" name="label-input-gap" value="0.5rem" datatype="text" label="Label Input Gap">
<meta type="xrm/designer/setting" name="label-color" value="#66b6e3" datatype="color" label="Label Text Color">
<meta type="xrm/designer/setting" name="input-height" value="30px" datatype="text" label="Input Height">
<meta type="xrm/designer/setting" name="input-border-radius" value="3px" datatype="text" label="Input Border Radius">
<meta type="xrm/designer/setting" name="input-border-width" value="1px" datatype="text" label="Input Border Width">
<meta type="xrm/designer/setting" name="input-border-color" value="#ccc" datatype="color" label="Input Border Color">
<meta type="xrm/designer/setting" name="input-focus-color" value="#2b95b8" datatype="color" label="Input Focused Color">
<meta type="xrm/designer/setting" name="button-background" value="#38C6F4" datatype="color" label="Button Background Color">
<meta type="xrm/designer/setting" name="button-text-color" value="#000" datatype="color" label="Button Text Color">
<meta type="xrm/designer/setting" name="button-background-hover" value="#2b95b8" datatype="color" label="Button Hover Background Color">
<meta type="xrm/designer/setting" name="button-text-size" value="1.25rem" datatype="text" label="Button Font Size">
<meta type="xrm/designer/setting" name="button-position" value="center" datatype="text" label="Submit Button Position (start,center,end)">

Now the end users and consultants can apply stylings to groups of elements without always bugging the developers. The users will need the developers' help only when they need additional controls enabled and this will be a rare occation if the available controls are planned out in the beginning. When a new marketing form is created using this template, the custom controls will be immediately available on the form designer and styling changes will be applied to new fields added too.

This solution greatly improves consistency of marketing form stylings and efficiency of individuals involved so I would definitely recommend trying out.

Manage your Dynamics 365 Marketing Forms properly

rainforss — Sun, 14 Aug 2022 18:20:48 +0000

Dynamics 365 for Marketing is an advanced marketing automation solution which has seen a great rise in popularity since its unveiling. Because of the complex nature of marketing tools (online marketing, email marketing and real-time marketing) and the fact that D365 Marketing module being "young" in its product lifecycle, this marketing automation solution is deemed to receive more critics than other Dynamics 365 products which are more "mature".

Here is one of the most prominent critiques I have heard working with clients and other D365 Marketing functional consultants:

Customizable can go too far, very involved to develop what is needed. Requires investment in both time and resources. No one seems to know how to get it to work, even the so-called experts.

This pain point is very valid since I have seen a many poorly implemented, customized and managed D365 Marketing instances. In this article, I will use Marketing Forms - one of the most used marketing tools for lead capturing - to prove this point and demonstrate how you can properly implement and maintain D365 Marketing Forms for clients to avoid unnecessary frustration.

The topics discussed in the following section require some knowledge about D365 Marketing Forms

Generally speaking, marketing forms need to follow a client's branding principles so they can be embedded in the client's public facing website seamlessly. In this case, the marketing forms need to styled accordingly unless your client is willing to use a marketing form which looks like it is coming from a 1980's webpage:

If you have used the marketing form designer provided by D365 Marketing, you probably know about - or have already complained about its lack of configurability for styling. Yes you can configure the form's layout, font size, text alignment, text color and paddings for form elements but these functionalities can barely make a web form which is no where near perfect.

The form editor itself is enough make a basic form somewhat dynamic, but things like perfectly centering the reCAPTCHA block and styling the submit button cannot be done using the user interface

Even if you can let the skewed reCAPTCHA block and unstyled button fly, any new fields being added to the marketing form need to be styled individually using the editor user interface. As a functional consultant, you would probably expect that your time is better spent elsewhere.

Luckily, the form designer provides an HTML editor which can be used by web developers (or anyone who has enough knowledge about HTML and CSS) to customize web form appearances to whatever the client requires. However, this customization can also bring us to some very messy scenarios which usually lead to the first part of the quoted critique - "Customizable can go too far so the solution requires investments in both time and resources".

Let's say that you - a functional consultant - has gathered the branding requirements from the customer and added all the fields to the marketing form using the form designer. However, the basic form designer cannot achieve the fancy design of the marketing form so the task is passed to a front-end developer within your development team so the form can be properly styled using custom CSS. This seems like no big deal because every front-end developer (event the juniors) can write CSS and style the form, right?

Unfortunaly, this implementation is very concerning, coming from a solution design perspective.

CSS can be added inline of any HTML element and it can also be added between the style tags using class names. If developers on your development team are not following the exact same enforced coding patterns and class naming principles, this implementation can lead to very a bloaded and messy HTML/CSS code base for all the marketing forms available
If the marketing form styling needs to be updated following a client's feedback (which happens very frequently) and the task is assigned to another developer, he/she might not be able to follow what the previous dev has done and simply proceeds to add his/her own CSS styling to override the previous styles, which leads to more unnecessary lines of code
If a new unstyled field is added to the existig form or if there is another similar form which needs some custom styling, you and your developers need to rinse and repeat this painfully inefficient and messy process.

A new text field is dropped onto the form, see the inconsistency of field size?

If we go down this route, we are relying on a group of people's (from all senority levels) diligence just to implement and maintain ONE marketing form. Also, the functional consultants (without solid HTML/CSS experience) will ALWAYS rely on the developers for custom styling changes. Furthermore, If people made the code base a giant pile of spaghetti, it'll be impossible for technical supports to troubleshoot and often customers quote "no one knows what is going on, even the so-called experts",

What could be a more elegant solution? Make use of form templates and custom attributes to enable additional form designer options for non-technical D365 Marketing users.

If you look closely to the right, you can notice the all the additional configurations which cannot be done using out-of-box form designer

Sometimes we do criticize Microsoft for poorly documented features but this would not be one of those occasions: the custom attribute feature is documented here under MS Docs for Dynamics 365 Marketing.

To summarize the implementation process, one of your developers would need to add all the commonly used form elements (text fields, dropdown fields, radio input, etc.) to a marketing form template and enable custom configuration attributes such as the gap between each form field and the color of the input labels. This can be done by adding meta tags to the form template's HTML and changing related CSS properties to a templated format. If you are interested about the details of setup, I will release a guide in the future.

So what are the benefits of this implementation?

The types of form fields are definite and the client would only need certain level of control over the form fields (field gaps, button color, button color when hovering, etc.), all of which can be implemented by a developer at once on a marketing form template. This means that you have less dependencies involved so the client does not continuous support on small things like changing a button's background color.
Admittedly, this implementation does not completely eliminate the risk of poorly coded CSS, but it does alleviate the severity of the issue by having developers implementing the code base only once.
The custom configuration can be generic so that you or the client can simply change, for example, the color or font size of all input labels by modifying only one field. The styling change will also be applied to new fields being added to marketing forms.
The migration process to another environment is simle and streamlined. Simply copy the HTML of the marketing form template and paste it into a new marketing form template created in another environment and you can start using the template for new forms.

_Let's see it in action. _

When you change the configuration, for example, the label text color using the color-picker, all labels change color.

When a new form element is added, it follows the same styling configuration as existing fields.

This implementation makes it easier for anyone (even without knowledge of HTML/CSS) to control advanced styling attributes, removing the need of constant support from developers. Also, if a client complains to you about certain configuration not working, you know that it won't be some "mysteries" hidden in HTML/CSS of the form.

As always, try not to settle with a solution which "just works". Design a solution that makes everyone's job hassle-free and it is what a marketing automation solution supposed to do.

D365 Marketing Email and Website Tracking

rainforss — Tue, 03 May 2022 21:33:13 +0000

Marketing plays a crucial role in the expansion and growth of a business and there has been a rise in popularity of the marketing automation tools/services. Serving marketing content to your target audience through your website or marketing email is only the beginning of the customer journey and implementing a personalized marketing strategy based on audience's reaction is most likely a more important piece of the puzzle. This is why the big wigs - D365 Marketing from Microsoft being one of them - of marketing automation tool/service providers offer both website traffic tracking and email tracking in their product bundle.

As a marketing specialist, it would be a blessing to know your audience's interactions and engage new conversations accordingly with the help of these tools. However, there are caveats that you might not be aware of if you do not have enough knowledge about how tracking - especially email tracking - is accomplished.

Website interaction tracking is generally more capable than email interaction tracking since it is realized using Javascript - a script that runs in the browser. Remember the code snippet you were asked to embed in your website to enable website traffic tracking?

<script async src="https://www.googletagmanager.com/gtag/js?id=GA_MEASUREMENT_ID"></script>
<script>
  window.dataLayer = window.dataLayer || [];
  function gtag(){window.dataLayer.push(arguments);}
  gtag('js', new Date());

  gtag('config', 'GA_MEASUREMENT_ID');
</script>

That code snippet was a piece of Javascript code, or sometimes a reference to some Javascript code hosted on the internet. When someone loads up your website in the browser, the code snippet starts running in the browser and the script has methods called "event listeners" that monitors all kinds of interactions. The abundance of event listeners in Javascript language makes it possible that almost any interaction can be captured, for example, user clicks on any section of the web page or user scrolls up/down the page.

Being very powerful at capturing user interactions, the website tracking has one major limitation - the user is (or should be) anonymous most of the time. Now you might be asking, "What about service offerings like Visitor Queue"? Yes some services like Visitor Queue can provide a bit more details about individual visitors to your website since their Javascript tracking code looks up the IP address of the visitors and tries to match the IP address with a company or organization. In this case, you would know that someone in a "Foo" company might be interested in your marketing content but you still do not have concrete information about the 'someone' who visited your website. Also, this type of tracking may or may not be legal depending on the territory where your business operates in so ALWAYS make sure that tracking services you are using would not bring you lawsuits.

In short, website tracking can bring you very detailed information about visitors' interactions but not the individual visitor. Therefore, most of the website tracking services (for example Google Analytics) provide you with analytical data which can guide you to better design and organize your website content and properly funnel your visitors to a CTA button or form submission.

Email tracking would be the opposite of website tracking. You know the exact person (because you have the person's email hence the contact record) who interacted with your marketing email but you will not be able to know if a user clicks on a paragraph or a heading of your email content.

Looking at the email click map provided by D365 Marketing email insights, you can notice that only clicks on hyperlinks are tracked. If a heading in the email does not have a hyperlink embedded, you would not see the click interactions on the heading at all. On the other hand, this same interaction can easily be captured in website tracking. But why?

Since emails contain very confidential information to users, email clients DO NOT ALLOW any scripts embedded in the email HTML. Without the help of scripting, email tracking services are not able to capture any interactions without a work-around that being the use of hyperlinks. To illustrate, imagine if you have embedded a hyperlink About Us in the marketing email and this hyperlink is actually an HTML element that looks like this:

<a href="www.betach.com/about">About Us</a>

When you save this hyperlink in the email editor, the marketing automation tool will further encrypt the original url (www.betach.com/about) into another url that points to a proxy server (most likely the server of the marketing automation service). Have a look at the following screenshot, you can notice that the url actually points to a server 'svc.dynamics.com' with some unique identifiers provided.

When this link is clicked, a browser tab will open to send an HTTP request to this url where the marketing application server intercepts the request, looks up the contact record and actual url using the unique identifiers, increments the user click count by 1 in database and then redirects the page to the actual url: www.betach.com/about.

With the above setup, email tracking services can record recipients' click on hyperlinks without any scripts embedded in the email body.

One of my clients who had some background knowledge in HTML tried to embed an HTML link which contains a sales-rep's email as follows:

<a href="mailto:xxx@xxx.com">Contact your sales rep</a>

Once the marketing email was set up, he wondered why he could not use the 'click of contact your sales rep button' as a criteria to segment the customer journey. With previous explanation, this is now easier to interpret:

Although this HTML element is also rendered as a hyperlink in the email clients, a click on the link would only open up a new email in the default email client on your desktop with the recipient field pre-populated with the email address. Since this behaviour is completely irrelevant to the browser, no HTTP request is made to the proxy marketing server and the marketing server has no way of recording the click event. Most of the time, the marketing automation's email editor will disregard this type of hyperlink and will not encrypt the link href property at all.

Another caveat worth mentioning is that 'email open' events tracked by email automation services might not be accurate as you anticipated. The most popular method to track email opens is implemented by a pixel image embedded in the marketing email body. The pixel image itself is hosted, as you might have guessed, in the marketing automation service's own server. When the email is opened in an email client, the email client attempts to send an HTTP request to the marketing server asking for the pixel image when the marketing server intercepts this request and record the event as email being opened. Because of security concerns, major email clients such as Gmail have started enforcing the email to not make HTTP request to remote resources unless the user explicitly allows so by clicking a button to start fetching the resources. With the enforcement in place, a good amount of email opens could have been missed since users could read the email content without downloading the extra image files.

The key takeaway here is that marketing specialists utilizing both tracking tools should have a good understanding of the underlying mechanisms. Without a solid background knowledge of these tools, you could mix up the capabilities of them and go down a pitfall where things are not working as you expect them to and you would have a hard time troubleshooting.

Going serverless with custom portal for D365 environments

rainforss — Fri, 22 Apr 2022 08:23:21 +0000

In the first article of the series, I "briefly" addressed the pros and cons of Microsoft's Power Apps Portal and the reason why it might not be a cost-efficient product for companies at all scales who need to open a controlled "portal" to a specific piece of their D365 data.

The standalone web server which is supporting the Power Apps Portal plays a major role depicting the product as a non-optimal solution for smaller organizations. Most of the time, a business operating in a single territory (or territories in a close proximity) does not need a web server which continuously listens for incoming requests since the traffic could be very localized and patterned. For example, Calgary Canucks Rugby Union - a non-profit organization operating in Calgary - tries to expose some matches information stored in Dynamics 365 through Power Apps Portal. It is very likely that traffic would hit the portal some time right after the rugby matches and rapidly die down until the next match result is posted. During this quiet period between the matches, utilization rate of the portal web server would be very low so the server will stay in an idle state awaiting requests. Therefore, smaller organizations using Power Apps Portal are essentially paying for a web server that's idling for more than 70% of the time.

One can propose the use of a lower tier web server (like what Heroku offers for the free hosting) which enters a sleep mode if no traffic come in for a period of time, but I am going to take it further by adopting a serverless solution, removing a traditional server from the scene entirely.

Being a huge fan of NextJS (an opinionated framework based on ReactJS) and serverless functions, I decided to use NextJS to power the front-end of my custom portal and NextJS API routes (together with Vercel's serverless functions) to orchestrate my portal's back-end services.

In short, serverless functions are invoked functions running in servers offered by larger organizations (AWS, Azure, Google, Vercel, etc.) whose profession is in cloud computing. Using serverless functions to access other web services (e.g., making HTTP requests to remote servers), the front-end application is enabled to do its job (providing user interactions) without the need of a dedicated traditional server. With the self-owned traditional server out of the solution, smaller organizations can focus on delivering user-centric content instead of sparing resources on maintaining a web server.

Now you might be asking, how do serverless functions communicate with Dynamics 365 environments to retrieve the data for the front-end web application? Making use of Azure App Registration and a application user in Dynamics 365, the serverless functions can obtain access tokens issued by Azure Active Directory OAuth2.0 service. Adding the access token to HTTP requests made by serverless functions, the web application is now authenticated and authorized as the application user to perform CRUD operations allowed by security role configurations. This is just a high-level overview of the integration, feel free to ask if you want to know more about the details.

Using the serverless functions, I was able to create a custom D365 portal with even registration and authentication functionalities for Calgary Canucks Rugby: Calgary Canucks Rugby Union

How does serverless function save companies' money?

Serverless functions are on-demand services provided by cloud computing companies so you only need to pay for the functions being invoked. In the previously illustrated scenario, Calgary Canucks Rugby simply wouldn't have any expenditure when no traffic is hitting (hence no back-end services involved) the public facing website if serverless function was the back-end solution.

To make the serverless solution even better, companies such as Vercel are also offering free serverless function usage per month and it is actually very rare for front-end applications with less traffic to use up the monthly quota.

By adopting the serverless function solution for a custom Dynamics 365 portal, Calgary Canucks Rugby and other similar organizations can save up a good amount of budget for other essential operations simply because they wouldn't be paying for web servers and web server maintenance. Sorry "webmasters" and "IT administrators" :(

In case you are wondering, this article is not sponsored by Vercel or NextJS. I simply love Vercel and NextJS that much :)

Using Redis Cloud in your NextJS application

rainforss — Mon, 18 Apr 2022 01:10:02 +0000

Recently, a Youtuber and Javascript influencer FireShip released a tutorial of utilizing Redis Enterprise Cloud as a high speed cloud data storage in a NextJS web application and the tutorial attracted hundreds of thousand of views on the internet. Although I was on the fence about using an in-memory database as the main database for a full-stack application, the Redis Enterprise Cloud service offer - together with a $200 voucher - had me interested enough to try it as a runtime cache of a NextJS application I built for a product demo to York University.

Arriving at Redis Enterprice Cloud service offering page, I picked the free tier since I was not certain if I would start spending the $200 credit yet. The free service tier - as you would imagine for anything that is free - does not offer a lot: 30MB RAM for 1 dedicated database with 30 maximum concurrent connections.

30 maximum connections may not seem like an issue as long as you are not building an application that has specific requirements for concurrency. This statement could be true if we are establishing connections between a Node server and a Redis cache since it is recommended that only one or two Redis client would be instantiated then reused in the Node server. In this case, there is a limited number of connections (clients are connections in Redis) needed when the server is running and communicating with Redis.

However, things are very different if you are using Redis Enterprise Cloud in a NextJS application and having your backend services orchestrated using the Serverless Functions. If you do not understand the basics of NextJS and Serverless Functions enough, you would encounter this error and never understand the cause of it:

ERR max number of clients reached

If I am only connecting to Redis with one application, how is it possible that all 30 connections are occupied? Let's look at some code before I reveal the answer.

import { Client, Entity, Schema, Repository } from 'redis-om';

const client = new Client();

async function connect() {
    if (!client.isOpen()) {
        await client.open(process.env.REDIS_URL);
    }
}

This is a code piece copied from Fireship's tutorial. It instantiates a Redis client with the help of "Redis-OM" library and declares a "connect" function which can be used to establish a connection between the client and Redis. As the code specifies, the "connect" function will only attempt to open a connection if no open connections exist between the client and Redis.

const client = createClient({ url: process.env.REDIS_URL });

export const connect = async () => {
  if (client.isOpen) {
    console.log("Already connected to Redis");
    return;
  }

  await client.connect();
  console.log("Connected successfully.");
};

This is another code piece if the "Node-Redis" library is being used. There are slight differences but the logic is the same: open a connection only if there is no existing connection established between the client and Redis.

Either piece of code makes sense on paper here: since the code only gets executed once when the application is up and running, only one instance of client is created and the "connect" function will create the connection only once no matter how many times it is called, right?

Wrong. Remember that NextJS's Serverless Functions and Edge Functions are executed in a highly dynamic environment based on resource allocation, meaning the environment where the functions are executed will not be the same as the environment where your NextJS was built and deployed in (e.g. Vercel, Netlify, etc.). Therefore, when you are creating NextJS API routes (these routes are realized by serverless functions if the NextJS app is deployed in Vercel, Netlify or similar platforms) under the /pages/api folder to have them call the "connect" function and then interact with data in Redis, it is almost inevitable that every serverless function execution will be in a different environment where the connection to Redis has not been opened before. With this setup, each serverless function invocation because of API calls will attempt to create a unique connection to your Redis Cloud database. Theoretically, 30 invocations of serverless functions which try to connect to Redis could exhaust the allowed number of connections for free tier. To make things worse, Redis does not have a default setting for connection timeout so the connections created will be kept alive unless your client-side code does the clean up.

If you are already stranded with Redis Cloud having maximum number of connections, connect to Redis Cloud using RedisInsights and run the CLIENT KILL TYPE normal command to evict all the basic connections.

In a serverless NextJS application setup, it is imperative to always close the connection after your code finishes interacting with Redis data. To complete my tutorial, here is an example setup I am using to handle Redis connections and interactions in a NextJS app supported by serverless functions:

//Initialize the Redis client in global execution context to ensure uniqueness of the client
let client = global.redis;

//If no Redis client is found, create the client using Redis connection string
if (!client) {
  client = global.redis = createClient({ url: process.env.REDIS_URL });
}

//Open connection only when there is no existing connection
export const connect = async () => {
  if (client.isOpen) {
    console.log("Already connected to Redis");
    return;
  }

  await client.connect();
  console.log("Connected successfully.");
};

//Close connection only when there is an existing connection
export const disconnect = async () => {
  if (!client.isOpen) {
    return;
  }
  await client.quit();
  console.log("Disconnected.");
};

//Retrieve the cached data
export const getcache = async () => {
  const cache = await redis.get("cacheBetachSite");
  return cache;
};

The comments are pretty self-explanatory so I will not go into too much details here. If you are wondering how these helper functions are being used in API routes or NextJS SSG/SSR functions, here are two examples:

NextJS SSG/SSR functions

export const getStaticProps: GetStaticProps = async () => {
  try {
    await connect();
    const cachedData = await getcache();
    await disconnect();
    return {
      props: {
        cachedData
      },
    };
  } catch (error: any) {
    console.log(error.message);
    return {
      props: {
        error,
      },
    };
  }
};

API routes supported by serverless functions

export default async function demoRoute(
  _req: NextApiRequest,
  res: NextApiResponse
) {
  try {
    await connect();
    const cachedData = await getcache();
    await disconnect();
    return res.status(200).json(cachedData);
  } catch (error: any) {
    return res.status(500).json({ error: "Unknown error." });
  }
}

Being a part of the IT community, we have the blessing that one can always find tutorials or documentations about a variety of technologies and tools, however, tutorials or documentations might not always fit into each individual's own use case seamlessly.

Be adventurous and diligent, always do your own recon before following others' footprints.

Is the Power Apps Portal a viable option as your customer facing website?

rainforss — Sun, 17 Apr 2022 04:40:03 +0000

Microsoft has always been promoting the Power Apps Portal (previously Dynamics Portal) as a "no-code" solution for external facing websites.

Together with the naming change, the pricing model of this Microsoft product has also seen significant changes.

Prior to 1st October 2020, a customer had to purchase a portal with a flat one-time payment and that pricing model was deemed less profitable since the major clients' portals had very high traffic volume of unauthenticated and authenticated visitors. From 1st October 2020, Microsoft has decided to adopt a consumption-based pricing scheme which makes customers pay monthly fees based on the number of page views and login sessions.

The following screenshot demonstrates the updated consumption-based pricing model:

If you found this pricing model confusing (which I did), this amazing article by Ben Hosking gives crystal clear descriptions about the updated pricing model, together with some beneficial background knowledge and thoughts about the portals.

Although the pay-as-you-go service model may seem cost-effective for consumers who have lower traffic volume and less registered portal users, bigger customers might not enjoy this change since they were always paying the same amount regardless of the amount of traffic hitting the portal. To make the situation even worse, there are some caveats that portal owners are required to pay at the beginning of the month and the purchased consumption amount (pageviews and logins) DOES NOT roll over to the next month if not used up.

Let's consider the following scenario to understand the grievance your customers might potentially have: A university is using a Power Apps Portal to manage student/staff/alumni logins and the IT department is managing the portal monthly payment based on the averaged number of logins per month. The employment center of the university is holding a career fair in the coming month which will result in a huge influx of authenticated users. However, not knowing the pricing model of Power App Portals, the employment center does not notify the IT department of the event and the greatly increased amount of logins would cause service disruptions of the portal. Furthermore, assuming the employment center learnt a lesson and notified IT department to increase the purchased amount of logins, there could be instances where the actual logins are way under the anticipated amount. With the purchased quota not being able to roll over, this would potentially result in thousands of purchased logins in vain.

The key takeaway of the previous example is that a customer can experience difficulties determine the proper amount of logins/pageviews and that can lead to either unforseeable service disruptions or wasted budgets. While this issue might not be as severe for bigger organizations since they could work with statistical data to make a decision on a payment plan and there is more expenditure budgetd, medium to small sized customers would find it hard to manage the traffic fluctuation.

There are many other scenarios we can examine to understand the concerns of choosing Power Apps Portal as your solution but I would leave that to future posts. I will quickly summarize my considerations and thoughts of features offered by Power Apps Portal in this article as follows:

Low-code or no-code solution to configure a website

This could either be a curse or blessing based on different situations. With the low-code or no-code solution, one can quickly start a "okay" public facing website using templates and basic configurations without much technical aptitude. Here is a basic demo portal landing page:

However, if more interactions are desired to make the website "lively" (see a "lively" custom coded website and a Power Apps Portal configured by North52 professionals), the use of HTML/CSS/Javascript coding is inevitable. Now you are facing a condumdrum of getting a developer to add custom code to a low-code/no-code platform which he/she likely has very little idea about (unless you can find a technical person who is proficient at both coding and Dynamics 365).

Microsoft takes care of the scaling aspect of Power Apps Portal

Scalability is a bitter-sweet problem to have (meaning that you are getting more traffic on your web application) and most of the organizations do not have the luxury or opportunity to ever have to worry about this problem. Moreover, the majority of public facing content is usually static (meaning it is likely to be stale for a long period of time) and this type of content can be cached to content-delivery networks (CDN), rendering the scalability unnecessary since the content is not served directly by web server when requested.

Of course there are way more aspects to look into but to summarize, the auto-scaling feature can certainly benefit large organizations while other organizations probably wouldn't make use of the feature at all.

Power Apps Portal can have a sophisticated security hierachy to manage different level of access.

Again, having a nested security configuration can bring in benefits or hassles based on the complexity of content you have on the external facing website. For the customers that I have previously worked with, only the large scale organizations are utilizing a complicated security role configuration while the relatively smaller ones only have two security levels - authenticated and unauthenticated.

Therefore, it is likely that you will be paying for features that are rarely used if you are a small to medium sized customer making use of Power Apps Portal.

Power Apps Portal is using a pay-as-you-go pricing model.

Based on the pricing model, the monthly cost to larger companies who have high traffic volume to the portals could be high and the cost will add up over time. However, these companies have higher budget than others so they can enjor all the benefits offered by Power Apps Portal while not being too concerned about the cost.

From the perspective of companies which do not have a good amount of visitors on external facing website, the monthly cost could look promising if we are aiming for under 100 logins and 100,000 pageviews per month. Does it mean that Power Apps Portal is a better choice than having some developers code a web application with integration to Dynamics 365? The answer is not definite.

You might be asking, "what's the catch?" Firstly, you definitely need to spend thousands if you are trying to have developers develop your website but the website's monthly cost will be much lower comparing to Power Apps Portal's. This means that a custom built website could still be the winner if we are talking about a long run. Secondly, moving away from Power Apps Portal once you are onboard will be a gruesome task because you have no access to the source code of Power Apps Portal.

There are way more features to be discussed but I will stop here for the sake of reader's sanity. I will potentially address more considerations about other features in the future.

In conclusion, I am not depicting a bad image of Power Apps Portal though it might seem that I am questioning certain aspects of Power Apps Portal. It is actually a solid product offering in the market when compared to other competitors such as Salesforce Portals. Nonetheless, this type of product offering might or might not be the optimal solution for everyone who is seeking an established public facing web application. Make your decision after careful considerations.

I sincerely hope that you would find this article inspiring or intuitive (at least part of it). I am happy to answer any questions in the comment or have some more discussions :)

So what would be a better solution if a web application which can securely surface data from Dynamics 365 environment (DataVerse) is desired? I will have this covered in another post, stay tuned.

The solution stack consists of NextJS, Vercel Serverless Functions, Redis Cloud and Dynamics 365 Web APIs.