DEV Community: Shuaib

Eliminate Clutter and Clean HTML Fluff in Microsoft Word Generated HTML Documents, specifically for Emails

Shuaib — Fri, 25 Jun 2021 01:20:23 +0000

This post is a distillation of lessons learned during reverse engineering an email template and making it somewhat editable. By the end of this post, you will learn how to tame a Microsoft Word-generated HTML template so that you can use it for your purposes.

Perhaps you had created a Microsoft Word document and used the HTML code generated by it and lost the original Word document and want to make changes in pure HTML code. Or you are in an unfortunate position where you don't have access to the original Word document but need a cleaner version of HTML code. Or you just want pure clean HTML code because it is a matter of principle. Or it's just a fascinating intellectual exercise. You could also just be using Word to create simple web pages or landing pages but know that having a cleaner and more semantic version of your HTML code will help you rank better on search engines as well as reduce page loading times for your users. Whatever the reason, you want clean HTML.

People at work asked me to automate an email that we need to send periodically. Of course, I said this won't be a problem, and it wasn't supposed to be. I asked them to provide an HTML of the email they'd like to send and I'll add some dynamic text and send it off.

Turns out the task I thought would be the simplest wasn't that simple.

The template I received was sent by email on Microsoft Outlook. There were some simple styling issues such as inconsistent fonts for some of the text which I thought would be a simple fix. When I viewed the source of the email, I was horrified to see it filled with junk I'm not usually used to. I attempted to employ some of my basic CSS skills to resolve these and thought I don't need to optimize the template as it's doing its job and no one is going to care about viewing the source as long as it renders beautifully. I didn't want spend time to manually handle CSS quirks since I'm not that great at CSS and I know I will get stuck in a back-and-forth loop of fixing minor quirks here and there which weren't that important.

In the first iteration, I made some minor changes as used online tools like https://html-cleaner.com/ to clean up some of the mess, as long as the formatting still staying intact.

In the second iteration, I attempted to redo the entire template by creating a CSS grid to ensure the right layout using https://grid.layoutit.com/. This wasn't too bad. I spent perhaps 20 minutes defining the grid. Considering I also learned this grid tool on the run, it was fairly quick.

This is the HTML it created:

CSS Grid Layout HTML

And this is the CSS that I was blessed with, good thing I don't have to spend too much brains on what's going on in here as long as it works.

CSS Grid Styles

I then manually populated the template with my desired content and basic CSS styling and was able to create a modern well-formatted HTML email template from scratch that was I proud of.

Until I tested sending an email with the template in Outlook. That's when I discovered that many email client software like Microsoft Outlook can't render modern CSS well. I spent a good few hours on this, but with no meaningful result.

Back to the drawing board.

Finally, I gave it another crack with HTML tidy. This is the tool the cleaned up the HTML well enough to maintain the semantic elements enough that a human can easily interpret them and make changes to the CSS. However, there is a trick to this that I will reveal shortly.

If you're tech-savvy, you can download and install the tidy utility on Windows, Linux, or Mac. Or you can use an online version of the tool at https://infohound.net/tidy/ (I've done both).

infohound.net/tidy

Select the 'Word 2000' option and click Tidy!

You'll get this page that shows the warning as well as cleaned HTML. Copy and paste that into JSBin.com. jsbin.com is a free web-based editor to quickly make and preview changes you make to an HTML document.

Just look at how clean that HTML is on the left and a clean version of it in the preview on the right.

And this the clean HTML that was generated. How clean!

But there is one problem that you may have noted. All the formatting is gone. Do you want to do all the formatting by hand? Well yes and no. There's no 100% automated way to do this that I know of, but you can get some of the styles back and then tweak them by hand.

But this requires dealing with an issue with tidy. See when you run tidy with the --clean y and --word-2000 y options selected, there are CSS style rules generated that preserve some of the original formattings, but are not embedded in the document. I don't know why, I assume it's a bug.

But if you first run the document with only the --clean y option, you do get a CSS style tag as below:

Now you copy the CSS styles and then run it with the --word-2000 option. Then copy the styles generated the first time manually to your document, you'll get something better.

You may not see it, but the footer text is bold. As I said it's not ideal, but with the additional CSS rules available you have a little more to start with. It didn't work entirely great in this fabricated example, but my original email worked fairly well with the background colors coming through. It's a hit and miss.

If you like this sort of reverse engineering and would like to get updates on random interesting stuff I'm working on and what I'm learning along the way, give me a follow on Twitter @shuuabe.🐤

Setting up Cron jobs to run bash scripts

Shuaib — Sun, 05 Apr 2020 16:04:14 +0000

The setup of cron jobs is fairly easy. However, I ran into problems running shell scripts using that create files as outputs. This post covers some of the gotchas to keep in mind.

How to setup Cron jobs

To setup a cronjob, you use a command called crontab. If you execute crontab -l you will see a list of currently installed cron jobs. Jobs are defined in a text file using crontab syntax that you can find or generate at https://crontab-generator.org/. Each line is a job that will be executed. To add/modify/delete a job run crontab -e and perform the deed.

Running a job as a root user

If you want to run a job as a root user, run sudo crontab -e to define a job. Once you save the crontab file, you will see an output on the console saying crontab: installing new crontab which confirms you newly defined changes have been configured.

Ensure you shell script is running with the right shell and environment variables

When you normally run a shell script, it runs under the context of your user profile's shell settings. So it knows which shell executor to use, the programs that are available in your PATH environment variable etc. However, when you run the same script with crontab, you may have a very different context/environment variables. It's best to specify these explicitly so that others and your future self can understand your state of mind and thinking if they every look at it.

You can determine the correct shell location by running which bash to get the shell location (in my case its bash on Ubuntu but you may have a different shell. Similarly, if you are using other programs such as docker, you can run which docker to get the exact file path of the program. Then in your shell script, instead of just using docker you specify the exact docker file you want the script to run. The alternative is the set environment variables in your script to keep the configuration dynamic, it all depends on your situation and use case.

Specify absolute paths in outputs

If your script is creating outputs, its a good idea to specify these in absolute terms. There is some confusion around what is the working directory where the files created by scripts run by crontab are placed. You should explicitly CD into the directory you'd like to work in. Here are a few cron jobs you can run to get an idea of you environment.



#Identify what in the path variable in the context of your script
* * * * * echo $PATH > ~/cron.log

#Identify the shell being use in your script
* * * * * echo $SHELL > ~/cron.log

#Identify the working directory your cron job natively executes in
* * * * * echo $PATH > ~/cron.log

View the outputs of these logs by inspecting the cron.log file the logs are output to cat ~/cron.log.

Make sure your script is executable and has the right permissions

Set a script to be executable by running chmod +x myscript.sh

Set the correct ownership by running chown myusername: myscript.sh

Inspect cron job runs

Run the following to see if the cronjob you've defined actually runs.



sudo grep CRON /var/log/syslog

Are there any other common mistakes you ran in to as a beginner when defining cronjobs to run shell scripts or cron jobs in general. Let me know in the comments.

Setting up Drone CI to work with self hosted Gitlab Repository

Shuaib — Sat, 04 Apr 2020 17:07:27 +0000

I have already setup Gitlab CI to build docker images which I'll be covering in a separate post. I may not have posted it to dev.to but you'll be able to find some material if you search on my blog.

I had successfully setup Drone CI to work with Gitea. However, I ran into trouble getting it to work with a Gitlab repository. Most of the problems result from SSL certificate configurations and proxy configurations in a self-hosted environment.

First off, once drone was connected to Gitlab repositories, it was able to sync the repositories. The problem came when I attempted to activate the repository for automated builds in Drone. I got the message failed to activate your repository and check this. Following the steps in that post did not resolve the issues for me. As always, I had a suspicion this is something related to communication between Gitlab and Drone not working because of SSL certificate not being configured correctly. Setting the DRONE_LOGS_DEBUG=true in my drone server and inspecting the logs gave some insight.

The problem was that Drone CI was attempting to create a webhook via Gitlab API that will be triggered everytime a commit occurs on the repo to trigger a build in Drone. However, it was not getting a response. This is related to a Gitlab setting that prevents Outbound requests from Gitlab. The solution is described here. You have to enable requests to the local network from web hooks and service.

Once this is done. You should be able to activate the repository in Drone. However, we're not done yet.

Even if the Omnibus docker distribution of Gitlab has the custom SSL certificates mapped, these are not applied to the webhooks (or atleast I haven't figure out yet how to make them be used by Gitlab webhooks for verification). Also by default, the project specific webhook created by Drone in Gitlab to trigger builds has SSL verification enabled by default. This results in the error ad depicted below.

The simple solution is to disable SSL verification at this point if you're OK to do that in your environment.

The last part you need to ensure is that all the required runners required for your projects have been configured correctly. For the docker image build and deploy setup, I require a docker-exec-runner and a docker-runner.

The final drone configuration for building a docker image and deploying via docker-compose on the same host is as follow. Deployment on other hosts could be achieved by tweaking the commands to login to remote host via ssh and perform the same actions.

kind: pipeline
type: docker
name: build
#cd
steps:
- name: publish
  pull: if-not-exists
  image: plugins/docker
  settings:
    registry: registry.mydomain.net
    username:
      from_secret: username
    password:
      from_secret: password
    repo: registry.mydomain.net/my-project
    tags: 
      - latest
    insecure: true
--------
kind: pipeline
type: exec
name: deploy

# Cloning the respository in deploy stage is necessary to copy docker-compose.yml
# clone:
# disable: true

steps:
- name: deploy
  environment:
    USERNAME:
      from_secret: username
    PASSWORD:
      from_secret: password
  commands:
  - echo $(pwd)
  #Be careful with command below to not overwrite an existing project
  - mkdir -p /home/docker/services/my-project
  - cp docker-compose.yml /home/docker/services/my-project/docker-compose.yml
  - cd /home/docker/services/my-project
  #Notice below the synatax for using environment variables source in the environment command above
  - docker login -u $USERNAME -p $PASSWORD registry.mydomain.net
  - docker-compose pull && docker-compose up -d
depends_on:
  - build

The Dockerfile that will be used in the build step must reference an image from the private repository if it does not have access to public docker repository.

FROM registry.mydomain.net/nginx:alpine
COPY /build /usr/share/nginx/html
EXPOSE 80
CMD ["nginx", "-g", "daemon off;"]

Add disqus comments and search functionality to your ghost blog

Shuaib — Thu, 12 Mar 2020 13:28:20 +0000

I have been meaning to add commenting functionality to this blog for a while because I am really interested in hearing feedback from readers on what they like and what can be improved. Adding disqus commenting facility to my site had been on the to do list for a while and I finally tackled it today. It was much quicker and easier to setup than I had expected and primarily just required setting up some configuration options. I have also changed the theme for the site to Attila which I came across reading another blog last night. The theme itself introduced me to the client-side search functionality via ghostHunter.

Detailed and clear setup instructions are provided in the links above so I'm not going to repeat them here but I will lay down high level steps that you will be able to follow:

Setup a disqus account and make note of your disqus short name that we will require for configuration.
Via the ghost integrations > custom intergrations, create a new custom integration called ghostHunter and copy the Content API Key
Clone the attila theme repository for GitHub from command line and cd into the directory /assets
Clone the ghostHunter repository as mentioned in the setup on the ghostHunter GitHub readme page.
Zip the resulting attila theme folder and upload it via Ghost Admin web interface under the Design section. Activate the theme.
Go to the Code integrations page an copy and paste the script tags mentioned in the attila GitHub repository readme in the Ghost Blog Header section. Add the disqus short name and Content API Key from earlier steps as relevant. Save changes.
Go to your blog home page. You will notice the search icon on the top right. Play with the search functionality to make sure it works. Navigate to one of the pages and go to the bottom of the page and verify that you have disqus section visible. Click the 'Show Comments' button to see that comment input form appears.

That's it. Everything should be working correctly. The whole setup will take 15 to 30 mins. I setup all this and wrote this blog post within my 1 hour train ride.