DEV Community: Darío Kondratiuk

UI testing with Puppeteer book officially released!

Darío Kondratiuk — Fri, 12 Mar 2021 19:13:08 +0000

I'm so thrilled to announce that my book UI testing with Puppeteer has been published!

You can find it on Amazon.com and PacktPub.com. You will find all the details on the book website. Special Thanks to Facundo Giuliani for helping me set up a Jamstack website in a few hours.

Why writing a book?

When Kaustubh Manglurkar from PacktPub reached me out asking me if I wanted to write a book, I didn't know what to say. I wasn't (and I don't consider myself) a writer. But Kaustubh told me something that turned my doubts into an absolute yes. He told me that there was no book about Puppeteer in the market.
That put extra pressure on the project. If I was going to write the first book about Puppeteer, it should cover most of the Puppeteer API.

I can make a connection with TV shows like House or Grey's Anatomy, where every chapter has its own plot, but there is an underlying story going on. The same happens with this book. Each chapter covers a topic, but the underlying story is the Puppeteer API.

What you will find in the book

This book will help you get started not only with Puppeteer but also with UI testing. Each chapter begins with the basic concepts before getting into the code. You will learn about automated testing in general. You will learn what test runners are and what you should expect from them.

In Chapter 4, where you will learn to interact with the page, but you will also learn about CSS selectors and XPath. You will even see a little bit about JavaScript closures in Chapter 7. Chapter 8 is about environments emulation, but I open the chapter talking about the internet ecosystem.

This book is also for web developers!!

In chapter 2, I talk about how web developers can use web automation as part of their workflow to test their UI components. In Chapter 7, We see how to generate content with Puppeteer. Then in chapter 9, I talk about web scraping and scraping ethics. The last chapter is about performance, which is something we should all care about.

Filling your toolbox

This book will give you more than a complete picture of API and UI testing. Many chapters include a bonus track, where you will learn about a tool that will be helpful in your automation journey.
These are the tools you will see in this book:

Github Actions.
VS Code debugging tools.
Checkly.
Checkly's headless recorder.
Google Lighthouse.
Puppeteer-Cluster.

Special thanks

It would be impossible to mention all the people involved in this project. I would miss many for sure. But I want to thank all the PacktPub crew, in special Kaustubh Manglurkar, for making all this possible and my editor Sofi Rogers who basically taught me to write, not just a book, to write :). You are the best Sofi!

The job of the technical reviewers was impressive! David Rubio Vidal and Jim Munro took the book to the next level.

Last but not least, thanks to my family and friends who supported me all these months. I love you all.

Final words

Maybe you are wondering, "but Dario, how about .NET? How about Playwright? weren't you working on Playwright?'" Well, if you are interested on seeing this book on .NET or Playwright, don't hesitate to send an email to PacktPub!

One final and personal thought. One thing I learned in this process is that writing a book is hard!!, really hard. It's easy to be mean on social media for many of us when we don't like something, and sometimes we forget to be thankful, even if we paid for the book we read.
To all tech authors around: thank you for your hard work!

Playwright Sharp joins the Microsoft family

Darío Kondratiuk — Thu, 24 Sep 2020 00:00:00 +0000

It is my pleasure to announce that today Microsoft forked Playwright Sharp!

Wait, did you give the project away?

No, I didn't. Because it was never mine.

It's our project

Playwright Sharp uses the MIT License. That means that Playwright Sharp is mine, and it's also yours, and it's also Microsoft's. It's ours.

You can do the same! You can fork the project and make it better.

Playwright team rocks!

They could just copy my code and ship it with a different name. But instead, they opened the doors of their team and embraced the Playwright Sharp project.

I think Playwright Sharp fits perfectly in the vision the team has for the product.

But, why?

I think it's what needs to be done to take this project to the next level.
The .NET community needs to grown in their relationship with the open-source. Aaron Stannard explained very well on his post The Next Decade of .NET Open Source.

When a .NET team needs to add a third-party tool to their toolbox, the management evaluates the long term support and the final decision tends to be: "I'd better use a tool from Microsoft than from a random Argentinian guy".

We developers also have some responsibility. The biggest risk for a .NET developer involved in open-source is Microsoft creating a similar project. And it's not Microsoft fault at all. But developers would pick Entity Framework over Nhibernate, the default ASP.NET DI over Windsor Castle, etc.

I love how Microsoft embraced open-source and became a big player in the open-source community. But I think the .NET community has a lot to learn yet.

Another reason to move the project is that, despite the relative popularity of Puppeteer-Sharp, I wasn't able to build a stable team.
My hope with this move is to serve way more developers compared to Puppeteer-Sharp and also get more people involved in the project.

Remember, it's not my project. It's OURS.

Are you leaving the project?

Hell no! The roadmap will now be aligned with Playwright, which will be great for Playwright Sharp. Arjun Attam is an awesome PM, and I hope he can help me take Playwright-Sharp to the next level.

The future

My dream is to make Playwright-Sharp a first-class automation tool for the .NET community, and there is no better partner than Microsoft to accomplish that.

Give some love!

So, Playwright Sharp has a new home: https://github.com/microsoft/playwright-sharp. Let's go there and give it some love ⭐️.

Happy automation!

Don't stop coding!

My programming language is the best!

Darío Kondratiuk — Wed, 16 Sep 2020 00:00:00 +0000

I've been porting Puppeteer code (to puppeteer-sharp) from javascript to C# since late 2017 and Playwright code (to playwright-sharp) since early 2020.

One of the things I learned is that programming languages are not about computer instructions, but human interaction.

Migrating code from Javascript/Typescript to C# is not about translating statements. It's about figuring out how C# developers express what a Javascript developer tried to communicate.

Each programming language has a community, a background, influencers, and frameworks that shape how programs are written.

We can't deny that javascript was born to solve simple HTML interactions. And we can't deny that C# was born as a new language to the Visual Basic community.

In the same way that spoken languages follow a community, their habits, and their way of living. Programming languages follow a programming community.

What makes a language good or bad is also biased. We consider French romantic because of French music. We might also consider other languages aggressive because that's the language of bad guys in Hollywood movies.

In the same way, we might consider some languages cooler than others because of influencers and Twitter hype.

My programming language is the best because it represents my community and how we communicate with each other.

The next time you get in touch with a language you are not familiar with, instead of thinking if it's bad or good, try to understand from that community and embrace this diversity we have in the industry.

Don't stop coding!

PlaywrightSharp v0.10 for Firefox is here!

Darío Kondratiuk — Fri, 19 Jun 2020 00:00:00 +0000

Firefox joined to the PlaywrightSharp family!

What can I do with PlaywrightSharp.Firefox?

You can do lots of things.

Seriously!

You should be able to do almost everything you would do with his older brother PuppeteerSharp or his younger brother PlaywrightSharp.Chromium.

This is how you can take a screenshot in PlaywrightSharp NOW!

var options = new LaunchOptions
{
    Headless = true
};

Console.WriteLine("Downloading Firefox");
var firefox = new FirefoxBrowserType();

await firefox.CreateBrowserFetcher().DownloadAsync();

Console.WriteLine("Navigating google");
await using var browser = await firefox.LaunchAsync(options);
var page = await browser.DefaultContext.NewPageAsync();
await page.GoToAsync("http://www.microsoft.com");

Console.WriteLine("Taking Screenshot");
File.WriteAllBytes(Path.Combine(Directory.GetCurrentDirectory(), "microsoft.png"), await page.ScreenshotAsync());

Console.WriteLine("Export completed");

Is there anything I can't do?

There are only two features not implemented on PlaywrightSharp.Firefox: PDF generator and Tracing.

Can I use this package in Production?

As I mentioned when we launched Playwrightsharp.Chromium, If you start using PlaywrightSharp now, the API is going to change, guaranteed, but you will jump into a project that will have way more support and new features.

PuppeteerSharp doesn't support Firefox yet. So I believe PlaywrightSharp.Firefox, even in its 0.10 version, is the best library to automate Firefox in .NET.

Final words

Now that we shipped our first version, I'm looking forward to getting more feedback. The issues tab in GitHub is open for all of you to share your ideas and thoughts.

Don't forget to follow me on Twitter @hardkoded to get more updates.

Don't stop coding!

Implementing interprocess communications between .NET and Node.JS

Darío Kondratiuk — Thu, 11 Jun 2020 00:00:00 +0000

Introduction

If you want to implement interprocess communication, you will end up reading about pipes and file descriptors. The problem I found is that .NET does a great job hiding all these concepts from us. I don't know about you, but I didn't even know much about these concepts until I needed to implement some interprocess communication.

Motivation

Most headless browsers, e.g. Chromium, have the ability to communicate with a parent process, whether using WebSockets or Pipes.

The problem is that I was never able to implement pipes on Puppeteer-Sharp, and I really want to implement it on Playwright-Sharp.

So let's see how far are we from getting this done.

Communicating two Node.JS processes

Implementing pipes in Node.JS it's pretty simple and straight forward.

Let's say I have a parent and a child app. When the parent app creates the child app process, it will be able to set up the child's stdin (File descriptor 0), stdout (File descriptor 1) and stderr (File descriptor 0), but not only that, it will also be able to create more file descriptors, which can be used as an extra set of pipes.

const server = childProcess.spawn(
  'node',
  ['../child/index.js'],
  {
    stdio : ['inherit', 'inherit', 'inherit', 'pipe', 'pipe']
  }
);

In this call, we are not only setting up stdin, stdout, and stderr, but we are also creating two pipes using the file descriptor 3 and file descriptor 4.

Let's create two small scripts to test this. We will write a parent app which will read an input from the terminal, it will send it to the child app, and the child app will echo it.

The parent app would look something like this:

const readline = require('readline');
const childProcess = require('child_process');

// Create child process
const server = childProcess.spawn(
  'node',
  ['../child/index.js'],
  {
    stdio : ['inherit', 'inherit', 'inherit', 'pipe', 'pipe']
  }
);

// We will use the File descriptor 3 as a writer and the File descriptor 4 as a reader.
const reader = server.stdio[4];
const writer = server.stdio[3]; 

const rl = readline.createInterface({
  input: process.stdin,
  output: process.stdout
});

// When we get data on the reader we print it
reader.on('data', data => console.log(data.toString()));

// We are going to send a message to the child
var waitForMessage = function () {
  rl.question('', (message) => {

    if(message === 'exit') {
      rl.close();
    }
    writer.write(message + '\n');
    waitForMessage();
  });
};

waitForMessage();

The child app will be super simple:

const readline = require('readline');
const fs = require('fs');

// We will create an stream reader from the file descriptor 3
const reader = fs.createReadStream(null, {fd: 3});

// And a writer from the file descriptor 4
const writer = fs.createWriteStream(null, {fd: 4});

// When we get a message on the reader we echo it in the writer
reader.on('data', data => writer.write('echo: ' + data + '\n'));

// This is how a .NET developer leaves an app open in Node.JS
setInterval(()=> {}, 1000 * 60 * 60);

So far, so good. It works

Now, let's get to the important part. Can I connect a .NET app there?

.

.

.

.

No, you can't. It's not possible to add more streams on a Process instance in C#. You will find a way better explanation here.

So, our next and final test would be trying to create a .NET app, create new pipes in there, and then passing those file descriptors as an argument to the child app.

The child app would be quite simple. We replace the hardkoded descriptors 3 and 4 with command line arguments:

const readline = require('readline');
const fs = require('fs');
const reader = fs.createReadStream(null, {fd: process.argv[2]});
const writer = fs.createWriteStream(null, {fd: process.argv[3]});

reader.on('data', data => writer.write('echo: ' + data + '\n'));


setInterval(()=> {}, 1000 * 60 * 60);

Now, let's go to the .NET world. We know there is something called Anonimous Pipes.
Let's see if that works:

static async Task Main(string[] args)
{
    // We are going to create two pipes, one writer and one reader.
    using var pipeWriter = new AnonymousPipeServerStream(PipeDirection.Out, HandleInheritability.Inheritable);
    using var pipeReader = new AnonymousPipeServerStream(PipeDirection.In, HandleInheritability.Inheritable);

    // We create a child process passing the pipes handles as string.
    Process client = new Process();

    client.StartInfo.FileName = "node";
    client.StartInfo.Arguments = "../../../../Childwitharguments/index.js " + pipeWriter.GetClientHandleAsString() + " " + pipeReader.GetClientHandleAsString();
    client.StartInfo.UseShellExecute = false;
    client.Start();

    // If microsoft docs tells me to call this method I will.
    pipeWriter.DisposeLocalCopyOfClientHandle();
    pipeReader.DisposeLocalCopyOfClientHandle();

    // We start listening to messages
    _ = StartReadingAsync(pipeReader);

    // We create a stream writer, and we will write messages on that stream.
    using var sw = new StreamWriter(pipeWriter)
    {
        AutoFlush = true
    };

    string message = Console.ReadLine();

    while (message != "exit")
    {
        await sw.WriteAsync(message);
        message = Console.ReadLine();
    }

    client.Close();
}

private static async Task StartReadingAsync(AnonymousPipeServerStream pipeReader)
{
    try
    {
        StreamReader sr = new StreamReader(pipeReader);

        // This method should get a CancellationToken so we use that instead of a while true.
        // But this will work now.
        while (true)
        {
            var message = await sr.ReadLineAsync();

            if (message != null)
            {
                Console.WriteLine(message);
            }
        }
    }
    catch (Exception ex)
    {
        Console.WriteLine(ex);
    }
}

Final Words

It was so cool to find that we can implement this kind of solution between both frameworks.
I know that implementing pipes is not something most of us need in our daily job, but you know, maybe someday you will need to connect to a browser using these pipes 😉.

If you want to take a look at the source code, you can find the repository on Github.

Don't stop coding!

Originally posted on harkoded.com

I'm a Microsoft MVP!

Darío Kondratiuk — Tue, 02 Jun 2020 00:00:00 +0000

I'm super excited to share with you that yesterday I have received the Microsoft MVP Award for the very first time!

I think the timing is perfect. Last year, I prepared a talk called "You can also be a hero". I was able to give that talk twice: At Age of Mobile in Buenos Aires (big thanks to Esteban and Leo) and vOpen UY (thanks to all the vOpen crew!). This year I will give the English version for the first time at HolyJS, Russia!!! (online 😞) and at the WebConf in Córdoba.

I think it's all about that. It's about being little heroes where you are. Did I dream about this award? Yes, sure! But I never worked for the award. I worked for the community, to give back, to be part of this amazing community. I won't talk more about this, that's what the hero talk is about 😜.

I want to thank Nico Milcoff for opening this door and guide me through the process, Thank you Nico!

I hope this award opens more doors to keep helping what I consider one of the best communities in the world: The developer's community.

Pushing your Docker images to Docker Hub

Darío Kondratiuk — Fri, 30 Aug 2019 16:14:46 +0000

In my previous post we created an image to get Puppeteer-Sharp running on Docker. Let's see if we can publish that image to Docker Hub and make that image available to the community.

I already have a user on Docker Hub, so I bet the first step is done.

I want to create one repository with two tags:

puppeteer-sharp-base:1.0
- puppeteer-sharp-base:1.0-sandboxed

If we go to the Create Repository page, we can see that we should be able to create a Github repository and make Docker build our images from there. We LOVE that kind of stuff, don't we?

So let's create our repo!

I'm going to commit there two images:

A base image without the user creation.
A sandboxed image, which will start from the base image and it create the user after that. I will put this file inside a "sandboxed" folder.

Time to go back to Docker Hub...

We are going to create two sets of builds:

One for the latest version, using the master branch.
One for tagged versions.

And we are going to build two images:

puppeteer-sharp-base:<version>
puppeteer-sharp-base:<version>-sandboxed

Let's Create and Build!

Now, we can create a 1.0 release on Github.

And voilà! We have a v1.0 on Docker!

Finally, the moment of truth, let's see if I can replace all the code I had y my previous Dockerfile and replace it only with the new image.

FROM hardkoded/puppeteer-sharp-base:latest

COPY bin/Release/netcoreapp2.1/publish/ /app/
ENTRYPOINT ["dotnet", "/app/PuppeteerSharpPdfDemo-Local.dll"]

Boom! We have our new image working!

Final Words

I hope this journey can help you to setup your own images, and also to start using Puppeteer-Sharp on Docker.
As you can see, I'm not a Docker expert. If you are, and you found something off on my post, please let me know!

Don't stop coding!

Originally posted on harkoded.com

Running Puppeteer-Sharp on Docker

Darío Kondratiuk — Fri, 16 Aug 2019 11:34:59 +0000

I get many questions about running Puppeteer-Sharp on Docker. Let's see if we can get a:

Let's take a look at the example provided by Puppeteer and see what we need to change there to make it work.

FROM node:10-slim

# Install latest chrome dev package and fonts to support major charsets (Chinese, Japanese, Arabic, Hebrew, Thai and a few others)
# Note: this installs the necessary libs to make the bundled version of Chromium that Puppeteer
# installs, work.
RUN wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add - \
    && sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list' \
    && apt-get update \
    && apt-get install -y google-chrome-unstable fonts-ipafont-gothic fonts-wqy-zenhei fonts-thai-tlwg fonts-kacst ttf-freefont \
      --no-install-recommends \
    && rm -rf /var/lib/apt/lists/*

# If running Docker >= 1.13.0 use docker run's --init arg to reap zombie processes, otherwise
# uncomment the following lines to have `dumb-init` as PID 1
# ADD https://github.com/Yelp/dumb-init/releases/download/v1.2.0/dumb-init_1.2.0_amd64 /usr/local/bin/dumb-init
# RUN chmod +x /usr/local/bin/dumb-init
# ENTRYPOINT ["dumb-init", "--"]

# Uncomment to skip the chromium download when installing puppeteer. If you do,
# you'll need to launch puppeteer with:
#     browser.launch({executablePath: 'google-chrome-unstable'})
# ENV PUPPETEER_SKIP_CHROMIUM_DOWNLOAD true

# Install puppeteer so it's available in the container.
RUN npm i puppeteer \
    # Add user so we don't need --no-sandbox.
    # same layer as npm install to keep re-chowned files from using up several hundred MBs more space
    && groupadd -r pptruser && useradd -r -g pptruser -G audio,video pptruser \
    && mkdir -p /home/pptruser/Downloads \
    && chown -R pptruser:pptruser /home/pptruser \
    && chown -R pptruser:pptruser /node_modules

# Run everything after as non-privileged user.
USER pptruser

CMD ["google-chrome-unstable"]

FROM

We need to change the base image. Instead of:

FROM node:10-slim

We will use:

FROM mcr.microsoft.com/dotnet/core/runtime:2.1

The minor version shouldn't matter here. We could use 2.0, 2.1 or 2.2

Puppeteer Recipe

So this is their setup recipe:

# Install latest chrome dev package and fonts to support major charsets (Chinese, Japanese, Arabic, Hebrew, Thai, and a few others)
# Note: this installs the necessary libs to make the bundled version of Chromium that Puppeteer
# installs, work.
RUN wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add - \
    && sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list' \
    && apt-get update \
    && apt-get install -y google-chrome-unstable fonts-ipafont-gothic fonts-wqy-zenhei fonts-thai-tlwg fonts-kacst ttf-freefont \
      --no-install-recommends \
    && rm -rf /var/lib/apt/lists/*

# If running Docker >= 1.13.0 use docker run's --init arg to reap zombie processes, otherwise
# uncomment the following lines to have `dumb-init` as PID 1
# ADD https://github.com/Yelp/dumb-init/releases/download/v1.2.0/dumb-init_1.2.0_amd64 /usr/local/bin/dumb-init
# RUN chmod +x /usr/local/bin/dumb-init
# ENTRYPOINT ["dumb-init", "--"]

# Uncomment to skip the chromium download when installing puppeteer. If you do,
# you'll need to launch puppeteer with:
#     browser.launch({executablePath: 'google-chrome-unstable'})
# ENV PUPPETEER_SKIP_CHROMIUM_DOWNLOAD true

# Install Puppeteer, so it's available in the container.
RUN npm i puppeteer \
    # Add user so we don't need --no-sandbox.
    # same layer as npm install to keep re-chowned files from using up several hundred MBs more space
    && groupadd -r pptruser && useradd -r -g pptruser -G audio,video pptruser \
    && mkdir -p /home/pptruser/Downloads \
    && chown -R pptruser:pptruser /home/pptruser \
    && chown -R pptruser:pptruser /node_modules

# Run everything after as non-privileged user.
USER pptruser

Here's where all the fun begins. If we run that setup using our Docker image we will get this:

/bin/sh: 1: wget: not found
E: gnupg, gnupg2 and gnupg1 do not seem to be installed, but one of them is required for this operation

I found that we need this before running that setup:

RUN apt-get update && apt-get -f install && apt-get -y install wget gnupg2 apt-utils

We also need to remove some Node stuff. Let's remove the npm i puppeteer and && chown -R pptruser:pptruser /node_modules.
That would leave our user setup like this:

RUN groupadd -r pptruser && useradd -r -g pptruser -G audio,video pptruser \
    && mkdir -p /home/pptruser/Downloads \
    && chown -R pptruser:pptruser /home/pptruser

Well, at least we are building our image now.

Notice two important things on this recipe:

We are downloading Chrome

If we take a closer look, we can see that we are already downloading Chrome there:

&& apt-get install -y google-chrome-unstable fonts-ipafont-gothic fonts-wqy-zenhei fonts-thai-tlwg fonts-kacst ttf-freefont \

We don't need to use BrowserFetcher in our app. This is important because BrowserFetcher is not going to download a valid version to be used on Docker.

So, how do we tell Puppeteer-Sharp to use that Chrome?

Easy, we can set an environment variable:

ENV PUPPETEER_EXECUTABLE_PATH "/usr/bin/google-chrome-unstable"

We are creating a new user.

We are not going talk about the --no-sandbox on this post. You will find many posts on the internet about the goods and bads of this flag. You can also take a look at the official doc.

But what you need to know here, is that we have two ways of setting up your image:

With --no-sandbox

If you are ok adding the --no-sandbox flag on your app, because you will browse a website you own or trust, you can remove all the user setup.

All this will be out:

# Add user, so we don't need --no-sandbox.
# same layer as npm install to keep re-chowned files from using up several hundred MBs more space    
RUN groupadd -r pptruser && useradd -r -g pptruser -G audio,video pptruser \
    && mkdir -p /home/pptruser/Downloads \
    && chown -R pptruser:pptruser /home/pptruser

# Run everything after as non-privileged user.
USER pptruser

And we'll need to add the --no-sandbox flag on our app.

var launchOptions = new LaunchOptions
{
    Headless = true,
    Args = new[]
    {
        "--no-sandbox"
    }
};

Without --no-sandbox

If you don't want to use the --no-sandbox flag, you will need to keep the user setup. But if you try to launch Chrome you might get this error:

Failed to move to new namespace: PID namespaces supported, Network namespace supported, but failed: errno = Operation not permitted

You will find many, many posts talking about this.

I found the solution on this post.
We'll need to run docker using the --security-opt=seccomp:unconfined

What else?

After doing all that setup, we need to do a normal .NET deploy to that Docker. You can copy the source code and make the publish inside the image or copy an existing publish there, just like this:

COPY bin/Release/netcoreapp2.1/publish/ /app/
ENTRYPOINT ["dotnet", "/app/PuppeteerSharpPdfDemo-Local.dll"]

Final Solution

This is how our new Dockerfile looks like:

FROM mcr.microsoft.com/dotnet/core/runtime:2.1

#####################
#PUPPETEER RECIPE
#####################
# Install latest chrome dev package and fonts to support major charsets (Chinese, Japanese, Arabic, Hebrew, Thai and a few others)
# Note: this installs the necessary libs to make the bundled version of Chromium that Puppeteer
# installs, work.
RUN apt-get update && apt-get -f install && apt-get -y install wget gnupg2 apt-utils
RUN wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add - \
    && sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list' \
    && apt-get update \
    && apt-get install -y google-chrome-unstable fonts-ipafont-gothic fonts-wqy-zenhei fonts-thai-tlwg fonts-kacst ttf-freefont \
      --no-install-recommends \
    && rm -rf /var/lib/apt/lists/*

# Add user, so we don't need --no-sandbox.
# same layer as npm install to keep re-chowned files from using up several hundred MBs more space    
RUN groupadd -r pptruser && useradd -r -g pptruser -G audio,video pptruser \
    && mkdir -p /home/pptruser/Downloads \
    && chown -R pptruser:pptruser /home/pptruser

# Run everything after as non-privileged user.
USER pptruser
#####################
#END PUPPETEER RECIPE
#####################

ENV PUPPETEER_EXECUTABLE_PATH "/usr/bin/google-chrome-unstable"
COPY bin/Release/netcoreapp2.1/publish/ /app/
ENTRYPOINT ["dotnet", "/app/PuppeteerSharpPdfDemo-Local.dll"]

Demo

Does it work?

Let's code that PuppeteerSharpPdfDemo-Local console app:

class MainClass
{
    public static async Task Main(string[] args)
    {
        using (var browser = await Puppeteer.LaunchAsync(new LaunchOptions()
        {
            Headless = true
        }))
        using (var page = await browser.NewPageAsync())
        {
            await page.GoToAsync("https://www.hardkoded.com");
            Console.WriteLine(
                await page
                    .QuerySelectorAsync(".page-subheading")
                    .EvaluateFunctionAsync<string>("el => el.innerText"));
        }
    }
}

Let's publish it:

dotnet publish PuppeteerSharpPdfDemo-Local.csproj -c Release

Build the docker image:

docker build --tag hardkoded/simple-docker-demo:v1.0.0 .

And run it!

docker run --security-opt=seccomp:unconfined -it hardkoded/simple-docker-demo:v1.0.0

Final words

I hope this post helps the community to start using Puppeteer-Sharp on Docker. I will see if I can publish these images to the Docker Repository, stay tuned! :)

Don't stop coding!

Originally posted on harkoded.com

Automating Microsoft Edge with Puppeteer-Sharp

Darío Kondratiuk — Tue, 23 Apr 2019 14:13:14 +0000

As you may have heard, Microsoft made available to the public its Chromium-powered Edge browser. Maybe this question won't qualify as technical interview question but:

If Puppeteer-Sharp automates Chromium, and Microsoft Edge (insider) is powered by chromium, that would mean that...

When you call Puppeteer.LaunchAsync, one of the values you can set in the LaunchOptions is the ExecutablePath.

So, what would happen if we launch Puppeteer passing our Microsoft Edge browser path?



var browser = await Puppeteer.LaunchAsync(new LaunchOptions
{
    Headless = true,
    ExecutablePath = "C:\\Program Files (x86)\\Microsoft\\Edge Dev\\Application\\msedge.exe"
});

Let's write a super simple example:



var browserOptions = new LaunchOptions
{
    Headless = false,
    ExecutablePath = "C:\\Program Files (x86)\\Microsoft\\Edge Dev\\Application\\msedge.exe"
};

var browser = await Puppeteer.LaunchAsync(browserOptions);
var page = await browser.NewPageAsync();

await page.SetContentAsync("<div>Testing</div>");

Voilà!

And after a few tweaks in puppeteer-sharp, and one similar change in puppeteer, we managed to get all tests running using Microsoft Edge!

So we can say that Puppeteer-Sharp v1.14 is fully compatible with Microsoft Edge Insider version 75.0.131.0.

Don't stop coding!

Originally posted on harkoded.com

Creating a WhatsApp bot with Puppeteer-Sharp

Darío Kondratiuk — Fri, 12 Apr 2019 11:35:32 +0000

Once upon a time

Not enough friends? Get a bot! was my second post on this blog. I was learning some Machine Learning back then. I found that many devs were using Marcovify and I thought it would be fun to create a bot to interact with my friends. But I found that it was not so simple to create a bot for WhatsApp. You need to set up a business account and pay for it. So I built it on Telegram.

There is one problem with Telegram. It wasn't our primary chat app. We all love it, but we use WhatsApp...

So well, our bot was there. Hosted on my local docker, so we could go to Telegram once in a while to have fun with it.

One guy automating VS Code

A few days ago I found a video where Jarrod Overson was automating VS Code using puppeteer! How cool is that?

After watching the video, I was like

If he was able to automate VS Code because it was an electron app, I should be able to do that with the WhatsApp app.

But then I realized that:

I don't need to hack electron. WhatsApp has a WebApp!!!

Let's get to work!

The bot

We want to create a basic chatbot. It would wait for a trigger word and respond accordingly.

Let's take a look at the WebApp.

This is what we need to do:

Open that page.
Search for a Group or a Person.
Select that Group or Person.
Start listening to messages.
Type a message when needed.

Exploring the page

I need to know how to search for a group, click on that group and type a message. DevTools is our best friend for this task.

After exploring the DOM, we found that:

All the page is inside a #pane-side div.
The search chat has a jN-F5 class.
Each person or group in the list has a _2wP_Y class.
The chat input is an editable DIV with a _2S1VP class.
The send message button has a _35EW6 class.
There is a DIV containing all the messages and it has a _9tCEa class.
Each message line is a DIV with a vW7d1 class.

Code time!

We are going to create a regular console app, using the Puppeteer-Sharp NuGet package, of course!

First of all, let's create a class so we can put there all CSS classes we found in the previous section:

internal class WhatsAppMetadata
{
    public const string WhatsAppURL = "https://web.whatsapp.com/";
    public const string MainPanel = "#pane-side";
    public const string SearchInput = ".jN-F5";
    public const string PersonItem = "._2wP_Y";
    public const string MessageLine = "vW7d1";
    public static string ChatContainer = "._9tCEa";
    public static string ChatInput = "._2S1VP";
    public static string SendMessageButton = "._35EW6";
}

Now, let's work based on the To-Do list we've made before:

Open the page.
Search for a Group or a Person.
Select that Group or Person.
Start listening to messages.
Type a message when needed.

Open the page

First, we need a browser.

await new BrowserFetcher().DownloadAsync(BrowserFetcher.DefaultRevision);
_browser = await Puppeteer.LaunchAsync(new LaunchOptions
{
    UserDataDir = Path.Combine(".", "user-data-dir"),
    Headless = false
});

We need a UserDataDir, so we know where we are going to store our data (a.k.a. cookies, localStorage, etc.)

We are setting Headless in false mainly because we need to scan the QR code with our Phone the first time. We can set that to true afterward.

Now, let's navigate to the page.

_whatsAppPage = await _browser.NewPageAsync();
await _whatsAppPage.GoToAsync(WhatsAppMetadata.WhatsAppURL);
await _whatsAppPage.WaitForSelectorAsync(WhatsAppMetadata.MainPanel);

.WaitForSelectorAsync(WhatsAppMetadata.MainPanel); will wait until the WebApp is loaded, this is a good time to scan the QR code if needed.

CommandLineParser

Before searching for a person or group, let's add the CommanLineParser package. I love this package when I need to code reusable Console Apps. It will not only create an instance of a class based on the arguments but also validate all those arguments.

This is the BotArguments class:

public class BotArguments
{
    [Option('t', "trigger", Required = true, HelpText = "Trigger word.")]
    public string TriggerWord { get; set; }
    [Option('c', "chat", Required = true, HelpText = "Chat name.")]
    public string ChatName { get; set; }
    [Option('r', "response", Required = true, HelpText = "Response template.")]
    public string ResponseTemplate { get; set; }
    [Option('l', "language", Required = true, HelpText = "Language.")]
    public string Language { get; set; }
    [Option('f', "file", Required = true, HelpText = "Source text file.")]
    public string SourceText { get; set; }
}

This is how my Main method looks like:

static async Task Main(string[] args)
{
    await Parser.Default.ParseArguments<BotArguments>(args).MapResult(
        async (BotArguments result) => await LaunchProcessAsync(result),
        _ => Task.FromResult<object>(null));
}

Now that we have our BotArgument class let's go back to our app.

Search for a group or person

Based on the person we got as an argument we can do this:

var input = await _whatsAppPage.QuerySelectorAsync(WhatsAppMetadata.SearchInput);
await input.TypeAsync(args.ChatName);
await _whatsAppPage.WaitForTimeoutAsync(500);

Pretty cool ah?
We query for an element, we type on it, and then we wait just a little bit for the DOM to be refreshed.

Select the item

If we assume that the person we are looking for will be the first one on our list, we will know that it will be the second item on that list, because "CHATS" will be the first item.

Now that we know that, we can do:

var menuItem = (await _whatsAppPage.QuerySelectorAllAsync(WhatsAppMetadata.PersonItem)).ElementAt(1);
await menuItem.ClickAsync();

We query all the items on the list, and we select the second one.

Start listening to messages

This is the fun part, and I think this something you will like to learn.
How can we start listening to new messages?

We will need two things:

A callback function on our side.
A DOM observer on the Browser/Javascript side with the ability to call our function.

ExposeFunctionAsync to the rescue

ExposeFunctionAsync will help us register a C# function on the Chromium side.

await _whatsAppPage.ExposeFunctionAsync("newChat", async (string text) =>
{
    Console.WriteLine(text);

    if (text.ToLower().Contains(args.TriggerWord) && !text.Contains(args.ResponseTemplate))
    {
        await RespondAsync(args, text);
    }

    text = text.Replace(args.ResponseTemplate, string.Empty);
    await File.AppendAllTextAsync(args.SourceText, text + "\n");
});

Done!
Now we have a function called newChat in Javascript.
When newChat is called we will:

Log the message.
Check if the message contains a trigger word.
Check that the message doesn't contain our response template (a.k.a. a message we just sent)
The last two lines are not so important now. But what they do is saving that message in a File, so we have more content to be able to create new messages.

Listen to new messages

If ExposeFunctionAsync was our best friend on the C# side, MutatorObserver will be our best friend on the Javascript side.

await _whatsAppPage.EvaluateFunctionAsync($@"() => {
    var observer = new MutationObserver((mutations) => {
        for(var mutation of mutations) {
            if(mutation.addedNodes.length &&
               mutation.addedNodes[0].classList.value === '{WhatsAppMetadata.MessageLine}') {
                newChat(mutation.addedNodes[0].querySelector('.copyable-text span').innerText);
            }
        }
    });
    observer.observe(
        document.querySelector('{WhatsAppMetadata.ChatContainer}'),
        {{ attributes: false, childList: true, subtree: true }});
}");
}

Note: The real code has double curly brackets. I removed them ~~because it breaks jekyll~~ so it's more clear

What we are doing here is observing changes on the childList of our WhatsAppMetadata.ChatContainer element. Inside the observer, we will filter items where the class value is our MessageLine const.
If we have a match, we call newChat sending that innerText to C#.

Type a message when needed.

I found that MarkovSharp could help me build some responses based on a chat export I have.

Setting up MarkupSharp is as easy as this:

var chat = await File.ReadAllLinesAsync(args.SourceText);
_model = new StringMarkov(5);
_model.Learn(chat);

I hope this is not too much for a blog post, but the RespondAsync method would be something like this.

We will get a curated list of words from the message using dotnet-stop-words.

string response = null;
var words = text.RemoveStopWords(args.Language).RemovePunctuation().Replace(args.TriggerWord, string.Empty).Split(' ');

Now we will walk through the list of words in reverse order trying to find a valid message from our Markov model.

The idea here is: If we get a message like "Hey this bot app is awesome". We will get [bot, app, awesome] as valid words, and we will try to make a message based on awesome first and then app and lastly bot.

for (var index = words.Length - 1; index >= 0; index--)
{
    response = _model.Walk(1, words[index]).First();

    if (response == words[index])
    {
        response = null;
    }
}

if (response == null)
{
    response = _model.Walk(1).First();
}

Once we have a "funny" message, we will send that message back.

cs await WriteChatAsync(args.ResponseTemplate + " " + response);

The method would look like this:

cs var chatInput = await _whatsAppPage.QuerySelectorAsync(WhatsAppMetadata.ChatInput); await chatInput.TypeAsync(text); await (await _whatsAppPage.QuerySelectorAsync(WhatsAppMetadata.SendMessageButton)).ClickAsync();

And voilà! We have our Bot!

Wrapping up

I hope you enjoyed reading this tutorial. You will find the repo on Github.
The idea of this post was not only showing off this bot but also presenting some techniques and tools you can use to automate a browser.

Don't stop coding!

Originally posted on harkoded.com

A fairy tale about async voids, events and error handling

Darío Kondratiuk — Thu, 22 Nov 2018 15:31:35 +0000

Let me tell you a story about async voids, SynchronizationContext, and async programming. A few days ago we got an issue on Puppeteer-Sharp describing two problems:

Puppeteer-Sharp crashes with exceptions which cannot be caught.
KeyNotFoundException trying to get a Frame The code was pretty simple:

var launchOptions = new LaunchOptions() { Headless = true };
var sites = new List<string>()
{
    "somesites.com",
}
// Act
try
{
    await new BrowserFetcher().DownloadAsync(BrowserFetcher.DefaultRevision);
    using (var browser = await Puppeteer.LaunchAsync(launchOptions))
    {
        var page = await browser.NewPageAsync();

        foreach (var site in sites)
        {
            try
            {
                await page.GoToAsync($"http://{site}");
                Console.WriteLine(await page.GetTitleAsync());
                await page.ScreenshotAsync($"D:\\bin\\screenshots\\{site}.png");
            }
            catch (Exception exception)
            {
                // Catches most exceptions such as timeouts but does not catch others, see below.
                Console.WriteLine($"Unable to take screenshot of: {site}. Exception: {exception.Message}");
            }
        }
    }
}
catch (Exception exception)
{
    // Never enters the catch.
    Console.WriteLine($"Unable to proceed: {exception.Message}");
}

How is it possible that a try-catch block can't catch an Exception?

Seriously, that's impossible. That's what try-catch blocks are for, right?
Well, Ben Adams gave me a hint here

I think most of us have read this rule at least once:

Avoid async void! It is possible to have an async method return void, but you should only do this if you’re writing an async event handler. A regular async method without a return value should return Task, not void.

Cleary, Stephen. Concurrency in C# Cookbook

But many times I got in a situation where I needed to code an async void event. It looked like a perfect excuse: "I know that async void is bad. But I need to code an async Event handler, so I have to use async voids. I’m following the rules, everything will be ok".

Well… No.

When an async void method propagates an exception, that exception is raised on the SynchronizationContext that was active at the time the async void method started executing. If your execution environment provides a SynchronizationContext, then it usually has a way to handle these top-level exceptions at a global scope. For example, WPF has Application.DispatcherUnhandledException, WinRT has Application.UnhandledException and ASP.NET has Application_Error.

Cleary, Stephen. Concurrency in C# Cookbook

But, as Ben said, in a console app, the exception will be propagated to the ThreadPool without being caught, taking down the entire process.

Where are those async voids?

You might be thinking: "Ok, but why are you talking about async voids? that piece of code has no async void?"
Well, async voids event handlers are being used internally by Puppeteer. So, from Puppeteer.LaunchAsync to the Dispose being called by the using block, many events will be triggered internally.
When a new message comes from Chromium, an IConnectionTransport would parse and then broadcast it using the MessageReceived event.
Many classes, like the Page class, would listen to events coming from the connection and perform asynchronous tasks.
And here we have our async voids.

End of story.

No just kidding we didn't even find the bug yet.

Is async void the problem?

This past week I was obsessed with this async void issue. I considered replacing those events calls with other tools, such us DataFlows or System.Reactive, but I wasn't able to find a clean and (most important) right solution.

As I finished the Concurrency C# cookbook a few weeks I ago, I decided to take my chances and contact Steve Cleary on Twitter. He was super friendly and replied my tweet. To my surprise, in the middle of the conversation, he said:

And I was like

How is it possible that the man who knows most about asynchronous programming tells me that? Isn’t async void the root of all evil?

But that weird question helped me to understand that I wasn't getting what my problem was. Let’s take a look at what happens under the hood when we have a simple piece of code like this one:

try
{
    using (var browser = await Puppeteer.LaunchAsync(options))
    using (var page = await browser.NewPageAsync())
    {
            await Page.ClickAsync("body");
    }
}
catch(Exception ex)
{
    Console.WriteLine(ex.Message);
}

A simplified sequence diagram will look like this.

So let's say that we fail to process Target.targetCreated. What line on the user code will fail? Easy, browser.NewPageAsync
But what if we fail to process Target.targetInfoChanged? There is no way we can send that exception to the user because the user didn't trigger that action.

This was a really ugly problem on Puppeteer-Sharp, not being able to communicate internal errors to the user.

If your library consumes events from another source you need to be extremely careful with your exception handling and design your API, so you are able to communicate those exceptions.

How did I solve this?

Puppeteer-Sharp has two types of connections. The Connection itself, one per process. And a Session connection, one per target.
So, we added a try-catch block on every MessageReceived event we have in the library. If we get any error, we close the connection, and we add that exception as a close reason.

private async void Client_MessageReceived(object sender, MessageEventArgs e)
{
    try
    {
        //Message Processing
    }
    catch (Exception ex)
    {
        var message = $"NetworkManager failed to process {e.MessageID}. {ex.Message}. {ex.StackTrace}";
        _logger.LogError(ex, message);
        _client.Close(message);
    }
}

Next issue.

KeyNotFoundException trying to get a Frame … What?

If you take a look at the sequence diagram again, you will see that the first message we get from Chromium is “Target.targetCreated”. How is it possible that we are getting a KeyNotFoundException exception?

I'll give you a hint, it begins with async and ends with void.

An IConnectionTransport will start receiving a stream of messages from Chromium, parse them and trigger a MessageReceived event.

The problem here is that MessageReceived?.Invoke will trigger and forget an async void event handler. Of Course! there is not await involved! This will be a fire and forget.

Now it makes more sense when you read this on the Using Asynchronous Methods in ASP.NET 4.5 post

The downside to async void events is that developers no longer have full control over when events execute. For example, if both an .aspx and a .Master define Page_Load events, and one or both of them are asynchronous, the order of execution can't be guaranteed. The same indeterminate order for non event handlers (such as async void Button_Click ) applies.

Of course, the order of execution can't be guaranteed, because **the Invoke method won't await your async event handlers.

Wait for frames to be created.

The Frames issue was easy to solve, though quite messy to implement. Basically. It was a matter of replacing all the:

Frames[someFrame];

With

await _frameManager.GetFrameAsync(someFrame);

Final words

Although this might sound too Puppeteer-Sharp specific, I think it gives us, some interesting things to consider when designing a library that consumes events for another source.

Don't stop coding!

Originally posted on harkoded.com

How to organize your git branches

Darío Kondratiuk — Mon, 17 Sep 2018 11:08:35 +0000

I remember implementing git in my team. I think it was more than 6 years ago. At that time, A successful git branching model by Vincent Driessen was required reading if you wanted to learn how to work with git effectively. Since then, I've been following the "Git Flow" style. But, this past year, I've been influenced by many developers from the community, and started using git in different ways depending on the projects I was working on.

Before getting into the different branching styles, it's important to remember that, although many git clients treat slashes ("/") as a directory separator, there is no such thing as folders or directories in the git specification. You will see this implemented in many UI clients, but I never found it on console clients, or on websites like Github or GitLab.

That being said, Let's explore some ways of organizing branches, so you don't get lost in a sea of code.

Gitflow

Although Gitflow doesn't mention branch folders, many devs use "Feature branches", "Hotfix branches" and "Release branches" and create folders accordingly.
So basically, a GitFlow organization would have these three folders:

feature[s]
hotfix[es]/fixes
release[s]

As there is no public document talking about this, I've seen some working copies using those folders in plural and others in singular.

Gitflow with steroids

I found myself adding two more folders to my Gitflow repos:

Docs: For branches related to markdown or release documents.
Task: For branches which are neither features nor fixes. Such as, clean up scripts or refactors.

BPMP (Branch-Push-Merge-Prune)

I’ve seen this a lot in many projects. The branch-push-merge-prune is the anti-folder method. I'm not saying that chaos is a way of organizing branches. People using this method like to have their working copy clean. They would just branch, push, create a pull request and then delete the branch (manually or via git fetch prune) as soon as the PR is merged.

Module-based

I found myself using what I call a "module-based" branching model on a big project where I got quite lost in a sea of features and fixes. So I started to create branches based on its modules.

You could mix Gitflow with this module-based approach, something like "backoffice/billing/fixes/billing-values", but that may be too much.

Version-based

I first saw Meir using this approach and I loved it. It’s great for projects like Puppeteer-sharp where the roadmap is clear.
In a version-based repo you create each branch inside a "vX.X" folder. What is cool about this is that it’s time-based, so it's easier to find branches and also it's super easy to delete old versions with this simple git command:

git branch | grep -e "vX.X/" | xargs git branch -D

Again, this could be mixed with Gitflow folders, but...

Ticket-based

If tickets numbers (tickets, issues or whatever you call them) are part of your team's language using a ticket-based system could be a perfect fit.
You could use a folder, such as "tickets/242" or "issues/242", or just simply call it "242".

Emoji-based

If none of these systems is for you, you can follow Nick's idea and implement an Emoji-based system :)

Final words

Two final thoughts to close this post. First, if you work on a team where you normally checkout each other branches, e.g. for local testing, I'd recommend you share the style with your team.
And finally, clean your working copy frequently. This will help you find your branches quickly, and also speed up your local repo.

Don't stop coding!

Originally posted on harkoded.com